[jira] [Assigned] (HAWQ-1583) Add vectorized executor extension and GNC

2018-01-23 Thread WANG Weinan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HAWQ-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

WANG Weinan reassigned HAWQ-1583:
-

Assignee: WANG Weinan  (was: Lei Chang)

> Add vectorized executor extension and GNC
> -
>
> Key: HAWQ-1583
> URL: https://issues.apache.org/jira/browse/HAWQ-1583
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Query Execution
>Reporter: Hongxu Ma
>Assignee: WANG Weinan
>Priority: Major
> Fix For: backlog
>
>
> The vectorized executor will be implemented as a extension (located at 
> contrib directory).
> And use a GNC to enable vectorized executor, e.g:
> {code:java}
> postgres=# set vectorized_executor_enable to on;
> // run the new vectorized executor
> postgres=# set vectorized_executor_enable to off;
> // run the original HAWQ executor
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HAWQ-1583) Add vectorized executor extension and GNC

2018-01-23 Thread Hongxu Ma (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HAWQ-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hongxu Ma updated HAWQ-1583:

Description: 
The vectorized executor will be implemented as a extension (located at contrib 
directory).

And use a GNC to enable vectorized executor, e.g:
{code:java}
postgres=# set vectorized_executor_enable to on;

// run the new vectorized executor

postgres=# set vectorized_executor_enable to off;

// run the original HAWQ executor
{code}

  was:
The vectorized executor will be implemented as a extension (located at contrib 
directory).

And use a GNC to enable vectorized executor, e.g:

{code}

postgres=# set vectorized_executor_enable to on;

// run the new vectorized executor

postgres=# set vectorized_executor_enable to off;

// run the original HAWQ executor


{code}


> Add vectorized executor extension and GNC
> -
>
> Key: HAWQ-1583
> URL: https://issues.apache.org/jira/browse/HAWQ-1583
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Query Execution
>Reporter: Hongxu Ma
>Assignee: Lei Chang
>Priority: Major
> Fix For: backlog
>
>
> The vectorized executor will be implemented as a extension (located at 
> contrib directory).
> And use a GNC to enable vectorized executor, e.g:
> {code:java}
> postgres=# set vectorized_executor_enable to on;
> // run the new vectorized executor
> postgres=# set vectorized_executor_enable to off;
> // run the original HAWQ executor
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HAWQ-1583) Add vectorized executor extension and GNC

2018-01-23 Thread Hongxu Ma (JIRA) 
Hongxu Ma created HAWQ-1583:
---

 Summary: Add vectorized executor extension and GNC
 Key: HAWQ-1583
 URL: https://issues.apache.org/jira/browse/HAWQ-1583
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Query Execution
Reporter: Hongxu Ma
Assignee: Lei Chang
 Fix For: backlog


The vectorized executor will be implemented as a extension (located at contrib 
directory).

And use a GNC to enable vectorized executor, e.g:

{code}

postgres=# set vectorized_executor_enable to on;

// run the new vectorized executor

postgres=# set vectorized_executor_enable to off;

// run the original HAWQ executor


{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] incubator-hawq issue #1333: HAWQ-1582. hawq ssh cmd bug when pipe in cmd

2018-01-23 Thread yaoj2 
Github user yaoj2 commented on the issue:

https://github.com/apache/incubator-hawq/pull/1333
  
The change looks good to me. 


---

[GitHub] incubator-hawq pull request #1333: HAWQ-1582. hawq ssh cmd bug when pipe in ...

2018-01-23 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/incubator-hawq/pull/1333


---

[GitHub] incubator-hawq issue #1333: HAWQ-1582. hawq ssh cmd bug when pipe in cmd

2018-01-23 Thread wcl14 
Github user wcl14 commented on the issue:

https://github.com/apache/incubator-hawq/pull/1333
  
+1


---

[GitHub] incubator-hawq issue #1333: HAWQ-1582. hawq ssh cmd bug when pipe in cmd

2018-01-23 Thread stanlyxiang 
Github user stanlyxiang commented on the issue:

https://github.com/apache/incubator-hawq/pull/1333
  
+1


---

[GitHub] incubator-hawq pull request #1333: HAWQ-1582. hawq ssh cmd bug when pipe in ...

2018-01-23 Thread YoungForest 
GitHub user YoungForest opened a pull request:

https://github.com/apache/incubator-hawq/pull/1333

HAWQ-1582. hawq ssh cmd bug when pipe in cmd

# Bug description
[HAWQ-1582](https://issues.apache.org/jira/browse/HAWQ-1582)

# Bug reason

https://github.com/apache/incubator-hawq/blob/037be68ffacdac09dbecef22951ffa97bd9acb4a/tools/bin/hawq#L94
Joining all arguments with space raises this bug when there is space in a 
argument. For example, argument`'ls -1 | wc -l'` becomes several arguments 
`'ls', '-1', '|', 'wc', '-l'`, which are feed into `subprocess.Popen`

https://github.com/apache/incubator-hawq/blob/037be68ffacdac09dbecef22951ffa97bd9acb4a/tools/bin/hawq#L41
For example, when executing `hawk ssh -h localhost -e 'ls -1 | wc -l'`, the 
`gpssh -h localhost -e ls -1 | wc -l` is feed into `subprocess.Popen` as cmd, 
while cmd is expected to be `gpssh -h localhost -e 'ls -1 | wc -l'`.

# How I fix it

[commit](https://github.com/YoungForest/incubator-hawq/commit/9ce5be82dcab0c53ab848178e1b4d8ec99a91a2e)
Not just join arguments with space simply, while keep its integrity when 
the argument has space in it.

Fix result:
``` bash
hawq ssh -h sdw2 -h localhost -e 'ls -1 | wc -l'
``` 
gets expected result.

``` bash
hawq ssh -h sdw2 -h localhost -e 'kill -9 \$(pgrep lava)'
```
gets expected result too after adding escape before `$`.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/YoungForest/incubator-hawq HAWQ-1582

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/1333.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1333


commit 9ce5be82dcab0c53ab848178e1b4d8ec99a91a2e
Author: YoungForest 
Date:   2018-01-24T02:46:18Z

HAWQ-1582. hawq ssh cmd bug when pipe in cmd




---

[GitHub] incubator-hawq pull request #1325: HAWQ-1125. Running pl/python related feat...

2018-01-23 Thread kuien 
Github user kuien closed the pull request at:

https://github.com/apache/incubator-hawq/pull/1325


---

[GitHub] incubator-hawq issue #1325: HAWQ-1125. Running pl/python related feature_tes...

2018-01-23 Thread kuien 
Github user kuien commented on the issue:

https://github.com/apache/incubator-hawq/pull/1325
  
Thanks @radarwave


---

[GitHub] incubator-hawq pull request #1324: HAWQ-1573. Clear debug_query_string in pr...

2018-01-23 Thread kuien 
Github user kuien closed the pull request at:

https://github.com/apache/incubator-hawq/pull/1324


---

[jira] [Created] (HAWQ-1582) hawq ssh cmd bug when pipe in cmd

2018-01-23 Thread Yang Sen (JIRA) 
Yang Sen created HAWQ-1582:
--

 Summary: hawq ssh cmd bug when pipe in cmd
 Key: HAWQ-1582
 URL: https://issues.apache.org/jira/browse/HAWQ-1582
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Command Line Tools
Reporter: Yang Sen
Assignee: Radar Lei
 Fix For: 2.3.0.0-incubating


h1. bug description
{code:bash}
hawq ssh -h sdw2 -h localhost -e 'ls -1 | wc -l'
{code}
When running this command, the expected action is that `ls -1 | wc -l` is 
executed in each host. The expected output is (the number may be different):
{code:bash}
[sdw2] ls -1 | wc -l
[sdw2] 23
[localhost] ls -1 | wc -l
[localhost] 20
{code}

While the output got is:
{code:bash}
45
{code}
The result looks like `ls -l` was executed in each host and the output of `hawq 
ssh -h sdw2 -h localhost -e 'ls -1'` was redirect to pipe to `wc -l`.

h2. Another related issue
{code:bash}
hawq ssh -h sdw2 -h localhost -e 'kill -9 $(pgrep lava)'
{code}
This command expects to kill process named lava in each host. While `$(pgrep 
lava)` is executed in localhost, and program gets the process id, for example 
5. And then `kill -9 5` is executed in each host, which is definitely 
not match with our expect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] incubator-hawq pull request #1332: HAWQ-1581. Separate PXF system parameters...

2018-01-23 Thread denalex 
Github user denalex commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1332#discussion_r163433442
  
--- Diff: 
pxf/pxf-api/src/test/java/org/apache/hawq/pxf/api/utilities/ProfilesConfTest.java
 ---
@@ -96,16 +96,18 @@ public void definedProfile() throws Exception {
 optionalFile.toURI().toURL());
 
 Map hbaseProfile = 
ProfilesConf.getProfilePluginsMap("HBase");
+System.out.println(hbaseProfile);
+System.out.println(hbaseProfile.get("X-GP-OPTIONS-PLUGIN1"));
--- End diff --

did you intend to have System.out here ?


---

[GitHub] incubator-hawq pull request #1332: HAWQ-1581. Separate PXF system parameters...

2018-01-23 Thread denalex 
Github user denalex commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1332#discussion_r163433718
  
--- Diff: 
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/JdbcPlugin.java ---
@@ -58,8 +58,8 @@ public JdbcPlugin(InputData input) throws 
UserDataException {
 super(input);
 jdbcDriver = input.getUserProperty("JDBC_DRIVER");
 dbUrl = input.getUserProperty("DB_URL");
-user = input.getUserProperty("USER");
-pass = input.getUserProperty("PASS");
+user = input.getUserProperty("DB_USER");
+pass = input.getUserProperty("DB_PASS");
--- End diff --

no longer necessary to change these property names


---

[GitHub] incubator-hawq pull request #1332: HAWQ-1581. Separate PXF system parameters...

2018-01-23 Thread denalex 
Github user denalex commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1332#discussion_r163434200
  
--- Diff: 
pxf/pxf-service/src/main/java/org/apache/hawq/pxf/service/utilities/ProtocolData.java
 ---
@@ -82,19 +83,26 @@ public ProtocolData(Map paramsMap) {
 parseTupleDescription();
 
 /*
- * accessor - will throw exception from getPropery() if 
outputFormat is
+ * accessor - will throw exception if outputFormat is
  * BINARY and the user did not supply accessor=... or profile=...
- * resolver - will throw exception from getPropery() if 
outputFormat is
+ * resolver - will throw exception if outputFormat is
  * BINARY and the user did not supply resolver=... or profile=...
  */
-profile = getOptionalProperty("PROFILE");
+profile = getUserProperty("PROFILE");
--- End diff --

should we preserve semantic of getUserProperty() that throws exception if 
the property is not present and getUserOptionalProperty() that returns null ?


---

[GitHub] incubator-hawq issue #1332: HAWQ-1581. Separate PXF system parameters from u...

2018-01-23 Thread lavjain 
Github user lavjain commented on the issue:

https://github.com/apache/incubator-hawq/pull/1332
  
pxf/pxf-service/src/scripts/pxf-service should not be part of this commit.
pxf/pxf-jdbc/README.md needs to be updated


---

[jira] [Assigned] (HAWQ-1581) Separate PXF system parameters from user configurable visible parameters

2018-01-23 Thread Shivram Mani (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HAWQ-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shivram Mani reassigned HAWQ-1581:
--

Assignee: Shivram Mani  (was: Ed Espino)

> Separate PXF system parameters from user configurable visible parameters
> 
>
> Key: HAWQ-1581
> URL: https://issues.apache.org/jira/browse/HAWQ-1581
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: PXF
>Reporter: Shivram Mani
>Assignee: Shivram Mani
>Priority: Major
>
> We need to modify our system such that user configurable options are kept 
> distinct form the internal parameters. The custom parameters are configured 
> in the {{LOCATION}} section of the external table DDL, is exposed to PXF 
> server as {{X-GP-}}.
> {{X-GP-USER}} is an internal parameter used to set the user information. When 
> the DDL has a custom parameter named {{user}} it ends up updating X-GP-USER 
> to also include the user configured in the DDL Location. This causes the JDBC 
> connector to fail.
> We will instead use \{X-GP-OPTIONS-} as the prefix for all user configurable 
> parameters to keep them isolated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] incubator-hawq pull request #1332: HAWQ-1581. Separate PXF system parameters...

2018-01-23 Thread shivzone 
GitHub user shivzone opened a pull request:

https://github.com/apache/incubator-hawq/pull/1332

HAWQ-1581. Separate PXF system parameters from user parameters



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shivzone/incubator-hawq HAWQ-1581

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/1332.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1332


commit f949858090dd5c7848df79434be5301f3c193489
Author: shivzone 
Date:   2018-01-24T00:29:41Z

HAWQ-1581. Separate PXF system parameters from user configurable visible 
parameters




---

[jira] [Created] (HAWQ-1581) Separate PXF system parameters from user configurable visible parameters

2018-01-23 Thread Shivram Mani (JIRA) 
Shivram Mani created HAWQ-1581:
--

 Summary: Separate PXF system parameters from user configurable 
visible parameters
 Key: HAWQ-1581
 URL: https://issues.apache.org/jira/browse/HAWQ-1581
 Project: Apache HAWQ
  Issue Type: Bug
  Components: PXF
Reporter: Shivram Mani
Assignee: Ed Espino


We need to modify our system such that user configurable options are kept 
distinct form the internal parameters. The custom parameters are configured in 
the {{LOCATION}} section of the external table DDL, is exposed to PXF server as 
{{X-GP-}}.

{{X-GP-USER}} is an internal parameter used to set the user information. When 
the DDL has a custom parameter named {{user}} it ends up updating X-GP-USER to 
also include the user configured in the DDL Location. This causes the JDBC 
connector to fail.

We will instead use \{X-GP-OPTIONS-} as the prefix for all user configurable 
parameters to keep them isolated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)