[GitHub] incubator-hawq pull request #1218: HAWQ-1431: Do not use StatsAccessor when ...

[sansanichfb]

GitHub user sansanichfb opened a pull request:

https://github.com/apache/incubator-hawq/pull/1218

HAWQ-1431: Do not use StatsAccessor when column specified in SELECT c…

…lause.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sansanichfb/incubator-hawq HAWQ-1431

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/1218.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1218


commit e864611591e4d2d5f6df238641cc04bd088d3d8c
Author: Oleksandr Diachenko 
Date:   2017-04-13T02:00:54Z

HAWQ-1431: Do not use StatsAccessor when column specified in SELECT clause.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1213: HAWQ-1426. Fix bug that hawq extract meet...

[wcl14]

Github user wcl14 closed the pull request at:

https://github.com/apache/incubator-hawq/pull/1213


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Issue Comment Deleted] (HAWQ-1432) document HAWQ's directory structure in HDFS

[Lisa Owen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lisa Owen updated HAWQ-1432:

Comment: was deleted

(was: thanks for the pointer, jon.  there is some great info in your response to
that question!


)

> document HAWQ's directory structure in HDFS
> ---
>
> Key: HAWQ-1432
> URL: https://issues.apache.org/jira/browse/HAWQ-1432
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Lisa Owen
>Assignee: David Yozie
>Priority: Minor
>
> add some documentation describing the HAWQ directory structure in HDFS and 
> how to locate the files for a specific table.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HAWQ-1432) document HAWQ's directory structure in HDFS

[Lisa Owen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lisa Owen updated HAWQ-1432:


thanks for the pointer, jon.  there is some great info in your response to
that question!




> document HAWQ's directory structure in HDFS
> ---
>
> Key: HAWQ-1432
> URL: https://issues.apache.org/jira/browse/HAWQ-1432
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Lisa Owen
>Assignee: David Yozie
>Priority: Minor
>
> add some documentation describing the HAWQ directory structure in HDFS and 
> how to locate the files for a specific table.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HAWQ-1432) document HAWQ's directory structure in HDFS

[Lisa Owen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966935#comment-15966935
 ] 

Lisa Owen commented on HAWQ-1432:
-

thanks for the pointer, jon.  i had come across that post previously.  

there is some great info in your response to that question!

> document HAWQ's directory structure in HDFS
> ---
>
> Key: HAWQ-1432
> URL: https://issues.apache.org/jira/browse/HAWQ-1432
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Lisa Owen
>Assignee: David Yozie
>Priority: Minor
>
> add some documentation describing the HAWQ directory structure in HDFS and 
> how to locate the files for a specific table.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HAWQ-1432) document HAWQ's directory structure in HDFS

[Jon Roberts (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966931#comment-15966931
 ] 

Jon Roberts commented on HAWQ-1432:
---

This might help get you started: 
http://stackoverflow.com/questions/42716111/where-can-i-find-the-location-of-distributed-file-on-slaves-using-apache-hawq


> document HAWQ's directory structure in HDFS
> ---
>
> Key: HAWQ-1432
> URL: https://issues.apache.org/jira/browse/HAWQ-1432
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Lisa Owen
>Assignee: David Yozie
>Priority: Minor
>
> add some documentation describing the HAWQ directory structure in HDFS and 
> how to locate the files for a specific table.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HAWQ-1432) document HAWQ's directory structure in HDFS

[Lisa Owen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lisa Owen updated HAWQ-1432:

Priority: Minor  (was: Major)

> document HAWQ's directory structure in HDFS
> ---
>
> Key: HAWQ-1432
> URL: https://issues.apache.org/jira/browse/HAWQ-1432
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Documentation
>Reporter: Lisa Owen
>Assignee: David Yozie
>Priority: Minor
>
> add some documentation describing the HAWQ directory structure in HDFS and 
> how to locate the files for a specific table.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HAWQ-1432) document HAWQ's directory structure in HDFS

[Lisa Owen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lisa Owen updated HAWQ-1432:

Issue Type: Improvement  (was: Bug)

> document HAWQ's directory structure in HDFS
> ---
>
> Key: HAWQ-1432
> URL: https://issues.apache.org/jira/browse/HAWQ-1432
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Lisa Owen
>Assignee: David Yozie
>Priority: Minor
>
> add some documentation describing the HAWQ directory structure in HDFS and 
> how to locate the files for a specific table.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1432) document HAWQ's directory structure in HDFS

[Lisa Owen (JIRA)]

Lisa Owen created HAWQ-1432:
---

 Summary: document HAWQ's directory structure in HDFS
 Key: HAWQ-1432
 URL: https://issues.apache.org/jira/browse/HAWQ-1432
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Documentation
Reporter: Lisa Owen
Assignee: David Yozie


add some documentation describing the HAWQ directory structure in HDFS and how 
to locate the files for a specific table.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HAWQ-1431) Do not use StatsAccessor when column specified in SELECT clause

[Oleksandr Diachenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleksandr Diachenko reassigned HAWQ-1431:
-

Assignee: Oleksandr Diachenko  (was: Ed Espino)

> Do not use StatsAccessor when column specified in SELECT clause
> ---
>
> Key: HAWQ-1431
> URL: https://issues.apache.org/jira/browse/HAWQ-1431
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
>
> StatsAccessor should be used only for queries like COUNT(*) and not 
> COUNT(column) because we use file-level statistics, which has a total number 
> of rows, not just number of not-null values.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1431) Do not use StatsAccessor when column specified in SELECT clause

[Oleksandr Diachenko (JIRA)]

Oleksandr Diachenko created HAWQ-1431:
-

 Summary: Do not use StatsAccessor when column specified in SELECT 
clause
 Key: HAWQ-1431
 URL: https://issues.apache.org/jira/browse/HAWQ-1431
 Project: Apache HAWQ
  Issue Type: Bug
  Components: PXF
Reporter: Oleksandr Diachenko
Assignee: Ed Espino


StatsAccessor should be used only for queries like COUNT(*) and not 
COUNT(column) because we use file-level statistics, which has a total number of 
rows, not just number of not-null values.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] incubator-hawq pull request #1216: HAWQ-1429. Do not use AggBridge when WHER...

[sansanichfb]

Github user sansanichfb closed the pull request at:

https://github.com/apache/incubator-hawq/pull/1216


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (HAWQ-1429) Do not use AggBridge when WHERE clause specified

[Oleksandr Diachenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleksandr Diachenko resolved HAWQ-1429.
---
Resolution: Fixed

> Do not use AggBridge when WHERE clause specified
> 
>
> Key: HAWQ-1429
> URL: https://issues.apache.org/jira/browse/HAWQ-1429
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: PXF
>Reporter: Oleksandr Diachenko
>Assignee: Ed Espino
> Fix For: 2.3.0.0-incubating
>
>
> Reduce the scope of AggBridge to the case when WHERE clause is not specified.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] incubator-hawq pull request #1216: HAWQ-1429. Do not use AggBridge when WHER...

[shivzone]

Github user shivzone commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1216#discussion_r111259106
  
--- Diff: 
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/utilities/Utilities.java ---
@@ -209,7 +209,8 @@ public static boolean useAggBridge(InputData inputData) 
{
 LOG.error("Unable to load accessor class: " + e.getMessage());
 return false;
 }
-return (inputData != null) && (inputData.getAggType() != null)
+return (inputData != null) && !inputData.hasFilter()
--- End diff --

Add an additional check to filter out queries that have aggregate that is 
not count(*)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (HAWQ-2) Build failed due to file missing during initial upload

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966589#comment-15966589
 ] 

ASF GitHub Bot commented on HAWQ-2:
---

GitHub user njayaram2 opened a pull request:

https://github.com/apache/incubator-madlib/pull/114

Bugfix: Install check failure on gpdb5 and HAWQ-2.2

MADLIB-1088

- Fixes concurrent delete issue with GPDB 5 on install check. This
also fixes the elastic net failure on cross validation, whose root
cause was drop and create table within the same query string.
- Fixes elastic net failure with IGD optimizer. Accessing warmup
lambdas was incorrect.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/njayaram2/incubator-madlib 
bugfix/elastic_net/igd

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-madlib/pull/114.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #114


commit 8430111ae2ac7b7d2c53f5a178d920b3afb74856
Author: Nandish Jayaram 
Date:   2017-04-11T22:45:33Z

potential bug fix in elastic net igd

commit 0e86538f25e96afed283f635627cbd267fd9450e
Author: Nandish Jayaram 
Date:   2017-04-12T20:43:26Z

Bugfix: Install check for elastic net fails on gpdb5

MADLIB-1088

- Fixes concurrent delete issue with GPDB 5 on install check. This
also fixes the elastic net failure on cross validation, whose root
cause was drop and create table within the same query string.
- Fixes elastic net failure with IGD optimizer. Accessing warmup
lambdas was incorrect.




> Build failed due to file missing during initial upload
> --
>
> Key: HAWQ-2
> URL: https://issues.apache.org/jira/browse/HAWQ-2
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Build
>Reporter: Lei Chang
>Assignee: Lei Chang
> Fix For: 2.0.0.0-incubating
>
>
> Detail info on link: https://hub.docker.com/r/mayjojo/hawq-dev/
> Errors reported when doing "make install":
> cd /hawq/tools/bin/pythonSrc/PyGreSQL-4.0 && CC="gcc" 
> CFLAGS="-I/opt/hawq-dep/include -I/opt/hawq-dep/python-2.6.2/include 
> -std=gnu99 -Wall -Wmissing-prototypes -Wpointer-arith -Wendif-labels 
> -Wformat-security -fno-strict-aliasing -fwrapv -I/opt/hawq-dep/include" 
> LDFLAGS="-L../../src/interfaces/libpq -L../../src/port -L../../src/port 
> -L/opt/hawq-dep/lib -L/opt/hawq-dep/lib64 -L/opt/hawq-dep/python-2.6.2/lib 
> -Wl,--as-needed -Wl,-rpath,'/opt/hawq/lib',--enable-new-dtags" python 
> setup.py build
> /bin/sh: line 0: cd: /hawq/tools/bin/pythonSrc/PyGreSQL-4.0: No such file or 
> directory



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] incubator-hawq pull request #1216: HAWQ-1429. Do not use AggBridge when WHER...

[sansanichfb]

Github user sansanichfb commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1216#discussion_r111216993
  
--- Diff: 
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/utilities/Utilities.java ---
@@ -209,7 +209,8 @@ public static boolean useAggBridge(InputData inputData) 
{
 LOG.error("Unable to load accessor class: " + e.getMessage());
 return false;
 }
-return (inputData != null) && (inputData.getAggType() != null)
+return (inputData != null) && !inputData.hasFilter()
--- End diff --

Sure, added



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1216: HAWQ-1429. Do not use AggBridge when WHER...

[kavinderd]

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1216#discussion_r90746
  
--- Diff: 
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/utilities/Utilities.java ---
@@ -209,7 +209,8 @@ public static boolean useAggBridge(InputData inputData) 
{
 LOG.error("Unable to load accessor class: " + e.getMessage());
 return false;
 }
-return (inputData != null) && (inputData.getAggType() != null)
+return (inputData != null) && !inputData.hasFilter()
--- End diff --

Maybe add a comment for why so many checks are required


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #1217: HAWQ-1430. Update ranger related log level to av...

[interma]

Github user interma commented on the issue:

https://github.com/apache/incubator-hawq/pull/1217
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1217: HAWQ-1430. Update ranger related log leve...

[stanlyxiang]

GitHub user stanlyxiang opened a pull request:

https://github.com/apache/incubator-hawq/pull/1217

HAWQ-1430. Update ranger related log level to avoid log flood

@interma @zhangh43 @ictmalili please help to review. thanks. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/stanlyxiang/incubator-hawq HAWQ-1430

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/1217.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1217


commit 2c763e2ba23a78b4ab1d9d0516b51543f1b93d8f
Author: stanlyxiang 
Date:   2017-04-12T09:17:53Z

HAWQ-1430. Update ranger related log level to avoid log flood




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (HAWQ-1430) Update ranger related log level to avoid log flood

[Xiang Sheng (JIRA)]

Xiang Sheng created HAWQ-1430:
-

 Summary: Update ranger related log level to avoid log flood
 Key: HAWQ-1430
 URL: https://issues.apache.org/jira/browse/HAWQ-1430
 Project: Apache HAWQ
  Issue Type: Improvement
  Components: Security
Reporter: Xiang Sheng
Assignee: Ed Espino
 Fix For: 2.3.0.0-incubating


I've been seeing following log flooding during various tests.

2017-04-10 07:31:36.570384 
PDT,"hawqsuperuser","olap_window",p28236,th-972322528,,"61316",2017-04-10 
07:31:36 PDT,0,con1311,,seg-1,"LOG","0","acl check type is 
standalone, the acl type value is 0.",,,0,,"postgres.c",4424,

Since this log level is LOG, and the rangerrest curl is session level, so we 
will check this for each session. This will cause a log flood under high 
concurrency. And this is not that necessary  to log with LOG level. So we 
should update it.  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HAWQ-1430) Update ranger related log level to avoid log flood

[Xiang Sheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiang Sheng reassigned HAWQ-1430:
-

Assignee: Xiang Sheng  (was: Ed Espino)

> Update ranger related log level to avoid log flood
> --
>
> Key: HAWQ-1430
> URL: https://issues.apache.org/jira/browse/HAWQ-1430
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Security
>Reporter: Xiang Sheng
>Assignee: Xiang Sheng
> Fix For: 2.3.0.0-incubating
>
>
> I've been seeing following log flooding during various tests.
> 2017-04-10 07:31:36.570384 
> PDT,"hawqsuperuser","olap_window",p28236,th-972322528,,"61316",2017-04-10 
> 07:31:36 PDT,0,con1311,,seg-1,"LOG","0","acl check type is 
> standalone, the acl type value is 0.",,,0,,"postgres.c",4424,
> Since this log level is LOG, and the rangerrest curl is session level, so we 
> will check this for each session. This will cause a log flood under high 
> concurrency. And this is not that necessary  to log with LOG level. So we 
> should update it.  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)