[jira] [Issue Comment Deleted] (HAWQ-999) Treat hash table as random when file count is not in proportion to bucket number of table.
[ https://issues.apache.org/jira/browse/HAWQ-999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang updated HAWQ-999: -- Comment: was deleted (was: [~jianlirong] We need to investigate why bucket number and file count mismatch happens. This JIRA is just to ensure even when mismatch happens, The query will not failed(of course catalog and physical file information must be consistent)) > Treat hash table as random when file count is not in proportion to bucket > number of table. > -- > > Key: HAWQ-999 > URL: https://issues.apache.org/jira/browse/HAWQ-999 > Project: Apache HAWQ > Issue Type: Improvement > Components: Core >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: 2.0.1.0-incubating > > > By definition, file count of a hash table should be equal to or a multiple of > the bucket number of the table. So if mismatch happens, we should not treat > it as hash table in data locality algorithm. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-hawq pull request #811: HAWQ-1000: Set dummy workfile pointer to N...
Github user liming01 closed the pull request at: https://github.com/apache/incubator-hawq/pull/811 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #811: HAWQ-1000: Set dummy workfile pointer to NULL aft...
Github user zhangh43 commented on the issue: https://github.com/apache/incubator-hawq/pull/811 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Closed] (HAWQ-980) hawq does not handle guc value with space properly
[ https://issues.apache.org/jira/browse/HAWQ-980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Guo closed HAWQ-980. - Resolution: Fixed > hawq does not handle guc value with space properly > -- > > Key: HAWQ-980 > URL: https://issues.apache.org/jira/browse/HAWQ-980 > Project: Apache HAWQ > Issue Type: Bug >Reporter: Paul Guo >Assignee: Paul Guo > Fix For: backlog > > > 1. Set guc value with space in "alter database" could lead subsequent query > fails. > e.g. > alter database test_db set datestyle to 'postgres, MDY'; > select * from test_table; > One example of the error output is: > ERROR: Error dispatching to seg0 localhost:4: connection pointer is NULL > > DETAIL: > > Master unable to connect to seg0 localhost:4: FATAL: invalid > > command-line arguments for server process > > HINT: Try "postgres --help" for more information. > 2. Currently hawq dispatch code addOneOption() removes all spaces for guc > value strings it handles. This is wrong since space in guc value could be > meaningful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HAWQ-1001) Implement HAWQ user ACL check through Ranger
Lili Ma created HAWQ-1001: - Summary: Implement HAWQ user ACL check through Ranger Key: HAWQ-1001 URL: https://issues.apache.org/jira/browse/HAWQ-1001 Project: Apache HAWQ Issue Type: Sub-task Components: Core Reporter: Lili Ma Assignee: Lei Chang When a user run some query, HAWQ can connect to Ranger to judge whether the user has the privilege to do that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HAWQ-1002) Implement a switch in hawq-site.xml to configure whether use Ranger or not for ACL
Lili Ma created HAWQ-1002: - Summary: Implement a switch in hawq-site.xml to configure whether use Ranger or not for ACL Key: HAWQ-1002 URL: https://issues.apache.org/jira/browse/HAWQ-1002 Project: Apache HAWQ Issue Type: Sub-task Components: Core Reporter: Lili Ma Assignee: Lei Chang Implement a switch in hawq-site.xml to configure whether use Ranger or not for ACL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1001) Implement HAWQ basic user ACL check through Ranger
[ https://issues.apache.org/jira/browse/HAWQ-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lili Ma updated HAWQ-1001: -- Summary: Implement HAWQ basic user ACL check through Ranger (was: Implement HAWQ user ACL check through Ranger) > Implement HAWQ basic user ACL check through Ranger > -- > > Key: HAWQ-1001 > URL: https://issues.apache.org/jira/browse/HAWQ-1001 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Core >Reporter: Lili Ma >Assignee: Lei Chang > Fix For: backlog > > > When a user run some query, HAWQ can connect to Ranger to judge whether the > user has the privilege to do that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HAWQ-1003) Implement enhanced hawq ACL check through Ranger
Lili Ma created HAWQ-1003: - Summary: Implement enhanced hawq ACL check through Ranger Key: HAWQ-1003 URL: https://issues.apache.org/jira/browse/HAWQ-1003 Project: Apache HAWQ Issue Type: Sub-task Components: Core Reporter: Lili Ma Assignee: Lei Chang Implement enhanced hawq ACL check through Ranger, which means, if a query contains several tables, we can combine the multiple table request together, to send just one REST request to Ranger REST API Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1001) Implement HAWQ basic user ACL check through Ranger
[ https://issues.apache.org/jira/browse/HAWQ-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lili Ma updated HAWQ-1001: -- Assignee: Hubert Zhang (was: Lei Chang) Description: When a user run some query, HAWQ can connect to Ranger to judge whether the user has the privilege to do that. For each object with unique oid, send one request to Ranger was:When a user run some query, HAWQ can connect to Ranger to judge whether the user has the privilege to do that. > Implement HAWQ basic user ACL check through Ranger > -- > > Key: HAWQ-1001 > URL: https://issues.apache.org/jira/browse/HAWQ-1001 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Core >Reporter: Lili Ma >Assignee: Hubert Zhang > Fix For: backlog > > > When a user run some query, HAWQ can connect to Ranger to judge whether the > user has the privilege to do that. > For each object with unique oid, send one request to Ranger -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HAWQ-1004) Decide How HAWQ connect Ranger, through which user, how to connect to REST Server
Lili Ma created HAWQ-1004: - Summary: Decide How HAWQ connect Ranger, through which user, how to connect to REST Server Key: HAWQ-1004 URL: https://issues.apache.org/jira/browse/HAWQ-1004 Project: Apache HAWQ Issue Type: Sub-task Components: Core Reporter: Lili Ma Assignee: Lei Chang Decide How HAWQ connect Ranger, through which user, how to connect to REST Server Acceptance Criteria: Provide an interface for HAWQ connecting Ranger REST Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418598#comment-15418598 ] Hubert Zhang commented on HAWQ-256: --- [~bosco] [~vineetgoel] [~lilima][~wlin] We revisited the HAWQ aclcheck related code, and found that in HAWQ if there is no acl information stored in an object(database, table...) catalog, different objects will have different default behaviours to do acl check. But in our original REST-API design, the checkPrivileges API return bool(allow or deny) which lacks a state of unknown to do default acl check. Here 'unknown' means there is no record in ranger about this request. So we propose two ways to handle this issue. 1 Set the type of return value of checkPrivileges to integer, 0 for deny, 1 for allow, 2 for unknown 2 Add another REST-API, isPrivilegeExist(). Any suggestion for which solution is better ? > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1004) Decide How HAWQ connect Ranger, through which user, how to connect to REST Server
[ https://issues.apache.org/jira/browse/HAWQ-1004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lili Ma updated HAWQ-1004: -- Assignee: Lin Wen (was: Lei Chang) > Decide How HAWQ connect Ranger, through which user, how to connect to REST > Server > - > > Key: HAWQ-1004 > URL: https://issues.apache.org/jira/browse/HAWQ-1004 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Core >Reporter: Lili Ma >Assignee: Lin Wen > Fix For: backlog > > > Decide How HAWQ connect Ranger, through which user, how to connect to REST > Server > Acceptance Criteria: > Provide an interface for HAWQ connecting Ranger REST Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418598#comment-15418598 ] Hubert Zhang edited comment on HAWQ-256 at 8/12/16 9:47 AM: [~bosco] [~vineetgoel] [~lilima][~wlin] We revisited the HAWQ aclcheck related code, and found that in HAWQ support default ack check. When there is no acl information stored in an object(database, table...) catalog, different objects will have different default behaviours to do acl check. But in our original REST-API design, the checkPrivileges API return bool(allow or deny) which lacks a state of unknown to do default acl check. Here 'unknown' means there is no record in ranger about this request. So we propose two ways to handle this issue. 1 Set the type of return value of checkPrivileges to integer, 0 for deny, 1 for allow, 2 for unknown 2 Add another REST-API, isPrivilegeExist(). Any suggestion for which solution is better ? was (Author: hubertzhang): [~bosco] [~vineetgoel] [~lilima][~wlin] We revisited the HAWQ aclcheck related code, and found that in HAWQ if there is no acl information stored in an object(database, table...) catalog, different objects will have different default behaviours to do acl check. But in our original REST-API design, the checkPrivileges API return bool(allow or deny) which lacks a state of unknown to do default acl check. Here 'unknown' means there is no record in ranger about this request. So we propose two ways to handle this issue. 1 Set the type of return value of checkPrivileges to integer, 0 for deny, 1 for allow, 2 for unknown 2 Add another REST-API, isPrivilegeExist(). Any suggestion for which solution is better ? > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418598#comment-15418598 ] Hubert Zhang edited comment on HAWQ-256 at 8/12/16 9:48 AM: [~bosco] [~vineetgoel] [~lilima][~wlin] We revisited the HAWQ aclcheck related code, and found that in HAWQ support default acl check. When there is no acl information stored in an object(database, table...) catalog, different objects will have different default behaviours to do acl check. But in our original REST-API design, the checkPrivileges API return bool(allow or deny) which lacks a state of unknown to do default acl check. Here 'unknown' means there is no record in ranger about this request. So we propose two ways to handle this issue. 1 Set the type of return value of checkPrivileges to integer, 0 for deny, 1 for allow, 2 for unknown 2 Add another REST-API, isPrivilegeExist(). Any suggestion for which solution is better ? was (Author: hubertzhang): [~bosco] [~vineetgoel] [~lilima][~wlin] We revisited the HAWQ aclcheck related code, and found that in HAWQ support default ack check. When there is no acl information stored in an object(database, table...) catalog, different objects will have different default behaviours to do acl check. But in our original REST-API design, the checkPrivileges API return bool(allow or deny) which lacks a state of unknown to do default acl check. Here 'unknown' means there is no record in ranger about this request. So we propose two ways to handle this issue. 1 Set the type of return value of checkPrivileges to integer, 0 for deny, 1 for allow, 2 for unknown 2 Add another REST-API, isPrivilegeExist(). Any suggestion for which solution is better ? > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-743) RPM conflict between apache-tomcat and pxf-service
[
https://issues.apache.org/jira/browse/HAWQ-743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15419410#comment-15419410
]
Goden Yao commented on HAWQ-743:
https://github.com/apache/incubator-hawq/pull/738/
> RPM conflict between apache-tomcat and pxf-service
> --
>
> Key: HAWQ-743
> URL: https://issues.apache.org/jira/browse/HAWQ-743
> Project: Apache HAWQ
> Issue Type: Bug
> Components: Build
>Affects Versions: 2.0.0.0-incubating
>Reporter: Zhanwei Wang
>Assignee: Shivram Mani
> Fix For: 2.0.1.0-incubating
>
>
> {code}
> ==
> Package 架构
> 版本源
> 大小
> ==
> 正在安装:
> pxf-service noarch
> 3.0.0-22126 HDB
> 212 k
> 事务概要
> ==
> 安装 1 软件包
> 总计：212 k
> 安装大小：371 k
> Is this ok [y/d/N]: y
> Downloading packages:
> Running transaction check
> Running transaction test
> Transaction check error:
> file /opt/pivotal from install of pxf-service-0:3.0.0-22126.noarch
> conflicts with file from package apache-tomcat-0:7.0.62-.noarch
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HAWQ-743) RPM conflict between apache-tomcat and pxf-service
[
https://issues.apache.org/jira/browse/HAWQ-743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Goden Yao resolved HAWQ-743.
Resolution: Fixed
> RPM conflict between apache-tomcat and pxf-service
> --
>
> Key: HAWQ-743
> URL: https://issues.apache.org/jira/browse/HAWQ-743
> Project: Apache HAWQ
> Issue Type: Bug
> Components: Build
>Affects Versions: 2.0.0.0-incubating
>Reporter: Zhanwei Wang
>Assignee: Shivram Mani
> Fix For: 2.0.1.0-incubating
>
>
> {code}
> ==
> Package 架构
> 版本源
> 大小
> ==
> 正在安装:
> pxf-service noarch
> 3.0.0-22126 HDB
> 212 k
> 事务概要
> ==
> 安装 1 软件包
> 总计：212 k
> 安装大小：371 k
> Is this ok [y/d/N]: y
> Downloading packages:
> Running transaction check
> Running transaction test
> Transaction check error:
> file /opt/pivotal from install of pxf-service-0:3.0.0-22126.noarch
> conflicts with file from package apache-tomcat-0:7.0.62-.noarch
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15419784#comment-15419784 ] Don Bosco Durai commented on HAWQ-256: -- + [~madhan.neethiraj] In the Ranger case, this might not be feasible. Because we go by "permissive" model, which means if there are no permissions, then it is "deny". So by absence of a permission, it will be difficult to determine "no privilege" v/s "deny". My suggestion would be for admins who want to set the default behaviors, can do it in Ranger itself. E.g. They can pick the resources (Database, table,etc) and give the desired permission to group "public". Which means, all users will at least get the permissions set in this policy. And they can have different defaults for different resources. It will be easy to manage these centrally, than trying to set the defaults in other config files or mechanism. This might be a better option, because now, the policies (including defaults) are in one place and it is easy to audit who set the default policies and how any end user got access to the resource. Would this be okay for the users? > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-hawq pull request #846: Hawq 991. Add support for "HAWQ register" ...
GitHub user xunzhang opened a pull request: https://github.com/apache/incubator-hawq/pull/846 Hawq 991. Add support for "HAWQ register" that could register tables by using "hawq extract" output. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xunzhang/incubator-hawq HAWQ-991 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/846.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #846 commit 7ed40d3e6924e606322276c0b61fb417e47ba19a Author: xunzhang Date: 2016-08-09T11:39:13Z HAWQ-991. Refactor hawqregister. commit b2e3679c55260fd8931ec2bcb4f834a4f5e6c931 Author: xunzhang Date: 2016-08-13T02:38:15Z HAWQ-991. Add schema info with Parquet format in hawqextract. commit cd7b71a0e59b01ac181bfbbd8fa4c6b14528ea80 Author: xunzhang Date: 2016-08-13T06:23:41Z HAWQ-991. Rewrite hawqregister to support registering from yaml file with parquet format. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (HAWQ-991) Add support for "HAWQ register" that could register tables by using "hawq extract" output
[ https://issues.apache.org/jira/browse/HAWQ-991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongwu updated HAWQ-991: Issue Type: New Feature (was: Improvement) > Add support for "HAWQ register" that could register tables by using "hawq > extract" output > - > > Key: HAWQ-991 > URL: https://issues.apache.org/jira/browse/HAWQ-991 > Project: Apache HAWQ > Issue Type: New Feature > Components: Command Line Tools, External Tables >Affects Versions: 2.0.1.0-incubating >Reporter: hongwu >Assignee: hongwu > Fix For: 2.0.1.0-incubating > > > User should be able to use HAWQ Register utility to register HAWQ table > files/directories into a new HAWQ cluster so that the data can be copied from > one cluster to another, and the HAWQ catalog metadata is synchronized with > these HDFS HAWQ files. > The ask for this feature is basically to pass `hawq register` an input file > (or set of files) containing the last-known-good metadata that it can use to > update the portion of the catalog managing HDFS blocks. Prior to every new > data load, the user can leverage the `hawq extract` command to snapshot the > metadata for every table to protect against corruption / divergence. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
