[jira] [Updated] (HAWQ-999) Log_error when file count is not in proportion to bucket number of hash table.
[ https://issues.apache.org/jira/browse/HAWQ-999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang updated HAWQ-999: -- Description: By definition, file count of a hash table should be equal to or a multiple of the bucket number of the table. So if mismatch happens, we should not treat it as hash table in data locality algorithm. The default behaviour should LOG_ERROR and tell the user "file count is not in proportion to bucket number for hash table with oid=X. Some data maybe lost, if you still want to continue the query, set GUC assign_randomly_for_mismatched_hash_table to true and try again." If GUC assign_randomly_for_mismatched_hash_table is on, the behaviour changed to treat the hash table as random table in data locality algorithm. was: By definition, file count of a hash table should be equal to or a multiple of the bucket number of the table. So if mismatch happens, we should not treat it as hash table in data locality algorithm. The default behaviour should LOG_ERROR and tell the user "" > Log_error when file count is not in proportion to bucket number of hash table. > -- > > Key: HAWQ-999 > URL: https://issues.apache.org/jira/browse/HAWQ-999 > Project: Apache HAWQ > Issue Type: Improvement > Components: Core >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: 2.0.1.0-incubating > > > By definition, file count of a hash table should be equal to or a multiple of > the bucket number of the table. So if mismatch happens, we should not treat > it as hash table in data locality algorithm. > The default behaviour should LOG_ERROR and tell the user "file count is not > in proportion to bucket number for hash table with oid=X. Some data maybe > lost, if you still want to continue the query, set GUC > assign_randomly_for_mismatched_hash_table to true and try again." > If GUC assign_randomly_for_mismatched_hash_table is on, the behaviour changed > to treat the hash table as random table in data locality algorithm. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-999) Log_error when file count is not in proportion to bucket number of hash table.
[ https://issues.apache.org/jira/browse/HAWQ-999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang updated HAWQ-999: -- Description: By definition, file count of a hash table should be equal to or a multiple of the bucket number of the table. So if mismatch happens, we should not treat it as hash table in data locality algorithm. The default behaviour should LOG_ERROR and tell the user "" was: By definition, file count of a hash table should be equal to or a multiple of the bucket number of the table. So if mismatch happens, we should not treat it as hash table in data locality algorithm. > Log_error when file count is not in proportion to bucket number of hash table. > -- > > Key: HAWQ-999 > URL: https://issues.apache.org/jira/browse/HAWQ-999 > Project: Apache HAWQ > Issue Type: Improvement > Components: Core >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: 2.0.1.0-incubating > > > By definition, file count of a hash table should be equal to or a multiple of > the bucket number of the table. So if mismatch happens, we should not treat > it as hash table in data locality algorithm. > The default behaviour should LOG_ERROR and tell the user "" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-999) Log_error when file count is not in proportion to bucket number of hash table.
[ https://issues.apache.org/jira/browse/HAWQ-999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang updated HAWQ-999: -- Summary: Log_error when file count is not in proportion to bucket number of hash table. (was: Treat hash table as random when file count is not in proportion to bucket number of table.) > Log_error when file count is not in proportion to bucket number of hash table. > -- > > Key: HAWQ-999 > URL: https://issues.apache.org/jira/browse/HAWQ-999 > Project: Apache HAWQ > Issue Type: Improvement > Components: Core >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: 2.0.1.0-incubating > > > By definition, file count of a hash table should be equal to or a multiple of > the bucket number of the table. So if mismatch happens, we should not treat > it as hash table in data locality algorithm. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420594#comment-15420594 ] Lili Ma commented on HAWQ-256: -- [~bosco] [~vVineet], Could you help confirm whether these APIs definitions are OK? Thanks :) > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HAWQ-256) Integrate Security with Apache Ranger
[
https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420591#comment-15420591
]
Hubert Zhang edited comment on HAWQ-256 at 8/15/16 3:42 AM:
HAWQ support two kinds of ACL check when there are more than one privilege in
one request.
One is ALL, which means that all the privileges are required to be allowed.
The other one is ANY, which means that one of the required privileges to be
allowed is sufficient.
{code}
{
“requestor” : “u1”,
[
{
"resource" : {“TABLE”: “t1”, “DATABASE”: “db1”},
"privilege" : ["select", "insert"]
"how": ALL
},
{
"resource" : {“TABLE”: “t2”, “DATABASE”: “db1”},
"privilege" : ["select"]
"how": ANY
}
]
}
{code}
was (Author: hubertzhang):
HAWQ support two kinds of ACL check when there are more than one privilege in
one request.
One is ALL, which means that all the privileges are required to be allowed.
The other one is ANY, which means that one of the required privileges to be
allowed is sufficient.
{code}
{
“requestor” : “u1”,
[
{
"resource" : {“TABLE”: “t1”, “DATABASE”: “db1”},
"privilege" : [“select”, "insert"]
"how": ALL
},
{
“resource” : {“TABLE”: “t2”, “DATABASE”: “db1”},
“privilege” : [“select”]
"how": ANY
}
]
}
{code}
> Integrate Security with Apache Ranger
> -
>
> Key: HAWQ-256
> URL: https://issues.apache.org/jira/browse/HAWQ-256
> Project: Apache HAWQ
> Issue Type: New Feature
> Components: PXF, Security
>Reporter: Michael Andre Pearce (IG)
>Assignee: Lili Ma
> Fix For: backlog
>
> Attachments: HAWQRangerSupportDesign.pdf
>
>
> Integrate security with Apache Ranger for a unified Hadoop security solution.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420590#comment-15420590 ] Lili Ma edited comment on HAWQ-256 at 8/15/16 3:43 AM: --- Below is the detailed list of default behaviors for the object operation if no privilege is specified. 1. A database is allowed for connecting and creating temp table. Which means, user A creates a database, everyone can connect to this database, and create temp table into it. 2. Language is allowed for using. Which means, user A creates a language, everyone can use this language. 3. Function is allowed for executing. Which means, user A defines a function, then everyone can execute the function. [~bosco], I think your suggestion on assigning a group "public" and assigns each resource the default behavior for this group makes sense. For the 'deny' operation, say, "I don't want user A to connect to database 1", we can simply add a "deny" record in Ranger 0.6, say, adding userA to the blacklist for database1. But for Ranger 0.5, what we can do for it?? For [~hubertzhang]'s solution for removing the user A out of 'public' group, then we need to keep one 'public' group for each object, I'm afraid there will be too many groups there... [~bosco], could you share the upgrade possibility for the user to upgrade Ranger 0.5 to Ranger 0.6? Does the user need many efforts to do the upgrade? If not, I think we can just simply consider supporting Ranger 0.6. Things will get more clear then, Thanks:) was (Author: lilima): Below is the detailed list of default behaviors for the object operation if no privilege is specified. 1. A database is allowed for connect and create temp table. Which means, user A creates a database, everyone can connect to this database, and create temp table into it. 2. Language is allowed to use. Which means, user A creates a language, everyone can use this language. 3. Function is allowed to execute. Which means, user A defines a function, then everyone can execute the function. [~bosco], I think your suggestion on assigning a group "public" and assigns each resource the default behavior for this group makes sense. For the 'deny' operation, say, "I don't want user A to connect to database 1", we can simply add a "deny" record in Ranger 0.6, say, adding userA to the blacklist for database1. But for Ranger 0.5, what we can do for it?? For [~hubertzhang]'s solution for removing the user A out of 'public' group, then we need to keep one 'public' group for each object, I'm afraid there will be too many groups there... [~bosco], could you share the upgrade possibility for the user to upgrade Ranger 0.5 to Ranger 0.6? Does the user need many efforts to do the upgrade? If not, I think we can just simply consider supporting Ranger 0.6. Things will get more clear then, Thanks:) > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[
https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420591#comment-15420591
]
Hubert Zhang commented on HAWQ-256:
---
HAWQ support two kinds of ACL check when there are more than one privilege in
one request.
One is ALL, which means that all the privileges are required to be allowed.
The other one is ANY, which means that one of the required privileges to be
allowed is sufficient.
{code}
{
“requestor” : “u1”,
[
{
"resource" : {“TABLE”: “t1”, “DATABASE”: “db1”},
"privilege" : [“select”, "insert"]
"how": ALL
},
{
“resource” : {“TABLE”: “t2”, “DATABASE”: “db1”},
“privilege” : [“select”]
"how": ANY
}
]
}
{code}
> Integrate Security with Apache Ranger
> -
>
> Key: HAWQ-256
> URL: https://issues.apache.org/jira/browse/HAWQ-256
> Project: Apache HAWQ
> Issue Type: New Feature
> Components: PXF, Security
>Reporter: Michael Andre Pearce (IG)
>Assignee: Lili Ma
> Fix For: backlog
>
> Attachments: HAWQRangerSupportDesign.pdf
>
>
> Integrate security with Apache Ranger for a unified Hadoop security solution.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420590#comment-15420590 ] Lili Ma commented on HAWQ-256: -- Below is the detailed list of default behaviors for the object operation if no privilege is specified. 1. A database is allowed for connect and create temp table. Which means, user A creates a database, everyone can connect to this database, and create temp table into it. 2. Language is allowed to use. Which means, user A creates a language, everyone can use this language. 3. Function is allowed to execute. Which means, user A defines a function, then everyone can execute the function. [~bosco], I think your suggestion on assigning a group "public" and assigns each resource the default behavior for this group makes sense. For the 'deny' operation, say, "I don't want user A to connect to database 1", we can simply add a "deny" record in Ranger 0.6, say, adding userA to the blacklist for database1. But for Ranger 0.5, what we can do for it?? For [~hubertzhang]'s solution for removing the user A out of 'public' group, then we need to keep one 'public' group for each object, I'm afraid there will be too many groups there... [~bosco], could you share the upgrade possibility for the user to upgrade Ranger 0.5 to Ranger 0.6? Does the user need many efforts to do the upgrade? If not, I think we can just simply consider supporting Ranger 0.6. Things will get more clear then, Thanks:) > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-hawq pull request #846: [WIP]Hawq 991. Add support for "HAWQ regis...
Github user ictmalili commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/846#discussion_r74715268
--- Diff: tools/bin/hawqregister ---
@@ -40,119 +38,126 @@ EXECNAME = os.path.split(__file__)[-1]
setup_tool_logging(EXECNAME,getLocalHostname(),getUserName())
-def create_opt_parser(version):
+def option_parser():
parser = OptParser(option_class=OptChecker,
usage='usage: %prog [options] database_name
table_name file_or_dir_path_in_hdfs',
- version=version)
+ version='%prog version $Revision: #1 $')
parser.remove_option('-h')
parser.add_option('-?', '--help', action='help')
-parser.add_option('-h', '--host', help="host of the target DB")
-parser.add_option('-p', '--port', help="port of the target DB",
type='int', default=0)
-parser.add_option('-U', '--user', help="username of the target DB")
-return parser
+parser.add_option('-h', '--host', help='host of the target DB')
+parser.add_option('-p', '--port', help='port of the target DB',
type='int', default=0)
+parser.add_option('-U', '--user', help='username of the target DB')
+parser.add_option('-d', '--database', default = 'postgres', dest =
'database', help='database name')
--- End diff --
@xunzhang , I think you can put the interface onto JIRA description, e.g.
the parameters description. Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #845: HAWQ-999. Treat hash table as random when file co...
Github user ztao1987 commented on the issue: https://github.com/apache/incubator-hawq/pull/845 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Comment Edited] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420550#comment-15420550 ] Hubert Zhang edited comment on HAWQ-256 at 8/15/16 1:40 AM: [~bosco]You suggest to replace Application(HAWQ) default ACL from application internal to Ranger policy too. It's another way to handle this problem. I have two questions: 1 Group "public" is a default internal type group in Ranger, I wonder whether the users synced from ldap belong to group "public" automatically? 2 Ranger 0.6 introduces deny condition, So even if a user inherits default privilege from group "public", It can also be revoked with specific deny operation. But how about Ranger 0.5? One method is to remove the user from "public" group. But the limitation of this method is that we have to create "public" group for each resource, and assign the new users(synced from ldap) to these groups. was (Author: hubertzhang): [~bosco]You suggest to replace Application(HAWQ) default ACL from application internal to Ranger policy too. It's another way to handle this problem. I have two questions: 1 Group "public" is a default internal type group in Ranger, I wonder whether the users synced from ldap belong to group "public" automatically? 2 Ranger 0.6 introduces deny condition, So even if a user inherits default privilege from group "public", It can also be revoked with specific deny operation. But how about Ranger 0.5? One method is to remove the user from "public" group. But the limitation of this method is that we have to create "public" group for each resource, and assign the new users(synced from ldap) to this group. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420550#comment-15420550 ] Hubert Zhang edited comment on HAWQ-256 at 8/15/16 1:40 AM: [~bosco]You suggest to replace Application(HAWQ) default ACL from application internal to Ranger policy too. It's another way to handle this problem. I have two questions: 1 Group "public" is a default internal type group in Ranger, I wonder whether the users synced from ldap belong to group "public" automatically? 2 Ranger 0.6 introduces deny condition, So even if a user inherits default privilege from group "public", It can also be revoked with specific deny operation. But how about Ranger 0.5? One method is to remove the user from "public" group. But the limitation of this method is that we have to create "public" group for each resource, and assign the new users(synced from ldap) to this group. was (Author: hubertzhang): [~bosco]You suggest to replace Application(HAWQ) default ACL from application internal to Ranger policy too. It's another way to handle this problem. I have two questions: 1 Group "public" is a default internal type group in Ranger, I wonder whether the users synced from ldap belong to group "public" 2 Ranger 0.6 introduces deny condition, So even if a user inherits default privilege from group "public", It can also be revoked with specific deny operation. But how about Ranger 0.5? One method is to remove the user from "public" group. But the limitation of this method is that we have to create "public" group for each resource, and assign the new users(synced from ldap) to this group. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420550#comment-15420550 ] Hubert Zhang commented on HAWQ-256: --- [~bosco]You suggest to replace Application(HAWQ) default ACL from application internal to Ranger policy too. It's another way to handle this problem. I have two questions: 1 Group "public" is a default internal type group in Ranger, I wonder whether the users synced from ldap belong to group "public" 2 Ranger 0.6 introduces deny condition, So even if a user inherits default privilege from group "public", It can also be revoked with specific deny operation. But how about Ranger 0.5? One method is to remove the user from "public" group. But the limitation of this method is that we have to create "public" group for each resource, and assign the new users(synced from ldap) to this group. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
