[jira] [Comment Edited] (HAWQ-1521) Idle QE Processes Can't Quit After An Interval
[
https://issues.apache.org/jira/browse/HAWQ-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148589#comment-16148589
]
Hubert Zhang edited comment on HAWQ-1521 at 8/31/17 7:48 AM:
-
HAWQ remove allocateGang to record gang information in the following variables.
{code}
static List *allocatedReaderGangsN = NIL;
static List *availableReaderGangsN = NIL;
static List *allocatedReaderGangs1 = NIL;
static List *availableReaderGangs1 = NIL;
static Gang *primaryWriterGang = NULL;
{code}
As a result, calling gangsExist will always return false.
But the cached gang information, to be more specific the cached QE information
could be found in other places at QD side.
The following struct ExecutorCache(the two pools) contains the all the cached
QE by a session. We could use this data structure to quit idle QE when session
is idled for a long time.
{code}
typedef struct ExecutorCache {
boolinit;
MemoryContext ctx;
struct PoolMgrState *pool;
struct PoolMgrState *entrydb_pool; // pool for entry db connection
int cached_num;
int allocated_num;
int takeover_num;
} ExecutorCache;
static ExecutorCacheexecutor_cache;
{code}
was (Author: hubertzhang):
HAWQ remove allocateGang to record gang information in the following variables.
static List *allocatedReaderGangsN = NIL;
static List *availableReaderGangsN = NIL;
static List *allocatedReaderGangs1 = NIL;
static List *availableReaderGangs1 = NIL;
static Gang *primaryWriterGang = NULL;
As a result, calling gangsExist will always return false.
But the cached gang information, to be more specific the cached QE information
could be found in other places at QD side.
The following struct ExecutorCache(the two pools) contains the all the cached
QE by a session. We could use this data structure to quit idle QE when session
is idled for a long time.
typedef struct ExecutorCache {
boolinit;
MemoryContext ctx;
struct PoolMgrState *pool;
struct PoolMgrState *entrydb_pool; // pool for entry db connection
int cached_num;
int allocated_num;
int takeover_num;
} ExecutorCache;
static ExecutorCacheexecutor_cache;
> Idle QE Processes Can't Quit After An Interval
> --
>
> Key: HAWQ-1521
> URL: https://issues.apache.org/jira/browse/HAWQ-1521
> Project: Apache HAWQ
> Issue Type: Bug
>Reporter: Lin Wen
>Assignee: Radar Lei
>
> After a query is finished, there are some idle QE processes on segments.
> These QE processes are expected to quit after a time interval, this interval
> is controlled by a GUC gp_vmem_idle_resource_timeout, the default value is 18
> seconds.
> However, this does't act as expected. Idle QE processes on segments always
> exist there, unless the QD process quit.
> The reason is in postgres.c, the codes to enable this timer can't get
> executed. function gangsExist() always return false, since gang related
> structures are all NULL.
> if (IdleSessionGangTimeout > 0 && gangsExist())
> if (!enable_sig_alarm( IdleSessionGangTimeout /* ms */, false))
> elog(FATAL, "could not set timer for client wait
> timeout");
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HAWQ-1521) Idle QE Processes Can't Quit After An Interval
[
https://issues.apache.org/jira/browse/HAWQ-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148589#comment-16148589
]
Hubert Zhang commented on HAWQ-1521:
HAWQ remove allocateGang to record gang information in the following variables.
static List *allocatedReaderGangsN = NIL;
static List *availableReaderGangsN = NIL;
static List *allocatedReaderGangs1 = NIL;
static List *availableReaderGangs1 = NIL;
static Gang *primaryWriterGang = NULL;
As a result, calling gangsExist will always return false.
But the cached gang information, to be more specific the cached QE information
could be found in other places at QD side.
The following struct ExecutorCache(the two pools) contains the all the cached
QE by a session. We could use this data structure to quit idle QE when session
is idled for a long time.
typedef struct ExecutorCache {
boolinit;
MemoryContext ctx;
struct PoolMgrState *pool;
struct PoolMgrState *entrydb_pool; // pool for entry db connection
int cached_num;
int allocated_num;
int takeover_num;
} ExecutorCache;
static ExecutorCacheexecutor_cache;
> Idle QE Processes Can't Quit After An Interval
> --
>
> Key: HAWQ-1521
> URL: https://issues.apache.org/jira/browse/HAWQ-1521
> Project: Apache HAWQ
> Issue Type: Bug
>Reporter: Lin Wen
>Assignee: Radar Lei
>
> After a query is finished, there are some idle QE processes on segments.
> These QE processes are expected to quit after a time interval, this interval
> is controlled by a GUC gp_vmem_idle_resource_timeout, the default value is 18
> seconds.
> However, this does't act as expected. Idle QE processes on segments always
> exist there, unless the QD process quit.
> The reason is in postgres.c, the codes to enable this timer can't get
> executed. function gangsExist() always return false, since gang related
> structures are all NULL.
> if (IdleSessionGangTimeout > 0 && gangsExist())
> if (!enable_sig_alarm( IdleSessionGangTimeout /* ms */, false))
> elog(FATAL, "could not set timer for client wait
> timeout");
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HAWQ-1476) Augment enable-ranger-plugin.sh to support kerberos.
[ https://issues.apache.org/jira/browse/HAWQ-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang reassigned HAWQ-1476: -- Assignee: Hubert Zhang (was: Ed Espino) > Augment enable-ranger-plugin.sh to support kerberos. > > > Key: HAWQ-1476 > URL: https://issues.apache.org/jira/browse/HAWQ-1476 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: backlog > > > Now ranger can lookup hawq resource in kerberized environment. So we also > need to change enable-ranger-plugin.sh to automatically fill the > authentication type and hawq kerberos service name fields -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HAWQ-1443) Implement Ranger lookup for HAWQ with Kerberos enabled.
[ https://issues.apache.org/jira/browse/HAWQ-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988295#comment-15988295 ] Hubert Zhang commented on HAWQ-1443: We need to change at least two aspects: 1 Ranger UI for hawq service. It should contain not only the username and password, but also the authentication type and principal. 2 HawqClient. This class should inherit from BaseClient of ranger just like what HiveClient and HbaseClient do. BaseClient include the common code to do kerberos authentication. jdbc parameter of kerberos principal also needs to be bind to the jdbc string. > Implement Ranger lookup for HAWQ with Kerberos enabled. > --- > > Key: HAWQ-1443 > URL: https://issues.apache.org/jira/browse/HAWQ-1443 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: backlog > > > When add a HAWQ service in Ranger, we also need to configure Ranger look up > service for HAWQ. Lookup service can be done through JDBC with username and > password. But It cannot support Kerberos authentication currently. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HAWQ-1443) Implement Ranger lookup for HAWQ with Kerberos enabled.
[ https://issues.apache.org/jira/browse/HAWQ-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang reassigned HAWQ-1443: -- Assignee: Hubert Zhang (was: Ed Espino) > Implement Ranger lookup for HAWQ with Kerberos enabled. > --- > > Key: HAWQ-1443 > URL: https://issues.apache.org/jira/browse/HAWQ-1443 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: backlog > > > When add a HAWQ service in Ranger, we also need to configure Ranger look up > service for HAWQ. Lookup service can be done through JDBC with username and > password. But It cannot support Kerberos authentication currently. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HAWQ-1443) Implement Ranger lookup for HAWQ with Kerberos enabled.
Hubert Zhang created HAWQ-1443: -- Summary: Implement Ranger lookup for HAWQ with Kerberos enabled. Key: HAWQ-1443 URL: https://issues.apache.org/jira/browse/HAWQ-1443 Project: Apache HAWQ Issue Type: Sub-task Reporter: Hubert Zhang Assignee: Ed Espino When add a HAWQ service in Ranger, we also need to configure Ranger look up service for HAWQ. Lookup service can be done through JDBC with username and password. But It cannot support Kerberos authentication currently. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (HAWQ-1375) Ranger should always using the current user to do privilege check.
[
https://issues.apache.org/jira/browse/HAWQ-1375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hubert Zhang closed HAWQ-1375.
--
Resolution: Invalid
Fix Version/s: 2.2.0.0-incubating
> Ranger should always using the current user to do privilege check.
> --
>
> Key: HAWQ-1375
> URL: https://issues.apache.org/jira/browse/HAWQ-1375
> Project: Apache HAWQ
> Issue Type: Bug
>Reporter: Hubert Zhang
>Assignee: Ed Espino
> Fix For: 2.2.0.0-incubating
>
>
> Failure Case:
> {code}
> user u1 create table a(i int);
> user u2 create view av as select * from a;
> user u3 select * from av.
> {code}
> When ORCA is on, u3 will first ask select privilege as user u3 to Ranger, and
> then ask select privilege as user u2 to Ranger.
> The second check should be removed, since there may be no privilege for u2 to
> select av even if av is created by u2 in Ranger mode.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HAWQ-1375) Ranger should always using the current user to do privilege check.
Hubert Zhang created HAWQ-1375:
--
Summary: Ranger should always using the current user to do
privilege check.
Key: HAWQ-1375
URL: https://issues.apache.org/jira/browse/HAWQ-1375
Project: Apache HAWQ
Issue Type: Bug
Reporter: Hubert Zhang
Assignee: Ed Espino
Failure Case:
{code}
user u1 create table a(i int);
user u2 create view av as select * from a;
user u3 select * from av.
{code}
When ORCA is on, u3 will first ask select privilege as user u3 to Ranger, and
then ask select privilege as user u2 to Ranger.
The second check should be removed, since there may be no privilege for u2 to
select av even if av is created by u2 in Ranger mode.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (HAWQ-1370) Misuse of regular expressions in init_file of feature test.
[
https://issues.apache.org/jira/browse/HAWQ-1370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hubert Zhang resolved HAWQ-1370.
Resolution: Fixed
Fix Version/s: 2.2.0.0-incubating
fixed
> Misuse of regular expressions in init_file of feature test.
> ---
>
> Key: HAWQ-1370
> URL: https://issues.apache.org/jira/browse/HAWQ-1370
> Project: Apache HAWQ
> Issue Type: Bug
>Reporter: Hubert Zhang
>Assignee: Hubert Zhang
> Fix For: 2.2.0.0-incubating
>
>
> in global_init_file of feature test, we want to skip expressions which
> include file and line number, e.g.(aclchk.c:123), or (aclchk.cpp:134).
> But currently, the regular expressions is {code}(.*c[p]+:\d+) {code} which
> need to be replaced by {code}(.*c[p]*:\d+) {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HAWQ-1370) Misuse of regular expressions in init_file of feature test.
[
https://issues.apache.org/jira/browse/HAWQ-1370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hubert Zhang reassigned HAWQ-1370:
--
Assignee: Hubert Zhang (was: Ed Espino)
> Misuse of regular expressions in init_file of feature test.
> ---
>
> Key: HAWQ-1370
> URL: https://issues.apache.org/jira/browse/HAWQ-1370
> Project: Apache HAWQ
> Issue Type: Bug
>Reporter: Hubert Zhang
>Assignee: Hubert Zhang
>
> in global_init_file of feature test, we want to skip expressions which
> include file and line number, e.g.(aclchk.c:123), or (aclchk.cpp:134).
> But currently, the regular expressions is {code}(.*c[p]+:\d+) {code} which
> need to be replaced by {code}(.*c[p]*:\d+) {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HAWQ-1370) Misuse of regular expressions in init_file of feature test.
[ https://issues.apache.org/jira/browse/HAWQ-1370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang updated HAWQ-1370: --- Description: in global_init_file of feature test, we want to skip expressions which include file and line number, e.g.(aclchk.c:123), or (aclchk.cpp:134). But currently, the regular expressions is`(.*c[p]+:\d+) ` which need to be replaced by `(.*c[p]*:\d+) ` was: in global_init_file of feature test, we want to skip expressions which include file and line number, e.g.(aclchk.c:123), or (aclchk.cpp:134). But currently, the regular expressions is \(.*c[p]+:\d+\) which need to be replaced by (.*c[p]*:\d+\) > Misuse of regular expressions in init_file of feature test. > --- > > Key: HAWQ-1370 > URL: https://issues.apache.org/jira/browse/HAWQ-1370 > Project: Apache HAWQ > Issue Type: Bug >Reporter: Hubert Zhang >Assignee: Ed Espino > > in global_init_file of feature test, we want to skip expressions which > include file and line number, e.g.(aclchk.c:123), or (aclchk.cpp:134). > But currently, the regular expressions is`(.*c[p]+:\d+) ` which need to be > replaced by `(.*c[p]*:\d+) ` -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HAWQ-1370) Misuse of regular expressions in init_file of feature test.
Hubert Zhang created HAWQ-1370: -- Summary: Misuse of regular expressions in init_file of feature test. Key: HAWQ-1370 URL: https://issues.apache.org/jira/browse/HAWQ-1370 Project: Apache HAWQ Issue Type: Bug Reporter: Hubert Zhang Assignee: Ed Espino in global_init_file of feature test, we want to skip expressions which include file and line number, e.g.(aclchk.c:123), or (aclchk.cpp:134). But currently, the regular expressions is \(.*c[p]+:\d+\) which need to be replaced by (.*c[p]*:\d+\) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HAWQ-1328) Add deny and exclude policy template for hawq service in ranger.
[ https://issues.apache.org/jira/browse/HAWQ-1328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang resolved HAWQ-1328. Resolution: Fixed Fix Version/s: (was: backlog) 2.2.0.0-incubating fixed > Add deny and exclude policy template for hawq service in ranger. > > > Key: HAWQ-1328 > URL: https://issues.apache.org/jira/browse/HAWQ-1328 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: 2.2.0.0-incubating > > > Currently, there is no template of deny and exclude policy for HAWQ service > in Ranger, we need to open this option in ranger-servicedef-hawq.json by > default. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HAWQ-1291) Fix the name of privilege when create temp table.
[ https://issues.apache.org/jira/browse/HAWQ-1291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang resolved HAWQ-1291. Resolution: Fixed Fix Version/s: 2.2.0.0-incubating fixed > Fix the name of privilege when create temp table. > - > > Key: HAWQ-1291 > URL: https://issues.apache.org/jira/browse/HAWQ-1291 > Project: Apache HAWQ > Issue Type: Bug > Components: Core >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: 2.2.0.0-incubating > > > When we do ranger check of query "create temp table", we need to PASS > privilege "TEMP" to RPS instead of "CREATE_TEMP". Or RPS can not recognise it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HAWQ-1358) Refactor gpfdist library in featuretest.
[ https://issues.apache.org/jira/browse/HAWQ-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang resolved HAWQ-1358. Resolution: Fixed Fix Version/s: 2.2.0.0-incubating fixed > Refactor gpfdist library in featuretest. > > > Key: HAWQ-1358 > URL: https://issues.apache.org/jira/browse/HAWQ-1358 > Project: Apache HAWQ > Issue Type: Improvement >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: 2.2.0.0-incubating > > > extract gpfdist code(which exists in exttable test case) as a common library, > to be used by both exttable and ranger feature test. > Main functions: > gpfdist_init: to setup gpfdist environment and start server. > gpfdist_finalize: to stop gpfdist server. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HAWQ-1357) Super user also need to check create privilege of public schema from Ranger.
[ https://issues.apache.org/jira/browse/HAWQ-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang resolved HAWQ-1357. Resolution: Fixed Fix Version/s: 2.2.0.0-incubating fixed > Super user also need to check create privilege of public schema from Ranger. > > > Key: HAWQ-1357 > URL: https://issues.apache.org/jira/browse/HAWQ-1357 > Project: Apache HAWQ > Issue Type: Bug >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: 2.2.0.0-incubating > > > [HAWQ-1318|https://issues.apache.org/jira/browse/HAWQ-1318] add create|usage > privilege of public schema to superuser to fix hawq stop BUG caused by > Resource Manager. > But RM only need the usage privilege of public schema to query HAWQ, So > create privilege need to be removed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HAWQ-1356) Add waring when user does not have usage privilege of namespace.
[ https://issues.apache.org/jira/browse/HAWQ-1356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang resolved HAWQ-1356. Resolution: Fixed Fix Version/s: 2.2.0.0-incubating fixed > Add waring when user does not have usage privilege of namespace. > > > Key: HAWQ-1356 > URL: https://issues.apache.org/jira/browse/HAWQ-1356 > Project: Apache HAWQ > Issue Type: Bug >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: 2.2.0.0-incubating > > > Currently, in Ranger mode, when a user doesn't have usage privilege on public > schema, she cannot see any tables, functions, schemas on public schema. When > she login into database and run a query "select * from a"; The error message > would be "table a doesn't exist", which makes user confuse. > This fix add a warning to tell user that she does not have the usage > privilege of a schema. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (HAWQ-1355) Namespace check may occur multiple times in first query.
[ https://issues.apache.org/jira/browse/HAWQ-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang closed HAWQ-1355. -- fixed > Namespace check may occur multiple times in first query. > > > Key: HAWQ-1355 > URL: https://issues.apache.org/jira/browse/HAWQ-1355 > Project: Apache HAWQ > Issue Type: Bug >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: 2.2.0.0-incubating > > > When running a query, HAWQ need to check namespace usage privilege in > function recomputeNamespacePath. This function will be called repeatedly but > check will be skipped when last_query_sign is equal to current_query_sign. > There is a bug that running the first query doesn't set the last_query_sign. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HAWQ-1359) Add policy test for HAWQ with Ranger enabled.
[ https://issues.apache.org/jira/browse/HAWQ-1359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887159#comment-15887159 ] Hubert Zhang edited comment on HAWQ-1359 at 2/28/17 3:24 AM: - PR @ https://github.com/apache/incubator-hawq/pull/1151 was (Author: hubertzhang): PR @ https://github.com/apache/incubator-hawq/pull/1144 > Add policy test for HAWQ with Ranger enabled. > - > > Key: HAWQ-1359 > URL: https://issues.apache.org/jira/browse/HAWQ-1359 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Hubert Zhang >Assignee: Chunling Wang > Fix For: backlog > > > Policy test includes different json requests(used by ranger) for different > quries, For example, for query "select * from a;", it needs usage privilege > of schema public, and select privilege of table a. > There are also queries can only be executed by superuser, we also test them > in policy test. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HAWQ-1360) Check privilege of sequence pass the wrong type to RPS.
[ https://issues.apache.org/jira/browse/HAWQ-1360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887164#comment-15887164 ] Hubert Zhang commented on HAWQ-1360: PR @ https://github.com/apache/incubator-hawq/pull/1144 > Check privilege of sequence pass the wrong type to RPS. > --- > > Key: HAWQ-1360 > URL: https://issues.apache.org/jira/browse/HAWQ-1360 > Project: Apache HAWQ > Issue Type: Bug >Reporter: Hubert Zhang >Assignee: Ed Espino > > When check privilege of query with ranger enabled, a json request will be > send to RPS, including information about: privilege type, resource type and > role name. > Currently, checking sequence privilege will send resource type "table" > instead of "sequence". -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HAWQ-1359) Add policy test for HAWQ with Ranger enabled.
[ https://issues.apache.org/jira/browse/HAWQ-1359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887159#comment-15887159 ] Hubert Zhang commented on HAWQ-1359: PR @ https://github.com/apache/incubator-hawq/pull/1144 > Add policy test for HAWQ with Ranger enabled. > - > > Key: HAWQ-1359 > URL: https://issues.apache.org/jira/browse/HAWQ-1359 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Hubert Zhang >Assignee: Chunling Wang > Fix For: backlog > > > Policy test includes different json requests(used by ranger) for different > quries, For example, for query "select * from a;", it needs usage privilege > of schema public, and select privilege of table a. > There are also queries can only be executed by superuser, we also test them > in policy test. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HAWQ-1360) Check privilege of sequence pass the wrong type to RPS.
Hubert Zhang created HAWQ-1360: -- Summary: Check privilege of sequence pass the wrong type to RPS. Key: HAWQ-1360 URL: https://issues.apache.org/jira/browse/HAWQ-1360 Project: Apache HAWQ Issue Type: Bug Reporter: Hubert Zhang Assignee: Ed Espino When check privilege of query with ranger enabled, a json request will be send to RPS, including information about: privilege type, resource type and role name. Currently, checking sequence privilege will send resource type "table" instead of "sequence". -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HAWQ-1358) Refactor gpfdist library in featuretest.
[ https://issues.apache.org/jira/browse/HAWQ-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang updated HAWQ-1358: --- Description: extract gpfdist code(which exists in exttable test case) as a common library, to be used by both exttable and ranger feature test. Main functions: gpfdist_init: to setup gpfdist environment and start server. gpfdist_finalize: to stop gpfdist server. was:extract gpfdist as a common library, to be used by both exttable and ranger feature test. > Refactor gpfdist library in featuretest. > > > Key: HAWQ-1358 > URL: https://issues.apache.org/jira/browse/HAWQ-1358 > Project: Apache HAWQ > Issue Type: Improvement >Reporter: Hubert Zhang >Assignee: Hubert Zhang > > extract gpfdist code(which exists in exttable test case) as a common library, > to be used by both exttable and ranger feature test. > Main functions: > gpfdist_init: to setup gpfdist environment and start server. > gpfdist_finalize: to stop gpfdist server. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HAWQ-1356) Add waring when user does not have usage privilege of namespace.
[ https://issues.apache.org/jira/browse/HAWQ-1356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang reassigned HAWQ-1356: -- Assignee: Hubert Zhang (was: Ed Espino) > Add waring when user does not have usage privilege of namespace. > > > Key: HAWQ-1356 > URL: https://issues.apache.org/jira/browse/HAWQ-1356 > Project: Apache HAWQ > Issue Type: Bug >Reporter: Hubert Zhang >Assignee: Hubert Zhang > > Currently, in Ranger mode, when a user doesn't have usage privilege on public > schema, she cannot see any tables, functions, schemas on public schema. When > she login into database and run a query "select * from a"; The error message > would be "table a doesn't exist", which makes user confuse. > This fix add a warning to tell user that she does not have the usage > privilege of a schema. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HAWQ-1357) Super user also need to check create privilege of public schema from Ranger.
[ https://issues.apache.org/jira/browse/HAWQ-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang reassigned HAWQ-1357: -- Assignee: Hubert Zhang (was: Ed Espino) > Super user also need to check create privilege of public schema from Ranger. > > > Key: HAWQ-1357 > URL: https://issues.apache.org/jira/browse/HAWQ-1357 > Project: Apache HAWQ > Issue Type: Bug >Reporter: Hubert Zhang >Assignee: Hubert Zhang > > [HAWQ-1318|https://issues.apache.org/jira/browse/HAWQ-1318] add create|usage > privilege of public schema to superuser to fix hawq stop BUG caused by > Resource Manager. > But RM only need the usage privilege of public schema to query HAWQ, So > create privilege need to be removed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HAWQ-1355) Namespace check may occur multiple times in first query.
[ https://issues.apache.org/jira/browse/HAWQ-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang reassigned HAWQ-1355: -- Assignee: Hubert Zhang (was: Ed Espino) > Namespace check may occur multiple times in first query. > > > Key: HAWQ-1355 > URL: https://issues.apache.org/jira/browse/HAWQ-1355 > Project: Apache HAWQ > Issue Type: Bug >Reporter: Hubert Zhang >Assignee: Hubert Zhang > > When running a query, HAWQ need to check namespace usage privilege in > function recomputeNamespacePath. This function will be called repeatedly but > check will be skipped when last_query_sign is equal to current_query_sign. > There is a bug that running the first query doesn't set the last_query_sign. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HAWQ-1359) Add policy test for HAWQ with Ranger enabled.
Hubert Zhang created HAWQ-1359: -- Summary: Add policy test for HAWQ with Ranger enabled. Key: HAWQ-1359 URL: https://issues.apache.org/jira/browse/HAWQ-1359 Project: Apache HAWQ Issue Type: Sub-task Reporter: Hubert Zhang Assignee: Ed Espino Policy test includes different json requests(used by ranger) for different quries, For example, for query "select * from a;", it needs usage privilege of schema public, and select privilege of table a. There are also queries can only be executed by superuser, we also test them in policy test. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HAWQ-1358) Refactor gpfdist library in featuretest.
Hubert Zhang created HAWQ-1358: -- Summary: Refactor gpfdist library in featuretest. Key: HAWQ-1358 URL: https://issues.apache.org/jira/browse/HAWQ-1358 Project: Apache HAWQ Issue Type: Improvement Reporter: Hubert Zhang Assignee: Ed Espino extract gpfdist as a common library, to be used by both exttable and ranger feature test. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HAWQ-1357) Super user also need to check create privilege of public schema from Ranger.
Hubert Zhang created HAWQ-1357: -- Summary: Super user also need to check create privilege of public schema from Ranger. Key: HAWQ-1357 URL: https://issues.apache.org/jira/browse/HAWQ-1357 Project: Apache HAWQ Issue Type: Bug Reporter: Hubert Zhang Assignee: Ed Espino [HAWQ-1318|https://issues.apache.org/jira/browse/HAWQ-1318] add create|usage privilege of public schema to superuser to fix hawq stop BUG caused by Resource Manager. But RM only need the usage privilege of public schema to query HAWQ, So create privilege need to be removed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HAWQ-1356) Add waring when user does not have usage privilege of namespace.
Hubert Zhang created HAWQ-1356: -- Summary: Add waring when user does not have usage privilege of namespace. Key: HAWQ-1356 URL: https://issues.apache.org/jira/browse/HAWQ-1356 Project: Apache HAWQ Issue Type: Bug Reporter: Hubert Zhang Assignee: Ed Espino Currently, in Ranger mode, when a user doesn't have usage privilege on public schema, she cannot see any tables, functions, schemas on public schema. When she login into database and run a query "select * from a"; The error message would be "table a doesn't exist", which makes user confuse. This fix add a warning to tell user that she does not have the usage privilege of a schema. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HAWQ-1328) Add deny and exclude policy template for hawq service in ranger.
[ https://issues.apache.org/jira/browse/HAWQ-1328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang reassigned HAWQ-1328: -- Assignee: Hubert Zhang (was: Ed Espino) > Add deny and exclude policy template for hawq service in ranger. > > > Key: HAWQ-1328 > URL: https://issues.apache.org/jira/browse/HAWQ-1328 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: backlog > > > Currently, there is no template of deny and exclude policy for HAWQ service > in Ranger, we need to open this option in ranger-servicedef-hawq.json by > default. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang reassigned HAWQ-256: - Assignee: Lili Ma (was: Hubert Zhang) > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf, > HAWQRangerSupportDesign_v0.2.pdf, HAWQRangerSupportDesign_v0.3.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang reassigned HAWQ-256: - Assignee: Hubert Zhang (was: Lili Ma) > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: Security >Reporter: Michael Andre Pearce (IG) >Assignee: Hubert Zhang > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf, > HAWQRangerSupportDesign_v0.2.pdf, HAWQRangerSupportDesign_v0.3.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HAWQ-1328) Add deny and exclude policy template for hawq service in ranger.
Hubert Zhang created HAWQ-1328: -- Summary: Add deny and exclude policy template for hawq service in ranger. Key: HAWQ-1328 URL: https://issues.apache.org/jira/browse/HAWQ-1328 Project: Apache HAWQ Issue Type: Sub-task Reporter: Hubert Zhang Assignee: Ed Espino Currently, there is no template of deny and exclude policy for HAWQ service in Ranger, we need to open this option in ranger-servicedef-hawq.json by default. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HAWQ-1311) Optimize the performance of hawq with ranger enabled.
[ https://issues.apache.org/jira/browse/HAWQ-1311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852704#comment-15852704 ] Hubert Zhang commented on HAWQ-1311: Step 1 add log to evaluate the cost of each rps call. > Optimize the performance of hawq with ranger enabled. > - > > Key: HAWQ-1311 > URL: https://issues.apache.org/jira/browse/HAWQ-1311 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: backlog > > > There are still more than one ranger plugin server(rps) call for a single > query. We need to first analyse the cost of ranger aclcheck and then using a > query level aclcache to minimise the times of rps call for each query. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HAWQ-1311) Optimize the performance of hawq with ranger enabled.
[ https://issues.apache.org/jira/browse/HAWQ-1311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang reassigned HAWQ-1311: -- Assignee: Hubert Zhang (was: Ed Espino) > Optimize the performance of hawq with ranger enabled. > - > > Key: HAWQ-1311 > URL: https://issues.apache.org/jira/browse/HAWQ-1311 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: backlog > > > There are still more than one ranger plugin server(rps) call for a single > query. We need to first analyse the cost of ranger aclcheck and then using a > query level aclcache to minimise the times of rps call for each query. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HAWQ-1311) Optimize the performance of hawq with ranger enabled.
Hubert Zhang created HAWQ-1311: -- Summary: Optimize the performance of hawq with ranger enabled. Key: HAWQ-1311 URL: https://issues.apache.org/jira/browse/HAWQ-1311 Project: Apache HAWQ Issue Type: Sub-task Reporter: Hubert Zhang Assignee: Ed Espino There are still more than one ranger plugin server(rps) call for a single query. We need to first analyse the cost of ranger aclcheck and then using a query level aclcache to minimise the times of rps call for each query. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HAWQ-1282) Shared Input Scan may result in endless loop.
[ https://issues.apache.org/jira/browse/HAWQ-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834026#comment-15834026 ] Hubert Zhang commented on HAWQ-1282: Here are some context. After running some queries(including shared input scan) There may be residual process left. The call stack of residual process is as follows: ``` #0 0x0032214e1523 in select () from /lib64/libc.so.6 #1 0x0069a072 in writer_wait_for_acks (share_id=0, xslice=Unhandled dwarf expression opcode 0xf3 ) at nodeShareInputScan.c:885 #2 0x0069ae92 in shareinput_writer_notifyready (share_id=0, xslice=3, planGen=PLANGEN_PLANNER) at nodeShareInputScan.c:854 #3 0x00693f48 in ExecMaterial (node=0x2e93a28) at nodeMaterial.c:248 #4 0x0066df31 in ExecProcNode (node=0x2e93a28) at execProcnode.c:971 #5 0x00699c17 in init_tuplestore_state (node=0x2e935c8) at nodeShareInputScan.c:100 #6 0x00699f99 in ShareInputNext (node=0x2e935c8) at nodeShareInputScan.c:210 #7 0x0066dd51 in ExecProcNode (node=0x2e935c8) at execProcnode.c:1003 #8 0x0069bd50 in ExecSort (node=0x2e93020) at nodeSort.c:335 ``` Function writer_wait_for_acks() will be in an endless loop if every time the select function returns 0 because of timeout or minus value. > Shared Input Scan may result in endless loop. > - > > Key: HAWQ-1282 > URL: https://issues.apache.org/jira/browse/HAWQ-1282 > Project: Apache HAWQ > Issue Type: Bug > Components: Core >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: 2.1.0.0-incubating > > > There are residual process after running some queries. Through the call > stack, we find that there is an endless loop in function > writer_wait_for_acks() in shared input scan. > We plan to add max retry times to avoid this problem. > Also, there are some file handler leaks in retry_read() and retry_write() of > shared input scan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (HAWQ-1282) Shared Input Scan may result in endless loop.
[ https://issues.apache.org/jira/browse/HAWQ-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang closed HAWQ-1282. -- Resolution: Fixed Fix Version/s: 2.1.0.0-incubating > Shared Input Scan may result in endless loop. > - > > Key: HAWQ-1282 > URL: https://issues.apache.org/jira/browse/HAWQ-1282 > Project: Apache HAWQ > Issue Type: Bug > Components: Core >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: 2.1.0.0-incubating > > > There are residual process after running some queries. Through the call > stack, we find that there is an endless loop in function > writer_wait_for_acks() in shared input scan. > We plan to add max retry times to avoid this problem. > Also, there are some file handler leaks in retry_read() and retry_write() of > shared input scan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1282) Shared Input Scan may result in endless loop.
[ https://issues.apache.org/jira/browse/HAWQ-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang updated HAWQ-1282: --- Summary: Shared Input Scan may result in endless loop. (was: Shared Input Scan may result in endless loop) > Shared Input Scan may result in endless loop. > - > > Key: HAWQ-1282 > URL: https://issues.apache.org/jira/browse/HAWQ-1282 > Project: Apache HAWQ > Issue Type: Bug > Components: Core >Reporter: Hubert Zhang >Assignee: Hubert Zhang > > There are residual process after running some queries. Through the call > stack, we find that there is an endless loop in function > writer_wait_for_acks() in shared input scan. > We plan to add max retry times to avoid this problem. > Also, there are some file handler leaks in retry_read() and retry_write() of > shared input scan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HAWQ-1282) Shared Input Scan may result in endless loop
Hubert Zhang created HAWQ-1282: -- Summary: Shared Input Scan may result in endless loop Key: HAWQ-1282 URL: https://issues.apache.org/jira/browse/HAWQ-1282 Project: Apache HAWQ Issue Type: Bug Components: Core Reporter: Hubert Zhang Assignee: Ed Espino There are residual process after running some queries. Through the call stack, we find that there is an endless loop in function writer_wait_for_acks() in shared input scan. We plan to add max retry times to avoid this problem. Also, we fix file handler leak in retry_read and retry_write. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1282) Shared Input Scan may result in endless loop
[ https://issues.apache.org/jira/browse/HAWQ-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang updated HAWQ-1282: --- Description: There are residual process after running some queries. Through the call stack, we find that there is an endless loop in function writer_wait_for_acks() in shared input scan. We plan to add max retry times to avoid this problem. Also, there are some file handler leaks in retry_read() and retry_write() of shared input scan. was: There are residual process after running some queries. Through the call stack, we find that there is an endless loop in function writer_wait_for_acks() in shared input scan. We plan to add max retry times to avoid this problem. Also, we fix file handler leak in retry_read and retry_write. > Shared Input Scan may result in endless loop > > > Key: HAWQ-1282 > URL: https://issues.apache.org/jira/browse/HAWQ-1282 > Project: Apache HAWQ > Issue Type: Bug > Components: Core >Reporter: Hubert Zhang >Assignee: Hubert Zhang > > There are residual process after running some queries. Through the call > stack, we find that there is an endless loop in function > writer_wait_for_acks() in shared input scan. > We plan to add max retry times to avoid this problem. > Also, there are some file handler leaks in retry_read() and retry_write() of > shared input scan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HAWQ-1282) Shared Input Scan may result in endless loop
[ https://issues.apache.org/jira/browse/HAWQ-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang reassigned HAWQ-1282: -- Assignee: Hubert Zhang (was: Ed Espino) > Shared Input Scan may result in endless loop > > > Key: HAWQ-1282 > URL: https://issues.apache.org/jira/browse/HAWQ-1282 > Project: Apache HAWQ > Issue Type: Bug > Components: Core >Reporter: Hubert Zhang >Assignee: Hubert Zhang > > There are residual process after running some queries. Through the call > stack, we find that there is an endless loop in function > writer_wait_for_acks() in shared input scan. > We plan to add max retry times to avoid this problem. > Also, we fix file handler leak in retry_read and retry_write. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HAWQ-1275) Check build-in catalogs, tables and functions in native aclcheck.
Hubert Zhang created HAWQ-1275: -- Summary: Check build-in catalogs, tables and functions in native aclcheck. Key: HAWQ-1275 URL: https://issues.apache.org/jira/browse/HAWQ-1275 Project: Apache HAWQ Issue Type: Sub-task Components: Security Reporter: Hubert Zhang Assignee: Ed Espino We plan to do privilege check in hawq side for build-in catalogs, tables and functions. The reasons are two folds; 1 Ranger mainly manage the user data, but build-in catalogs and tables are not related to user data(note that some of them contain statistics information of user data such as catalog table pg_aoseg_*). 2 We haven't finish the code of merge of all the privilege check requests into one big request. Without it query such as "\d" and "analyze" will lead to hundreds of RPS request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (HAWQ-1243) Add suffix name for ranger restful service.
[ https://issues.apache.org/jira/browse/HAWQ-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang closed HAWQ-1243. -- Resolution: Fixed Fix Version/s: 2.2.0.0-incubating > Add suffix name for ranger restful service. > --- > > Key: HAWQ-1243 > URL: https://issues.apache.org/jira/browse/HAWQ-1243 > Project: Apache HAWQ > Issue Type: Improvement >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: 2.2.0.0-incubating > > > Except rps_addr_host and rps_addr_port, we also need rps_addr_suffix to > We will use this GUC to construct rest service address: > http://rps_addr_host:rps_addr_port/rps_addr_suffix -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HAWQ-1243) Add suffix name for ranger restful service.
[ https://issues.apache.org/jira/browse/HAWQ-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang reassigned HAWQ-1243: -- Assignee: Hubert Zhang (was: Ed Espino) > Add suffix name for ranger restful service. > --- > > Key: HAWQ-1243 > URL: https://issues.apache.org/jira/browse/HAWQ-1243 > Project: Apache HAWQ > Issue Type: Improvement >Reporter: Hubert Zhang >Assignee: Hubert Zhang > > Except rps_addr_host and rps_addr_port, we also need rps_addr_suffix to > We will use this GUC to construct rest service address: > http://rps_addr_host:rps_addr_port/rps_addr_suffix -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HAWQ-1243) Add suffix name for ranger restful service.
Hubert Zhang created HAWQ-1243: -- Summary: Add suffix name for ranger restful service. Key: HAWQ-1243 URL: https://issues.apache.org/jira/browse/HAWQ-1243 Project: Apache HAWQ Issue Type: Improvement Reporter: Hubert Zhang Assignee: Ed Espino Except rps_addr_host and rps_addr_port, we also need rps_addr_suffix to We will use this GUC to construct rest service address: http://rps_addr_host:rps_addr_port/rps_addr_suffix -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (HAWQ-999) Log_error when file count is not in proportion to bucket number of hash table.
[ https://issues.apache.org/jira/browse/HAWQ-999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang closed HAWQ-999. - Resolution: Fixed > Log_error when file count is not in proportion to bucket number of hash table. > -- > > Key: HAWQ-999 > URL: https://issues.apache.org/jira/browse/HAWQ-999 > Project: Apache HAWQ > Issue Type: Improvement > Components: Core >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: 2.0.1.0-incubating > > > By definition, file count of a hash table should be equal to or a multiple of > the bucket number of the table. So if mismatch happens, we should not treat > it as hash table in data locality algorithm. > The default behaviour should LOG_ERROR and tell the user "file count is not > in proportion to bucket number for hash table with oid=X. Some data maybe > lost, if you still want to continue the query, set GUC > assign_randomly_for_mismatched_hash_table to true and try again." > If GUC assign_randomly_for_mismatched_hash_table is on, the behaviour changed > to treat the hash table as random table in data locality algorithm. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HAWQ-1039) Add test case of bucket number may not be consistent with parent table.
[ https://issues.apache.org/jira/browse/HAWQ-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang reassigned HAWQ-1039: -- Assignee: Hubert Zhang (was: Lei Chang) > Add test case of bucket number may not be consistent with parent table. > --- > > Key: HAWQ-1039 > URL: https://issues.apache.org/jira/browse/HAWQ-1039 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Core >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: 2.0.1.0-incubating > > > add test case for HAWQ-1032 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (HAWQ-1039) Add test case of bucket number may not be consistent with parent table.
[ https://issues.apache.org/jira/browse/HAWQ-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang closed HAWQ-1039. -- Resolution: Fixed > Add test case of bucket number may not be consistent with parent table. > --- > > Key: HAWQ-1039 > URL: https://issues.apache.org/jira/browse/HAWQ-1039 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Core >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: 2.0.1.0-incubating > > > add test case for HAWQ-1032 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (HAWQ-1032) Bucket number of newly added partition is not consistent with parent table.
[
https://issues.apache.org/jira/browse/HAWQ-1032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hubert Zhang closed HAWQ-1032.
--
Resolution: Fixed
> Bucket number of newly added partition is not consistent with parent table.
> ---
>
> Key: HAWQ-1032
> URL: https://issues.apache.org/jira/browse/HAWQ-1032
> Project: Apache HAWQ
> Issue Type: Bug
> Components: Core
>Reporter: Hubert Zhang
>Assignee: Hubert Zhang
> Fix For: 2.0.1.0-incubating
>
>
> Failure Case
> {code}
> set deafult_hash_table_bucket_number = 12;
> CREATE TABLE sales3 (id int, date date, amt decimal(10,2))
> DISTRIBUTED BY (id)
> PARTITION BY RANGE (date)
> ( START (date '2008-01-01') INCLUSIVE
>END (date '2009-01-01') EXCLUSIVE
>EVERY (INTERVAL '1 day') );
> set default_hash_table_bucket_number = 16;
> ALTER TABLE sales3 ADD PARTITION START
> (date '2009-03-01') INCLUSIVE END
> (date '2009-04-01') EXCLUSIVE;
> {code}
> The newly added partition with buckcet number 16 is not consistent with
> parent partition.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (HAWQ-1001) Implement HAWQ basic user ACL check through Ranger
[ https://issues.apache.org/jira/browse/HAWQ-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang closed HAWQ-1001. -- Resolution: Fixed > Implement HAWQ basic user ACL check through Ranger > -- > > Key: HAWQ-1001 > URL: https://issues.apache.org/jira/browse/HAWQ-1001 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Core >Reporter: Lili Ma >Assignee: Hubert Zhang > Fix For: backlog > > > When a user run some query, HAWQ can connect to Ranger to judge whether the > user has the privilege to do that. > For each object with unique oid, send one request to Ranger -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HAWQ-1220) Support ranger plugin server HA in hawq side.
[ https://issues.apache.org/jira/browse/HAWQ-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang reassigned HAWQ-1220: -- Assignee: Hubert Zhang (was: Ed Espino) > Support ranger plugin server HA in hawq side. > - > > Key: HAWQ-1220 > URL: https://issues.apache.org/jira/browse/HAWQ-1220 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: backlog > > > RPS will run both at master and at standby master, If connection to master > RPS failed, we should try to connect to standby master instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HAWQ-1220) Support ranger plugin server HA in hawq side.
Hubert Zhang created HAWQ-1220: -- Summary: Support ranger plugin server HA in hawq side. Key: HAWQ-1220 URL: https://issues.apache.org/jira/browse/HAWQ-1220 Project: Apache HAWQ Issue Type: Sub-task Components: Security Reporter: Hubert Zhang Assignee: Ed Espino RPS will run both at master and at standby master, If connection to master RPS failed, we should try to connect to standby master instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1003) Implement batched ACL check through Ranger.
[ https://issues.apache.org/jira/browse/HAWQ-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang updated HAWQ-1003: --- Summary: Implement batched ACL check through Ranger. (was: Implement bached ACL check through Ranger.) > Implement batched ACL check through Ranger. > --- > > Key: HAWQ-1003 > URL: https://issues.apache.org/jira/browse/HAWQ-1003 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Core >Reporter: Lili Ma >Assignee: hongwu > Fix For: backlog > > > Implement enhanced hawq ACL check through Ranger, which means, if a query > contains several tables, we can combine the multiple table request together, > to send just one REST request to Ranger REST API Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (HAWQ-1205) Change hawq start script once finding enable_ranger GUC is on.
[ https://issues.apache.org/jira/browse/HAWQ-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang closed HAWQ-1205. -- Resolution: Fixed > Change hawq start script once finding enable_ranger GUC is on. > -- > > Key: HAWQ-1205 > URL: https://issues.apache.org/jira/browse/HAWQ-1205 > Project: Apache HAWQ > Issue Type: Sub-task > Components: PXF, Security >Reporter: Lili Ma >Assignee: Lili Ma > Fix For: backlog > > > If hawq start finds enable_ranger GUC is on, it needs to start RPS service. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-1205) Change hawq start script once finding enable_ranger GUC is on.
[ https://issues.apache.org/jira/browse/HAWQ-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15741109#comment-15741109 ] Hubert Zhang commented on HAWQ-1205: Based on design doc, Hawq inits without ranger configured, so we can close this issue. > Change hawq start script once finding enable_ranger GUC is on. > -- > > Key: HAWQ-1205 > URL: https://issues.apache.org/jira/browse/HAWQ-1205 > Project: Apache HAWQ > Issue Type: Sub-task > Components: PXF, Security >Reporter: Lili Ma >Assignee: Lili Ma > Fix For: backlog > > > If hawq start finds enable_ranger GUC is on, it needs to start RPS service. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1206) Process catalog table ACL on Ranger.
[ https://issues.apache.org/jira/browse/HAWQ-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang updated HAWQ-1206: --- Summary: Process catalog table ACL on Ranger. (was: Process catalog table ACL on Ranger) > Process catalog table ACL on Ranger. > > > Key: HAWQ-1206 > URL: https://issues.apache.org/jira/browse/HAWQ-1206 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Lili Ma >Assignee: Lin Wen > Fix For: backlog > > > There are a lot of catalog tables in HAWQ which also need to go through ACL > check. We need find out how to process there tables once Ranger is configured. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1206) Process catalog table ACL on Ranger.
[ https://issues.apache.org/jira/browse/HAWQ-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang updated HAWQ-1206: --- Assignee: Hubert Zhang (was: Lin Wen) > Process catalog table ACL on Ranger. > > > Key: HAWQ-1206 > URL: https://issues.apache.org/jira/browse/HAWQ-1206 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Lili Ma >Assignee: Hubert Zhang > Fix For: backlog > > > There are a lot of catalog tables in HAWQ which also need to go through ACL > check. We need find out how to process there tables once Ranger is configured. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1002) Implement configuration GUC related to Ranger.
[ https://issues.apache.org/jira/browse/HAWQ-1002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang updated HAWQ-1002: --- Summary: Implement configuration GUC related to Ranger. (was: Implement a switch in hawq-site.xml to configure whether use Ranger or not for ACL) > Implement configuration GUC related to Ranger. > -- > > Key: HAWQ-1002 > URL: https://issues.apache.org/jira/browse/HAWQ-1002 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Core >Reporter: Lili Ma >Assignee: Hubert Zhang > Fix For: backlog > > > Implement a switch in hawq-site.xml to configure whether use Ranger or not > for ACL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1003) Implement bached ACL check through Ranger.
[ https://issues.apache.org/jira/browse/HAWQ-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang updated HAWQ-1003: --- Assignee: hongwu (was: Hubert Zhang) > Implement bached ACL check through Ranger. > -- > > Key: HAWQ-1003 > URL: https://issues.apache.org/jira/browse/HAWQ-1003 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Core >Reporter: Lili Ma >Assignee: hongwu > Fix For: backlog > > > Implement enhanced hawq ACL check through Ranger, which means, if a query > contains several tables, we can combine the multiple table request together, > to send just one REST request to Ranger REST API Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1003) Implement bached ACL check through Ranger.
[ https://issues.apache.org/jira/browse/HAWQ-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang updated HAWQ-1003: --- Summary: Implement bached ACL check through Ranger. (was: Implement enhanced hawq ACL check through Ranger) > Implement bached ACL check through Ranger. > -- > > Key: HAWQ-1003 > URL: https://issues.apache.org/jira/browse/HAWQ-1003 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Core >Reporter: Lili Ma >Assignee: Hubert Zhang > Fix For: backlog > > > Implement enhanced hawq ACL check through Ranger, which means, if a query > contains several tables, we can combine the multiple table request together, > to send just one REST request to Ranger REST API Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1004) Implement calling Ranger REST Service using libcurl
[ https://issues.apache.org/jira/browse/HAWQ-1004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang updated HAWQ-1004: --- Summary: Implement calling Ranger REST Service using libcurl (was: Implement calling Ranger REST Service -- use mock server) > Implement calling Ranger REST Service using libcurl > --- > > Key: HAWQ-1004 > URL: https://issues.apache.org/jira/browse/HAWQ-1004 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Core >Reporter: Lili Ma >Assignee: Lin Wen > Fix For: backlog > > > Decide How HAWQ connect Ranger, through which user, how to connect to REST > Server > Acceptance Criteria: > Provide an interface for HAWQ connecting Ranger REST Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1004) Implement calling Ranger REST Service using libcurl.
[ https://issues.apache.org/jira/browse/HAWQ-1004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang updated HAWQ-1004: --- Summary: Implement calling Ranger REST Service using libcurl. (was: Implement calling Ranger REST Service using libcurl) > Implement calling Ranger REST Service using libcurl. > > > Key: HAWQ-1004 > URL: https://issues.apache.org/jira/browse/HAWQ-1004 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Core >Reporter: Lili Ma >Assignee: Lin Wen > Fix For: backlog > > > Decide How HAWQ connect Ranger, through which user, how to connect to REST > Server > Acceptance Criteria: > Provide an interface for HAWQ connecting Ranger REST Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HAWQ-1171) Support upgrade for hawq register.
[ https://issues.apache.org/jira/browse/HAWQ-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang reassigned HAWQ-1171: -- Assignee: Hubert Zhang (was: Lei Chang) > Support upgrade for hawq register. > -- > > Key: HAWQ-1171 > URL: https://issues.apache.org/jira/browse/HAWQ-1171 > Project: Apache HAWQ > Issue Type: New Feature > Components: Core >Reporter: Hubert Zhang >Assignee: Hubert Zhang > > For Hawq register feature, we need to add some build-in functions to support > some catalog changes. This could be done by a hawqupgrade script. > User interface: > Hawq upgrade. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang updated HAWQ-256: -- Attachment: HAWQRangerSupportDesign_v0.3.pdf > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf, > HAWQRangerSupportDesign_v0.2.pdf, HAWQRangerSupportDesign_v0.3.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HAWQ-1143) Libhdfs create semantic is not consistent with posix standard.
Hubert Zhang created HAWQ-1143: -- Summary: Libhdfs create semantic is not consistent with posix standard. Key: HAWQ-1143 URL: https://issues.apache.org/jira/browse/HAWQ-1143 Project: Apache HAWQ Issue Type: Bug Components: libhdfs Reporter: Hubert Zhang Assignee: Lei Chang Open a file under posix standard, if o_create flag is set to true and the file exists, no side effect except O_EXCL is also be set true. Open a file in HDFS with hdfs::create flag will report errors if file exists. In libhdfs, the o_create flag is interpreted to hdfs::create, which leads to errors if file exists no matter O_EXCL is set or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HAWQ-1143) Libhdfs create semantic is not consistent with posix standard.
[ https://issues.apache.org/jira/browse/HAWQ-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang reassigned HAWQ-1143: -- Assignee: Hubert Zhang (was: Lei Chang) > Libhdfs create semantic is not consistent with posix standard. > -- > > Key: HAWQ-1143 > URL: https://issues.apache.org/jira/browse/HAWQ-1143 > Project: Apache HAWQ > Issue Type: Bug > Components: libhdfs >Reporter: Hubert Zhang >Assignee: Hubert Zhang > > Open a file under posix standard, if o_create flag is set to true and the > file exists, no side effect except O_EXCL is also be set true. > Open a file in HDFS with hdfs::create flag will report errors if file exists. > In libhdfs, the o_create flag is interpreted to hdfs::create, which leads to > errors if file exists no matter O_EXCL is set or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1135) MADlib: Raising exception leads to database connection termination.
[
https://issues.apache.org/jira/browse/HAWQ-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hubert Zhang updated HAWQ-1135:
---
Summary: MADlib: Raising exception leads to database connection
termination. (was: MADlib: Raising exception leads to database connection
termination)
> MADlib: Raising exception leads to database connection termination.
> ---
>
> Key: HAWQ-1135
> URL: https://issues.apache.org/jira/browse/HAWQ-1135
> Project: Apache HAWQ
> Issue Type: Bug
> Components: Core
>Reporter: Ming LI
>Assignee: Hubert Zhang
>
> MADlib tests on HAWQ 2.0 Nightly builds fails due to server terminating it's
> connection. The failed tests are testing for bad input by returning an
> exception on specific user inputs. These exceptions are raised cleanly in
> other platforms including HAWQ 2.0 and all Greenplum DBs.
> Reproduction Steps
> Install MADlib using the RPM and HAWQ install script.
> Run attached script (called hawq_2.0.1_test.sql)
> Current error message is
> {{
> server closed the connection unexpectedly
> This probably means the server terminated abnormally
> before or while processing the request.
> }}
> Expected error is
> {{
> ERROR: spiexceptions.InvalidParameterValue: Function
> "madlib.lmf_igd_transition(double precision[],integer,integer,double
> precision,double precision[],integer,integer,integer,double precision,double
> precision)": Invalid type conversion. Null where not expected.
> }}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HAWQ-1135) MADlib: Raising exception leads to database connection termination
[
https://issues.apache.org/jira/browse/HAWQ-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hubert Zhang reassigned HAWQ-1135:
--
Assignee: Hubert Zhang (was: Lei Chang)
> MADlib: Raising exception leads to database connection termination
> --
>
> Key: HAWQ-1135
> URL: https://issues.apache.org/jira/browse/HAWQ-1135
> Project: Apache HAWQ
> Issue Type: Bug
> Components: Core
>Reporter: Ming LI
>Assignee: Hubert Zhang
>
> MADlib tests on HAWQ 2.0 Nightly builds fails due to server terminating it's
> connection. The failed tests are testing for bad input by returning an
> exception on specific user inputs. These exceptions are raised cleanly in
> other platforms including HAWQ 2.0 and all Greenplum DBs.
> Reproduction Steps
> Install MADlib using the RPM and HAWQ install script.
> Run attached script (called hawq_2.0.1_test.sql)
> Current error message is
> {{
> server closed the connection unexpectedly
> This probably means the server terminated abnormally
> before or while processing the request.
> }}
> Expected error is
> {{
> ERROR: spiexceptions.InvalidParameterValue: Function
> "madlib.lmf_igd_transition(double precision[],integer,integer,double
> precision,double precision[],integer,integer,integer,double precision,double
> precision)": Invalid type conversion. Null where not expected.
> }}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HAWQ-1034) add --repair option for hawq register
[ https://issues.apache.org/jira/browse/HAWQ-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang updated HAWQ-1034: --- Assignee: hongwu (was: Hubert Zhang) > add --repair option for hawq register > - > > Key: HAWQ-1034 > URL: https://issues.apache.org/jira/browse/HAWQ-1034 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Command Line Tools >Affects Versions: 2.0.1.0-incubating >Reporter: Lili Ma >Assignee: hongwu > Fix For: 2.0.1.0-incubating > > > add --repair option for hawq register > Will change both file folder and catalog table pg_aoseg.pg_paqseg_$relid to > the state which .yml file configures. Note may some new generated files since > the checkpoint may be deleted here. Also note the all the files in .yml file > should all under the table folder on HDFS. Limitation: Do not support cases > for hash table redistribution, table truncate and table drop. This is for > scenario rollback of table: Do checkpoints somewhere, and need to rollback to > previous checkpoint. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HAWQ-1035) support partition table register
[ https://issues.apache.org/jira/browse/HAWQ-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang reassigned HAWQ-1035: -- Assignee: Hubert Zhang (was: hongwu) > support partition table register > > > Key: HAWQ-1035 > URL: https://issues.apache.org/jira/browse/HAWQ-1035 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Command Line Tools >Reporter: Lili Ma >Assignee: Hubert Zhang > Fix For: 2.0.1.0-incubating > > > Support partition table register, limited to 1 level partition table, since > hawq extract only supports 1-level partition table. > Expected behavior: > 1. Create a partition table in HAWQ, then extract the information out to .yml > file > 2. Call hawq register and specify identified .yml file and a new table name, > the files should be registered into the new table. > Work can be detailed down to implement partition table register: > 1. modify .yml configuration file parsing function, add content for partition > table. > 2. construct partition table DDL regards to .yml configuration file > 3. map sub partition table name to the table list in .yml configuration file > 4. register the subpartition table one by one -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HAWQ-1034) add --repair option for hawq register
[ https://issues.apache.org/jira/browse/HAWQ-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang reassigned HAWQ-1034: -- Assignee: Hubert Zhang (was: hongwu) > add --repair option for hawq register > - > > Key: HAWQ-1034 > URL: https://issues.apache.org/jira/browse/HAWQ-1034 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Command Line Tools >Affects Versions: 2.0.1.0-incubating >Reporter: Lili Ma >Assignee: Hubert Zhang > Fix For: 2.0.1.0-incubating > > > add --repair option for hawq register > Will change both file folder and catalog table pg_aoseg.pg_paqseg_$relid to > the state which .yml file configures. Note may some new generated files since > the checkpoint may be deleted here. Also note the all the files in .yml file > should all under the table folder on HDFS. Limitation: Do not support cases > for hash table redistribution, table truncate and table drop. This is for > scenario rollback of table: Do checkpoints somewhere, and need to rollback to > previous checkpoint. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HAWQ-1039) Add test case of bucket number may not be consistent with parent table.
Hubert Zhang created HAWQ-1039: -- Summary: Add test case of bucket number may not be consistent with parent table. Key: HAWQ-1039 URL: https://issues.apache.org/jira/browse/HAWQ-1039 Project: Apache HAWQ Issue Type: Sub-task Components: Core Reporter: Hubert Zhang Assignee: Lei Chang add test case for HAWQ-1032 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-1032) Bucket number of newly added partition is not consistent with parent table.
[
https://issues.apache.org/jira/browse/HAWQ-1032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15453989#comment-15453989
]
Hubert Zhang commented on HAWQ-1032:
not partition number, but bucket number of sub partition is not consistent with
bucket number of root partition.
In the above case, the bucket number of root partition is 12, while the new
added partition's bucket number is 16.
The query will failed if bucket numbers of every partitions are not the same
> Bucket number of newly added partition is not consistent with parent table.
> ---
>
> Key: HAWQ-1032
> URL: https://issues.apache.org/jira/browse/HAWQ-1032
> Project: Apache HAWQ
> Issue Type: Bug
> Components: Core
>Reporter: Hubert Zhang
>Assignee: Hubert Zhang
> Fix For: 2.0.1.0-incubating
>
>
> Failure Case
> {code}
> set deafult_hash_table_bucket_number = 12;
> CREATE TABLE sales3 (id int, date date, amt decimal(10,2))
> DISTRIBUTED BY (id)
> PARTITION BY RANGE (date)
> ( START (date '2008-01-01') INCLUSIVE
>END (date '2009-01-01') EXCLUSIVE
>EVERY (INTERVAL '1 day') );
> set deafult_hash_table_bucket_number = 16;
> ALTER TABLE sales3 ADD PARTITION START
> (date '2009-03-01') INCLUSIVE END
> (date '2009-04-01') EXCLUSIVE;
> {code}
> The newly added partition with buckcet number 16 is not consistent with
> parent partition.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15451720#comment-15451720 ] Hubert Zhang commented on HAWQ-256: --- +1 for two stage authorization. Hawq ranger plugin(REST service) manages the access privilege of hawq object, include database, table, function, language and so on. While HDFS ranger plugin manages the access privilege of hdfs file. They are not conflicted with each other. User must first have the privilege to access hawq object(calculated in planner), next user also need to have the privilege to access the hdfs file. Currently, hawq use the admin user to create/append hdfs file, this is convenient for hawq user management. For example, user A own table t1, and if user A grant select and insert privilege of table t1 to user B, user B can directly access table t1, because on HDFS, the files of table t1 are created and accessed both by admin. But user-identity passing down will lead to table t1 is created by user A and user B cannot access file directly, unless add user B to user A's group, or change the file privilege. I do agree "user-identity passing down" is useful especially in hadoop eco, but when implementing it, pay attention to the problem I mentioned above.(Also this is beyond the discussion of issue256) > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf, > HAWQRangerSupportDesign_v0.2.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HAWQ-1003) Implement enhanced hawq ACL check through Ranger
[ https://issues.apache.org/jira/browse/HAWQ-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang reassigned HAWQ-1003: -- Assignee: Hubert Zhang (was: Lei Chang) > Implement enhanced hawq ACL check through Ranger > > > Key: HAWQ-1003 > URL: https://issues.apache.org/jira/browse/HAWQ-1003 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Core >Reporter: Lili Ma >Assignee: Hubert Zhang > Fix For: backlog > > > Implement enhanced hawq ACL check through Ranger, which means, if a query > contains several tables, we can combine the multiple table request together, > to send just one REST request to Ranger REST API Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HAWQ-1002) Implement a switch in hawq-site.xml to configure whether use Ranger or not for ACL
[ https://issues.apache.org/jira/browse/HAWQ-1002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang reassigned HAWQ-1002: -- Assignee: Hubert Zhang (was: Lei Chang) > Implement a switch in hawq-site.xml to configure whether use Ranger or not > for ACL > -- > > Key: HAWQ-1002 > URL: https://issues.apache.org/jira/browse/HAWQ-1002 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Core >Reporter: Lili Ma >Assignee: Hubert Zhang > Fix For: backlog > > > Implement a switch in hawq-site.xml to configure whether use Ranger or not > for ACL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HAWQ-1032) Bucket number of new added partition is not consistent with parent table.
Hubert Zhang created HAWQ-1032: -- Summary: Bucket number of new added partition is not consistent with parent table. Key: HAWQ-1032 URL: https://issues.apache.org/jira/browse/HAWQ-1032 Project: Apache HAWQ Issue Type: Bug Components: Core Reporter: Hubert Zhang Assignee: Lei Chang Failure Case set deafult_hash_table_bucket_number = 12; CREATE TABLE sales3 (id int, date date, amt decimal(10,2)) DISTRIBUTED BY (id) PARTITION BY RANGE (date) ( START (date '2008-01-01') INCLUSIVEEND (date '2009-01-01') EXCLUSIVE EVERY (INTERVAL '1 day') ); set deafult_hash_table_bucket_number = 16; ALTER TABLE sales3 ADD PARTITION START (date '2009-03-01') INCLUSIVE END (date '2009-04-01') EXCLUSIVE; The new added partition with bukcet number 16 which is not consistent with parent partition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HAWQ-1032) Bucket number of new added partition is not consistent with parent table.
[ https://issues.apache.org/jira/browse/HAWQ-1032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang reassigned HAWQ-1032: -- Assignee: Hubert Zhang (was: Lei Chang) > Bucket number of new added partition is not consistent with parent table. > - > > Key: HAWQ-1032 > URL: https://issues.apache.org/jira/browse/HAWQ-1032 > Project: Apache HAWQ > Issue Type: Bug > Components: Core >Reporter: Hubert Zhang >Assignee: Hubert Zhang > > Failure Case > set deafult_hash_table_bucket_number = 12; > CREATE TABLE sales3 (id int, date date, amt decimal(10,2)) > DISTRIBUTED BY (id) > PARTITION BY RANGE (date) > ( START (date '2008-01-01') INCLUSIVE >END (date '2009-01-01') EXCLUSIVE >EVERY (INTERVAL '1 day') ); > set deafult_hash_table_bucket_number = 16; > ALTER TABLE sales3 ADD PARTITION START > (date '2009-03-01') INCLUSIVE END > (date '2009-04-01') EXCLUSIVE; > The new added partition with bukcet number 16 which is not consistent with > parent partition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang updated HAWQ-256: -- Attachment: HAWQRangerSupportDesign_v0.2.pdf > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf, > HAWQRangerSupportDesign_v0.2.pdf, HAWQRangerSupportDesign_v0.2.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang updated HAWQ-256: -- Comment: was deleted (was: Agree with [~vVineet] If enable_ranger is ON in Hawq, we should disable grant/revoke/reassign statement on CLI. Here reassign is a Hawq statement which changes the ownership of database objects owned by a database role to other role.) > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436134#comment-15436134 ] Hubert Zhang commented on HAWQ-256: --- Agree with [~vVineet] If enable_ranger is ON in Hawq, we should disable grant/revoke/reassign statement on CLI. Here reassign is a Hawq statement which changes the ownership of database objects owned by a database role to other role. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436135#comment-15436135 ] Hubert Zhang commented on HAWQ-256: --- Agree with [~vVineet] If enable_ranger is ON in Hawq, we should disable grant/revoke/reassign statement on CLI. Here reassign is a Hawq statement which changes the ownership of database objects owned by a database role to other role. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420803#comment-15420803 ] Hubert Zhang commented on HAWQ-256: --- i review the Ranger doc for ranger policy evaluation flow, It said that if no allow-condition matches the request, the access result will be undertermined. In this case, most components will deny the access. However, components like HDFS and YARN fallback to their native ACL to determine the access So I think it's also reasonable for HAWQ to follow this way. No record leads to unknown return value, and let HAWQ to handle the behaviour. Also, this suggest only works in Ranger0.6+ > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-999) Log_error when file count is not in proportion to bucket number of hash table.
[ https://issues.apache.org/jira/browse/HAWQ-999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang updated HAWQ-999: -- Description: By definition, file count of a hash table should be equal to or a multiple of the bucket number of the table. So if mismatch happens, we should not treat it as hash table in data locality algorithm. The default behaviour should LOG_ERROR and tell the user "file count is not in proportion to bucket number for hash table with oid=X. Some data maybe lost, if you still want to continue the query, set GUC assign_randomly_for_mismatched_hash_table to true and try again." If GUC assign_randomly_for_mismatched_hash_table is on, the behaviour changed to treat the hash table as random table in data locality algorithm. was: By definition, file count of a hash table should be equal to or a multiple of the bucket number of the table. So if mismatch happens, we should not treat it as hash table in data locality algorithm. The default behaviour should LOG_ERROR and tell the user "" > Log_error when file count is not in proportion to bucket number of hash table. > -- > > Key: HAWQ-999 > URL: https://issues.apache.org/jira/browse/HAWQ-999 > Project: Apache HAWQ > Issue Type: Improvement > Components: Core >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: 2.0.1.0-incubating > > > By definition, file count of a hash table should be equal to or a multiple of > the bucket number of the table. So if mismatch happens, we should not treat > it as hash table in data locality algorithm. > The default behaviour should LOG_ERROR and tell the user "file count is not > in proportion to bucket number for hash table with oid=X. Some data maybe > lost, if you still want to continue the query, set GUC > assign_randomly_for_mismatched_hash_table to true and try again." > If GUC assign_randomly_for_mismatched_hash_table is on, the behaviour changed > to treat the hash table as random table in data locality algorithm. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-999) Log_error when file count is not in proportion to bucket number of hash table.
[ https://issues.apache.org/jira/browse/HAWQ-999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang updated HAWQ-999: -- Description: By definition, file count of a hash table should be equal to or a multiple of the bucket number of the table. So if mismatch happens, we should not treat it as hash table in data locality algorithm. The default behaviour should LOG_ERROR and tell the user "" was: By definition, file count of a hash table should be equal to or a multiple of the bucket number of the table. So if mismatch happens, we should not treat it as hash table in data locality algorithm. > Log_error when file count is not in proportion to bucket number of hash table. > -- > > Key: HAWQ-999 > URL: https://issues.apache.org/jira/browse/HAWQ-999 > Project: Apache HAWQ > Issue Type: Improvement > Components: Core >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: 2.0.1.0-incubating > > > By definition, file count of a hash table should be equal to or a multiple of > the bucket number of the table. So if mismatch happens, we should not treat > it as hash table in data locality algorithm. > The default behaviour should LOG_ERROR and tell the user "" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-999) Log_error when file count is not in proportion to bucket number of hash table.
[ https://issues.apache.org/jira/browse/HAWQ-999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang updated HAWQ-999: -- Summary: Log_error when file count is not in proportion to bucket number of hash table. (was: Treat hash table as random when file count is not in proportion to bucket number of table.) > Log_error when file count is not in proportion to bucket number of hash table. > -- > > Key: HAWQ-999 > URL: https://issues.apache.org/jira/browse/HAWQ-999 > Project: Apache HAWQ > Issue Type: Improvement > Components: Core >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: 2.0.1.0-incubating > > > By definition, file count of a hash table should be equal to or a multiple of > the bucket number of the table. So if mismatch happens, we should not treat > it as hash table in data locality algorithm. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HAWQ-256) Integrate Security with Apache Ranger
[
https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420591#comment-15420591
]
Hubert Zhang edited comment on HAWQ-256 at 8/15/16 3:42 AM:
HAWQ support two kinds of ACL check when there are more than one privilege in
one request.
One is ALL, which means that all the privileges are required to be allowed.
The other one is ANY, which means that one of the required privileges to be
allowed is sufficient.
{code}
{
“requestor” : “u1”,
[
{
"resource" : {“TABLE”: “t1”, “DATABASE”: “db1”},
"privilege" : ["select", "insert"]
"how": ALL
},
{
"resource" : {“TABLE”: “t2”, “DATABASE”: “db1”},
"privilege" : ["select"]
"how": ANY
}
]
}
{code}
was (Author: hubertzhang):
HAWQ support two kinds of ACL check when there are more than one privilege in
one request.
One is ALL, which means that all the privileges are required to be allowed.
The other one is ANY, which means that one of the required privileges to be
allowed is sufficient.
{code}
{
“requestor” : “u1”,
[
{
"resource" : {“TABLE”: “t1”, “DATABASE”: “db1”},
"privilege" : [“select”, "insert"]
"how": ALL
},
{
“resource” : {“TABLE”: “t2”, “DATABASE”: “db1”},
“privilege” : [“select”]
"how": ANY
}
]
}
{code}
> Integrate Security with Apache Ranger
> -
>
> Key: HAWQ-256
> URL: https://issues.apache.org/jira/browse/HAWQ-256
> Project: Apache HAWQ
> Issue Type: New Feature
> Components: PXF, Security
>Reporter: Michael Andre Pearce (IG)
>Assignee: Lili Ma
> Fix For: backlog
>
> Attachments: HAWQRangerSupportDesign.pdf
>
>
> Integrate security with Apache Ranger for a unified Hadoop security solution.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420550#comment-15420550 ] Hubert Zhang edited comment on HAWQ-256 at 8/15/16 1:40 AM: [~bosco]You suggest to replace Application(HAWQ) default ACL from application internal to Ranger policy too. It's another way to handle this problem. I have two questions: 1 Group "public" is a default internal type group in Ranger, I wonder whether the users synced from ldap belong to group "public" automatically? 2 Ranger 0.6 introduces deny condition, So even if a user inherits default privilege from group "public", It can also be revoked with specific deny operation. But how about Ranger 0.5? One method is to remove the user from "public" group. But the limitation of this method is that we have to create "public" group for each resource, and assign the new users(synced from ldap) to these groups. was (Author: hubertzhang): [~bosco]You suggest to replace Application(HAWQ) default ACL from application internal to Ranger policy too. It's another way to handle this problem. I have two questions: 1 Group "public" is a default internal type group in Ranger, I wonder whether the users synced from ldap belong to group "public" automatically? 2 Ranger 0.6 introduces deny condition, So even if a user inherits default privilege from group "public", It can also be revoked with specific deny operation. But how about Ranger 0.5? One method is to remove the user from "public" group. But the limitation of this method is that we have to create "public" group for each resource, and assign the new users(synced from ldap) to this group. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420550#comment-15420550 ] Hubert Zhang edited comment on HAWQ-256 at 8/15/16 1:40 AM: [~bosco]You suggest to replace Application(HAWQ) default ACL from application internal to Ranger policy too. It's another way to handle this problem. I have two questions: 1 Group "public" is a default internal type group in Ranger, I wonder whether the users synced from ldap belong to group "public" automatically? 2 Ranger 0.6 introduces deny condition, So even if a user inherits default privilege from group "public", It can also be revoked with specific deny operation. But how about Ranger 0.5? One method is to remove the user from "public" group. But the limitation of this method is that we have to create "public" group for each resource, and assign the new users(synced from ldap) to this group. was (Author: hubertzhang): [~bosco]You suggest to replace Application(HAWQ) default ACL from application internal to Ranger policy too. It's another way to handle this problem. I have two questions: 1 Group "public" is a default internal type group in Ranger, I wonder whether the users synced from ldap belong to group "public" 2 Ranger 0.6 introduces deny condition, So even if a user inherits default privilege from group "public", It can also be revoked with specific deny operation. But how about Ranger 0.5? One method is to remove the user from "public" group. But the limitation of this method is that we have to create "public" group for each resource, and assign the new users(synced from ldap) to this group. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418598#comment-15418598 ] Hubert Zhang commented on HAWQ-256: --- [~bosco] [~vineetgoel] [~lilima][~wlin] We revisited the HAWQ aclcheck related code, and found that in HAWQ if there is no acl information stored in an object(database, table...) catalog, different objects will have different default behaviours to do acl check. But in our original REST-API design, the checkPrivileges API return bool(allow or deny) which lacks a state of unknown to do default acl check. Here 'unknown' means there is no record in ranger about this request. So we propose two ways to handle this issue. 1 Set the type of return value of checkPrivileges to integer, 0 for deny, 1 for allow, 2 for unknown 2 Add another REST-API, isPrivilegeExist(). Any suggestion for which solution is better ? > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HAWQ-999) Treat hash table as random when file count is not in proportion to bucket number of table.
[ https://issues.apache.org/jira/browse/HAWQ-999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang updated HAWQ-999: -- Comment: was deleted (was: [~jianlirong] We need to investigate why bucket number and file count mismatch happens. This JIRA is just to ensure even when mismatch happens, The query will not failed(of course catalog and physical file information must be consistent)) > Treat hash table as random when file count is not in proportion to bucket > number of table. > -- > > Key: HAWQ-999 > URL: https://issues.apache.org/jira/browse/HAWQ-999 > Project: Apache HAWQ > Issue Type: Improvement > Components: Core >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: 2.0.1.0-incubating > > > By definition, file count of a hash table should be equal to or a multiple of > the bucket number of the table. So if mismatch happens, we should not treat > it as hash table in data locality algorithm. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-999) Treat hash table as random when file count is not in proportion to bucket number of table.
[ https://issues.apache.org/jira/browse/HAWQ-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418251#comment-15418251 ] Hubert Zhang commented on HAWQ-999: --- [~jianlirong] We need to investigate why bucket number and file count mismatch happens. This JIRA is just to ensure even when mismatch happens, The query will not failed(of course catalog and physical file information must be consistent) > Treat hash table as random when file count is not in proportion to bucket > number of table. > -- > > Key: HAWQ-999 > URL: https://issues.apache.org/jira/browse/HAWQ-999 > Project: Apache HAWQ > Issue Type: Improvement > Components: Core >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: 2.0.1.0-incubating > > > By definition, file count of a hash table should be equal to or a multiple of > the bucket number of the table. So if mismatch happens, we should not treat > it as hash table in data locality algorithm. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-999) Treat hash table as random when file count is not in proportion to bucket number of table.
[ https://issues.apache.org/jira/browse/HAWQ-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418252#comment-15418252 ] Hubert Zhang commented on HAWQ-999: --- Lirong Jian We need to investigate why bucket number and file count mismatch happens. This JIRA is just to ensure even when mismatch happens, The query will not failed(of course catalog and physical file information must be consistent) > Treat hash table as random when file count is not in proportion to bucket > number of table. > -- > > Key: HAWQ-999 > URL: https://issues.apache.org/jira/browse/HAWQ-999 > Project: Apache HAWQ > Issue Type: Improvement > Components: Core >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: 2.0.1.0-incubating > > > By definition, file count of a hash table should be equal to or a multiple of > the bucket number of the table. So if mismatch happens, we should not treat > it as hash table in data locality algorithm. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HAWQ-999) Treat hash table as random when file count is not in proportion to bucket number of table.
Hubert Zhang created HAWQ-999: - Summary: Treat hash table as random when file count is not in proportion to bucket number of table. Key: HAWQ-999 URL: https://issues.apache.org/jira/browse/HAWQ-999 Project: Apache HAWQ Issue Type: Improvement Components: Core Reporter: Hubert Zhang Assignee: Lei Chang By definition, file count of a hash table should be equal to or a multiple of the bucket number of the table. So if mismatch happens, we should not treat it as hash table in data locality algorithm. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HAWQ-999) Treat hash table as random when file count is not in proportion to bucket number of table.
[ https://issues.apache.org/jira/browse/HAWQ-999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang reassigned HAWQ-999: - Assignee: Hubert Zhang (was: Lei Chang) > Treat hash table as random when file count is not in proportion to bucket > number of table. > -- > > Key: HAWQ-999 > URL: https://issues.apache.org/jira/browse/HAWQ-999 > Project: Apache HAWQ > Issue Type: Improvement > Components: Core >Reporter: Hubert Zhang >Assignee: Hubert Zhang > > By definition, file count of a hash table should be equal to or a multiple of > the bucket number of the table. So if mismatch happens, we should not treat > it as hash table in data locality algorithm. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HAWQ-256) Integrate Security with Apache Ranger
[
https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411542#comment-15411542
]
Hubert Zhang edited comment on HAWQ-256 at 8/8/16 9:07 AM:
---
Agree. We can use JSON array to represent it.
{code}
{
“requestor” : “u1”,
[
{
“resource” : {“TABLE”: “t1”, “DATABASE”: “db1”},
“privilege” : [“select”, "insert"]
},
{
“resource” : {“TABLE”: “t2”, “DATABASE”: “db1”},
“privilege” : [“select”]
}
]
}
{code}
was (Author: hubertzhang):
Agree. We can use JSON array to represent it.
{
“requestor” : “u1”,
[
{
“resource” : {“TABLE”: “t1”, “DATABASE”: “db1”},
“privilege” : [“select”, "insert"]
},
{
“resource” : {“TABLE”: “t2”, “DATABASE”: “db1”},
“privilege” : [“select”]
}
]
}
> Integrate Security with Apache Ranger
> -
>
> Key: HAWQ-256
> URL: https://issues.apache.org/jira/browse/HAWQ-256
> Project: Apache HAWQ
> Issue Type: New Feature
> Components: PXF, Security
>Reporter: Michael Andre Pearce (IG)
>Assignee: Lili Ma
> Fix For: backlog
>
> Attachments: HAWQRangerSupportDesign.pdf
>
>
> Integrate security with Apache Ranger for a unified Hadoop security solution.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[
https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411542#comment-15411542
]
Hubert Zhang commented on HAWQ-256:
---
Agree. We can use JSON array to represent it.
{
“requestor” : “u1”,
[
{
“resource” : {“TABLE”: “t1”, “DATABASE”: “db1”},
“privilege” : [“select”, "insert"]
},
{
“resource” : {“TABLE”: “t2”, “DATABASE”: “db1”},
“privilege” : [“select”]
}
]
}
> Integrate Security with Apache Ranger
> -
>
> Key: HAWQ-256
> URL: https://issues.apache.org/jira/browse/HAWQ-256
> Project: Apache HAWQ
> Issue Type: New Feature
> Components: PXF, Security
>Reporter: Michael Andre Pearce (IG)
>Assignee: Lili Ma
> Fix For: backlog
>
> Attachments: HAWQRangerSupportDesign.pdf
>
>
> Integrate security with Apache Ranger for a unified Hadoop security solution.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382037#comment-15382037 ] Hubert Zhang commented on HAWQ-256: --- [~bosco] [~Lili Ma] I skim the Authorizer code in Hive and Ranger. I found that the behaviour of RangerHiveAuthorizer is limited. In detail, RangerHiveAuthorizer is subclass of HiveAuthorizer(abstract interface), but it only implemented a subset of member functions. I conclude them as follows: Implemented in ranger: grantPrivileges revokePrivileges checkPrivileges applyRowFilterAndColumnMasking needTransform filterListCmdObjects Not implemented in ranger: createRole dropRole getPrincipalGrantInfoForRole getRoleGrantInfoForPrincipal grantRole revokeRole getAllRoles showPrivileges getCurrentRoleNames setCurrentRole applyAuthorizationConfigPolicy getHiveAuthorizationTranslator So could I get a conclusion that when user config Hive with Ranger enabled, uses cannot create or drop roles in Hive? > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
