[jira] [Commented] (HAWQ-1212) Upgrade libhdfs3 with upstream hadoop/hdp
[ https://issues.apache.org/jira/browse/HAWQ-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15743616#comment-15743616 ] hongwu commented on HAWQ-1212: -- Got, thanks. > Upgrade libhdfs3 with upstream hadoop/hdp > - > > Key: HAWQ-1212 > URL: https://issues.apache.org/jira/browse/HAWQ-1212 > Project: Apache HAWQ > Issue Type: New Feature > Components: libhdfs >Reporter: hongwu >Assignee: Lei Chang > > Since current libhdfs3 implementation is based on old version of HDFS, it is > necessary to upgrade libhdfs3 with latest HDFS of lots of new features and > configuration parameters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-1212) Upgrade libhdfs3 with upstream hadoop/hdp
[ https://issues.apache.org/jira/browse/HAWQ-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15743612#comment-15743612 ] Roman Shaposhnik commented on HAWQ-1212: My plan is to help you with mentorship required to merge it with Apache Hadoop codebase -- somebody in HAWQ will still have to commit to doing all the actual work. > Upgrade libhdfs3 with upstream hadoop/hdp > - > > Key: HAWQ-1212 > URL: https://issues.apache.org/jira/browse/HAWQ-1212 > Project: Apache HAWQ > Issue Type: New Feature > Components: libhdfs >Reporter: hongwu >Assignee: Lei Chang > > Since current libhdfs3 implementation is based on old version of HDFS, it is > necessary to upgrade libhdfs3 with latest HDFS of lots of new features and > configuration parameters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-1202) docs - add PXF server configuration parameters to reference
[ https://issues.apache.org/jira/browse/HAWQ-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15743475#comment-15743475 ] ASF GitHub Bot commented on HAWQ-1202: -- Github user dyozie commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/70#discussion_r92063656 --- Diff: reference/guc/parameter_definitions.html.md.erb --- @@ -2740,9 +2754,26 @@ Determines whether `ANALYZE` collects statistics for readable PXF tables. If tru |-|-|-| | Boolean | true| master, session, reload | +## pxf\_remote\_service\_login + +Temporary configuration parameter identifying the login credential to forward to remote services requiring authentication. + +| Value Range | Default | Set Classifications | +|-|-|-| +| string | | master, session, reload | + +## pxf\_remote\_service\_secret + +Temporary configuration parameter identifying the password credential to forward to remote services requiring authentication. + +| Value Range | Default | Set Classifications | +|-|-|-| +| string | | master, session, reload | + --- End diff -- What does "temporary" mean for the above two parameters? > docs - add PXF server configuration parameters to reference > --- > > Key: HAWQ-1202 > URL: https://issues.apache.org/jira/browse/HAWQ-1202 > Project: Apache HAWQ > Issue Type: Improvement > Components: Documentation >Reporter: Lisa Owen >Assignee: David Yozie >Priority: Minor > > certain PXF server configuration parameters are referenced in the > documentation. most of these are not included in the server configuration > parameter reference. add them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1215) PXF HiveORC profile doesn't handle complex types correctly
[ https://issues.apache.org/jira/browse/HAWQ-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivram Mani updated HAWQ-1215: --- Affects Version/s: 2.0.0.0-incubating > PXF HiveORC profile doesn't handle complex types correctly > -- > > Key: HAWQ-1215 > URL: https://issues.apache.org/jira/browse/HAWQ-1215 > Project: Apache HAWQ > Issue Type: Bug > Components: PXF >Affects Versions: 2.0.0.0-incubating >Reporter: Shivram Mani >Assignee: Shivram Mani > > The new HiveORC profile has an issue with handling complex hive types > (array,map,struct,union,etc). The object inspector being used marks all these > complex types as string and hence during resolution time, PXF treats them as > primitive data types and fails. > We get the following exception > {code} > 2016-12-12 10:13:37.0579 DEBUG tomcat-http--13 > org.apache.hawq.pxf.service.rest.BridgeResource - Starting streaming fragment > 0 of resource /hive/warehouse/hive_collections_table_orc/00_0 > 2016-12-12 10:13:37.0580 ERROR tomcat-http--13 > org.apache.hawq.pxf.service.rest.BridgeResource - Exception thrown when > streaming > java.lang.ClassCastException: java.util.ArrayList cannot be cast to > org.apache.hadoop.io.Text > at > org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.getPrimitiveJavaObject(WritableStringObjectInspector.java:46) > at > org.apache.hawq.pxf.plugins.hive.HiveResolver.resolvePrimitive(HiveResolver.java:563) > at > org.apache.hawq.pxf.plugins.hive.HiveResolver.traverseTuple(HiveResolver.java:368) > at > org.apache.hawq.pxf.plugins.hive.HiveResolver.traverseStruct(HiveResolver.java:470) > at > org.apache.hawq.pxf.plugins.hive.HiveORCSerdeResolver.getFields(HiveORCSerdeResolver.java:81) > at org.apache.hawq.pxf.service.ReadBridge.getNext(ReadBridge.java:104) > at > org.apache.hawq.pxf.service.rest.BridgeResource$1.write(BridgeResource.java:140) > {code} > HiveORC profile uses the column types from the schema definition in HAWQ. > Complex fields are defined as text in HAWQ and hence is treated as string and > results in this error. This should be modified to use the schema definition > from Fragmenter metadata instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1215) PXF HiveORC profile doesn't handle complex types correctly
[ https://issues.apache.org/jira/browse/HAWQ-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivram Mani updated HAWQ-1215: --- Description: The new HiveORC profile has an issue with handling complex hive types (array,map,struct,union,etc). The object inspector being used marks all these complex types as string and hence during resolution time, PXF treats them as primitive data types and fails. We get the following exception {code} 2016-12-12 10:13:37.0579 DEBUG tomcat-http--13 org.apache.hawq.pxf.service.rest.BridgeResource - Starting streaming fragment 0 of resource /hive/warehouse/hive_collections_table_orc/00_0 2016-12-12 10:13:37.0580 ERROR tomcat-http--13 org.apache.hawq.pxf.service.rest.BridgeResource - Exception thrown when streaming java.lang.ClassCastException: java.util.ArrayList cannot be cast to org.apache.hadoop.io.Text at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.getPrimitiveJavaObject(WritableStringObjectInspector.java:46) at org.apache.hawq.pxf.plugins.hive.HiveResolver.resolvePrimitive(HiveResolver.java:563) at org.apache.hawq.pxf.plugins.hive.HiveResolver.traverseTuple(HiveResolver.java:368) at org.apache.hawq.pxf.plugins.hive.HiveResolver.traverseStruct(HiveResolver.java:470) at org.apache.hawq.pxf.plugins.hive.HiveORCSerdeResolver.getFields(HiveORCSerdeResolver.java:81) at org.apache.hawq.pxf.service.ReadBridge.getNext(ReadBridge.java:104) at org.apache.hawq.pxf.service.rest.BridgeResource$1.write(BridgeResource.java:140) {code} HiveORC profile uses the column types from the schema definition in HAWQ. Complex fields are defined as text in HAWQ and hence is treated as string and results in this error. This should be modified to use the schema definition from Fragmenter metadata instead. was: The new HiveORC profile has an issue with handling complex hive types (array,map,struct,union,etc). The object inspector being used marks all these complex types as string and hence during resolution time, PXF treats them as primitive data types and fails. We get the following exception {code} 2016-12-12 10:13:37.0579 DEBUG tomcat-http--13 org.apache.hawq.pxf.service.rest.BridgeResource - Starting streaming fragment 0 of resource /hive/warehouse/hive_collections_table_orc/00_0 2016-12-12 10:13:37.0580 ERROR tomcat-http--13 org.apache.hawq.pxf.service.rest.BridgeResource - Exception thrown when streaming java.lang.ClassCastException: java.util.ArrayList cannot be cast to org.apache.hadoop.io.Text at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.getPrimitiveJavaObject(WritableStringObjectInspector.java:46) at org.apache.hawq.pxf.plugins.hive.HiveResolver.resolvePrimitive(HiveResolver.java:563) at org.apache.hawq.pxf.plugins.hive.HiveResolver.traverseTuple(HiveResolver.java:368) at org.apache.hawq.pxf.plugins.hive.HiveResolver.traverseStruct(HiveResolver.java:470) at org.apache.hawq.pxf.plugins.hive.HiveORCSerdeResolver.getFields(HiveORCSerdeResolver.java:81) at org.apache.hawq.pxf.service.ReadBridge.getNext(ReadBridge.java:104) at org.apache.hawq.pxf.service.rest.BridgeResource$1.write(BridgeResource.java:140) {code} > PXF HiveORC profile doesn't handle complex types correctly > -- > > Key: HAWQ-1215 > URL: https://issues.apache.org/jira/browse/HAWQ-1215 > Project: Apache HAWQ > Issue Type: Bug > Components: PXF >Reporter: Shivram Mani >Assignee: Lei Chang > > The new HiveORC profile has an issue with handling complex hive types > (array,map,struct,union,etc). The object inspector being used marks all these > complex types as string and hence during resolution time, PXF treats them as > primitive data types and fails. > We get the following exception > {code} > 2016-12-12 10:13:37.0579 DEBUG tomcat-http--13 > org.apache.hawq.pxf.service.rest.BridgeResource - Starting streaming fragment > 0 of resource /hive/warehouse/hive_collections_table_orc/00_0 > 2016-12-12 10:13:37.0580 ERROR tomcat-http--13 > org.apache.hawq.pxf.service.rest.BridgeResource - Exception thrown when > streaming > java.lang.ClassCastException: java.util.ArrayList cannot be cast to > org.apache.hadoop.io.Text > at > org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.getPrimitiveJavaObject(WritableStringObjectInspector.java:46) > at > org.apache.hawq.pxf.plugins.hive.HiveResolver.resolvePrimitive(HiveResolver.java:563) > at > org.apache.hawq.pxf.plugins.hive.HiveResolver.traverseTuple(HiveResolver.java:368) > at > org.apache.hawq.pxf.plugins.hive.HiveResolver.traverseStruct(HiveResolver.java:470) > at > org.apache.hawq.pxf.plugins.hive.HiveORCSerdeResolver.getFields(HiveORCSe
[jira] [Assigned] (HAWQ-1215) PXF HiveORC profile doesn't handle complex types correctly
[ https://issues.apache.org/jira/browse/HAWQ-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivram Mani reassigned HAWQ-1215: -- Assignee: Shivram Mani (was: Lei Chang) > PXF HiveORC profile doesn't handle complex types correctly > -- > > Key: HAWQ-1215 > URL: https://issues.apache.org/jira/browse/HAWQ-1215 > Project: Apache HAWQ > Issue Type: Bug > Components: PXF >Reporter: Shivram Mani >Assignee: Shivram Mani > > The new HiveORC profile has an issue with handling complex hive types > (array,map,struct,union,etc). The object inspector being used marks all these > complex types as string and hence during resolution time, PXF treats them as > primitive data types and fails. > We get the following exception > {code} > 2016-12-12 10:13:37.0579 DEBUG tomcat-http--13 > org.apache.hawq.pxf.service.rest.BridgeResource - Starting streaming fragment > 0 of resource /hive/warehouse/hive_collections_table_orc/00_0 > 2016-12-12 10:13:37.0580 ERROR tomcat-http--13 > org.apache.hawq.pxf.service.rest.BridgeResource - Exception thrown when > streaming > java.lang.ClassCastException: java.util.ArrayList cannot be cast to > org.apache.hadoop.io.Text > at > org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.getPrimitiveJavaObject(WritableStringObjectInspector.java:46) > at > org.apache.hawq.pxf.plugins.hive.HiveResolver.resolvePrimitive(HiveResolver.java:563) > at > org.apache.hawq.pxf.plugins.hive.HiveResolver.traverseTuple(HiveResolver.java:368) > at > org.apache.hawq.pxf.plugins.hive.HiveResolver.traverseStruct(HiveResolver.java:470) > at > org.apache.hawq.pxf.plugins.hive.HiveORCSerdeResolver.getFields(HiveORCSerdeResolver.java:81) > at org.apache.hawq.pxf.service.ReadBridge.getNext(ReadBridge.java:104) > at > org.apache.hawq.pxf.service.rest.BridgeResource$1.write(BridgeResource.java:140) > {code} > HiveORC profile uses the column types from the schema definition in HAWQ. > Complex fields are defined as text in HAWQ and hence is treated as string and > results in this error. This should be modified to use the schema definition > from Fragmenter metadata instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HAWQ-1215) PXF HiveORC profile doesn't handle complex types correctly
Shivram Mani created HAWQ-1215: -- Summary: PXF HiveORC profile doesn't handle complex types correctly Key: HAWQ-1215 URL: https://issues.apache.org/jira/browse/HAWQ-1215 Project: Apache HAWQ Issue Type: Bug Components: PXF Reporter: Shivram Mani Assignee: Lei Chang The new HiveORC profile has an issue with handling complex hive types (array,map,struct,union,etc). The object inspector being used marks all these complex types as string and hence during resolution time, PXF treats them as primitive data types and fails. We get the following exception {code} 2016-12-12 10:13:37.0579 DEBUG tomcat-http--13 org.apache.hawq.pxf.service.rest.BridgeResource - Starting streaming fragment 0 of resource /hive/warehouse/hive_collections_table_orc/00_0 2016-12-12 10:13:37.0580 ERROR tomcat-http--13 org.apache.hawq.pxf.service.rest.BridgeResource - Exception thrown when streaming java.lang.ClassCastException: java.util.ArrayList cannot be cast to org.apache.hadoop.io.Text at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.getPrimitiveJavaObject(WritableStringObjectInspector.java:46) at org.apache.hawq.pxf.plugins.hive.HiveResolver.resolvePrimitive(HiveResolver.java:563) at org.apache.hawq.pxf.plugins.hive.HiveResolver.traverseTuple(HiveResolver.java:368) at org.apache.hawq.pxf.plugins.hive.HiveResolver.traverseStruct(HiveResolver.java:470) at org.apache.hawq.pxf.plugins.hive.HiveORCSerdeResolver.getFields(HiveORCSerdeResolver.java:81) at org.apache.hawq.pxf.service.ReadBridge.getNext(ReadBridge.java:104) at org.apache.hawq.pxf.service.rest.BridgeResource$1.write(BridgeResource.java:140) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-1205) Change hawq start script once finding enable_ranger GUC is on.
[ https://issues.apache.org/jira/browse/HAWQ-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15742675#comment-15742675 ] Alexander Denissov commented on HAWQ-1205: -- I think there are 2 points here: 1. HAWQ init() should run only if Ranger GUC is turned off. To this extend, if init() finds the GUC is to be ON, it should stop and error out with the proper message. 2. HAWQ start() should analyze the GUC and start RPS if the GUC is ON, which is what this story is about. Why is it closed ? > Change hawq start script once finding enable_ranger GUC is on. > -- > > Key: HAWQ-1205 > URL: https://issues.apache.org/jira/browse/HAWQ-1205 > Project: Apache HAWQ > Issue Type: Sub-task > Components: PXF, Security >Reporter: Lili Ma >Assignee: Lili Ma > Fix For: backlog > > > If hawq start finds enable_ranger GUC is on, it needs to start RPS service. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-1210) Documentation regarding usage of libhdfs3 in concurrent environment
[ https://issues.apache.org/jira/browse/HAWQ-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15742603#comment-15742603 ] William Forson commented on HAWQ-1210: -- Hi Zhanwei Wang, Unfortunately, I don't think I will have the bandwidth to debug this further for at least a few weeks. So far, I've been using libhdfs3 as a black-box component (i.e. I've really only looked at {{hdfs.h}} and build logic), so I will have to get myself up to speed on the basic organization of the codebase, etc. However, since there is a decent chance I will be using libhdfs3 as a production dependency, in a multi-threaded environment, I would definitely like to understand what is going on here. So I will try to look into this as soon as I have the time. Thanks! > Documentation regarding usage of libhdfs3 in concurrent environment > --- > > Key: HAWQ-1210 > URL: https://issues.apache.org/jira/browse/HAWQ-1210 > Project: Apache HAWQ > Issue Type: Bug > Components: libhdfs >Reporter: William Forson >Assignee: Lei Chang > Attachments: hdfs_fs_concurrent_test.cpp > > > Hi, > I've been using libhdfs3 in a single-threaded environment for several months > now, without any problems. However, as soon as I tried using the library > concurrently from multiple threads: hello, segfaults. > Although the source of these segfaults is annoyingly subtle, I've managed to > isolate it to a relatively small block of my code that does nothing > interesting aside from using libhdfs3 to download a single hdfs file. > To be clear: I assume that the mistake here is mine -- that is, that I am > using your library incorrectly. However, I have been unable to find any > documentation as to how the libhdfs3 API _should_ be used in a multi-threaded > environment. I initially interpreted this to mean, "go to town, it's all more > or less thread-safe", but I am now questioning that interpretation. > So, I have a question, and a request. > Question: Are there any known, non-obvious concurrency gotchas regarding the > usage of libhdfs3 (or whatever it's currently called)? > Request: Could you please add some documentation, to the README and/or > hdfs.h, regarding usage in a concurrent environment? (ideally, such notes > would annotate individual components of the API in hdfs.h, but if the answer > to my question above is, "No", then this could perhaps be a single sentence > in the README which affirmatively states that the library is generally safe > for concurrent usage without additional/explicit synchronization -- anything > would be better than nothing :)) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-hawq issue #1040: HAWQ-1195. Fixed error "Two or more external tab...
Github user liming01 commented on the issue: https://github.com/apache/incubator-hawq/pull/1040 When we found the same error table used for the same query, we should assign a new sets of seg files for INSERT INTO ERROR TABLE, which means different relation will insert into the different sets of segfiles. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (HAWQ-1214) Remove resource_parameters
[ https://issues.apache.org/jira/browse/HAWQ-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Guo updated HAWQ-1214: --- Summary: Remove resource_parameters (was: Remove resource_parameters in calculate_planner_segment_num()) > Remove resource_parameters > -- > > Key: HAWQ-1214 > URL: https://issues.apache.org/jira/browse/HAWQ-1214 > Project: Apache HAWQ > Issue Type: Bug >Reporter: Paul Guo >Assignee: Paul Guo > > The reasons are: > 1) It is not used anymore. > 2) Have seen crash due to this. The scenario is that: > > The datalocality memory context on which resource_parameters is palloc-ed is > reset before calculate_planner_segment_num() finishes, thus later access to > resource_parameters could lead to crash or wrong results. A code path example > which could lead to segfault is: > ProcessUtility()->PerformCursorOpen()->_copyPlannedStmt()->_copyQueryResourceParameters() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HAWQ-1214) Remove resource_parameters in calculate_planner_segment_num()
Paul Guo created HAWQ-1214: -- Summary: Remove resource_parameters in calculate_planner_segment_num() Key: HAWQ-1214 URL: https://issues.apache.org/jira/browse/HAWQ-1214 Project: Apache HAWQ Issue Type: Bug Reporter: Paul Guo Assignee: Lei Chang The reasons are: 1) It is not used anymore. 2) Have seen crash due to this. The scenario is that: The datalocality memory context on which resource_parameters is palloc-ed is reset before calculate_planner_segment_num() finishes, thus later access to resource_parameters could lead to crash or wrong results. A code path example which could lead to segfault is: ProcessUtility()->PerformCursorOpen()->_copyPlannedStmt()->_copyQueryResourceParameters() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HAWQ-1214) Remove resource_parameters in calculate_planner_segment_num()
[ https://issues.apache.org/jira/browse/HAWQ-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Guo reassigned HAWQ-1214: -- Assignee: Paul Guo (was: Lei Chang) > Remove resource_parameters in calculate_planner_segment_num() > - > > Key: HAWQ-1214 > URL: https://issues.apache.org/jira/browse/HAWQ-1214 > Project: Apache HAWQ > Issue Type: Bug >Reporter: Paul Guo >Assignee: Paul Guo > > The reasons are: > 1) It is not used anymore. > 2) Have seen crash due to this. The scenario is that: > > The datalocality memory context on which resource_parameters is palloc-ed is > reset before calculate_planner_segment_num() finishes, thus later access to > resource_parameters could lead to crash or wrong results. A code path example > which could lead to segfault is: > ProcessUtility()->PerformCursorOpen()->_copyPlannedStmt()->_copyQueryResourceParameters() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-1212) Upgrade libhdfs3 with upstream hadoop/hdp
[ https://issues.apache.org/jira/browse/HAWQ-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15741377#comment-15741377 ] hongwu commented on HAWQ-1212: -- [~rvs], yes. There needs more investigation to list these details of new features. For libhdfs3, you plan to merge its codes to Apache Hadoop codebase? > Upgrade libhdfs3 with upstream hadoop/hdp > - > > Key: HAWQ-1212 > URL: https://issues.apache.org/jira/browse/HAWQ-1212 > Project: Apache HAWQ > Issue Type: New Feature > Components: libhdfs >Reporter: hongwu >Assignee: Lei Chang > > Since current libhdfs3 implementation is based on old version of HDFS, it is > necessary to upgrade libhdfs3 with latest HDFS of lots of new features and > configuration parameters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-hawq pull request #1048: HAWQ-1140. Parallelize test cases for haw...
GitHub user wcl14 opened a pull request: https://github.com/apache/incubator-hawq/pull/1048 HAWQ-1140. Parallelize test cases for hawqregister usage2. You can merge this pull request into a Git repository by running: $ git pull https://github.com/wcl14/incubator-hawq HAWQ-1140 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1048.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1048 commit 8da85b198f70df5edbecb1cf1e4a780f65f4b357 Author: Chunling Wang Date: 2016-12-12T08:56:58Z HAWQ-1140. Parallelize test cases for hawqregister usage2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1047: HAWQ-1003. Implement batched ACL check through R...
Github user paul-guo- commented on the issue: https://github.com/apache/incubator-hawq/pull/1047 Seems there are file conflicts with https://github.com/apache/incubator-hawq/pull/1046 At least rangerrest.h --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1040: HAWQ-1195. Fixed error "Two or more external tab...
Github user liming01 commented on the issue: https://github.com/apache/incubator-hawq/pull/1040 Revert this commit because running installcheck-good failed. It is my fault, I had thought that we have move all testcases in installcheck-good to featuretest, so I only ran feature test. Need more time to investigate why it conflict at the same relation file number. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1040: HAWQ-1195. Fixed error "Two or more exter...
GitHub user liming01 reopened a pull request: https://github.com/apache/incubator-hawq/pull/1040 HAWQ-1195. Fixed error "Two or more external tables use the same erro⦠â¦r table " The error table should be works same as normal user table on hdfs, which support multiple INSERTs. Hi @wangzw, could you help me to review it? Thanks. You can merge this pull request into a Git repository by running: $ git pull https://github.com/liming01/incubator-hawq mli/errtab_chk Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1040.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1040 commit 2a8eaa7ef6ac025f65d03510a2428825cd383341 Author: Ming LI Date: 2016-12-06T09:17:50Z HAWQ-1195. Fixed error "Two or more external tables use the same error table " The error table should be works same as normal user table on hdfs, which support multiple INSERTs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Reopened] (HAWQ-1195) Synchrony:Union not working on external tables ERROR:"Two or more external tables use the same error table ""xxxxxxx"" in a statement (execMain.c:274)"
[ https://issues.apache.org/jira/browse/HAWQ-1195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming LI reopened HAWQ-1195: --- Revert this commit because running installcheck-good failed. It is my fault, I had thought that we have move all testcases in installcheck-good to featuretest, so I only ran feature test. Need more time to investigate why it conflict at the same relation file number. > Synchrony:Union not working on external tables ERROR:"Two or more external > tables use the same error table ""xxx"" in a statement (execMain.c:274)" > --- > > Key: HAWQ-1195 > URL: https://issues.apache.org/jira/browse/HAWQ-1195 > Project: Apache HAWQ > Issue Type: Bug > Components: External Tables >Reporter: Ming LI >Assignee: Ming LI > Fix For: backlog > > > Hello, > User create an external table and define the error table. Then he do the > union on the same external table with different where condition. Then return > the error:ERROR: Two or more external tables use the same error table > "err_ext_pdr_cdci_pivotal_request_43448" in a statement (execMain.c:274) > Below is the master log when I reproduce: (the whole log file attached in > attachement) > {code} > 2016-11-29 22:49:51.976864 > PST,"gpadmin","postgres",p769199,th-2123704032,"[local]",,2016-11-29 22:46:14 > PST,1260,con72,cmd10,seg-1,,,x1260,sx1,"ERROR","XX000","Two or more external > tables use the same error table ""err_ext_pdr_cdci_pivotal_request_43448"" in > a statement (execMain.c:274)",,"select current_account_nbr,yearmonthint, > bank_name, first_date_open, max_cr_limit, care_credit_flag, cc1_flag, > partition_value, 'US' as loc from pdr_cdci_pivotal_request_43448 where > care_credit_flag<1 > union > select current_account_nbr,yearmonthint, bank_name, first_date_open, > max_cr_limit, care_credit_flag, cc1_flag, partition_value, 'Non-US' as loc > from pdr_cdci_pivotal_request_43448 where > care_credit_flag=1;",0,,"execMain.c",274,"Stack trace: > 10x8c5858 postgres errstart (??:0) > 20x8c75db postgres elog_finish (??:0) > 30x65f669 postgres (??:0) > 40x77d06a postgres walk_plan_node_fields (??:0) > 50x77e3ee postgres plan_tree_walker (??:0) > 60x77c70a postgres expression_tree_walker (??:0) > 70x77e35d postgres plan_tree_walker (??:0) > 80x77d06a postgres walk_plan_node_fields (??:0) > 90x77dfe6 postgres plan_tree_walker (??:0) > 10 0x77d06a postgres walk_plan_node_fields (??:0) > 11 0x77e1e5 postgres plan_tree_walker (??:0) > 12 0x77d06a postgres walk_plan_node_fields (??:0) > 13 0x77dfe6 postgres plan_tree_walker (??:0) > 14 0x77d06a postgres walk_plan_node_fields (??:0) > 15 0x77e1e5 postgres plan_tree_walker (??:0) > 16 0x66079b postgres ExecutorStart (??:0) > 17 0x7ebf1d postgres PortalStart (??:0) > 18 0x7e4288 postgres (??:0) > 19 0x7e54c2 postgres PostgresMain (??:0) > 20 0x797d50 postgres (??:0) > 21 0x79ab19 postgres PostmasterMain (??:0) > 22 0x4a4069 postgres main (??:0) > 23 0x7fd97d486d5d libc.so.6 __libc_start_main (??:0) > 24 0x4a40e9 postgres (??:0) > " > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-hawq pull request #1047: HAWQ-1003. Implement batched ACL check th...
Github user paul-guo- commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1047#discussion_r91898169 --- Diff: src/backend/utils/misc/guc.c --- @@ -8154,6 +8174,15 @@ static struct config_string ConfigureNamesString[] = }, { +{"hawq_rps_address_host", PGC_POSTMASTER, PRESET_OPTIONS, + gettext_noop("rps server address hostname"), + NULL +}, +&rps_addr_host, +"localhost", NULL, NULL + }, + + { --- End diff -- Same as before. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1047: HAWQ-1003. Implement batched ACL check th...
Github user paul-guo- commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1047#discussion_r91897627 --- Diff: src/backend/parser/parse_relation.c --- @@ -2713,15 +2712,90 @@ warnAutoRange(ParseState *pstate, RangeVar *relation, int location) void ExecCheckRTPerms(List *rangeTable) { + if (enable_ranger) + { +if(rangeTable!=NULL) + ExecCheckRTPermsWithRanger(rangeTable); +return; + } ListCell *l; - foreach(l, rangeTable) { ExecCheckRTEPerms((RangeTblEntry *) lfirst(l)); } } /* + * ExecCheckRTPerms + * Batch implementation: Check access permissions for all relations listed in a range table with enable_ranger is true. + */ +void +ExecCheckRTPermsWithRanger(List *rangeTable) +{ + List *ranger_check_args = NIL; + ListCell *l; + foreach(l, rangeTable) + { + +AclMode requiredPerms; +Oid relOid; +Oid userid; +RangeTblEntry *rte = (RangeTblEntry *) lfirst(l); + +if (rte->rtekind != RTE_RELATION) + return; +requiredPerms = rte->requiredPerms; +if (requiredPerms == 0) + return; + +relOid = rte->relid; +userid = rte->checkAsUser ? rte->checkAsUser : GetUserId(); + +RangerPrivilegeArgs *ranger_check_arg = (RangerPrivilegeArgs *) palloc(sizeof(RangerPrivilegeArgs)); +ranger_check_arg->objkind = ACL_KIND_CLASS; +ranger_check_arg->object_oid = relOid; +ranger_check_arg->roleid = userid; +ranger_check_arg->mask = requiredPerms; +ranger_check_arg->how = ACLMASK_ALL; +ranger_check_args = lappend(ranger_check_args, ranger_check_arg); + + } // foreach + + // ranger ACL check with package Oids + List *aclresults = NIL; + aclresults = pg_rangercheck_batch(ranger_check_args); + if (aclresults == NIL) + { +elog(ERROR, "ERROR\n"); --- End diff -- I would think this log does not help debugging much. Could we provide more info? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1047: HAWQ-1003. Implement batched ACL check th...
Github user paul-guo- commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1047#discussion_r91898078 --- Diff: src/include/utils/rangerrest.h --- @@ -0,0 +1,39 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/*- + * + * rangerrest.h + * routines to interact with Ranger REST API --- End diff -- REST API for Ranger interaction? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1047: HAWQ-1003. Implement batched ACL check th...
Github user paul-guo- commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1047#discussion_r91898147 --- Diff: src/backend/utils/misc/guc.c --- @@ -6250,6 +6261,15 @@ static struct config_int ConfigureNamesInt[] = }, { +{"hawq_rps_address_port", PGC_POSTMASTER, PRESET_OPTIONS, + gettext_noop("rps server address port number"), + NULL +}, +&rps_addr_port, +1, 1, 65535, NULL, NULL + }, + + { --- End diff -- What is rps? If this is Ranger specific, please specify in the descriptions. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---