[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15241991#comment-15241991
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user shivzone commented on the pull request:

https://github.com/apache/incubator-hawq/pull/412#issuecomment-210170271
  
This pull request is closed as it is outdated.
Refer to the below 2 pull requests which solves it more generically for 
both readable and writable tables.
https://github.com/apache/incubator-hawq/pull/503
https://github.com/apache/incubator-hawq/pull/604


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15240587#comment-15240587
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user ictmalili commented on the pull request:

https://github.com/apache/incubator-hawq/pull/503#issuecomment-209759590
  
Adding @ztao1987 's comments, LGTM.


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15240545#comment-15240545
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user ztao1987 commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/503#discussion_r59660545
  
--- Diff: src/bin/gpfusion/gpbridgeapi.c ---
@@ -536,3 +536,8 @@ void free_token_resources(PxfInputData *inputData)
 
pfree(inputData->token);
 }
+
+void free_dfs_address()
+{
+   free(dfs_address);
+}
--- End diff --

Yes, just check NULL condition is enough.


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15240541#comment-15240541
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/503#discussion_r59660368
  
--- Diff: src/bin/gpfusion/gpbridgeapi.c ---
@@ -536,3 +536,8 @@ void free_token_resources(PxfInputData *inputData)
 
pfree(inputData->token);
 }
+
+void free_dfs_address()
+{
+   free(dfs_address);
+}
--- End diff --

Oh right, good catch. I will add a check in `free_dfs_address` for secure 
filesystem. Sound good?


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15240538#comment-15240538
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user ztao1987 commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/503#discussion_r59660246
  
--- Diff: src/bin/gpfusion/gpbridgeapi.c ---
@@ -536,3 +536,8 @@ void free_token_resources(PxfInputData *inputData)
 
pfree(inputData->token);
 }
+
+void free_dfs_address()
+{
+   free(dfs_address);
+}
--- End diff --

I checked free_dfs_address is called by gpbridge_export_start and 
gpbridge_import_start, do you mean this two funtions are only called in the 
secure mode? If not, you are freeing NULL pointer.


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15240532#comment-15240532
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/503#discussion_r59659935
  
--- Diff: src/bin/gpfusion/gpbridgeapi.c ---
@@ -536,3 +536,8 @@ void free_token_resources(PxfInputData *inputData)
 
pfree(inputData->token);
 }
+
+void free_dfs_address()
+{
+   free(dfs_address);
+}
--- End diff --

Yes only in the secure mode. However it is only dispatched from master in 
the secure mode as well so it is never allocated in any other case


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15240527#comment-15240527
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user ztao1987 commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/503#discussion_r59659812
  
--- Diff: src/bin/gpfusion/gpbridgeapi.c ---
@@ -536,3 +536,8 @@ void free_token_resources(PxfInputData *inputData)
 
pfree(inputData->token);
 }
+
+void free_dfs_address()
+{
+   free(dfs_address);
+}
--- End diff --

Will free_dfs_address only be called in secure mode? Otherwise it will 
coredump


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15240521#comment-15240521
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user ztao1987 commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/503#discussion_r59659407
  
--- Diff: src/backend/cdb/cdbquerycontextdispatching.c ---
@@ -770,6 +770,30 @@ RebuildTupleForRelation(QueryContextInfo *cxt)
 }
 
 /*
+ * Deserialize the Namespace data
+ */
+static void
+RebuildNamespace(QueryContextInfo *cxt)
+{
+
+   int len;
+   char buffer[4], *binary;
+   ReadData(cxt, buffer, sizeof(buffer), TRUE);
+
+   len = (int) ntohl(*(uint32 *) buffer);
+   binary = palloc(len);
+   if(ReadData(cxt, binary, len, TRUE))
+   {
+   StringInfoData buffer;
+   initStringInfoOfString(, binary, len);
+   dfs_address = strdup(buffer.data);
--- End diff --

Why not just strdup(binary) to dfs_address? 


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225689#comment-15225689
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user kavinderd commented on the pull request:

https://github.com/apache/incubator-hawq/pull/503#issuecomment-205644754
  
@ictmalili Oh ok cool. Thanks for taking care of it


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225604#comment-15225604
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user ictmalili commented on the pull request:

https://github.com/apache/incubator-hawq/pull/503#issuecomment-205623491
  
@kavinderd  I have fixed this. Could you see the pull request? 
https://github.com/apache/incubator-hawq/pull/554


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15224882#comment-15224882
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user kavinderd closed the pull request at:

https://github.com/apache/incubator-hawq/pull/503


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218903#comment-15218903
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user GodenYao commented on the pull request:

https://github.com/apache/incubator-hawq/pull/503#issuecomment-203644405
  
+1


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218878#comment-15218878
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user GodenYao commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/503#discussion_r57964920
  
--- Diff: src/backend/cdb/cdbquerycontextdispatching.c ---
@@ -1747,6 +1774,7 @@ 
prepareDispatchedCatalogExternalTable(QueryContextInfo *cxt,
{
Insist(array_size == 1);
AddFileSystemCredentialForPxfTable(location);
--- End diff --

is this still needed? AddFileSystemCredentialForPxfTable as you dispatch it 
through query context now.


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218560#comment-15218560
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/503#discussion_r57940794
  
--- Diff: src/backend/cdb/cdbquerycontextdispatching.c ---
@@ -801,6 +825,9 @@ RebuildQueryContext(QueryContextInfo *cxt, HTAB 
**currentFilesystemCredentials,
RebuildFilesystemCredentials(cxt, currentFilesystemCredentials,
currentFilesystemCredentialsMemoryContext);
break;
+case Namespace:
+RebuildNamespace(cxt);
--- End diff --

No, it won't. The `Namespace` enum is only added to the buffer in the case 
of a secure filesystem


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218411#comment-15218411
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user shivzone commented on the pull request:

https://github.com/apache/incubator-hawq/pull/503#issuecomment-203543666
  
+1


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216923#comment-15216923
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user shivzone commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/503#discussion_r57808336
  
--- Diff: src/backend/cdb/cdbquerycontextdispatching.c ---
@@ -801,6 +825,9 @@ RebuildQueryContext(QueryContextInfo *cxt, HTAB 
**currentFilesystemCredentials,
RebuildFilesystemCredentials(cxt, currentFilesystemCredentials,
currentFilesystemCredentialsMemoryContext);
break;
+case Namespace:
+RebuildNamespace(cxt);
--- End diff --

Will this case be invoked in the non secure case where you don't set the 
namespace


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216912#comment-15216912
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user shivzone commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/503#discussion_r57807058
  
--- Diff: src/backend/cdb/cdbquerycontextdispatching.c ---
@@ -2984,3 +3012,21 @@ GetResultRelSegFileInfos(Oid relid, List *segnomaps, 
List *existing_seginfomaps)
 
return existing_seginfomaps;
 }
+
+void
+prepareDfsAddressForDispatch(QueryContextInfo* cxt)
--- End diff --

add comment to describe the purpose of this function


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211402#comment-15211402
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user kavinderd commented on the pull request:

https://github.com/apache/incubator-hawq/pull/503#issuecomment-201131567
  
In the non-secure case the `dfs_address` is not serialized and dispatched 
from the master so on the segment the global var will just be `NULL`


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211398#comment-15211398
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/503#discussion_r57418511
  
--- Diff: src/backend/cdb/cdbquerycontextdispatching.c ---
@@ -2984,3 +3013,20 @@ GetResultRelSegFileInfos(Oid relid, List *segnomaps, 
List *existing_seginfomaps)
 
return existing_seginfomaps;
 }
+
+void
+prepareDfsAddressForDispatch(QueryContextInfo* cxt){
+// if (enable_secure_filesystem)
--- End diff --

oh yeah, oops


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211294#comment-15211294
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/503#discussion_r57412948
  
--- Diff: src/backend/cdb/cdbquerycontextdispatching.c ---
@@ -770,6 +770,31 @@ RebuildTupleForRelation(QueryContextInfo *cxt)
 }
 
 /*
+ * Deserialize the Namespace data
+ */
+
--- End diff --

remove new line


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211254#comment-15211254
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

GitHub user kavinderd opened a pull request:

https://github.com/apache/incubator-hawq/pull/503

HAWQ-462. Dispatch dfs_address from master to segment in pxf secure f…

…ilesystem case

Set global var dfs_address on segment after deserialization

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kavinderd/incubator-hawq HAWQ-462

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/503.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #503


commit 0585eb0de9d5203335656796534463630942ff84
Author: Kavinder Dhaliwal 
Date:   2016-03-25T01:24:35Z

HAWQ-462. Dispatch dfs_address from master to segment in pxf secure 
filesystem case

Set global var dfs_address on segment after deserialization




> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186285#comment-15186285
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user shivzone commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/412#discussion_r55460638
  
--- Diff: src/backend/access/external/pxfuriparser.c ---
@@ -634,9 +638,20 @@ GPHDUri_parse_segwork(GPHDUri *uri, const char 
*uri_str)
segwork = strstr(uri_str, segwork_substring);
if (segwork == NULL)
return;
-
segwork += strlen(segwork_substring);
 
+   if (enable_secure_filesystem)
+   {
+   /* parse dfs address */
+   size_end = strchr(segwork, segwork_dfs_separator);
+   if(size_end != NULL)
--- End diff --

Should this be checked in the master instead before we add this info to the 
segwork data ?


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186275#comment-15186275
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user hornn commented on the pull request:

https://github.com/apache/incubator-hawq/pull/412#issuecomment-194056909
  
Looks good. Do you plan to open a separate story for writable?


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186268#comment-15186268
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/412#discussion_r55459533
  
--- Diff: src/backend/access/external/pxfuriparser.c ---
@@ -634,9 +638,20 @@ GPHDUri_parse_segwork(GPHDUri *uri, const char 
*uri_str)
segwork = strstr(uri_str, segwork_substring);
if (segwork == NULL)
return;
-
segwork += strlen(segwork_substring);
 
+   if (enable_secure_filesystem)
+   {
+   /* parse dfs address */
+   size_end = strchr(segwork, segwork_dfs_separator);
+   if(size_end != NULL)
--- End diff --

if it's a secured environment, if there is no dfs address data maybe we 
should error out. What do you think?


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186266#comment-15186266
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/412#discussion_r55459477
  
--- Diff: src/backend/access/external/pxfuriparser.c ---
@@ -744,6 +759,9 @@ GPHDUri_free_fragments(GPHDUri *uri)
pfree(data->authority);
pfree(data->index);
pfree(data->source_name);
+   pfree(data->fragment_md);
+   if(data->user_data)
--- End diff --

missing space after if


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15185896#comment-15185896
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user GodenYao commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/412#discussion_r55433977
  
--- Diff: src/backend/access/external/hd_work_mgr.c ---
@@ -768,6 +769,12 @@ make_allocation_output_string(List *segment_fragments)
initStringInfo();
appendStringInfoString(, SEGWORK_PREFIX);

+   /* Add dfs_address from pg_filespace to the segment data. Fixes 
HAWQ-462 *//* dfs_address from pg_filespace entry */
+   char* dfs_address = NULL;
+   get_hdfs_location_from_filespace(_address);
--- End diff --

probably add check to see if it's secure or not. enable_secure_filesystem 
== true


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15185879#comment-15185879
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user GodenYao commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/412#discussion_r55432353
  
--- Diff: src/include/access/pxfuriparser.h ---
@@ -76,6 +76,9 @@ typedef struct GPHDUri
 * NNHAConf  will also occupy  and  members
 */
NNHAConf*ha_nodes;
+
+   /* dfs address from pg_filespace (optional) */
+   char*dfs_address;
--- End diff --

indention seems off.


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180587#comment-15180587
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user shivzone commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/412#discussion_r55093127
  
--- Diff: src/backend/access/external/pxfuriparser.c ---
@@ -719,7 +727,7 @@ GPHDUri_parse_fragment(char* fragment, List* fragments)
has_user_data = true;
*value_end = '\0';
}
-   fragment_data->fragment_md = pstrdup(value_start);
+   fragment_data->dfs_address = pstrdup(value_start);
--- End diff --

Done


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180588#comment-15180588
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user shivzone commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/412#discussion_r55093144
  
--- Diff: src/backend/access/external/pxfuriparser.c ---
@@ -709,7 +709,15 @@ GPHDUri_parse_fragment(char* fragment, List* fragments)
*value_end = '\0';
--- End diff --

Added


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180576#comment-15180576
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user shivzone commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/412#discussion_r55092296
  
--- Diff: src/backend/access/external/hd_work_mgr.c ---
@@ -783,6 +786,10 @@ make_allocation_output_string(List *segment_fragments)
appendStringInfoChar(_str, SEGWORK_IN_PAIR_DELIM);
if (frag->fragment_md)
appendStringInfo(_str, "%s", 
frag->fragment_md);
+   /* Adding dfs_address from pg_filespace entry required for 
HAWQ-462 */
+   appendStringInfoChar(_str, SEGWORK_IN_PAIR_DELIM);
+   if (dfs_address)
--- End diff --

Yes, this is the first cut solution, the optimal solution is to pass it as 
part of the segment level data instead of fragment. Wasn't quite able to figure 
out how to add additional metadata with the segment data. Wanted to first 
certify against secure cluster before investigating addition into segment data.


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180561#comment-15180561
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user GodenYao commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/412#discussion_r55091155
  
--- Diff: src/backend/access/external/hd_work_mgr.c ---
@@ -783,6 +786,10 @@ make_allocation_output_string(List *segment_fragments)
appendStringInfoChar(_str, SEGWORK_IN_PAIR_DELIM);
if (frag->fragment_md)
appendStringInfo(_str, "%s", 
frag->fragment_md);
+   /* Adding dfs_address from pg_filespace entry required for 
HAWQ-462 */
+   appendStringInfoChar(_str, SEGWORK_IN_PAIR_DELIM);
+   if (dfs_address)
--- End diff --

Second that. it'll be an overhead for each fragment to have the duplicated 
info.


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180560#comment-15180560
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user GodenYao commented on the pull request:

https://github.com/apache/incubator-hawq/pull/412#issuecomment-192472457
  
can you also check if you differentiate secure and non-secure queries - for 
non-secure query we don't need dfs_url info


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180485#comment-15180485
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user hornn commented on the pull request:

https://github.com/apache/incubator-hawq/pull/412#issuecomment-192448360
  
General question - does it also solve the problem for writable? I thought 
for writable tables the code path didn't pass through hd_work_mgr.


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180484#comment-15180484
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/412#discussion_r55083101
  
--- Diff: src/backend/access/external/pxfuriparser.c ---
@@ -719,7 +727,7 @@ GPHDUri_parse_fragment(char* fragment, List* fragments)
has_user_data = true;
*value_end = '\0';
}
-   fragment_data->fragment_md = pstrdup(value_start);
+   fragment_data->dfs_address = pstrdup(value_start);
--- End diff --

please make sure to pfree it in GPHDUri_free_fragments.
I also noticed that we don't free fragment_md and user_data.



> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180475#comment-15180475
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/412#discussion_r55082713
  
--- Diff: src/backend/access/external/pxfuriparser.c ---
@@ -709,7 +709,15 @@ GPHDUri_parse_fragment(char* fragment, List* fragments)
*value_end = '\0';
--- End diff --

in line 666, please update the expected syntax.



> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180362#comment-15180362
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

GitHub user shivzone opened a pull request:

https://github.com/apache/incubator-hawq/pull/412

HAWQ-462. Adding dfs_address from pgfilespace along with fragment dat…

HAWQ-317 attempts to use the dfs_address from pg_filespace_entry. The 
segments (gpbridge) do not have catalog access. Adding the dfs_address along 
with the fragment data passed from the master to the segments. This fixes the 
pxf handling of secure isilon and hcatalog with or without HA

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/incubator-hawq HAWQ-462

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/412.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #412


commit 50926e7470e4d5a6c6857d57063f8b8f5634d6f8
Author: Shivram Mani 
Date:   2016-03-04T18:49:13Z

HAWQ-462. Adding dfs_address from pgfilespace along with fragment data sent 
to segments




> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[Goden Yao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15176556#comment-15176556
 ] 

Goden Yao commented on HAWQ-462:


incorporate comments from [~nhorn] 
{quote}
I don't know if the filespace info is available from the segment or not - but 
from your description it sounds like it's not.

In that case, I think the solution might be to propagate the dfs_url from the 
master to the segment as part of the external scan info. For readable table it 
should be easy - the data can be added in hd_work_mgr, together with the 
fragments info. For writable tables it's more tricky, because there is no 
pxf-specific code that is called in the master for writable tables. So I think 
a better solution would be to always make this info available to external 
protocol code, by making it part of the external scan, or perhaps populating a 
global variable when the segment starts.
{quote}

> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[Shivram Mani (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15176071#comment-15176071
 ] 

Shivram Mani commented on HAWQ-462:
---

On further investigation, we noticed that invoking 
get_hdfs_location_from_filespace() from gpbridge doesn't return the expected 
result as this runs on the hawq segment. This causes both hive and hcatalog 
access to fail in secure environment.
We will alternatively extract the namenode location from the context 
information available provided to the segment.

> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[Noa Horn (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15169769#comment-15169769
 ] 

Noa Horn commented on HAWQ-462:
---

Merged HAWQ-317 fix to master 
(https://github.com/apache/incubator-hawq/pull/279).

> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Kavinder Dhaliwal
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-462) Querying Hcatalog in HA Secure Environment Fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15169200#comment-15169200
 ] 

ASF GitHub Bot commented on HAWQ-462:
-

Github user shivzone commented on the pull request:

https://github.com/apache/incubator-hawq/pull/279#issuecomment-189326238
  
@hornn this patch is required to fix 
https://issues.apache.org/jira/browse/HAWQ-462
We did the RCA for the hcatalog failure against secure cluster and this 
patch fixes that issue as well.
Can you please merge it ?


> Querying Hcatalog in HA Secure Environment Fails
> 
>
> Key: HAWQ-462
> URL: https://issues.apache.org/jira/browse/HAWQ-462
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Kavinder Dhaliwal
>Assignee: Kavinder Dhaliwal
> Fix For: 2.0.0
>
>
> On an HA Secure Cluster querying a hive external table works:
> {code}
> create external table pxf_hive(s1 text, n1 int, d1 float, bg bigint, b 
> boolean) location 
> ('pxf://ip-10-32-38-119.ore1.vpc.pivotal.io:51200/hive_table?profile=Hive') 
> format 'custom' (formatter='pxfwritable_import');
> select * from pxf_hive;
> {code}
> but querying the same table via hcatalog does not
> {code}
> SELECT * FROM hcatalog.default.hive_table;
> ERROR:  Failed to acquire a delegation token for uri hdfs://localhost:8020/ 
> (hd_work_mgr.c:930)
> {code}
> This should be fixed by the PR for 
> https://issues.apache.org/jira/browse/HAWQ-317



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)