[
https://issues.apache.org/jira/browse/HAWQ-257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15066757#comment-15066757
]
Noa Horn edited comment on HAWQ-257 at 12/21/15 9:48 PM:
---------------------------------------------------------
How to test:
1. Change pxf_service_port GUC
The GUC needs to be changed on both master and all nodes. It can be done by
running
{noformat}
hawq config -c pxf_service_port -v 1234 --skipvalidation
hawq restart cluster
{noformat}
2. Check pxf_service_port value
{noformat}
psql
show pxf_service_port;
{noformat}
3. Create external read/write tables
{noformat}
DROP EXTERNAL TABLE IF EXISTS ext_r;
CREATE EXTERNAL TABLE ext_r (num float, eng text, ger text)
LOCATION('pxf://localhost:51201/table.txt?profile=HdfsTextSimple')
FORMAT 'TEXT' (delimiter = ',');
DROP EXTERNAL TABLE IF EXISTS ext_w;
CREATE WRITABLE EXTERNAL TABLE ext_w (num float, eng text, ger text)
LOCATION('pxf://localhost:51201/w/?profile=HdfsTextSimple')
FORMAT 'TEXT' (delimiter = ',');
{noformat}
4. Run read/write on above tables:
{noformat}
SELECT * FROM ext_r;
INSERT INTO ext_w SELECT * FROM ext_r;
ANALYZE ext_r;
{noformat}
5. HA mode: create tables while using the nameservice in the address, and
repeat step (4) to test.
{noformat}
DROP EXTERNAL TABLE IF EXISTS ext_r;
CREATE EXTERNAL TABLE ext_r (num float, eng text, ger text)
LOCATION('pxf://singlecluster/table.txt?profile=HdfsTextSimple')
FORMAT 'TEXT' (delimiter = ',');
DROP EXTERNAL TABLE IF EXISTS ext_w;
CREATE WRITABLE EXTERNAL TABLE ext_w (num float, eng text, ger text)
LOCATION('pxf://singlecluster/w/?profile=HdfsTextSimple')
FORMAT 'TEXT' (delimiter = ',');
{noformat}
was (Author: nhorn):
How to test:
1. Change pxf_service_port GUC
The GUC needs to be changed on both master and all nodes. It can be done by
running
{noformat}
hawq config -c pxf_service_port -v 1234 --skipvalidation
hawq restart cluster
{noformat}
2. Check pxf_service_port value
{noformat}
psql
show pxf_service_port;
{noformat}
3. Create external read/write tables
{noformat}
DROP EXTERNAL TABLE IF EXISTS ext_r;
CREATE EXTERNAL TABLE ext_r (num float, eng text, ger text)
LOCATION('pxf://localhost:51201/table.txt?profile=HdfsTextSimple')
FORMAT 'TEXT' (delimiter = ',');
DROP EXTERNAL TABLE IF EXISTS ext_w;
CREATE WRITABLE EXTERNAL TABLE ext_w (num float, eng text, ger text)
LOCATION('pxf://localhost:51201/w/?profile=HdfsTextSimple')
FORMAT 'TEXT' (delimiter = ',');
{noformat}
4. Run read/write on above tables:
{noformat}
SELECT * FROM ext_r;
INSERT INTO ext_w SELECT * FROM ext_r;
{noformat}
5. HA mode: create tables while using the nameservice in the address, and
repeat step (4) to test.
{noformat}
DROP EXTERNAL TABLE IF EXISTS ext_r;
CREATE EXTERNAL TABLE ext_r (num float, eng text, ger text)
LOCATION('pxf://singlecluster/table.txt?profile=HdfsTextSimple')
FORMAT 'TEXT' (delimiter = ',');
DROP EXTERNAL TABLE IF EXISTS ext_w;
CREATE WRITABLE EXTERNAL TABLE ext_w (num float, eng text, ger text)
LOCATION('pxf://singlecluster/w/?profile=HdfsTextSimple')
FORMAT 'TEXT' (delimiter = ',');
{noformat}
> PXF to honor user defined port in location URL
> ----------------------------------------------
>
> Key: HAWQ-257
> URL: https://issues.apache.org/jira/browse/HAWQ-257
> Project: Apache HAWQ
> Issue Type: Improvement
> Components: PXF
> Reporter: Goden Yao
> Assignee: Noa Horn
>
> I should be able to configure which port PXF service is listening to and
> input the same value in DDL to create tables
> {code}
> CREATE EXTERNAL TABLE ext_table <attr list, ...>
> LOCATION('pxf://<host>:<port>/path/to/data?...
> {code}
> *Background information*
> The reason that PXF currently ignores the port value in the DDL is due to
> legacy issues: 1) user can only configure this port that PXF service listens
> to through GUC (pxf_service_port), which was a hidden secret. 2) For HA
> consideration, when you don't put port in the URI, we use 51200 to connect to
> the pxf service.
> *Acceptance Criteria*
> 1) If users put a port in URL, pxf should use that port to initiate
> connection and error out if it cannot connect.
> 2) If users don't have port in URL, we assume it's HA scenario and connect to
> name service via default port : 51200.
> 3) Users can still modify the GUC to change the default value, but this is
> only for HA name service scenario.
> *Source Code*
> The current logic is located at: [pxfuriparser.c |
> https://github.com/apache/incubator-hawq/blob/7be5148934994df819ea53127c209a82eba40f40/src/backend/access/external/pxfuriparser.c],
> which replace user input port in the url with pxf_service_port
> {code}
> ...
> GPHDUri*
> parseGPHDUri(const char *uri_str)
> {
> GPHDUri *uri = (GPHDUri *)palloc0(sizeof(GPHDUri));
> char *cursor;
> uri->uri = GPHDUri_dup_without_segwork(uri_str);
> cursor = uri->uri;
> GPHDUri_parse_segwork(uri, uri_str);
> GPHDUri_parse_protocol(uri, &cursor);
> GPHDUri_parse_authority(uri, &cursor);
> GPHDUri_parse_data(uri, &cursor);
> GPHDUri_parse_options(uri, &cursor);
> port_to_str(&(uri->port), pxf_service_port);
> return uri;
> }
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
