[ https://issues.apache.org/jira/browse/HAWQ-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866266#comment-15866266 ]
Kyle R Dunn edited comment on HAWQ-1234 at 2/14/17 6:01 PM: ------------------------------------------------------------ Additionally, I observed all this using the following command, on the HAWQ master. First, start the {{tcpdump}} trace, then invoke a SELECT from a previously defined PXF table, either using HCatalog directly, or manually defining a PXF external table. A similar approach could be used to observe datanode traffic during read/write operations. {code} $ tcpdump -n port 51200 -A {code} Here is the conversation output: {code} 21:20:14.632410 IP 127.0.0.1.20416 > 127.0.0.1.51200: Flags [S], seq 3498721498, win 65483, options [mss 65495,sackOK,TS val 1901602390 ecr 1901547904,nop,wscale 9], length 0 E..<f.@.@..-........O.....@.........._......... qX&VqWQ.... 21:20:14.632421 IP 127.0.0.1.51200 > 127.0.0.1.20416: Flags [S.], seq 3752275736, ack 3498721499, win 65483, options [mss 65495,sackOK,TS val 1901602390 ecr 1901602390,nop,wscale 9], length 0 E..<..@.@.<...........O.../...@................ qX&VqX&V... 21:20:14.632428 IP 127.0.0.1.20416 > 127.0.0.1.51200: Flags [.], ack 1, win 128, options [nop,nop,TS val 1901602390 ecr 1901602390], length 0 E..4f.@.@..4........O.....@.../............ qX&VqX&V 21:20:14.632602 IP 127.0.0.1.20416 > 127.0.0.1.51200: Flags [P.], seq 1:318, ack 1, win 128, options [nop,nop,TS val 1901602390 ecr 1901602390], length 317 E..qf.@.@...........O.....@.../......e..... qX&VqX&VGET /pxf/v14/Metadata/getMetadata?profile=Hive&pattern=default.kdtest HTTP/1.1 Host: localhost:51200 Accept: application/json X-GP-SEGMENT-ID: -100005432 X-GP-SEGMENT-COUNT: 0 X-GP-XID: 2725021 X-GP-ALIGNMENT: 8 X-GP-URL-HOST: localhost X-GP-URL-PORT: 51200 X-GP-URI: localhost:51200/ X-GP-HAS-FILTER: 0 21:20:14.632607 IP 127.0.0.1.51200 > 127.0.0.1.20416: Flags [.], ack 318, win 130, options [nop,nop,TS val 1901602390 ecr 1901602390], length 0 E..4 O@.@.3s..........O.../...B............ qX&VqX&V 21:20:15.084890 IP 127.0.0.1.51200 > 127.0.0.1.20416: Flags [P.], seq 1:269, ack 318, win 130, options [nop,nop,TS val 1901602843 ecr 1901602390], length 268 E..@ P@.@.2f..........O.../...B......4..... qX(.qX&VHTTP/1.1 200 OK Server: Apache-Coyote/1.1 Content-Type: application/json Content-Length: 132 Date: Tue, 14 Feb 2017 05:20:15 GMT {"PXFMetadata":[{"item":{"path":"default","name":"kdtest"},"fields":[{"name":"key","type":"text"},{"name":"value","type":"text"}]}]} 21:20:15.084900 IP 127.0.0.1.20416 > 127.0.0.1.51200: Flags [.], ack 269, win 130, options [nop,nop,TS val 1901602843 ecr 1901602843], length 0 E..4f.@.@..2........O.....B...0%.....!..... qX(.qX(. 21:20:15.085229 IP 127.0.0.1.20416 > 127.0.0.1.51200: Flags [F.], seq 318, ack 269, win 130, options [nop,nop,TS val 1901602843 ecr 1901602843], length 0 E..4f.@.@..1........O.....B...0%..... ..... qX(.qX(. 21:20:15.085286 IP 127.0.0.1.51200 > 127.0.0.1.20416: Flags [F.], seq 269, ack 319, win 130, options [nop,nop,TS val 1901602843 ecr 1901602843], length 0 E..4 Q@.@.3q..........O...0%..B............ qX(.qX(. 21:20:15.085294 IP 127.0.0.1.20416 > 127.0.0.1.51200: Flags [.], ack 270, win 130, options [nop,nop,TS val 1901602843 ecr 1901602843], length 0 E..4f.@.@..0........O.....B...0&........... qX(.qX(. 21:20:15.112739 IP 127.0.0.1.20422 > 127.0.0.1.51200: Flags [S], seq 222439143, win 65483, options [mss 65495,sackOK,TS val 1901602870 ecr 1901602843,nop,wscale 9], length 0 B&...................... qX(6qX(.... 21:20:15.112765 IP 127.0.0.1.51200 > 127.0.0.1.20422: Flags [S.], seq 2606634976, ack 222439144, win 65483, options [mss 65495,sackOK,TS val 1901602871 ecr 1901602870,nop,wscale 9], length 0 B&....................O..^.. qX(7qX(6... 21:20:15.112773 IP 127.0.0.1.20422 > 127.0.0.1.51200: Flags [.], ack 1, win 128, options [nop,nop,TS val 1901602871 ecr 1901602871], length 0 B&..^......E........O... qX(7qX(7 21:20:15.112809 IP 127.0.0.1.20422 > 127.0.0.1.51200: Flags [P.], seq 1:581, ack 1, win 128, options [nop,nop,TS val 1901602871 ecr 1901602871], length 580 B&..^.......m.......O... qX(7qX(7GET /pxf/v14/Fragmenter/getFragments?path=default.kdtest HTTP/1.1 Host: localhost:51200 Accept: application/json X-GP-FORMAT: GPDBWritable X-GP-ATTRS: 2 X-GP-ATTR-NAME0: key X-GP-ATTR-TYPECODE0: 25 X-GP-ATTR-TYPENAME0: text X-GP-ATTR-NAME1: value X-GP-ATTR-TYPECODE1: 25 X-GP-ATTR-TYPENAME1: text X-GP-SEGMENT-ID: -100005432 X-GP-SEGMENT-COUNT: 0 X-GP-XID: 2725021 X-GP-ALIGNMENT: 8 X-GP-URL-HOST: localhost X-GP-URL-PORT: 51200 X-GP-DATA-DIR: default.kdtest X-GP-Profile: Hive X-GP-URI: pxf://localhost:51200/default.kdtest?Profile=Hive X-GP-HAS-FILTER: 0 21:20:15.112813 IP 127.0.0.1.51200 > 127.0.0.1.20422: Flags [.], ack 581, win 131, options [nop,nop,TS val 1901602871 ecr 1901602871], length 0 B),....C..............O..^.. qX(7qX(7 21:20:15.305723 IP 127.0.0.1.51200 > 127.0.0.1.20422: Flags [P.], seq 1:1443, ack 581, win 131, options [nop,nop,TS val 1901603063 ecr 1901602871], length 1442 B),...................O..^.. qX(.qX(7HTTP/1.1 200 OK Server: Apache-Coyote/1.1 Content-Type: application/json Content-Length: 1305 Date: Tue, 14 Feb 2017 05:20:15 GMT {"PXFFragments":[{"sourceName":"/apps/hive/warehouse/kdtest/hive-test-data.txt","index":0,"replicas":["10.215.181.12","10.215.181.11"],"metadata":"rO0ABXcQAAAAAAAAAAAAAAAAAAAAN3VyABNbTGphdmEubGFuZy5TdHJpbmc7rdJW5+kde0cCAAB4cAAAAAJ0AB1jbHBxbjFwZGhkYmRuMDIuaW5mb3NvbGNvLm5ldHQAHWNscHFuMXBkaGRiZG4wMS5pbmZvc29sY28ubmV0","userData":"b3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlRleHRJbnB1dEZvcm1hdCFIVUREIW9yZy5hcGFjaGUuaGFkb29wLmhpdmUuc2VyZGUyLmxhenkuTGF6eVNpbXBsZVNlckRlIUhVREQhIwojTW9uIEZlYiAxMyAyMToyMDoxNSBQU1QgMjAxNwpuYW1lPWRlZmF1bHQua2R0ZXN0Cm51bUZpbGVzPTEKZmllbGQuZGVsaW09LApjb2x1bW5zLnR5cGVzPXN0cmluZ1w6c3RyaW5nCnNlcmlhbGl6YXRpb24uZGRsPXN0cnVjdCBrZHRlc3QgeyBzdHJpbmcga2V5LCBzdHJpbmcgdmFsdWV9CmNvbHVtbnM9a2V5LHZhbHVlCnNlcmlhbGl6YXRpb24uZm9ybWF0PSwKY29sdW1ucy5jb21tZW50cz1cdTAwMDAKYnVja2V0X2NvdW50PS0xCnNlcmlhbGl6YXRpb24ubGliPW9yZy5hcGFjaGUuaGFkb29wLmhpdmUuc2VyZGUyLmxhenkuTGF6eVNpbXBsZVNlckRlCkNPTFVNTl9TVEFUU19BQ0NVUkFURT10cnVlCmZpbGUuaW5wdXRmb3JtYXQ9b3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlRleHRJbnB1dEZvcm1hdAp0b3RhbFNpemU9NTUKZmlsZS5vdXRwdXRmb3JtYXQ9b3JnLmFwYWNoZS5oYWRvb3AuaGl2ZS5xbC5pby5IaXZlSWdub3JlS2V5VGV4dE91dHB1dEZvcm1hdApsb2NhdGlvbj1oZGZzXDovL2NscHFuMXBkaGRibW4wMS5pbmZvc29sY28ubmV0XDo4MDIwL2FwcHMvaGl2ZS93YXJlaG91c2Uva2R0ZXN0CnRyYW5zaWVudF9sYXN0RGRsVGltZT0xNDg3MDA2NDg4CiFIVUREISFITlBUISFIVUREIWZhbHNl"}]} 21:20:15.305733 IP 127.0.0.1.20422 > 127.0.0.1.51200: Flags [.], ack 1443, win 134, options [nop,nop,TS val 1901603063 ecr 1901603063], length 0 B),.^......<........O... qX(.qX(. 21:20:15.305897 IP 127.0.0.1.20422 > 127.0.0.1.51200: Flags [F.], seq 581, ack 1443, win 134, options [nop,nop,TS val 1901603064 ecr 1901603063], length 0 B),.^......<}.......O... qX(.qX(. 21:20:15.305961 IP 127.0.0.1.51200 > 127.0.0.1.20422: Flags [F.], seq 1443, ack 582, win 131, options [nop,nop,TS val 1901603064 ecr 1901603064], length 0 B)-....<~.............O..^.. qX(.qX(. 21:20:15.305968 IP 127.0.0.1.20422 > 127.0.0.1.51200: Flags [.], ack 1444, win 134, options [nop,nop,TS val 1901603064 ecr 1901603064], length 0 B)-.^......<{.......O... qX(.qX(. 21:20:15.348058 IP 127.0.0.1.20432 > 127.0.0.1.51200: Flags [S], seq 1316022209, win 65483, options [mss 65495,sackOK,TS val 1901603106 ecr 1901603064,nop,wscale 9], length 0 E..<..@.@.m.........O...Np...........=......... qX)"qX(.... 21:20:15.348069 IP 127.0.0.1.51200 > 127.0.0.1.20432: Flags [S.], seq 124605876, ack 1316022210, win 65483, options [mss 65495,sackOK,TS val 1901603106 ecr 1901603106,nop,wscale 9], length 0 E..<..@.@.<...........O..mU.Np......o.......... qX)"qX)"... 21:20:15.348078 IP 127.0.0.1.20432 > 127.0.0.1.51200: Flags [.], ack 1, win 128, options [nop,nop,TS val 1901603106 ecr 1901603106], length 0 E..4..@.@.m.........O...Np...mU............ qX)"qX)" 21:20:15.348138 IP 127.0.0.1.20432 > 127.0.0.1.51200: Flags [P.], seq 1:581, ack 1, win 128, options [nop,nop,TS val 1901603106 ecr 1901603106], length 580 E..x..@.@.k.........O...Np...mU......m..... qX)"qX)"GET /pxf/v14/Fragmenter/getFragments?path=default.kdtest HTTP/1.1 Host: localhost:51200 Accept: application/json X-GP-FORMAT: GPDBWritable X-GP-ATTRS: 2 X-GP-ATTR-NAME0: key X-GP-ATTR-TYPECODE0: 25 X-GP-ATTR-TYPENAME0: text X-GP-ATTR-NAME1: value X-GP-ATTR-TYPECODE1: 25 X-GP-ATTR-TYPENAME1: text X-GP-SEGMENT-ID: -100005432 X-GP-SEGMENT-COUNT: 0 X-GP-XID: 2725021 X-GP-ALIGNMENT: 8 X-GP-URL-HOST: localhost X-GP-URL-PORT: 51200 X-GP-DATA-DIR: default.kdtest X-GP-Profile: Hive X-GP-URI: pxf://localhost:51200/default.kdtest?Profile=Hive X-GP-HAS-FILTER: 0 21:20:15.348144 IP 127.0.0.1.51200 > 127.0.0.1.20432: Flags [.], ack 581, win 131, options [nop,nop,TS val 1901603106 ecr 1901603106], length 0 E..4|.@.@..*..........O..mU.Np............. qX)"qX)" 21:20:15.523187 IP 127.0.0.1.51200 > 127.0.0.1.20432: Flags [P.], seq 1:1443, ack 581, win 131, options [nop,nop,TS val 1901603281 ecr 1901603106], length 1442 E...|.@.@.............O..mU.Np............. qX).qX)"HTTP/1.1 200 OK Server: Apache-Coyote/1.1 Content-Type: application/json Content-Length: 1305 Date: Tue, 14 Feb 2017 05:20:15 GMT {"PXFFragments":[{"sourceName":"/apps/hive/warehouse/kdtest/hive-test-data.txt","index":0,"replicas":["10.215.181.12","10.215.181.11"],"metadata":"rO0ABXcQAAAAAAAAAAAAAAAAAAAAN3VyABNbTGphdmEubGFuZy5TdHJpbmc7rdJW5+kde0cCAAB4cAAAAAJ0AB1jbHBxbjFwZGhkYmRuMDIuaW5mb3NvbGNvLm5ldHQAHWNscHFuMXBkaGRiZG4wMS5pbmZvc29sY28ubmV0","userData":"b3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlRleHRJbnB1dEZvcm1hdCFIVUREIW9yZy5hcGFjaGUuaGFkb29wLmhpdmUuc2VyZGUyLmxhenkuTGF6eVNpbXBsZVNlckRlIUhVREQhIwojTW9uIEZlYiAxMyAyMToyMDoxNSBQU1QgMjAxNwpuYW1lPWRlZmF1bHQua2R0ZXN0Cm51bUZpbGVzPTEKZmllbGQuZGVsaW09LApjb2x1bW5zLnR5cGVzPXN0cmluZ1w6c3RyaW5nCnNlcmlhbGl6YXRpb24uZGRsPXN0cnVjdCBrZHRlc3QgeyBzdHJpbmcga2V5LCBzdHJpbmcgdmFsdWV9CmNvbHVtbnM9a2V5LHZhbHVlCnNlcmlhbGl6YXRpb24uZm9ybWF0PSwKY29sdW1ucy5jb21tZW50cz1cdTAwMDAKYnVja2V0X2NvdW50PS0xCnNlcmlhbGl6YXRpb24ubGliPW9yZy5hcGFjaGUuaGFkb29wLmhpdmUuc2VyZGUyLmxhenkuTGF6eVNpbXBsZVNlckRlCkNPTFVNTl9TVEFUU19BQ0NVUkFURT10cnVlCmZpbGUuaW5wdXRmb3JtYXQ9b3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlRleHRJbnB1dEZvcm1hdAp0b3RhbFNpemU9NTUKZmlsZS5vdXRwdXRmb3JtYXQ9b3JnLmFwYWNoZS5oYWRvb3AuaGl2ZS5xbC5pby5IaXZlSWdub3JlS2V5VGV4dE91dHB1dEZvcm1hdApsb2NhdGlvbj1oZGZzXDovL2NscHFuMXBkaGRibW4wMS5pbmZvc29sY28ubmV0XDo4MDIwL2FwcHMvaGl2ZS93YXJlaG91c2Uva2R0ZXN0CnRyYW5zaWVudF9sYXN0RGRsVGltZT0xNDg3MDA2NDg4CiFIVUREISFITlBUISFIVUREIWZhbHNl"}]} 21:20:15.523197 IP 127.0.0.1.20432 > 127.0.0.1.51200: Flags [.], ack 1443, win 134, options [nop,nop,TS val 1901603281 ecr 1901603281], length 0 E..4..@.@.m.........O...Np...m[W........... qX).qX). 21:20:15.523402 IP 127.0.0.1.20432 > 127.0.0.1.51200: Flags [F.], seq 581, ack 1443, win 134, options [nop,nop,TS val 1901603281 ecr 1901603281], length 0 E..4..@.@.m.........O...Np...m[W........... qX).qX). 21:20:15.523457 IP 127.0.0.1.51200 > 127.0.0.1.20432: Flags [F.], seq 1443, ack 582, win 131, options [nop,nop,TS val 1901603281 ecr 1901603281], length 0 E..4|.@.@..(..........O..m[WNp............. qX).qX). 21:20:15.523464 IP 127.0.0.1.20432 > 127.0.0.1.51200: Flags [.], ack 1444, win 134, options [nop,nop,TS val 1901603281 ecr 1901603281], length 0 E..4..@.@.m.........O...Np...m[X........... qX).qX). {code} was (Author: kdunn926): Additionally, I observed all this using the following command, on the HAWQ master. First, start the {{tcpdump}} trace, then invoke a SELECT from a previously defined PXF table, either using HCatalog directly, or manually defining a PXF external table. {code} $ tcpdump -n port 51200 -A {code} Here is the conversation output: {code} 21:20:14.632410 IP 127.0.0.1.20416 > 127.0.0.1.51200: Flags [S], seq 3498721498, win 65483, options [mss 65495,sackOK,TS val 1901602390 ecr 1901547904,nop,wscale 9], length 0 E..<f.@.@..-........O.....@.........._......... qX&VqWQ.... 21:20:14.632421 IP 127.0.0.1.51200 > 127.0.0.1.20416: Flags [S.], seq 3752275736, ack 3498721499, win 65483, options [mss 65495,sackOK,TS val 1901602390 ecr 1901602390,nop,wscale 9], length 0 E..<..@.@.<...........O.../...@................ qX&VqX&V... 21:20:14.632428 IP 127.0.0.1.20416 > 127.0.0.1.51200: Flags [.], ack 1, win 128, options [nop,nop,TS val 1901602390 ecr 1901602390], length 0 E..4f.@.@..4........O.....@.../............ qX&VqX&V 21:20:14.632602 IP 127.0.0.1.20416 > 127.0.0.1.51200: Flags [P.], seq 1:318, ack 1, win 128, options [nop,nop,TS val 1901602390 ecr 1901602390], length 317 E..qf.@.@...........O.....@.../......e..... qX&VqX&VGET /pxf/v14/Metadata/getMetadata?profile=Hive&pattern=default.kdtest HTTP/1.1 Host: localhost:51200 Accept: application/json X-GP-SEGMENT-ID: -100005432 X-GP-SEGMENT-COUNT: 0 X-GP-XID: 2725021 X-GP-ALIGNMENT: 8 X-GP-URL-HOST: localhost X-GP-URL-PORT: 51200 X-GP-URI: localhost:51200/ X-GP-HAS-FILTER: 0 21:20:14.632607 IP 127.0.0.1.51200 > 127.0.0.1.20416: Flags [.], ack 318, win 130, options [nop,nop,TS val 1901602390 ecr 1901602390], length 0 E..4 O@.@.3s..........O.../...B............ qX&VqX&V 21:20:15.084890 IP 127.0.0.1.51200 > 127.0.0.1.20416: Flags [P.], seq 1:269, ack 318, win 130, options [nop,nop,TS val 1901602843 ecr 1901602390], length 268 E..@ P@.@.2f..........O.../...B......4..... qX(.qX&VHTTP/1.1 200 OK Server: Apache-Coyote/1.1 Content-Type: application/json Content-Length: 132 Date: Tue, 14 Feb 2017 05:20:15 GMT {"PXFMetadata":[{"item":{"path":"default","name":"kdtest"},"fields":[{"name":"key","type":"text"},{"name":"value","type":"text"}]}]} 21:20:15.084900 IP 127.0.0.1.20416 > 127.0.0.1.51200: Flags [.], ack 269, win 130, options [nop,nop,TS val 1901602843 ecr 1901602843], length 0 E..4f.@.@..2........O.....B...0%.....!..... qX(.qX(. 21:20:15.085229 IP 127.0.0.1.20416 > 127.0.0.1.51200: Flags [F.], seq 318, ack 269, win 130, options [nop,nop,TS val 1901602843 ecr 1901602843], length 0 E..4f.@.@..1........O.....B...0%..... ..... qX(.qX(. 21:20:15.085286 IP 127.0.0.1.51200 > 127.0.0.1.20416: Flags [F.], seq 269, ack 319, win 130, options [nop,nop,TS val 1901602843 ecr 1901602843], length 0 E..4 Q@.@.3q..........O...0%..B............ qX(.qX(. 21:20:15.085294 IP 127.0.0.1.20416 > 127.0.0.1.51200: Flags [.], ack 270, win 130, options [nop,nop,TS val 1901602843 ecr 1901602843], length 0 E..4f.@.@..0........O.....B...0&........... qX(.qX(. 21:20:15.112739 IP 127.0.0.1.20422 > 127.0.0.1.51200: Flags [S], seq 222439143, win 65483, options [mss 65495,sackOK,TS val 1901602870 ecr 1901602843,nop,wscale 9], length 0 B&...................... qX(6qX(.... 21:20:15.112765 IP 127.0.0.1.51200 > 127.0.0.1.20422: Flags [S.], seq 2606634976, ack 222439144, win 65483, options [mss 65495,sackOK,TS val 1901602871 ecr 1901602870,nop,wscale 9], length 0 B&....................O..^.. qX(7qX(6... 21:20:15.112773 IP 127.0.0.1.20422 > 127.0.0.1.51200: Flags [.], ack 1, win 128, options [nop,nop,TS val 1901602871 ecr 1901602871], length 0 B&..^......E........O... qX(7qX(7 21:20:15.112809 IP 127.0.0.1.20422 > 127.0.0.1.51200: Flags [P.], seq 1:581, ack 1, win 128, options [nop,nop,TS val 1901602871 ecr 1901602871], length 580 B&..^.......m.......O... qX(7qX(7GET /pxf/v14/Fragmenter/getFragments?path=default.kdtest HTTP/1.1 Host: localhost:51200 Accept: application/json X-GP-FORMAT: GPDBWritable X-GP-ATTRS: 2 X-GP-ATTR-NAME0: key X-GP-ATTR-TYPECODE0: 25 X-GP-ATTR-TYPENAME0: text X-GP-ATTR-NAME1: value X-GP-ATTR-TYPECODE1: 25 X-GP-ATTR-TYPENAME1: text X-GP-SEGMENT-ID: -100005432 X-GP-SEGMENT-COUNT: 0 X-GP-XID: 2725021 X-GP-ALIGNMENT: 8 X-GP-URL-HOST: localhost X-GP-URL-PORT: 51200 X-GP-DATA-DIR: default.kdtest X-GP-Profile: Hive X-GP-URI: pxf://localhost:51200/default.kdtest?Profile=Hive X-GP-HAS-FILTER: 0 21:20:15.112813 IP 127.0.0.1.51200 > 127.0.0.1.20422: Flags [.], ack 581, win 131, options [nop,nop,TS val 1901602871 ecr 1901602871], length 0 B),....C..............O..^.. qX(7qX(7 21:20:15.305723 IP 127.0.0.1.51200 > 127.0.0.1.20422: Flags [P.], seq 1:1443, ack 581, win 131, options [nop,nop,TS val 1901603063 ecr 1901602871], length 1442 B),...................O..^.. qX(.qX(7HTTP/1.1 200 OK Server: Apache-Coyote/1.1 Content-Type: application/json Content-Length: 1305 Date: Tue, 14 Feb 2017 05:20:15 GMT {"PXFFragments":[{"sourceName":"/apps/hive/warehouse/kdtest/hive-test-data.txt","index":0,"replicas":["10.215.181.12","10.215.181.11"],"metadata":"rO0ABXcQAAAAAAAAAAAAAAAAAAAAN3VyABNbTGphdmEubGFuZy5TdHJpbmc7rdJW5+kde0cCAAB4cAAAAAJ0AB1jbHBxbjFwZGhkYmRuMDIuaW5mb3NvbGNvLm5ldHQAHWNscHFuMXBkaGRiZG4wMS5pbmZvc29sY28ubmV0","userData":"b3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlRleHRJbnB1dEZvcm1hdCFIVUREIW9yZy5hcGFjaGUuaGFkb29wLmhpdmUuc2VyZGUyLmxhenkuTGF6eVNpbXBsZVNlckRlIUhVREQhIwojTW9uIEZlYiAxMyAyMToyMDoxNSBQU1QgMjAxNwpuYW1lPWRlZmF1bHQua2R0ZXN0Cm51bUZpbGVzPTEKZmllbGQuZGVsaW09LApjb2x1bW5zLnR5cGVzPXN0cmluZ1w6c3RyaW5nCnNlcmlhbGl6YXRpb24uZGRsPXN0cnVjdCBrZHRlc3QgeyBzdHJpbmcga2V5LCBzdHJpbmcgdmFsdWV9CmNvbHVtbnM9a2V5LHZhbHVlCnNlcmlhbGl6YXRpb24uZm9ybWF0PSwKY29sdW1ucy5jb21tZW50cz1cdTAwMDAKYnVja2V0X2NvdW50PS0xCnNlcmlhbGl6YXRpb24ubGliPW9yZy5hcGFjaGUuaGFkb29wLmhpdmUuc2VyZGUyLmxhenkuTGF6eVNpbXBsZVNlckRlCkNPTFVNTl9TVEFUU19BQ0NVUkFURT10cnVlCmZpbGUuaW5wdXRmb3JtYXQ9b3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlRleHRJbnB1dEZvcm1hdAp0b3RhbFNpemU9NTUKZmlsZS5vdXRwdXRmb3JtYXQ9b3JnLmFwYWNoZS5oYWRvb3AuaGl2ZS5xbC5pby5IaXZlSWdub3JlS2V5VGV4dE91dHB1dEZvcm1hdApsb2NhdGlvbj1oZGZzXDovL2NscHFuMXBkaGRibW4wMS5pbmZvc29sY28ubmV0XDo4MDIwL2FwcHMvaGl2ZS93YXJlaG91c2Uva2R0ZXN0CnRyYW5zaWVudF9sYXN0RGRsVGltZT0xNDg3MDA2NDg4CiFIVUREISFITlBUISFIVUREIWZhbHNl"}]} 21:20:15.305733 IP 127.0.0.1.20422 > 127.0.0.1.51200: Flags [.], ack 1443, win 134, options [nop,nop,TS val 1901603063 ecr 1901603063], length 0 B),.^......<........O... qX(.qX(. 21:20:15.305897 IP 127.0.0.1.20422 > 127.0.0.1.51200: Flags [F.], seq 581, ack 1443, win 134, options [nop,nop,TS val 1901603064 ecr 1901603063], length 0 B),.^......<}.......O... qX(.qX(. 21:20:15.305961 IP 127.0.0.1.51200 > 127.0.0.1.20422: Flags [F.], seq 1443, ack 582, win 131, options [nop,nop,TS val 1901603064 ecr 1901603064], length 0 B)-....<~.............O..^.. qX(.qX(. 21:20:15.305968 IP 127.0.0.1.20422 > 127.0.0.1.51200: Flags [.], ack 1444, win 134, options [nop,nop,TS val 1901603064 ecr 1901603064], length 0 B)-.^......<{.......O... qX(.qX(. 21:20:15.348058 IP 127.0.0.1.20432 > 127.0.0.1.51200: Flags [S], seq 1316022209, win 65483, options [mss 65495,sackOK,TS val 1901603106 ecr 1901603064,nop,wscale 9], length 0 E..<..@.@.m.........O...Np...........=......... qX)"qX(.... 21:20:15.348069 IP 127.0.0.1.51200 > 127.0.0.1.20432: Flags [S.], seq 124605876, ack 1316022210, win 65483, options [mss 65495,sackOK,TS val 1901603106 ecr 1901603106,nop,wscale 9], length 0 E..<..@.@.<...........O..mU.Np......o.......... qX)"qX)"... 21:20:15.348078 IP 127.0.0.1.20432 > 127.0.0.1.51200: Flags [.], ack 1, win 128, options [nop,nop,TS val 1901603106 ecr 1901603106], length 0 E..4..@.@.m.........O...Np...mU............ qX)"qX)" 21:20:15.348138 IP 127.0.0.1.20432 > 127.0.0.1.51200: Flags [P.], seq 1:581, ack 1, win 128, options [nop,nop,TS val 1901603106 ecr 1901603106], length 580 E..x..@.@.k.........O...Np...mU......m..... qX)"qX)"GET /pxf/v14/Fragmenter/getFragments?path=default.kdtest HTTP/1.1 Host: localhost:51200 Accept: application/json X-GP-FORMAT: GPDBWritable X-GP-ATTRS: 2 X-GP-ATTR-NAME0: key X-GP-ATTR-TYPECODE0: 25 X-GP-ATTR-TYPENAME0: text X-GP-ATTR-NAME1: value X-GP-ATTR-TYPECODE1: 25 X-GP-ATTR-TYPENAME1: text X-GP-SEGMENT-ID: -100005432 X-GP-SEGMENT-COUNT: 0 X-GP-XID: 2725021 X-GP-ALIGNMENT: 8 X-GP-URL-HOST: localhost X-GP-URL-PORT: 51200 X-GP-DATA-DIR: default.kdtest X-GP-Profile: Hive X-GP-URI: pxf://localhost:51200/default.kdtest?Profile=Hive X-GP-HAS-FILTER: 0 21:20:15.348144 IP 127.0.0.1.51200 > 127.0.0.1.20432: Flags [.], ack 581, win 131, options [nop,nop,TS val 1901603106 ecr 1901603106], length 0 E..4|.@.@..*..........O..mU.Np............. qX)"qX)" 21:20:15.523187 IP 127.0.0.1.51200 > 127.0.0.1.20432: Flags [P.], seq 1:1443, ack 581, win 131, options [nop,nop,TS val 1901603281 ecr 1901603106], length 1442 E...|.@.@.............O..mU.Np............. qX).qX)"HTTP/1.1 200 OK Server: Apache-Coyote/1.1 Content-Type: application/json Content-Length: 1305 Date: Tue, 14 Feb 2017 05:20:15 GMT {"PXFFragments":[{"sourceName":"/apps/hive/warehouse/kdtest/hive-test-data.txt","index":0,"replicas":["10.215.181.12","10.215.181.11"],"metadata":"rO0ABXcQAAAAAAAAAAAAAAAAAAAAN3VyABNbTGphdmEubGFuZy5TdHJpbmc7rdJW5+kde0cCAAB4cAAAAAJ0AB1jbHBxbjFwZGhkYmRuMDIuaW5mb3NvbGNvLm5ldHQAHWNscHFuMXBkaGRiZG4wMS5pbmZvc29sY28ubmV0","userData":"b3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlRleHRJbnB1dEZvcm1hdCFIVUREIW9yZy5hcGFjaGUuaGFkb29wLmhpdmUuc2VyZGUyLmxhenkuTGF6eVNpbXBsZVNlckRlIUhVREQhIwojTW9uIEZlYiAxMyAyMToyMDoxNSBQU1QgMjAxNwpuYW1lPWRlZmF1bHQua2R0ZXN0Cm51bUZpbGVzPTEKZmllbGQuZGVsaW09LApjb2x1bW5zLnR5cGVzPXN0cmluZ1w6c3RyaW5nCnNlcmlhbGl6YXRpb24uZGRsPXN0cnVjdCBrZHRlc3QgeyBzdHJpbmcga2V5LCBzdHJpbmcgdmFsdWV9CmNvbHVtbnM9a2V5LHZhbHVlCnNlcmlhbGl6YXRpb24uZm9ybWF0PSwKY29sdW1ucy5jb21tZW50cz1cdTAwMDAKYnVja2V0X2NvdW50PS0xCnNlcmlhbGl6YXRpb24ubGliPW9yZy5hcGFjaGUuaGFkb29wLmhpdmUuc2VyZGUyLmxhenkuTGF6eVNpbXBsZVNlckRlCkNPTFVNTl9TVEFUU19BQ0NVUkFURT10cnVlCmZpbGUuaW5wdXRmb3JtYXQ9b3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlRleHRJbnB1dEZvcm1hdAp0b3RhbFNpemU9NTUKZmlsZS5vdXRwdXRmb3JtYXQ9b3JnLmFwYWNoZS5oYWRvb3AuaGl2ZS5xbC5pby5IaXZlSWdub3JlS2V5VGV4dE91dHB1dEZvcm1hdApsb2NhdGlvbj1oZGZzXDovL2NscHFuMXBkaGRibW4wMS5pbmZvc29sY28ubmV0XDo4MDIwL2FwcHMvaGl2ZS93YXJlaG91c2Uva2R0ZXN0CnRyYW5zaWVudF9sYXN0RGRsVGltZT0xNDg3MDA2NDg4CiFIVUREISFITlBUISFIVUREIWZhbHNl"}]} 21:20:15.523197 IP 127.0.0.1.20432 > 127.0.0.1.51200: Flags [.], ack 1443, win 134, options [nop,nop,TS val 1901603281 ecr 1901603281], length 0 E..4..@.@.m.........O...Np...m[W........... qX).qX). 21:20:15.523402 IP 127.0.0.1.20432 > 127.0.0.1.51200: Flags [F.], seq 581, ack 1443, win 134, options [nop,nop,TS val 1901603281 ecr 1901603281], length 0 E..4..@.@.m.........O...Np...m[W........... qX).qX). 21:20:15.523457 IP 127.0.0.1.51200 > 127.0.0.1.20432: Flags [F.], seq 1443, ack 582, win 131, options [nop,nop,TS val 1901603281 ecr 1901603281], length 0 E..4|.@.@..(..........O..m[WNp............. qX).qX). 21:20:15.523464 IP 127.0.0.1.20432 > 127.0.0.1.51200: Flags [.], ack 1444, win 134, options [nop,nop,TS val 1901603281 ecr 1901603281], length 0 E..4..@.@.m.........O...Np...m[X........... qX).qX). {code} > Document HAWQ to PXF APIs > ------------------------- > > Key: HAWQ-1234 > URL: https://issues.apache.org/jira/browse/HAWQ-1234 > Project: Apache HAWQ > Issue Type: Sub-task > Components: PXF > Reporter: Roman Shaposhnik > Assignee: Roman Shaposhnik > Attachments: PXFAdvancedStatsplan.pdf > > > It would be very useful to start documenting HAWQ to PXF APIs. The right > places to start are: > * libcurl (a thin wrapper for making HAWQ C code be able to do REST calls): > https://github.com/apache/incubator-hawq/blob/master/src/include/access/libchurl.h > https://github.com/apache/incubator-hawq/blob/master/src/backend/access/external/libchurl.c > * pxfmasterapi (mostly metadata calls that master is doing): > https://github.com/apache/incubator-hawq/blob/master/src/backend/access/external/pxfmasterapi.c > Here you will find how HAWQ via PXF pulls using a REST API to get external > metadata and some logic to parse the JSON response. > * gpbridgeapi (segment calls to PXF): > https://github.com/apache/incubator-hawq/blob/master/src/bin/gpfusion/gpbridgeapi.c > Here you will find other examples of (read and write calls) used to fetch > external data. > Design doc on PXF's support for analyze (pxf's analyzer) is attached -- This message was sent by Atlassian JIRA (v6.3.15#6346)