[jira] [Commented] (HAWQ-304) Support update and delete on non-heap tables
[ https://issues.apache.org/jira/browse/HAWQ-304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936618#comment-15936618 ] Christian Tzolov commented on HAWQ-304: --- Glad to hear it works [~hongxu ma]! We are still working on correctness vs. efficiency implementation of the INSERT rule. We may end-up implementing two different InsertRule strategies and allow the user to configure the prefered one: https://github.com/tzolov/calcite-sql-rewriter/issues/2 Another observation. It seems like HAWQ doesn't support OVER-PARTITION-BY without an explicit ORDER BY! Furthermore it seems that HAWQ allays performs internal Sort My interpretation of Postgres spec: https://www.postgresql.org/docs/current/static/sql-expressions.html#SYNTAX-WINDOW-FUNCTIONS is that a missing `frame_clause` is equal to `ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING`. In both cases you shouldn't need ORDER BY. I assume that unnecessary SORT would incur undesired performance overhead. > Support update and delete on non-heap tables > > > Key: HAWQ-304 > URL: https://issues.apache.org/jira/browse/HAWQ-304 > Project: Apache HAWQ > Issue Type: Wish > Components: Storage >Reporter: Lei Chang >Assignee: Lili Ma > Fix For: 3.0.0.0 > > Attachments: mutable_table.sql > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HAWQ-304) Support update and delete on non-heap tables
[ https://issues.apache.org/jira/browse/HAWQ-304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926286#comment-15926286 ] Christian Tzolov commented on HAWQ-304: --- FYI, we have been working on this workaround: https://github.com/tzolov/calcite-sql-rewriter It leverages Apache Calcite to implement JDBC driver that converts the INSERT, UPDATE and DELETE statements into append-only INSERTs. E.g. instead of updating rows in-place it inserts the new version of the row along with version metadata. > Support update and delete on non-heap tables > > > Key: HAWQ-304 > URL: https://issues.apache.org/jira/browse/HAWQ-304 > Project: Apache HAWQ > Issue Type: Wish > Components: Storage >Reporter: Lei Chang >Assignee: Lili Ma > Fix For: 3.0.0.0 > > Attachments: mutable_table.sql > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HAWQ-178) Add JSON plugin support in code base
[ https://issues.apache.org/jira/browse/HAWQ-178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219067#comment-15219067 ] Christian Tzolov edited comment on HAWQ-178 at 3/31/16 7:05 AM: [~hellstorm] i'm experimenting with the JsonPath (https://github.com/jayway/JsonPath) library. There is test pxf-json plugin prototype based on JsonPath that allows you to crawl and flatten nested json structures including arrays. But this is an experimental work and will not be part of the first pxf-json release. Still if you are interested you can give it try (and bring feedback). Mind though that it might never became part of the official code. *Update:* Note that pxf-json only allows you to to define the column names as selectors to extract particular JSON element(s) or attributes(s). The selected result _must_ be mappable to _known HAWQ Column Type_. There are not Array column types! So jsonpath will only help you to write more sophisticated expressions to select members of nested arrays or other structures. Still if you need to extract lengthy arrays it will not be sufficient to extract each array member one by one (running the column name expressions). Instead you should consider the approach explained by [~shivram] above or run MapReduce-like ETL to flatter the json before applying pxf (Apache Crunch is my favorite tool for the second). was (Author: tzolov): [~hellstorm] i'm experimenting with the JsonPath (https://github.com/jayway/JsonPath) library. There is test pxf-json plugin prototype based on JsonPath that allows you to crawl and flatten nested json structures including arrays. But this is an experimental work and will not be part of the first pxf-json release. Still if you are interested you can give it try (and bring feedback). Mind though that it might never became part of the official code. > Add JSON plugin support in code base > > > Key: HAWQ-178 > URL: https://issues.apache.org/jira/browse/HAWQ-178 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF >Reporter: Goden Yao >Assignee: Christian Tzolov > Fix For: backlog > > Attachments: PXFJSONPluginforHAWQ2.0andPXF3.0.0.pdf, > PXFJSONPluginforHAWQ2.0andPXF3.0.0v.2.pdf, > PXFJSONPluginforHAWQ2.0andPXF3.0.0v.3.pdf > > > JSON has been a popular format used in HDFS as well as in the community, > there has been a few JSON PXF plugins developed by the community and we'd > like to see it being incorporated into the code base as an optional package. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-178) Add JSON plugin support in code base
[ https://issues.apache.org/jira/browse/HAWQ-178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219067#comment-15219067 ] Christian Tzolov commented on HAWQ-178: --- [~hellstorm] i'm experimenting with the JsonPath (https://github.com/jayway/JsonPath) library. There is test pxf-json plugin prototype based on JsonPath that allows you to crawl and flatten nested json structures including arrays. But this is an experimental work and will not be part of the first pxf-json release. Still if you are interested you can give it try (and bring feedback). Mind though that it might never became part of the official code. > Add JSON plugin support in code base > > > Key: HAWQ-178 > URL: https://issues.apache.org/jira/browse/HAWQ-178 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF >Reporter: Goden Yao >Assignee: Christian Tzolov > Fix For: backlog > > Attachments: PXFJSONPluginforHAWQ2.0andPXF3.0.0.pdf, > PXFJSONPluginforHAWQ2.0andPXF3.0.0v.2.pdf, > PXFJSONPluginforHAWQ2.0andPXF3.0.0v.3.pdf > > > JSON has been a popular format used in HDFS as well as in the community, > there has been a few JSON PXF plugins developed by the community and we'd > like to see it being incorporated into the code base as an optional package. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-178) Add JSON plugin support in code base
[ https://issues.apache.org/jira/browse/HAWQ-178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128758#comment-15128758 ] Christian Tzolov commented on HAWQ-178: --- [~GodenYao] should i assign this ticket to me? > Add JSON plugin support in code base > > > Key: HAWQ-178 > URL: https://issues.apache.org/jira/browse/HAWQ-178 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF >Reporter: Goden Yao >Assignee: Goden Yao > Fix For: backlog > > Attachments: PXFJSONPluginforHAWQ2.0andPXF3.0.0.pdf, > PXFJSONPluginforHAWQ2.0andPXF3.0.0v.2.pdf > > > JSON has been a popular format used in HDFS as well as in the community, > there has been a few JSON PXF plugins developed by the community and we'd > like to see it being incorporated into the code base as an optional package. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-178) Add JSON plugin support in code base
[ https://issues.apache.org/jira/browse/HAWQ-178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christian Tzolov updated HAWQ-178: -- Attachment: PXFJSONPluginforHAWQ2.0andPXF3.0.0v.3.pdf Update the how-to guide to reflect the new multiline json support > Add JSON plugin support in code base > > > Key: HAWQ-178 > URL: https://issues.apache.org/jira/browse/HAWQ-178 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF >Reporter: Goden Yao >Assignee: Christian Tzolov > Fix For: backlog > > Attachments: PXFJSONPluginforHAWQ2.0andPXF3.0.0.pdf, > PXFJSONPluginforHAWQ2.0andPXF3.0.0v.2.pdf, > PXFJSONPluginforHAWQ2.0andPXF3.0.0v.3.pdf > > > JSON has been a popular format used in HDFS as well as in the community, > there has been a few JSON PXF plugins developed by the community and we'd > like to see it being incorporated into the code base as an optional package. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-178) Add JSON plugin support in code base
[ https://issues.apache.org/jira/browse/HAWQ-178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christian Tzolov updated HAWQ-178: -- Attachment: PXFJSONPluginforHAWQ2.0andPXF3.0.0v.2.pdf Updated documentation > Add JSON plugin support in code base > > > Key: HAWQ-178 > URL: https://issues.apache.org/jira/browse/HAWQ-178 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF >Reporter: Goden Yao >Assignee: Goden Yao > Fix For: backlog > > Attachments: PXFJSONPluginforHAWQ2.0andPXF3.0.0.pdf, > PXFJSONPluginforHAWQ2.0andPXF3.0.0v.2.pdf > > > JSON has been a popular format used in HDFS as well as in the community, > there has been a few JSON PXF plugins developed by the community and we'd > like to see it being incorporated into the code base as an optional package. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-178) Add JSON plugin support in code base
[ https://issues.apache.org/jira/browse/HAWQ-178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christian Tzolov updated HAWQ-178: -- Attachment: PXFJSONPluginforHAWQ2.0andPXF3.0.0.pdf Instructions how to install pxf-json manually. > Add JSON plugin support in code base > > > Key: HAWQ-178 > URL: https://issues.apache.org/jira/browse/HAWQ-178 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF >Reporter: Goden Yao >Assignee: Goden Yao > Fix For: backlog > > Attachments: PXFJSONPluginforHAWQ2.0andPXF3.0.0.pdf > > > JSON has been a popular format used in HDFS as well as in the community, > there has been a few JSON PXF plugins developed by the community and we'd > like to see it being incorporated into the code base as an optional package. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-178) Add JSON plugin support in code base
[ https://issues.apache.org/jira/browse/HAWQ-178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15081312#comment-15081312 ] Christian Tzolov commented on HAWQ-178: --- The initial work on the pxf-json port is here: https://github.com/tzolov/incubator-hawq/tree/HAWQ-178 But before i open a new PR I would like to test it in a real HAWQ environment. I've build the docker HAWQ image but not sure how to install it on running HDP or PHD distro? > Add JSON plugin support in code base > > > Key: HAWQ-178 > URL: https://issues.apache.org/jira/browse/HAWQ-178 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF >Reporter: Goden Yao >Assignee: Goden Yao > > JSON has been a popular format used in HDFS as well as in the community, > there has been a few JSON PXF plugins developed by the community and we'd > like to see it being incorporated into the code base as an optional package. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
