[jira] [Commented] (HIVE-5325) Implement statistics providing ORC writer and reader interfaces
[
https://issues.apache.org/jira/browse/HIVE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782686#comment-13782686
]
Hive QA commented on HIVE-5325:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12606028/HIVE-5325.3.patch.txt
{color:green}SUCCESS:{color} +1 4077 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/978/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/978/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
Implement statistics providing ORC writer and reader interfaces
---
Key: HIVE-5325
URL: https://issues.apache.org/jira/browse/HIVE-5325
Project: Hive
Issue Type: New Feature
Affects Versions: 0.13.0
Reporter: Prasanth J
Assignee: Prasanth J
Labels: orcfile, statistics
Fix For: 0.13.0
Attachments: HIVE-5325.1.patch.txt, HIVE-5325.2.patch.txt,
HIVE-5325.3.patch.txt, HIVE-5325-java-only.1.patch.txt,
HIVE-5325-java-only.2.patch.txt, HIVE-5325-java-only.3.patch.txt
HIVE-5324 adds new interfaces that can be implemented by ORC reader/writer to
provide statistics. Writer provided statistics is used to update
table/partition level statistics in metastore. Reader provided statistics can
be used for reducer estimation, CBO etc. in the absence of metastore
statistics.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-5325) Implement statistics providing ORC writer and reader interfaces
[ https://issues.apache.org/jira/browse/HIVE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783022#comment-13783022 ] Hudson commented on HIVE-5325: -- FAILURE: Integrated in Hive-trunk-h0.21 #2370 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2370/]) HIVE-5325 : Implement statistics providing ORC writer and reader interfaces (Prasanth J via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1528108) * /hive/trunk/ql/src/gen/protobuf/gen-java/org/apache/hadoop/hive/ql/io/orc/OrcProto.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/BinaryColumnStatistics.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ColumnStatisticsImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/StringColumnStatistics.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/util/JavaDataModel.java * /hive/trunk/ql/src/protobuf/org/apache/hadoop/hive/ql/io/orc/orc_proto.proto * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcNullOptimization.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSerDeStats.java * /hive/trunk/ql/src/test/resources/orc-file-dump-dictionary-threshold.out * /hive/trunk/ql/src/test/resources/orc-file-dump.out Implement statistics providing ORC writer and reader interfaces --- Key: HIVE-5325 URL: https://issues.apache.org/jira/browse/HIVE-5325 Project: Hive Issue Type: New Feature Affects Versions: 0.13.0 Reporter: Prasanth J Assignee: Prasanth J Labels: orcfile, statistics Fix For: 0.13.0 Attachments: HIVE-5325.1.patch.txt, HIVE-5325.2.patch.txt, HIVE-5325.3.patch.txt, HIVE-5325-java-only.1.patch.txt, HIVE-5325-java-only.2.patch.txt, HIVE-5325-java-only.3.patch.txt HIVE-5324 adds new interfaces that can be implemented by ORC reader/writer to provide statistics. Writer provided statistics is used to update table/partition level statistics in metastore. Reader provided statistics can be used for reducer estimation, CBO etc. in the absence of metastore statistics. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5325) Implement statistics providing ORC writer and reader interfaces
[ https://issues.apache.org/jira/browse/HIVE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783251#comment-13783251 ] Hudson commented on HIVE-5325: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #122 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/122/]) HIVE-5325 : Implement statistics providing ORC writer and reader interfaces (Prasanth J via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1528108) * /hive/trunk/ql/src/gen/protobuf/gen-java/org/apache/hadoop/hive/ql/io/orc/OrcProto.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/BinaryColumnStatistics.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ColumnStatisticsImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/StringColumnStatistics.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/util/JavaDataModel.java * /hive/trunk/ql/src/protobuf/org/apache/hadoop/hive/ql/io/orc/orc_proto.proto * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcNullOptimization.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSerDeStats.java * /hive/trunk/ql/src/test/resources/orc-file-dump-dictionary-threshold.out * /hive/trunk/ql/src/test/resources/orc-file-dump.out Implement statistics providing ORC writer and reader interfaces --- Key: HIVE-5325 URL: https://issues.apache.org/jira/browse/HIVE-5325 Project: Hive Issue Type: New Feature Affects Versions: 0.13.0 Reporter: Prasanth J Assignee: Prasanth J Labels: orcfile, statistics Fix For: 0.13.0 Attachments: HIVE-5325.1.patch.txt, HIVE-5325.2.patch.txt, HIVE-5325.3.patch.txt, HIVE-5325-java-only.1.patch.txt, HIVE-5325-java-only.2.patch.txt, HIVE-5325-java-only.3.patch.txt HIVE-5324 adds new interfaces that can be implemented by ORC reader/writer to provide statistics. Writer provided statistics is used to update table/partition level statistics in metastore. Reader provided statistics can be used for reducer estimation, CBO etc. in the absence of metastore statistics. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5325) Implement statistics providing ORC writer and reader interfaces
[ https://issues.apache.org/jira/browse/HIVE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783363#comment-13783363 ] Hudson commented on HIVE-5325: -- SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #188 (See [https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/188/]) HIVE-5325 : Implement statistics providing ORC writer and reader interfaces (Prasanth J via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1528108) * /hive/trunk/ql/src/gen/protobuf/gen-java/org/apache/hadoop/hive/ql/io/orc/OrcProto.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/BinaryColumnStatistics.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ColumnStatisticsImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/StringColumnStatistics.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/util/JavaDataModel.java * /hive/trunk/ql/src/protobuf/org/apache/hadoop/hive/ql/io/orc/orc_proto.proto * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcNullOptimization.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSerDeStats.java * /hive/trunk/ql/src/test/resources/orc-file-dump-dictionary-threshold.out * /hive/trunk/ql/src/test/resources/orc-file-dump.out Implement statistics providing ORC writer and reader interfaces --- Key: HIVE-5325 URL: https://issues.apache.org/jira/browse/HIVE-5325 Project: Hive Issue Type: New Feature Affects Versions: 0.13.0 Reporter: Prasanth J Assignee: Prasanth J Labels: orcfile, statistics Fix For: 0.13.0 Attachments: HIVE-5325.1.patch.txt, HIVE-5325.2.patch.txt, HIVE-5325.3.patch.txt, HIVE-5325-java-only.1.patch.txt, HIVE-5325-java-only.2.patch.txt, HIVE-5325-java-only.3.patch.txt HIVE-5324 adds new interfaces that can be implemented by ORC reader/writer to provide statistics. Writer provided statistics is used to update table/partition level statistics in metastore. Reader provided statistics can be used for reducer estimation, CBO etc. in the absence of metastore statistics. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5325) Implement statistics providing ORC writer and reader interfaces
[ https://issues.apache.org/jira/browse/HIVE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13781975#comment-13781975 ] Ashutosh Chauhan commented on HIVE-5325: Left some comments on RB. Implement statistics providing ORC writer and reader interfaces --- Key: HIVE-5325 URL: https://issues.apache.org/jira/browse/HIVE-5325 Project: Hive Issue Type: New Feature Affects Versions: 0.13.0 Reporter: Prasanth J Assignee: Prasanth J Labels: orcfile, statistics Fix For: 0.13.0 Attachments: HIVE-5325.1.patch.txt, HIVE-5325.2.patch.txt, HIVE-5325-java-only.1.patch.txt, HIVE-5325-java-only.2.patch.txt HIVE-5324 adds new interfaces that can be implemented by ORC reader/writer to provide statistics. Writer provided statistics is used to update table/partition level statistics in metastore. Reader provided statistics can be used for reducer estimation, CBO etc. in the absence of metastore statistics. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5325) Implement statistics providing ORC writer and reader interfaces
[ https://issues.apache.org/jira/browse/HIVE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782558#comment-13782558 ] Prasanth J commented on HIVE-5325: -- [~ashutoshc] which among the two patch will HiveQA pick? I have java-only patch and protobuf generated code patch. Implement statistics providing ORC writer and reader interfaces --- Key: HIVE-5325 URL: https://issues.apache.org/jira/browse/HIVE-5325 Project: Hive Issue Type: New Feature Affects Versions: 0.13.0 Reporter: Prasanth J Assignee: Prasanth J Labels: orcfile, statistics Fix For: 0.13.0 Attachments: HIVE-5325.1.patch.txt, HIVE-5325.2.patch.txt, HIVE-5325.3.patch.txt, HIVE-5325-java-only.1.patch.txt, HIVE-5325-java-only.2.patch.txt, HIVE-5325-java-only.3.patch.txt HIVE-5324 adds new interfaces that can be implemented by ORC reader/writer to provide statistics. Writer provided statistics is used to update table/partition level statistics in metastore. Reader provided statistics can be used for reducer estimation, CBO etc. in the absence of metastore statistics. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5325) Implement statistics providing ORC writer and reader interfaces
[ https://issues.apache.org/jira/browse/HIVE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782577#comment-13782577 ] Ashutosh Chauhan commented on HIVE-5325: +1 It will pick the latest patch which conforms to naming convention. Implement statistics providing ORC writer and reader interfaces --- Key: HIVE-5325 URL: https://issues.apache.org/jira/browse/HIVE-5325 Project: Hive Issue Type: New Feature Affects Versions: 0.13.0 Reporter: Prasanth J Assignee: Prasanth J Labels: orcfile, statistics Fix For: 0.13.0 Attachments: HIVE-5325.1.patch.txt, HIVE-5325.2.patch.txt, HIVE-5325.3.patch.txt, HIVE-5325-java-only.1.patch.txt, HIVE-5325-java-only.2.patch.txt, HIVE-5325-java-only.3.patch.txt HIVE-5324 adds new interfaces that can be implemented by ORC reader/writer to provide statistics. Writer provided statistics is used to update table/partition level statistics in metastore. Reader provided statistics can be used for reducer estimation, CBO etc. in the absence of metastore statistics. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5325) Implement statistics providing ORC writer and reader interfaces
[
https://issues.apache.org/jira/browse/HIVE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782651#comment-13782651
]
Hive QA commented on HIVE-5325:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12606028/HIVE-5325.3.patch.txt
{color:green}SUCCESS:{color} +1 4077 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/977/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/977/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
Implement statistics providing ORC writer and reader interfaces
---
Key: HIVE-5325
URL: https://issues.apache.org/jira/browse/HIVE-5325
Project: Hive
Issue Type: New Feature
Affects Versions: 0.13.0
Reporter: Prasanth J
Assignee: Prasanth J
Labels: orcfile, statistics
Fix For: 0.13.0
Attachments: HIVE-5325.1.patch.txt, HIVE-5325.2.patch.txt,
HIVE-5325.3.patch.txt, HIVE-5325-java-only.1.patch.txt,
HIVE-5325-java-only.2.patch.txt, HIVE-5325-java-only.3.patch.txt
HIVE-5324 adds new interfaces that can be implemented by ORC reader/writer to
provide statistics. Writer provided statistics is used to update
table/partition level statistics in metastore. Reader provided statistics can
be used for reducer estimation, CBO etc. in the absence of metastore
statistics.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-5325) Implement statistics providing ORC writer and reader interfaces
[ https://issues.apache.org/jira/browse/HIVE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13776836#comment-13776836 ] Prasanth J commented on HIVE-5325: -- [~ashutoshc] refreshed the patch. Updated RB as well. Its now ready for review :) Implement statistics providing ORC writer and reader interfaces --- Key: HIVE-5325 URL: https://issues.apache.org/jira/browse/HIVE-5325 Project: Hive Issue Type: New Feature Affects Versions: 0.13.0 Reporter: Prasanth J Assignee: Prasanth J Labels: orcfile, statistics Fix For: 0.13.0 Attachments: HIVE-5325.1.patch.txt, HIVE-5325.2.patch.txt, HIVE-5325-java-only.1.patch.txt, HIVE-5325-java-only.2.patch.txt HIVE-5324 adds new interfaces that can be implemented by ORC reader/writer to provide statistics. Writer provided statistics is used to update table/partition level statistics in metastore. Reader provided statistics can be used for reducer estimation, CBO etc. in the absence of metastore statistics. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5325) Implement statistics providing ORC writer and reader interfaces
[ https://issues.apache.org/jira/browse/HIVE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13772551#comment-13772551 ] Prasanth J commented on HIVE-5325: -- The attached patch is generated on top of HIVE-5324. Here is RB link https://reviews.apache.org/r/14243/ Implement statistics providing ORC writer and reader interfaces --- Key: HIVE-5325 URL: https://issues.apache.org/jira/browse/HIVE-5325 Project: Hive Issue Type: New Feature Affects Versions: 0.13.0 Reporter: Prasanth J Assignee: Prasanth J Labels: orcfile, statistics Fix For: 0.13.0 Attachments: HIVE-5325.1.patch.txt, HIVE-5325-java-only.1.patch.txt HIVE-5324 adds new interfaces that can be implemented by ORC reader/writer to provide statistics. Writer provided statistics is used to update table/partition level statistics in metastore. Reader provided statistics can be used for reducer estimation, CBO etc. in the absence of metastore statistics. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
