[jira] [Commented] (HIVE-5325) Implement statistics providing ORC writer and reader interfaces
[ https://issues.apache.org/jira/browse/HIVE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13783698#comment-13783698 ] Hudson commented on HIVE-5325: -- ABORTED: Integrated in Hive-trunk-hadoop2 #468 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/468/]) HIVE-5325 : Implement statistics providing ORC writer and reader interfaces (Prasanth J via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1528108) * /hive/trunk/ql/src/gen/protobuf/gen-java/org/apache/hadoop/hive/ql/io/orc/OrcProto.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/BinaryColumnStatistics.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ColumnStatisticsImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/StringColumnStatistics.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/util/JavaDataModel.java * /hive/trunk/ql/src/protobuf/org/apache/hadoop/hive/ql/io/orc/orc_proto.proto * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcNullOptimization.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSerDeStats.java * /hive/trunk/ql/src/test/resources/orc-file-dump-dictionary-threshold.out * /hive/trunk/ql/src/test/resources/orc-file-dump.out > Implement statistics providing ORC writer and reader interfaces > --- > > Key: HIVE-5325 > URL: https://issues.apache.org/jira/browse/HIVE-5325 > Project: Hive > Issue Type: New Feature >Affects Versions: 0.13.0 >Reporter: Prasanth J >Assignee: Prasanth J > Labels: orcfile, statistics > Fix For: 0.13.0 > > Attachments: HIVE-5325.1.patch.txt, HIVE-5325.2.patch.txt, > HIVE-5325.3.patch.txt, HIVE-5325-java-only.1.patch.txt, > HIVE-5325-java-only.2.patch.txt, HIVE-5325-java-only.3.patch.txt > > > HIVE-5324 adds new interfaces that can be implemented by ORC reader/writer to > provide statistics. Writer provided statistics is used to update > table/partition level statistics in metastore. Reader provided statistics can > be used for reducer estimation, CBO etc. in the absence of metastore > statistics. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5325) Implement statistics providing ORC writer and reader interfaces
[ https://issues.apache.org/jira/browse/HIVE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13783363#comment-13783363 ] Hudson commented on HIVE-5325: -- SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #188 (See [https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/188/]) HIVE-5325 : Implement statistics providing ORC writer and reader interfaces (Prasanth J via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1528108) * /hive/trunk/ql/src/gen/protobuf/gen-java/org/apache/hadoop/hive/ql/io/orc/OrcProto.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/BinaryColumnStatistics.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ColumnStatisticsImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/StringColumnStatistics.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/util/JavaDataModel.java * /hive/trunk/ql/src/protobuf/org/apache/hadoop/hive/ql/io/orc/orc_proto.proto * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcNullOptimization.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSerDeStats.java * /hive/trunk/ql/src/test/resources/orc-file-dump-dictionary-threshold.out * /hive/trunk/ql/src/test/resources/orc-file-dump.out > Implement statistics providing ORC writer and reader interfaces > --- > > Key: HIVE-5325 > URL: https://issues.apache.org/jira/browse/HIVE-5325 > Project: Hive > Issue Type: New Feature >Affects Versions: 0.13.0 >Reporter: Prasanth J >Assignee: Prasanth J > Labels: orcfile, statistics > Fix For: 0.13.0 > > Attachments: HIVE-5325.1.patch.txt, HIVE-5325.2.patch.txt, > HIVE-5325.3.patch.txt, HIVE-5325-java-only.1.patch.txt, > HIVE-5325-java-only.2.patch.txt, HIVE-5325-java-only.3.patch.txt > > > HIVE-5324 adds new interfaces that can be implemented by ORC reader/writer to > provide statistics. Writer provided statistics is used to update > table/partition level statistics in metastore. Reader provided statistics can > be used for reducer estimation, CBO etc. in the absence of metastore > statistics. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5325) Implement statistics providing ORC writer and reader interfaces
[ https://issues.apache.org/jira/browse/HIVE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13783251#comment-13783251 ] Hudson commented on HIVE-5325: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #122 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/122/]) HIVE-5325 : Implement statistics providing ORC writer and reader interfaces (Prasanth J via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1528108) * /hive/trunk/ql/src/gen/protobuf/gen-java/org/apache/hadoop/hive/ql/io/orc/OrcProto.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/BinaryColumnStatistics.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ColumnStatisticsImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/StringColumnStatistics.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/util/JavaDataModel.java * /hive/trunk/ql/src/protobuf/org/apache/hadoop/hive/ql/io/orc/orc_proto.proto * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcNullOptimization.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSerDeStats.java * /hive/trunk/ql/src/test/resources/orc-file-dump-dictionary-threshold.out * /hive/trunk/ql/src/test/resources/orc-file-dump.out > Implement statistics providing ORC writer and reader interfaces > --- > > Key: HIVE-5325 > URL: https://issues.apache.org/jira/browse/HIVE-5325 > Project: Hive > Issue Type: New Feature >Affects Versions: 0.13.0 >Reporter: Prasanth J >Assignee: Prasanth J > Labels: orcfile, statistics > Fix For: 0.13.0 > > Attachments: HIVE-5325.1.patch.txt, HIVE-5325.2.patch.txt, > HIVE-5325.3.patch.txt, HIVE-5325-java-only.1.patch.txt, > HIVE-5325-java-only.2.patch.txt, HIVE-5325-java-only.3.patch.txt > > > HIVE-5324 adds new interfaces that can be implemented by ORC reader/writer to > provide statistics. Writer provided statistics is used to update > table/partition level statistics in metastore. Reader provided statistics can > be used for reducer estimation, CBO etc. in the absence of metastore > statistics. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5325) Implement statistics providing ORC writer and reader interfaces
[ https://issues.apache.org/jira/browse/HIVE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13783022#comment-13783022 ] Hudson commented on HIVE-5325: -- FAILURE: Integrated in Hive-trunk-h0.21 #2370 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2370/]) HIVE-5325 : Implement statistics providing ORC writer and reader interfaces (Prasanth J via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1528108) * /hive/trunk/ql/src/gen/protobuf/gen-java/org/apache/hadoop/hive/ql/io/orc/OrcProto.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/BinaryColumnStatistics.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ColumnStatisticsImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/StringColumnStatistics.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/util/JavaDataModel.java * /hive/trunk/ql/src/protobuf/org/apache/hadoop/hive/ql/io/orc/orc_proto.proto * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcNullOptimization.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSerDeStats.java * /hive/trunk/ql/src/test/resources/orc-file-dump-dictionary-threshold.out * /hive/trunk/ql/src/test/resources/orc-file-dump.out > Implement statistics providing ORC writer and reader interfaces > --- > > Key: HIVE-5325 > URL: https://issues.apache.org/jira/browse/HIVE-5325 > Project: Hive > Issue Type: New Feature >Affects Versions: 0.13.0 >Reporter: Prasanth J >Assignee: Prasanth J > Labels: orcfile, statistics > Fix For: 0.13.0 > > Attachments: HIVE-5325.1.patch.txt, HIVE-5325.2.patch.txt, > HIVE-5325.3.patch.txt, HIVE-5325-java-only.1.patch.txt, > HIVE-5325-java-only.2.patch.txt, HIVE-5325-java-only.3.patch.txt > > > HIVE-5324 adds new interfaces that can be implemented by ORC reader/writer to > provide statistics. Writer provided statistics is used to update > table/partition level statistics in metastore. Reader provided statistics can > be used for reducer estimation, CBO etc. in the absence of metastore > statistics. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5325) Implement statistics providing ORC writer and reader interfaces
[ https://issues.apache.org/jira/browse/HIVE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782686#comment-13782686 ] Hive QA commented on HIVE-5325: --- {color:green}Overall{color}: +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12606028/HIVE-5325.3.patch.txt {color:green}SUCCESS:{color} +1 4077 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/978/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/978/console Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. > Implement statistics providing ORC writer and reader interfaces > --- > > Key: HIVE-5325 > URL: https://issues.apache.org/jira/browse/HIVE-5325 > Project: Hive > Issue Type: New Feature >Affects Versions: 0.13.0 >Reporter: Prasanth J >Assignee: Prasanth J > Labels: orcfile, statistics > Fix For: 0.13.0 > > Attachments: HIVE-5325.1.patch.txt, HIVE-5325.2.patch.txt, > HIVE-5325.3.patch.txt, HIVE-5325-java-only.1.patch.txt, > HIVE-5325-java-only.2.patch.txt, HIVE-5325-java-only.3.patch.txt > > > HIVE-5324 adds new interfaces that can be implemented by ORC reader/writer to > provide statistics. Writer provided statistics is used to update > table/partition level statistics in metastore. Reader provided statistics can > be used for reducer estimation, CBO etc. in the absence of metastore > statistics. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5325) Implement statistics providing ORC writer and reader interfaces
[ https://issues.apache.org/jira/browse/HIVE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782651#comment-13782651 ] Hive QA commented on HIVE-5325: --- {color:green}Overall{color}: +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12606028/HIVE-5325.3.patch.txt {color:green}SUCCESS:{color} +1 4077 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/977/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/977/console Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. > Implement statistics providing ORC writer and reader interfaces > --- > > Key: HIVE-5325 > URL: https://issues.apache.org/jira/browse/HIVE-5325 > Project: Hive > Issue Type: New Feature >Affects Versions: 0.13.0 >Reporter: Prasanth J >Assignee: Prasanth J > Labels: orcfile, statistics > Fix For: 0.13.0 > > Attachments: HIVE-5325.1.patch.txt, HIVE-5325.2.patch.txt, > HIVE-5325.3.patch.txt, HIVE-5325-java-only.1.patch.txt, > HIVE-5325-java-only.2.patch.txt, HIVE-5325-java-only.3.patch.txt > > > HIVE-5324 adds new interfaces that can be implemented by ORC reader/writer to > provide statistics. Writer provided statistics is used to update > table/partition level statistics in metastore. Reader provided statistics can > be used for reducer estimation, CBO etc. in the absence of metastore > statistics. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5325) Implement statistics providing ORC writer and reader interfaces
[ https://issues.apache.org/jira/browse/HIVE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782577#comment-13782577 ] Ashutosh Chauhan commented on HIVE-5325: +1 It will pick the latest patch which conforms to naming convention. > Implement statistics providing ORC writer and reader interfaces > --- > > Key: HIVE-5325 > URL: https://issues.apache.org/jira/browse/HIVE-5325 > Project: Hive > Issue Type: New Feature >Affects Versions: 0.13.0 >Reporter: Prasanth J >Assignee: Prasanth J > Labels: orcfile, statistics > Fix For: 0.13.0 > > Attachments: HIVE-5325.1.patch.txt, HIVE-5325.2.patch.txt, > HIVE-5325.3.patch.txt, HIVE-5325-java-only.1.patch.txt, > HIVE-5325-java-only.2.patch.txt, HIVE-5325-java-only.3.patch.txt > > > HIVE-5324 adds new interfaces that can be implemented by ORC reader/writer to > provide statistics. Writer provided statistics is used to update > table/partition level statistics in metastore. Reader provided statistics can > be used for reducer estimation, CBO etc. in the absence of metastore > statistics. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5325) Implement statistics providing ORC writer and reader interfaces
[ https://issues.apache.org/jira/browse/HIVE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782558#comment-13782558 ] Prasanth J commented on HIVE-5325: -- [~ashutoshc] which among the two patch will HiveQA pick? I have java-only patch and protobuf generated code patch. > Implement statistics providing ORC writer and reader interfaces > --- > > Key: HIVE-5325 > URL: https://issues.apache.org/jira/browse/HIVE-5325 > Project: Hive > Issue Type: New Feature >Affects Versions: 0.13.0 >Reporter: Prasanth J >Assignee: Prasanth J > Labels: orcfile, statistics > Fix For: 0.13.0 > > Attachments: HIVE-5325.1.patch.txt, HIVE-5325.2.patch.txt, > HIVE-5325.3.patch.txt, HIVE-5325-java-only.1.patch.txt, > HIVE-5325-java-only.2.patch.txt, HIVE-5325-java-only.3.patch.txt > > > HIVE-5324 adds new interfaces that can be implemented by ORC reader/writer to > provide statistics. Writer provided statistics is used to update > table/partition level statistics in metastore. Reader provided statistics can > be used for reducer estimation, CBO etc. in the absence of metastore > statistics. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5325) Implement statistics providing ORC writer and reader interfaces
[ https://issues.apache.org/jira/browse/HIVE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13781975#comment-13781975 ] Ashutosh Chauhan commented on HIVE-5325: Left some comments on RB. > Implement statistics providing ORC writer and reader interfaces > --- > > Key: HIVE-5325 > URL: https://issues.apache.org/jira/browse/HIVE-5325 > Project: Hive > Issue Type: New Feature >Affects Versions: 0.13.0 >Reporter: Prasanth J >Assignee: Prasanth J > Labels: orcfile, statistics > Fix For: 0.13.0 > > Attachments: HIVE-5325.1.patch.txt, HIVE-5325.2.patch.txt, > HIVE-5325-java-only.1.patch.txt, HIVE-5325-java-only.2.patch.txt > > > HIVE-5324 adds new interfaces that can be implemented by ORC reader/writer to > provide statistics. Writer provided statistics is used to update > table/partition level statistics in metastore. Reader provided statistics can > be used for reducer estimation, CBO etc. in the absence of metastore > statistics. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5325) Implement statistics providing ORC writer and reader interfaces
[ https://issues.apache.org/jira/browse/HIVE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776836#comment-13776836 ] Prasanth J commented on HIVE-5325: -- [~ashutoshc] refreshed the patch. Updated RB as well. Its now ready for review :) > Implement statistics providing ORC writer and reader interfaces > --- > > Key: HIVE-5325 > URL: https://issues.apache.org/jira/browse/HIVE-5325 > Project: Hive > Issue Type: New Feature >Affects Versions: 0.13.0 >Reporter: Prasanth J >Assignee: Prasanth J > Labels: orcfile, statistics > Fix For: 0.13.0 > > Attachments: HIVE-5325.1.patch.txt, HIVE-5325.2.patch.txt, > HIVE-5325-java-only.1.patch.txt, HIVE-5325-java-only.2.patch.txt > > > HIVE-5324 adds new interfaces that can be implemented by ORC reader/writer to > provide statistics. Writer provided statistics is used to update > table/partition level statistics in metastore. Reader provided statistics can > be used for reducer estimation, CBO etc. in the absence of metastore > statistics. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5325) Implement statistics providing ORC writer and reader interfaces
[ https://issues.apache.org/jira/browse/HIVE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13772551#comment-13772551 ] Prasanth J commented on HIVE-5325: -- The attached patch is generated on top of HIVE-5324. Here is RB link https://reviews.apache.org/r/14243/ > Implement statistics providing ORC writer and reader interfaces > --- > > Key: HIVE-5325 > URL: https://issues.apache.org/jira/browse/HIVE-5325 > Project: Hive > Issue Type: New Feature >Affects Versions: 0.13.0 >Reporter: Prasanth J >Assignee: Prasanth J > Labels: orcfile, statistics > Fix For: 0.13.0 > > Attachments: HIVE-5325.1.patch.txt, HIVE-5325-java-only.1.patch.txt > > > HIVE-5324 adds new interfaces that can be implemented by ORC reader/writer to > provide statistics. Writer provided statistics is used to update > table/partition level statistics in metastore. Reader provided statistics can > be used for reducer estimation, CBO etc. in the absence of metastore > statistics. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira