[jira] [Commented] (HIVE-11160) Auto-gather column stats
[
https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284189#comment-16284189
]
Hive QA commented on HIVE-11160:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12799200/HIVE-11160.09.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8148/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8148/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8148/
Messages:
{noformat}
This message was trimmed, see log for full details
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-8148/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-12-08 20:52:55.963
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 36f0d89 HIVE-17333 Oracle does not allow change from VARCHAR2 to
CLOB for upgrade (Naveen Gangam, reviewed by Aihua Xu)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 36f0d89 HIVE-17333 Oracle does not allow change from VARCHAR2 to
CLOB for upgrade (Naveen Gangam, reviewed by Aihua Xu)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-12-08 20:53:00.664
+ rm -rf ../yetus
+ mkdir ../yetus
+ cp -R . ../yetus
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-8148/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: No such
file or directory
error: a/data/conf/hive-site.xml: No such file or directory
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:
No such file or directory
error: a/itests/src/test/resources/testconfiguration.properties: No such file
or directory
error: a/metastore/if/hive_metastore.thrift: No such file or directory
error: a/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp: No such file
or directory
error: a/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h: No such file
or directory
error:
a/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp: No
such file or directory
error: a/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp: No such
file or directory
error: a/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h: No such file
or directory
error:
a/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SetPartitionsStatsRequest.java:
No such file or directory
error:
a/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java:
No such file or directory
error: a/metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php: No
such file or directory
error: a/metastore/src/gen/thrift/gen-php/metastore/Types.php: No such file or
directory
error:
a/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote: No
such file or directory
error: a/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py:
No such file or directory
error: a/metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py: No such file
or directory
error: a/metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb: No such file
or directory
error: a/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb: No such file
or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java: No
such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
No such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java: No
such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java: No
such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/StatsCache.java: No
such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BinaryColumnStatsAggregator.java:
No such file or directory
error:
[jira] [Commented] (HIVE-11160) Auto-gather column stats
[
https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260912#comment-16260912
]
Hive QA commented on HIVE-11160:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12799200/HIVE-11160.09.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7942/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7942/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7942/
Messages:
{noformat}
This message was trimmed, see log for full details
error: a/itests/src/test/resources/testconfiguration.properties: No such file
or directory
error: a/metastore/if/hive_metastore.thrift: No such file or directory
error: a/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp: No such file
or directory
error: a/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h: No such file
or directory
error:
a/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp: No
such file or directory
error: a/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp: No such
file or directory
error: a/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h: No such file
or directory
error:
a/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SetPartitionsStatsRequest.java:
No such file or directory
error:
a/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java:
No such file or directory
error: a/metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php: No
such file or directory
error: a/metastore/src/gen/thrift/gen-php/metastore/Types.php: No such file or
directory
error:
a/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote: No
such file or directory
error: a/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py:
No such file or directory
error: a/metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py: No such file
or directory
error: a/metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb: No such file
or directory
error: a/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb: No such file
or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java: No
such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
No such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java: No
such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java: No
such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/StatsCache.java: No
such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BinaryColumnStatsAggregator.java:
No such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BooleanColumnStatsAggregator.java:
No such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregator.java:
No such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregatorFactory.java:
No such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DecimalColumnStatsAggregator.java:
No such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DoubleColumnStatsAggregator.java:
No such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/IExtrapolatePartStatus.java:
No such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/LongColumnStatsAggregator.java:
No such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/StringColumnStatsAggregator.java:
No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java: No
such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java: No such
file or directory
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java:
No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java: No such file
or directory
error:
a/ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java:
No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java: No
such file or directory
error:
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenMRSkewJoinProcessor.java:
No such file or directory
error:
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java: No
such file or directory
error:
[jira] [Commented] (HIVE-11160) Auto-gather column stats
[
https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145161#comment-16145161
]
Hive QA commented on HIVE-11160:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12799200/HIVE-11160.09.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6579/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6579/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6579/
Messages:
{noformat}
This message was trimmed, see log for full details
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-6579/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-08-29 11:43:09.752
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at dd04a92 HIVE-17375: stddev_samp,var_samp standard compliance
(Zoltan Haindrich, reviewed by Ashutosh Chauhan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at dd04a92 HIVE-17375: stddev_samp,var_samp standard compliance
(Zoltan Haindrich, reviewed by Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-08-29 11:43:14.700
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: No such
file or directory
error: a/data/conf/hive-site.xml: No such file or directory
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:
No such file or directory
error: a/itests/src/test/resources/testconfiguration.properties: No such file
or directory
error: a/metastore/if/hive_metastore.thrift: No such file or directory
error: a/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp: No such file
or directory
error: a/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h: No such file
or directory
error:
a/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp: No
such file or directory
error: a/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp: No such
file or directory
error: a/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h: No such file
or directory
error:
a/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SetPartitionsStatsRequest.java:
No such file or directory
error:
a/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java:
No such file or directory
error: a/metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php: No
such file or directory
error: a/metastore/src/gen/thrift/gen-php/metastore/Types.php: No such file or
directory
error:
a/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote: No
such file or directory
error: a/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py:
No such file or directory
error: a/metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py: No such file
or directory
error: a/metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb: No such file
or directory
error: a/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb: No such file
or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java: No
such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
No such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java: No
such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java: No
such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/StatsCache.java: No
such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BinaryColumnStatsAggregator.java:
No such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BooleanColumnStatsAggregator.java:
No such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregator.java:
No such file or directory
error:
[jira] [Commented] (HIVE-11160) Auto-gather column stats
[
https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942008#comment-15942008
]
Hive QA commented on HIVE-11160:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12799200/HIVE-11160.09.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4372/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4372/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4372/
Messages:
{noformat}
This message was trimmed, see log for full details
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 24ad21e HIVE-16298: Add config to specify multi-column joins
have correlated columns (Jason Dere, reviewed by Prasanth Jayachandran)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-03-25 22:28:41.480
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: No such
file or directory
error: a/data/conf/hive-site.xml: No such file or directory
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:
No such file or directory
error: a/itests/src/test/resources/testconfiguration.properties: No such file
or directory
error: a/metastore/if/hive_metastore.thrift: No such file or directory
error: a/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp: No such file
or directory
error: a/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h: No such file
or directory
error:
a/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp: No
such file or directory
error: a/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp: No such
file or directory
error: a/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h: No such file
or directory
error:
a/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SetPartitionsStatsRequest.java:
No such file or directory
error:
a/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java:
No such file or directory
error: a/metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php: No
such file or directory
error: a/metastore/src/gen/thrift/gen-php/metastore/Types.php: No such file or
directory
error:
a/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote: No
such file or directory
error: a/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py:
No such file or directory
error: a/metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py: No such file
or directory
error: a/metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb: No such file
or directory
error: a/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb: No such file
or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java: No
such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
No such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java: No
such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java: No
such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/StatsCache.java: No
such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BinaryColumnStatsAggregator.java:
No such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BooleanColumnStatsAggregator.java:
No such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregator.java:
No such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregatorFactory.java:
No such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DecimalColumnStatsAggregator.java:
No such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DoubleColumnStatsAggregator.java:
No such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/IExtrapolatePartStatus.java:
No such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/LongColumnStatsAggregator.java:
No such file or directory
error:
[jira] [Commented] (HIVE-11160) Auto-gather column stats
[ https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15308413#comment-15308413 ] Jesus Camacho Rodriguez commented on HIVE-11160: Removing 2.1.0 target as issue is not tagged as Critical/Blocker and the RC will be created tomorrow. Please feel free to commit to branch-2.1 anyway and fix for 2.1.0 if this happens before the release. > Auto-gather column stats > > > Key: HIVE-11160 > URL: https://issues.apache.org/jira/browse/HIVE-11160 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11160.01.patch, HIVE-11160.02.patch, > HIVE-11160.03.patch, HIVE-11160.04.patch, HIVE-11160.05.patch, > HIVE-11160.06.patch, HIVE-11160.07.patch, HIVE-11160.08.patch, > HIVE-11160.09.patch > > > Hive will collect table stats when set hive.stats.autogather=true during the > INSERT OVERWRITE command. And then the users need to collect the column stats > themselves using "Analyze" command. In this patch, the column stats will also > be collected automatically. More specifically, INSERT OVERWRITE will > automatically create new column stats. INSERT INTO will automatically merge > new column stats with existing ones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11160) Auto-gather column stats
[ https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15250376#comment-15250376 ] Ashutosh Chauhan commented on HIVE-11160: - I think it will be good to separate metastore thrift changes in a different jira, so that we are committing smaller contained changeset. > Auto-gather column stats > > > Key: HIVE-11160 > URL: https://issues.apache.org/jira/browse/HIVE-11160 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11160.01.patch, HIVE-11160.02.patch, > HIVE-11160.03.patch, HIVE-11160.04.patch, HIVE-11160.05.patch, > HIVE-11160.06.patch, HIVE-11160.07.patch, HIVE-11160.08.patch, > HIVE-11160.09.patch > > > Hive will collect table stats when set hive.stats.autogather=true during the > INSERT OVERWRITE command. And then the users need to collect the column stats > themselves using "Analyze" command. In this patch, the column stats will also > be collected automatically. More specifically, INSERT OVERWRITE will > automatically create new column stats. INSERT INTO will automatically merge > new column stats with existing ones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11160) Auto-gather column stats
[
https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193250#comment-15193250
]
Hive QA commented on HIVE-11160:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12793216/HIVE-11160.07.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7262/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7262/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7262/
Messages:
{noformat}
This message was trimmed, see log for full details
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-metastore ---
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/metastore/target
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/metastore
(includes = [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
hive-metastore ---
[INFO]
[INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @
hive-metastore ---
[INFO] Source directory:
/data/hive-ptest/working/apache-github-source-source/metastore/src/model added.
[INFO] Source directory:
/data/hive-ptest/working/apache-github-source-source/metastore/src/gen/thrift/gen-javabean
added.
[INFO] Source directory:
/data/hive-ptest/working/apache-github-source-source/metastore/src/gen/protobuf/gen-java
added.
[INFO]
[INFO] --- antlr3-maven-plugin:3.4:antlr (default) @ hive-metastore ---
[INFO] ANTLR: Processing source directory
/data/hive-ptest/working/apache-github-source-source/metastore/src/java
ANTLR Parser Generator Version 3.4
org/apache/hadoop/hive/metastore/parser/Filter.g
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-metastore
---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
hive-metastore ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-metastore ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-metastore
---
[INFO] Compiling 301 source files to
/data/hive-ptest/working/apache-github-source-source/metastore/target/classes
[INFO] -
[WARNING] COMPILATION WARNING :
[INFO] -
[WARNING]
/data/hive-ptest/working/apache-github-source-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:
Some input files use or override a deprecated API.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:
Recompile with -Xlint:deprecation for details.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java:
Some input files use unchecked or unsafe operations.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java:
Recompile with -Xlint:unchecked for details.
[INFO] 4 warnings
[INFO] -
[INFO] -
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/data/hive-ptest/working/apache-github-source-source/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java:[112838,12]
an enum switch case label must be the unqualified name of an enumeration
constant
[ERROR]
/data/hive-ptest/working/apache-github-source-source/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java:[112839,16]
cannot find symbol
symbol: method getRequest()
location: class
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore.update_partition_column_statistics_args
[ERROR]
/data/hive-ptest/working/apache-github-source-source/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java:[112852,12]
an enum switch case label must be the unqualified name of an enumeration
constant
[ERROR]
/data/hive-ptest/working/apache-github-source-source/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java:[112853,16]
cannot find symbol
symbol: method isSetRequest()
location: class
[jira] [Commented] (HIVE-11160) Auto-gather column stats
[
https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15182828#comment-15182828
]
Hive QA commented on HIVE-11160:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12791691/HIVE-11160.06.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 556 failed/errored test(s), 9790 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_vectorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_stats_orc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_numbuckets_partitioned_table2_h23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_numbuckets_partitioned_table_h23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_coltype
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_serde2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_analyze_table_null_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_array_map_access_nonconstant
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_delete
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_delete_own_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_index
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_update
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_update_own_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join17
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join26
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_reordering_values
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_charvarchar
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_date
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_nullable_fields
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_timestamp
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_binary_output_format
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative
[jira] [Commented] (HIVE-11160) Auto-gather column stats
[ https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15182396#comment-15182396 ] Pengcheng Xiong commented on HIVE-11160: (1)Sure (2)Yes, that is true. But, in the static partition case, although we only run it for a single static partition, we still need a GBY (see the explain insert overwrite table alter5 partition (dt='a') select key from src) in the new patch. In the dynamic partition case (or a mix of static and dynamic), we group by the partition key and run it for all the partitions because we do not know which partition the new data is going to go to. > Auto-gather column stats > > > Key: HIVE-11160 > URL: https://issues.apache.org/jira/browse/HIVE-11160 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11160.01.patch, HIVE-11160.02.patch, > HIVE-11160.03.patch, HIVE-11160.04.patch, HIVE-11160.05.patch > > > Hive will collect table stats when set hive.stats.autogather=true during the > INSERT OVERWRITE command. And then the users need to collect the column stats > themselves using "Analyze" command. In this patch, the column stats will also > be collected automatically. More specifically, INSERT OVERWRITE will > automatically create new column stats. INSERT INTO will automatically merge > new column stats with existing ones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11160) Auto-gather column stats
[ https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15181323#comment-15181323 ] Ashutosh Chauhan commented on HIVE-11160: - * Can you add explain for insert statements ? * It seems we run analyze table for all partitions, we should run it only for new partitions getting generated in the query ? > Auto-gather column stats > > > Key: HIVE-11160 > URL: https://issues.apache.org/jira/browse/HIVE-11160 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11160.01.patch, HIVE-11160.02.patch, > HIVE-11160.03.patch, HIVE-11160.04.patch, HIVE-11160.05.patch > > > Hive will collect table stats when set hive.stats.autogather=true during the > INSERT OVERWRITE command. And then the users need to collect the column stats > themselves using "Analyze" command. In this patch, the column stats will also > be collected automatically. More specifically, INSERT OVERWRITE will > automatically create new column stats. INSERT INTO will automatically merge > new column stats with existing ones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11160) Auto-gather column stats
[ https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180193#comment-15180193 ] Ashutosh Chauhan commented on HIVE-11160: - It will be good to have test run on this with config value to set. Can you set that to true in data/conf/hive-site.xml ? > Auto-gather column stats > > > Key: HIVE-11160 > URL: https://issues.apache.org/jira/browse/HIVE-11160 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11160.01.patch, HIVE-11160.02.patch, > HIVE-11160.03.patch, HIVE-11160.04.patch > > > Hive will collect table stats when set hive.stats.autogather=true during the > INSERT OVERWRITE command. And then the users need to collect the column stats > themselves using "Analyze" command. In this patch, the column stats will also > be collected automatically. More specifically, INSERT OVERWRITE will > automatically create new column stats. INSERT INTO will automatically merge > new column stats with existing ones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11160) Auto-gather column stats
[
https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149587#comment-15149587
]
Hive QA commented on HIVE-11160:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12788058/HIVE-11160.04.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 9794 tests executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-unionDistinct_1.q-insert_update_delete.q-selectDistinctStar.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.hbase.TestHBaseAggregateStatsCacheWithBitVector.allPartitions
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7004/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7004/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7004/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12788058 - PreCommit-HIVE-TRUNK-Build
> Auto-gather column stats
>
>
> Key: HIVE-11160
> URL: https://issues.apache.org/jira/browse/HIVE-11160
> Project: Hive
> Issue Type: New Feature
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-11160.01.patch, HIVE-11160.02.patch,
> HIVE-11160.03.patch, HIVE-11160.04.patch
>
>
> Hive will collect table stats when set hive.stats.autogather=true during the
> INSERT OVERWRITE command. And then the users need to collect the column stats
> themselves using "Analyze" command. In this patch, the column stats will also
> be collected automatically. More specifically, INSERT OVERWRITE will
> automatically create new column stats. INSERT INTO will automatically merge
> new column stats with existing ones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11160) Auto-gather column stats
[
https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648420#comment-14648420
]
Hive QA commented on HIVE-11160:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12748041/HIVE-11160.03.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 9277 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_partitioned
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Json
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4765/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4765/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4765/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12748041 - PreCommit-HIVE-TRUNK-Build
Auto-gather column stats
Key: HIVE-11160
URL: https://issues.apache.org/jira/browse/HIVE-11160
Project: Hive
Issue Type: New Feature
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong
Attachments: HIVE-11160.01.patch, HIVE-11160.02.patch,
HIVE-11160.03.patch
Hive will collect table stats when set hive.stats.autogather=true during the
INSERT OVERWRITE command. And then the users need to collect the column stats
themselves using Analyze command. In this patch, the column stats will also
be collected automatically.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11160) Auto-gather column stats
[
https://issues.apache.org/jira/browse/HIVE-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622508#comment-14622508
]
Hive QA commented on HIVE-11160:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12744642/HIVE-11160.02.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 9151 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchEmptyCommit
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4571/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4571/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4571/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12744642 - PreCommit-HIVE-TRUNK-Build
Auto-gather column stats
Key: HIVE-11160
URL: https://issues.apache.org/jira/browse/HIVE-11160
Project: Hive
Issue Type: New Feature
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong
Attachments: HIVE-11160.01.patch, HIVE-11160.02.patch
Hive will collect table stats when set hive.stats.autogather=true during the
INSERT OVERWRITE command. And then the users need to collect the column stats
themselves using Analyze command. In this patch, the column stats will also
be collected automatically.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
