[jira] [Commented] (HIVE-13542) Missing stats for tables in TPCDS performance regression suite
[
https://issues.apache.org/jira/browse/HIVE-13542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275237#comment-15275237
]
Hive QA commented on HIVE-13542:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12802469/HIVE-13542.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/204/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/204/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-204/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-204/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 3f3aa2a HIVE-12827: Vectorization:
VectorCopyRow/VectorAssignRow/VectorDeserializeRow assign needs explicit
isNull[offset] modification (errata.txt)
+ git clean -f -d
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at 3f3aa2a HIVE-12827: Vectorization:
VectorCopyRow/VectorAssignRow/VectorDeserializeRow assign needs explicit
isNull[offset] modification (errata.txt)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12802469 - PreCommit-HIVE-MASTER-Build
> Missing stats for tables in TPCDS performance regression suite
> --
>
> Key: HIVE-13542
> URL: https://issues.apache.org/jira/browse/HIVE-13542
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Affects Versions: 2.0.0
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Fix For: 2.1.0
>
> Attachments: HIVE-13542.1.patch, HIVE-13542.2.patch
>
>
> These are the tables whose stats are missing in
> data/files/tpcds-perf/metastore_export/csv/TAB_COL_STATS.txt:
> * catalog_returns
> * catalog_sales
> * inventory
> * store_returns
> * store_sales
> * web_returns
> * web_sales
> Thanks to [~jcamachorodriguez] for discovering this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13542) Missing stats for tables in TPCDS performance regression suite
[ https://issues.apache.org/jira/browse/HIVE-13542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15272902#comment-15272902 ] Ashutosh Chauhan commented on HIVE-13542: - +1 pending tests > Missing stats for tables in TPCDS performance regression suite > -- > > Key: HIVE-13542 > URL: https://issues.apache.org/jira/browse/HIVE-13542 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13542.1.patch, HIVE-13542.2.patch > > > These are the tables whose stats are missing in > data/files/tpcds-perf/metastore_export/csv/TAB_COL_STATS.txt: > * catalog_returns > * catalog_sales > * inventory > * store_returns > * store_sales > * web_returns > * web_sales > Thanks to [~jcamachorodriguez] for discovering this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13542) Missing stats for tables in TPCDS performance regression suite
[ https://issues.apache.org/jira/browse/HIVE-13542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270504#comment-15270504 ] Ashutosh Chauhan commented on HIVE-13542: - I am fine either ways. [~hsubramaniyan] ? > Missing stats for tables in TPCDS performance regression suite > -- > > Key: HIVE-13542 > URL: https://issues.apache.org/jira/browse/HIVE-13542 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13542.1.patch > > > These are the tables whose stats are missing in > data/files/tpcds-perf/metastore_export/csv/TAB_COL_STATS.txt: > * catalog_returns > * catalog_sales > * inventory > * store_returns > * store_sales > * web_returns > * web_sales > Thanks to [~jcamachorodriguez] for discovering this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13542) Missing stats for tables in TPCDS performance regression suite
[ https://issues.apache.org/jira/browse/HIVE-13542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270429#comment-15270429 ] Jesus Camacho Rodriguez commented on HIVE-13542: [~hsubramaniyan], [~ashutoshc], if this and HIVE-13061 are taking a bit of time, is it possible to set hive.metastore.fastpath to false and add the missing stats so TestPerfCliDriver works properly? That would unblock HIVE-13269 while TestPerfCliDriver is being migrated, right? > Missing stats for tables in TPCDS performance regression suite > -- > > Key: HIVE-13542 > URL: https://issues.apache.org/jira/browse/HIVE-13542 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13542.1.patch > > > These are the tables whose stats are missing in > data/files/tpcds-perf/metastore_export/csv/TAB_COL_STATS.txt: > * catalog_returns > * catalog_sales > * inventory > * store_returns > * store_sales > * web_returns > * web_sales > Thanks to [~jcamachorodriguez] for discovering this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13542) Missing stats for tables in TPCDS performance regression suite
[ https://issues.apache.org/jira/browse/HIVE-13542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270256#comment-15270256 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-13542: -- This jira is blocked by the fact that we are do the following: 1. Move the TestPerfCliDriver to TestMiniTezCliDriver 2. Move from derby to hbase metastore (as part of point 1). I will create a new jira and link it to this one when I start working on it. The fix once available for the above points, will invalidate this one. Thanks Hari > Missing stats for tables in TPCDS performance regression suite > -- > > Key: HIVE-13542 > URL: https://issues.apache.org/jira/browse/HIVE-13542 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13542.1.patch > > > These are the tables whose stats are missing in > data/files/tpcds-perf/metastore_export/csv/TAB_COL_STATS.txt: > * catalog_returns > * catalog_sales > * inventory > * store_returns > * store_sales > * web_returns > * web_sales > Thanks to [~jcamachorodriguez] for discovering this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13542) Missing stats for tables in TPCDS performance regression suite
[ https://issues.apache.org/jira/browse/HIVE-13542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15248021#comment-15248021 ] Jesus Camacho Rodriguez commented on HIVE-13542: [~hsubramaniyan], still same problem. Can be reproduced running e.g. query7.q. > Missing stats for tables in TPCDS performance regression suite > -- > > Key: HIVE-13542 > URL: https://issues.apache.org/jira/browse/HIVE-13542 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13542.1.patch > > > These are the tables whose stats are missing in > data/files/tpcds-perf/metastore_export/csv/TAB_COL_STATS.txt: > * catalog_returns > * catalog_sales > * inventory > * store_returns > * store_sales > * web_returns > * web_sales > Thanks to [~jcamachorodriguez] for discovering this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13542) Missing stats for tables in TPCDS performance regression suite
[
https://issues.apache.org/jira/browse/HIVE-13542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15247977#comment-15247977
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-13542:
--
[~jcamachorodriguez] Thanks.
It seems there is a corrupt entry in TAB_COL_STATS.txt, can you please replace
the line 101 (one with cd_demo_sk) to
{code}
default,customer_demographics,cd_demo_sk,int,1,1920800,1835839,0,1434571729,6296,_customer_demographics_
{code}
and see if it resolves the issue.
Thanks
Hari
> Missing stats for tables in TPCDS performance regression suite
> --
>
> Key: HIVE-13542
> URL: https://issues.apache.org/jira/browse/HIVE-13542
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13542.1.patch
>
>
> These are the tables whose stats are missing in
> data/files/tpcds-perf/metastore_export/csv/TAB_COL_STATS.txt:
> * catalog_returns
> * catalog_sales
> * inventory
> * store_returns
> * store_sales
> * web_returns
> * web_sales
> Thanks to [~jcamachorodriguez] for discovering this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13542) Missing stats for tables in TPCDS performance regression suite
[
https://issues.apache.org/jira/browse/HIVE-13542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15247892#comment-15247892
]
Jesus Camacho Rodriguez commented on HIVE-13542:
Thanks [~hsubramaniyan].
I have been running some tests using your patch e.g. query7.q. Examining the
logs I see the following message that seems to indicate we still have some kind
of problem with the stats:
{noformat}
Caused by: org.apache.hadoop.hive.metastore.DeadlineException: The threadlocal
Deadline is null, please register it first.
at
org.apache.hadoop.hive.metastore.Deadline.checkTimeout(Deadline.java:149)
~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore$7.getJdoResult(ObjectStore.java:6686)
~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore$7.getJdoResult(ObjectStore.java:6664)
~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2550)
~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:6663)
~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatistics(ObjectStore.java:6657)
~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_statistics_req(HiveMetaStore.java:4327)
~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableColumnStatistics(HiveMetaStoreClient.java:1570)
~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTableColumnStatistics(SessionHiveMetaStoreClient.java:347)
~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.metadata.Hive.getTableColumnStatistics(Hive.java:3317)
~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
... 178 more
2016-04-19T07:23:58,718 ERROR [ee2214e0-ff62-4617-a51c-45dfba3ae1c0 main[]]:
calcite.RelOptHiveTable (RelOptHiveTable.java:updateColStats(395)) - No Stats
for default@customer_demographics, Columns: cd_demo_sk
2016-04-19T07:23:58,719 WARN [ee2214e0-ff62-4617-a51c-45dfba3ae1c0 main[]]:
parse.CalcitePlanner (CalcitePlanner.java:apply(993)) - Missing column stats
(see previous messages), skipping join reordering in CBO
{noformat}
In particular, that column is present in
{{data/files/tpcds-perf/metastore_export/csv/TAB_COL_STATS.txt}}.
I could avoid the DeadlineException by setting {{hive.metastore.fastpath}} to
false in {{data/conf/perf-reg/hive-site.xml}}, thus avoiding bypassing the raw
store proxy as we are using the ObjectStore in the PerfCliDriver. However, even
after doing that, I still see the missing stats message.
> Missing stats for tables in TPCDS performance regression suite
> --
>
> Key: HIVE-13542
> URL: https://issues.apache.org/jira/browse/HIVE-13542
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13542.1.patch
>
>
> These are the tables whose stats are missing in
> data/files/tpcds-perf/metastore_export/csv/TAB_COL_STATS.txt:
> * catalog_returns
> * catalog_sales
> * inventory
> * store_returns
> * store_sales
> * web_returns
> * web_sales
> Thanks to [~jcamachorodriguez] for discovering this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
