[jira] [Commented] (HIVE-6639) Vectorization: Partition column names are not picked up.
[
https://issues.apache.org/jira/browse/HIVE-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13939613#comment-13939613
]
Harish Butani commented on HIVE-6639:
-
+1 for 0.13
> Vectorization: Partition column names are not picked up.
>
>
> Key: HIVE-6639
> URL: https://issues.apache.org/jira/browse/HIVE-6639
> Project: Hive
> Issue Type: Bug
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
> Attachments: HIVE-6639.2.patch, HIVE-6639.3.patch, HIVE-6639.4.patch,
> HIVE-6639.5.patch, HIVE-6639.5.patch, HIVE-6639.6.patch
>
>
> The vectorized plan generation finds the list of partitioning columns from
> pruned-partition-list using table scan operator. In some cases the list is
> coming as null. TPCDS query 27 can reproduce this issue if the store_sales
> table is partitioned on ss_store_sk. The exception stacktrace is :
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getInputColumnIndex(VectorizationContext.java:166)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getColumnVectorExpression(VectorizationContext.java:240)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpression(VectorizationContext.java:287)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpressions(VectorizationContext.java:267)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpressions(VectorizationContext.java:255)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.(VectorMapJoinOperator.java:116)
> ... 42 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6639) Vectorization: Partition column names are not picked up.
[
https://issues.apache.org/jira/browse/HIVE-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13939608#comment-13939608
]
Jitendra Nath Pandey commented on HIVE-6639:
I have committed this to trunk.
[~rhbutani] This bug affects hive-0.13 and fails queries having partitioned
columns but no filters. This should be fixed in branch-0.13 as well.
> Vectorization: Partition column names are not picked up.
>
>
> Key: HIVE-6639
> URL: https://issues.apache.org/jira/browse/HIVE-6639
> Project: Hive
> Issue Type: Bug
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
> Attachments: HIVE-6639.2.patch, HIVE-6639.3.patch, HIVE-6639.4.patch,
> HIVE-6639.5.patch, HIVE-6639.5.patch, HIVE-6639.6.patch
>
>
> The vectorized plan generation finds the list of partitioning columns from
> pruned-partition-list using table scan operator. In some cases the list is
> coming as null. TPCDS query 27 can reproduce this issue if the store_sales
> table is partitioned on ss_store_sk. The exception stacktrace is :
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getInputColumnIndex(VectorizationContext.java:166)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getColumnVectorExpression(VectorizationContext.java:240)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpression(VectorizationContext.java:287)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpressions(VectorizationContext.java:267)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpressions(VectorizationContext.java:255)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.(VectorMapJoinOperator.java:116)
> ... 42 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6639) Vectorization: Partition column names are not picked up.
[
https://issues.apache.org/jira/browse/HIVE-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13938195#comment-13938195
]
Jitendra Nath Pandey commented on HIVE-6639:
Ran tests locally. Only failures were show_create_table_serde.q and
metadata_only_queries_with_filters.q which are unrelated to this patch.
> Vectorization: Partition column names are not picked up.
>
>
> Key: HIVE-6639
> URL: https://issues.apache.org/jira/browse/HIVE-6639
> Project: Hive
> Issue Type: Bug
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
> Attachments: HIVE-6639.2.patch, HIVE-6639.3.patch, HIVE-6639.4.patch,
> HIVE-6639.5.patch, HIVE-6639.5.patch, HIVE-6639.6.patch
>
>
> The vectorized plan generation finds the list of partitioning columns from
> pruned-partition-list using table scan operator. In some cases the list is
> coming as null. TPCDS query 27 can reproduce this issue if the store_sales
> table is partitioned on ss_store_sk. The exception stacktrace is :
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getInputColumnIndex(VectorizationContext.java:166)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getColumnVectorExpression(VectorizationContext.java:240)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpression(VectorizationContext.java:287)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpressions(VectorizationContext.java:267)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpressions(VectorizationContext.java:255)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.(VectorMapJoinOperator.java:116)
> ... 42 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6639) Vectorization: Partition column names are not picked up.
[
https://issues.apache.org/jira/browse/HIVE-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13938120#comment-13938120
]
Vikram Dixit K commented on HIVE-6639:
--
LGTM +1.
> Vectorization: Partition column names are not picked up.
>
>
> Key: HIVE-6639
> URL: https://issues.apache.org/jira/browse/HIVE-6639
> Project: Hive
> Issue Type: Bug
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
> Attachments: HIVE-6639.2.patch, HIVE-6639.3.patch, HIVE-6639.4.patch,
> HIVE-6639.5.patch, HIVE-6639.5.patch, HIVE-6639.6.patch
>
>
> The vectorized plan generation finds the list of partitioning columns from
> pruned-partition-list using table scan operator. In some cases the list is
> coming as null. TPCDS query 27 can reproduce this issue if the store_sales
> table is partitioned on ss_store_sk. The exception stacktrace is :
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getInputColumnIndex(VectorizationContext.java:166)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getColumnVectorExpression(VectorizationContext.java:240)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpression(VectorizationContext.java:287)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpressions(VectorizationContext.java:267)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpressions(VectorizationContext.java:255)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.(VectorMapJoinOperator.java:116)
> ... 42 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6639) Vectorization: Partition column names are not picked up.
[
https://issues.apache.org/jira/browse/HIVE-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13936272#comment-13936272
]
Hive QA commented on HIVE-6639:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12634926/HIVE-6639.6.patch
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1837/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1837/console
Messages:
{noformat}
This message was trimmed, see log for full details
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-svn-trunk-source/hwi/src/test/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-hwi ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/hwi/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/hwi/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/hwi/target/tmp/conf
[copy] Copying 5 files to
/data/hive-ptest/working/apache-svn-trunk-source/hwi/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hive-hwi ---
[INFO] Compiling 2 source files to
/data/hive-ptest/working/apache-svn-trunk-source/hwi/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-hwi ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-hwi ---
[INFO] Building jar:
/data/hive-ptest/working/apache-svn-trunk-source/hwi/target/hive-hwi-0.14.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
hive-hwi ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-hwi ---
[INFO] Installing
/data/hive-ptest/working/apache-svn-trunk-source/hwi/target/hive-hwi-0.14.0-SNAPSHOT.jar
to
/data/hive-ptest/working/maven/org/apache/hive/hive-hwi/0.14.0-SNAPSHOT/hive-hwi-0.14.0-SNAPSHOT.jar
[INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/hwi/pom.xml
to
/data/hive-ptest/working/maven/org/apache/hive/hive-hwi/0.14.0-SNAPSHOT/hive-hwi-0.14.0-SNAPSHOT.pom
[INFO]
[INFO]
[INFO] Building Hive ODBC 0.14.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-odbc ---
[INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/odbc (includes
= [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-odbc ---
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-odbc ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-odbc ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/odbc/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/odbc/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/odbc/target/tmp/conf
[copy] Copying 5 files to
/data/hive-ptest/working/apache-svn-trunk-source/odbc/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
hive-odbc ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-odbc ---
[INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/odbc/pom.xml
to
/data/hive-ptest/working/maven/org/apache/hive/hive-odbc/0.14.0-SNAPSHOT/hive-odbc-0.14.0-SNAPSHOT.pom
[INFO]
[INFO]
[INFO] Building Hive Shims Aggregator 0.14.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-shims-aggregator
---
[INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/shims
(includes = [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @
hive-shims-aggregator ---
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @
hive-shims-aggregator ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @
hive-shims-aggregator ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-pt
[jira] [Commented] (HIVE-6639) Vectorization: Partition column names are not picked up.
[
https://issues.apache.org/jira/browse/HIVE-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13936042#comment-13936042
]
Hive QA commented on HIVE-6639:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12634679/HIVE-6639.5.patch
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1790/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1790/console
Messages:
{noformat}
This message was trimmed, see log for full details
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-svn-trunk-source/hwi/src/test/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-hwi ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/hwi/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/hwi/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/hwi/target/tmp/conf
[copy] Copying 5 files to
/data/hive-ptest/working/apache-svn-trunk-source/hwi/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hive-hwi ---
[INFO] Compiling 2 source files to
/data/hive-ptest/working/apache-svn-trunk-source/hwi/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-hwi ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-hwi ---
[INFO] Building jar:
/data/hive-ptest/working/apache-svn-trunk-source/hwi/target/hive-hwi-0.14.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
hive-hwi ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-hwi ---
[INFO] Installing
/data/hive-ptest/working/apache-svn-trunk-source/hwi/target/hive-hwi-0.14.0-SNAPSHOT.jar
to
/data/hive-ptest/working/maven/org/apache/hive/hive-hwi/0.14.0-SNAPSHOT/hive-hwi-0.14.0-SNAPSHOT.jar
[INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/hwi/pom.xml
to
/data/hive-ptest/working/maven/org/apache/hive/hive-hwi/0.14.0-SNAPSHOT/hive-hwi-0.14.0-SNAPSHOT.pom
[INFO]
[INFO]
[INFO] Building Hive ODBC 0.14.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-odbc ---
[INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/odbc (includes
= [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-odbc ---
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-odbc ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-odbc ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/odbc/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/odbc/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/odbc/target/tmp/conf
[copy] Copying 5 files to
/data/hive-ptest/working/apache-svn-trunk-source/odbc/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
hive-odbc ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-odbc ---
[INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/odbc/pom.xml
to
/data/hive-ptest/working/maven/org/apache/hive/hive-odbc/0.14.0-SNAPSHOT/hive-odbc-0.14.0-SNAPSHOT.pom
[INFO]
[INFO]
[INFO] Building Hive Shims Aggregator 0.14.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-shims-aggregator
---
[INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/shims
(includes = [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @
hive-shims-aggregator ---
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @
hive-shims-aggregator ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @
hive-shims-aggregator ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-pt
[jira] [Commented] (HIVE-6639) Vectorization: Partition column names are not picked up.
[ https://issues.apache.org/jira/browse/HIVE-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934805#comment-13934805 ] Jitendra Nath Pandey commented on HIVE-6639: Review board: https://reviews.apache.org/r/19219/ > Vectorization: Partition column names are not picked up. > > > Key: HIVE-6639 > URL: https://issues.apache.org/jira/browse/HIVE-6639 > Project: Hive > Issue Type: Bug >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Attachments: HIVE-6639.2.patch, HIVE-6639.3.patch, HIVE-6639.4.patch, > HIVE-6639.5.patch > > > The vectorized plan generation finds the list of partitioning columns from > pruned-partition-list using table scan operator. In some cases the list is > coming as null. TPCDS query 27 can reproduce this issue if the store_sales > table is partitioned on ss_store_sk. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6639) Vectorization: Partition column names are not picked up.
[ https://issues.apache.org/jira/browse/HIVE-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934296#comment-13934296 ] Jitendra Nath Pandey commented on HIVE-6639: The patch gets all the columns from the row schema eliminating the need to look at pruned partition list. > Vectorization: Partition column names are not picked up. > > > Key: HIVE-6639 > URL: https://issues.apache.org/jira/browse/HIVE-6639 > Project: Hive > Issue Type: Bug >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Attachments: HIVE-6639.2.patch, HIVE-6639.3.patch, HIVE-6639.4.patch > > > The vectorized plan generation finds the list of partitioning columns from > pruned-partition-list using table scan operator. In some cases the list is > coming as null. -- This message was sent by Atlassian JIRA (v6.2#6252)
