[ https://issues.apache.org/jira/browse/HIVE-7723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14559849#comment-14559849 ]
Hive QA commented on HIVE-7723: ------------------------------- {color:red}Overall{color}: -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12735389/HIVE-7723.11.patch Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4046/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4046/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4046/ Messages: {noformat} **** This message was trimmed, see log for full details **** [WARNING] /data/hive-ptest/working/apache-github-source-source/spark-client/src/main/java/org/apache/hive/spark/client/rpc/RpcDispatcher.java: Recompile with -Xlint:unchecked for details. [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ spark-client --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 1 resource [INFO] Copying 3 resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ spark-client --- [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/spark-client/target/warehouse [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf [copy] Copying 11 files to /data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ spark-client --- [INFO] Compiling 5 source files to /data/hive-ptest/working/apache-github-source-source/spark-client/target/test-classes [INFO] [INFO] --- maven-dependency-plugin:2.8:copy (copy-guava-14) @ spark-client --- [INFO] Configured Artifact: com.google.guava:guava:14.0.1:jar [INFO] Copying guava-14.0.1.jar to /data/hive-ptest/working/apache-github-source-source/spark-client/target/dependency/guava-14.0.1.jar [INFO] [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ spark-client --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ spark-client --- [INFO] Building jar: /data/hive-ptest/working/apache-github-source-source/spark-client/target/spark-client-1.3.0-SNAPSHOT.jar [INFO] [INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ spark-client --- [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ spark-client --- [INFO] Installing /data/hive-ptest/working/apache-github-source-source/spark-client/target/spark-client-1.3.0-SNAPSHOT.jar to /home/hiveptest/.m2/repository/org/apache/hive/spark-client/1.3.0-SNAPSHOT/spark-client-1.3.0-SNAPSHOT.jar [INFO] Installing /data/hive-ptest/working/apache-github-source-source/spark-client/pom.xml to /home/hiveptest/.m2/repository/org/apache/hive/spark-client/1.3.0-SNAPSHOT/spark-client-1.3.0-SNAPSHOT.pom [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Hive Query Language 1.3.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-exec --- [INFO] Deleting /data/hive-ptest/working/apache-github-source-source/ql/target [INFO] Deleting /data/hive-ptest/working/apache-github-source-source/ql (includes = [datanucleus.log, derby.log], excludes = []) [INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @ hive-exec --- [INFO] [INFO] --- maven-antrun-plugin:1.7:run (generate-sources) @ hive-exec --- [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/ql/target/generated-test-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen Generating vector expression code Generating vector expression test code [INFO] Executed tasks [INFO] [INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @ hive-exec --- [INFO] Source directory: /data/hive-ptest/working/apache-github-source-source/ql/src/gen/protobuf/gen-java added. [INFO] Source directory: /data/hive-ptest/working/apache-github-source-source/ql/src/gen/thrift/gen-javabean added. [INFO] Source directory: /data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java added. [INFO] [INFO] --- antlr3-maven-plugin:3.4:antlr (default) @ hive-exec --- [INFO] ANTLR: Processing source directory /data/hive-ptest/working/apache-github-source-source/ql/src/java ANTLR Parser Generator Version 3.4 org/apache/hadoop/hive/ql/parse/HiveLexer.g org/apache/hadoop/hive/ql/parse/HiveParser.g warning(200): IdentifiersParser.g:455:5: Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_ORDER KW_BY" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:455:5: Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_GROUP KW_BY" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:455:5: Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_CLUSTER KW_BY" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:455:5: Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_UNION KW_MAP" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:455:5: Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_UNION KW_SELECT" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:455:5: Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_MAP LPAREN" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:455:5: Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_UNION KW_REDUCE" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:455:5: Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_LATERAL KW_VIEW" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:455:5: Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_SORT KW_BY" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:455:5: Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_INSERT KW_INTO" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:455:5: Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_UNION KW_ALL" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:455:5: Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_UNION KW_FROM" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:455:5: Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_INSERT KW_OVERWRITE" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:455:5: Decision can match input such as "{KW_REGEXP, KW_RLIKE} KW_DISTRIBUTE KW_BY" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-exec --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hive-exec --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 2 resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-exec --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ hive-exec --- [INFO] Compiling 2385 source files to /data/hive-ptest/working/apache-github-source-source/ql/target/classes [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /data/hive-ptest/working/apache-github-source-source/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:[578,13] error: cannot find symbol [INFO] 1 error [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Hive .............................................. SUCCESS [18.020s] [INFO] Hive Shims Common ................................. SUCCESS [13.792s] [INFO] Hive Shims 0.20S .................................. SUCCESS [3.313s] [INFO] Hive Shims 0.23 ................................... SUCCESS [12.029s] [INFO] Hive Shims Scheduler .............................. SUCCESS [1.974s] [INFO] Hive Shims ........................................ SUCCESS [2.660s] [INFO] Hive Common ....................................... SUCCESS [14.956s] [INFO] Hive Serde ........................................ SUCCESS [16.633s] [INFO] Hive Metastore .................................... SUCCESS [36.064s] [INFO] Hive Ant Utilities ................................ SUCCESS [2.404s] [INFO] Spark Remote Client ............................... SUCCESS [19.045s] [INFO] Hive Query Language ............................... FAILURE [1:37.997s] [INFO] Hive Service ...................................... SKIPPED [INFO] Hive Accumulo Handler ............................. SKIPPED [INFO] Hive JDBC ......................................... SKIPPED [INFO] Hive Beeline ...................................... SKIPPED [INFO] Hive CLI .......................................... SKIPPED [INFO] Hive Contrib ...................................... SKIPPED [INFO] Hive HBase Handler ................................ SKIPPED [INFO] Hive HCatalog ..................................... SKIPPED [INFO] Hive HCatalog Core ................................ SKIPPED [INFO] Hive HCatalog Pig Adapter ......................... SKIPPED [INFO] Hive HCatalog Server Extensions ................... SKIPPED [INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED [INFO] Hive HCatalog Webhcat ............................. SKIPPED [INFO] Hive HCatalog Streaming ........................... SKIPPED [INFO] Hive HWI .......................................... SKIPPED [INFO] Hive ODBC ......................................... SKIPPED [INFO] Hive Shims Aggregator ............................. SKIPPED [INFO] Hive TestUtils .................................... SKIPPED [INFO] Hive Packaging .................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 4:02.443s [INFO] Finished at: Tue May 26 16:52:42 EDT 2015 [INFO] Final Memory: 109M/628M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.3.2:compile (default-compile) on project hive-exec: Compilation failure [ERROR] /data/hive-ptest/working/apache-github-source-source/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:[578,13] error: cannot find symbol [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hive-exec + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12735389 - PreCommit-HIVE-TRUNK-Build > Explain plan for complex query with lots of partitions is slow due to > in-efficient collection used to find a matching ReadEntity > -------------------------------------------------------------------------------------------------------------------------------- > > Key: HIVE-7723 > URL: https://issues.apache.org/jira/browse/HIVE-7723 > Project: Hive > Issue Type: Bug > Components: CLI, Physical Optimizer > Affects Versions: 0.13.1 > Reporter: Mostafa Mokhtar > Assignee: Mostafa Mokhtar > Attachments: HIVE-7723.1.patch, HIVE-7723.10.patch, > HIVE-7723.11.patch, HIVE-7723.2.patch, HIVE-7723.3.patch, HIVE-7723.4.patch, > HIVE-7723.5.patch, HIVE-7723.6.patch, HIVE-7723.7.patch, HIVE-7723.8.patch, > HIVE-7723.9.patch > > > Explain on TPC-DS query 64 took 11 seconds, when the CLI was profiled it > showed that ReadEntity.equals is taking ~40% of the CPU. > ReadEntity.equals is called from the snippet below. > Again and again the set is iterated over to get the actual match, a HashMap > is a better option for this case as Set doesn't have a Get method. > Also for ReadEntity equals is case-insensitive while hash is , which is an > undesired behavior. > {code} > public static ReadEntity addInput(Set<ReadEntity> inputs, ReadEntity > newInput) { > // If the input is already present, make sure the new parent is added to > the input. > if (inputs.contains(newInput)) { > for (ReadEntity input : inputs) { > if (input.equals(newInput)) { > if ((newInput.getParents() != null) && > (!newInput.getParents().isEmpty())) { > input.getParents().addAll(newInput.getParents()); > input.setDirect(input.isDirect() || newInput.isDirect()); > } > return input; > } > } > assert false; > } else { > inputs.add(newInput); > return newInput; > } > // make compile happy > return null; > } > {code} > This is the query used : > {code} > select cs1.product_name ,cs1.store_name ,cs1.store_zip ,cs1.b_street_number > ,cs1.b_streen_name ,cs1.b_city > ,cs1.b_zip ,cs1.c_street_number ,cs1.c_street_name ,cs1.c_city > ,cs1.c_zip ,cs1.syear ,cs1.cnt > ,cs1.s1 ,cs1.s2 ,cs1.s3 > ,cs2.s1 ,cs2.s2 ,cs2.s3 ,cs2.syear ,cs2.cnt > from > (select i_product_name as product_name ,i_item_sk as item_sk ,s_store_name as > store_name > ,s_zip as store_zip ,ad1.ca_street_number as b_street_number > ,ad1.ca_street_name as b_streen_name > ,ad1.ca_city as b_city ,ad1.ca_zip as b_zip ,ad2.ca_street_number as > c_street_number > ,ad2.ca_street_name as c_street_name ,ad2.ca_city as c_city ,ad2.ca_zip > as c_zip > ,d1.d_year as syear ,d2.d_year as fsyear ,d3.d_year as s2year ,count(*) > as cnt > ,sum(ss_wholesale_cost) as s1 ,sum(ss_list_price) as s2 > ,sum(ss_coupon_amt) as s3 > FROM store_sales > JOIN store_returns ON store_sales.ss_item_sk = > store_returns.sr_item_sk and store_sales.ss_ticket_number = > store_returns.sr_ticket_number > JOIN customer ON store_sales.ss_customer_sk = customer.c_customer_sk > JOIN date_dim d1 ON store_sales.ss_sold_date_sk = d1.d_date_sk > JOIN date_dim d2 ON customer.c_first_sales_date_sk = d2.d_date_sk > JOIN date_dim d3 ON customer.c_first_shipto_date_sk = d3.d_date_sk > JOIN store ON store_sales.ss_store_sk = store.s_store_sk > JOIN customer_demographics cd1 ON store_sales.ss_cdemo_sk= > cd1.cd_demo_sk > JOIN customer_demographics cd2 ON customer.c_current_cdemo_sk = > cd2.cd_demo_sk > JOIN promotion ON store_sales.ss_promo_sk = promotion.p_promo_sk > JOIN household_demographics hd1 ON store_sales.ss_hdemo_sk = > hd1.hd_demo_sk > JOIN household_demographics hd2 ON customer.c_current_hdemo_sk = > hd2.hd_demo_sk > JOIN customer_address ad1 ON store_sales.ss_addr_sk = > ad1.ca_address_sk > JOIN customer_address ad2 ON customer.c_current_addr_sk = > ad2.ca_address_sk > JOIN income_band ib1 ON hd1.hd_income_band_sk = ib1.ib_income_band_sk > JOIN income_band ib2 ON hd2.hd_income_band_sk = ib2.ib_income_band_sk > JOIN item ON store_sales.ss_item_sk = item.i_item_sk > JOIN > (select cs_item_sk > ,sum(cs_ext_list_price) as > sale,sum(cr_refunded_cash+cr_reversed_charge+cr_store_credit) as refund > from catalog_sales JOIN catalog_returns > ON catalog_sales.cs_item_sk = catalog_returns.cr_item_sk > and catalog_sales.cs_order_number = catalog_returns.cr_order_number > group by cs_item_sk > having > sum(cs_ext_list_price)>2*sum(cr_refunded_cash+cr_reversed_charge+cr_store_credit)) > cs_ui > ON store_sales.ss_item_sk = cs_ui.cs_item_sk > WHERE > cd1.cd_marital_status <> cd2.cd_marital_status and > i_color in ('maroon','burnished','dim','steel','navajo','chocolate') > and > i_current_price between 35 and 35 + 10 and > i_current_price between 35 + 1 and 35 + 15 > and ss_sold_date between '2000-01-01' and '2000-12-31' > group by i_product_name ,i_item_sk ,s_store_name ,s_zip ,ad1.ca_street_number > ,ad1.ca_street_name ,ad1.ca_city ,ad1.ca_zip ,ad2.ca_street_number > ,ad2.ca_street_name ,ad2.ca_city ,ad2.ca_zip ,d1.d_year ,d2.d_year > ,d3.d_year > ) cs1 > JOIN > (select i_product_name as product_name ,i_item_sk as item_sk ,s_store_name as > store_name > ,s_zip as store_zip ,ad1.ca_street_number as b_street_number > ,ad1.ca_street_name as b_streen_name > ,ad1.ca_city as b_city ,ad1.ca_zip as b_zip ,ad2.ca_street_number as > c_street_number > ,ad2.ca_street_name as c_street_name ,ad2.ca_city as c_city ,ad2.ca_zip > as c_zip > ,d1.d_year as syear ,d2.d_year as fsyear ,d3.d_year as s2year ,count(*) > as cnt > ,sum(ss_wholesale_cost) as s1 ,sum(ss_list_price) as s2 > ,sum(ss_coupon_amt) as s3 > FROM store_sales > JOIN store_returns ON store_sales.ss_item_sk = > store_returns.sr_item_sk and store_sales.ss_ticket_number = > store_returns.sr_ticket_number > JOIN customer ON store_sales.ss_customer_sk = customer.c_customer_sk > JOIN date_dim d1 ON store_sales.ss_sold_date_sk = d1.d_date_sk > JOIN date_dim d2 ON customer.c_first_sales_date_sk = d2.d_date_sk > JOIN date_dim d3 ON customer.c_first_shipto_date_sk = d3.d_date_sk > JOIN store ON store_sales.ss_store_sk = store.s_store_sk > JOIN customer_demographics cd1 ON store_sales.ss_cdemo_sk= > cd1.cd_demo_sk > JOIN customer_demographics cd2 ON customer.c_current_cdemo_sk = > cd2.cd_demo_sk > JOIN promotion ON store_sales.ss_promo_sk = promotion.p_promo_sk > JOIN household_demographics hd1 ON store_sales.ss_hdemo_sk = > hd1.hd_demo_sk > JOIN household_demographics hd2 ON customer.c_current_hdemo_sk = > hd2.hd_demo_sk > JOIN customer_address ad1 ON store_sales.ss_addr_sk = > ad1.ca_address_sk > JOIN customer_address ad2 ON customer.c_current_addr_sk = > ad2.ca_address_sk > JOIN income_band ib1 ON hd1.hd_income_band_sk = ib1.ib_income_band_sk > JOIN income_band ib2 ON hd2.hd_income_band_sk = ib2.ib_income_band_sk > JOIN item ON store_sales.ss_item_sk = item.i_item_sk > JOIN > (select cs_item_sk > ,sum(cs_ext_list_price) as > sale,sum(cr_refunded_cash+cr_reversed_charge+cr_store_credit) as refund > from catalog_sales JOIN catalog_returns > ON catalog_sales.cs_item_sk = catalog_returns.cr_item_sk > and catalog_sales.cs_order_number = catalog_returns.cr_order_number > group by cs_item_sk > having > sum(cs_ext_list_price)>2*sum(cr_refunded_cash+cr_reversed_charge+cr_store_credit)) > cs_ui > ON store_sales.ss_item_sk = cs_ui.cs_item_sk > WHERE > cd1.cd_marital_status <> cd2.cd_marital_status and > i_color in ('maroon','burnished','dim','steel','navajo','chocolate') > and > i_current_price between 35 and 35 + 10 and > i_current_price between 35 + 1 and 35 + 15 > and ss_sold_date between '2001-01-01' and '2001-12-31' > group by i_product_name ,i_item_sk ,s_store_name ,s_zip ,ad1.ca_street_number > ,ad1.ca_street_name ,ad1.ca_city ,ad1.ca_zip ,ad2.ca_street_number > ,ad2.ca_street_name ,ad2.ca_city ,ad2.ca_zip ,d1.d_year ,d2.d_year > ,d3.d_year > ) cs2 > ON cs1.item_sk=cs2.item_sk > where > cs1.syear = 2000 and > cs2.syear = 2000 + 1 and > cs2.cnt <= cs1.cnt and > cs1.store_name = cs2.store_name and > cs1.store_zip = cs2.store_zip > order by cs1.product_name ,cs1.store_name ,cs2.cnt; > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)