[jira] [Commented] (HIVE-15331) Decimal multiplication with high precision/scale often returns NULL
[
https://issues.apache.org/jira/browse/HIVE-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731436#comment-15731436
]
Hive QA commented on HIVE-15331:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12842239/HIVE-15331.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10782 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=92)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2481/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2481/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2481/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12842239 - PreCommit-HIVE-Build
> Decimal multiplication with high precision/scale often returns NULL
> ---
>
> Key: HIVE-15331
> URL: https://issues.apache.org/jira/browse/HIVE-15331
> Project: Hive
> Issue Type: Bug
> Components: Types
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-15331.1.patch, HIVE-15331.2.patch,
> HIVE-15331.3.patch
>
>
> {noformat}
> create temporary table dec (a decimal(38,18));
> insert into dec values(100.0);
> hive> select a*a from dec;
> OK
> NULL
> Time taken: 0.165 seconds, Fetched: 1 row(s)
> {noformat}
> Looks like the reason is because the result of decimal(38,18) *
> decimal(38,18) only has 2 digits of precision for integers:
> {noformat}
> hive> set hive.explain.user=false;
> hive> explain select a*a from dec;
> OK
> STAGE DEPENDENCIES:
> Stage-0 is a root stage
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> TableScan
> alias: dec
> Select Operator
> expressions: (a * a) (type: decimal(38,36))
> outputColumnNames: _col0
> ListSink
> Time taken: 0.039 seconds, Fetched: 15 row(s)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15383) Add additional info to 'desc function extended' output
[
https://issues.apache.org/jira/browse/HIVE-15383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731323#comment-15731323
]
Hive QA commented on HIVE-15383:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12842229/HIVE-15383.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 222 failed/errored test(s), 10782 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_udf_percentile2]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_udf_percentile]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_udf_max] (batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_func1]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[str_to_map] (batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_collect_set]
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_corr] (batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_covar_pop]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_covar_samp]
(batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_E] (batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_PI] (batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_abs] (batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_acos] (batchId=43)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_add] (batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_add_months]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_aes_decrypt]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_aes_encrypt]
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_array] (batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_array_contains]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_ascii] (batchId=43)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_asin] (batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_atan] (batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_avg] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_between] (batchId=65)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_bigint] (batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_bin] (batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_bitwise_and]
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_bitwise_not]
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_bitwise_or]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_bitwise_shiftleft]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_bitwise_shiftright]
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_bitwise_shiftrightunsigned]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_bitwise_xor]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_boolean] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_bround] (batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_case] (batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_cbrt] (batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_ceil] (batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_ceiling] (batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_chr] (batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_coalesce]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_concat] (batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_concat_ws]
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_conv] (batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_cos] (batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_count] (batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_crc32] (batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_current_user]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_date_add]
(batchId=43)
org.apache.hadoop.hive.cl
[jira] [Commented] (HIVE-15333) Add a FetchTask to REPL DUMP plan for reading dump uri, last repl id as ResultSet
[
https://issues.apache.org/jira/browse/HIVE-15333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731228#comment-15731228
]
Hive QA commented on HIVE-15333:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12842230/HIVE-15333.4.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 10748 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=143)
[vectorized_rcfile_columnar.q,vector_elt.q,explainuser_1.q,multi_insert.q,tez_dml.q,vector_bround.q,schema_evol_orc_acid_table.q,vector_when_case_null.q,orc_ppd_schema_evol_1b.q,vector_join30.q,vectorization_11.q,cte_3.q,update_tmp_table.q,vector_decimal_cast.q,groupby_grouping_id2.q,vector_decimal_round.q,tez_smb_empty.q,orc_merge6.q,vector_decimal_trailing.q,cte_5.q,tez_union.q,cbo_rp_subq_not_in.q,vector_decimal_2.q,columnStatsUpdateForStatsOptimizer_1.q,vector_outer_join3.q,schema_evol_text_vec_part_all_complex.q,tez_dynpart_hashjoin_2.q,auto_sortmerge_join_12.q,offset_limit.q,tez_union_multiinsert.q]
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.org.apache.hadoop.hive.cli.TestBlobstoreCliDriver
(batchId=229)
org.apache.hadoop.hive.cli.TestBlobstoreNegativeCliDriver.org.apache.hadoop.hive.cli.TestBlobstoreNegativeCliDriver
(batchId=229)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=91)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2479/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2479/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2479/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12842230 - PreCommit-HIVE-Build
> Add a FetchTask to REPL DUMP plan for reading dump uri, last repl id as
> ResultSet
> -
>
> Key: HIVE-15333
> URL: https://issues.apache.org/jira/browse/HIVE-15333
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-15333.1.patch, HIVE-15333.2.patch,
> HIVE-15333.3.patch, HIVE-15333.4.patch
>
>
> We're writing the return values to a file, but we don't add FetchTask while
> planning.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15381) don't log the callstack for reduce.xml-doesn't-exist
[
https://issues.apache.org/jira/browse/HIVE-15381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731226#comment-15731226
]
Ashutosh Chauhan commented on HIVE-15381:
-
+1
> don't log the callstack for reduce.xml-doesn't-exist
>
>
> Key: HIVE-15381
> URL: https://issues.apache.org/jira/browse/HIVE-15381
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Trivial
> Attachments: HIVE-15381.patch
>
>
> Pointless exception in the logs:
> {noformat}
> java.io.FileNotFoundException: File
> file:[snip]/itests/qtest/target/tmp/localscratchdir/bcc7fce3-b9a3-4d5a-bf52-4e3b70ad9fed/hive_2016-12-07_09-53-18_167_8716888773328063866-1/-mr-10002/3fb6d7bd-d8b6-4238-bc08-90b2f0217197/reduce.xml
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:609)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:822)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:599)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:140)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341)
> ~[hadoop-common-2.7.2.jar:?]
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:421)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getReduceWork(Utilities.java:313)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:292)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.HiveOutputFormatImpl.checkOutputSpecs(HiveOutputFormatImpl.java:61)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:268)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_102]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_102]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_102]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_102]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:401)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:151)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2166)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1822)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1
[jira] [Updated] (HIVE-15386) Expose Spark task counts and stage Ids information in SparkTask from SparkJobMonitor
[ https://issues.apache.org/jira/browse/HIVE-15386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated HIVE-15386: - Status: Patch Available (was: Open) > Expose Spark task counts and stage Ids information in SparkTask from > SparkJobMonitor > > > Key: HIVE-15386 > URL: https://issues.apache.org/jira/browse/HIVE-15386 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 2.2.0 >Reporter: zhihai xu >Assignee: zhihai xu > Attachments: HIVE-15386.000.patch > > > Expose Spark task counts and stage Ids information in SparkTask from > SparkJobMonitor. So these information can be used by hive hook to monitor > spark jobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15386) Expose Spark task counts and stage Ids information in SparkTask from SparkJobMonitor
[ https://issues.apache.org/jira/browse/HIVE-15386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated HIVE-15386: - Attachment: HIVE-15386.000.patch > Expose Spark task counts and stage Ids information in SparkTask from > SparkJobMonitor > > > Key: HIVE-15386 > URL: https://issues.apache.org/jira/browse/HIVE-15386 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 2.2.0 >Reporter: zhihai xu >Assignee: zhihai xu > Attachments: HIVE-15386.000.patch > > > Expose Spark task counts and stage Ids information in SparkTask from > SparkJobMonitor. So these information can be used by hive hook to monitor > spark jobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15378) clean up HADOOP_USER_CLASSPATH_FIRST in bin scripts
[ https://issues.apache.org/jira/browse/HIVE-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731180#comment-15731180 ] Fei Hui commented on HIVE-15378: cc [~spena] > clean up HADOOP_USER_CLASSPATH_FIRST in bin scripts > --- > > Key: HIVE-15378 > URL: https://issues.apache.org/jira/browse/HIVE-15378 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 2.2.0 >Reporter: Fei Hui >Assignee: Fei Hui > Attachments: HIVE-15378.1.patch > > > beeline, hive, hplsql have this statement > export HADOOP_USER_CLASSPATH_FIRST=true > beeline and hplsql use 'hive --service' to start, so it is uselese in beeline > and hplsql > add export HADOOP_USER_CLASSPATH_FIRST=true to hive.cmd -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15381) don't log the callstack for reduce.xml-doesn't-exist
[
https://issues.apache.org/jira/browse/HIVE-15381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731089#comment-15731089
]
Hive QA commented on HIVE-15381:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12842227/HIVE-15381.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10752 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=141)
[acid_vectorization_missing_cols.q,orc_merge9.q,vector_acid3.q,delete_where_no_match.q,vector_reduce1.q,stats_only_null.q,vectorization_part_project.q,vectorization_6.q,count.q,tez_vector_dynpart_hashjoin_2.q,parallel.q,delete_all_non_partitioned.q,delete_all_partitioned.q,vectorization_10.q,insert1.q,custom_input_output_format.q,vectorized_bucketmapjoin1.q,cbo_rp_windowing_2.q,vector_reduce3.q,smb_cache.q,hybridgrace_hashjoin_1.q,vector_count_distinct.q,schema_evol_orc_acid_part.q,hybridgrace_hashjoin_2.q,cross_join.q,parquet_predicate_pushdown.q,vector_varchar_mapjoin1.q,tez_smb_main.q,quotedid_smb.q,vector_bucket.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=92)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_4]
(batchId=92)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2478/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2478/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2478/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12842227 - PreCommit-HIVE-Build
> don't log the callstack for reduce.xml-doesn't-exist
>
>
> Key: HIVE-15381
> URL: https://issues.apache.org/jira/browse/HIVE-15381
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Trivial
> Attachments: HIVE-15381.patch
>
>
> Pointless exception in the logs:
> {noformat}
> java.io.FileNotFoundException: File
> file:[snip]/itests/qtest/target/tmp/localscratchdir/bcc7fce3-b9a3-4d5a-bf52-4e3b70ad9fed/hive_2016-12-07_09-53-18_167_8716888773328063866-1/-mr-10002/3fb6d7bd-d8b6-4238-bc08-90b2f0217197/reduce.xml
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:609)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:822)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:599)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:140)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341)
> ~[hadoop-common-2.7.2.jar:?]
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:421)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getReduceWork(Utilities.java:313)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:292)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.HiveOutputFormatImpl.checkOutputSpecs(HiveOutputFormatImpl.java:61)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:268)
[jira] [Updated] (HIVE-15076) Improve scalability of LDAP authentication provider group filter
[ https://issues.apache.org/jira/browse/HIVE-15076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Illya Yalovyy updated HIVE-15076: - Status: In Progress (was: Patch Available) > Improve scalability of LDAP authentication provider group filter > > > Key: HIVE-15076 > URL: https://issues.apache.org/jira/browse/HIVE-15076 > Project: Hive > Issue Type: Improvement > Components: Authentication >Affects Versions: 2.1.0 >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: HIVE-15076.1.patch, HIVE-15076.2.patch, > HIVE-15076.3.patch > > > Current implementation uses following algorithm: > # For a given user find all groups that user is a member of. (A list of > LDAP groups is constructed as a result of that request) > # Match this list of groups with provided group filter. > > Time/Memory complexity of this approach is O(N) on client side, where N – is > a number of groups the user has membership in. On a large directory (800+ > groups per user) we can observe up to 2x performance degradation and failures > because of size of LDAP response (LDAP: error code 4 - Sizelimit Exceeded). > > Some Directory Services (Microsoft Active Directory for instance) provide a > virtual attribute for User Object that contains a list of groups that user > belongs to. This attribute can be used to quickly determine whether this user > passes or fails the group filter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15076) Improve scalability of LDAP authentication provider group filter
[ https://issues.apache.org/jira/browse/HIVE-15076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Illya Yalovyy updated HIVE-15076: - Status: Patch Available (was: In Progress) > Improve scalability of LDAP authentication provider group filter > > > Key: HIVE-15076 > URL: https://issues.apache.org/jira/browse/HIVE-15076 > Project: Hive > Issue Type: Improvement > Components: Authentication >Affects Versions: 2.1.0 >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: HIVE-15076.1.patch, HIVE-15076.2.patch, > HIVE-15076.3.patch > > > Current implementation uses following algorithm: > # For a given user find all groups that user is a member of. (A list of > LDAP groups is constructed as a result of that request) > # Match this list of groups with provided group filter. > > Time/Memory complexity of this approach is O(N) on client side, where N – is > a number of groups the user has membership in. On a large directory (800+ > groups per user) we can observe up to 2x performance degradation and failures > because of size of LDAP response (LDAP: error code 4 - Sizelimit Exceeded). > > Some Directory Services (Microsoft Active Directory for instance) provide a > virtual attribute for User Object that contains a list of groups that user > belongs to. This attribute can be used to quickly determine whether this user > passes or fails the group filter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15385) Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., false) causes queries to fail
[
https://issues.apache.org/jira/browse/HIVE-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731059#comment-15731059
]
Sahil Takiar commented on HIVE-15385:
-
HIVE-7015 changed the logic so that failures to inherit permissions don't fail
queries.
> Failure to inherit permissions when running HdfsUtils.setFullFileStatus(...,
> false) causes queries to fail
> --
>
> Key: HIVE-15385
> URL: https://issues.apache.org/jira/browse/HIVE-15385
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> According to
> https://cwiki.apache.org/confluence/display/Hive/Permission+Inheritance+in+Hive,
> failure to inherit permissions should not cause queries to fail.
> It looks like this was the case until HIVE-13716, which added some code to
> use {{fs.setOwner}}, {{fs.setAcl}}, and {{fs.setPermission}} to set
> permissions instead of shelling out and running {{-chgrp -R ...}}.
> When shelling out, the return status of each command is ignored, so if there
> are any failures when inheriting permissions, a warning is logged, but the
> query still succeeds.
> However, when invoked the {{FileSystem}} API, any failures will be propagated
> up to the caller, and the query will fail.
> This is problematic because {{setFulFileStatus}} shells out when the
> {{recursive}} parameter is set to {{true}}, and when it is false it invokes
> the {{FileSystem}} API. So the behavior is inconsistent depending on the
> value of {{recursive}}.
> We should decide whether or not permission inheritance should fail queries or
> not, and then ensure the code consistently follows that decision.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15385) Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., false) causes queries to fail
[
https://issues.apache.org/jira/browse/HIVE-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731056#comment-15731056
]
Sahil Takiar commented on HIVE-15385:
-
There are also a few places throughout {{Hive.java}} where exceptions thrown by
{{setFullFileStatus}} are swallowed, but there also a bunch of places where
exceptions are propagated to the caller.
> Failure to inherit permissions when running HdfsUtils.setFullFileStatus(...,
> false) causes queries to fail
> --
>
> Key: HIVE-15385
> URL: https://issues.apache.org/jira/browse/HIVE-15385
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> According to
> https://cwiki.apache.org/confluence/display/Hive/Permission+Inheritance+in+Hive,
> failure to inherit permissions should not cause queries to fail.
> It looks like this was the case until HIVE-13716, which added some code to
> use {{fs.setOwner}}, {{fs.setAcl}}, and {{fs.setPermission}} to set
> permissions instead of shelling out and running {{-chgrp -R ...}}.
> When shelling out, the return status of each command is ignored, so if there
> are any failures when inheriting permissions, a warning is logged, but the
> query still succeeds.
> However, when invoked the {{FileSystem}} API, any failures will be propagated
> up to the caller, and the query will fail.
> This is problematic because {{setFulFileStatus}} shells out when the
> {{recursive}} parameter is set to {{true}}, and when it is false it invokes
> the {{FileSystem}} API. So the behavior is inconsistent depending on the
> value of {{recursive}}.
> We should decide whether or not permission inheritance should fail queries or
> not, and then ensure the code consistently follows that decision.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15367) CTAS with LOCATION should write temp data under location directory rather than database location
[
https://issues.apache.org/jira/browse/HIVE-15367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730995#comment-15730995
]
Hive QA commented on HIVE-15367:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12842225/HIVE-15367.4.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10784 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_1]
(batchId=91)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_4]
(batchId=92)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2477/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2477/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2477/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12842225 - PreCommit-HIVE-Build
> CTAS with LOCATION should write temp data under location directory rather
> than database location
>
>
> Key: HIVE-15367
> URL: https://issues.apache.org/jira/browse/HIVE-15367
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-15367.1.patch, HIVE-15367.2.patch,
> HIVE-15367.3.patch, HIVE-15367.4.patch
>
>
> For regular CTAS queries, temp data from a SELECT query will be written to to
> a staging directory under the database location. The code to control this is
> in {{SemanticAnalyzer.java}}
> {code}
> // allocate a temporary output dir on the location of the table
> String tableName = getUnescapedName((ASTNode) ast.getChild(0));
> String[] names = Utilities.getDbTableName(tableName);
> Path location;
> try {
> Warehouse wh = new Warehouse(conf);
> //Use destination table's db location.
> String destTableDb = qb.getTableDesc() != null?
> qb.getTableDesc().getDatabaseName(): null;
> if (destTableDb == null) {
> destTableDb = names[0];
> }
> location = wh.getDatabasePath(db.getDatabase(destTableDb));
> } catch (MetaException e) {
> throw new SemanticException(e);
> }
> {code}
> However, CTAS queries allow specifying a {{LOCATION}} for the new table. Its
> possible for this location to be on a different filesystem than the database
> location. If this happens temp data will be written to the database
> filesystem and will be copied to the table filesystem in {{MoveTask}}.
> This extra copying of data can drastically affect performance. Rather than
> always use the database location as the staging dir for CTAS queries, Hive
> should first check if there is an explicit {{LOCATION}} specified in the CTAS
> query. If there is, staging data should be stored under the {{LOCATION}}
> directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15376) Improve heartbeater scheduling for transactions
[
https://issues.apache.org/jira/browse/HIVE-15376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730918#comment-15730918
]
Hive QA commented on HIVE-15376:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12842224/HIVE-15376.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 10736 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=143)
[vectorized_rcfile_columnar.q,vector_elt.q,explainuser_1.q,multi_insert.q,tez_dml.q,vector_bround.q,schema_evol_orc_acid_table.q,vector_when_case_null.q,orc_ppd_schema_evol_1b.q,vector_join30.q,vectorization_11.q,cte_3.q,update_tmp_table.q,vector_decimal_cast.q,groupby_grouping_id2.q,vector_decimal_round.q,tez_smb_empty.q,orc_merge6.q,vector_decimal_trailing.q,cte_5.q,tez_union.q,cbo_rp_subq_not_in.q,vector_decimal_2.q,columnStatsUpdateForStatsOptimizer_1.q,vector_outer_join3.q,schema_evol_text_vec_part_all_complex.q,tez_dynpart_hashjoin_2.q,auto_sortmerge_join_12.q,offset_limit.q,tez_union_multiinsert.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=102)
[skewjoinopt19.q,order.q,join_merge_multi_expressions.q,skewjoinopt10.q,union5.q,insert_into1.q,vectorized_math_funcs.q,vectorization_4.q,vectorization_2.q,skewjoinopt6.q,decimal_1_1.q,join14.q,outer_join_ppr.q,rcfile_bigdata.q,load_dyn_part10.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=132)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=92)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_4]
(batchId=92)
org.apache.hadoop.hive.ql.TestTxnCommands2.testFailHeartbeater (batchId=260)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdate.testFailHeartbeater
(batchId=270)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testFailHeartbeater
(batchId=267)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2476/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2476/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2476/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12842224 - PreCommit-HIVE-Build
> Improve heartbeater scheduling for transactions
> ---
>
> Key: HIVE-15376
> URL: https://issues.apache.org/jira/browse/HIVE-15376
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15376.1.patch, HIVE-15376.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15337) Enhance Show Compactions output with JobId and start time for "attempted" state
[ https://issues.apache.org/jira/browse/HIVE-15337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-15337: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Enhance Show Compactions output with JobId and start time for "attempted" > state > --- > > Key: HIVE-15337 > URL: https://issues.apache.org/jira/browse/HIVE-15337 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 2.2.0 > > Attachments: HIVE-15337.01.patch, HIVE-15337.02.patch, > HIVE-15337.03.addendum.patch, HIVE-15337.03.patch > > > W/o this SHOW COMPACTIONS output is not as useful > Also, add Hadoop Job ID to SHOW COMPACTIONS output -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15337) Enhance Show Compactions output with JobId and start time for "attempted" state
[ https://issues.apache.org/jira/browse/HIVE-15337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730840#comment-15730840 ] Eugene Koifman commented on HIVE-15337: --- committed HIVE-15337.03.addendum.patch to master > Enhance Show Compactions output with JobId and start time for "attempted" > state > --- > > Key: HIVE-15337 > URL: https://issues.apache.org/jira/browse/HIVE-15337 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 2.2.0 > > Attachments: HIVE-15337.01.patch, HIVE-15337.02.patch, > HIVE-15337.03.addendum.patch, HIVE-15337.03.patch > > > W/o this SHOW COMPACTIONS output is not as useful > Also, add Hadoop Job ID to SHOW COMPACTIONS output -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15307) Hive MERGE: "when matched then update" allows invalid column names.
[
https://issues.apache.org/jira/browse/HIVE-15307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-15307:
--
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
committed to master
Thanks Alan for the review
> Hive MERGE: "when matched then update" allows invalid column names.
> ---
>
> Key: HIVE-15307
> URL: https://issues.apache.org/jira/browse/HIVE-15307
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 2.2.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 2.2.0
>
> Attachments: HIVE-15307.01.patch, HIVE-15307.02.patch
>
>
> {noformat}
> create table target (
> id int, val int
> )
> CLUSTERED BY (id) INTO 2 BUCKETS STORED AS ORC TBLPROPERTIES
> ("transactional"="true");
> create table source2 (
> id int, val int
> );
> insert into source2 values (2, 25), (3, 35), (4, 45);
> merge into target
> using source2 sub on sub.id = target.id
> when matched then update set invalid = sub.val;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15337) Enhance Show Compactions output with JobId and start time for "attempted" state
[ https://issues.apache.org/jira/browse/HIVE-15337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730799#comment-15730799 ] Eugene Koifman commented on HIVE-15337: --- Test failures not related > Enhance Show Compactions output with JobId and start time for "attempted" > state > --- > > Key: HIVE-15337 > URL: https://issues.apache.org/jira/browse/HIVE-15337 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 2.2.0 > > Attachments: HIVE-15337.01.patch, HIVE-15337.02.patch, > HIVE-15337.03.addendum.patch, HIVE-15337.03.patch > > > W/o this SHOW COMPACTIONS output is not as useful > Also, add Hadoop Job ID to SHOW COMPACTIONS output -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10937) LLAP: make ObjectCache for plans work properly in the daemon
[
https://issues.apache.org/jira/browse/HIVE-10937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730777#comment-15730777
]
Hive QA commented on HIVE-10937:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12772892/HIVE-10937.05.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2475/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2475/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2475/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2016-12-08 01:48:10.592
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-2475/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2016-12-08 01:48:10.594
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at c097884 HIVE-15361: INSERT dynamic partition on S3 fails with a
MoveTask failure (Sergio Pena, reviewed by Mohit Sabharwal and Illya Yalovvy)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at c097884 HIVE-15361: INSERT dynamic partition on S3 fails with a
MoveTask failure (Sergio Pena, reviewed by Mohit Sabharwal and Illya Yalovvy)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2016-12-08 01:48:11.550
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:2348
error: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: patch does
not apply
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java:25
error: ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java:
patch does not apply
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java:32
error: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java:
patch does not apply
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileRecordProcessor.java:64
error:
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileRecordProcessor.java:
patch does not apply
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java:28
error:
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java:
patch does not apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12772892 - PreCommit-HIVE-Build
> LLAP: make ObjectCache for plans work properly in the daemon
>
>
> Key: HIVE-10937
> URL: https://issues.apache.org/jira/browse/HIVE-10937
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-10937.01.patch, HIVE-10937.02.patch,
> HIVE-10937.03.patch, HIVE-10937.04.patch, HIVE-10937.05.patch,
> HIVE-10937.patch
>
>
> There's perf hit otherwise, esp. when stupid planner creates 1009 reducers of
> 4Mb each.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15337) Enhance Show Compactions output with JobId and start time for "attempted" state
[
https://issues.apache.org/jira/browse/HIVE-15337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730770#comment-15730770
]
Hive QA commented on HIVE-15337:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12842170/HIVE-15337.03.addendum.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10781 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=92)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2474/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2474/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2474/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12842170 - PreCommit-HIVE-Build
> Enhance Show Compactions output with JobId and start time for "attempted"
> state
> ---
>
> Key: HIVE-15337
> URL: https://issues.apache.org/jira/browse/HIVE-15337
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 2.2.0
>
> Attachments: HIVE-15337.01.patch, HIVE-15337.02.patch,
> HIVE-15337.03.addendum.patch, HIVE-15337.03.patch
>
>
> W/o this SHOW COMPACTIONS output is not as useful
> Also, add Hadoop Job ID to SHOW COMPACTIONS output
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Status: Patch Available (was: Open) > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, > HIVE-14731.6.patch, HIVE-14731.7.patch, HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15338) Wrong result from non-vectorized DATEDIFF with scalar parameter of type DATE/TIMESTAMP
[ https://issues.apache.org/jira/browse/HIVE-15338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-15338: Attachment: HIVE-15338.03.patch > Wrong result from non-vectorized DATEDIFF with scalar parameter of type > DATE/TIMESTAMP > -- > > Key: HIVE-15338 > URL: https://issues.apache.org/jira/browse/HIVE-15338 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15338.01.patch, HIVE-15338.02.patch, > HIVE-15338.03.patch > > > Vectorization in vectorized DATEDIFF accidentally treated the scalar > parameter is type DATE (e.g. CURRENT_DATE) as 0. > Current Q file test vectorized_date_funcs.q DOES NOT test the DATE/TIMESTAMP > scalar type case. > And, non-vectorized cases of DATEDIFF are using UTF and returning the wrong > results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15338) Wrong result from non-vectorized DATEDIFF with scalar parameter of type DATE/TIMESTAMP
[ https://issues.apache.org/jira/browse/HIVE-15338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-15338: Status: Patch Available (was: In Progress) > Wrong result from non-vectorized DATEDIFF with scalar parameter of type > DATE/TIMESTAMP > -- > > Key: HIVE-15338 > URL: https://issues.apache.org/jira/browse/HIVE-15338 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15338.01.patch, HIVE-15338.02.patch, > HIVE-15338.03.patch > > > Vectorization in vectorized DATEDIFF accidentally treated the scalar > parameter is type DATE (e.g. CURRENT_DATE) as 0. > Current Q file test vectorized_date_funcs.q DOES NOT test the DATE/TIMESTAMP > scalar type case. > And, non-vectorized cases of DATEDIFF are using UTF and returning the wrong > results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated HIVE-14731: Attachment: HIVE-14731.9.patch Upload new patch to fix the outer join issue. XPROD_EDGE is no longer used for outer join. > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, > HIVE-14731.6.patch, HIVE-14731.7.patch, HIVE-14731.8.patch, HIVE-14731.9.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15338) Wrong result from non-vectorized DATEDIFF with scalar parameter of type DATE/TIMESTAMP
[ https://issues.apache.org/jira/browse/HIVE-15338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-15338: Status: In Progress (was: Patch Available) > Wrong result from non-vectorized DATEDIFF with scalar parameter of type > DATE/TIMESTAMP > -- > > Key: HIVE-15338 > URL: https://issues.apache.org/jira/browse/HIVE-15338 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15338.01.patch, HIVE-15338.02.patch > > > Vectorization in vectorized DATEDIFF accidentally treated the scalar > parameter is type DATE (e.g. CURRENT_DATE) as 0. > Current Q file test vectorized_date_funcs.q DOES NOT test the DATE/TIMESTAMP > scalar type case. > And, non-vectorized cases of DATEDIFF are using UTF and returning the wrong > results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13306) Better Decimal vectorization
[
https://issues.apache.org/jira/browse/HIVE-13306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730629#comment-15730629
]
Hive QA commented on HIVE-13306:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12842166/HIVE-13306.5.patch
{color:green}SUCCESS:{color} +1 due to 11 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10767 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=143)
[vectorized_rcfile_columnar.q,vector_elt.q,explainuser_1.q,multi_insert.q,tez_dml.q,vector_bround.q,schema_evol_orc_acid_table.q,vector_when_case_null.q,orc_ppd_schema_evol_1b.q,vector_join30.q,vectorization_11.q,cte_3.q,update_tmp_table.q,vector_decimal_cast.q,groupby_grouping_id2.q,vector_decimal_round.q,tez_smb_empty.q,orc_merge6.q,vector_decimal_trailing.q,cte_5.q,tez_union.q,cbo_rp_subq_not_in.q,vector_decimal_2.q,columnStatsUpdateForStatsOptimizer_1.q,vector_outer_join3.q,schema_evol_text_vec_part_all_complex.q,tez_dynpart_hashjoin_2.q,auto_sortmerge_join_12.q,offset_limit.q,tez_union_multiinsert.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision]
(batchId=150)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2473/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2473/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2473/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12842166 - PreCommit-HIVE-Build
> Better Decimal vectorization
>
>
> Key: HIVE-13306
> URL: https://issues.apache.org/jira/browse/HIVE-13306
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Teddy Choi
>Priority: Critical
> Attachments: HIVE-13306.1.patch, HIVE-13306.2.patch,
> HIVE-13306.3.patch, HIVE-13306.4.patch, HIVE-13306.5.patch
>
>
> Decimal Vectorization Requirements
> • Today, the LongColumnVector, DoubleColumnVector, BytesColumnVector,
> TimestampColumnVector classes store the data as primitive Java data types
> long, double, or byte arrays for efficiency.
> • DecimalColumnVector is different - it has an array of Object references
> to HiveDecimal objects.
> • The HiveDecimal object uses an internal object BigDecimal for its
> implementation. Further, BigDecimal itself uses an internal object
> BigInteger for its implementation, and BigInteger uses an int array. 4
> objects total.
> • And, HiveDecimal is an immutable object which means arithmetic and
> other operations produce new HiveDecimal object with 3 new objects underneath.
> • A major reason Vectorization is fast is the ColumnVector classes except
> DecimalColumnVector do not have to allocate additional memory per row. This
> avoids memory fragmentation and pressure on the Java Garbage Collector that
> DecimalColumnVector can generate. It is very significant.
> • What can be done with DecimalColumnVector to make it much more
> efficient?
> o Design several new decimal classes that allow the caller to manage the
> decimal storage.
> o If it takes N int values to store a decimal (e.g. N=1..5), then a new
> DecimalColumnVector would have an int[] of length N*1024 (where 1024 is the
> default column vector size).
> o Why store a decimal in separate int values?
> • Java does not support 128 bit integers.
> • Java does not support unsigned integers.
> • In order to do multiplication of a decimal represented in a long you
> need twice the storage (i.e. 128 bits). So you need to represent parts in 32
> bit integers.
> • But really since we do not have unsigned, really you can only
[jira] [Updated] (HIVE-15192) Use Calcite to de-correlate and plan subqueries
[ https://issues.apache.org/jira/browse/HIVE-15192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-15192: --- Status: Patch Available (was: Open) > Use Calcite to de-correlate and plan subqueries > --- > > Key: HIVE-15192 > URL: https://issues.apache.org/jira/browse/HIVE-15192 > Project: Hive > Issue Type: Task > Components: Logical Optimizer >Reporter: Vineet Garg >Assignee: Vineet Garg > Labels: sub-query > Attachments: HIVE-15192.2.patch, HIVE-15192.3.patch, > HIVE-15192.4.patch, HIVE-15192.5.patch, HIVE-15192.patch > > > Currently support of subqueries is limited [Link to original spec | > https://issues.apache.org/jira/secure/attachment/12614003/SubQuerySpec.pdf]. > Using Calcite to plan and de-correlate subqueries will help Hive get rid of > these limitations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15192) Use Calcite to de-correlate and plan subqueries
[ https://issues.apache.org/jira/browse/HIVE-15192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-15192: --- Status: Open (was: Patch Available) > Use Calcite to de-correlate and plan subqueries > --- > > Key: HIVE-15192 > URL: https://issues.apache.org/jira/browse/HIVE-15192 > Project: Hive > Issue Type: Task > Components: Logical Optimizer >Reporter: Vineet Garg >Assignee: Vineet Garg > Labels: sub-query > Attachments: HIVE-15192.2.patch, HIVE-15192.3.patch, > HIVE-15192.4.patch, HIVE-15192.5.patch, HIVE-15192.patch > > > Currently support of subqueries is limited [Link to original spec | > https://issues.apache.org/jira/secure/attachment/12614003/SubQuerySpec.pdf]. > Using Calcite to plan and de-correlate subqueries will help Hive get rid of > these limitations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15192) Use Calcite to de-correlate and plan subqueries
[ https://issues.apache.org/jira/browse/HIVE-15192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-15192: --- Attachment: HIVE-15192.5.patch > Use Calcite to de-correlate and plan subqueries > --- > > Key: HIVE-15192 > URL: https://issues.apache.org/jira/browse/HIVE-15192 > Project: Hive > Issue Type: Task > Components: Logical Optimizer >Reporter: Vineet Garg >Assignee: Vineet Garg > Labels: sub-query > Attachments: HIVE-15192.2.patch, HIVE-15192.3.patch, > HIVE-15192.4.patch, HIVE-15192.5.patch, HIVE-15192.patch > > > Currently support of subqueries is limited [Link to original spec | > https://issues.apache.org/jira/secure/attachment/12614003/SubQuerySpec.pdf]. > Using Calcite to plan and de-correlate subqueries will help Hive get rid of > these limitations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11812) datediff sometimes returns incorrect results when called with dates
[
https://issues.apache.org/jira/browse/HIVE-11812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730478#comment-15730478
]
Jason Dere commented on HIVE-11812:
---
Ok, I think the issue is the fact that the date formatter is using UTC timezone
when parsing dates from string. I believe this allows results to work correctly
when both parameters are of string type, but i think this messes up results
when the one parameter is of string/char/varchar type and the other is
date/timestamp, because the date/timestamp value will have a millis result
based on the date for the local timezone. I believe [~mmccline]'s patch for
HIVE-15338 may be able to handle this properly since it is parsing dates using
the local timezone (to match Date/Timestamp type), and will be using
dateToDays() (which hopefully works correctly, otherwise this may need fixes),
rather than trying to do some milliseconds-to-days conversion here in the UDF.
> datediff sometimes returns incorrect results when called with dates
> ---
>
> Key: HIVE-11812
> URL: https://issues.apache.org/jira/browse/HIVE-11812
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 2.0.0
>Reporter: Nicholas Brenwald
>Assignee: Chetna Chaudhari
>Priority: Minor
> Attachments: HIVE-11812.1.patch
>
>
> DATEDIFF returns an incorrect result when one of the arguments is a date
> type.
> The Hive Language Manual provides the following signature for datediff:
> {code}
> int datediff(string enddate, string startdate)
> {code}
> I think datediff should either throw an error (if date types are not
> supported), or return the correct result.
> To reproduce, create a table:
> {code}
> create table t (c1 string, c2 date);
> {code}
> Assuming you have a table x containing some data, populate table t with 1 row:
> {code}
> insert into t select '2015-09-15', '2015-09-15' from x limit 1;
> {code}
> Then run the following 12 test queries:
> {code}
> select datediff(c1, '2015-09-14') from t;
> select datediff(c1, '2015-09-15') from t;
> select datediff(c1, '2015-09-16') from t;
> select datediff('2015-09-14', c1) from t;
> select datediff('2015-09-15', c1) from t;
> select datediff('2015-09-16', c1) from t;
> select datediff(c2, '2015-09-14') from t;
> select datediff(c2, '2015-09-15') from t;
> select datediff(c2, '2015-09-16') from t;
> select datediff('2015-09-14', c2) from t;
> select datediff('2015-09-15', c2) from t;
> select datediff('2015-09-16', c2) from t;
> {code}
> The below table summarises the result. All results for column c1 (which is a
> string) are correct, but when using c2 (which is a date), two of the results
> are incorrect.
> || Test || Expected Result || Actual Result || Passed / Failed ||
> |datediff(c1, '2015-09-14')| 1 | 1| Passed |
> |datediff(c1, '2015-09-15')| 0 | 0| Passed |
> |datediff(c1, '2015-09-16') | -1 | -1| Passed |
> |datediff('2015-09-14', c1) | -1 | -1| Passed |
> |datediff('2015-09-15', c1)| 0 | 0| Passed |
> |datediff('2015-09-16', c1)| 1 | 1| Passed |
> |datediff(c2, '2015-09-14')| 1 | 0| {color:red}Failed{color} |
> |datediff(c2, '2015-09-15')| 0 | 0| Passed |
> |datediff(c2, '2015-09-16') | -1 | -1| Passed |
> |datediff('2015-09-14', c2) | -1 | 0 | {color:red}Failed{color} |
> |datediff('2015-09-15', c2)| 0 | 0| Passed |
> |datediff('2015-09-16', c2)| 1 | 1| Passed |
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15076) Improve scalability of LDAP authentication provider group filter
[ https://issues.apache.org/jira/browse/HIVE-15076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Illya Yalovyy updated HIVE-15076: - Attachment: HIVE-15076.3.patch > Improve scalability of LDAP authentication provider group filter > > > Key: HIVE-15076 > URL: https://issues.apache.org/jira/browse/HIVE-15076 > Project: Hive > Issue Type: Improvement > Components: Authentication >Affects Versions: 2.1.0 >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: HIVE-15076.1.patch, HIVE-15076.2.patch, > HIVE-15076.3.patch > > > Current implementation uses following algorithm: > # For a given user find all groups that user is a member of. (A list of > LDAP groups is constructed as a result of that request) > # Match this list of groups with provided group filter. > > Time/Memory complexity of this approach is O(N) on client side, where N – is > a number of groups the user has membership in. On a large directory (800+ > groups per user) we can observe up to 2x performance degradation and failures > because of size of LDAP response (LDAP: error code 4 - Sizelimit Exceeded). > > Some Directory Services (Microsoft Active Directory for instance) provide a > virtual attribute for User Object that contains a list of groups that user > belongs to. This attribute can be used to quickly determine whether this user > passes or fails the group filter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15385) Failure to inherit permissions when running HdfsUtils.setFullFileStatus(..., false) causes queries to fail
[
https://issues.apache.org/jira/browse/HIVE-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730435#comment-15730435
]
Sahil Takiar commented on HIVE-15385:
-
[~spena] does my explanation of the situation make sense? Any comments on
whether or not failure to inherit permissions should cause query failures or
not?
> Failure to inherit permissions when running HdfsUtils.setFullFileStatus(...,
> false) causes queries to fail
> --
>
> Key: HIVE-15385
> URL: https://issues.apache.org/jira/browse/HIVE-15385
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> According to
> https://cwiki.apache.org/confluence/display/Hive/Permission+Inheritance+in+Hive,
> failure to inherit permissions should not cause queries to fail.
> It looks like this was the case until HIVE-13716, which added some code to
> use {{fs.setOwner}}, {{fs.setAcl}}, and {{fs.setPermission}} to set
> permissions instead of shelling out and running {{-chgrp -R ...}}.
> When shelling out, the return status of each command is ignored, so if there
> are any failures when inheriting permissions, a warning is logged, but the
> query still succeeds.
> However, when invoked the {{FileSystem}} API, any failures will be propagated
> up to the caller, and the query will fail.
> This is problematic because {{setFulFileStatus}} shells out when the
> {{recursive}} parameter is set to {{true}}, and when it is false it invokes
> the {{FileSystem}} API. So the behavior is inconsistent depending on the
> value of {{recursive}}.
> We should decide whether or not permission inheritance should fail queries or
> not, and then ensure the code consistently follows that decision.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15367) CTAS with LOCATION should write temp data under location directory rather than database location
[
https://issues.apache.org/jira/browse/HIVE-15367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730383#comment-15730383
]
Sahil Takiar commented on HIVE-15367:
-
Thanks for the review Sergio!
> CTAS with LOCATION should write temp data under location directory rather
> than database location
>
>
> Key: HIVE-15367
> URL: https://issues.apache.org/jira/browse/HIVE-15367
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-15367.1.patch, HIVE-15367.2.patch,
> HIVE-15367.3.patch, HIVE-15367.4.patch
>
>
> For regular CTAS queries, temp data from a SELECT query will be written to to
> a staging directory under the database location. The code to control this is
> in {{SemanticAnalyzer.java}}
> {code}
> // allocate a temporary output dir on the location of the table
> String tableName = getUnescapedName((ASTNode) ast.getChild(0));
> String[] names = Utilities.getDbTableName(tableName);
> Path location;
> try {
> Warehouse wh = new Warehouse(conf);
> //Use destination table's db location.
> String destTableDb = qb.getTableDesc() != null?
> qb.getTableDesc().getDatabaseName(): null;
> if (destTableDb == null) {
> destTableDb = names[0];
> }
> location = wh.getDatabasePath(db.getDatabase(destTableDb));
> } catch (MetaException e) {
> throw new SemanticException(e);
> }
> {code}
> However, CTAS queries allow specifying a {{LOCATION}} for the new table. Its
> possible for this location to be on a different filesystem than the database
> location. If this happens temp data will be written to the database
> filesystem and will be copied to the table filesystem in {{MoveTask}}.
> This extra copying of data can drastically affect performance. Rather than
> always use the database location as the staging dir for CTAS queries, Hive
> should first check if there is an explicit {{LOCATION}} specified in the CTAS
> query. If there is, staging data should be stored under the {{LOCATION}}
> directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7847) query orc partitioned table fail when table column type change
[
https://issues.apache.org/jira/browse/HIVE-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730350#comment-15730350
]
Hive QA commented on HIVE-7847:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12680989/vector_alter_partition_change_col.q
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2472/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2472/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2472/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2016-12-07 23:51:28.741
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-2472/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2016-12-07 23:51:28.743
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at c097884 HIVE-15361: INSERT dynamic partition on S3 fails with a
MoveTask failure (Sergio Pena, reviewed by Mohit Sabharwal and Illya Yalovvy)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at c097884 HIVE-15361: INSERT dynamic partition on S3 fails with a
MoveTask failure (Sergio Pena, reviewed by Mohit Sabharwal and Illya Yalovvy)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2016-12-07 23:51:29.686
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
fatal: unrecognized input
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12680989 - PreCommit-HIVE-Build
> query orc partitioned table fail when table column type change
> --
>
> Key: HIVE-7847
> URL: https://issues.apache.org/jira/browse/HIVE-7847
> Project: Hive
> Issue Type: Bug
> Components: File Formats
>Affects Versions: 0.11.0, 0.12.0, 0.13.0
>Reporter: Zhichun Wu
>Assignee: Zhichun Wu
> Attachments: HIVE-7847.1.patch, vector_alter_partition_change_col.q
>
>
> I use the following script to test orc column type change with partitioned
> table on branch-0.13:
> {code}
> use test;
> DROP TABLE if exists orc_change_type_staging;
> DROP TABLE if exists orc_change_type;
> CREATE TABLE orc_change_type_staging (
> id int
> );
> CREATE TABLE orc_change_type (
> id int
> ) PARTITIONED BY (`dt` string)
> stored as orc;
> --- load staging table
> LOAD DATA LOCAL INPATH '../hive/examples/files/int.txt' OVERWRITE INTO TABLE
> orc_change_type_staging;
> --- populate orc hive table
> INSERT OVERWRITE TABLE orc_change_type partition(dt='20140718') select * FROM
> orc_change_type_staging limit 1;
> --- change column id from int to bigint
> ALTER TABLE orc_change_type CHANGE id id bigint;
> INSERT OVERWRITE TABLE orc_change_type partition(dt='20140719') select * FROM
> orc_change_type_staging limit 1;
> SELECT id FROM orc_change_type where dt between '20140718' and '20140719';
> {code}
> if fails in the last query "SELECT id FROM orc_change_type where dt between
> '20140718' and '20140719';" with exception:
> {code}
> Error: java.io.IOException: java.io.IOException:
> java.lang.ClassCastException: org.apache.hadoop.io.IntWritable cannot be cast
> to org.apache.hadoop.io.Lon
[jira] [Commented] (HIVE-15338) Wrong result from non-vectorized DATEDIFF with scalar parameter of type DATE/TIMESTAMP
[
https://issues.apache.org/jira/browse/HIVE-15338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730345#comment-15730345
]
Hive QA commented on HIVE-15338:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12842135/HIVE-15338.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 10781 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[date_udf] (batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_between_in]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_date_funcs]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_1]
(batchId=91)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=92)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[date_udf]
(batchId=106)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_between_in]
(batchId=117)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2471/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2471/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2471/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12842135 - PreCommit-HIVE-Build
> Wrong result from non-vectorized DATEDIFF with scalar parameter of type
> DATE/TIMESTAMP
> --
>
> Key: HIVE-15338
> URL: https://issues.apache.org/jira/browse/HIVE-15338
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-15338.01.patch, HIVE-15338.02.patch
>
>
> Vectorization in vectorized DATEDIFF accidentally treated the scalar
> parameter is type DATE (e.g. CURRENT_DATE) as 0.
> Current Q file test vectorized_date_funcs.q DOES NOT test the DATE/TIMESTAMP
> scalar type case.
> And, non-vectorized cases of DATEDIFF are using UTF and returning the wrong
> results.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15384) Compressor plugin framework
[ https://issues.apache.org/jira/browse/HIVE-15384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ziyang Zhao updated HIVE-15384: --- Issue Type: Sub-task (was: Task) Parent: HIVE-14549 > Compressor plugin framework > --- > > Key: HIVE-15384 > URL: https://issues.apache.org/jira/browse/HIVE-15384 > Project: Hive > Issue Type: Sub-task >Reporter: Ziyang Zhao > > Splitting server framework into separate JIRA from compressor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15331) Decimal multiplication with high precision/scale often returns NULL
[
https://issues.apache.org/jira/browse/HIVE-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-15331:
--
Attachment: HIVE-15331.3.patch
Updating golden files.
> Decimal multiplication with high precision/scale often returns NULL
> ---
>
> Key: HIVE-15331
> URL: https://issues.apache.org/jira/browse/HIVE-15331
> Project: Hive
> Issue Type: Bug
> Components: Types
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-15331.1.patch, HIVE-15331.2.patch,
> HIVE-15331.3.patch
>
>
> {noformat}
> create temporary table dec (a decimal(38,18));
> insert into dec values(100.0);
> hive> select a*a from dec;
> OK
> NULL
> Time taken: 0.165 seconds, Fetched: 1 row(s)
> {noformat}
> Looks like the reason is because the result of decimal(38,18) *
> decimal(38,18) only has 2 digits of precision for integers:
> {noformat}
> hive> set hive.explain.user=false;
> hive> explain select a*a from dec;
> OK
> STAGE DEPENDENCIES:
> Stage-0 is a root stage
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> TableScan
> alias: dec
> Select Operator
> expressions: (a * a) (type: decimal(38,36))
> outputColumnNames: _col0
> ListSink
> Time taken: 0.039 seconds, Fetched: 15 row(s)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15367) CTAS with LOCATION should write temp data under location directory rather than database location
[
https://issues.apache.org/jira/browse/HIVE-15367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730232#comment-15730232
]
Sergio Peña commented on HIVE-15367:
Thanks Sahil. I agree with your comment on the RB. The patch looks good.
+1
> CTAS with LOCATION should write temp data under location directory rather
> than database location
>
>
> Key: HIVE-15367
> URL: https://issues.apache.org/jira/browse/HIVE-15367
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-15367.1.patch, HIVE-15367.2.patch,
> HIVE-15367.3.patch, HIVE-15367.4.patch
>
>
> For regular CTAS queries, temp data from a SELECT query will be written to to
> a staging directory under the database location. The code to control this is
> in {{SemanticAnalyzer.java}}
> {code}
> // allocate a temporary output dir on the location of the table
> String tableName = getUnescapedName((ASTNode) ast.getChild(0));
> String[] names = Utilities.getDbTableName(tableName);
> Path location;
> try {
> Warehouse wh = new Warehouse(conf);
> //Use destination table's db location.
> String destTableDb = qb.getTableDesc() != null?
> qb.getTableDesc().getDatabaseName(): null;
> if (destTableDb == null) {
> destTableDb = names[0];
> }
> location = wh.getDatabasePath(db.getDatabase(destTableDb));
> } catch (MetaException e) {
> throw new SemanticException(e);
> }
> {code}
> However, CTAS queries allow specifying a {{LOCATION}} for the new table. Its
> possible for this location to be on a different filesystem than the database
> location. If this happens temp data will be written to the database
> filesystem and will be copied to the table filesystem in {{MoveTask}}.
> This extra copying of data can drastically affect performance. Rather than
> always use the database location as the staging dir for CTAS queries, Hive
> should first check if there is an explicit {{LOCATION}} specified in the CTAS
> query. If there is, staging data should be stored under the {{LOCATION}}
> directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-6638) Hive needs to implement recovery for Application Master restart
[
https://issues.apache.org/jira/browse/HIVE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730225#comment-15730225
]
Hive QA commented on HIVE-6638:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12637702/HIVE-6638.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2470/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2470/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2470/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2016-12-07 23:03:53.778
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-2470/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2016-12-07 23:03:53.780
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at c097884 HIVE-15361: INSERT dynamic partition on S3 fails with a
MoveTask failure (Sergio Pena, reviewed by Mohit Sabharwal and Illya Yalovvy)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at c097884 HIVE-15361: INSERT dynamic partition on S3 fails with a
MoveTask failure (Sergio Pena, reviewed by Mohit Sabharwal and Illya Yalovvy)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2016-12-07 23:03:54.787
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: No such
file or directory
error:
a/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java: No
such file or directory
error:
a/shims/common/src/main/java/org/apache/hadoop/hive/shims/ShimLoader.java: No
such file or directory
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12637702 - PreCommit-HIVE-Build
> Hive needs to implement recovery for Application Master restart
>
>
> Key: HIVE-6638
> URL: https://issues.apache.org/jira/browse/HIVE-6638
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 0.11.0, 0.12.0, 0.13.0
>Reporter: Ashutosh Chauhan
>Assignee: Mohammad Kamrul Islam
> Attachments: HIVE-6638.1.patch, HIVE-6638.2.patch
>
>
> Currently, if AM restarts, whole job is restarted. Although, job and
> subsequently query would still finish to completion, it would be nice if Hive
> don't need to redo all the work done under previous AM.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15378) clean up HADOOP_USER_CLASSPATH_FIRST in bin scripts
[
https://issues.apache.org/jira/browse/HIVE-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730221#comment-15730221
]
Hive QA commented on HIVE-15378:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12842122/HIVE-15378.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10777 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision]
(batchId=150)
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver
(batchId=228)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2469/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2469/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2469/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12842122 - PreCommit-HIVE-Build
> clean up HADOOP_USER_CLASSPATH_FIRST in bin scripts
> ---
>
> Key: HIVE-15378
> URL: https://issues.apache.org/jira/browse/HIVE-15378
> Project: Hive
> Issue Type: Bug
> Components: Build Infrastructure
>Affects Versions: 2.2.0
>Reporter: Fei Hui
>Assignee: Fei Hui
> Attachments: HIVE-15378.1.patch
>
>
> beeline, hive, hplsql have this statement
> export HADOOP_USER_CLASSPATH_FIRST=true
> beeline and hplsql use 'hive --service' to start, so it is uselese in beeline
> and hplsql
> add export HADOOP_USER_CLASSPATH_FIRST=true to hive.cmd
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15307) Hive MERGE: "when matched then update" allows invalid column names.
[
https://issues.apache.org/jira/browse/HIVE-15307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730214#comment-15730214
]
Alan Gates commented on HIVE-15307:
---
+1
> Hive MERGE: "when matched then update" allows invalid column names.
> ---
>
> Key: HIVE-15307
> URL: https://issues.apache.org/jira/browse/HIVE-15307
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 2.2.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-15307.01.patch, HIVE-15307.02.patch
>
>
> {noformat}
> create table target (
> id int, val int
> )
> CLUSTERED BY (id) INTO 2 BUCKETS STORED AS ORC TBLPROPERTIES
> ("transactional"="true");
> create table source2 (
> id int, val int
> );
> insert into source2 values (2, 25), (3, 35), (4, 45);
> merge into target
> using source2 sub on sub.id = target.id
> when matched then update set invalid = sub.val;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15383) Add additional info to 'desc function extended' output
[ https://issues.apache.org/jira/browse/HIVE-15383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-15383: Status: Patch Available (was: Open) Initial patch: there are many unit test cases to be fixed. But here is the code part. > Add additional info to 'desc function extended' output > -- > > Key: HIVE-15383 > URL: https://issues.apache.org/jira/browse/HIVE-15383 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Trivial > Attachments: HIVE-15383.1.patch > > > Add additional info to the output to 'desc function extended'. The resources > would be helpful for the user to check which jars are referred. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15333) Add a FetchTask to REPL DUMP plan for reading dump uri, last repl id as ResultSet
[ https://issues.apache.org/jira/browse/HIVE-15333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15333: Attachment: HIVE-15333.4.patch > Add a FetchTask to REPL DUMP plan for reading dump uri, last repl id as > ResultSet > - > > Key: HIVE-15333 > URL: https://issues.apache.org/jira/browse/HIVE-15333 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-15333.1.patch, HIVE-15333.2.patch, > HIVE-15333.3.patch, HIVE-15333.4.patch > > > We're writing the return values to a file, but we don't add FetchTask while > planning. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15383) Add additional info to 'desc function extended' output
[ https://issues.apache.org/jira/browse/HIVE-15383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-15383: Attachment: HIVE-15383.1.patch > Add additional info to 'desc function extended' output > -- > > Key: HIVE-15383 > URL: https://issues.apache.org/jira/browse/HIVE-15383 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Trivial > Attachments: HIVE-15383.1.patch > > > Add additional info to the output to 'desc function extended'. The resources > would be helpful for the user to check which jars are referred. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-15381) don't log the callstack for reduce.xml-doesn't-exist
[
https://issues.apache.org/jira/browse/HIVE-15381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-15381:
---
Assignee: Sergey Shelukhin
> don't log the callstack for reduce.xml-doesn't-exist
>
>
> Key: HIVE-15381
> URL: https://issues.apache.org/jira/browse/HIVE-15381
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Trivial
> Attachments: HIVE-15381.patch
>
>
> Pointless exception in the logs:
> {noformat}
> java.io.FileNotFoundException: File
> file:[snip]/itests/qtest/target/tmp/localscratchdir/bcc7fce3-b9a3-4d5a-bf52-4e3b70ad9fed/hive_2016-12-07_09-53-18_167_8716888773328063866-1/-mr-10002/3fb6d7bd-d8b6-4238-bc08-90b2f0217197/reduce.xml
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:609)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:822)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:599)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:140)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341)
> ~[hadoop-common-2.7.2.jar:?]
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:421)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getReduceWork(Utilities.java:313)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:292)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.HiveOutputFormatImpl.checkOutputSpecs(HiveOutputFormatImpl.java:61)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:268)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_102]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_102]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_102]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_102]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:401)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:151)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2166)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1822)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1510)
> [hive-exec-2.2.
[jira] [Updated] (HIVE-15381) don't log the callstack for reduce.xml-doesn't-exist
[
https://issues.apache.org/jira/browse/HIVE-15381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-15381:
Status: Patch Available (was: Open)
> don't log the callstack for reduce.xml-doesn't-exist
>
>
> Key: HIVE-15381
> URL: https://issues.apache.org/jira/browse/HIVE-15381
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Trivial
> Attachments: HIVE-15381.patch
>
>
> Pointless exception in the logs:
> {noformat}
> java.io.FileNotFoundException: File
> file:[snip]/itests/qtest/target/tmp/localscratchdir/bcc7fce3-b9a3-4d5a-bf52-4e3b70ad9fed/hive_2016-12-07_09-53-18_167_8716888773328063866-1/-mr-10002/3fb6d7bd-d8b6-4238-bc08-90b2f0217197/reduce.xml
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:609)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:822)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:599)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:140)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341)
> ~[hadoop-common-2.7.2.jar:?]
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:421)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getReduceWork(Utilities.java:313)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:292)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.HiveOutputFormatImpl.checkOutputSpecs(HiveOutputFormatImpl.java:61)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:268)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_102]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_102]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_102]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_102]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:401)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:151)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2166)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1822)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1510)
> [hive-exec-2
[jira] [Updated] (HIVE-15381) don't log the callstack for reduce.xml-doesn't-exist
[
https://issues.apache.org/jira/browse/HIVE-15381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-15381:
Attachment: HIVE-15381.patch
[~ashutoshc] can you take a look? one-line patch
> don't log the callstack for reduce.xml-doesn't-exist
>
>
> Key: HIVE-15381
> URL: https://issues.apache.org/jira/browse/HIVE-15381
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Trivial
> Attachments: HIVE-15381.patch
>
>
> Pointless exception in the logs:
> {noformat}
> java.io.FileNotFoundException: File
> file:[snip]/itests/qtest/target/tmp/localscratchdir/bcc7fce3-b9a3-4d5a-bf52-4e3b70ad9fed/hive_2016-12-07_09-53-18_167_8716888773328063866-1/-mr-10002/3fb6d7bd-d8b6-4238-bc08-90b2f0217197/reduce.xml
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:609)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:822)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:599)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:140)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341)
> ~[hadoop-common-2.7.2.jar:?]
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:421)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getReduceWork(Utilities.java:313)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:292)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.HiveOutputFormatImpl.checkOutputSpecs(HiveOutputFormatImpl.java:61)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:268)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_102]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_102]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_102]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_102]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:401)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:151)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2166)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1822)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1510)
> [hive-exec
[jira] [Commented] (HIVE-15367) CTAS with LOCATION should write temp data under location directory rather than database location
[
https://issues.apache.org/jira/browse/HIVE-15367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730152#comment-15730152
]
Sahil Takiar commented on HIVE-15367:
-
[~spena] comments addressed
> CTAS with LOCATION should write temp data under location directory rather
> than database location
>
>
> Key: HIVE-15367
> URL: https://issues.apache.org/jira/browse/HIVE-15367
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-15367.1.patch, HIVE-15367.2.patch,
> HIVE-15367.3.patch, HIVE-15367.4.patch
>
>
> For regular CTAS queries, temp data from a SELECT query will be written to to
> a staging directory under the database location. The code to control this is
> in {{SemanticAnalyzer.java}}
> {code}
> // allocate a temporary output dir on the location of the table
> String tableName = getUnescapedName((ASTNode) ast.getChild(0));
> String[] names = Utilities.getDbTableName(tableName);
> Path location;
> try {
> Warehouse wh = new Warehouse(conf);
> //Use destination table's db location.
> String destTableDb = qb.getTableDesc() != null?
> qb.getTableDesc().getDatabaseName(): null;
> if (destTableDb == null) {
> destTableDb = names[0];
> }
> location = wh.getDatabasePath(db.getDatabase(destTableDb));
> } catch (MetaException e) {
> throw new SemanticException(e);
> }
> {code}
> However, CTAS queries allow specifying a {{LOCATION}} for the new table. Its
> possible for this location to be on a different filesystem than the database
> location. If this happens temp data will be written to the database
> filesystem and will be copied to the table filesystem in {{MoveTask}}.
> This extra copying of data can drastically affect performance. Rather than
> always use the database location as the staging dir for CTAS queries, Hive
> should first check if there is an explicit {{LOCATION}} specified in the CTAS
> query. If there is, staging data should be stored under the {{LOCATION}}
> directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15367) CTAS with LOCATION should write temp data under location directory rather than database location
[
https://issues.apache.org/jira/browse/HIVE-15367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-15367:
Attachment: HIVE-15367.4.patch
> CTAS with LOCATION should write temp data under location directory rather
> than database location
>
>
> Key: HIVE-15367
> URL: https://issues.apache.org/jira/browse/HIVE-15367
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-15367.1.patch, HIVE-15367.2.patch,
> HIVE-15367.3.patch, HIVE-15367.4.patch
>
>
> For regular CTAS queries, temp data from a SELECT query will be written to to
> a staging directory under the database location. The code to control this is
> in {{SemanticAnalyzer.java}}
> {code}
> // allocate a temporary output dir on the location of the table
> String tableName = getUnescapedName((ASTNode) ast.getChild(0));
> String[] names = Utilities.getDbTableName(tableName);
> Path location;
> try {
> Warehouse wh = new Warehouse(conf);
> //Use destination table's db location.
> String destTableDb = qb.getTableDesc() != null?
> qb.getTableDesc().getDatabaseName(): null;
> if (destTableDb == null) {
> destTableDb = names[0];
> }
> location = wh.getDatabasePath(db.getDatabase(destTableDb));
> } catch (MetaException e) {
> throw new SemanticException(e);
> }
> {code}
> However, CTAS queries allow specifying a {{LOCATION}} for the new table. Its
> possible for this location to be on a different filesystem than the database
> location. If this happens temp data will be written to the database
> filesystem and will be copied to the table filesystem in {{MoveTask}}.
> This extra copying of data can drastically affect performance. Rather than
> always use the database location as the staging dir for CTAS queries, Hive
> should first check if there is an explicit {{LOCATION}} specified in the CTAS
> query. If there is, staging data should be stored under the {{LOCATION}}
> directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15376) Improve heartbeater scheduling for transactions
[ https://issues.apache.org/jira/browse/HIVE-15376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15376: - Status: Patch Available (was: Open) > Improve heartbeater scheduling for transactions > --- > > Key: HIVE-15376 > URL: https://issues.apache.org/jira/browse/HIVE-15376 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-15376.1.patch, HIVE-15376.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15376) Improve heartbeater scheduling for transactions
[ https://issues.apache.org/jira/browse/HIVE-15376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15376: - Status: Open (was: Patch Available) > Improve heartbeater scheduling for transactions > --- > > Key: HIVE-15376 > URL: https://issues.apache.org/jira/browse/HIVE-15376 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-15376.1.patch, HIVE-15376.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15376) Improve heartbeater scheduling for transactions
[ https://issues.apache.org/jira/browse/HIVE-15376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15376: - Attachment: HIVE-15376.2.patch > Improve heartbeater scheduling for transactions > --- > > Key: HIVE-15376 > URL: https://issues.apache.org/jira/browse/HIVE-15376 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-15376.1.patch, HIVE-15376.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15339) Prefetch column stats for fields needed in FilterSelectivityEstimator
[
https://issues.apache.org/jira/browse/HIVE-15339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730100#comment-15730100
]
Hive QA commented on HIVE-15339:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12842116/HIVE-15339.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 765 failed/errored test(s), 6738 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_custom_key2]
(batchId=217)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_custom_key]
(batchId=217)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_joins]
(batchId=217)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=217)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=217)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
(batchId=217)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_dynamic_partitions]
(batchId=229)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_table]
(batchId=229)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_directory]
(batchId=229)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions]
(batchId=229)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_table]
(batchId=229)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[write_final_output_blobstore]
(batchId=229)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=39)
org.apache.hadoop.
[jira] [Commented] (HIVE-15368) consider optimizing Utilities::handleMmTableFinalPath
[
https://issues.apache.org/jira/browse/HIVE-15368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730097#comment-15730097
]
Sergey Shelukhin commented on HIVE-15368:
-
pool.shutdown/shutdownNow seems to be called twice (before the future.get-s and
in finally)
I am not sure the change is valid... it seems like we don't clean up MM
directories at all now if we found all committed files in the recursive variant
of get... am I missing something? we still need to look at files that are NOT
committed, and delete them.
Also, the logic is now kind of non-transparent. globStatus path returns
directories, but the recursive path actually modifies the committed set on the
fly and then directories are ignored. Perhaps the paths should be separated on
higher level than get... method.
> consider optimizing Utilities::handleMmTableFinalPath
> -
>
> Key: HIVE-15368
> URL: https://issues.apache.org/jira/browse/HIVE-15368
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: hive-14535
>Reporter: Rajesh Balamohan
> Attachments: HIVE-15368.branch.14535.1.patch
>
>
> Branch: hive-14535
> https://github.com/apache/hive/blob/hive-14535/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L4049
> When running "insert overwrite...on partitioned table" with 2000+ partitions,
> good amount of time (~245 seconds) was spent in iterating every mmDirectory
> entry and checking its file listings in S3. Creating this jira to consider
> optimizing this codepath, as information from {{getMmDirectoryCandidates}}
> could be used in terms of reducing the number of times S3 needs to be
> contacted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15066) push down projection is mismatched in HiveInputFormat
[ https://issues.apache.org/jira/browse/HIVE-15066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730091#comment-15730091 ] Prasanth Jayachandran commented on HIVE-15066: -- I think this will again introduce the perf issue that [~rajesh.balamohan]'s patch fixed in HIVE-7052. Since we can't catch perf regression easily I think we should be cautious that this doesn't introduce the perf issue again. > push down projection is mismatched in HiveInputFormat > - > > Key: HIVE-15066 > URL: https://issues.apache.org/jira/browse/HIVE-15066 > Project: Hive > Issue Type: Bug > Components: File Formats >Affects Versions: 2.1.0 >Reporter: Jeongdae Kim >Assignee: Jeongdae Kim > Attachments: HIVE-15066.1.patch > > > if HiveInputFormat.getSplits() is called with more than one table, > inputFormatClass.getSplits() for the first table will be called with read > column information for the second table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15066) push down projection is mismatched in HiveInputFormat
[ https://issues.apache.org/jira/browse/HIVE-15066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730076#comment-15730076 ] Sergey Shelukhin commented on HIVE-15066: - [~prasanth_j] should this be committed? I can commit if you think that is ok > push down projection is mismatched in HiveInputFormat > - > > Key: HIVE-15066 > URL: https://issues.apache.org/jira/browse/HIVE-15066 > Project: Hive > Issue Type: Bug > Components: File Formats >Affects Versions: 2.1.0 >Reporter: Jeongdae Kim >Assignee: Jeongdae Kim > Attachments: HIVE-15066.1.patch > > > if HiveInputFormat.getSplits() is called with more than one table, > inputFormatClass.getSplits() for the first table will be called with read > column information for the second table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15112) Implement Parquet vectorization reader for Struct type
[
https://issues.apache.org/jira/browse/HIVE-15112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730029#comment-15730029
]
Hive QA commented on HIVE-15112:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841963/HIVE-15112.1.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10793 tests
executed
*Failed tests:*
{noformat}
TestVectorizedColumnReaderBase - did not produce a TEST-*.xml file (likely
timed out) (batchId=249)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision]
(batchId=150)
org.apache.hadoop.hive.ql.io.parquet.TestVectorizedColumnReader.decimalRead
(batchId=249)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2467/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2467/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2467/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841963 - PreCommit-HIVE-Build
> Implement Parquet vectorization reader for Struct type
> --
>
> Key: HIVE-15112
> URL: https://issues.apache.org/jira/browse/HIVE-15112
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ferdinand Xu
>Assignee: Ferdinand Xu
> Attachments: HIVE-15112.1.patch, HIVE-15112.patch
>
>
> Like HIVE-14815, we need support Parquet vectorized reader for struct type.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14900) fix entry for hive.exec.max.dynamic.partitions in config whitelist for sql std auth
[
https://issues.apache.org/jira/browse/HIVE-14900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729911#comment-15729911
]
Hive QA commented on HIVE-14900:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12842102/HIVE-14900.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 10730 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=143)
[vectorized_rcfile_columnar.q,vector_elt.q,explainuser_1.q,multi_insert.q,tez_dml.q,vector_bround.q,schema_evol_orc_acid_table.q,vector_when_case_null.q,orc_ppd_schema_evol_1b.q,vector_join30.q,vectorization_11.q,cte_3.q,update_tmp_table.q,vector_decimal_cast.q,groupby_grouping_id2.q,vector_decimal_round.q,tez_smb_empty.q,orc_merge6.q,vector_decimal_trailing.q,cte_5.q,tez_union.q,cbo_rp_subq_not_in.q,vector_decimal_2.q,columnStatsUpdateForStatsOptimizer_1.q,vector_outer_join3.q,schema_evol_text_vec_part_all_complex.q,tez_dynpart_hashjoin_2.q,auto_sortmerge_join_12.q,offset_limit.q,tez_union_multiinsert.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=122)
[groupby_complex_types.q,multigroupby_singlemr.q,mapjoin_decimal.q,groupby7.q,join5.q,bucketmapjoin_negative2.q,vectorization_div0.q,union_script.q,add_part_multiple.q,limit_pushdown.q,union_remove_17.q,uniquejoin.q,metadata_only_queries_with_filters.q,union25.q,load_dyn_part13.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=92)
org.apache.hadoop.hive.metastore.TestRemoteUGIHiveMetaStoreIpAddress.testIpAddress
(batchId=198)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2466/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2466/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2466/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12842102 - PreCommit-HIVE-Build
> fix entry for hive.exec.max.dynamic.partitions in config whitelist for sql
> std auth
> ---
>
> Key: HIVE-14900
> URL: https://issues.apache.org/jira/browse/HIVE-14900
> Project: Hive
> Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-14900.1.patch, HIVE-14900.1.patch
>
>
> HiveConf.java has -
> {code}
> static final String [] sqlStdAuthSafeVarNameRegexes = new String [] {
> ...
> "hive\\.exec\\..*\\.dynamic\\.partitions\\..*",
> {code}
> The regex doesn't work for hive.exec.max.dynamic.partitions as there is a "."
> at the end.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15373) DbNotificationListener should use thread-local RawStore
[ https://issues.apache.org/jira/browse/HIVE-15373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729870#comment-15729870 ] Alexander Kolbasov commented on HIVE-15373: --- I see, makes sense. Is there a specific reason not to create persistence manager per transaction which then should take care of nested transactions (which should create their own persistence manager and open transaction within its context)? > DbNotificationListener should use thread-local RawStore > --- > > Key: HIVE-15373 > URL: https://issues.apache.org/jira/browse/HIVE-15373 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Alexander Kolbasov > > ObjectStore.java has several important calls which are not thread-safe: > * openTransaction() > * commitTransaction() > * rollbackTransaction() > These should be made thread-safe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15359) skip.footer.line.count doesnt work properly for certain situations
[
https://issues.apache.org/jira/browse/HIVE-15359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729820#comment-15729820
]
Aihua Xu commented on HIVE-15359:
-
The patch looks good.
It would be good if we can add some comments to
{noformat}
if (preReader != null && preReader instanceof CombineHiveRecordReader
73 && ((CombineHiveRecordReader)preReader).getFooterBuffer() !=
null) {
74if (partition != 0 && hsplit.getPaths()[partition
-1].equals(hsplit.getPaths()[partition])) //the split is from same file as
previous split.
75
this.setFooterBuffer(((CombineHiveRecordReader)preReader).getFooterBuffer());
{noformat}
+1.
> skip.footer.line.count doesnt work properly for certain situations
> --
>
> Key: HIVE-15359
> URL: https://issues.apache.org/jira/browse/HIVE-15359
> Project: Hive
> Issue Type: Bug
> Components: Reader
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-15359.1.patch
>
>
> This issue's reproduce is very like HIVE-12718 , but the data file is larger
> than 128M . In this case, even make sure only one mapper is used, the footer
> is still wrongly skipped.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-15361) INSERT dynamic partition on S3 fails with a MoveTask failure
[
https://issues.apache.org/jira/browse/HIVE-15361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña resolved HIVE-15361.
Resolution: Fixed
Fix Version/s: 2.2.0
Thanks [~yalovyyi] [~mohitsabharwal] for the review. I committed to master.
> INSERT dynamic partition on S3 fails with a MoveTask failure
>
>
> Key: HIVE-15361
> URL: https://issues.apache.org/jira/browse/HIVE-15361
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
>Priority: Critical
> Fix For: 2.2.0
>
> Attachments: HIVE-15361.1.patch, HIVE-15361.2.patch,
> HIVE-15361.3.patch, HIVE-15361.4.patch, HIVE-15361.5.patch
>
>
> The following failure is due to the patch that merges two MoveTask found on
> the ConditionalTask (See HIVE-15114)
> {panel:title=Repro steps}
> CREATE EXTERNAL TABLE external_1k0jU (name STRING, age INT) PARTITIONED BY
> (country STRING, state STRING);
> ALTER TABLE external_1k0jU ADD PARTITION (COUNTRY='USA', STATE='CA');
> INSERT INTO external_1k0jU PARTITION (country='USA', state='CA') values
> ('John Doe', 23), ('Jane Doe', 22);
> CREATE EXTERNAL TABLE external_P3kiT (name STRING, age INT) PARTITIONED BY
> (country STRING, state STRING) location 's3a://hive-on-s3/foo/bar/';
> set hive.exec.dynamic.partition.mode=nonstrict;
> INSERT INTO TABLE external_P3kiT PARTITION (country, state) SELECT * FROM
> external_1k0jU;
> {panel}
> {panel:title=Error & stack trace}
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask
> INFO : MapReduce Jobs Launched:
> INFO : Stage-Stage-1: Map: 1 Cumulative CPU: 3.64 sec HDFS Read: 3656
> HDFS Write: 99 SUCCESS
> INFO : Total MapReduce CPU Time Spent: 3 seconds 640 msec
> INFO : Completed executing
> command(queryId=hive_20161201113939_d64df5d7-a4c4-4885-846f-10f0223fcf4c);
> Time taken: 23.227 seconds
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask (state=08S01,code=1)
> INFO : Loading data to table default.external_p3kit partition (country=null,
> state=null) from
> s3a://hive-on-s3/foo/bar/.hive-staging_hive_2016-12-01_11-39-48_741_6724911837889341086-13/-ext-10002
> {code}
> ERROR : Failed with exception MetaException(message:Invalid partition key &
> values; keys [country, state, ], values [])
> org.apache.hadoop.hive.ql.metadata.HiveException:
> MetaException(message:Invalid partition key & values; keys [country, state,
> ], values [])
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:1902)
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:1834)
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1428)
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1388)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:453)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1976)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1689)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1421)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1205)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1200)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:293)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:306)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: MetaException(message:Invalid partition key & values; keys
> [country, state, ], values [])
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partition_with_auth_result$get_partition_with_auth_resultStandardScheme.read(ThriftHiveMetastore.java:65142)
[jira] [Updated] (HIVE-15279) map join dummy operators are not set up correctly in certain cases with merge join
[ https://issues.apache.org/jira/browse/HIVE-15279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-15279: Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to master > map join dummy operators are not set up correctly in certain cases with merge > join > -- > > Key: HIVE-15279 > URL: https://issues.apache.org/jira/browse/HIVE-15279 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.2.0 > > Attachments: HIVE-15279.01.patch, HIVE-15279.02.patch, > HIVE-15279.03.patch, HIVE-15279.patch > > > As a result, MapJoin is not initialized and there's NPE later. > Tez-specific. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15379) Get the key of hive.metastore.* values should be consistent with Hive Metastore Server.
[
https://issues.apache.org/jira/browse/HIVE-15379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729794#comment-15729794
]
Sergey Shelukhin commented on HIVE-15379:
-
This looks like a setup-specific issue. CDH is not part of Hive ;)
> Get the key of hive.metastore.* values should be consistent with Hive
> Metastore Server.
>
>
> Key: HIVE-15379
> URL: https://issues.apache.org/jira/browse/HIVE-15379
> Project: Hive
> Issue Type: Bug
> Components: Beeline, CLI
>Affects Versions: 1.1.0
>Reporter: Yuming Wang
>Assignee: Fei Hui
>Priority: Minor
>
> Hive Metastore Server's {{hive.metastore.try.direct.sql=false}} when using
> Cloudera Manager. But cli or beeline read the client configure and return
> true. It is meaningless.
> {code}
> hive> set hive.metastore.try.direct.sql;
> hive.metastore.try.direct.sql=true
> hive>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-15379) Get the key of hive.metastore.* values should be consistent with Hive Metastore Server.
[
https://issues.apache.org/jira/browse/HIVE-15379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729794#comment-15729794
]
Sergey Shelukhin edited comment on HIVE-15379 at 12/7/16 8:13 PM:
--
This looks like a setup-specific issue. Cloudera Manager is not part of Hive ;)
was (Author: sershe):
This looks like a setup-specific issue. CDH is not part of Hive ;)
> Get the key of hive.metastore.* values should be consistent with Hive
> Metastore Server.
>
>
> Key: HIVE-15379
> URL: https://issues.apache.org/jira/browse/HIVE-15379
> Project: Hive
> Issue Type: Bug
> Components: Beeline, CLI
>Affects Versions: 1.1.0
>Reporter: Yuming Wang
>Assignee: Fei Hui
>Priority: Minor
>
> Hive Metastore Server's {{hive.metastore.try.direct.sql=false}} when using
> Cloudera Manager. But cli or beeline read the client configure and return
> true. It is meaningless.
> {code}
> hive> set hive.metastore.try.direct.sql;
> hive.metastore.try.direct.sql=true
> hive>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15279) map join dummy operators are not set up correctly in certain cases with merge join
[ https://issues.apache.org/jira/browse/HIVE-15279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729773#comment-15729773 ] Sergey Shelukhin commented on HIVE-15279: - Filed HIVE-15382 for one test (failed a lot in other JIRAs too, so unrelated to this) > map join dummy operators are not set up correctly in certain cases with merge > join > -- > > Key: HIVE-15279 > URL: https://issues.apache.org/jira/browse/HIVE-15279 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-15279.01.patch, HIVE-15279.02.patch, > HIVE-15279.03.patch, HIVE-15279.patch > > > As a result, MapJoin is not initialized and there's NPE later. > Tez-specific. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15381) don't log the callstack for reduce.xml-doesn't-exist
[
https://issues.apache.org/jira/browse/HIVE-15381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-15381:
Summary: don't log the callstack for reduce.xml-doesn't-exist (was: don't
log the callstack for reduce.xml doesn't exist)
> don't log the callstack for reduce.xml-doesn't-exist
>
>
> Key: HIVE-15381
> URL: https://issues.apache.org/jira/browse/HIVE-15381
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Trivial
>
> Pointless exception in the logs:
> {noformat}
> java.io.FileNotFoundException: File
> file:/home/hiveptest/104.197.81.77-hiveptest-1/apache-github-source-source/itests/qtest/target/tmp/localscratchdir/bcc7fce3-b9a3-4d5a-bf52-4e3b70ad9fed/hive_2016-12-07_09-53-18_167_8716888773328063866-1/-mr-10002/3fb6d7bd-d8b6-4238-bc08-90b2f0217197/reduce.xml
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:609)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:822)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:599)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:140)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341)
> ~[hadoop-common-2.7.2.jar:?]
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
> ~[hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:421)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getReduceWork(Utilities.java:313)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:292)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.HiveOutputFormatImpl.checkOutputSpecs(HiveOutputFormatImpl.java:61)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:268)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_102]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_102]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_102]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_102]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561)
> [hadoop-mapreduce-client-core-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:401)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:151)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2166)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1822)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at org.apache
[jira] [Updated] (HIVE-15381) don't log the callstack for reduce.xml-doesn't-exist
[
https://issues.apache.org/jira/browse/HIVE-15381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-15381:
Description:
Pointless exception in the logs:
{noformat}
java.io.FileNotFoundException: File
file:[snip]/itests/qtest/target/tmp/localscratchdir/bcc7fce3-b9a3-4d5a-bf52-4e3b70ad9fed/hive_2016-12-07_09-53-18_167_8716888773328063866-1/-mr-10002/3fb6d7bd-d8b6-4238-bc08-90b2f0217197/reduce.xml
does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:609)
~[hadoop-common-2.7.2.jar:?]
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:822)
~[hadoop-common-2.7.2.jar:?]
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:599)
~[hadoop-common-2.7.2.jar:?]
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
~[hadoop-common-2.7.2.jar:?]
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:140)
~[hadoop-common-2.7.2.jar:?]
at
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341)
~[hadoop-common-2.7.2.jar:?]
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
~[hadoop-common-2.7.2.jar:?]
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:421)
[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.Utilities.getReduceWork(Utilities.java:313)
[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:292)
[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.io.HiveOutputFormatImpl.checkOutputSpecs(HiveOutputFormatImpl.java:61)
[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:268)
[hadoop-mapreduce-client-core-2.7.2.jar:?]
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139)
[hadoop-mapreduce-client-core-2.7.2.jar:?]
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
[hadoop-mapreduce-client-core-2.7.2.jar:?]
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
[hadoop-mapreduce-client-core-2.7.2.jar:?]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_102]
at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_102]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
[hadoop-common-2.7.2.jar:?]
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
[hadoop-mapreduce-client-core-2.7.2.jar:?]
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)
[hadoop-mapreduce-client-core-2.7.2.jar:?]
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)
[hadoop-mapreduce-client-core-2.7.2.jar:?]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_102]
at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_102]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
[hadoop-common-2.7.2.jar:?]
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570)
[hadoop-mapreduce-client-core-2.7.2.jar:?]
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561)
[hadoop-mapreduce-client-core-2.7.2.jar:?]
at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:401)
[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:151)
[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2166)
[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1822)
[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1510)
[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1221)
[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1211)
[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
[hive-cli-2.2.0-SNAPSHOT.jar:?]
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
[hive-cli-2.2.0-SNAPSHOT.jar:?]
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.
[jira] [Commented] (HIVE-15376) Improve heartbeater scheduling for transactions
[
https://issues.apache.org/jira/browse/HIVE-15376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729742#comment-15729742
]
Hive QA commented on HIVE-15376:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12842095/HIVE-15376.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 341 failed/errored test(s), 10717 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=143)
[vectorized_rcfile_columnar.q,vector_elt.q,explainuser_1.q,multi_insert.q,tez_dml.q,vector_bround.q,schema_evol_orc_acid_table.q,vector_when_case_null.q,orc_ppd_schema_evol_1b.q,vector_join30.q,vectorization_11.q,cte_3.q,update_tmp_table.q,vector_decimal_cast.q,groupby_grouping_id2.q,vector_decimal_round.q,tez_smb_empty.q,orc_merge6.q,vector_decimal_trailing.q,cte_5.q,tez_union.q,cbo_rp_subq_not_in.q,vector_decimal_2.q,columnStatsUpdateForStatsOptimizer_1.q,vector_outer_join3.q,schema_evol_text_vec_part_all_complex.q,tez_dynpart_hashjoin_2.q,auto_sortmerge_join_12.q,offset_limit.q,tez_union_multiinsert.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=122)
[groupby_complex_types.q,multigroupby_singlemr.q,mapjoin_decimal.q,groupby7.q,join5.q,bucketmapjoin_negative2.q,vectorization_div0.q,union_script.q,add_part_multiple.q,limit_pushdown.q,union_remove_17.q,uniquejoin.q,metadata_only_queries_with_filters.q,union25.q,load_dyn_part13.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_join] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_vectorization]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_vectorization_partition]
(batchId=65)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_vectorization_project]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_delete]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_delete_own_table]
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_update]
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_update_own_table]
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_abort]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_compact1]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_compact2]
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_compact3]
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_ddl1]
(batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_query1]
(batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_query2]
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_query3]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_query4]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_query5]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_all_non_partitioned]
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_all_partitioned]
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_orig_table]
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_tmp_table]
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_where_no_match]
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_where_non_partitioned]
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_where_partitioned]
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_whole_partition]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization_acid2]
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_acid_dynamic_partition]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_acid_not_bucketed]
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_nonacid_from_acid]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_orig_table]
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_update_delete]
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values
[jira] [Updated] (HIVE-15296) AM may lose task failures and not reschedule when scheduling to LLAP
[
https://issues.apache.org/jira/browse/HIVE-15296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-15296:
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Committed to master. Thanks for the review!
> AM may lose task failures and not reschedule when scheduling to LLAP
>
>
> Key: HIVE-15296
> URL: https://issues.apache.org/jira/browse/HIVE-15296
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 2.2.0
>
> Attachments: HIVE-15296.01.patch, HIVE-15296.patch, HIVE-15296.patch
>
>
> First attempt and failure detection:
> {noformat}
> 2016-11-18 20:20:01,980 [INFO] [TaskSchedulerEventHandlerThread]
> |tezplugins.LlapTaskSchedulerService|: Received allocateRequest.
> task=attempt_1478967587833_2622_1_06_31_0, priority=65,
> capability=, hosts=[3n01]
> 2016-11-18 20:20:01,982 [INFO] [LlapScheduler]
> |tezplugins.LlapTaskSchedulerService|: Assigned task
> TaskInfo{task=attempt_1478967587833_2622_1_06_31_0, priority=65,
> startTime=0, containerId=null, assignedInstance=null, uniqueId=55,
> localityDelayTimeout=9223372036854775807} to container
> container_1_2622_01_56 on node=DynamicServiceInstance
> [alive=true, host=3n01:15001 with resources=,
> shufflePort=15551, servicesAddress=http://3n01:15002, mgmtPort=15004]
> 2016-11-18 20:20:01,982 [INFO] [LlapScheduler]
> |tezplugins.LlapTaskSchedulerService|: ScheduleResult for Task:
> TaskInfo{task=attempt_1478967587833_2622_1_06_31_0, priority=65,
> startTime=10550817928, containerId=container_1_2622_01_56,
> assignedInstance=DynamicServiceInstance [alive=true, host=3n01:15001 with
> resources=, shufflePort=15551,
> servicesAddress=http://3n01:15002, mgmtPort=15004], uniqueId=55,
> localityDelayTimeout=9223372036854775807} = SCHEDULED
> 2016-11-18 20:20:03,427 [INFO] [Dispatcher thread {Central}]
> |impl.TaskAttemptImpl|: TaskAttempt:
> [attempt_1478967587833_2622_1_06_31_0] started. Is using containerId:
> [container_1_2622_01_56] on NM: [3n01:15001]
> 2016-11-18 20:20:03,427 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1478967587833_2622_1][Event:TASK_ATTEMPT_STARTED]:
> vertexName=Map 1, taskAttemptId=attempt_1478967587833_2622_1_06_31_0,
> startTime=1479500403427, containerId=container_1_2622_01_56,
> nodeId=3n01:15001
> 2016-11-18 20:20:03,430 [INFO] [TaskCommunicator # 1]
> |tezplugins.LlapTaskCommunicator|: Successfully launched task:
> attempt_1478967587833_2622_1_06_31_0
> 2016-11-18 20:20:03,434 [INFO] [IPC Server handler 11 on 43092]
> |impl.TaskImpl|: TaskAttempt:attempt_1478967587833_2622_1_06_31_0 sent
> events: (0-1).
> 2016-11-18 20:20:03,434 [INFO] [IPC Server handler 11 on 43092]
> |impl.VertexImpl|: Sending attempt_1478967587833_2622_1_06_31_0 24 events
> [0,24) total 24 vertex_1478967587833_2622_1_06 [Map 1]
> 2016-11-18 20:25:43,249 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1478967587833_2622_1][Event:TASK_ATTEMPT_FINISHED]:
> vertexName=Map 1, taskAttemptId=attempt_1478967587833_2622_1_06_31_0,
> creationTime=1479500401929, allocationTime=1479500403426,
> startTime=1479500403427, finishTime=1479500743249, timeTaken=339822,
> status=FAILED, taskFailureType=NON_FATAL, errorEnum=TASK_HEARTBEAT_ERROR,
> diagnostics=AttemptID:attempt_1478967587833_2622_1_06_31_0 Timed out
> after 300 secs, nodeHttpAddress=http://3n01:15002, counters=Counters: 1,
> org.apache.tez.common.counters.DAGCounter, DATA_LOCAL_TASKS=1
> 2016-11-18 20:25:43,255 [INFO] [TaskSchedulerEventHandlerThread]
> |tezplugins.LlapTaskSchedulerService|: Processing de-allocate request for
> task=attempt_1478967587833_2622_1_06_31_0, state=ASSIGNED, endReason=OTHER
> 2016-11-18 20:25:43,259 [INFO] [Dispatcher thread {Central}]
> |node.AMNodeImpl|: Attempt failed on node: 3n01:15001 TA:
> attempt_1478967587833_2622_1_06_31_0 failed: true container:
> container_1_2622_01_56 numFailedTAs: 7
> 2016-11-18 20:25:43,262 [INFO] [Dispatcher thread {Central}]
> |impl.VertexImpl|: Source task attempt completed for vertex:
> vertex_1478967587833_2622_1_07 [Reducer 2] attempt:
> attempt_1478967587833_2622_1_06_31_0 with state: FAILED vertexState:
> RUNNING
> {noformat}
> Second attempt:
> {noformat}
> 2016-11-18 20:25:43,267 [INFO] [TaskSchedulerEventHandlerThread]
> |tezplugins.LlapTaskSchedulerService|: Received allocateRequest.
> task=attempt_1478967587833_2622_1_06_31_1, priority=64,
> capability=
[jira] [Commented] (HIVE-15149) Add additional information to ATSHook for Tez UI
[ https://issues.apache.org/jira/browse/HIVE-15149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729639#comment-15729639 ] Sergey Shelukhin commented on HIVE-15149: - +1 assuming that if we query partitions, the table entities would still be in the list. I thought only partitions would be, but it seems like it should be ok. > Add additional information to ATSHook for Tez UI > > > Key: HIVE-15149 > URL: https://issues.apache.org/jira/browse/HIVE-15149 > Project: Hive > Issue Type: Improvement > Components: Hooks >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-15149.1.patch, HIVE-15149.2.patch, > HIVE-15149.3.patch, HIVE-15149.4.patch, HIVE-15149.5.patch > > > Additional query details wanted for TEZ-3530. The additional details > discussed include the following: > Publish the following info ( in addition to existing bits published today): > Application Id to which the query was submitted (primary filter) > DAG Id (primary filter) > Hive query name (primary filter) > Hive Configs (everything a set command would provide except for sensitive > credential info) > Potentially publish source of config i.e. set in hive query script vs > hive-site.xml, etc. > Which HiveServer2 the query was submitted to > *Which IP/host the query was submitted from - not sure what filter support > will be available. > Which execution mode the query is running in (primary filter) > What submission mode was used (cli/beeline/jdbc, etc) > User info ( running as, actual end user, etc) - not sure if already present > Perf logger events. The data published should be able to create a timeline > view of the query i.e. actual submission time, query compile timestamps, > execution timestamps, post-exec data moves, etc. > Explain plan with enough details for visualizing. > Databases and tables being queried (primary filter) > Yarn queue info (primary filter) > Caller context (primary filter) > Original source i.e. submitter > Thread info in HS2 if needed ( I believe Vikram may have added this earlier ) > Query time taken (with filter support ) > Additional context info e.g. llap instance name and appId if required. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15368) consider optimizing Utilities::handleMmTableFinalPath
[
https://issues.apache.org/jira/browse/HIVE-15368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729614#comment-15729614
]
Sergey Shelukhin commented on HIVE-15368:
-
Will look later today. Tests are not set up. Can you try running all the tests
that start with mm_ on mini-llap, locally?
> consider optimizing Utilities::handleMmTableFinalPath
> -
>
> Key: HIVE-15368
> URL: https://issues.apache.org/jira/browse/HIVE-15368
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: hive-14535
>Reporter: Rajesh Balamohan
> Attachments: HIVE-15368.branch.14535.1.patch
>
>
> Branch: hive-14535
> https://github.com/apache/hive/blob/hive-14535/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L4049
> When running "insert overwrite...on partitioned table" with 2000+ partitions,
> good amount of time (~245 seconds) was spent in iterating every mmDirectory
> entry and checking its file listings in S3. Creating this jira to consider
> optimizing this codepath, as information from {{getMmDirectoryCandidates}}
> could be used in terms of reducing the number of times S3 needs to be
> contacted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10937) LLAP: make ObjectCache for plans work properly in the daemon
[ https://issues.apache.org/jira/browse/HIVE-10937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729593#comment-15729593 ] Sergey Shelukhin commented on HIVE-10937: - Happy birthday patch! > LLAP: make ObjectCache for plans work properly in the daemon > > > Key: HIVE-10937 > URL: https://issues.apache.org/jira/browse/HIVE-10937 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-10937.01.patch, HIVE-10937.02.patch, > HIVE-10937.03.patch, HIVE-10937.04.patch, HIVE-10937.05.patch, > HIVE-10937.patch > > > There's perf hit otherwise, esp. when stupid planner creates 1009 reducers of > 4Mb each. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15373) DbNotificationListener should use thread-local RawStore
[ https://issues.apache.org/jira/browse/HIVE-15373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729574#comment-15729574 ] Sergey Shelukhin commented on HIVE-15373: - The state is thread-specific. Each thread calls open, then does stuff (potentially with nested open/close), then calls commit/rollback ... if 2 threads happen to be using the same rawstore, we don't want their open/close and their operations to mix. So to make it thread-safe, the entire transaction needs to be isolated, or the txn state would need to be separated into some sort of context object > DbNotificationListener should use thread-local RawStore > --- > > Key: HIVE-15373 > URL: https://issues.apache.org/jira/browse/HIVE-15373 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Alexander Kolbasov > > ObjectStore.java has several important calls which are not thread-safe: > * openTransaction() > * commitTransaction() > * rollbackTransaction() > These should be made thread-safe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-15373) DbNotificationListener should use thread-local RawStore
[ https://issues.apache.org/jira/browse/HIVE-15373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729574#comment-15729574 ] Sergey Shelukhin edited comment on HIVE-15373 at 12/7/16 6:52 PM: -- The state is thread-specific. Each thread calls open, then does stuff (potentially with nested open/close), then calls commit/rollback ... if 2 threads happen to be using the same rawstore, we don't want their open/close and their operations to mix. So to make it thread-safe, the entire top-level transaction needs to be isolated, or the txn state would need to be separated into some sort of context object was (Author: sershe): The state is thread-specific. Each thread calls open, then does stuff (potentially with nested open/close), then calls commit/rollback ... if 2 threads happen to be using the same rawstore, we don't want their open/close and their operations to mix. So to make it thread-safe, the entire transaction needs to be isolated, or the txn state would need to be separated into some sort of context object > DbNotificationListener should use thread-local RawStore > --- > > Key: HIVE-15373 > URL: https://issues.apache.org/jira/browse/HIVE-15373 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Alexander Kolbasov > > ObjectStore.java has several important calls which are not thread-safe: > * openTransaction() > * commitTransaction() > * rollbackTransaction() > These should be made thread-safe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-15377) Driver::acquireWriteIds can be expensive trying to get details from MS
[
https://issues.apache.org/jira/browse/HIVE-15377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727406#comment-15727406
]
Sergey Shelukhin edited comment on HIVE-15377 at 12/7/16 6:47 PM:
--
[~rajesh.balamohan] this is going to be replaced by ACID transaction IDs after
ACID merge. cc [~wzheng]
Also, in preparation of ACID merge, advancing watermark as not implemented
[update] so the fetching of many write IDs may be even slower.
Is the slowness ORM-related? ACID doesn't use ORM so that should hopefully be
faster.
was (Author: sershe):
[~rajesh.balamohan] this is going to be replaced by ACID transaction IDs after
ACID merge. cc [~wzheng]
Also, in preparation of ACID merge, advancing watermark as not implemented.
Is the slowness ORM-related? ACID doesn't use ORM so that should hopefully be
faster.
> Driver::acquireWriteIds can be expensive trying to get details from MS
> --
>
> Key: HIVE-15377
> URL: https://issues.apache.org/jira/browse/HIVE-15377
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: hive-14535
>Reporter: Rajesh Balamohan
>Assignee: Wei Zheng
>
> Branch: hive-14535
> Populated data in tpc-ds web_returns table. Select queries take longer time
> trying to acquire writeIds from MS.
> {noformat}
> hive> select * from web_returns_hive_commit limit 10;
> select * from web_returns_hive_commit limit 10
> ...
>
> Time taken: 52.494 seconds, Fetched: 10 row(s)
> {noformat}
> Without commit feature, same query would execute in ~6 seconds.
> Attaching the stacktrace for reference:
> {noformat}
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at
> org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:143)
> at
> org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:112)
> at
> org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:70)
> at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:283)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1799)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:200)
> - locked <0x000223192988> (a
> org.postgresql.core.v3.QueryExecutorImpl)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:424)
> at
> org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:161)
> at
> org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:114)
> at
> com.jolbox.bonecp.PreparedStatementHandle.executeQuery(PreparedStatementHandle.java:174)
> at
> org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeQuery(ParamLoggingPreparedStatement.java:375)
> at
> org.datanucleus.store.rdbms.SQLController.executeStatementQuery(SQLController.java:552)
> at
> org.datanucleus.store.rdbms.scostore.ElementContainerStore.getSize(ElementContainerStore.java:660)
> at
> org.datanucleus.store.rdbms.scostore.ElementContainerStore.size(ElementContainerStore.java:606)
> at org.datanucleus.store.types.wrappers.backed.List.size(List.java:542)
> - locked <0x00078e8b4f60> (a
> org.datanucleus.store.types.wrappers.backed.List)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToOrders(ObjectStore.java:1665)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1710)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1725)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToTable(ObjectStore.java:1578)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1274)
> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
> at com.sun.proxy.$Proxy47.getTable(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_valid_write_ids(HiveMetaStore.java:6874)
> at sun.reflect.GeneratedMethodAccessor82.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(Retrying
[jira] [Commented] (HIVE-15279) map join dummy operators are not set up correctly in certain cases with merge join
[
https://issues.apache.org/jira/browse/HIVE-15279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729536#comment-15729536
]
Hive QA commented on HIVE-15279:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12842075/HIVE-15279.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 10760 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=102)
[skewjoinopt19.q,order.q,join_merge_multi_expressions.q,skewjoinopt10.q,union5.q,insert_into1.q,vectorized_math_funcs.q,vectorization_4.q,vectorization_2.q,skewjoinopt6.q,decimal_1_1.q,join14.q,outer_join_ppr.q,rcfile_bigdata.q,load_dyn_part10.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_bitmap_auto_partitioned]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_4]
(batchId=92)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2464/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2464/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2464/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12842075 - PreCommit-HIVE-Build
> map join dummy operators are not set up correctly in certain cases with merge
> join
> --
>
> Key: HIVE-15279
> URL: https://issues.apache.org/jira/browse/HIVE-15279
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15279.01.patch, HIVE-15279.02.patch,
> HIVE-15279.03.patch, HIVE-15279.patch
>
>
> As a result, MapJoin is not initialized and there's NPE later.
> Tez-specific.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15338) Wrong result from non-vectorized DATEDIFF with scalar parameter of type DATE/TIMESTAMP
[ https://issues.apache.org/jira/browse/HIVE-15338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729442#comment-15729442 ] Matt McCline commented on HIVE-15338: - And is the correct approach to switch to java.sql.Date from java.util.Date? > Wrong result from non-vectorized DATEDIFF with scalar parameter of type > DATE/TIMESTAMP > -- > > Key: HIVE-15338 > URL: https://issues.apache.org/jira/browse/HIVE-15338 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15338.01.patch, HIVE-15338.02.patch > > > Vectorization in vectorized DATEDIFF accidentally treated the scalar > parameter is type DATE (e.g. CURRENT_DATE) as 0. > Current Q file test vectorized_date_funcs.q DOES NOT test the DATE/TIMESTAMP > scalar type case. > And, non-vectorized cases of DATEDIFF are using UTF and returning the wrong > results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11812) datediff sometimes returns incorrect results when called with dates
[
https://issues.apache.org/jira/browse/HIVE-11812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729429#comment-15729429
]
Matt McCline commented on HIVE-11812:
-
Interesting I'm working with Jason on a DATEDIFF fix in
https://issues.apache.org/jira/browse/HIVE-15338 that is probably the same
problem.
> datediff sometimes returns incorrect results when called with dates
> ---
>
> Key: HIVE-11812
> URL: https://issues.apache.org/jira/browse/HIVE-11812
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 2.0.0
>Reporter: Nicholas Brenwald
>Assignee: Chetna Chaudhari
>Priority: Minor
> Attachments: HIVE-11812.1.patch
>
>
> DATEDIFF returns an incorrect result when one of the arguments is a date
> type.
> The Hive Language Manual provides the following signature for datediff:
> {code}
> int datediff(string enddate, string startdate)
> {code}
> I think datediff should either throw an error (if date types are not
> supported), or return the correct result.
> To reproduce, create a table:
> {code}
> create table t (c1 string, c2 date);
> {code}
> Assuming you have a table x containing some data, populate table t with 1 row:
> {code}
> insert into t select '2015-09-15', '2015-09-15' from x limit 1;
> {code}
> Then run the following 12 test queries:
> {code}
> select datediff(c1, '2015-09-14') from t;
> select datediff(c1, '2015-09-15') from t;
> select datediff(c1, '2015-09-16') from t;
> select datediff('2015-09-14', c1) from t;
> select datediff('2015-09-15', c1) from t;
> select datediff('2015-09-16', c1) from t;
> select datediff(c2, '2015-09-14') from t;
> select datediff(c2, '2015-09-15') from t;
> select datediff(c2, '2015-09-16') from t;
> select datediff('2015-09-14', c2) from t;
> select datediff('2015-09-15', c2) from t;
> select datediff('2015-09-16', c2) from t;
> {code}
> The below table summarises the result. All results for column c1 (which is a
> string) are correct, but when using c2 (which is a date), two of the results
> are incorrect.
> || Test || Expected Result || Actual Result || Passed / Failed ||
> |datediff(c1, '2015-09-14')| 1 | 1| Passed |
> |datediff(c1, '2015-09-15')| 0 | 0| Passed |
> |datediff(c1, '2015-09-16') | -1 | -1| Passed |
> |datediff('2015-09-14', c1) | -1 | -1| Passed |
> |datediff('2015-09-15', c1)| 0 | 0| Passed |
> |datediff('2015-09-16', c1)| 1 | 1| Passed |
> |datediff(c2, '2015-09-14')| 1 | 0| {color:red}Failed{color} |
> |datediff(c2, '2015-09-15')| 0 | 0| Passed |
> |datediff(c2, '2015-09-16') | -1 | -1| Passed |
> |datediff('2015-09-14', c2) | -1 | 0 | {color:red}Failed{color} |
> |datediff('2015-09-15', c2)| 0 | 0| Passed |
> |datediff('2015-09-16', c2)| 1 | 1| Passed |
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15377) Driver::acquireWriteIds can be expensive trying to get details from MS
[
https://issues.apache.org/jira/browse/HIVE-15377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729401#comment-15729401
]
Wei Zheng commented on HIVE-15377:
--
Yes this call will not be there with ACID transaction id replacement
> Driver::acquireWriteIds can be expensive trying to get details from MS
> --
>
> Key: HIVE-15377
> URL: https://issues.apache.org/jira/browse/HIVE-15377
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: hive-14535
>Reporter: Rajesh Balamohan
>Assignee: Wei Zheng
>
> Branch: hive-14535
> Populated data in tpc-ds web_returns table. Select queries take longer time
> trying to acquire writeIds from MS.
> {noformat}
> hive> select * from web_returns_hive_commit limit 10;
> select * from web_returns_hive_commit limit 10
> ...
>
> Time taken: 52.494 seconds, Fetched: 10 row(s)
> {noformat}
> Without commit feature, same query would execute in ~6 seconds.
> Attaching the stacktrace for reference:
> {noformat}
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at
> org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:143)
> at
> org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:112)
> at
> org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:70)
> at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:283)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1799)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:200)
> - locked <0x000223192988> (a
> org.postgresql.core.v3.QueryExecutorImpl)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:424)
> at
> org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:161)
> at
> org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:114)
> at
> com.jolbox.bonecp.PreparedStatementHandle.executeQuery(PreparedStatementHandle.java:174)
> at
> org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeQuery(ParamLoggingPreparedStatement.java:375)
> at
> org.datanucleus.store.rdbms.SQLController.executeStatementQuery(SQLController.java:552)
> at
> org.datanucleus.store.rdbms.scostore.ElementContainerStore.getSize(ElementContainerStore.java:660)
> at
> org.datanucleus.store.rdbms.scostore.ElementContainerStore.size(ElementContainerStore.java:606)
> at org.datanucleus.store.types.wrappers.backed.List.size(List.java:542)
> - locked <0x00078e8b4f60> (a
> org.datanucleus.store.types.wrappers.backed.List)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToOrders(ObjectStore.java:1665)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1710)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1725)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToTable(ObjectStore.java:1578)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1274)
> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
> at com.sun.proxy.$Proxy47.getTable(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_valid_write_ids(HiveMetaStore.java:6874)
> at sun.reflect.GeneratedMethodAccessor82.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at com.sun.proxy.$Proxy50.get_valid_write_ids(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getValidWriteIds(HiveMetaStoreClient.java:2480)
> at sun.reflect.GeneratedMethodAccessor81.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(
[jira] [Assigned] (HIVE-15377) Driver::acquireWriteIds can be expensive trying to get details from MS
[
https://issues.apache.org/jira/browse/HIVE-15377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng reassigned HIVE-15377:
Assignee: Wei Zheng
> Driver::acquireWriteIds can be expensive trying to get details from MS
> --
>
> Key: HIVE-15377
> URL: https://issues.apache.org/jira/browse/HIVE-15377
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: hive-14535
>Reporter: Rajesh Balamohan
>Assignee: Wei Zheng
>
> Branch: hive-14535
> Populated data in tpc-ds web_returns table. Select queries take longer time
> trying to acquire writeIds from MS.
> {noformat}
> hive> select * from web_returns_hive_commit limit 10;
> select * from web_returns_hive_commit limit 10
> ...
>
> Time taken: 52.494 seconds, Fetched: 10 row(s)
> {noformat}
> Without commit feature, same query would execute in ~6 seconds.
> Attaching the stacktrace for reference:
> {noformat}
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at
> org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:143)
> at
> org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:112)
> at
> org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:70)
> at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:283)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1799)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:200)
> - locked <0x000223192988> (a
> org.postgresql.core.v3.QueryExecutorImpl)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:424)
> at
> org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:161)
> at
> org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:114)
> at
> com.jolbox.bonecp.PreparedStatementHandle.executeQuery(PreparedStatementHandle.java:174)
> at
> org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeQuery(ParamLoggingPreparedStatement.java:375)
> at
> org.datanucleus.store.rdbms.SQLController.executeStatementQuery(SQLController.java:552)
> at
> org.datanucleus.store.rdbms.scostore.ElementContainerStore.getSize(ElementContainerStore.java:660)
> at
> org.datanucleus.store.rdbms.scostore.ElementContainerStore.size(ElementContainerStore.java:606)
> at org.datanucleus.store.types.wrappers.backed.List.size(List.java:542)
> - locked <0x00078e8b4f60> (a
> org.datanucleus.store.types.wrappers.backed.List)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToOrders(ObjectStore.java:1665)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1710)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1725)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToTable(ObjectStore.java:1578)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1274)
> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
> at com.sun.proxy.$Proxy47.getTable(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_valid_write_ids(HiveMetaStore.java:6874)
> at sun.reflect.GeneratedMethodAccessor82.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at com.sun.proxy.$Proxy50.get_valid_write_ids(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getValidWriteIds(HiveMetaStoreClient.java:2480)
> at sun.reflect.GeneratedMethodAccessor81.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:162)
> at com.sun.proxy.$Proxy51.getValidWriteIds(Unknown Sou
[jira] [Commented] (HIVE-15296) AM may lose task failures and not reschedule when scheduling to LLAP
[
https://issues.apache.org/jira/browse/HIVE-15296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729365#comment-15729365
]
Hive QA commented on HIVE-15296:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12842074/HIVE-15296.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 10741 tests
executed
*Failed tests:*
{noformat}
TestHS2ImpersonationWithRemoteMS - did not produce a TEST-*.xml file (likely
timed out) (batchId=211)
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=158)
[infer_bucket_sort_reducers_power_two.q,list_bucket_dml_10.q,orc_merge9.q,orc_merge6.q,leftsemijoin_mr.q,bucket6.q,bucketmapjoin7.q,uber_reduce.q,empty_dir_in_table.q,vector_outer_join3.q,index_bitmap_auto.q,vector_outer_join2.q,vector_outer_join1.q,orc_merge1.q,orc_merge_diff_fs.q,load_hdfs_file_with_space_in_the_name.q,scriptfile1_win.q,quotedid_smb.q,truncate_column_buckets.q,orc_merge3.q]
TestOperationLoggingAPIWithMr - did not produce a TEST-*.xml file (likely timed
out) (batchId=211)
TestOperationLoggingLayout - did not produce a TEST-*.xml file (likely timed
out) (batchId=211)
TestXSRFFilter - did not produce a TEST-*.xml file (likely timed out)
(batchId=211)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=92)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2463/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2463/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2463/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12842074 - PreCommit-HIVE-Build
> AM may lose task failures and not reschedule when scheduling to LLAP
>
>
> Key: HIVE-15296
> URL: https://issues.apache.org/jira/browse/HIVE-15296
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15296.01.patch, HIVE-15296.patch, HIVE-15296.patch
>
>
> First attempt and failure detection:
> {noformat}
> 2016-11-18 20:20:01,980 [INFO] [TaskSchedulerEventHandlerThread]
> |tezplugins.LlapTaskSchedulerService|: Received allocateRequest.
> task=attempt_1478967587833_2622_1_06_31_0, priority=65,
> capability=, hosts=[3n01]
> 2016-11-18 20:20:01,982 [INFO] [LlapScheduler]
> |tezplugins.LlapTaskSchedulerService|: Assigned task
> TaskInfo{task=attempt_1478967587833_2622_1_06_31_0, priority=65,
> startTime=0, containerId=null, assignedInstance=null, uniqueId=55,
> localityDelayTimeout=9223372036854775807} to container
> container_1_2622_01_56 on node=DynamicServiceInstance
> [alive=true, host=3n01:15001 with resources=,
> shufflePort=15551, servicesAddress=http://3n01:15002, mgmtPort=15004]
> 2016-11-18 20:20:01,982 [INFO] [LlapScheduler]
> |tezplugins.LlapTaskSchedulerService|: ScheduleResult for Task:
> TaskInfo{task=attempt_1478967587833_2622_1_06_31_0, priority=65,
> startTime=10550817928, containerId=container_1_2622_01_56,
> assignedInstance=DynamicServiceInstance [alive=true, host=3n01:15001 with
> resources=, shufflePort=15551,
> servicesAddress=http://3n01:15002, mgmtPort=15004], uniqueId=55,
> localityDelayTimeout=9223372036854775807} = SCHEDULED
> 2016-11-18 20:20:03,427 [INFO] [Dispatcher thread {Central}]
> |impl.TaskAttemptImpl|: TaskAttempt:
> [attempt_1478967587833_2622_1_06_31_0] started. Is using containerId:
> [container_1_2622_01_56] on NM: [3n01:15001]
> 2016-11-18 20:20:03,427 [INFO] [Dispatcher thread {Central}]
> |history.Histo
[jira] [Commented] (HIVE-15367) CTAS with LOCATION should write temp data under location directory rather than database location
[
https://issues.apache.org/jira/browse/HIVE-15367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729367#comment-15729367
]
Sahil Takiar commented on HIVE-15367:
-
Test failures look unrelated and are failing in other Hive QA runs.
[~spena], [~ychena] could you review, RB is here:
https://reviews.apache.org/r/54451/
> CTAS with LOCATION should write temp data under location directory rather
> than database location
>
>
> Key: HIVE-15367
> URL: https://issues.apache.org/jira/browse/HIVE-15367
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-15367.1.patch, HIVE-15367.2.patch,
> HIVE-15367.3.patch
>
>
> For regular CTAS queries, temp data from a SELECT query will be written to to
> a staging directory under the database location. The code to control this is
> in {{SemanticAnalyzer.java}}
> {code}
> // allocate a temporary output dir on the location of the table
> String tableName = getUnescapedName((ASTNode) ast.getChild(0));
> String[] names = Utilities.getDbTableName(tableName);
> Path location;
> try {
> Warehouse wh = new Warehouse(conf);
> //Use destination table's db location.
> String destTableDb = qb.getTableDesc() != null?
> qb.getTableDesc().getDatabaseName(): null;
> if (destTableDb == null) {
> destTableDb = names[0];
> }
> location = wh.getDatabasePath(db.getDatabase(destTableDb));
> } catch (MetaException e) {
> throw new SemanticException(e);
> }
> {code}
> However, CTAS queries allow specifying a {{LOCATION}} for the new table. Its
> possible for this location to be on a different filesystem than the database
> location. If this happens temp data will be written to the database
> filesystem and will be copied to the table filesystem in {{MoveTask}}.
> This extra copying of data can drastically affect performance. Rather than
> always use the database location as the staging dir for CTAS queries, Hive
> should first check if there is an explicit {{LOCATION}} specified in the CTAS
> query. If there is, staging data should be stored under the {{LOCATION}}
> directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15361) INSERT dynamic partition on S3 fails with a MoveTask failure
[
https://issues.apache.org/jira/browse/HIVE-15361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729267#comment-15729267
]
Mohit Sabharwal commented on HIVE-15361:
LGTM, +1
> INSERT dynamic partition on S3 fails with a MoveTask failure
>
>
> Key: HIVE-15361
> URL: https://issues.apache.org/jira/browse/HIVE-15361
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
>Priority: Critical
> Attachments: HIVE-15361.1.patch, HIVE-15361.2.patch,
> HIVE-15361.3.patch, HIVE-15361.4.patch, HIVE-15361.5.patch
>
>
> The following failure is due to the patch that merges two MoveTask found on
> the ConditionalTask (See HIVE-15114)
> {panel:title=Repro steps}
> CREATE EXTERNAL TABLE external_1k0jU (name STRING, age INT) PARTITIONED BY
> (country STRING, state STRING);
> ALTER TABLE external_1k0jU ADD PARTITION (COUNTRY='USA', STATE='CA');
> INSERT INTO external_1k0jU PARTITION (country='USA', state='CA') values
> ('John Doe', 23), ('Jane Doe', 22);
> CREATE EXTERNAL TABLE external_P3kiT (name STRING, age INT) PARTITIONED BY
> (country STRING, state STRING) location 's3a://hive-on-s3/foo/bar/';
> set hive.exec.dynamic.partition.mode=nonstrict;
> INSERT INTO TABLE external_P3kiT PARTITION (country, state) SELECT * FROM
> external_1k0jU;
> {panel}
> {panel:title=Error & stack trace}
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask
> INFO : MapReduce Jobs Launched:
> INFO : Stage-Stage-1: Map: 1 Cumulative CPU: 3.64 sec HDFS Read: 3656
> HDFS Write: 99 SUCCESS
> INFO : Total MapReduce CPU Time Spent: 3 seconds 640 msec
> INFO : Completed executing
> command(queryId=hive_20161201113939_d64df5d7-a4c4-4885-846f-10f0223fcf4c);
> Time taken: 23.227 seconds
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask (state=08S01,code=1)
> INFO : Loading data to table default.external_p3kit partition (country=null,
> state=null) from
> s3a://hive-on-s3/foo/bar/.hive-staging_hive_2016-12-01_11-39-48_741_6724911837889341086-13/-ext-10002
> {code}
> ERROR : Failed with exception MetaException(message:Invalid partition key &
> values; keys [country, state, ], values [])
> org.apache.hadoop.hive.ql.metadata.HiveException:
> MetaException(message:Invalid partition key & values; keys [country, state,
> ], values [])
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:1902)
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:1834)
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1428)
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1388)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:453)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1976)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1689)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1421)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1205)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1200)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:293)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:306)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: MetaException(message:Invalid partition key & values; keys
> [country, state, ], values [])
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partition_with_auth_result$get_partition_with_auth_resultStandardScheme.read(ThriftHiveMetastore.java:65142)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partition_
[jira] [Updated] (HIVE-15361) INSERT dynamic partition on S3 fails with a MoveTask failure
[
https://issues.apache.org/jira/browse/HIVE-15361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-15361:
---
Attachment: HIVE-15361.5.patch
Attaching this file that removes this code was left unnecessary.
try {
Hive hive = Hive.get(conf);
} catch (HiveException e) {
}
No tests need to run again.
> INSERT dynamic partition on S3 fails with a MoveTask failure
>
>
> Key: HIVE-15361
> URL: https://issues.apache.org/jira/browse/HIVE-15361
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
>Priority: Critical
> Attachments: HIVE-15361.1.patch, HIVE-15361.2.patch,
> HIVE-15361.3.patch, HIVE-15361.4.patch, HIVE-15361.5.patch
>
>
> The following failure is due to the patch that merges two MoveTask found on
> the ConditionalTask (See HIVE-15114)
> {panel:title=Repro steps}
> CREATE EXTERNAL TABLE external_1k0jU (name STRING, age INT) PARTITIONED BY
> (country STRING, state STRING);
> ALTER TABLE external_1k0jU ADD PARTITION (COUNTRY='USA', STATE='CA');
> INSERT INTO external_1k0jU PARTITION (country='USA', state='CA') values
> ('John Doe', 23), ('Jane Doe', 22);
> CREATE EXTERNAL TABLE external_P3kiT (name STRING, age INT) PARTITIONED BY
> (country STRING, state STRING) location 's3a://hive-on-s3/foo/bar/';
> set hive.exec.dynamic.partition.mode=nonstrict;
> INSERT INTO TABLE external_P3kiT PARTITION (country, state) SELECT * FROM
> external_1k0jU;
> {panel}
> {panel:title=Error & stack trace}
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask
> INFO : MapReduce Jobs Launched:
> INFO : Stage-Stage-1: Map: 1 Cumulative CPU: 3.64 sec HDFS Read: 3656
> HDFS Write: 99 SUCCESS
> INFO : Total MapReduce CPU Time Spent: 3 seconds 640 msec
> INFO : Completed executing
> command(queryId=hive_20161201113939_d64df5d7-a4c4-4885-846f-10f0223fcf4c);
> Time taken: 23.227 seconds
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask (state=08S01,code=1)
> INFO : Loading data to table default.external_p3kit partition (country=null,
> state=null) from
> s3a://hive-on-s3/foo/bar/.hive-staging_hive_2016-12-01_11-39-48_741_6724911837889341086-13/-ext-10002
> {code}
> ERROR : Failed with exception MetaException(message:Invalid partition key &
> values; keys [country, state, ], values [])
> org.apache.hadoop.hive.ql.metadata.HiveException:
> MetaException(message:Invalid partition key & values; keys [country, state,
> ], values [])
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:1902)
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:1834)
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1428)
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1388)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:453)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1976)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1689)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1421)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1205)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1200)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:293)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:306)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: MetaException(message:Invalid partition key & values; keys
> [country, state, ], values [])
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partition_with_auth_result$get_partition_with_auth_resu
[jira] [Updated] (HIVE-15361) INSERT dynamic partition on S3 fails with a MoveTask failure
[
https://issues.apache.org/jira/browse/HIVE-15361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-15361:
---
Status: In Progress (was: Patch Available)
> INSERT dynamic partition on S3 fails with a MoveTask failure
>
>
> Key: HIVE-15361
> URL: https://issues.apache.org/jira/browse/HIVE-15361
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
>Priority: Critical
> Attachments: HIVE-15361.1.patch, HIVE-15361.2.patch,
> HIVE-15361.3.patch, HIVE-15361.4.patch
>
>
> The following failure is due to the patch that merges two MoveTask found on
> the ConditionalTask (See HIVE-15114)
> {panel:title=Repro steps}
> CREATE EXTERNAL TABLE external_1k0jU (name STRING, age INT) PARTITIONED BY
> (country STRING, state STRING);
> ALTER TABLE external_1k0jU ADD PARTITION (COUNTRY='USA', STATE='CA');
> INSERT INTO external_1k0jU PARTITION (country='USA', state='CA') values
> ('John Doe', 23), ('Jane Doe', 22);
> CREATE EXTERNAL TABLE external_P3kiT (name STRING, age INT) PARTITIONED BY
> (country STRING, state STRING) location 's3a://hive-on-s3/foo/bar/';
> set hive.exec.dynamic.partition.mode=nonstrict;
> INSERT INTO TABLE external_P3kiT PARTITION (country, state) SELECT * FROM
> external_1k0jU;
> {panel}
> {panel:title=Error & stack trace}
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask
> INFO : MapReduce Jobs Launched:
> INFO : Stage-Stage-1: Map: 1 Cumulative CPU: 3.64 sec HDFS Read: 3656
> HDFS Write: 99 SUCCESS
> INFO : Total MapReduce CPU Time Spent: 3 seconds 640 msec
> INFO : Completed executing
> command(queryId=hive_20161201113939_d64df5d7-a4c4-4885-846f-10f0223fcf4c);
> Time taken: 23.227 seconds
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask (state=08S01,code=1)
> INFO : Loading data to table default.external_p3kit partition (country=null,
> state=null) from
> s3a://hive-on-s3/foo/bar/.hive-staging_hive_2016-12-01_11-39-48_741_6724911837889341086-13/-ext-10002
> {code}
> ERROR : Failed with exception MetaException(message:Invalid partition key &
> values; keys [country, state, ], values [])
> org.apache.hadoop.hive.ql.metadata.HiveException:
> MetaException(message:Invalid partition key & values; keys [country, state,
> ], values [])
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:1902)
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:1834)
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1428)
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1388)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:453)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1976)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1689)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1421)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1205)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1200)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:293)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:306)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: MetaException(message:Invalid partition key & values; keys
> [country, state, ], values [])
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partition_with_auth_result$get_partition_with_auth_resultStandardScheme.read(ThriftHiveMetastore.java:65142)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partition_with_auth_result$get_partition_with_auth_result
[jira] [Commented] (HIVE-15367) CTAS with LOCATION should write temp data under location directory rather than database location
[
https://issues.apache.org/jira/browse/HIVE-15367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729178#comment-15729178
]
Hive QA commented on HIVE-15367:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12842069/HIVE-15367.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10747 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=143)
[vectorized_rcfile_columnar.q,vector_elt.q,explainuser_1.q,multi_insert.q,tez_dml.q,vector_bround.q,schema_evol_orc_acid_table.q,vector_when_case_null.q,orc_ppd_schema_evol_1b.q,vector_join30.q,vectorization_11.q,cte_3.q,update_tmp_table.q,vector_decimal_cast.q,groupby_grouping_id2.q,vector_decimal_round.q,tez_smb_empty.q,orc_merge6.q,vector_decimal_trailing.q,cte_5.q,tez_union.q,cbo_rp_subq_not_in.q,vector_decimal_2.q,columnStatsUpdateForStatsOptimizer_1.q,vector_outer_join3.q,schema_evol_text_vec_part_all_complex.q,tez_dynpart_hashjoin_2.q,auto_sortmerge_join_12.q,offset_limit.q,tez_union_multiinsert.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision]
(batchId=150)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2462/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2462/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2462/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12842069 - PreCommit-HIVE-Build
> CTAS with LOCATION should write temp data under location directory rather
> than database location
>
>
> Key: HIVE-15367
> URL: https://issues.apache.org/jira/browse/HIVE-15367
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-15367.1.patch, HIVE-15367.2.patch,
> HIVE-15367.3.patch
>
>
> For regular CTAS queries, temp data from a SELECT query will be written to to
> a staging directory under the database location. The code to control this is
> in {{SemanticAnalyzer.java}}
> {code}
> // allocate a temporary output dir on the location of the table
> String tableName = getUnescapedName((ASTNode) ast.getChild(0));
> String[] names = Utilities.getDbTableName(tableName);
> Path location;
> try {
> Warehouse wh = new Warehouse(conf);
> //Use destination table's db location.
> String destTableDb = qb.getTableDesc() != null?
> qb.getTableDesc().getDatabaseName(): null;
> if (destTableDb == null) {
> destTableDb = names[0];
> }
> location = wh.getDatabasePath(db.getDatabase(destTableDb));
> } catch (MetaException e) {
> throw new SemanticException(e);
> }
> {code}
> However, CTAS queries allow specifying a {{LOCATION}} for the new table. Its
> possible for this location to be on a different filesystem than the database
> location. If this happens temp data will be written to the database
> filesystem and will be copied to the table filesystem in {{MoveTask}}.
> This extra copying of data can drastically affect performance. Rather than
> always use the database location as the staging dir for CTAS queries, Hive
> should first check if there is an explicit {{LOCATION}} specified in the CTAS
> query. If there is, staging data should be stored under the {{LOCATION}}
> directory.
--
[jira] [Updated] (HIVE-15275) "beeline -f " will throw NPE
[
https://issues.apache.org/jira/browse/HIVE-15275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-15275:
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks Vihang and Yongzhi for reviewing.
> "beeline -f " will throw NPE
> ---
>
> Key: HIVE-15275
> URL: https://issues.apache.org/jira/browse/HIVE-15275
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 2.2.0
>
> Attachments: HIVE-15275.1.patch
>
>
> Execute {{"beeline -f "}} and the command will throw the following NPE
> exception.
> {noformat}
> 2016-11-23T13:34:54,367 WARN [Thread-1]
> org.apache.hadoop.util.ShutdownHookManager - ShutdownHook '' failed,
> java.lang.NullPointerException
> java.lang.NullPointerException
> at org.apache.hive.beeline.BeeLine$1.run(BeeLine.java:1247)
> ~[hive-beeline-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
> [hadoop-common-2.7.3.jar:?]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15337) Enhance Show Compactions output with JobId and start time for "attempted" state
[ https://issues.apache.org/jira/browse/HIVE-15337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-15337: -- Status: Patch Available (was: Reopened) > Enhance Show Compactions output with JobId and start time for "attempted" > state > --- > > Key: HIVE-15337 > URL: https://issues.apache.org/jira/browse/HIVE-15337 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 2.2.0 > > Attachments: HIVE-15337.01.patch, HIVE-15337.02.patch, > HIVE-15337.03.addendum.patch, HIVE-15337.03.patch > > > W/o this SHOW COMPACTIONS output is not as useful > Also, add Hadoop Job ID to SHOW COMPACTIONS output -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15337) Enhance Show Compactions output with JobId and start time for "attempted" state
[ https://issues.apache.org/jira/browse/HIVE-15337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-15337: -- Attachment: HIVE-15337.03.addendum.patch > Enhance Show Compactions output with JobId and start time for "attempted" > state > --- > > Key: HIVE-15337 > URL: https://issues.apache.org/jira/browse/HIVE-15337 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 2.2.0 > > Attachments: HIVE-15337.01.patch, HIVE-15337.02.patch, > HIVE-15337.03.addendum.patch, HIVE-15337.03.patch > > > W/o this SHOW COMPACTIONS output is not as useful > Also, add Hadoop Job ID to SHOW COMPACTIONS output -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HIVE-15337) Enhance Show Compactions output with JobId and start time for "attempted" state
[ https://issues.apache.org/jira/browse/HIVE-15337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reopened HIVE-15337: --- > Enhance Show Compactions output with JobId and start time for "attempted" > state > --- > > Key: HIVE-15337 > URL: https://issues.apache.org/jira/browse/HIVE-15337 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 2.2.0 > > Attachments: HIVE-15337.01.patch, HIVE-15337.02.patch, > HIVE-15337.03.patch > > > W/o this SHOW COMPACTIONS output is not as useful > Also, add Hadoop Job ID to SHOW COMPACTIONS output -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15307) Hive MERGE: "when matched then update" allows invalid column names.
[
https://issues.apache.org/jira/browse/HIVE-15307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729070#comment-15729070
]
Eugene Koifman commented on HIVE-15307:
---
[~alangates] could you review please
> Hive MERGE: "when matched then update" allows invalid column names.
> ---
>
> Key: HIVE-15307
> URL: https://issues.apache.org/jira/browse/HIVE-15307
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 2.2.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-15307.01.patch, HIVE-15307.02.patch
>
>
> {noformat}
> create table target (
> id int, val int
> )
> CLUSTERED BY (id) INTO 2 BUCKETS STORED AS ORC TBLPROPERTIES
> ("transactional"="true");
> create table source2 (
> id int, val int
> );
> insert into source2 values (2, 25), (3, 35), (4, 45);
> merge into target
> using source2 sub on sub.id = target.id
> when matched then update set invalid = sub.val;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15285) err info for itests mvn building is not correct
[ https://issues.apache.org/jira/browse/HIVE-15285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729063#comment-15729063 ] Sergio Peña commented on HIVE-15285: The HIVE-14735 approach will have a flag to disable the spark download. Something like -DskipSparkAssemblyDeploy so it might help on network issues too. > err info for itests mvn building is not correct > --- > > Key: HIVE-15285 > URL: https://issues.apache.org/jira/browse/HIVE-15285 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 2.1.0, 2.2.0 >Reporter: Fei Hui >Assignee: Fei Hui > Attachments: HIVE-15285.1.patch > > > When i build itests, i found some err info > ''' > [exec] cp: cannot stat > `./target/../../..//data/conf/spark/log4j2.properties': No such file or > directory > ''' > But i found that the real reason is spark down load error. The msg above > confuse users ， it is not root cause -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13306) Better Decimal vectorization
[ https://issues.apache.org/jira/browse/HIVE-13306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-13306: -- Attachment: HIVE-13306.5.patch > Better Decimal vectorization > > > Key: HIVE-13306 > URL: https://issues.apache.org/jira/browse/HIVE-13306 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Teddy Choi >Priority: Critical > Attachments: HIVE-13306.1.patch, HIVE-13306.2.patch, > HIVE-13306.3.patch, HIVE-13306.4.patch, HIVE-13306.5.patch > > > Decimal Vectorization Requirements > • Today, the LongColumnVector, DoubleColumnVector, BytesColumnVector, > TimestampColumnVector classes store the data as primitive Java data types > long, double, or byte arrays for efficiency. > • DecimalColumnVector is different - it has an array of Object references > to HiveDecimal objects. > • The HiveDecimal object uses an internal object BigDecimal for its > implementation. Further, BigDecimal itself uses an internal object > BigInteger for its implementation, and BigInteger uses an int array. 4 > objects total. > • And, HiveDecimal is an immutable object which means arithmetic and > other operations produce new HiveDecimal object with 3 new objects underneath. > • A major reason Vectorization is fast is the ColumnVector classes except > DecimalColumnVector do not have to allocate additional memory per row. This > avoids memory fragmentation and pressure on the Java Garbage Collector that > DecimalColumnVector can generate. It is very significant. > • What can be done with DecimalColumnVector to make it much more > efficient? > o Design several new decimal classes that allow the caller to manage the > decimal storage. > o If it takes N int values to store a decimal (e.g. N=1..5), then a new > DecimalColumnVector would have an int[] of length N*1024 (where 1024 is the > default column vector size). > o Why store a decimal in separate int values? > • Java does not support 128 bit integers. > • Java does not support unsigned integers. > • In order to do multiplication of a decimal represented in a long you > need twice the storage (i.e. 128 bits). So you need to represent parts in 32 > bit integers. > • But really since we do not have unsigned, really you can only do > multiplications on N-1 bits or 31 bits. > • So, 5 ints are needed for decimal storage... of 38 digits. > o It makes sense to have just one algorithm for decimals rather than one > for HiveDecimal and another for DecimalColumnVector. So, make HiveDecimal > store N int values, too. > o A lower level primitive decimal class would accept decimals stored as > int arrays and produces results into int arrays. It would be used by > HiveDecimal and DecimalColumnVector. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15149) Add additional information to ATSHook for Tez UI
[
https://issues.apache.org/jira/browse/HIVE-15149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729000#comment-15729000
]
Hive QA commented on HIVE-15149:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12842063/HIVE-15149.5.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 10760 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=112)
[bucketmapjoin3.q,union_date.q,cbo_gby.q,auto_join31.q,auto_sortmerge_join_1.q,join_cond_pushdown_unqual1.q,ppd_outer_join3.q,bucket_map_join_spark3.q,union28.q,statsfs.q,escape_sortby1.q,leftsemijoin.q,groupby_multi_single_reducer3.q,union_remove_6.q,join29.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters]
(batchId=136)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=92)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_4]
(batchId=92)
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery
(batchId=215)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2461/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2461/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2461/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12842063 - PreCommit-HIVE-Build
> Add additional information to ATSHook for Tez UI
>
>
> Key: HIVE-15149
> URL: https://issues.apache.org/jira/browse/HIVE-15149
> Project: Hive
> Issue Type: Improvement
> Components: Hooks
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-15149.1.patch, HIVE-15149.2.patch,
> HIVE-15149.3.patch, HIVE-15149.4.patch, HIVE-15149.5.patch
>
>
> Additional query details wanted for TEZ-3530. The additional details
> discussed include the following:
> Publish the following info ( in addition to existing bits published today):
> Application Id to which the query was submitted (primary filter)
> DAG Id (primary filter)
> Hive query name (primary filter)
> Hive Configs (everything a set command would provide except for sensitive
> credential info)
> Potentially publish source of config i.e. set in hive query script vs
> hive-site.xml, etc.
> Which HiveServer2 the query was submitted to
> *Which IP/host the query was submitted from - not sure what filter support
> will be available.
> Which execution mode the query is running in (primary filter)
> What submission mode was used (cli/beeline/jdbc, etc)
> User info ( running as, actual end user, etc) - not sure if already present
> Perf logger events. The data published should be able to create a timeline
> view of the query i.e. actual submission time, query compile timestamps,
> execution timestamps, post-exec data moves, etc.
> Explain plan with enough details for visualizing.
> Databases and tables being queried (primary filter)
> Yarn queue info (primary filter)
> Caller context (primary filter)
> Original source i.e. submitter
> Thread info in HS2 if needed ( I believe Vikram may have added this earlier )
> Query time taken (with filter support )
> Additional context info e.g. llap instance name and appId if required.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15359) skip.footer.line.count doesnt work properly for certain situations
[ https://issues.apache.org/jira/browse/HIVE-15359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15728918#comment-15728918 ] Yongzhi Chen commented on HIVE-15359: - The failures are not related. > skip.footer.line.count doesnt work properly for certain situations > -- > > Key: HIVE-15359 > URL: https://issues.apache.org/jira/browse/HIVE-15359 > Project: Hive > Issue Type: Bug > Components: Reader >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Attachments: HIVE-15359.1.patch > > > This issue's reproduce is very like HIVE-12718 , but the data file is larger > than 128M . In this case, even make sure only one mapper is used, the footer > is still wrongly skipped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14007) Replace ORC module with ORC release
[
https://issues.apache.org/jira/browse/HIVE-14007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15728879#comment-15728879
]
Hive QA commented on HIVE-14007:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12842058/HIVE-14007.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 50 failed/errored test(s), 9907 tests
executed
*Failed tests:*
{noformat}
TestBitFieldReader - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
TestBitPack - did not produce a TEST-*.xml file (likely timed out) (batchId=235)
TestColumnStatistics - did not produce a TEST-*.xml file (likely timed out)
(batchId=233)
TestColumnStatisticsImpl - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestDataReaderProperties - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestDynamicArray - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestFileDump - did not produce a TEST-*.xml file (likely timed out)
(batchId=233)
TestInStream - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestIntegerCompressionReader - did not produce a TEST-*.xml file (likely timed
out) (batchId=234)
TestJsonFileDump - did not produce a TEST-*.xml file (likely timed out)
(batchId=233)
TestMemoryManager - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=132)
[mapreduce2.q,orc_llap_counters1.q,bucket6.q,insert_into1.q,empty_dir_in_table.q,orc_merge1.q,script_env_var1.q,orc_merge_diff_fs.q,llapdecider.q,load_hdfs_file_with_space_in_the_name.q,llap_nullscan.q,orc_ppd_basic.q,transform_ppr1.q,rcfile_merge4.q,orc_merge3.q]
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=136)
[orc_merge2.q,insert_into2.q,reduce_deduplicate.q,orc_llap_counters.q,cte_4.q,schemeAuthority2.q,file_with_header_footer.q,rcfile_merge3.q]
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=143)
[vectorized_rcfile_columnar.q,vector_elt.q,explainuser_1.q,multi_insert.q,tez_dml.q,vector_bround.q,schema_evol_orc_acid_table.q,vector_when_case_null.q,orc_ppd_schema_evol_1b.q,vector_join30.q,vectorization_11.q,cte_3.q,update_tmp_table.q,vector_decimal_cast.q,groupby_grouping_id2.q,vector_decimal_round.q,tez_smb_empty.q,orc_merge6.q,vector_decimal_trailing.q,cte_5.q,tez_union.q,cbo_rp_subq_not_in.q,vector_decimal_2.q,columnStatsUpdateForStatsOptimizer_1.q,vector_outer_join3.q,schema_evol_text_vec_part_all_complex.q,tez_dynpart_hashjoin_2.q,auto_sortmerge_join_12.q,offset_limit.q,tez_union_multiinsert.q]
TestNewIntegerEncoding - did not produce a TEST-*.xml file (likely timed out)
(batchId=236)
TestOrcNullOptimization - did not produce a TEST-*.xml file (likely timed out)
(batchId=233)
TestOrcTimezone1 - did not produce a TEST-*.xml file (likely timed out)
(batchId=233)
TestOrcTimezone2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=233)
TestOrcTimezone3 - did not produce a TEST-*.xml file (likely timed out)
(batchId=233)
TestOrcWideTable - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestOutStream - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
TestRLEv2 - did not produce a TEST-*.xml file (likely timed out) (batchId=234)
TestReaderImpl - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
TestRecordReaderImpl - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
TestRunLengthByteReader - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestRunLengthIntegerReader - did not produce a TEST-*.xml file (likely timed
out) (batchId=235)
TestSchemaEvolution - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
TestSerializationUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[parallel_join1.q,union27.q,union12.q,groupby7_map_multi_single_reducer.q,varchar_join1.q,join7.q,join_reorder4.q,skewjoinopt2.q,bucketsortoptimize_insert_2.q,smb_mapjoin_17.q,script_env_var1.q,groupby7_map.q,groupby3.q,bucketsortoptimize_insert_8.q,union20.q]
TestStreamName - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestStringDictionary - did not produce a TEST-*.xml file (likely timed out)
(batchId=233)
TestStringRedBlackTree - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestTypeDescription - did not produce a TEST-*.xml file (likely timed out)
(batchId=236)
TestUnrolledBitPack - did not produce a TEST-*.xml file (likely timed out)
(batchId=233)
TestVectorOrcFile - did not produce a TEST-*.xml file (likely timed out)
(batch
[jira] [Commented] (HIVE-15335) Fast Decimal
[
https://issues.apache.org/jira/browse/HIVE-15335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15728726#comment-15728726
]
Hive QA commented on HIVE-15335:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12842055/HIVE-15335.05.patch
{color:green}SUCCESS:{color} +1 due to 11 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10856 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_ppd_decimal]
(batchId=138)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision]
(batchId=150)
org.apache.hive.hcatalog.data.TestDefaultHCatRecord.testRYW (batchId=180)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2459/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2459/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2459/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12842055 - PreCommit-HIVE-Build
> Fast Decimal
>
>
> Key: HIVE-15335
> URL: https://issues.apache.org/jira/browse/HIVE-15335
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-15335.01.patch, HIVE-15335.02.patch,
> HIVE-15335.03.patch, HIVE-15335.04.patch, HIVE-15335.05.patch
>
>
> Replace HiveDecimal implementation that currently represents the decimal
> internally as a BigDecimal with a faster version that does not allocate extra
> objects
> Replace HiveDecimalWritable implementation with a faster version that has new
> mutable* calls (e.g. mutableAdd, mutableEnforcePrecisionScale, etc) and
> stores the result as a fast decimal instead of a slow byte array containing a
> serialized BigInteger.
> Provide faster ways to serialize/deserialize decimals.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15374) Hive column comments disappearing/being replaced by "from deserializer"
[
https://issues.apache.org/jira/browse/HIVE-15374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15728640#comment-15728640
]
Hive QA commented on HIVE-15374:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12842048/HIVE-15374.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10775 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision]
(batchId=150)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_table_wrong_regex]
(batchId=84)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2458/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2458/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2458/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12842048 - PreCommit-HIVE-Build
> Hive column comments disappearing/being replaced by "from deserializer"
> ---
>
> Key: HIVE-15374
> URL: https://issues.apache.org/jira/browse/HIVE-15374
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-15374.patch
>
>
> After creating a table in hive with column comments, running show create
> table or describe [formatted], on the same table gives "from deserializer"
> instead of the original comments.
> CREATE TABLE `test`(
> `stringid` string COMMENT 'string id',
> `value` string COMMENT 'description')
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
> WITH SERDEPROPERTIES (
> 'input.regex'='(.{1})');
> The comments appear to be stored correctly in the HMS backend DB. Just the
> fetching of this metadata seems to be incorrect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-15379) Get the key of hive.metastore.* values should be consistent with Hive Metastore Server.
[
https://issues.apache.org/jira/browse/HIVE-15379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-15379 started by Fei Hui.
--
> Get the key of hive.metastore.* values should be consistent with Hive
> Metastore Server.
>
>
> Key: HIVE-15379
> URL: https://issues.apache.org/jira/browse/HIVE-15379
> Project: Hive
> Issue Type: Bug
> Components: Beeline, CLI
>Affects Versions: 1.1.0
>Reporter: Yuming Wang
>Assignee: Fei Hui
>Priority: Minor
>
> Hive Metastore Server's {{hive.metastore.try.direct.sql=false}} when using
> Cloudera Manager. But cli or beeline read the client configure and return
> true. It is meaningless.
> {code}
> hive> set hive.metastore.try.direct.sql;
> hive.metastore.try.direct.sql=true
> hive>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15379) Get the key of hive.metastore.* values should be consistent with Hive Metastore Server.
[
https://issues.apache.org/jira/browse/HIVE-15379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15728620#comment-15728620
]
Fei Hui commented on HIVE-15379:
i.e.
I mean that metastore can startup with --hiveconf
hive.metastore.try.direct.sql=false ,then hive.metastore.try.direct.sql is
false.
Cli startup with no args, it will set hive.metastore.try.direct.sql default
value
> Get the key of hive.metastore.* values should be consistent with Hive
> Metastore Server.
>
>
> Key: HIVE-15379
> URL: https://issues.apache.org/jira/browse/HIVE-15379
> Project: Hive
> Issue Type: Bug
> Components: Beeline, CLI
>Affects Versions: 1.1.0
>Reporter: Yuming Wang
>Assignee: Fei Hui
>Priority: Minor
>
> Hive Metastore Server's {{hive.metastore.try.direct.sql=false}} when using
> Cloudera Manager. But cli or beeline read the client configure and return
> true. It is meaningless.
> {code}
> hive> set hive.metastore.try.direct.sql;
> hive.metastore.try.direct.sql=true
> hive>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
