[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850285#comment-15850285 ] Hive QA commented on HIVE-12767: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12786152/TestNanoTimeUtils.java {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3331/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3331/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3331/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2017-02-02 18:33:04.296 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-3331/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2017-02-02 18:33:04.300 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 7cca097 HIVE-14420: Fix orc_llap_counters.q test failure in master (Prasanth Jayachandran reviewed by Siddharth Seth) + git clean -f -d Removing ql/src/test/queries/clientpositive/parquet_ppd_multifiles.q Removing ql/src/test/results/clientpositive/parquet_ppd_multifiles.q.out + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 7cca097 HIVE-14420: Fix orc_llap_counters.q test failure in master (Prasanth Jayachandran reviewed by Siddharth Seth) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2017-02-02 18:33:05.296 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch patch: Only garbage was found in the patch input. patch: Only garbage was found in the patch input. patch: Only garbage was found in the patch input. fatal: unrecognized input The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12786152 - PreCommit-HIVE-Build > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-12767.3.patch, HIVE-12767.4.patch, > TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15854565#comment-15854565 ] Hive QA commented on HIVE-12767: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12851198/HIVE-12767.5.patch {color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10224 tests executed *Failed tests:* {noformat} TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235) TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=100) [skewjoinopt15.q,auto_join18.q,list_bucket_dml_2.q,input1_limit.q,load_dyn_part3.q,union_remove_14.q,auto_sortmerge_join_14.q,auto_sortmerge_join_15.q,union10.q,bucket_map_join_tez2.q,groupby5_map_skew.q,join_reorder.q,sample1.q,bucketmapjoin8.q,union34.q] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_external_time] (batchId=12) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_int96_timestamp] (batchId=33) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys] (batchId=159) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=223) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=223) org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testWaitQueuePreemption (batchId=278) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3396/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3396/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3396/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12851198 - PreCommit-HIVE-Build > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-12767.3.patch, HIVE-12767.4.patch, > HIVE-12767.5.patch, TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856234#comment-15856234 ] Hive QA commented on HIVE-12767: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12851409/HIVE-12767.6.patch {color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10142 tests executed *Failed tests:* {noformat} TestCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=17) [list_bucket_dml_1.q,ppd_join3.q,auto_join23.q,list_bucket_dml_11.q,join10.q,avro_type_evolution.q,create_struct_table.q,cbo_const.q,skewjoin_mapjoin9.q,hook_context_cs.q,subquery_unqualcolumnrefs.q,exim_22_import_exist_authsuccess.q,groupby1.q,cbo_rp_udf_udaf.q,udf_regexp_replace.q,vector_decimal_aggregate.q,authorization_grant_public_role.q,partition_wise_fileformat.q,sort_merge_join_desc_5.q,union_ppr.q,spark_combine_equivalent_work.q,stats_partial_size.q,partition_date2.q,join32.q,list_bucket_dml_14.q,input34.q,insert_values_acid_not_bucketed.q,udf_parse_url.q,schema_evol_text_nonvec_part.q,ctas_char.q] TestCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=18) [encryption_move_tbl.q,cbo_rp_udf_percentile2.q,udf_lower.q,union_remove_11.q,transform_acid.q,constprog_dp.q,join45.q,subquery_multi.q,orc_merge1.q,groupby_multi_single_reducer2.q,offset_limit_global_optimizer.q,dbtxnmgr_query4.q,multi_column_in.q,create_func1.q,create_skewed_table1.q,authorization_view_1.q,show_tables.q,orc_empty_files.q,ba_table2.q,acid_vectorization_project.q,union_remove_19.q,vector_decimal_3.q,stats3.q,update_where_no_match.q,parquet_join.q,exim_24_import_nonexist_authsuccess.q,outer_join_ppr.q,quotedid_partition.q,ansi_sql_arithmetic.q,join26.q] TestCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=19) [cp_mj_rc.q,order.q,udf_bitwise_shiftleft.q,groupby1_limit.q,insert_values_non_partitioned.q,skewjoinopt10.q,extrapolate_part_stats_date.q,udf_sin.q,vectorized_math_funcs.q,vectorization_2.q,join14.q,query_result_fileformat.q,udf3.q,cbo_union_view.q,drop_function.q,tez_union_decimal.q,exim_23_import_part_authsuccess.q,load_dyn_part10.q,nonmr_fetch.q,order_null.q,cbo_rp_views.q,lvj_mapjoin.q,insert_acid_dynamic_partition.q,skewjoinopt6.q,cbo_rp_cross_product_check_2.q,mapreduce7.q,update_two_cols.q,correlationoptimizer11.q,stats_empty_partition.q,authorization_2.q] TestCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=22) [multi_insert_mixed.q,specialChar.q,join_cond_pushdown_3.q,cast_on_constant.q,union_remove_10.q,skewjoinopt21.q,input_part0.q,vector_join_nulls.q,combine2_hadoop20.q,infer_join_preds.q,udf_hex.q,reducesink_dedup.q,insert1.q,input16.q,udf_in_file.q,vector_empty_where.q,udf_variance.q,join42.q,auto_join12.q,groupby_sort_2.q,subquery_alias.q,cte_mat_3.q,parenthesis_star_by.q,vector_decimal_round_2.q,udf_conv.q,column_names_with_leading_and_trailing_spaces.q,vectorized_mapjoin2.q,union_stats.q,nullgroup4.q,authorization_view_disable_cbo_2.q] TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_int96_timestamp] (batchId=33) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys] (batchId=159) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=223) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3417/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3417/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3417/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12851409 - PreCommit-HIVE-Build > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-12767.3.patch, HIVE-12767.4.patch, > HIVE-12767.5.patch, HIVE-12767.6.patch, TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15858175#comment-15858175 ] Hive QA commented on HIVE-12767: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12851654/HIVE-12767.7.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3442/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3442/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3442/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2017-02-08 16:13:13.361 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-3442/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2017-02-08 16:13:13.364 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 8a06b9e HIVE-15626: beeline should not exit after canceling the query on ctrl-c (Vihang Karajgaonkar via Chaoyu Tang) + git clean -f -d Removing ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RepartitionShuffler.java + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 8a06b9e HIVE-15626: beeline should not exit after canceling the query on ctrl-c (Vihang Karajgaonkar via Chaoyu Tang) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2017-02-08 16:13:14.446 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch fatal: git diff header lacks filename information when removing 0 leading pathname components (line 20) The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12851654 - PreCommit-HIVE-Build > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-12767.3.patch, HIVE-12767.4.patch, > HIVE-12767.5.patch, HIVE-12767.6.patch, HIVE-12767.7.patch, > TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sen
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860144#comment-15860144 ] Hive QA commented on HIVE-12767: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12851905/HIVE-12767.8.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3471/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3471/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3471/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2017-02-09 20:31:57.050 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-3471/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2017-02-09 20:31:57.053 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 36ff484 HIVE-15851 CompactorMR.launchCompactionJob() should use JobClient.submitJob() not runJob (Eugene Koifman, reviewed by Wei Zheng) + git clean -f -d + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 36ff484 HIVE-15851 CompactorMR.launchCompactionJob() should use JobClient.submitJob() not runJob (Eugene Koifman, reviewed by Wei Zheng) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2017-02-09 20:31:58.075 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java:127 error: ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java: patch does not apply The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12851905 - PreCommit-HIVE-Build > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-12767.3.patch, HIVE-12767.4.patch, > HIVE-12767.5.patch, HIVE-12767.6.patch, HIVE-12767.7.patch, > HIVE-12767.8.patch, TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables create
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15861350#comment-15861350 ] Hive QA commented on HIVE-12767: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12852050/HIVE-12767.8.patch {color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10258 tests executed *Failed tests:* {noformat} TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys] (batchId=159) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=223) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3490/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3490/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3490/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12852050 - PreCommit-HIVE-Build > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-12767.3.patch, HIVE-12767.4.patch, > HIVE-12767.5.patch, HIVE-12767.6.patch, HIVE-12767.7.patch, > HIVE-12767.8.patch, TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15861366#comment-15861366 ] Barna Zsombor Klara commented on HIVE-12767: query14 and encryption_join_with_different_encryption_keys are know flaky tests (HIVE-15744, HIVE-15696) > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-12767.3.patch, HIVE-12767.4.patch, > HIVE-12767.5.patch, HIVE-12767.6.patch, HIVE-12767.7.patch, > HIVE-12767.8.patch, TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15863723#comment-15863723 ] Hive QA commented on HIVE-12767: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12852347/HIVE-12767.9.patch {color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10250 tests executed *Failed tests:* {noformat} TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] (batchId=140) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=223) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=223) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3517/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3517/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3517/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12852347 - PreCommit-HIVE-Build > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-12767.10.patch, HIVE-12767.3.patch, > HIVE-12767.4.patch, HIVE-12767.5.patch, HIVE-12767.6.patch, > HIVE-12767.7.patch, HIVE-12767.8.patch, HIVE-12767.9.patch, > TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15863788#comment-15863788 ] Hive QA commented on HIVE-12767: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12852362/HIVE-12767.10.patch {color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10250 tests executed *Failed tests:* {noformat} TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys] (batchId=159) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=223) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=223) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3518/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3518/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3518/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12852362 - PreCommit-HIVE-Build > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-12767.10.patch, HIVE-12767.3.patch, > HIVE-12767.4.patch, HIVE-12767.5.patch, HIVE-12767.6.patch, > HIVE-12767.7.patch, HIVE-12767.8.patch, HIVE-12767.9.patch, > TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15863951#comment-15863951 ] Hive QA commented on HIVE-12767: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12852366/HIVE-12767.10.patch {color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10236 tests executed *Failed tests:* {noformat} TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys] (batchId=159) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=223) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=223) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=101) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3520/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3520/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3520/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12852366 - PreCommit-HIVE-Build > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-12767.10.patch, HIVE-12767.3.patch, > HIVE-12767.4.patch, HIVE-12767.5.patch, HIVE-12767.6.patch, > HIVE-12767.7.patch, HIVE-12767.8.patch, HIVE-12767.9.patch, > TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15878798#comment-15878798 ] Sergio Peña commented on HIVE-12767: Great, thanks [~zsombor.klara] for continuing on this work. I just left a couple of comments on the RB. > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-12767.10.patch, HIVE-12767.3.patch, > HIVE-12767.4.patch, HIVE-12767.5.patch, HIVE-12767.6.patch, > HIVE-12767.7.patch, HIVE-12767.8.patch, HIVE-12767.9.patch, > TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15883733#comment-15883733 ] Hive QA commented on HIVE-12767: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12854485/HIVE-12767.11.patch {color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10274 tests executed *Failed tests:* {noformat} TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=223) org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel (batchId=211) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3764/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3764/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3764/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12854485 - PreCommit-HIVE-Build > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-12767.10.patch, HIVE-12767.11.patch, > HIVE-12767.3.patch, HIVE-12767.4.patch, HIVE-12767.5.patch, > HIVE-12767.6.patch, HIVE-12767.7.patch, HIVE-12767.8.patch, > HIVE-12767.9.patch, TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15886739#comment-15886739 ] Sergio Peña commented on HIVE-12767: +1 Thanks [~zsombor.klara]. I think the patch is good now. Thanks for continuing the work on this patch. > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-12767.10.patch, HIVE-12767.11.patch, > HIVE-12767.3.patch, HIVE-12767.4.patch, HIVE-12767.5.patch, > HIVE-12767.6.patch, HIVE-12767.7.patch, HIVE-12767.8.patch, > HIVE-12767.9.patch, TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15889747#comment-15889747 ] Lefty Leverenz commented on HIVE-12767: --- Here's where the documentation belongs: * [Parquet | https://cwiki.apache.org/confluence/display/Hive/Parquet] * [Configuration Properties -- Parquet | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Parquet] for *parquet.mr.int96.enable.utc.write.zone* * [Table Properties | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-listTableProperties] Question: Shouldn't the config name start with "hive." as in "hive.parquet.mr.int96.enable.utc.write.zone"? > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-12767.10.patch, HIVE-12767.11.patch, > HIVE-12767.3.patch, HIVE-12767.4.patch, HIVE-12767.5.patch, > HIVE-12767.6.patch, HIVE-12767.7.patch, HIVE-12767.8.patch, > HIVE-12767.9.patch, TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15889906#comment-15889906 ] Barna Zsombor Klara commented on HIVE-12767: Thanks [~leftylev] for the help, I updated the docs you mentioned. As for the "hive." prefix, I don't know if this is a strict convention or not. While most configuration properties start with the "hive" prefix, others do not, like "parquet.memory.pool.ratio", and I can't see a clear rule clarifying when the prefix is needed. [~spena] should we (should I) rename it or was is meant to be like this? > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-12767.10.patch, HIVE-12767.11.patch, > HIVE-12767.3.patch, HIVE-12767.4.patch, HIVE-12767.5.patch, > HIVE-12767.6.patch, HIVE-12767.7.patch, HIVE-12767.8.patch, > HIVE-12767.9.patch, TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16016900#comment-16016900 ] Lefty Leverenz commented on HIVE-12767: --- Thanks for the docs, [~zsombor.klara]. I changed the version numbers from 2.2.0 to 2.4.0 in the wiki and removed the TODOC2.2 label from this issue. Here are the doc links again: * [Parquet -- Hive2.4.0 | https://cwiki.apache.org/confluence/display/Hive/Parquet#Parquet-Hive2.4.0] * [Configuration Properties -- hive.parquet.mr.int96.enable.utc.write.zone | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.parquet.mr.int96.enable.utc.write.zone] * [DDL -- TBLPROPERTIES | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-listTableProperties] > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Fix For: 2.4.0 > > Attachments: HIVE-12767.10.patch, HIVE-12767.11.patch, > HIVE-12767.3.patch, HIVE-12767.4.patch, HIVE-12767.5.patch, > HIVE-12767.6.patch, HIVE-12767.7.patch, HIVE-12767.8.patch, > HIVE-12767.9.patch, TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077227#comment-16077227 ] Lefty Leverenz commented on HIVE-12767: --- Added a TODOC2.4 label in case we need to revert the documentation (3 wikidocs listed in comment 19/May/17). But let's wait a while and see how the issue is solved. > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Labels: TODOC2.4 > Attachments: HIVE-12767.10.patch, HIVE-12767.11.patch, > HIVE-12767.3.patch, HIVE-12767.4.patch, HIVE-12767.5.patch, > HIVE-12767.6.patch, HIVE-12767.7.patch, HIVE-12767.8.patch, > HIVE-12767.9.patch, TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153172#comment-16153172 ] Lefty Leverenz commented on HIVE-12767: --- The documentation was reverted (thanks, [~zsombor.klara]) so I'll remove the TODOC2.4 label. > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-12767.10.patch, HIVE-12767.11.patch, > HIVE-12767.3.patch, HIVE-12767.4.patch, HIVE-12767.5.patch, > HIVE-12767.6.patch, HIVE-12767.7.patch, HIVE-12767.8.patch, > HIVE-12767.9.patch, TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891949#comment-15891949 ] Barna Zsombor Klara commented on HIVE-12767: I've raised HIVE-16088 to fix the misnamed configuration property. > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-12767.10.patch, HIVE-12767.11.patch, > HIVE-12767.3.patch, HIVE-12767.4.patch, HIVE-12767.5.patch, > HIVE-12767.6.patch, HIVE-12767.7.patch, HIVE-12767.8.patch, > HIVE-12767.9.patch, TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15075538#comment-15075538 ] Sergio Peña commented on HIVE-12767: cc: [~kamrul] > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Sergio Peña > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15075595#comment-15075595 ] Hive QA commented on HIVE-12767: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12780066/HIVE-12767.1.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6506/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6506/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6506/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]] + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6506/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 1a460b5 HIVE-12372 : Improve to support the multibyte character at lpad and rpad (Shinichi Yamashita via Ashutosh Chauhan) + git clean -f -d + git checkout master Already on 'master' + git reset --hard origin/master HEAD is now at 1a460b5 HIVE-12372 : Improve to support the multibyte character at lpad and rpad (Shinichi Yamashita via Ashutosh Chauhan) + git merge --ff-only origin/master Already up-to-date. + git gc + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12780066 - PreCommit-HIVE-TRUNK-Build > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-12767.1.patch > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15086549#comment-15086549 ] Hive QA commented on HIVE-12767: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12780780/HIVE-12767.2.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6533/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6533/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6533/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]] + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6533/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + cd apache-github-source-source + git fetch origin >From https://github.com/apache/hive 05e6096..95f2bd8 branch-1 -> origin/branch-1 8069b59..cb17456 master -> origin/master + git reset --hard HEAD HEAD is now at 8069b59 HIVE-12597 : LLAP - allow using elevator without cache (Sergey Shelukhin, reviewed by Prasanth Jayachandran) + git clean -f -d Removing ql/src/test/queries/clientpositive/parquet_join2.q Removing ql/src/test/results/clientpositive/parquet_join2.q.out + git checkout master Already on 'master' Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded. + git reset --hard origin/master HEAD is now at cb17456 HIVE-12796: Switch to 32-bits containers for HMS upgrade tests. + git merge --ff-only origin/master Already up-to-date. + git gc + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch patch: Only garbage was found in the patch input. patch: Only garbage was found in the patch input. patch: Only garbage was found in the patch input. The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12780780 - PreCommit-HIVE-TRUNK-Build > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-12767.2.patch > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089760#comment-15089760 ] Hive QA commented on HIVE-12767: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12780997/HIVE-12767.3.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 10001 tests executed *Failed tests:* {noformat} TestHWISessionManager - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_ppd_char org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_ppd_varchar org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch org.apache.hive.jdbc.TestSSL.testSSLVersion org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles org.apache.hive.spark.client.TestSparkClient.testCounters org.apache.hive.spark.client.TestSparkClient.testErrorJob org.apache.hive.spark.client.TestSparkClient.testJobSubmission org.apache.hive.spark.client.TestSparkClient.testMetricsCollection org.apache.hive.spark.client.TestSparkClient.testRemoteClient org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob org.apache.hive.spark.client.TestSparkClient.testSyncRpc {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6554/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6554/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6554/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 16 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12780997 - PreCommit-HIVE-TRUNK-Build > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-12767.3.patch > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097385#comment-15097385 ] Hive QA commented on HIVE-12767: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12782125/HIVE-12767.4.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10005 tests executed *Failed tests:* {noformat} TestHWISessionManager - did not produce a TEST-*.xml file TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union9 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse org.apache.hive.jdbc.TestSSL.testSSLVersion {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6616/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6616/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6616/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12782125 - PreCommit-HIVE-TRUNK-Build > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-12767.3.patch, HIVE-12767.4.patch > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15131453#comment-15131453 ] Ryan Blue commented on HIVE-12767: -- [~spena], the q tests from the last patch look great and make me confident that the create table and property behavior are correct. I still think that the time zone conversion isn't implementing what it should (see below). I also found a couple of minor things: # Minor: The CTAS test says the zone property on the resulting table should be PST, but is is (and should be) UTC. I think the comment is wrong and the behavior is correct. # It would be nice to validate that the table property always overrides the global option # Is there a test that files are created with the table property in key/value metadata? I think that the copy from one table to another with a different property set wouldn't work otherwise, but I want to make sure. # What is the Hive 2.0 test doing? It looks like it is validating that setting the table property to UTC converts values back to local time? Or was the file written with UTC? (If so, we should have one written in another zone.) The Hive 2.0 test I think demonstrates that the time zone conversion isn't correct. It is correct for the with and without conversion cases. But, it isn't a correct implementation for the meaning of the table property's time zone. In the Hive 2.0 test, it is set to UTC, but the value is read in local (if I'm reading this correctly). Here's the problem: the back-end converts julian/nanos to year/month/day/hour values, then sets those on a calendar. It then uses the backing time in milliseconds to create a Timestamp. That timestamp is in _local_ time because of HIVE-12192, so to have the time _not_ adjusted, the calendar needs to also use the local zone. So using the local zone ends up not adjusting the value. The problem is that the current patch uses this implementation to adjust values to the table property zone. That doesn't work because the "no offset" zone for the table property is UTC, not the local zone. I'm attaching a file that demonstrates the problem. It uses [Spark's implementation of fromJulianDay|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala#L180] to demonstrate what I'm talking about. I chose a date/time at random and tested out the two implementations. Tests that show _correct_ behavior: * {{testKnownTimestampWithoutConversion}} shows that when UTC is used, the fromJulianDay value is the same as the "skip conversion" value returned by NanoTimeUtils. * {{testKnownTimestampWithConversion}} shows that when the local zone is used (for me, PST), the fromJulianDay value is the same as the "do not skip conversion" value returned by NanoTimeUtils. Tests that show _incorrect_ behavior: * {{testKnownWithZoneArgumentUTC}} shows that passing a UTC calendar to NanoTimeUtils does not match the fromJulianDay/UTC value that matched the "skip conversion" in tests above. This doesn't pass, but should. * {{testKnownWithWrongZone}} shows that passing a PST calendar to NanoTimeUtils matches the fromJulianDay/UTC value that corresponds to "skip conversion". This shouldn't pass, but does. I'm attaching my test to this and can help if you have any questions about it. > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-12767.3.patch, HIVE-12767.4.patch > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global