[jira] [Commented] (HIVE-13773) Stats state is not captured correctly in dynpart_sort_optimization_acid.q
[
https://issues.apache.org/jira/browse/HIVE-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15290459#comment-15290459
]
Hive QA commented on HIVE-13773:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804536/HIVE-13773.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 92 failed/errored test(s), 10068 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_join1.q-schema_evol_text_vec_mapwork_part_all_complex.q-vector_complex_join.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-order.q-auto_join18_multi_distinct.q-union2.q-and-12-more -
did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join21
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cross_product_check_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cte_mat_4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_llapdecider
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part_all_primitive
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_script_env_var1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_fsstat
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union_with_udf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_interval_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_null_projection
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_12
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_ptf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_shufflejoin
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby3_map
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert_into1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin_test_outer
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_pcr
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_skewjoinopt8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_udf_max
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_short_regress
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tez.
[jira] [Updated] (HIVE-13788) hive msck listpartitions need to make use of directSQL instead of datanucleus
[ https://issues.apache.org/jira/browse/HIVE-13788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-13788: Attachment: msck_stack_trace.png > hive msck listpartitions need to make use of directSQL instead of datanucleus > - > > Key: HIVE-13788 > URL: https://issues.apache.org/jira/browse/HIVE-13788 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Minor > Attachments: msck_stack_trace.png > > > Currently, for tables having 1000s of partitions too many DB calls are made > via datanucleus. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13520) Don't allow any test to run for longer than 60minutes in the ptest setup
[ https://issues.apache.org/jira/browse/HIVE-13520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15290403#comment-15290403 ] Siddharth Seth commented on HIVE-13520: --- [~ashutoshc] - could you please take a look again. I'm not bothering with submitting this patch to jenkins since it isn't used till a restart, and does not fail tests. The unit tests for testutils/ptest2 pass except for the standard failure related to enabling/disabling spark in TestScripts > Don't allow any test to run for longer than 60minutes in the ptest setup > > > Key: HIVE-13520 > URL: https://issues.apache.org/jira/browse/HIVE-13520 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13520.01.txt, HIVE-13520.02.txt, HIVE-13520.03.patch > > > Current timeout for batches is 2hours. This needs to be lowered. 1hour may be > too much as well. We can start with this, and reduce timeouts further. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13520) Don't allow any test to run for longer than 60minutes in the ptest setup
[ https://issues.apache.org/jira/browse/HIVE-13520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13520: -- Issue Type: Sub-task (was: Improvement) Parent: HIVE-13503 > Don't allow any test to run for longer than 60minutes in the ptest setup > > > Key: HIVE-13520 > URL: https://issues.apache.org/jira/browse/HIVE-13520 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13520.01.txt, HIVE-13520.02.txt, HIVE-13520.03.patch > > > Current timeout for batches is 2hours. This needs to be lowered. 1hour may be > too much as well. We can start with this, and reduce timeouts further. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13787) LLAP: bug in recent security patches
[ https://issues.apache.org/jira/browse/HIVE-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13787: Attachment: HIVE-13787.patch [~sseth] can you take a look? I am not sure this fixes the token issue, still testing > LLAP: bug in recent security patches > > > Key: HIVE-13787 > URL: https://issues.apache.org/jira/browse/HIVE-13787 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13787.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13616) Investigate renaming a table without invalidating the column stats
[
https://issues.apache.org/jira/browse/HIVE-13616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15290299#comment-15290299
]
Hive QA commented on HIVE-13616:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804409/HIVE-13616.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 64 failed/errored test(s), 9996 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_sortmerge_join_7.q-orc_merge9.q-tez_union_dynamic_partition.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-constprog_dpp.q-dynamic_partition_pruning.q-vectorization_10.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorized_parquet.q-insert_values_non_partitioned.q-schema_evol_orc_nonvec_mapwork_part.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby2_noskew_multi_distinct.q-vectorization_10.q-list_bucket_dml_2.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby6_map.q-join13.q-join_reorder3.q-and-12-more - did
not produce a TEST-*.xml file
TestSparkCliDriver-join9.q-join_casesensitive.q-filter_join_breaktask.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-multi_insert.q-join5.q-groupby6.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join19
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join22
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join31
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_avro_compression_enabled_native
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_date_udf
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby1_map_nomap
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_sort_skew_1_23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert_into2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join19
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_louter_join_ppr
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_transform
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_subquery_exists
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_timestamp_lazy
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union25
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union29
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union30
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_elt
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_orderby_5
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorized_case
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.
[jira] [Assigned] (HIVE-13384) Failed to create HiveMetaStoreClient object with proxy user when Kerberos enabled
[
https://issues.apache.org/jira/browse/HIVE-13384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bing Li reassigned HIVE-13384:
--
Assignee: Bing Li
> Failed to create HiveMetaStoreClient object with proxy user when Kerberos
> enabled
> -
>
> Key: HIVE-13384
> URL: https://issues.apache.org/jira/browse/HIVE-13384
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Bing Li
>Assignee: Bing Li
>
> I wrote a Java client to talk with HiveMetaStore. (Hive 1.2.0)
> But found that it can't new a HiveMetaStoreClient object successfully via a
> proxy using in Kerberos env.
> ===
> 15/10/13 00:14:38 ERROR transport.TSaslTransport: SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
> ==
> When I debugging on Hive, I found that the error came from open() method in
> HiveMetaStoreClient class.
> Around line 406,
> transport = UserGroupInformation.getCurrentUser().doAs(new
> PrivilegedExceptionAction() { //FAILED, because the current user
> doesn't have the cridential
> But it will work if I change above line to
> transport = UserGroupInformation.getCurrentUser().getRealUser().doAs(new
> PrivilegedExceptionAction() { //PASS
> I found DRILL-3413 fixes this error in Drill side as a workaround. But if I
> submit a mapreduce job via Pig/HCatalog, it runs into the same issue again
> when initialize the object via HCatalog.
> It would be better to fix this issue in Hive side.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-13384) Failed to create HiveMetaStoreClient object with proxy user when Kerberos enabled
[
https://issues.apache.org/jira/browse/HIVE-13384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bing Li resolved HIVE-13384.
Resolution: Won't Fix
> Failed to create HiveMetaStoreClient object with proxy user when Kerberos
> enabled
> -
>
> Key: HIVE-13384
> URL: https://issues.apache.org/jira/browse/HIVE-13384
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Bing Li
>
> I wrote a Java client to talk with HiveMetaStore. (Hive 1.2.0)
> But found that it can't new a HiveMetaStoreClient object successfully via a
> proxy using in Kerberos env.
> ===
> 15/10/13 00:14:38 ERROR transport.TSaslTransport: SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
> ==
> When I debugging on Hive, I found that the error came from open() method in
> HiveMetaStoreClient class.
> Around line 406,
> transport = UserGroupInformation.getCurrentUser().doAs(new
> PrivilegedExceptionAction() { //FAILED, because the current user
> doesn't have the cridential
> But it will work if I change above line to
> transport = UserGroupInformation.getCurrentUser().getRealUser().doAs(new
> PrivilegedExceptionAction() { //PASS
> I found DRILL-3413 fixes this error in Drill side as a workaround. But if I
> submit a mapreduce job via Pig/HCatalog, it runs into the same issue again
> when initialize the object via HCatalog.
> It would be better to fix this issue in Hive side.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13622) WriteSet tracking optimizations
[ https://issues.apache.org/jira/browse/HIVE-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15290196#comment-15290196 ] Alan Gates commented on HIVE-13622: --- +1 > WriteSet tracking optimizations > --- > > Key: HIVE-13622 > URL: https://issues.apache.org/jira/browse/HIVE-13622 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0, 2.1.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-13622.2.patch, HIVE-13622.3.patch, > HIVE-13622.4.patch > > > HIVE-13395 solves the the lost update problem with some inefficiencies. > 1. TxhHandler.OperationType is currently derived from LockType. This doesn't > distinguish between Update and Delete but would be useful. See comments in > TxnHandler. Should be able to pass in Insert/Update/Delete info from client > into TxnHandler. > 2. TxnHandler.addDynamicPartitions() should know the OperationType as well > from the client. It currently extrapolates it from TXN_COMPONENTS. This > works but requires extra SQL statements and is thus less performant. It will > not work multi-stmt txns. See comments in the code. > 3. TxnHandler.checkLock() see more comments around > "isPartOfDynamicPartitionInsert". If TxnHandler knew whether it is being > called as part of an op running with dynamic partitions, it could be more > efficient. In that case we don't have to write to TXN_COMPONENTS at all > during lock acquisition. Conversely, if not running with DynPart then, we > can kill current txn on lock grant rather than wait until commit time. > 4. TxnHandler.addDynamicPartitions() - the insert stmt here should combing > multiple rows into single SQL stmt (but with a limit for extreme cases) > 5. TxnHandler.enqueueLockWithRetry() - this currently adds components that > are only being read to TXN_COMPONENTS. This is useless at best since read > op don't generate anything to compact. For example, delete from T where t1 > in (select c1 from C) - no reason to add C to txn_components but we do. > > All of these require some Thrift changes > Once done, re-enable TestDbTxnHandler2.testWriteSetTracking11() > Also see comments in > [here|https://issues.apache.org/jira/browse/HIVE-13395?focusedCommentId=15271712&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15271712] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13763) Update smart-apply-patch.sh with ability to use patches from git
[
https://issues.apache.org/jira/browse/HIVE-13763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15290138#comment-15290138
]
Alan Gates commented on HIVE-13763:
---
+1
> Update smart-apply-patch.sh with ability to use patches from git
>
>
> Key: HIVE-13763
> URL: https://issues.apache.org/jira/browse/HIVE-13763
> Project: Hive
> Issue Type: Improvement
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: HIVE-13763.patch
>
>
> Currently, the smart-apply-patch.sh doesn't understand git patches. It is
> relatively easy to make it understand patches generated by:
> {code}
> % git format-patch apache/master --stdout > HIVE-999.patch
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13763) Update smart-apply-patch.sh with ability to use patches from git
[
https://issues.apache.org/jira/browse/HIVE-13763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15290137#comment-15290137
]
Alan Gates commented on HIVE-13763:
---
+1
> Update smart-apply-patch.sh with ability to use patches from git
>
>
> Key: HIVE-13763
> URL: https://issues.apache.org/jira/browse/HIVE-13763
> Project: Hive
> Issue Type: Improvement
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: HIVE-13763.patch
>
>
> Currently, the smart-apply-patch.sh doesn't understand git patches. It is
> relatively easy to make it understand patches generated by:
> {code}
> % git format-patch apache/master --stdout > HIVE-999.patch
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13708) Create table should verify datatypes supported by the serde
[
https://issues.apache.org/jira/browse/HIVE-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13708:
-
Status: Patch Available (was: Open)
> Create table should verify datatypes supported by the serde
> ---
>
> Key: HIVE-13708
> URL: https://issues.apache.org/jira/browse/HIVE-13708
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Thejas M Nair
>Assignee: Hari Sankar Sivarama Subramaniyan
>Priority: Critical
> Attachments: HIVE-13708.1.patch, HIVE-13708.2.patch,
> HIVE-13708.3.patch
>
>
> As [~Goldshuv] mentioned in HIVE-.
> Create table with serde such as OpenCSVSerde allows for creation of table
> with columns of arbitrary types. But 'describe table' would still return
> string datatypes, and so does selects on the table.
> This is misleading and would result in users not getting intended results.
> The create table ideally should disallow the creation of such tables with
> unsupported types.
> Example posted by [~Goldshuv] in HIVE- -
> {noformat}
> CREATE EXTERNAL TABLE test (totalprice DECIMAL(38,10))
> ROW FORMAT SERDE 'com.bizo.hive.serde.csv.CSVSerde' with
> serdeproperties ("separatorChar" = ",","quoteChar"= "'","escapeChar"= "\\")
> STORED AS TEXTFILE
> LOCATION ''
> tblproperties ("skip.header.line.count"="1");
> {noformat}
> Now consider this sql:
> hive> select min(totalprice) from test;
> in this case given my data, the result should have been 874.89, but the
> actual result became 11.57 (as it is first according to byte ordering of
> a string type). this is a wrong result.
> hive> desc extended test;
> OK
> o_totalprice string from deserializer
> ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13708) Create table should verify datatypes supported by the serde
[
https://issues.apache.org/jira/browse/HIVE-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13708:
-
Attachment: HIVE-13708.3.patch
> Create table should verify datatypes supported by the serde
> ---
>
> Key: HIVE-13708
> URL: https://issues.apache.org/jira/browse/HIVE-13708
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Thejas M Nair
>Assignee: Hari Sankar Sivarama Subramaniyan
>Priority: Critical
> Attachments: HIVE-13708.1.patch, HIVE-13708.2.patch,
> HIVE-13708.3.patch
>
>
> As [~Goldshuv] mentioned in HIVE-.
> Create table with serde such as OpenCSVSerde allows for creation of table
> with columns of arbitrary types. But 'describe table' would still return
> string datatypes, and so does selects on the table.
> This is misleading and would result in users not getting intended results.
> The create table ideally should disallow the creation of such tables with
> unsupported types.
> Example posted by [~Goldshuv] in HIVE- -
> {noformat}
> CREATE EXTERNAL TABLE test (totalprice DECIMAL(38,10))
> ROW FORMAT SERDE 'com.bizo.hive.serde.csv.CSVSerde' with
> serdeproperties ("separatorChar" = ",","quoteChar"= "'","escapeChar"= "\\")
> STORED AS TEXTFILE
> LOCATION ''
> tblproperties ("skip.header.line.count"="1");
> {noformat}
> Now consider this sql:
> hive> select min(totalprice) from test;
> in this case given my data, the result should have been 874.89, but the
> actual result became 11.57 (as it is first according to byte ordering of
> a string type). this is a wrong result.
> hive> desc extended test;
> OK
> o_totalprice string from deserializer
> ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13708) Create table should verify datatypes supported by the serde
[
https://issues.apache.org/jira/browse/HIVE-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13708:
-
Status: Open (was: Patch Available)
> Create table should verify datatypes supported by the serde
> ---
>
> Key: HIVE-13708
> URL: https://issues.apache.org/jira/browse/HIVE-13708
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Thejas M Nair
>Assignee: Hari Sankar Sivarama Subramaniyan
>Priority: Critical
> Attachments: HIVE-13708.1.patch, HIVE-13708.2.patch,
> HIVE-13708.3.patch
>
>
> As [~Goldshuv] mentioned in HIVE-.
> Create table with serde such as OpenCSVSerde allows for creation of table
> with columns of arbitrary types. But 'describe table' would still return
> string datatypes, and so does selects on the table.
> This is misleading and would result in users not getting intended results.
> The create table ideally should disallow the creation of such tables with
> unsupported types.
> Example posted by [~Goldshuv] in HIVE- -
> {noformat}
> CREATE EXTERNAL TABLE test (totalprice DECIMAL(38,10))
> ROW FORMAT SERDE 'com.bizo.hive.serde.csv.CSVSerde' with
> serdeproperties ("separatorChar" = ",","quoteChar"= "'","escapeChar"= "\\")
> STORED AS TEXTFILE
> LOCATION ''
> tblproperties ("skip.header.line.count"="1");
> {noformat}
> Now consider this sql:
> hive> select min(totalprice) from test;
> in this case given my data, the result should have been 874.89, but the
> actual result became 11.57 (as it is first according to byte ordering of
> a string type). this is a wrong result.
> hive> desc extended test;
> OK
> o_totalprice string from deserializer
> ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13750) Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer when possible
[
https://issues.apache.org/jira/browse/HIVE-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15290128#comment-15290128
]
Ashutosh Chauhan commented on HIVE-13750:
-
Compiler side changes look good. I left some comments on RB.
But I wonder if this breaks any assumption for FS operator about order in which
it expects rows to arrive to be written out. Since earlier all rows for a
corresponding partition in a Reducer needs to come sorted in a single batch,
but now they may come sorted but in multiple batches. [~prasanth_j] Can you
also please take a look at patch and comment?
> Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer
> when possible
> --
>
> Key: HIVE-13750
> URL: https://issues.apache.org/jira/browse/HIVE-13750
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13750.01.patch, HIVE-13750.02.patch,
> HIVE-13750.patch, HIVE-13750.patch
>
>
> Extend ReduceDedup to remove additional shuffle stage created by sorted
> dynamic partition optimizer when possible, thus avoiding unnecessary work.
> By [~ashutoshc]:
> {quote}
> Currently, if config is on Sorted Dynamic Partition Optimizer (SDPO)
> unconditionally adds an extra shuffle stage. If sort columns of previous
> shuffle and partitioning columns of table match, reduce sink deduplication
> optimizer removes extra shuffle stage, thus bringing down overhead to zero.
> However, if they don’t match, we end up doing extra shuffle. This can be
> improved since we can add table partition columns as a sort columns on
> earlier shuffle and avoid this extra shuffle. This ensures that in cases
> query already has a shuffle stage, we are not shuffling data again.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13269) Simplify comparison expressions using column stats
[
https://issues.apache.org/jira/browse/HIVE-13269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15290105#comment-15290105
]
Hive QA commented on HIVE-13269:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804461/HIVE-13269.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 63 failed/errored test(s), 1 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_join30.q-vector_decimal_10_0.q-acid_globallimit.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_sortmerge_join_7.q-orc_merge9.q-tez_union_dynamic_partition.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-schema_evol_text_nonvec_mapwork_table.q-vector_decimal_trailing.q-subquery_in.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-script_pipe.q-vector_decimal_aggregate.q-vector_data_types.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-smb_cache.q-transform_ppr2.q-vector_outer_join0.q-and-5-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby2_noskew_multi_distinct.q-vectorization_10.q-list_bucket_dml_2.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_vc.q-input1_limit.q-join16.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-table_access_keys_stats.q-bucketsortoptimize_insert_4.q-runtime_skewjoin_mapjoin_spark.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_serde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_remove_exprs_stats
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cte_4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_all_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_empty_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_groupby1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_groupby3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_into2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_partition_column_names_with_leading_and_trailing_spaces
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ptf_streaming
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part_all_primitive
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_inner_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_non_string_partition
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query88
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_rc
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_skewjoin
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_17
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStoreStatsMerge.testStatsMerge
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testA
[jira] [Updated] (HIVE-13443) LLAP: signing for the second state of submit (the event)
[ https://issues.apache.org/jira/browse/HIVE-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13443: Attachment: (was: HIVE-13443.WIP.patch) > LLAP: signing for the second state of submit (the event) > > > Key: HIVE-13443 > URL: https://issues.apache.org/jira/browse/HIVE-13443 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13443.WIP.nogen.patch, HIVE-13443.patch, > HIVE-13443.wo.13444.13675.nogen.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13443) LLAP: signing for the second state of submit (the event)
[ https://issues.apache.org/jira/browse/HIVE-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13443: Attachment: HIVE-13443.patch HIVE-13443.wo.13444.13675.nogen.patch Rebased the patch. > LLAP: signing for the second state of submit (the event) > > > Key: HIVE-13443 > URL: https://issues.apache.org/jira/browse/HIVE-13443 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13443.WIP.nogen.patch, HIVE-13443.patch, > HIVE-13443.wo.13444.13675.nogen.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13675) LLAP: add HMAC signatures to LLAPIF splits
[ https://issues.apache.org/jira/browse/HIVE-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13675: Attachment: HIVE-13675.01.patch HIVE-13675.wo.13444.patch Rebased the patch, added the missing functionality based on recently committed and updated patches. > LLAP: add HMAC signatures to LLAPIF splits > -- > > Key: HIVE-13675 > URL: https://issues.apache.org/jira/browse/HIVE-13675 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13675.01.patch, HIVE-13675.WIP.patch, > HIVE-13675.wo.13444.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13723) Executing join query on type Float using Thrift Serde will result in Float cast to Double error
[
https://issues.apache.org/jira/browse/HIVE-13723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-13723:
Issue Type: Sub-task (was: Bug)
Parent: HIVE-12427
> Executing join query on type Float using Thrift Serde will result in Float
> cast to Double error
> ---
>
> Key: HIVE-13723
> URL: https://issues.apache.org/jira/browse/HIVE-13723
> Project: Hive
> Issue Type: Sub-task
> Components: Serializers/Deserializers
>Affects Versions: 2.1.0
>Reporter: Ziyang Zhao
>
> After enable thrift Serde, execute the following queries in beeline,
> >create table test1 (a int);
> >create table test2 (b float);
> >insert into test1 values (1);
> >insert into test2 values (1);
> >select * from test1 join test2 on test1.a=test2.b;
> this will give the error:
> java.lang.Exception: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"b":1.0}
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> [hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"b":1.0}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:168)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[?:1.7.0_95]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[?:1.7.0_95]
> at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_95]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"b":1.0}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:568)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:159)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[?:1.7.0_95]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[?:1.7.0_95]
> at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_95]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected
> exception from MapJoinOperator :
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.ClassCastException:
> java.lang.Float cannot be cast to java.lang.Double
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:454)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:126)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
>
[jira] [Assigned] (HIVE-13191) DummyTable map joins mix up columns between tables
[
https://issues.apache.org/jira/browse/HIVE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong reassigned HIVE-13191:
--
Assignee: Pengcheng Xiong (was: Vikram Dixit K)
> DummyTable map joins mix up columns between tables
> --
>
> Key: HIVE-13191
> URL: https://issues.apache.org/jira/browse/HIVE-13191
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Gopal V
>Assignee: Pengcheng Xiong
> Attachments: tez.q
>
>
> {code}
> SELECT
> a.key,
> a.a_one,
> b.b_one,
> a.a_zero,
> b.b_zero
> FROM
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 a_one,
> 0 a_zero
> ) a
> LEFT JOIN
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 b_one,
> 0 b_zero
> ) b
> ON a.key = b.key
> ;
> 11 1 0 0 1
> {code}
> This should be 11, 1, 1, 0, 0 instead.
> Disabling map-joins & using shuffle-joins returns the right result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13770) Improve Thrift result set streaming when serializing thrift ResultSets in tasks
[ https://issues.apache.org/jira/browse/HIVE-13770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15290014#comment-15290014 ] Vaibhav Gumashta commented on HIVE-13770: - cc [~thejas] [~gopalv] > Improve Thrift result set streaming when serializing thrift ResultSets in > tasks > --- > > Key: HIVE-13770 > URL: https://issues.apache.org/jira/browse/HIVE-13770 > Project: Hive > Issue Type: Sub-task >Reporter: Holman Lan > > When serializing the Thrift result set in final task, i.e. the > hive.server2.thrift.resultset.serialize.in.tasks property is set to true, HS2 > does not start sending the results until the entire result set has been > written to HDFS. > This is not efficient and we should find a way for HS2 to start sending the > results as soon as a block of result becomes available. The advantage for > this is two folds. One, the client can start consuming the results much > sooner. Two, we can start reclaiming the storage space in HDFS used by a > particular result set block as soon as the result set block has been > successfully sent to the client. > It's worth checking if this is also the case when not serializing the Thrift > result set in final task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13770) Improve Thrift result set streaming when serializing thrift ResultSets in tasks
[ https://issues.apache.org/jira/browse/HIVE-13770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-13770: Issue Type: Sub-task (was: Improvement) Parent: HIVE-12427 > Improve Thrift result set streaming when serializing thrift ResultSets in > tasks > --- > > Key: HIVE-13770 > URL: https://issues.apache.org/jira/browse/HIVE-13770 > Project: Hive > Issue Type: Sub-task >Reporter: Holman Lan > > When serializing the Thrift result set in final task, i.e. the > hive.server2.thrift.resultset.serialize.in.tasks property is set to true, HS2 > does not start sending the results until the entire result set has been > written to HDFS. > This is not efficient and we should find a way for HS2 to start sending the > results as soon as a block of result becomes available. The advantage for > this is two folds. One, the client can start consuming the results much > sooner. Two, we can start reclaiming the storage space in HDFS used by a > particular result set block as soon as the result set block has been > successfully sent to the client. > It's worth checking if this is also the case when not serializing the Thrift > result set in final task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13191) DummyTable map joins mix up columns between tables
[
https://issues.apache.org/jira/browse/HIVE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K updated HIVE-13191:
--
Attachment: tez.q
Along with this addition
{code}
diff --git a/itests/src/test/resources/testconfiguration.properties
b/itests/src/test/resources/testconfiguration.properties
index c891d40..a499b0a 100644
--- a/itests/src/test/resources/testconfiguration.properties
+++ b/itests/src/test/resources/testconfiguration.properties
@@ -70,6 +70,7 @@ disabled.query.files=ql_rewrite_gbtoidx.q,\
smb_mapjoin_8.q
minitez.query.files.shared=acid_globallimit.q,\
+ tez.q,\
deleteAnalyze.q,\
empty_join.q,\
alter_merge_2_orc.q,\
{code}
> DummyTable map joins mix up columns between tables
> --
>
> Key: HIVE-13191
> URL: https://issues.apache.org/jira/browse/HIVE-13191
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Gopal V
>Assignee: Vikram Dixit K
> Attachments: tez.q
>
>
> {code}
> SELECT
> a.key,
> a.a_one,
> b.b_one,
> a.a_zero,
> b.b_zero
> FROM
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 a_one,
> 0 a_zero
> ) a
> LEFT JOIN
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 b_one,
> 0 b_zero
> ) b
> ON a.key = b.key
> ;
> 11 1 0 0 1
> {code}
> This should be 11, 1, 1, 0, 0 instead.
> Disabling map-joins & using shuffle-joins returns the right result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13782) Compile async query asynchronously
[ https://issues.apache.org/jira/browse/HIVE-13782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-13782: --- Status: Patch Available (was: Open) > Compile async query asynchronously > -- > > Key: HIVE-13782 > URL: https://issues.apache.org/jira/browse/HIVE-13782 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-13782.1.patch > > > Currently, when an async query is submitted to HS2, HS2 does the preparation > synchronously. One of the preparation step is to compile the query, which may > take some time. It will be helpful to provide an option to do the compilation > asynchronously. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13539) HiveHFileOutputFormat searching the wrong directory for HFiles
[
https://issues.apache.org/jira/browse/HIVE-13539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289928#comment-15289928
]
Matt McCline commented on HIVE-13539:
-
[~timrobertson100] Can you write a Unit Test and/or Q file (e.g. using your
repro) that verifies this fix? I'm looking at reviewing this change so it get
committed.
> HiveHFileOutputFormat searching the wrong directory for HFiles
> --
>
> Key: HIVE-13539
> URL: https://issues.apache.org/jira/browse/HIVE-13539
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Affects Versions: 1.1.0
> Environment: Built into CDH 5.4.7
>Reporter: Tim Robertson
>Assignee: Tim Robertson
>Priority: Blocker
>
> When creating HFiles for a bulkload in HBase I believe it is looking in the
> wrong directory to find the HFiles, resulting in the following exception:
> {code}
> Error: java.lang.RuntimeException: Hive Runtime Error while closing
> operators: java.io.IOException: Multiple family directories found in
> hdfs://c1n1.gbif.org:8020/user/hive/warehouse/tim.db/coords_hbase/_temporary/2/_temporary
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:295)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:453)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: Multiple family directories found in
> hdfs://c1n1.gbif.org:8020/user/hive/warehouse/tim.db/coords_hbase/_temporary/2/_temporary
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.closeWriters(FileSinkOperator.java:188)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:958)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:598)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:287)
> ... 7 more
> Caused by: java.io.IOException: Multiple family directories found in
> hdfs://c1n1.gbif.org:8020/user/hive/warehouse/tim.db/coords_hbase/_temporary/2/_temporary
> at
> org.apache.hadoop.hive.hbase.HiveHFileOutputFormat$1.close(HiveHFileOutputFormat.java:158)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.closeWriters(FileSinkOperator.java:185)
> ... 11 more
> {code}
> The issue is that is looks for the HFiles in
> {{hdfs://c1n1.gbif.org:8020/user/hive/warehouse/tim.db/coords_hbase/_temporary/2/_temporary}}
> when I believe it should be looking in the task attempt subfolder, such as
> {{hdfs://c1n1.gbif.org:8020/user/hive/warehouse/tim.db/coords_hbase/_temporary/2/_temporary/attempt_1461004169450_0002_r_00_1000}}.
> This can be reproduced in any HFile creation such as:
> {code:sql}
> CREATE TABLE coords_hbase(id INT, x DOUBLE, y DOUBLE)
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
> 'hbase.columns.mapping' = ':key,o:x,o:y',
> 'hbase.table.default.storage.type' = 'binary');
> SET hfile.family.path=/tmp/coords_hfiles/o;
> SET hive.hbase.generatehfiles=true;
> INSERT OVERWRITE TABLE coords_hbase
> SELECT id, decimalLongitude, decimalLatitude
> FROM source
> CLUSTER BY id;
> {code}
> Any advice greatly appreciated
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13750) Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer when possible
[
https://issues.apache.org/jira/browse/HIVE-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289902#comment-15289902
]
Jesus Camacho Rodriguez commented on HIVE-13750:
Regenerated two q files in new patch:
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
{noformat}
[~ashutoshc], could you take a look? thanks!
> Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer
> when possible
> --
>
> Key: HIVE-13750
> URL: https://issues.apache.org/jira/browse/HIVE-13750
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13750.01.patch, HIVE-13750.02.patch,
> HIVE-13750.patch, HIVE-13750.patch
>
>
> Extend ReduceDedup to remove additional shuffle stage created by sorted
> dynamic partition optimizer when possible, thus avoiding unnecessary work.
> By [~ashutoshc]:
> {quote}
> Currently, if config is on Sorted Dynamic Partition Optimizer (SDPO)
> unconditionally adds an extra shuffle stage. If sort columns of previous
> shuffle and partitioning columns of table match, reduce sink deduplication
> optimizer removes extra shuffle stage, thus bringing down overhead to zero.
> However, if they don’t match, we end up doing extra shuffle. This can be
> improved since we can add table partition columns as a sort columns on
> earlier shuffle and avoid this extra shuffle. This ensures that in cases
> query already has a shuffle stage, we are not shuffling data again.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13750) Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer when possible
[
https://issues.apache.org/jira/browse/HIVE-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13750:
---
Status: Patch Available (was: In Progress)
> Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer
> when possible
> --
>
> Key: HIVE-13750
> URL: https://issues.apache.org/jira/browse/HIVE-13750
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13750.01.patch, HIVE-13750.02.patch,
> HIVE-13750.patch, HIVE-13750.patch
>
>
> Extend ReduceDedup to remove additional shuffle stage created by sorted
> dynamic partition optimizer when possible, thus avoiding unnecessary work.
> By [~ashutoshc]:
> {quote}
> Currently, if config is on Sorted Dynamic Partition Optimizer (SDPO)
> unconditionally adds an extra shuffle stage. If sort columns of previous
> shuffle and partitioning columns of table match, reduce sink deduplication
> optimizer removes extra shuffle stage, thus bringing down overhead to zero.
> However, if they don’t match, we end up doing extra shuffle. This can be
> improved since we can add table partition columns as a sort columns on
> earlier shuffle and avoid this extra shuffle. This ensures that in cases
> query already has a shuffle stage, we are not shuffling data again.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-13750) Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer when possible
[
https://issues.apache.org/jira/browse/HIVE-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-13750 started by Jesus Camacho Rodriguez.
--
> Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer
> when possible
> --
>
> Key: HIVE-13750
> URL: https://issues.apache.org/jira/browse/HIVE-13750
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13750.01.patch, HIVE-13750.02.patch,
> HIVE-13750.patch, HIVE-13750.patch
>
>
> Extend ReduceDedup to remove additional shuffle stage created by sorted
> dynamic partition optimizer when possible, thus avoiding unnecessary work.
> By [~ashutoshc]:
> {quote}
> Currently, if config is on Sorted Dynamic Partition Optimizer (SDPO)
> unconditionally adds an extra shuffle stage. If sort columns of previous
> shuffle and partitioning columns of table match, reduce sink deduplication
> optimizer removes extra shuffle stage, thus bringing down overhead to zero.
> However, if they don’t match, we end up doing extra shuffle. This can be
> improved since we can add table partition columns as a sort columns on
> earlier shuffle and avoid this extra shuffle. This ensures that in cases
> query already has a shuffle stage, we are not shuffling data again.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13750) Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer when possible
[
https://issues.apache.org/jira/browse/HIVE-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13750:
---
Attachment: HIVE-13750.02.patch
> Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer
> when possible
> --
>
> Key: HIVE-13750
> URL: https://issues.apache.org/jira/browse/HIVE-13750
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13750.01.patch, HIVE-13750.02.patch,
> HIVE-13750.patch, HIVE-13750.patch
>
>
> Extend ReduceDedup to remove additional shuffle stage created by sorted
> dynamic partition optimizer when possible, thus avoiding unnecessary work.
> By [~ashutoshc]:
> {quote}
> Currently, if config is on Sorted Dynamic Partition Optimizer (SDPO)
> unconditionally adds an extra shuffle stage. If sort columns of previous
> shuffle and partitioning columns of table match, reduce sink deduplication
> optimizer removes extra shuffle stage, thus bringing down overhead to zero.
> However, if they don’t match, we end up doing extra shuffle. This can be
> improved since we can add table partition columns as a sort columns on
> earlier shuffle and avoid this extra shuffle. This ensures that in cases
> query already has a shuffle stage, we are not shuffling data again.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13750) Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer when possible
[
https://issues.apache.org/jira/browse/HIVE-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13750:
---
Status: Open (was: Patch Available)
> Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer
> when possible
> --
>
> Key: HIVE-13750
> URL: https://issues.apache.org/jira/browse/HIVE-13750
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13750.01.patch, HIVE-13750.patch, HIVE-13750.patch
>
>
> Extend ReduceDedup to remove additional shuffle stage created by sorted
> dynamic partition optimizer when possible, thus avoiding unnecessary work.
> By [~ashutoshc]:
> {quote}
> Currently, if config is on Sorted Dynamic Partition Optimizer (SDPO)
> unconditionally adds an extra shuffle stage. If sort columns of previous
> shuffle and partitioning columns of table match, reduce sink deduplication
> optimizer removes extra shuffle stage, thus bringing down overhead to zero.
> However, if they don’t match, we end up doing extra shuffle. This can be
> improved since we can add table partition columns as a sort columns on
> earlier shuffle and avoid this extra shuffle. This ensures that in cases
> query already has a shuffle stage, we are not shuffling data again.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13786) Fix the unit test failure org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
[ https://issues.apache.org/jira/browse/HIVE-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-13786: Status: Patch Available (was: Open) > Fix the unit test failure > org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle > - > > Key: HIVE-13786 > URL: https://issues.apache.org/jira/browse/HIVE-13786 > Project: Hive > Issue Type: Sub-task > Components: Test >Affects Versions: 2.1.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13786.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13786) Fix the unit test failure org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
[ https://issues.apache.org/jira/browse/HIVE-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-13786: Attachment: HIVE-13786.patch Change the type to long to fix the test failure. > Fix the unit test failure > org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle > - > > Key: HIVE-13786 > URL: https://issues.apache.org/jira/browse/HIVE-13786 > Project: Hive > Issue Type: Sub-task > Components: Test >Affects Versions: 2.1.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13786.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13730) Avoid double spilling the same partition when memory threshold is set very low
[
https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289834#comment-15289834
]
Wei Zheng commented on HIVE-13730:
--
I just observed the Hive QA run difference before and after this patch was
committed. It can be seen that this patch got rid of three sets of xml error
mesages.
Before (HIVE-6131:
https://issues.apache.org/jira/browse/HIVE-6131?focusedCommentId=15289556&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15289556)
{code}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_sortmerge_join_7.q-orc_merge9.q-tez_union_dynamic_partition.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-union5.q-and-12-more - did not
produce a TEST-*.xml file
TestMiniTezCliDriver-load_dyn_part2.q-selectDistinctStar.q-vector_decimal_5.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-mapjoin_mapjoin.q-insert_into1.q-vector_decimal_2.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_distinct_2.q-tez_joins_explain.q-cte_mat_1.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_interval_2.q-schema_evol_text_nonvec_mapwork_part_all_primitive.q-tez_fsstat.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorized_parquet.q-insert_values_non_partitioned.q-schema_evol_orc_nonvec_mapwork_part.q-and-12-more
- did not produce a TEST-*.xml file
{code}
After (HIVE-13750:
https://issues.apache.org/jira/browse/HIVE-13750?focusedCommentId=15289795&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15289795)
{code}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-constprog_dpp.q-dynamic_partition_pruning.q-vectorization_10.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-order_null.q-vector_acid3.q-orc_merge10.q-and-12-more -
did not produce a TEST-*.xml file
TestMiniTezCliDriver-tez_union_group_by.q-vector_auto_smb_mapjoin_14.q-union_fast_stats.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_coalesce.q-cbo_windowing.q-tez_join.q-and-12-more -
did not produce a TEST-*.xml file
{code}
> Avoid double spilling the same partition when memory threshold is set very low
> --
>
> Key: HIVE-13730
> URL: https://issues.apache.org/jira/browse/HIVE-13730
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 2.1.0
>Reporter: Vikram Dixit K
>Assignee: Wei Zheng
>Priority: Blocker
> Fix For: 2.1.0
>
> Attachments: HIVE-13730.1.patch, HIVE-13730.2.patch,
> HIVE-13730.3.patch, HIVE-13730.4.patch
>
>
> I am seeing hybridgrace_hashjoin_1.q getting stuck on master.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13783) No secondary prompt
[
https://issues.apache.org/jira/browse/HIVE-13783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-13783:
---
Status: Patch Available (was: In Progress)
> No secondary prompt
> ---
>
> Key: HIVE-13783
> URL: https://issues.apache.org/jira/browse/HIVE-13783
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Affects Versions: 2.0.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-13783.01.patch
>
>
> {noformat}
> # beeline -u jdbc:hive2://localhost:1
> [...]
> Beeline version 1.1.0-cdh5.4.5 by Apache Hive
> 0: jdbc:hive2://localhost:1> "
> 0: jdbc:hive2://localhost:1> select * from foo;
> Error: Error while compiling statement: FAILED: ParseException line 2:17
> character '' not supported here (state=42000,code=4)
> 0: jdbc:hive2://localhost:1>
> {noformat}
> After (accidentally) entering a lonely quote character on its own line and
> pressing Enter, I get back the normal prompt. This easily makes me believe
> I'm about to type a new command from scratch, e.g. a select query as in the
> example, which ends up not working due to parsing error.
> Expected behavior: When a previous command is continued, or a quote is opened
> or anything like this, a differently looking secondary prompt should be
> displayed rather than the normal prompt; as this is done in e.g. hive,
> impala, mysql, bash..., e.g.:
> {noformat}
> # beeline -u jdbc:hive2://localhost:1
> [...]
> Beeline version 1.1.0-cdh5.4.5 by Apache Hive
> 0: jdbc:hive2://localhost:1> "
>> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13783) No secondary prompt
[
https://issues.apache.org/jira/browse/HIVE-13783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-13783:
---
Attachment: HIVE-13783.01.patch
Change is very small and trivial. Ran beeline unit tests successfully.
> No secondary prompt
> ---
>
> Key: HIVE-13783
> URL: https://issues.apache.org/jira/browse/HIVE-13783
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Affects Versions: 2.0.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-13783.01.patch
>
>
> {noformat}
> # beeline -u jdbc:hive2://localhost:1
> [...]
> Beeline version 1.1.0-cdh5.4.5 by Apache Hive
> 0: jdbc:hive2://localhost:1> "
> 0: jdbc:hive2://localhost:1> select * from foo;
> Error: Error while compiling statement: FAILED: ParseException line 2:17
> character '' not supported here (state=42000,code=4)
> 0: jdbc:hive2://localhost:1>
> {noformat}
> After (accidentally) entering a lonely quote character on its own line and
> pressing Enter, I get back the normal prompt. This easily makes me believe
> I'm about to type a new command from scratch, e.g. a select query as in the
> example, which ends up not working due to parsing error.
> Expected behavior: When a previous command is continued, or a quote is opened
> or anything like this, a differently looking secondary prompt should be
> displayed rather than the normal prompt; as this is done in e.g. hive,
> impala, mysql, bash..., e.g.:
> {noformat}
> # beeline -u jdbc:hive2://localhost:1
> [...]
> Beeline version 1.1.0-cdh5.4.5 by Apache Hive
> 0: jdbc:hive2://localhost:1> "
>> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13782) Compile async query asynchronously
[ https://issues.apache.org/jira/browse/HIVE-13782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-13782: --- Attachment: HIVE-13782.1.patch > Compile async query asynchronously > -- > > Key: HIVE-13782 > URL: https://issues.apache.org/jira/browse/HIVE-13782 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-13782.1.patch > > > Currently, when an async query is submitted to HS2, HS2 does the preparation > synchronously. One of the preparation step is to compile the query, which may > take some time. It will be helpful to provide an option to do the compilation > asynchronously. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13068: --- Attachment: HIVE-13068.05.patch > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch, > HIVE-13068.02.patch, HIVE-13068.03.patch, HIVE-13068.04.patch, > HIVE-13068.04.patch, HIVE-13068.05.patch, HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13068: --- Attachment: (was: HIVE-13068.05.patch) > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch, > HIVE-13068.02.patch, HIVE-13068.03.patch, HIVE-13068.04.patch, > HIVE-13068.04.patch, HIVE-13068.05.patch, HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work stopped] (HIVE-13783) No secondary prompt
[
https://issues.apache.org/jira/browse/HIVE-13783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-13783 stopped by Vihang Karajgaonkar.
--
> No secondary prompt
> ---
>
> Key: HIVE-13783
> URL: https://issues.apache.org/jira/browse/HIVE-13783
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Affects Versions: 2.0.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
>
> {noformat}
> # beeline -u jdbc:hive2://localhost:1
> [...]
> Beeline version 1.1.0-cdh5.4.5 by Apache Hive
> 0: jdbc:hive2://localhost:1> "
> 0: jdbc:hive2://localhost:1> select * from foo;
> Error: Error while compiling statement: FAILED: ParseException line 2:17
> character '' not supported here (state=42000,code=4)
> 0: jdbc:hive2://localhost:1>
> {noformat}
> After (accidentally) entering a lonely quote character on its own line and
> pressing Enter, I get back the normal prompt. This easily makes me believe
> I'm about to type a new command from scratch, e.g. a select query as in the
> example, which ends up not working due to parsing error.
> Expected behavior: When a previous command is continued, or a quote is opened
> or anything like this, a differently looking secondary prompt should be
> displayed rather than the normal prompt; as this is done in e.g. hive,
> impala, mysql, bash..., e.g.:
> {noformat}
> # beeline -u jdbc:hive2://localhost:1
> [...]
> Beeline version 1.1.0-cdh5.4.5 by Apache Hive
> 0: jdbc:hive2://localhost:1> "
>> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-13783) No secondary prompt
[
https://issues.apache.org/jira/browse/HIVE-13783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-13783 started by Vihang Karajgaonkar.
--
> No secondary prompt
> ---
>
> Key: HIVE-13783
> URL: https://issues.apache.org/jira/browse/HIVE-13783
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Affects Versions: 2.0.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
>
> {noformat}
> # beeline -u jdbc:hive2://localhost:1
> [...]
> Beeline version 1.1.0-cdh5.4.5 by Apache Hive
> 0: jdbc:hive2://localhost:1> "
> 0: jdbc:hive2://localhost:1> select * from foo;
> Error: Error while compiling statement: FAILED: ParseException line 2:17
> character '' not supported here (state=42000,code=4)
> 0: jdbc:hive2://localhost:1>
> {noformat}
> After (accidentally) entering a lonely quote character on its own line and
> pressing Enter, I get back the normal prompt. This easily makes me believe
> I'm about to type a new command from scratch, e.g. a select query as in the
> example, which ends up not working due to parsing error.
> Expected behavior: When a previous command is continued, or a quote is opened
> or anything like this, a differently looking secondary prompt should be
> displayed rather than the normal prompt; as this is done in e.g. hive,
> impala, mysql, bash..., e.g.:
> {noformat}
> # beeline -u jdbc:hive2://localhost:1
> [...]
> Beeline version 1.1.0-cdh5.4.5 by Apache Hive
> 0: jdbc:hive2://localhost:1> "
>> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-13783) No secondary prompt
[
https://issues.apache.org/jira/browse/HIVE-13783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-13783 started by Vihang Karajgaonkar.
--
> No secondary prompt
> ---
>
> Key: HIVE-13783
> URL: https://issues.apache.org/jira/browse/HIVE-13783
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Affects Versions: 2.0.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
>
> {noformat}
> # beeline -u jdbc:hive2://localhost:1
> [...]
> Beeline version 1.1.0-cdh5.4.5 by Apache Hive
> 0: jdbc:hive2://localhost:1> "
> 0: jdbc:hive2://localhost:1> select * from foo;
> Error: Error while compiling statement: FAILED: ParseException line 2:17
> character '' not supported here (state=42000,code=4)
> 0: jdbc:hive2://localhost:1>
> {noformat}
> After (accidentally) entering a lonely quote character on its own line and
> pressing Enter, I get back the normal prompt. This easily makes me believe
> I'm about to type a new command from scratch, e.g. a select query as in the
> example, which ends up not working due to parsing error.
> Expected behavior: When a previous command is continued, or a quote is opened
> or anything like this, a differently looking secondary prompt should be
> displayed rather than the normal prompt; as this is done in e.g. hive,
> impala, mysql, bash..., e.g.:
> {noformat}
> # beeline -u jdbc:hive2://localhost:1
> [...]
> Beeline version 1.1.0-cdh5.4.5 by Apache Hive
> 0: jdbc:hive2://localhost:1> "
>> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13782) Compile async query asynchronously
[ https://issues.apache.org/jira/browse/HIVE-13782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-13782: --- Priority: Minor (was: Major) > Compile async query asynchronously > -- > > Key: HIVE-13782 > URL: https://issues.apache.org/jira/browse/HIVE-13782 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > > Currently, when an async query is submitted to HS2, HS2 does the preparation > synchronously. One of the preparation step is to compile the query, which may > take some time. It will be helpful to provide an option to do the compilation > asynchronously. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13783) No secondary prompt
[
https://issues.apache.org/jira/browse/HIVE-13783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar reassigned HIVE-13783:
--
Assignee: Vihang Karajgaonkar
> No secondary prompt
> ---
>
> Key: HIVE-13783
> URL: https://issues.apache.org/jira/browse/HIVE-13783
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Affects Versions: 2.0.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
>
> {noformat}
> # beeline -u jdbc:hive2://localhost:1
> [...]
> Beeline version 1.1.0-cdh5.4.5 by Apache Hive
> 0: jdbc:hive2://localhost:1> "
> 0: jdbc:hive2://localhost:1> select * from foo;
> Error: Error while compiling statement: FAILED: ParseException line 2:17
> character '' not supported here (state=42000,code=4)
> 0: jdbc:hive2://localhost:1>
> {noformat}
> After (accidentally) entering a lonely quote character on its own line and
> pressing Enter, I get back the normal prompt. This easily makes me believe
> I'm about to type a new command from scratch, e.g. a select query as in the
> example, which ends up not working due to parsing error.
> Expected behavior: When a previous command is continued, or a quote is opened
> or anything like this, a differently looking secondary prompt should be
> displayed rather than the normal prompt; as this is done in e.g. hive,
> impala, mysql, bash..., e.g.:
> {noformat}
> # beeline -u jdbc:hive2://localhost:1
> [...]
> Beeline version 1.1.0-cdh5.4.5 by Apache Hive
> 0: jdbc:hive2://localhost:1> "
>> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13750) Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer when possible
[
https://issues.apache.org/jira/browse/HIVE-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289795#comment-15289795
]
Hive QA commented on HIVE-13750:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804385/HIVE-13750.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 50 failed/errored test(s), 9228 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-constprog_dpp.q-dynamic_partition_pruning.q-vectorization_10.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-order_null.q-vector_acid3.q-orc_merge10.q-and-12-more -
did not produce a TEST-*.xml file
TestMiniTezCliDriver-tez_union_group_by.q-vector_auto_smb_mapjoin_14.q-union_fast_stats.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_coalesce.q-cbo_windowing.q-tez_join.q-and-12-more -
did not produce a TEST-*.xml file
TestNegativeCliDriver-udf_invalid.q-nopart_insert.q-insert_into_with_schema.q-and-734-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-bucketmapjoin10.q-join_rc.q-skewjoinopt13.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-groupby2_noskew_multi_distinct.q-vectorization_10.q-list_bucket_dml_2.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join9.q-join_casesensitive.q-filter_join_breaktask.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_nullsafe
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_join2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_multi_insert
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ptf_streaming
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_runtime_skewjoin_mapjoin_spark
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union20
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_20
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_9
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableF
[jira] [Updated] (HIVE-13785) Investigate recent test failures.
[ https://issues.apache.org/jira/browse/HIVE-13785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-13785: Description: Seems at least the following tests are failing recently. http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/315/testReport/ org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_22 57 sec 3 org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges 1 min 3 sec 3 org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries 16 min 14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload 5.7 sec 21 org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec 54 ms 21 org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle 15 sec 21 org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure 2.5 sec 21 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 5.6 sec 21 was: Seems at least the following tests are failing recently. org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_22 57 sec 3 org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges 1 min 3 sec 3 org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries 16 min 14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload 5.7 sec 21 org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec 54 ms 21 org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle 15 sec 21 org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure 2.5 sec 21 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 5.6 sec 21 > Investigate recent test failures. > - > > Key: HIVE-13785 > URL: https://issues.apache.org/jira/browse/HIVE-13785 > Project: Hive > Issue Type: Test >Reporter: Aihua Xu > > Seems at least the following tests are failing recently. > http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/315/testReport/ > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_22 > 57 sec 3 > > org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges > 1 min 3 sec 3 > org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries > 16 min 14 > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload 5.7 sec > 21 > org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec > 54 ms 21 > > org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle > 15 sec 21 > > org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure > 2.5 sec 21 > > org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 > 5.6 sec 21 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13699) Make JavaDataModel#get thread safe for parallel compilation
[ https://issues.apache.org/jira/browse/HIVE-13699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289762#comment-15289762 ] Ashutosh Chauhan commented on HIVE-13699: - +1 > Make JavaDataModel#get thread safe for parallel compilation > --- > > Key: HIVE-13699 > URL: https://issues.apache.org/jira/browse/HIVE-13699 > Project: Hive > Issue Type: Bug > Components: HiveServer2, storage-api >Affects Versions: 2.0.0 >Reporter: Peter Slawski >Assignee: Peter Slawski >Priority: Minor > Attachments: HIVE-13699.1.patch, HIVE-13699.2.patch > > > The class JavaDataModel has a static method, #get, that is not thread safe. > This may be an issue when parallel query compilation is enabled because two > threads may attempt to call JavaDataModel#get at the same time, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13444) LLAP: add HMAC signatures to LLAP; verify them on LLAP side
[ https://issues.apache.org/jira/browse/HIVE-13444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289761#comment-15289761 ] Sergey Shelukhin commented on HIVE-13444: - [~sseth] [~jdere] this is now ready for review. Thanks :) > LLAP: add HMAC signatures to LLAP; verify them on LLAP side > --- > > Key: HIVE-13444 > URL: https://issues.apache.org/jira/browse/HIVE-13444 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13444.01.patch, HIVE-13444.WIP.patch, > HIVE-13444.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13444) LLAP: add HMAC signatures to LLAP; verify them on LLAP side
[ https://issues.apache.org/jira/browse/HIVE-13444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13444: Attachment: HIVE-13444.01.patch Rebased the patch and added separate flag handling for local and remote token creation > LLAP: add HMAC signatures to LLAP; verify them on LLAP side > --- > > Key: HIVE-13444 > URL: https://issues.apache.org/jira/browse/HIVE-13444 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13444.01.patch, HIVE-13444.WIP.patch, > HIVE-13444.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13149) Remove some unnecessary HMS connections from HS2
[
https://issues.apache.org/jira/browse/HIVE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-13149:
Attachment: HIVE-13149.8.patch
reattach the patch-8
> Remove some unnecessary HMS connections from HS2
> -
>
> Key: HIVE-13149
> URL: https://issues.apache.org/jira/browse/HIVE-13149
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 2.1.0
>
> Attachments: HIVE-13149.1.patch, HIVE-13149.2.patch,
> HIVE-13149.3.patch, HIVE-13149.4.patch, HIVE-13149.5.patch,
> HIVE-13149.6.patch, HIVE-13149.7.patch, HIVE-13149.8.patch
>
>
> In SessionState class, currently we will always try to get a HMS connection
> in {{start(SessionState startSs, boolean isAsync, LogHelper console)}}
> regardless of if the connection will be used later or not.
> When SessionState is accessed by the tasks in TaskRunner.java, although most
> of the tasks other than some like StatsTask, don't need to access HMS.
> Currently a new HMS connection will be established for each Task thread. If
> HiveServer2 is configured to run in parallel and the query involves many
> tasks, then the connections are created but unused.
> {noformat}
> @Override
> public void run() {
> runner = Thread.currentThread();
> try {
> OperationLog.setCurrentOperationLog(operationLog);
> SessionState.start(ss);
> runSequential();
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10815) Let HiveMetaStoreClient Choose MetaStore Randomly
[ https://issues.apache.org/jira/browse/HIVE-10815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289702#comment-15289702 ] Thejas M Nair commented on HIVE-10815: -- The change looks good to me. bq. Should it be done once per process though, to minimize the number of connections? [~sershe] I am not sure what you mean. There is no connection pooling anyway. With this change, you could have n connections from a process talking to x number of metastore instead of all talking to single metastore. But I don't see that as an issue. > Let HiveMetaStoreClient Choose MetaStore Randomly > - > > Key: HIVE-10815 > URL: https://issues.apache.org/jira/browse/HIVE-10815 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Metastore >Affects Versions: 1.2.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-10815.1.patch, HIVE-10815.patch > > > Currently HiveMetaStoreClient using a fixed order to choose MetaStore URIs > when multiple metastores configured. > Choosing MetaStore Randomly will be good for load balance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10815) Let HiveMetaStoreClient Choose MetaStore Randomly
[ https://issues.apache.org/jira/browse/HIVE-10815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289706#comment-15289706 ] Thejas M Nair commented on HIVE-10815: -- The unit test state is such a mess. Its hard to understand if any of these failures could be related to the change. > Let HiveMetaStoreClient Choose MetaStore Randomly > - > > Key: HIVE-10815 > URL: https://issues.apache.org/jira/browse/HIVE-10815 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Metastore >Affects Versions: 1.2.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-10815.1.patch, HIVE-10815.patch > > > Currently HiveMetaStoreClient using a fixed order to choose MetaStore URIs > when multiple metastores configured. > Choosing MetaStore Randomly will be good for load balance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10815) Let HiveMetaStoreClient Choose MetaStore Randomly
[ https://issues.apache.org/jira/browse/HIVE-10815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289704#comment-15289704 ] Thejas M Nair commented on HIVE-10815: -- The unit test state is such a mess. Its hard to understand if any of these failures could be related to the change. > Let HiveMetaStoreClient Choose MetaStore Randomly > - > > Key: HIVE-10815 > URL: https://issues.apache.org/jira/browse/HIVE-10815 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Metastore >Affects Versions: 1.2.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-10815.1.patch, HIVE-10815.patch > > > Currently HiveMetaStoreClient using a fixed order to choose MetaStore URIs > when multiple metastores configured. > Choosing MetaStore Randomly will be good for load balance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13502) Beeline doesnt support session parameters in JDBC URL as documentation states.
[ https://issues.apache.org/jira/browse/HIVE-13502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-13502: - Status: Open (was: Patch Available) > Beeline doesnt support session parameters in JDBC URL as documentation states. > -- > > Key: HIVE-13502 > URL: https://issues.apache.org/jira/browse/HIVE-13502 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 1.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-13502.1.patch, HIVE-13502.2.patch, > HIVE-13502.3.patch, HIVE-13502.3.patch, HIVE-13502.4.patch, HIVE-13502.patch > > > https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-ConnectionURLs > documents that sessions variables like credentials etc are accepted as part > of the URL. However, Beeline does not support such URLs today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13502) Beeline doesnt support session parameters in JDBC URL as documentation states.
[ https://issues.apache.org/jira/browse/HIVE-13502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-13502: - Attachment: HIVE-13502.5.patch > Beeline doesnt support session parameters in JDBC URL as documentation states. > -- > > Key: HIVE-13502 > URL: https://issues.apache.org/jira/browse/HIVE-13502 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 1.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-13502.1.patch, HIVE-13502.2.patch, > HIVE-13502.3.patch, HIVE-13502.3.patch, HIVE-13502.4.patch, > HIVE-13502.5.patch, HIVE-13502.patch > > > https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-ConnectionURLs > documents that sessions variables like credentials etc are accepted as part > of the URL. However, Beeline does not support such URLs today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13784) Hive Metastore start failed on Oracle DB
[
https://issues.apache.org/jira/browse/HIVE-13784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13784:
-
Resolution: Fixed
Fix Version/s: 2.1.0
Status: Resolved (was: Patch Available)
Thanks [~sershe] for the review.
> Hive Metastore start failed on Oracle DB
>
>
> Key: HIVE-13784
> URL: https://issues.apache.org/jira/browse/HIVE-13784
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
>Priority: Blocker
> Fix For: 2.1.0
>
> Attachments: HIVE-13784.1.patch
>
>
> NO PRECOMMIT TESTS
> During hive metastore start, hive is trying to execute this script
> {{hive-schema-2.1.0.oracle.sql}}:
> {code}
> Metastore connection URL: jdbc:oracle:thin:@//172.22.114.220:1521/XE
> Metastore Connection Driver : oracle.jdbc.driver.OracleDriver
> Metastore connection User: hiveuser
> Starting metastore schema initialization to 2.1.0
> Initialization script hive-schema-2.1.0.oracle.sql
> Connecting to jdbc:oracle:thin:@//172.22.114.220:1521/XE
> Connected to: Oracle (version Oracle Database 11g Express Edition Release
> 11.2.0.2.0 - 64bit Production)
> Driver: Oracle JDBC driver (version 11.2.0.2.0)
> Transaction isolation: TRANSACTION_READ_COMMITTED
> 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> !autocommit on
> Autocommit status: true
> 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> CREATE TABLE SEQUENCE_TABLE (
> SEQ
> UENCE_NAME VARCHAR2(255) NOT NULL, NEXT_VAL NUMBER NOT NULL )
> No rows affected (0.208 seconds)
> ..
> {code}
> but inthe end of this script we are getting Error:
> {code}
> 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> CREATE INDEX
> CONSTRAINTS_PARENT_T
> BL_ID_INDEX ON KEY_CONSTRAINTS(PARENT_TBL_ID)
> Error: ORA-00972: identifier is too long (state=42000,code=972)
> Closing: 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE
> org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
> FAILED! Metastore state would be inconsistent !!
> org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
> FAILED! Metastore state would be inconsistent !!
> at
> org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:291)
> at
> org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:264)
> at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:505)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException: Schema script failed, errorcode 2
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:390)
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:347)
> at
> org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:287)
> ... 8 more
> *** schemaTool failed ***
> {code}
> We have a problem with too long identifier:
> {{CONSTRAINTS_PARENT_TBL_ID_INDEX}} (31 symbol), it should be 30 symbols of
> max length.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13784) Hive Metastore start failed on Oracle DB
[
https://issues.apache.org/jira/browse/HIVE-13784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13784:
-
Description:
NO PRECOMMIT TESTS
During hive metastore start, hive is trying to execute this script
{{hive-schema-2.1.0.oracle.sql}}:
{code}
Metastore connection URL:jdbc:oracle:thin:@//172.22.114.220:1521/XE
Metastore Connection Driver :oracle.jdbc.driver.OracleDriver
Metastore connection User: hiveuser
Starting metastore schema initialization to 2.1.0
Initialization script hive-schema-2.1.0.oracle.sql
Connecting to jdbc:oracle:thin:@//172.22.114.220:1521/XE
Connected to: Oracle (version Oracle Database 11g Express Edition Release
11.2.0.2.0 - 64bit Production)
Driver: Oracle JDBC driver (version 11.2.0.2.0)
Transaction isolation: TRANSACTION_READ_COMMITTED
0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> !autocommit on
Autocommit status: true
0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> CREATE TABLE SEQUENCE_TABLE (
SEQ
UENCE_NAME VARCHAR2(255) NOT NULL, NEXT_VAL NUMBER NOT NULL )
No rows affected (0.208 seconds)
..
{code}
but inthe end of this script we are getting Error:
{code}
0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> CREATE INDEX
CONSTRAINTS_PARENT_T
BL_ID_INDEX ON KEY_CONSTRAINTS(PARENT_TBL_ID)
Error: ORA-00972: identifier is too long (state=42000,code=972)
Closing: 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
FAILED! Metastore state would be inconsistent !!
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
FAILED! Metastore state would be inconsistent !!
at
org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:291)
at
org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:264)
at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:505)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: Schema script failed, errorcode 2
at
org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:390)
at
org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:347)
at
org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:287)
... 8 more
*** schemaTool failed ***
{code}
We have a problem with too long identifier: {{CONSTRAINTS_PARENT_TBL_ID_INDEX}}
(31 symbol), it should be 30 symbols of max length.
was:
During hive metastore start, hive is trying to execute this script
{{hive-schema-2.1.0.oracle.sql}}:
{code}
Metastore connection URL:jdbc:oracle:thin:@//172.22.114.220:1521/XE
Metastore Connection Driver :oracle.jdbc.driver.OracleDriver
Metastore connection User: hiveuser
Starting metastore schema initialization to 2.1.0
Initialization script hive-schema-2.1.0.oracle.sql
Connecting to jdbc:oracle:thin:@//172.22.114.220:1521/XE
Connected to: Oracle (version Oracle Database 11g Express Edition Release
11.2.0.2.0 - 64bit Production)
Driver: Oracle JDBC driver (version 11.2.0.2.0)
Transaction isolation: TRANSACTION_READ_COMMITTED
0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> !autocommit on
Autocommit status: true
0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> CREATE TABLE SEQUENCE_TABLE (
SEQ
UENCE_NAME VARCHAR2(255) NOT NULL, NEXT_VAL NUMBER NOT NULL )
No rows affected (0.208 seconds)
..
{code}
but inthe end of this script we are getting Error:
{code}
0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> CREATE INDEX
CONSTRAINTS_PARENT_T
BL_ID_INDEX ON KEY_CONSTRAINTS(PARENT_TBL_ID)
Error: ORA-00972: identifier is too long (state=42000,code=972)
Closing: 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
FAILED! Metastore state would be inconsistent !!
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
FAILED! Metastore state would be inconsistent !!
at
org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:291)
at
org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:264)
at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:505)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethod
[jira] [Updated] (HIVE-13699) Make JavaDataModel#get thread safe for parallel compilation
[ https://issues.apache.org/jira/browse/HIVE-13699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Slawski updated HIVE-13699: - Attachment: HIVE-13699.2.patch Attached updated patch with fix to use SLF4J logger. > Make JavaDataModel#get thread safe for parallel compilation > --- > > Key: HIVE-13699 > URL: https://issues.apache.org/jira/browse/HIVE-13699 > Project: Hive > Issue Type: Bug > Components: HiveServer2, storage-api >Affects Versions: 2.0.0 >Reporter: Peter Slawski >Assignee: Peter Slawski >Priority: Minor > Attachments: HIVE-13699.1.patch, HIVE-13699.2.patch > > > The class JavaDataModel has a static method, #get, that is not thread safe. > This may be an issue when parallel query compilation is enabled because two > threads may attempt to call JavaDataModel#get at the same time, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-13777) NPE is thrown when select table after change column from string to decimal
[
https://issues.apache.org/jira/browse/HIVE-13777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito resolved HIVE-13777.
---
Resolution: Cannot Reproduce
The issue was not reproduced with later code.
> NPE is thrown when select table after change column from string to decimal
> --
>
> Key: HIVE-13777
> URL: https://issues.apache.org/jira/browse/HIVE-13777
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>
> Run the followings to change a column from string to decimail:
> {noformat}
> drop table if exists
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal;
> create table
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal
> stored as orc as select s, s_dc from schemaevo;
> alter table
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal
> change column s_dc s_dc decimal(12,4);
> select count(*) from
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal;
> {noformat}
> select count works fine but just select throws NPE:
> {noformat}
> 0: jdbc:hive2://os-r6-0517-hiveserver2-1-1.op> select count(*) from
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal;
> +-+--+
> | c0|
> +-+--+
> | 10 |
> +-+--+
> 1 row selected (13.856 seconds)
> 0: jdbc:hive2://os-r6-0517-hiveserver2-1-1.op> select * from
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal;
> Error: java.io.IOException: java.lang.NullPointerException (state=,code=0)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13502) Beeline doesnt support session parameters in JDBC URL as documentation states.
[ https://issues.apache.org/jira/browse/HIVE-13502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-13502: - Status: Patch Available (was: Open) > Beeline doesnt support session parameters in JDBC URL as documentation states. > -- > > Key: HIVE-13502 > URL: https://issues.apache.org/jira/browse/HIVE-13502 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 1.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-13502.1.patch, HIVE-13502.2.patch, > HIVE-13502.3.patch, HIVE-13502.3.patch, HIVE-13502.4.patch, > HIVE-13502.5.patch, HIVE-13502.patch > > > https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-ConnectionURLs > documents that sessions variables like credentials etc are accepted as part > of the URL. However, Beeline does not support such URLs today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13784) Hive Metastore start failed on Oracle DB
[
https://issues.apache.org/jira/browse/HIVE-13784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289657#comment-15289657
]
Sergey Shelukhin commented on HIVE-13784:
-
+1. Another day, another dumb Oracle error :)
> Hive Metastore start failed on Oracle DB
>
>
> Key: HIVE-13784
> URL: https://issues.apache.org/jira/browse/HIVE-13784
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
>Priority: Blocker
> Attachments: HIVE-13784.1.patch
>
>
> During hive metastore start, hive is trying to execute this script
> {{hive-schema-2.1.0.oracle.sql}}:
> {code}
> Metastore connection URL: jdbc:oracle:thin:@//172.22.114.220:1521/XE
> Metastore Connection Driver : oracle.jdbc.driver.OracleDriver
> Metastore connection User: hiveuser
> Starting metastore schema initialization to 2.1.0
> Initialization script hive-schema-2.1.0.oracle.sql
> Connecting to jdbc:oracle:thin:@//172.22.114.220:1521/XE
> Connected to: Oracle (version Oracle Database 11g Express Edition Release
> 11.2.0.2.0 - 64bit Production)
> Driver: Oracle JDBC driver (version 11.2.0.2.0)
> Transaction isolation: TRANSACTION_READ_COMMITTED
> 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> !autocommit on
> Autocommit status: true
> 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> CREATE TABLE SEQUENCE_TABLE (
> SEQ
> UENCE_NAME VARCHAR2(255) NOT NULL, NEXT_VAL NUMBER NOT NULL )
> No rows affected (0.208 seconds)
> ..
> {code}
> but inthe end of this script we are getting Error:
> {code}
> 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> CREATE INDEX
> CONSTRAINTS_PARENT_T
> BL_ID_INDEX ON KEY_CONSTRAINTS(PARENT_TBL_ID)
> Error: ORA-00972: identifier is too long (state=42000,code=972)
> Closing: 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE
> org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
> FAILED! Metastore state would be inconsistent !!
> org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
> FAILED! Metastore state would be inconsistent !!
> at
> org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:291)
> at
> org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:264)
> at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:505)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException: Schema script failed, errorcode 2
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:390)
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:347)
> at
> org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:287)
> ... 8 more
> *** schemaTool failed ***
> {code}
> We have a problem with too long identifier:
> {{CONSTRAINTS_PARENT_TBL_ID_INDEX}} (31 symbol), it should be 30 symbols of
> max length.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13448) LLAP: check ZK acls for ZKSM and fail if they are too permissive
[ https://issues.apache.org/jira/browse/HIVE-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13448: Attachment: HIVE-13448.02.patch Rebased the patch > LLAP: check ZK acls for ZKSM and fail if they are too permissive > > > Key: HIVE-13448 > URL: https://issues.apache.org/jira/browse/HIVE-13448 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13448.01.patch, HIVE-13448.02.patch, > HIVE-13448.patch > > > Only the current user should have any access. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13784) Hive Metastore start failed on Oracle DB
[
https://issues.apache.org/jira/browse/HIVE-13784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289594#comment-15289594
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-13784:
--
[~thejas] or [~ashutoshc] Its a trivial change, can either of you please review
it so that we can close this one ASAP.
Thanks
Hari
> Hive Metastore start failed on Oracle DB
>
>
> Key: HIVE-13784
> URL: https://issues.apache.org/jira/browse/HIVE-13784
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
>Priority: Blocker
> Attachments: HIVE-13784.1.patch
>
>
> During hive metastore start, hive is trying to execute this script
> {{hive-schema-2.1.0.oracle.sql}}:
> {code}
> Metastore connection URL: jdbc:oracle:thin:@//172.22.114.220:1521/XE
> Metastore Connection Driver : oracle.jdbc.driver.OracleDriver
> Metastore connection User: hiveuser
> Starting metastore schema initialization to 2.1.0
> Initialization script hive-schema-2.1.0.oracle.sql
> Connecting to jdbc:oracle:thin:@//172.22.114.220:1521/XE
> Connected to: Oracle (version Oracle Database 11g Express Edition Release
> 11.2.0.2.0 - 64bit Production)
> Driver: Oracle JDBC driver (version 11.2.0.2.0)
> Transaction isolation: TRANSACTION_READ_COMMITTED
> 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> !autocommit on
> Autocommit status: true
> 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> CREATE TABLE SEQUENCE_TABLE (
> SEQ
> UENCE_NAME VARCHAR2(255) NOT NULL, NEXT_VAL NUMBER NOT NULL )
> No rows affected (0.208 seconds)
> ..
> {code}
> but inthe end of this script we are getting Error:
> {code}
> 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> CREATE INDEX
> CONSTRAINTS_PARENT_T
> BL_ID_INDEX ON KEY_CONSTRAINTS(PARENT_TBL_ID)
> Error: ORA-00972: identifier is too long (state=42000,code=972)
> Closing: 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE
> org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
> FAILED! Metastore state would be inconsistent !!
> org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
> FAILED! Metastore state would be inconsistent !!
> at
> org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:291)
> at
> org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:264)
> at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:505)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException: Schema script failed, errorcode 2
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:390)
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:347)
> at
> org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:287)
> ... 8 more
> *** schemaTool failed ***
> {code}
> We have a problem with too long identifier:
> {{CONSTRAINTS_PARENT_TBL_ID_INDEX}} (31 symbol), it should be 30 symbols of
> max length.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13784) Hive Metastore start failed on Oracle DB
[
https://issues.apache.org/jira/browse/HIVE-13784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13784:
-
Status: Patch Available (was: Open)
> Hive Metastore start failed on Oracle DB
>
>
> Key: HIVE-13784
> URL: https://issues.apache.org/jira/browse/HIVE-13784
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
>Priority: Blocker
> Attachments: HIVE-13784.1.patch
>
>
> During hive metastore start, hive is trying to execute this script
> {{hive-schema-2.1.0.oracle.sql}}:
> {code}
> Metastore connection URL: jdbc:oracle:thin:@//172.22.114.220:1521/XE
> Metastore Connection Driver : oracle.jdbc.driver.OracleDriver
> Metastore connection User: hiveuser
> Starting metastore schema initialization to 2.1.0
> Initialization script hive-schema-2.1.0.oracle.sql
> Connecting to jdbc:oracle:thin:@//172.22.114.220:1521/XE
> Connected to: Oracle (version Oracle Database 11g Express Edition Release
> 11.2.0.2.0 - 64bit Production)
> Driver: Oracle JDBC driver (version 11.2.0.2.0)
> Transaction isolation: TRANSACTION_READ_COMMITTED
> 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> !autocommit on
> Autocommit status: true
> 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> CREATE TABLE SEQUENCE_TABLE (
> SEQ
> UENCE_NAME VARCHAR2(255) NOT NULL, NEXT_VAL NUMBER NOT NULL )
> No rows affected (0.208 seconds)
> ..
> {code}
> but inthe end of this script we are getting Error:
> {code}
> 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> CREATE INDEX
> CONSTRAINTS_PARENT_T
> BL_ID_INDEX ON KEY_CONSTRAINTS(PARENT_TBL_ID)
> Error: ORA-00972: identifier is too long (state=42000,code=972)
> Closing: 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE
> org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
> FAILED! Metastore state would be inconsistent !!
> org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
> FAILED! Metastore state would be inconsistent !!
> at
> org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:291)
> at
> org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:264)
> at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:505)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException: Schema script failed, errorcode 2
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:390)
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:347)
> at
> org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:287)
> ... 8 more
> *** schemaTool failed ***
> {code}
> We have a problem with too long identifier:
> {{CONSTRAINTS_PARENT_TBL_ID_INDEX}} (31 symbol), it should be 30 symbols of
> max length.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13784) Hive Metastore start failed on Oracle DB
[
https://issues.apache.org/jira/browse/HIVE-13784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13784:
-
Component/s: Metastore
> Hive Metastore start failed on Oracle DB
>
>
> Key: HIVE-13784
> URL: https://issues.apache.org/jira/browse/HIVE-13784
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
>Priority: Blocker
> Attachments: HIVE-13784.1.patch
>
>
> During hive metastore start, hive is trying to execute this script
> {{hive-schema-2.1.0.oracle.sql}}:
> {code}
> Metastore connection URL: jdbc:oracle:thin:@//172.22.114.220:1521/XE
> Metastore Connection Driver : oracle.jdbc.driver.OracleDriver
> Metastore connection User: hiveuser
> Starting metastore schema initialization to 2.1.0
> Initialization script hive-schema-2.1.0.oracle.sql
> Connecting to jdbc:oracle:thin:@//172.22.114.220:1521/XE
> Connected to: Oracle (version Oracle Database 11g Express Edition Release
> 11.2.0.2.0 - 64bit Production)
> Driver: Oracle JDBC driver (version 11.2.0.2.0)
> Transaction isolation: TRANSACTION_READ_COMMITTED
> 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> !autocommit on
> Autocommit status: true
> 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> CREATE TABLE SEQUENCE_TABLE (
> SEQ
> UENCE_NAME VARCHAR2(255) NOT NULL, NEXT_VAL NUMBER NOT NULL )
> No rows affected (0.208 seconds)
> ..
> {code}
> but inthe end of this script we are getting Error:
> {code}
> 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> CREATE INDEX
> CONSTRAINTS_PARENT_T
> BL_ID_INDEX ON KEY_CONSTRAINTS(PARENT_TBL_ID)
> Error: ORA-00972: identifier is too long (state=42000,code=972)
> Closing: 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE
> org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
> FAILED! Metastore state would be inconsistent !!
> org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
> FAILED! Metastore state would be inconsistent !!
> at
> org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:291)
> at
> org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:264)
> at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:505)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException: Schema script failed, errorcode 2
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:390)
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:347)
> at
> org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:287)
> ... 8 more
> *** schemaTool failed ***
> {code}
> We have a problem with too long identifier:
> {{CONSTRAINTS_PARENT_TBL_ID_INDEX}} (31 symbol), it should be 30 symbols of
> max length.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13784) Hive Metastore start failed on Oracle DB
[
https://issues.apache.org/jira/browse/HIVE-13784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13784:
-
Affects Version/s: 2.1.0
> Hive Metastore start failed on Oracle DB
>
>
> Key: HIVE-13784
> URL: https://issues.apache.org/jira/browse/HIVE-13784
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13784.1.patch
>
>
> During hive metastore start, hive is trying to execute this script
> {{hive-schema-2.1.0.oracle.sql}}:
> {code}
> Metastore connection URL: jdbc:oracle:thin:@//172.22.114.220:1521/XE
> Metastore Connection Driver : oracle.jdbc.driver.OracleDriver
> Metastore connection User: hiveuser
> Starting metastore schema initialization to 2.1.0
> Initialization script hive-schema-2.1.0.oracle.sql
> Connecting to jdbc:oracle:thin:@//172.22.114.220:1521/XE
> Connected to: Oracle (version Oracle Database 11g Express Edition Release
> 11.2.0.2.0 - 64bit Production)
> Driver: Oracle JDBC driver (version 11.2.0.2.0)
> Transaction isolation: TRANSACTION_READ_COMMITTED
> 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> !autocommit on
> Autocommit status: true
> 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> CREATE TABLE SEQUENCE_TABLE (
> SEQ
> UENCE_NAME VARCHAR2(255) NOT NULL, NEXT_VAL NUMBER NOT NULL )
> No rows affected (0.208 seconds)
> ..
> {code}
> but inthe end of this script we are getting Error:
> {code}
> 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> CREATE INDEX
> CONSTRAINTS_PARENT_T
> BL_ID_INDEX ON KEY_CONSTRAINTS(PARENT_TBL_ID)
> Error: ORA-00972: identifier is too long (state=42000,code=972)
> Closing: 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE
> org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
> FAILED! Metastore state would be inconsistent !!
> org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
> FAILED! Metastore state would be inconsistent !!
> at
> org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:291)
> at
> org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:264)
> at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:505)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException: Schema script failed, errorcode 2
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:390)
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:347)
> at
> org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:287)
> ... 8 more
> *** schemaTool failed ***
> {code}
> We have a problem with too long identifier:
> {{CONSTRAINTS_PARENT_TBL_ID_INDEX}} (31 symbol), it should be 30 symbols of
> max length.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13784) Hive Metastore start failed on Oracle DB
[
https://issues.apache.org/jira/browse/HIVE-13784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13784:
-
Attachment: HIVE-13784.1.patch
> Hive Metastore start failed on Oracle DB
>
>
> Key: HIVE-13784
> URL: https://issues.apache.org/jira/browse/HIVE-13784
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
>Priority: Blocker
> Attachments: HIVE-13784.1.patch
>
>
> During hive metastore start, hive is trying to execute this script
> {{hive-schema-2.1.0.oracle.sql}}:
> {code}
> Metastore connection URL: jdbc:oracle:thin:@//172.22.114.220:1521/XE
> Metastore Connection Driver : oracle.jdbc.driver.OracleDriver
> Metastore connection User: hiveuser
> Starting metastore schema initialization to 2.1.0
> Initialization script hive-schema-2.1.0.oracle.sql
> Connecting to jdbc:oracle:thin:@//172.22.114.220:1521/XE
> Connected to: Oracle (version Oracle Database 11g Express Edition Release
> 11.2.0.2.0 - 64bit Production)
> Driver: Oracle JDBC driver (version 11.2.0.2.0)
> Transaction isolation: TRANSACTION_READ_COMMITTED
> 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> !autocommit on
> Autocommit status: true
> 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> CREATE TABLE SEQUENCE_TABLE (
> SEQ
> UENCE_NAME VARCHAR2(255) NOT NULL, NEXT_VAL NUMBER NOT NULL )
> No rows affected (0.208 seconds)
> ..
> {code}
> but inthe end of this script we are getting Error:
> {code}
> 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> CREATE INDEX
> CONSTRAINTS_PARENT_T
> BL_ID_INDEX ON KEY_CONSTRAINTS(PARENT_TBL_ID)
> Error: ORA-00972: identifier is too long (state=42000,code=972)
> Closing: 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE
> org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
> FAILED! Metastore state would be inconsistent !!
> org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
> FAILED! Metastore state would be inconsistent !!
> at
> org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:291)
> at
> org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:264)
> at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:505)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException: Schema script failed, errorcode 2
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:390)
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:347)
> at
> org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:287)
> ... 8 more
> *** schemaTool failed ***
> {code}
> We have a problem with too long identifier:
> {{CONSTRAINTS_PARENT_TBL_ID_INDEX}} (31 symbol), it should be 30 symbols of
> max length.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13784) Hive Metastore start failed on Oracle DB
[
https://issues.apache.org/jira/browse/HIVE-13784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13784:
-
Priority: Blocker (was: Major)
> Hive Metastore start failed on Oracle DB
>
>
> Key: HIVE-13784
> URL: https://issues.apache.org/jira/browse/HIVE-13784
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
>Priority: Blocker
> Attachments: HIVE-13784.1.patch
>
>
> During hive metastore start, hive is trying to execute this script
> {{hive-schema-2.1.0.oracle.sql}}:
> {code}
> Metastore connection URL: jdbc:oracle:thin:@//172.22.114.220:1521/XE
> Metastore Connection Driver : oracle.jdbc.driver.OracleDriver
> Metastore connection User: hiveuser
> Starting metastore schema initialization to 2.1.0
> Initialization script hive-schema-2.1.0.oracle.sql
> Connecting to jdbc:oracle:thin:@//172.22.114.220:1521/XE
> Connected to: Oracle (version Oracle Database 11g Express Edition Release
> 11.2.0.2.0 - 64bit Production)
> Driver: Oracle JDBC driver (version 11.2.0.2.0)
> Transaction isolation: TRANSACTION_READ_COMMITTED
> 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> !autocommit on
> Autocommit status: true
> 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> CREATE TABLE SEQUENCE_TABLE (
> SEQ
> UENCE_NAME VARCHAR2(255) NOT NULL, NEXT_VAL NUMBER NOT NULL )
> No rows affected (0.208 seconds)
> ..
> {code}
> but inthe end of this script we are getting Error:
> {code}
> 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> CREATE INDEX
> CONSTRAINTS_PARENT_T
> BL_ID_INDEX ON KEY_CONSTRAINTS(PARENT_TBL_ID)
> Error: ORA-00972: identifier is too long (state=42000,code=972)
> Closing: 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE
> org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
> FAILED! Metastore state would be inconsistent !!
> org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
> FAILED! Metastore state would be inconsistent !!
> at
> org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:291)
> at
> org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:264)
> at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:505)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException: Schema script failed, errorcode 2
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:390)
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:347)
> at
> org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:287)
> ... 8 more
> *** schemaTool failed ***
> {code}
> We have a problem with too long identifier:
> {{CONSTRAINTS_PARENT_TBL_ID_INDEX}} (31 symbol), it should be 30 symbols of
> max length.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13149) Remove some unnecessary HMS connections from HS2
[
https://issues.apache.org/jira/browse/HIVE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-13149:
Attachment: (was: HIVE-13149.8.patch)
> Remove some unnecessary HMS connections from HS2
> -
>
> Key: HIVE-13149
> URL: https://issues.apache.org/jira/browse/HIVE-13149
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 2.1.0
>
> Attachments: HIVE-13149.1.patch, HIVE-13149.2.patch,
> HIVE-13149.3.patch, HIVE-13149.4.patch, HIVE-13149.5.patch,
> HIVE-13149.6.patch, HIVE-13149.7.patch
>
>
> In SessionState class, currently we will always try to get a HMS connection
> in {{start(SessionState startSs, boolean isAsync, LogHelper console)}}
> regardless of if the connection will be used later or not.
> When SessionState is accessed by the tasks in TaskRunner.java, although most
> of the tasks other than some like StatsTask, don't need to access HMS.
> Currently a new HMS connection will be established for each Task thread. If
> HiveServer2 is configured to run in parallel and the query involves many
> tasks, then the connections are created but unused.
> {noformat}
> @Override
> public void run() {
> runner = Thread.currentThread();
> try {
> OperationLog.setCurrentOperationLog(operationLog);
> SessionState.start(ss);
> runSequential();
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13502) Beeline doesnt support session parameters in JDBC URL as documentation states.
[
https://issues.apache.org/jira/browse/HIVE-13502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289566#comment-15289566
]
Hive QA commented on HIVE-13502:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804348/HIVE-13502.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/317/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/317/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-317/
Messages:
{noformat}
This message was trimmed, see log for full details
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-it-util ---
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/itests/util/target
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/itests/util (includes =
[datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
hive-it-util ---
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (download-spark) @ hive-it-util ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-it-util
---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
hive-it-util ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-github-source-source/itests/util/src/main/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-it-util ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-it-util
---
[INFO] Compiling 52 source files to
/data/hive-ptest/working/apache-github-source-source/itests/util/target/classes
[WARNING]
/data/hive-ptest/working/apache-github-source-source/itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java:
Some input files use or override a deprecated API.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java:
Recompile with -Xlint:deprecation for details.
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
hive-it-util ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-github-source-source/itests/util/src/test/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-it-util ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/itests/util/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/itests/util/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/itests/util/target/tmp/conf
[copy] Copying 15 files to
/data/hive-ptest/working/apache-github-source-source/itests/util/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hive-it-util ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-it-util ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-it-util ---
[INFO] Building jar:
/data/hive-ptest/working/apache-github-source-source/itests/util/target/hive-it-util-2.1.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
hive-it-util ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-it-util ---
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/itests/util/target/hive-it-util-2.1.0-SNAPSHOT.jar
to
/home/hiveptest/.m2/repository/org/apache/hive/hive-it-util/2.1.0-SNAPSHOT/hive-it-util-2.1.0-SNAPSHOT.jar
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/itests/util/pom.xml to
/home/hiveptest/.m2/repository/org/apache/hive/hive-it-util/2.1.0-SNAPSHOT/hive-it-util-2.1.0-SNAPSHOT.pom
[INFO]
[INFO]
[INFO] Building Hive Integration - Unit Tests 2.1.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-it-unit ---
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/itests/hive-unit/target
[INFO] Deleting
/data/hive-ptest/wo
[jira] [Updated] (HIVE-13453) Support ORDER BY and windowing clause in partitioning clause with distinct function
[ https://issues.apache.org/jira/browse/HIVE-13453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-13453: Attachment: HIVE-13453.4.patch Patch-4: add the condition to only handle windowing distinct as a special case since regular distinct will be handled in groupby. > Support ORDER BY and windowing clause in partitioning clause with distinct > function > --- > > Key: HIVE-13453 > URL: https://issues.apache.org/jira/browse/HIVE-13453 > Project: Hive > Issue Type: Sub-task > Components: PTF-Windowing >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13453.1.patch, HIVE-13453.2.patch, > HIVE-13453.3.patch, HIVE-13453.4.patch > > > Current distinct function on partitioning doesn't support order by and > windowing clause due to performance reason. Explore an efficient way to > support that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6131) New columns after table alter result in null values despite data
[
https://issues.apache.org/jira/browse/HIVE-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289556#comment-15289556
]
Hive QA commented on HIVE-6131:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12637927/HIVE-6131.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 69 failed/errored test(s), 9899 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_sortmerge_join_7.q-orc_merge9.q-tez_union_dynamic_partition.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-union5.q-and-12-more - did not
produce a TEST-*.xml file
TestMiniTezCliDriver-load_dyn_part2.q-selectDistinctStar.q-vector_decimal_5.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-mapjoin_mapjoin.q-insert_into1.q-vector_decimal_2.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_distinct_2.q-tez_joins_explain.q-cte_mat_1.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_interval_2.q-schema_evol_text_nonvec_mapwork_part_all_primitive.q-tez_fsstat.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorized_parquet.q-insert_values_non_partitioned.q-schema_evol_orc_nonvec_mapwork_part.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join4.q-groupby_cube1.q-auto_join20.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-parallel_join1.q-escape_distributeby1.q-auto_sortmerge_join_7.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_change_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_cascade
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_data_after_schema_update
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part_all_complex
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part_all_primitive
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part_all_complex
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part_all_primitive
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part_all_complex
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part_all_primitive
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_partition_diff_num_cols
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part_all_complex
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part_all_complex
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part_all_primitive
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part_all_complex
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part_all_primitive
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_partition_diff_num_cols
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_many
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join18
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_reordering_values
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cbo_udf_udaf
org.apache.hadoop.hive.cli.TestSparkCliDriver.
[jira] [Commented] (HIVE-13736) View's input/output formats are TEXT by default
[ https://issues.apache.org/jira/browse/HIVE-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289543#comment-15289543 ] Yongzhi Chen commented on HIVE-13736: - The failures are not related. The spark failures are all: RPC channel is closed. TestJdbcNonKrbSASLWithMiniKdc failures: Connection refused. All other failures age 2 or more. [~ctang.ma], could you review the change? Thanks > View's input/output formats are TEXT by default > --- > > Key: HIVE-13736 > URL: https://issues.apache.org/jira/browse/HIVE-13736 > Project: Hive > Issue Type: New Feature > Components: Hive >Affects Versions: 1.2.1 >Reporter: Pavas Garg >Assignee: Yongzhi Chen >Priority: Minor > Attachments: HIVE-13736.1.patch > > > Feature request where Hive View's input/output formats are text by default in > order to help 3rd party compatibility -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HIVE-13049) Hive Thrift python lib not at latest version
[ https://issues.apache.org/jira/browse/HIVE-13049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sailesh Mukil reopened HIVE-13049: -- > Hive Thrift python lib not at latest version > > > Key: HIVE-13049 > URL: https://issues.apache.org/jira/browse/HIVE-13049 > Project: Hive > Issue Type: Bug > Components: Authentication, Security >Reporter: Sailesh Mukil >Priority: Blocker > > The Thrift python lib in Hive does not seem to be of the latest Thrift > version. We found this while running tests with SSL on our cluster. We have a > python script that runs our workloads and we couldn't get SSL authentication. > Looking at the Hive python thrift library, I noticed that the TSSLSocket.py > file is missing: > https://github.com/apache/hive/tree/master/service/lib/py/thrift/transport > From the thrift repo, we need TSSLSocket.py: > https://github.com/apache/thrift/tree/master/lib/py/src/transport -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-13049) Hive Thrift python lib not at latest version
[ https://issues.apache.org/jira/browse/HIVE-13049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sailesh Mukil resolved HIVE-13049. -- Resolution: Fixed Turns out that the older version of the Thrift library was there just for backwards compatibility and Hive fetches Thrift as a dependency. > Hive Thrift python lib not at latest version > > > Key: HIVE-13049 > URL: https://issues.apache.org/jira/browse/HIVE-13049 > Project: Hive > Issue Type: Bug > Components: Authentication, Security >Reporter: Sailesh Mukil >Priority: Blocker > > The Thrift python lib in Hive does not seem to be of the latest Thrift > version. We found this while running tests with SSL on our cluster. We have a > python script that runs our workloads and we couldn't get SSL authentication. > Looking at the Hive python thrift library, I noticed that the TSSLSocket.py > file is missing: > https://github.com/apache/hive/tree/master/service/lib/py/thrift/transport > From the thrift repo, we need TSSLSocket.py: > https://github.com/apache/thrift/tree/master/lib/py/src/transport -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-13049) Hive Thrift python lib not at latest version
[ https://issues.apache.org/jira/browse/HIVE-13049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sailesh Mukil resolved HIVE-13049. -- Resolution: Not A Bug > Hive Thrift python lib not at latest version > > > Key: HIVE-13049 > URL: https://issues.apache.org/jira/browse/HIVE-13049 > Project: Hive > Issue Type: Bug > Components: Authentication, Security >Reporter: Sailesh Mukil >Priority: Blocker > > The Thrift python lib in Hive does not seem to be of the latest Thrift > version. We found this while running tests with SSL on our cluster. We have a > python script that runs our workloads and we couldn't get SSL authentication. > Looking at the Hive python thrift library, I noticed that the TSSLSocket.py > file is missing: > https://github.com/apache/hive/tree/master/service/lib/py/thrift/transport > From the thrift repo, we need TSSLSocket.py: > https://github.com/apache/thrift/tree/master/lib/py/src/transport -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13249) Hard upper bound on number of open transactions
[ https://issues.apache.org/jira/browse/HIVE-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13249: - Attachment: HIVE-13249.11.patch Patch 11 moved two tests TestCompactionTxnHandler.java and TestTxnHandler.java from metastore/src/test/org/apache/hadoop/hive/metastore/txn/ to ql/src/test/org/apache/hadoop/hive/metastore/txn/ The reason is due to the change in TxnHandler.openTxns, the class "org.apache.hadoop.hive.ql.txn.AcidOpenTxnsCounterService" in ql package needs to be loaded from metastore package. This is OK during runtime, when we have hive-exec.jar that's needed for this. But for unit test things are different. We don't have the jar from which to load the class. So the solution is to move the affected unit test classes from metastore directory to ql directory, while they still belong to metastore package. > Hard upper bound on number of open transactions > --- > > Key: HIVE-13249 > URL: https://issues.apache.org/jira/browse/HIVE-13249 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13249.1.patch, HIVE-13249.10.patch, > HIVE-13249.11.patch, HIVE-13249.2.patch, HIVE-13249.3.patch, > HIVE-13249.4.patch, HIVE-13249.5.patch, HIVE-13249.6.patch, > HIVE-13249.7.patch, HIVE-13249.8.patch, HIVE-13249.9.patch > > > We need to have a safeguard by adding an upper bound for open transactions to > avoid huge number of open-transaction requests, usually due to improper > configuration of clients such as Storm. > Once that limit is reached, clients will start failing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13778) DROP TABLE PURGE on S3A table with too many files does not delete the files
[ https://issues.apache.org/jira/browse/HIVE-13778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289504#comment-15289504 ] Sailesh Mukil edited comment on HIVE-13778 at 5/18/16 6:18 PM: --- [~ashutoshc] I noticed that this happens usually around 25+ files. was (Author: sailesh): I noticed that this happens usually around 25+ files. > DROP TABLE PURGE on S3A table with too many files does not delete the files > --- > > Key: HIVE-13778 > URL: https://issues.apache.org/jira/browse/HIVE-13778 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Sailesh Mukil >Priority: Critical > Labels: metastore, s3 > > I've noticed that when we do a DROP TABLE tablename PURGE on a table on S3A > that has many files, the files never get deleted. However, the Hive metastore > logs do say that the path was deleted: > "Not moving [path] to trash" > "Deleted the diretory [path]" > I initially thought that this was due to the eventually consistent nature of > S3 for deletes, however, a week later, the files still exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13778) DROP TABLE PURGE on S3A table with too many files does not delete the files
[ https://issues.apache.org/jira/browse/HIVE-13778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289504#comment-15289504 ] Sailesh Mukil commented on HIVE-13778: -- I noticed that this happens usually around 25+ files. > DROP TABLE PURGE on S3A table with too many files does not delete the files > --- > > Key: HIVE-13778 > URL: https://issues.apache.org/jira/browse/HIVE-13778 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Sailesh Mukil >Priority: Critical > Labels: metastore, s3 > > I've noticed that when we do a DROP TABLE tablename PURGE on a table on S3A > that has many files, the files never get deleted. However, the Hive metastore > logs do say that the path was deleted: > "Not moving [path] to trash" > "Deleted the diretory [path]" > I initially thought that this was due to the eventually consistent nature of > S3 for deletes, however, a week later, the files still exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13562) Enable vector bridge for all non-vectorized udfs
[
https://issues.apache.org/jira/browse/HIVE-13562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-13562:
Attachment: HIVE-13562.03.patch
> Enable vector bridge for all non-vectorized udfs
>
>
> Key: HIVE-13562
> URL: https://issues.apache.org/jira/browse/HIVE-13562
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Reporter: Ashutosh Chauhan
>Assignee: Matt McCline
> Attachments: HIVE-13562.01.patch, HIVE-13562.02.patch,
> HIVE-13562.03.patch
>
>
> Mechanism already exists for this via {{VectorUDFAdaptor}} but we have
> arbitrarily hand picked few udfs to go through it. I think we should enable
> this by default for all udfs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13562) Enable vector bridge for all non-vectorized udfs
[
https://issues.apache.org/jira/browse/HIVE-13562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-13562:
Attachment: (was: HIVE-13562.03.patch)
> Enable vector bridge for all non-vectorized udfs
>
>
> Key: HIVE-13562
> URL: https://issues.apache.org/jira/browse/HIVE-13562
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Reporter: Ashutosh Chauhan
>Assignee: Matt McCline
> Attachments: HIVE-13562.01.patch, HIVE-13562.02.patch
>
>
> Mechanism already exists for this via {{VectorUDFAdaptor}} but we have
> arbitrarily hand picked few udfs to go through it. I think we should enable
> this by default for all udfs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13343: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Need to disable hybrid grace hash join in llap mode except for dynamically > partitioned hash join > > > Key: HIVE-13343 > URL: https://issues.apache.org/jira/browse/HIVE-13343 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13343.1.patch, HIVE-13343.2.patch, > HIVE-13343.3.patch, HIVE-13343.4.patch, HIVE-13343.5.patch, > HIVE-13343.6.patch, HIVE-13343.7.patch > > > Due to performance reasons, we should disable use of hybrid grace hash join > in llap when dynamic partition hash join is not used. With dynamic partition > hash join, we need hybrid grace hash join due to the possibility of skews. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13699) Make JavaDataModel#get thread safe for parallel compilation
[ https://issues.apache.org/jira/browse/HIVE-13699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289449#comment-15289449 ] Peter Slawski commented on HIVE-13699: -- Yeah, I should be using SLF4J. I will correct and post an updated patch. This is a preemptive patch found by doing static analysis on the code path for Driver#compile. > Make JavaDataModel#get thread safe for parallel compilation > --- > > Key: HIVE-13699 > URL: https://issues.apache.org/jira/browse/HIVE-13699 > Project: Hive > Issue Type: Bug > Components: HiveServer2, storage-api >Affects Versions: 2.0.0 >Reporter: Peter Slawski >Assignee: Peter Slawski >Priority: Minor > Attachments: HIVE-13699.1.patch > > > The class JavaDataModel has a static method, #get, that is not thread safe. > This may be an issue when parallel query compilation is enabled because two > threads may attempt to call JavaDataModel#get at the same time, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13777) NPE is thrown when select table after change column from string to decimal
[
https://issues.apache.org/jira/browse/HIVE-13777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-13777:
--
Target Version/s: 2.1.0 (was: 1.2.1, 2.1.0)
> NPE is thrown when select table after change column from string to decimal
> --
>
> Key: HIVE-13777
> URL: https://issues.apache.org/jira/browse/HIVE-13777
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>
> Run the followings to change a column from string to decimail:
> {noformat}
> drop table if exists
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal;
> create table
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal
> stored as orc as select s, s_dc from schemaevo;
> alter table
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal
> change column s_dc s_dc decimal(12,4);
> select count(*) from
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal;
> {noformat}
> select count works fine but just select throws NPE:
> {noformat}
> 0: jdbc:hive2://os-r6-0517-hiveserver2-1-1.op> select count(*) from
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal;
> +-+--+
> | c0|
> +-+--+
> | 10 |
> +-+--+
> 1 row selected (13.856 seconds)
> 0: jdbc:hive2://os-r6-0517-hiveserver2-1-1.op> select * from
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal;
> Error: java.io.IOException: java.lang.NullPointerException (state=,code=0)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13777) NPE is thrown when select table after change column from string to decimal
[
https://issues.apache.org/jira/browse/HIVE-13777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-13777:
--
Affects Version/s: (was: 1.2.1)
> NPE is thrown when select table after change column from string to decimal
> --
>
> Key: HIVE-13777
> URL: https://issues.apache.org/jira/browse/HIVE-13777
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>
> Run the followings to change a column from string to decimail:
> {noformat}
> drop table if exists
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal;
> create table
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal
> stored as orc as select s, s_dc from schemaevo;
> alter table
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal
> change column s_dc s_dc decimal(12,4);
> select count(*) from
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal;
> {noformat}
> select count works fine but just select throws NPE:
> {noformat}
> 0: jdbc:hive2://os-r6-0517-hiveserver2-1-1.op> select count(*) from
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal;
> +-+--+
> | c0|
> +-+--+
> | 10 |
> +-+--+
> 1 row selected (13.856 seconds)
> 0: jdbc:hive2://os-r6-0517-hiveserver2-1-1.op> select * from
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal;
> Error: java.io.IOException: java.lang.NullPointerException (state=,code=0)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13773) Stats state is not captured correctly in dynpart_sort_optimization_acid.q
[ https://issues.apache.org/jira/browse/HIVE-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289434#comment-15289434 ] Pengcheng Xiong commented on HIVE-13773: [~ashutoshc], what i have observed is this. In the q file that I attached, there is an insert into. It reads from a table and then insert into a partition table. There are two configurations, hive.optimize.sort.dynamic.partition and also ACID. If we turn on only one of them, the stats of insert into works as we expected. However, if we turn on both of them, the stats of insert into got screwed up. Note that, the data is reading correctly. I suspect that HIVE-6455 introduced prevFsp and it may be wrongly configured when ACID is on. The remove of the related code makes the stats of insert into work. But I need the original author [~prasanth_j] to confirm. Thanks. > Stats state is not captured correctly in dynpart_sort_optimization_acid.q > - > > Key: HIVE-13773 > URL: https://issues.apache.org/jira/browse/HIVE-13773 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13773.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-13691) No record with CQ_ID=0 found in COMPACTION_QUEUE
[
https://issues.apache.org/jira/browse/HIVE-13691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman resolved HIVE-13691.
---
Resolution: Fixed
committed to branch-1 and master
thanks Wei for the review
> No record with CQ_ID=0 found in COMPACTION_QUEUE
>
>
> Key: HIVE-13691
> URL: https://issues.apache.org/jira/browse/HIVE-13691
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13691.2.patch, HIVE-13691.3.patch, HIVE-13691.patch
>
>
> {noformat}
> 2016-04-29 18:49:31,594 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(141)) - Caught exception while trying to determine if we
> should compact id:0,dbname:default,tableName:service_logs_v2,par
> tName:ds=2016-04-21,state:^@,type:null,runAs:null,tooManyAborts:false,highestTxnId:0.
> Marking clean to avoid repeated failures, MetaException(message:Timeout
> when executing method: getTable)
> at
> org.apache.hadoop.hive.metastore.Deadline.newMetaException(Deadline.java:187)
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:177)
> at
> org.apache.hadoop.hive.metastore.Deadline.checkTimeout(Deadline.java:160)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToParts(ObjectStore.java:1839)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsViaOrmFilter(ObjectStore.java:2255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$300(ObjectStore.java:165)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2051)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2400)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNamesInternal(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNames(ObjectStore.java:2037)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy0.getPartitionsByNames(Unknown Source)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread.resolvePartition(CompactorThread.java:111)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:129)
> Caused by: org.apache.hadoop.hive.metastore.DeadlineException: Timeout when
> executing method: getTable
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:174)
> ... 16 more
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> {noformat}
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> is triggered by _DeadlineException: Timeout when executing method_ but is
> nonetheless an issue.
> We should be able to record an entry in completed_compaction_queue to
> represent a failed compaction even if an entry in compaction_queue was never
> made, as is the case here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13699) Make JavaDataModel#get thread safe for parallel compilation
[ https://issues.apache.org/jira/browse/HIVE-13699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289407#comment-15289407 ] Ashutosh Chauhan commented on HIVE-13699: - Please use slf4j Logger instead of apache-commons Log. Otherwise, patch looks good. Have you seen issues because of this static method in parallel compilation? If so, how it manifested or is it just a preemptive patch. > Make JavaDataModel#get thread safe for parallel compilation > --- > > Key: HIVE-13699 > URL: https://issues.apache.org/jira/browse/HIVE-13699 > Project: Hive > Issue Type: Bug > Components: HiveServer2, storage-api >Affects Versions: 2.0.0 >Reporter: Peter Slawski >Assignee: Peter Slawski >Priority: Minor > Attachments: HIVE-13699.1.patch > > > The class JavaDataModel has a static method, #get, that is not thread safe. > This may be an issue when parallel query compilation is enabled because two > threads may attempt to call JavaDataModel#get at the same time, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13691) No record with CQ_ID=0 found in COMPACTION_QUEUE
[
https://issues.apache.org/jira/browse/HIVE-13691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-13691:
--
Attachment: HIVE-13691.3.patch
patch 3 removes unused imports - no other changes
> No record with CQ_ID=0 found in COMPACTION_QUEUE
>
>
> Key: HIVE-13691
> URL: https://issues.apache.org/jira/browse/HIVE-13691
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13691.2.patch, HIVE-13691.3.patch, HIVE-13691.patch
>
>
> {noformat}
> 2016-04-29 18:49:31,594 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(141)) - Caught exception while trying to determine if we
> should compact id:0,dbname:default,tableName:service_logs_v2,par
> tName:ds=2016-04-21,state:^@,type:null,runAs:null,tooManyAborts:false,highestTxnId:0.
> Marking clean to avoid repeated failures, MetaException(message:Timeout
> when executing method: getTable)
> at
> org.apache.hadoop.hive.metastore.Deadline.newMetaException(Deadline.java:187)
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:177)
> at
> org.apache.hadoop.hive.metastore.Deadline.checkTimeout(Deadline.java:160)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.convertToParts(ObjectStore.java:1839)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsViaOrmFilter(ObjectStore.java:2255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$300(ObjectStore.java:165)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2051)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$3.getJdoResult(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2400)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNamesInternal(ObjectStore.java:2043)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNames(ObjectStore.java:2037)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy0.getPartitionsByNames(Unknown Source)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread.resolvePartition(CompactorThread.java:111)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:129)
> Caused by: org.apache.hadoop.hive.metastore.DeadlineException: Timeout when
> executing method: getTable
> at org.apache.hadoop.hive.metastore.Deadline.check(Deadline.java:174)
> ... 16 more
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> {noformat}
> 2016-04-29 18:49:31,595 ERROR [Thread-11]: compactor.Initiator
> (Initiator.java:run(154)) - Initiator loop caught unexpected exception this
> time through the loop: java.lang.IllegalStateException: No record with
> CQ_ID=0 found in COMPACTION_QUEUE
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.markFailed(CompactionTxnHandler.java:861)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:144)
> {noformat}
> is triggered by _DeadlineException: Timeout when executing method_ but is
> nonetheless an issue.
> We should be able to record an entry in completed_compaction_queue to
> represent a failed compaction even if an entry in compaction_queue was never
> made, as is the case here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13699) Make JavaDataModel#get thread safe for parallel compilation
[ https://issues.apache.org/jira/browse/HIVE-13699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289398#comment-15289398 ] Peter Slawski commented on HIVE-13699: -- [~ashutoshc], would have a moment to take a look at this patch? It is fixing another thread safety for parallel compilation. Thank you! > Make JavaDataModel#get thread safe for parallel compilation > --- > > Key: HIVE-13699 > URL: https://issues.apache.org/jira/browse/HIVE-13699 > Project: Hive > Issue Type: Bug > Components: HiveServer2, storage-api >Affects Versions: 2.0.0 >Reporter: Peter Slawski >Assignee: Peter Slawski >Priority: Minor > Attachments: HIVE-13699.1.patch > > > The class JavaDataModel has a static method, #get, that is not thread safe. > This may be an issue when parallel query compilation is enabled because two > threads may attempt to call JavaDataModel#get at the same time, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13730) Avoid double spilling the same partition when memory threshold is set very low
[ https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13730: - Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to master. Thanks Vikram for the review! > Avoid double spilling the same partition when memory threshold is set very low > -- > > Key: HIVE-13730 > URL: https://issues.apache.org/jira/browse/HIVE-13730 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Wei Zheng >Priority: Blocker > Fix For: 2.1.0 > > Attachments: HIVE-13730.1.patch, HIVE-13730.2.patch, > HIVE-13730.3.patch, HIVE-13730.4.patch > > > I am seeing hybridgrace_hashjoin_1.q getting stuck on master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13730) Avoid double spilling the same partition when memory threshold is set very low
[ https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13730: - Summary: Avoid double spilling the same partition when memory threshold is set very low (was: hybridgrace_hashjoin_1.q test gets stuck) > Avoid double spilling the same partition when memory threshold is set very low > -- > > Key: HIVE-13730 > URL: https://issues.apache.org/jira/browse/HIVE-13730 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Wei Zheng >Priority: Blocker > Attachments: HIVE-13730.1.patch, HIVE-13730.2.patch, > HIVE-13730.3.patch, HIVE-13730.4.patch > > > I am seeing hybridgrace_hashjoin_1.q getting stuck on master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13730) hybridgrace_hashjoin_1.q test gets stuck
[ https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289292#comment-15289292 ] Vikram Dixit K commented on HIVE-13730: --- +1 > hybridgrace_hashjoin_1.q test gets stuck > > > Key: HIVE-13730 > URL: https://issues.apache.org/jira/browse/HIVE-13730 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Wei Zheng >Priority: Blocker > Attachments: HIVE-13730.1.patch, HIVE-13730.2.patch, > HIVE-13730.3.patch, HIVE-13730.4.patch > > > I am seeing hybridgrace_hashjoin_1.q getting stuck on master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13730) hybridgrace_hashjoin_1.q test gets stuck
[
https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289258#comment-15289258
]
Wei Zheng commented on HIVE-13730:
--
I ran the three sets of tests that failed to produce TEST-*.xml files, as
commented above by Hive QA. All of them passed locally. So Should have nothing
to do with patch 4.
{code}
enforce_order.q,vector_partition_diff_num_cols.q,unionDistinct_1.q,tez_smb_empty.q,vectorized_timestamp.q,vectorized_rcfile_columnar.q,tez_dml.q,vector_join_nulls.q,delete_tmp_table.q,schema_evol_orc_nonvec_fetchwork_part.q,vectorization_part_varchar.q,load_dyn_part1.q,auto_sortmerge_join_3.q,vector_reduce_groupby_decimal.q,union_type_chk.q
groupby2.q,tez_dynpart_hashjoin_1.q,custom_input_output_format.q,schema_evol_orc_nonvec_fetchwork_table.q,schema_evol_orc_nonvec_mapwork_part_all_complex.q,tez_multi_union.q,vector_between_in.q,vector_char_4.q,dynamic_partition_pruning_2.q,vector_decimal_math_funcs.q,union7.q,vector_char_simple.q,auto_sortmerge_join_8.q,schema_evol_orc_nonvec_mapwork_table.q,merge2.q
vectorization_16.q,vector_decimal_round.q,orc_merge6.q,vector_multi_insert.q,tez_union.q,vector_decimal_precision.q,alter_merge_2_orc.q,auto_sortmerge_join_14.q,vector_aggregate_9.q,vector_reduce1.q,vector_count_distinct.q,auto_join0.q,cross_join.q,vector_coalesce_2.q,vector_varchar_simple.q
{code}
> hybridgrace_hashjoin_1.q test gets stuck
>
>
> Key: HIVE-13730
> URL: https://issues.apache.org/jira/browse/HIVE-13730
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 2.1.0
>Reporter: Vikram Dixit K
>Assignee: Wei Zheng
>Priority: Blocker
> Attachments: HIVE-13730.1.patch, HIVE-13730.2.patch,
> HIVE-13730.3.patch, HIVE-13730.4.patch
>
>
> I am seeing hybridgrace_hashjoin_1.q getting stuck on master.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13029) NVDIMM support for LLAP Cache
[ https://issues.apache.org/jira/browse/HIVE-13029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289249#comment-15289249 ] Sergey Shelukhin commented on HIVE-13029: - Btw, one question I had, how is mem mapped file memory attributed by YARN? As far as I can tell we have no control over it, right? As long as YARN doesn't see it it should be ok, otherwise the containers might get killed. > NVDIMM support for LLAP Cache > - > > Key: HIVE-13029 > URL: https://issues.apache.org/jira/browse/HIVE-13029 > Project: Hive > Issue Type: New Feature > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Critical > Attachments: HIVE-13029.1.patch, HIVE-13029.2.patch > > > LLAP cache has been designed so that the cache can be offloaded easily to a > pmem API without restart coherence. > The tricky part about NVDIMMs are restart coherence, while most of the cache > gains can be obtained without keeping state across refreshes, since LLAP is > not the system of record, HDFS is. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13730) hybridgrace_hashjoin_1.q test gets stuck
[ https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289201#comment-15289201 ] Ashutosh Chauhan commented on HIVE-13730: - We don't have a good run for TestMiniLlapCliDriver & for few batches of TestMiniTezCliDriver for few weeks now. Shall we disable this test, if this gonna take more time to review and commit? cc: [~vikram.dixit] > hybridgrace_hashjoin_1.q test gets stuck > > > Key: HIVE-13730 > URL: https://issues.apache.org/jira/browse/HIVE-13730 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Wei Zheng >Priority: Blocker > Attachments: HIVE-13730.1.patch, HIVE-13730.2.patch, > HIVE-13730.3.patch, HIVE-13730.4.patch > > > I am seeing hybridgrace_hashjoin_1.q getting stuck on master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289173#comment-15289173 ] Jesus Camacho Rodriguez commented on HIVE-13068: [~ashutoshc], I uploaded a new patch and updated the RB. I solved every possible, and left further comments on some that possibly need follow-up work. Thanks > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch, > HIVE-13068.02.patch, HIVE-13068.03.patch, HIVE-13068.04.patch, > HIVE-13068.04.patch, HIVE-13068.05.patch, HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10815) Let HiveMetaStoreClient Choose MetaStore Randomly
[
https://issues.apache.org/jira/browse/HIVE-10815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289141#comment-15289141
]
Hive QA commented on HIVE-10815:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804354/HIVE-10815.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 9979 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-dynpart_sort_optimization2.q-tez_dynpart_hashjoin_3.q-orc_vectorization_ppd.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-union5.q-and-12-more - did not
produce a TEST-*.xml file
TestMiniTezCliDriver-script_pipe.q-vector_decimal_aggregate.q-vector_data_types.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_grouping_sets.q-update_all_partitioned.q-cte_5.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorization_13.q-auto_sortmerge_join_13.q-tez_bmj_schema_evolution.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-bucketmapjoin10.q-join_rc.q-skewjoinopt13.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-union_top_level.q-join11.q-auto_join1.q-and-12-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_22
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorParallelism
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hive.hcatalog.listener.TestDbNotificationListener.cleanupNotifs
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/314/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/314/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-314/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 23 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12804354 - PreCommit-HIVE-MASTER-Build
> Let HiveMetaStoreClient Choose MetaStore Randomly
> -
>
> Key: HIVE-10815
> URL: https://issues.apache.org/jira/browse/HIVE-10815
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, Metastore
>Affects Versions: 1.2.0
>Reporter: Nemon Lou
>Assignee: Nemon Lou
> Attachments: HIVE-10815.1.patch, HIVE-10815.patch
>
>
> Currently HiveMetaStoreClient using a fixed order to choose MetaStore URIs
> when multiple metastores configured.
> Choosing MetaStore Randomly will be good for load balance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13068: --- Status: Patch Available (was: In Progress) > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch, > HIVE-13068.02.patch, HIVE-13068.03.patch, HIVE-13068.04.patch, > HIVE-13068.04.patch, HIVE-13068.05.patch, HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13068: --- Attachment: HIVE-13068.05.patch > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch, > HIVE-13068.02.patch, HIVE-13068.03.patch, HIVE-13068.04.patch, > HIVE-13068.04.patch, HIVE-13068.05.patch, HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
