[jira] [Commented] (HIVE-9264) Merge encryption branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-9264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271977#comment-15271977 ] lushuai commented on HIVE-9264: --- How to create a table encryption table, for example, by specified DDL table encryption attributes, implementation, and secure area bound. By implementing MetaStoreEventListener in the onCreate Table, Drop Table, onAlterTable, onAlterTable etc. and in combination with transparent encryption. IS OK??? > Merge encryption branch to trunk > > > Key: HIVE-9264 > URL: https://issues.apache.org/jira/browse/HIVE-9264 > Project: Hive > Issue Type: Sub-task >Affects Versions: 0.15.0 >Reporter: Brock Noland >Assignee: Brock Noland > Labels: TODOC15 > Fix For: 1.1.0 > > Attachments: HIVE-9264.1.patch, HIVE-9264.2.patch, HIVE-9264.2.patch, > HIVE-9264.2.patch, HIVE-9264.3.patch, HIVE-9264.3.patch, HIVE-9264.3.patch, > HIVE-9264.addendum.patch > > > The team working on the encryption branch would like to merge their work to > trunk. This jira will track that effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12837) Better memory estimation/allocation for hybrid grace hash join during hash table loading
[ https://issues.apache.org/jira/browse/HIVE-12837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-12837: - Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Thanks [~vikram.dixit] and [~sershe] for the review. Committed to master. > Better memory estimation/allocation for hybrid grace hash join during hash > table loading > > > Key: HIVE-12837 > URL: https://issues.apache.org/jira/browse/HIVE-12837 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Fix For: 2.1.0 > > Attachments: HIVE-12837.1.patch, HIVE-12837.2.patch, > HIVE-12837.3.patch, HIVE-12837.4.patch, HIVE-12837.5.patch > > > This is to avoid an edge case when the memory available is very little (less > than a single write buffer size), and we start loading the hash table. Since > the write buffer is lazily allocated, we will easily run out of memory before > even checking if we should spill any hash partition. > e.g. > Total memory available: 210 MB > Size of ref array of BytesBytesMultiHashMap for each hash partition: ~16 MB > Size of write buffer: 8 MB (lazy allocation) > Number of hash partitions: 16 > Number of hash partitions created in memory: 13 > Number of hash partitions created on disk: 3 > Available memory left after HybridHashTableContainer initialization: > 210-16*13=2MB > Now let's say a row is to be loaded into a hash partition in memory, it will > try to allocate an 8MB write buffer for it, but we only have 2MB, thus OOM. > Solution is to perform the check for possible spilling earlier so we can > spill partitions if memory is about to be full, to avoid OOM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13216) ORC Reader will leave file open until GC when opening a malformed ORC file
[
https://issues.apache.org/jira/browse/HIVE-13216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271928#comment-15271928
]
Yuxing Yao commented on HIVE-13216:
---
Why not put this fix into 2.0.1 release?
> ORC Reader will leave file open until GC when opening a malformed ORC file
> --
>
> Key: HIVE-13216
> URL: https://issues.apache.org/jira/browse/HIVE-13216
> Project: Hive
> Issue Type: Bug
> Components: File Formats, ORC
>Affects Versions: 1.2.0, 1.2.1, 2.0.0
>Reporter: Yuxing Yao
>Assignee: Sergey Shelukhin
>Priority: Minor
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13216.patch
>
>
> In ORC extractMetaInfoFromFooter method of ReaderImpl.java:
> A new input stream is open without try-catch-finally to enforce closing.
> Once the footer parse has some exception, the stream close will miss.
> Until GC happen to close the stream.
> private static FileMetaInfo extractMetaInfoFromFooter(FileSystem fs,
> Path path,
> long maxFileLength
> ) throws IOException {
> FSDataInputStream file = fs.open(path);
> ...
> file.close();
> return new FileMetaInfo(
> ps.getCompression().toString(),
> (int) ps.getCompressionBlockSize(),
> (int) ps.getMetadataLength(),
> buffer,
> ps.getVersionList(),
> writerVersion
> );
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13507) Improved logging for ptest
[
https://issues.apache.org/jira/browse/HIVE-13507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271881#comment-15271881
]
Hive QA commented on HIVE-13507:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12802003/HIVE-13507.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 53 failed/errored test(s), 10029 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_serde
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefaultEmptyString
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfOverridden
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfUnknownPreperty
org.apache.hadoop.hive.metastore.TestMetaStoreAuthorization.testMetaStoreAuthorization
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.checkExpectedLocks
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.insertOverwriteCreate
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testDummyTxnManagerOnAcidTable
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testLocksInSubquery
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.updateSelectUpdate
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.hcatalog.api.TestHCatClient.testBasicDDLCommands
org.apache.hive.hcatalog.api.TestHCatClient.testDatabaseLocation
org.apache.hive.hcatalog.api.TestHCatClient.testDropPartitionsWithPartialSpec
org.apache.hive.hcatalog.api.TestHCatClient.testDropTableException
org.apache.hive.hcatalog.api.TestHCatClient.testEmptyTableInstantiation
org.apache.hive.hcatalog.api.TestHCatClient.testGetPartitionsWithPartialSpec
org.apache.hive.hcatalog.api.TestHCatClient.testObjectNotFoundException
org.apache.hive.hcatalog.api.TestHCatClient.testRenameTable
org.apache.hive.hcatalog.api.TestHCatClient.testReplicationTaskIter
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropTable
org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.org.apache.
[jira] [Updated] (HIVE-13671) Add PerfLogger to log4j2.properties logger
[ https://issues.apache.org/jira/browse/HIVE-13671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13671: - Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed patch to master > Add PerfLogger to log4j2.properties logger > -- > > Key: HIVE-13671 > URL: https://issues.apache.org/jira/browse/HIVE-13671 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Fix For: 2.1.0 > > Attachments: HIVE-13671.1.patch > > > To enable perflogging, root logging has to be set to DEBUG. Provide a way to > to independently configure perflogger and root logger levels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13671) Add PerfLogger to log4j2.properties logger
[ https://issues.apache.org/jira/browse/HIVE-13671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271807#comment-15271807 ] Prasanth Jayachandran commented on HIVE-13671: -- All 28 MiniLlap failures runs successfully locally. Other test failures are unrelated to this change. > Add PerfLogger to log4j2.properties logger > -- > > Key: HIVE-13671 > URL: https://issues.apache.org/jira/browse/HIVE-13671 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13671.1.patch > > > To enable perflogging, root logging has to be set to DEBUG. Provide a way to > to independently configure perflogger and root logger levels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13625) Hive Prepared Statement when executed with escape characters in parameter fails
[
https://issues.apache.org/jira/browse/HIVE-13625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271806#comment-15271806
]
Hive QA commented on HIVE-13625:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12801946/HIVE-13625.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 71 failed/errored test(s), 9985 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_sortmerge_join_7.q-tez_union_group_by.q-orc_merge9.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-mapjoin_mapjoin.q-insert_into1.q-vector_decimal_2.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-script_pipe.q-vector_decimal_aggregate.q-acid_globallimit.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_bucket_map_join_tez2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_3
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_5
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_3
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_4
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_5
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llap_nullscan
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llapdecider
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dml
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_hash
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_tests
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_joins_explain
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_smb_main
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestMetaStoreAuthorization.testMetaStoreAuthorization
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.lockmgr.TestDb
[jira] [Updated] (HIVE-13448) LLAP: check ZK acls for ZKSM and fail if they are too permissive
[ https://issues.apache.org/jira/browse/HIVE-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13448: Attachment: HIVE-13448.01.patch Rebased the patch... [~prasanth_j] [~sseth] can you review please? The number of patches in this area causes a lot of conflicts, I want to commit small patches first so the large patches won't need to be constantly rebased. > LLAP: check ZK acls for ZKSM and fail if they are too permissive > > > Key: HIVE-13448 > URL: https://issues.apache.org/jira/browse/HIVE-13448 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13448.01.patch, HIVE-13448.patch > > > Only the current user should have any access. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13656) need to set direct memory limit higher in LlapServiceDriver for certain edge case configurations
[ https://issues.apache.org/jira/browse/HIVE-13656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271787#comment-15271787 ] Sergey Shelukhin commented on HIVE-13656: - No, it sets the value as is in bytes > need to set direct memory limit higher in LlapServiceDriver for certain edge > case configurations > > > Key: HIVE-13656 > URL: https://issues.apache.org/jira/browse/HIVE-13656 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13656.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13449) LLAP: HS2 should get the token directly, rather than from LLAP
[ https://issues.apache.org/jira/browse/HIVE-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13449: Attachment: HIVE-13449.02.WIP.patch Actually, since the LLAP tokens were scoped to the cluster, there is a problem - HS2 doesn't know the full LLAP cluster name including the appId; also the local and remote paths diverge enough to probably require separate APIs. For now, attaching the rebased WIP patch. > LLAP: HS2 should get the token directly, rather than from LLAP > -- > > Key: HIVE-13449 > URL: https://issues.apache.org/jira/browse/HIVE-13449 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13449.01.patch, HIVE-13449.02.WIP.patch, > HIVE-13449.02.patch, HIVE-13449.patch > > > HS2 doesn't need a roundtrip to LLAP; it can instantiate the SecretManager > directly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13657) Spark driver stderr logs should appear in hive client logs
[
https://issues.apache.org/jira/browse/HIVE-13657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-13657:
---
Attachment: HIVE-13657.1.patch
> Spark driver stderr logs should appear in hive client logs
> --
>
> Key: HIVE-13657
> URL: https://issues.apache.org/jira/browse/HIVE-13657
> Project: Hive
> Issue Type: Bug
>Reporter: Mohit Sabharwal
>Assignee: Mohit Sabharwal
> Attachments: HIVE-13657.1.patch, HIVE-13657.patch
>
>
> Currently, spark driver exceptions are not getting logged in beeline.
> Instead, the users sees the not-so-useful:
> {code}
> ERROR : Failed to execute spark task, with exception
> 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark
> client.)'
>
> {code}
> The user has to look at HS2 logs to discover the root cause:
> {code}
> 2015-04-01 11:33:16,048 INFO org.apache.hive.spark.client.SparkClientImpl:
> 15/04/01 11:33:16 WARN UserGroupInformation: PriviledgedActionException
> as:foo (auth:PROXY) via hive (auth:SIMPLE)
> cause:org.apache.hadoop.security.AccessControlException: Permission denied:
> user=foo, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
> ...
> {code}
> We should surface these critical errors in hive client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13657) Spark driver stderr logs should appear in hive client logs
[
https://issues.apache.org/jira/browse/HIVE-13657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271777#comment-15271777
]
Mohit Sabharwal commented on HIVE-13657:
Updated patch with some cleanup.
> Spark driver stderr logs should appear in hive client logs
> --
>
> Key: HIVE-13657
> URL: https://issues.apache.org/jira/browse/HIVE-13657
> Project: Hive
> Issue Type: Bug
>Reporter: Mohit Sabharwal
>Assignee: Mohit Sabharwal
> Attachments: HIVE-13657.1.patch, HIVE-13657.patch
>
>
> Currently, spark driver exceptions are not getting logged in beeline.
> Instead, the users sees the not-so-useful:
> {code}
> ERROR : Failed to execute spark task, with exception
> 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark
> client.)'
>
> {code}
> The user has to look at HS2 logs to discover the root cause:
> {code}
> 2015-04-01 11:33:16,048 INFO org.apache.hive.spark.client.SparkClientImpl:
> 15/04/01 11:33:16 WARN UserGroupInformation: PriviledgedActionException
> as:foo (auth:PROXY) via hive (auth:SIMPLE)
> cause:org.apache.hadoop.security.AccessControlException: Permission denied:
> user=foo, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
> ...
> {code}
> We should surface these critical errors in hive client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13657) Spark driver stderr logs should appear in hive client logs
[
https://issues.apache.org/jira/browse/HIVE-13657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271776#comment-15271776
]
Mohit Sabharwal commented on HIVE-13657:
Thanks, [~szehon]. In my tests, the stderr simply contained the exception
thrown. However, you're right that the stderr may potentially contain
irrelevant info and also too much info. For the latter, I've tried to limit the
number of lines sent back. For the former, I don't see any simple way of
filtering out irrelevant info. We can explore capturing spark driver log4j logs
corresponding to this query and use verbosity levels like we do in HIVE-10119
for MR as a follow-up item.
> Spark driver stderr logs should appear in hive client logs
> --
>
> Key: HIVE-13657
> URL: https://issues.apache.org/jira/browse/HIVE-13657
> Project: Hive
> Issue Type: Bug
>Reporter: Mohit Sabharwal
>Assignee: Mohit Sabharwal
> Attachments: HIVE-13657.patch
>
>
> Currently, spark driver exceptions are not getting logged in beeline.
> Instead, the users sees the not-so-useful:
> {code}
> ERROR : Failed to execute spark task, with exception
> 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark
> client.)'
>
> {code}
> The user has to look at HS2 logs to discover the root cause:
> {code}
> 2015-04-01 11:33:16,048 INFO org.apache.hive.spark.client.SparkClientImpl:
> 15/04/01 11:33:16 WARN UserGroupInformation: PriviledgedActionException
> as:foo (auth:PROXY) via hive (auth:SIMPLE)
> cause:org.apache.hadoop.security.AccessControlException: Permission denied:
> user=foo, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
> ...
> {code}
> We should surface these critical errors in hive client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12837) Better memory estimation/allocation for hybrid grace hash join during hash table loading
[ https://issues.apache.org/jira/browse/HIVE-12837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271764#comment-15271764 ] Vikram Dixit K commented on HIVE-12837: --- +1 LGTM. > Better memory estimation/allocation for hybrid grace hash join during hash > table loading > > > Key: HIVE-12837 > URL: https://issues.apache.org/jira/browse/HIVE-12837 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-12837.1.patch, HIVE-12837.2.patch, > HIVE-12837.3.patch, HIVE-12837.4.patch, HIVE-12837.5.patch > > > This is to avoid an edge case when the memory available is very little (less > than a single write buffer size), and we start loading the hash table. Since > the write buffer is lazily allocated, we will easily run out of memory before > even checking if we should spill any hash partition. > e.g. > Total memory available: 210 MB > Size of ref array of BytesBytesMultiHashMap for each hash partition: ~16 MB > Size of write buffer: 8 MB (lazy allocation) > Number of hash partitions: 16 > Number of hash partitions created in memory: 13 > Number of hash partitions created on disk: 3 > Available memory left after HybridHashTableContainer initialization: > 210-16*13=2MB > Now let's say a row is to be loaded into a hash partition in memory, it will > try to allocate an 8MB write buffer for it, but we only have 2MB, thus OOM. > Solution is to perform the check for possible spilling earlier so we can > spill partitions if memory is about to be full, to avoid OOM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13634) Hive-on-Spark performed worse than Hive-on-MR, for queries with external scripts
[ https://issues.apache.org/jira/browse/HIVE-13634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271760#comment-15271760 ] Rui Li commented on HIVE-13634: --- I'll look into this one. > Hive-on-Spark performed worse than Hive-on-MR, for queries with external > scripts > > > Key: HIVE-13634 > URL: https://issues.apache.org/jira/browse/HIVE-13634 > Project: Hive > Issue Type: Bug >Reporter: Xin Hao >Assignee: Rui Li > > Hive-on-Spark performed worse than Hive-on-MR, for queries with external > scripts. > For TPCx-BB Q2/Q3/Q4, they are Python Streaming related cases and will call > external scripts to handle reduce tasks. We found that for these 3 queries > Hive-on-Spark shows lower performance than Hive-on-MR when processing reduce > tasks with external (Python) scripts. So ‘Improve HoS performance for queries > with external scripts’ seems a performance optimization opportunity. > The following shows the Q2/Q3/Q4 test result on 8-worker-node cluster with > TPCx-BB 3TB data size. > TPCx-BB Query 2 > (1)Hive-on-MR > Total Query Execution Time (sec): 2172.180 > Execution Time of External Scripts (sec): 736 > (2)Hive-on-Spark > Total Query Execution Time (sec): 2283.604 > Execution Time of External Scripts (sec): 1197 > TPCx-BB Query 3 > (1)Hive-on-MR > Total Query Execution Time (sec): 1070.632 > Execution Time of External Scripts (sec): 513 > (2)Hive-on-Spark > Total Query Execution Time (sec): 1287.679 > Execution Time of External Scripts (sec): 919 > TPCx-BB Query 4 > (1)Hive-on-MR > Total Query Execution Time (sec): 1781.864 > Execution Time of External Scripts (sec): 1518 > (2)Hive-on-Spark > Total Query Execution Time (sec): 2028.023 > Execution Time of External Scripts (sec): 1599 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13634) Hive-on-Spark performed worse than Hive-on-MR, for queries with external scripts
[ https://issues.apache.org/jira/browse/HIVE-13634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-13634: - Assignee: Rui Li > Hive-on-Spark performed worse than Hive-on-MR, for queries with external > scripts > > > Key: HIVE-13634 > URL: https://issues.apache.org/jira/browse/HIVE-13634 > Project: Hive > Issue Type: Bug >Reporter: Xin Hao >Assignee: Rui Li > > Hive-on-Spark performed worse than Hive-on-MR, for queries with external > scripts. > For TPCx-BB Q2/Q3/Q4, they are Python Streaming related cases and will call > external scripts to handle reduce tasks. We found that for these 3 queries > Hive-on-Spark shows lower performance than Hive-on-MR when processing reduce > tasks with external (Python) scripts. So ‘Improve HoS performance for queries > with external scripts’ seems a performance optimization opportunity. > The following shows the Q2/Q3/Q4 test result on 8-worker-node cluster with > TPCx-BB 3TB data size. > TPCx-BB Query 2 > (1)Hive-on-MR > Total Query Execution Time (sec): 2172.180 > Execution Time of External Scripts (sec): 736 > (2)Hive-on-Spark > Total Query Execution Time (sec): 2283.604 > Execution Time of External Scripts (sec): 1197 > TPCx-BB Query 3 > (1)Hive-on-MR > Total Query Execution Time (sec): 1070.632 > Execution Time of External Scripts (sec): 513 > (2)Hive-on-Spark > Total Query Execution Time (sec): 1287.679 > Execution Time of External Scripts (sec): 919 > TPCx-BB Query 4 > (1)Hive-on-MR > Total Query Execution Time (sec): 1781.864 > Execution Time of External Scripts (sec): 1518 > (2)Hive-on-Spark > Total Query Execution Time (sec): 2028.023 > Execution Time of External Scripts (sec): 1599 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13159) TxnHandler should support datanucleus.connectionPoolingType = None
[ https://issues.apache.org/jira/browse/HIVE-13159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271756#comment-15271756 ] Eugene Koifman commented on HIVE-13159: --- +1 pending tests > TxnHandler should support datanucleus.connectionPoolingType = None > -- > > Key: HIVE-13159 > URL: https://issues.apache.org/jira/browse/HIVE-13159 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Sergey Shelukhin >Assignee: Alan Gates > Attachments: HIVE-13159.2.patch, HIVE-13159.patch > > > Right now, one has to choose bonecp or dbcp. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13598) Describe extended table should show the primary keys/foreign keys associated with the table
[ https://issues.apache.org/jira/browse/HIVE-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13598: - Status: Patch Available (was: Open) > Describe extended table should show the primary keys/foreign keys associated > with the table > --- > > Key: HIVE-13598 > URL: https://issues.apache.org/jira/browse/HIVE-13598 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13598.1.patch, HIVE-13598.2.patch, > HIVE-13598.3.patch, HIVE-13598.4.patch, HIVE-13598.5.patch > > > After HIVE-13290 is committed, we need to show the constraints as part of > table description when extended label is used. Currently, the constraints > would not be shown as part of table description since Constraint is a > separate entity. > The purpose of the jira is to modify Hive.describeTable() to enable the user > to view the constraints associated with the table when the user does a > "describe extended table"; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13598) Describe extended table should show the primary keys/foreign keys associated with the table
[ https://issues.apache.org/jira/browse/HIVE-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13598: - Status: Open (was: Patch Available) > Describe extended table should show the primary keys/foreign keys associated > with the table > --- > > Key: HIVE-13598 > URL: https://issues.apache.org/jira/browse/HIVE-13598 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13598.1.patch, HIVE-13598.2.patch, > HIVE-13598.3.patch, HIVE-13598.4.patch, HIVE-13598.5.patch > > > After HIVE-13290 is committed, we need to show the constraints as part of > table description when extended label is used. Currently, the constraints > would not be shown as part of table description since Constraint is a > separate entity. > The purpose of the jira is to modify Hive.describeTable() to enable the user > to view the constraints associated with the table when the user does a > "describe extended table"; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13598) Describe extended table should show the primary keys/foreign keys associated with the table
[ https://issues.apache.org/jira/browse/HIVE-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13598: - Attachment: HIVE-13598.5.patch > Describe extended table should show the primary keys/foreign keys associated > with the table > --- > > Key: HIVE-13598 > URL: https://issues.apache.org/jira/browse/HIVE-13598 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13598.1.patch, HIVE-13598.2.patch, > HIVE-13598.3.patch, HIVE-13598.4.patch, HIVE-13598.5.patch > > > After HIVE-13290 is committed, we need to show the constraints as part of > table description when extended label is used. Currently, the constraints > would not be shown as part of table description since Constraint is a > separate entity. > The purpose of the jira is to modify Hive.describeTable() to enable the user > to view the constraints associated with the table when the user does a > "describe extended table"; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13681) Update README with latest Hive functionality
[ https://issues.apache.org/jira/browse/HIVE-13681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-13681: -- Attachment: HIVE-13681.2.patch Correct second patch (previous one was only a diff from first patch instead of trunk) NO PRECOMMIT TESTS > Update README with latest Hive functionality > > > Key: HIVE-13681 > URL: https://issues.apache.org/jira/browse/HIVE-13681 > Project: Hive > Issue Type: Improvement > Components: Documentation >Affects Versions: 2.0.0 >Reporter: Alan Gates >Assignee: Alan Gates > Attachments: HIVE-13681.2.patch, HIVE-13681.patch > > > In line with the latest changes to the website (see HIVE-13607) and the wiki > to remove references to Hive QL/HQL and update use cases Hive is appropriate > for we need to update the README file as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13159) TxnHandler should support datanucleus.connectionPoolingType = None
[ https://issues.apache.org/jira/browse/HIVE-13159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-13159: -- Attachment: HIVE-13159.2.patch New version of the patch with UnsupportedOperationException thrown by unimplemented parts of the connection pool. > TxnHandler should support datanucleus.connectionPoolingType = None > -- > > Key: HIVE-13159 > URL: https://issues.apache.org/jira/browse/HIVE-13159 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Sergey Shelukhin >Assignee: Alan Gates > Attachments: HIVE-13159.2.patch, HIVE-13159.patch > > > Right now, one has to choose bonecp or dbcp. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13681) Update README with latest Hive functionality
[ https://issues.apache.org/jira/browse/HIVE-13681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-13681: -- Attachment: (was: HIVE-13681.2.patch) > Update README with latest Hive functionality > > > Key: HIVE-13681 > URL: https://issues.apache.org/jira/browse/HIVE-13681 > Project: Hive > Issue Type: Improvement > Components: Documentation >Affects Versions: 2.0.0 >Reporter: Alan Gates >Assignee: Alan Gates > Attachments: HIVE-13681.patch > > > In line with the latest changes to the website (see HIVE-13607) and the wiki > to remove references to Hive QL/HQL and update use cases Hive is appropriate > for we need to update the README file as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13395) Lost Update problem in ACID
[ https://issues.apache.org/jira/browse/HIVE-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271728#comment-15271728 ] Alan Gates commented on HIVE-13395: --- Looks good. If you fix the indentation on this I'm +1 on it. > Lost Update problem in ACID > --- > > Key: HIVE-13395 > URL: https://issues.apache.org/jira/browse/HIVE-13395 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.2.0, 2.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-13395.11.patch, HIVE-13395.12.patch, > HIVE-13395.13.patch, HIVE-13395.14.patch, HIVE-13395.15.patch, > HIVE-13395.6.patch, HIVE-13395.7.patch, HIVE-13395.8.patch > > > ACID users can run into Lost Update problem. > In Hive 1.2, Driver.recordValidTxns() (which records the snapshot to use for > the query) is called in Driver.compile(). > Now suppose to concurrent "update T set x = x + 1" are executed. (for > simplicity assume there is exactly 1 row in T) > What can happen is that both compile at the same time (more precisely before > acquireLocksAndOpenTxn() in runInternal() is called) and thus will lock in > the same snapshot, say the value of x = 7 in this snapshot. > Now 1 will get the lock on the row, the second will block. > Now 1, makes x = 8 and commits. > Now 2 proceeds and makes x = 8 again since in it's snapshot x is still 7. > This specific issue is solved in Hive 1.3/2.0 (HIVE-11077 which is a large > patch that deals with multi-statement txns) by moving recordValidTxns() after > locks are acquired which reduces the likelihood of this but doesn't eliminate > the problem. > > Even in 1.3 version of the code, you could have the same issue. Assume the > same 2 queries: > Both start a txn, say txnid 9 and 10. Say 10 gets the lock first, 9 blocks. > 10 updates the row (so x = 8) and thus ReaderKey.currentTransactionId=10. > 10 commits. > Now 9 can proceed and it will get a snapshot that includes 10, i.e. it will > see x = 8 and it will write x = 9, but it will set > ReaderKey.currentTransactionId = 9. Thus when merge logic runs, it will see > x = 8 is the later version of this row, i.e. lost update. > The problem is that locks alone are insufficient for MVCC architecture. > > At lower level Row ID has (originalTransactionId, rowid, bucket id, > currentTransactionId) and since on update/delete we do a table scan, we could > check that we are about to write a row with currentTransactionId < > (currentTransactionId of row we've read) and fail the query. Currently, > currentTransactionId is not surfaced at higher level where this check can be > made. > This would not work (efficiently) longer term where we want to support fast > update on user defined PK vis streaming ingest. > Also, this would not work with multi statement txns since in that case we'd > lock in the snapshot at the start of the txn, but then 2nd, 3rd etc queries > would use the same snapshot and the locks for these queries would be acquired > after the snapshot is locked in so this would be the same situation as pre > HIVE-11077. > > > A more robust solution (commonly used with MVCC) is to keep track of start > and commit time (logical counter) or each transaction to detect if two txns > overlap. The 2nd part is to keep track of write-set, i.e. which data (rows, > partitions, whatever appropriate level of granularity is) were modified by > any txn and if 2 txns overlap in time and wrote the same element, abort later > one. This is called first-committer-wins rule. This requires a MS DB schema > change > It would be most convenient to use the same sequence for txnId, start and > commit time (in which case txnid=start time). In this case we'd need to add > 1 filed to TXNS table. The complication here is that we'll be using elements > of the sequence faster and they are used as part of file name of delta and > base dir and currently limited to 7 digits which can be exceeded. So this > would require some thought to handling upgrade/migration. > Also, write-set tracking requires either additional metastore table or > keeping info in HIVE_LOCKS around longer with new state. > > In the short term, on SQL side of things we could (in auto commit mode only) > acquire the locks first and then open the txn AND update these locks with txn > id. > This implies another Thrift change to pass in lockId to openTxn. > The same would not work for Streaming API since it opens several txns at once > and then acquires locks for each. > (Not sure if that's is an issue or not since Streaming only does Insert). > Either way this feels hacky. > > Here is one simple example why we need Write-Set tracking for multi-statement > txns > Consider transactions T ~1~
[jira] [Commented] (HIVE-13395) Lost Update problem in ACID
[ https://issues.apache.org/jira/browse/HIVE-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271723#comment-15271723 ] Eugene Koifman commented on HIVE-13395: --- I have a followup HIVE-13622 which I hope to get to get shortly - I believe it covers all your concerns (except for the indentation). Let me know if you disagree. > Lost Update problem in ACID > --- > > Key: HIVE-13395 > URL: https://issues.apache.org/jira/browse/HIVE-13395 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.2.0, 2.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-13395.11.patch, HIVE-13395.12.patch, > HIVE-13395.13.patch, HIVE-13395.14.patch, HIVE-13395.15.patch, > HIVE-13395.6.patch, HIVE-13395.7.patch, HIVE-13395.8.patch > > > ACID users can run into Lost Update problem. > In Hive 1.2, Driver.recordValidTxns() (which records the snapshot to use for > the query) is called in Driver.compile(). > Now suppose to concurrent "update T set x = x + 1" are executed. (for > simplicity assume there is exactly 1 row in T) > What can happen is that both compile at the same time (more precisely before > acquireLocksAndOpenTxn() in runInternal() is called) and thus will lock in > the same snapshot, say the value of x = 7 in this snapshot. > Now 1 will get the lock on the row, the second will block. > Now 1, makes x = 8 and commits. > Now 2 proceeds and makes x = 8 again since in it's snapshot x is still 7. > This specific issue is solved in Hive 1.3/2.0 (HIVE-11077 which is a large > patch that deals with multi-statement txns) by moving recordValidTxns() after > locks are acquired which reduces the likelihood of this but doesn't eliminate > the problem. > > Even in 1.3 version of the code, you could have the same issue. Assume the > same 2 queries: > Both start a txn, say txnid 9 and 10. Say 10 gets the lock first, 9 blocks. > 10 updates the row (so x = 8) and thus ReaderKey.currentTransactionId=10. > 10 commits. > Now 9 can proceed and it will get a snapshot that includes 10, i.e. it will > see x = 8 and it will write x = 9, but it will set > ReaderKey.currentTransactionId = 9. Thus when merge logic runs, it will see > x = 8 is the later version of this row, i.e. lost update. > The problem is that locks alone are insufficient for MVCC architecture. > > At lower level Row ID has (originalTransactionId, rowid, bucket id, > currentTransactionId) and since on update/delete we do a table scan, we could > check that we are about to write a row with currentTransactionId < > (currentTransactionId of row we've read) and fail the query. Currently, > currentTransactionId is not surfaced at higher level where this check can be > made. > This would not work (efficiently) longer term where we want to support fast > update on user defined PK vis streaming ingest. > Also, this would not work with multi statement txns since in that case we'd > lock in the snapshot at the start of the txn, but then 2nd, 3rd etc queries > would use the same snapshot and the locks for these queries would be acquired > after the snapshot is locked in so this would be the same situation as pre > HIVE-11077. > > > A more robust solution (commonly used with MVCC) is to keep track of start > and commit time (logical counter) or each transaction to detect if two txns > overlap. The 2nd part is to keep track of write-set, i.e. which data (rows, > partitions, whatever appropriate level of granularity is) were modified by > any txn and if 2 txns overlap in time and wrote the same element, abort later > one. This is called first-committer-wins rule. This requires a MS DB schema > change > It would be most convenient to use the same sequence for txnId, start and > commit time (in which case txnid=start time). In this case we'd need to add > 1 filed to TXNS table. The complication here is that we'll be using elements > of the sequence faster and they are used as part of file name of delta and > base dir and currently limited to 7 digits which can be exceeded. So this > would require some thought to handling upgrade/migration. > Also, write-set tracking requires either additional metastore table or > keeping info in HIVE_LOCKS around longer with new state. > > In the short term, on SQL side of things we could (in auto commit mode only) > acquire the locks first and then open the txn AND update these locks with txn > id. > This implies another Thrift change to pass in lockId to openTxn. > The same would not work for Streaming API since it opens several txns at once > and then acquires locks for each. > (Not sure if that's is an issue or not since Streaming only does Insert). > Either way this feels hacky. > > Her
[jira] [Updated] (HIVE-13681) Update README with latest Hive functionality
[ https://issues.apache.org/jira/browse/HIVE-13681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-13681: -- Attachment: HIVE-13681.2.patch NO PRECOMMIT TESTS > Update README with latest Hive functionality > > > Key: HIVE-13681 > URL: https://issues.apache.org/jira/browse/HIVE-13681 > Project: Hive > Issue Type: Improvement > Components: Documentation >Affects Versions: 2.0.0 >Reporter: Alan Gates >Assignee: Alan Gates > Attachments: HIVE-13681.2.patch, HIVE-13681.patch > > > In line with the latest changes to the website (see HIVE-13607) and the wiki > to remove references to Hive QL/HQL and update use cases Hive is appropriate > for we need to update the README file as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13395) Lost Update problem in ACID
[ https://issues.apache.org/jira/browse/HIVE-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271712#comment-15271712 ] Alan Gates commented on HIVE-13395: --- TxnHandler.java in OperationType: it seems odd to create a new enum with a deprecated method. In TxnHandler.commitTxn, would it make sense to rearrange this so that the check is made whether there are any operations that could conflict before the mutex is obtained and the transaction id checked? If there's nothing to record in the write sets I don't see why you need to hold the mutex or even record a commit txn id. TxnHandler.addDynamicPartitions we should fix this so that the operations is carried in the dynamic partition message now rather than fetched from components table. We can do it in a separate JIRA but we should do it quickly. Fetching back another row to answer that question is bogus. TxnHandler.checkLock IIUC the if (!writeSet.isEmpty()) (line 2176) will never be triggered right now. I'm not a fan of blocks of dead code. When do you plan to alter checkLock so that it knows whether the requesting locker is dynamic or static? TxnHandler lines 2229 through 2290 should be indented to match the following lines. Right now they are 1 stop too far to the left. > Lost Update problem in ACID > --- > > Key: HIVE-13395 > URL: https://issues.apache.org/jira/browse/HIVE-13395 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.2.0, 2.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-13395.11.patch, HIVE-13395.12.patch, > HIVE-13395.13.patch, HIVE-13395.14.patch, HIVE-13395.15.patch, > HIVE-13395.6.patch, HIVE-13395.7.patch, HIVE-13395.8.patch > > > ACID users can run into Lost Update problem. > In Hive 1.2, Driver.recordValidTxns() (which records the snapshot to use for > the query) is called in Driver.compile(). > Now suppose to concurrent "update T set x = x + 1" are executed. (for > simplicity assume there is exactly 1 row in T) > What can happen is that both compile at the same time (more precisely before > acquireLocksAndOpenTxn() in runInternal() is called) and thus will lock in > the same snapshot, say the value of x = 7 in this snapshot. > Now 1 will get the lock on the row, the second will block. > Now 1, makes x = 8 and commits. > Now 2 proceeds and makes x = 8 again since in it's snapshot x is still 7. > This specific issue is solved in Hive 1.3/2.0 (HIVE-11077 which is a large > patch that deals with multi-statement txns) by moving recordValidTxns() after > locks are acquired which reduces the likelihood of this but doesn't eliminate > the problem. > > Even in 1.3 version of the code, you could have the same issue. Assume the > same 2 queries: > Both start a txn, say txnid 9 and 10. Say 10 gets the lock first, 9 blocks. > 10 updates the row (so x = 8) and thus ReaderKey.currentTransactionId=10. > 10 commits. > Now 9 can proceed and it will get a snapshot that includes 10, i.e. it will > see x = 8 and it will write x = 9, but it will set > ReaderKey.currentTransactionId = 9. Thus when merge logic runs, it will see > x = 8 is the later version of this row, i.e. lost update. > The problem is that locks alone are insufficient for MVCC architecture. > > At lower level Row ID has (originalTransactionId, rowid, bucket id, > currentTransactionId) and since on update/delete we do a table scan, we could > check that we are about to write a row with currentTransactionId < > (currentTransactionId of row we've read) and fail the query. Currently, > currentTransactionId is not surfaced at higher level where this check can be > made. > This would not work (efficiently) longer term where we want to support fast > update on user defined PK vis streaming ingest. > Also, this would not work with multi statement txns since in that case we'd > lock in the snapshot at the start of the txn, but then 2nd, 3rd etc queries > would use the same snapshot and the locks for these queries would be acquired > after the snapshot is locked in so this would be the same situation as pre > HIVE-11077. > > > A more robust solution (commonly used with MVCC) is to keep track of start > and commit time (logical counter) or each transaction to detect if two txns > overlap. The 2nd part is to keep track of write-set, i.e. which data (rows, > partitions, whatever appropriate level of granularity is) were modified by > any txn and if 2 txns overlap in time and wrote the same element, abort later > one. This is called first-committer-wins rule. This requires a MS DB schema > change > It would be most convenient to use the same sequence for txnId, start and > commit time (in which case txnid=start time). In this
[jira] [Commented] (HIVE-13449) LLAP: HS2 should get the token directly, rather than from LLAP
[ https://issues.apache.org/jira/browse/HIVE-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271706#comment-15271706 ] Sergey Shelukhin commented on HIVE-13449: - [~hagleitn] [~vikram.dixit] ping? > LLAP: HS2 should get the token directly, rather than from LLAP > -- > > Key: HIVE-13449 > URL: https://issues.apache.org/jira/browse/HIVE-13449 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13449.01.patch, HIVE-13449.02.patch, > HIVE-13449.patch > > > HS2 doesn't need a roundtrip to LLAP; it can instantiate the SecretManager > directly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13597) revert HIVE-12892
[ https://issues.apache.org/jira/browse/HIVE-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13597: Attachment: HIVE-13597.01.patch > revert HIVE-12892 > - > > Key: HIVE-13597 > URL: https://issues.apache.org/jira/browse/HIVE-13597 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13597.01.patch, HIVE-13597.01.patch, > HIVE-13597.nogen.patch, HIVE-13597.patch > > > Updating functions in LLAP via a metastore call is not going to be committed > in the current form due to other design considerations. > For now, we should revert the changes made in HIVE-12892; there's a DB change > involved there, and we don't want that to be included in a released version > before we decide on the final design of the update feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13597) revert HIVE-12892
[ https://issues.apache.org/jira/browse/HIVE-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271703#comment-15271703 ] Sergey Shelukhin commented on HIVE-13597: - Will commit after HiveQA > revert HIVE-12892 > - > > Key: HIVE-13597 > URL: https://issues.apache.org/jira/browse/HIVE-13597 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13597.01.patch, HIVE-13597.01.patch, > HIVE-13597.nogen.patch, HIVE-13597.patch > > > Updating functions in LLAP via a metastore call is not going to be committed > in the current form due to other design considerations. > For now, we should revert the changes made in HIVE-12892; there's a DB change > involved there, and we don't want that to be included in a released version > before we decide on the final design of the update feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13597) revert HIVE-12892
[ https://issues.apache.org/jira/browse/HIVE-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271690#comment-15271690 ] Ashutosh Chauhan commented on HIVE-13597: - +1 > revert HIVE-12892 > - > > Key: HIVE-13597 > URL: https://issues.apache.org/jira/browse/HIVE-13597 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13597.01.patch, HIVE-13597.nogen.patch, > HIVE-13597.patch > > > Updating functions in LLAP via a metastore call is not going to be committed > in the current form due to other design considerations. > For now, we should revert the changes made in HIVE-12892; there's a DB change > involved there, and we don't want that to be included in a released version > before we decide on the final design of the update feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13592) metastore calls map is not thread safe
[ https://issues.apache.org/jira/browse/HIVE-13592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13592: Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to master after rebase. Thanks for the review! > metastore calls map is not thread safe > -- > > Key: HIVE-13592 > URL: https://issues.apache.org/jira/browse/HIVE-13592 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.1.0 > > Attachments: HIVE-13592.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-12579) add support for datanucleus.connectionPoolingType=None in TxnHandler.setupJdbcConnectionPool()
[ https://issues.apache.org/jira/browse/HIVE-12579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman resolved HIVE-12579. --- Resolution: Fixed this is fixed in HIVE-13159 > add support for datanucleus.connectionPoolingType=None in > TxnHandler.setupJdbcConnectionPool() > -- > > Key: HIVE-12579 > URL: https://issues.apache.org/jira/browse/HIVE-12579 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > "None" is a valid option for datanucleus.connectionPoolingType > http://www.datanucleus.org/products/accessplatform_2_2/rdbms/connection_pooling.html#Manual. > TxnHandler.setupJdbcConnectionPool() doesn't support it. > If nothing else, this is useful for debugging. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12403) Too many delta files during Compaction - OOM Part Deux
[ https://issues.apache.org/jira/browse/HIVE-12403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-12403: -- Assignee: (was: Eugene Koifman) > Too many delta files during Compaction - OOM Part Deux > -- > > Key: HIVE-12403 > URL: https://issues.apache.org/jira/browse/HIVE-12403 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0 >Reporter: Eugene Koifman > > HIVE-11540 solution is somewhat incomplete. > Suppose hive.compactor.max.num.delta=10 and there are 1000 delta files to > compact. > CompactorMR will currently run 100 jobs on 10 files each to create 100 wider > delta files but then it will run 1 job which will include 100 of the wider > deltas. > it's a simple fix to make sure the max.num.delta is always respected -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-6191) remove explicit Joda dependency from itests/hcatalog-unit/pom.xml
[ https://issues.apache.org/jira/browse/HIVE-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-6191: - Assignee: (was: Eugene Koifman) > remove explicit Joda dependency from itests/hcatalog-unit/pom.xml > - > > Key: HIVE-6191 > URL: https://issues.apache.org/jira/browse/HIVE-6191 > Project: Hive > Issue Type: Sub-task > Components: HCatalog, Tests >Affects Versions: 0.13.0 >Reporter: Eugene Koifman > > Joda library is used by Pig and should automatically be pulled down by MVN. > Unfortunately Pig 12 is missing the relevant attribute from it's build file > (PIG-3516) so I added Joda explicitly to itests/hcatalog-unit/pom.xml. This > should be removed once Pig 13 is released and HCat dependency is upgraded. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10986) Check of fs.trash.interval in HiveMetaStore should be consistent with Trash.moveToAppropriateTrash()
[ https://issues.apache.org/jira/browse/HIVE-10986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-10986: -- Assignee: (was: Eugene Koifman) > Check of fs.trash.interval in HiveMetaStore should be consistent with > Trash.moveToAppropriateTrash() > > > Key: HIVE-10986 > URL: https://issues.apache.org/jira/browse/HIVE-10986 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 1.2.1 >Reporter: Eugene Koifman > Attachments: HIVE-10986.2.patch, HIVE-10986.3.patch, HIVE-10986.patch > > > This is a followup to HIVE-10629. > Trash.moveToAppropriateTrash() takes core-site.xml but HiveMetaStore checks > "hiveConf" which is a problem when they disagree. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-12725) CompactionTxnHandler.findNextToCompact() may produce "Operation not allowed after ResultSet closed"
[
https://issues.apache.org/jira/browse/HIVE-12725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman resolved HIVE-12725.
---
Resolution: Fixed
Fix Version/s: 2.1.0
1.3.0
this was fixed in HIVE-11388
> CompactionTxnHandler.findNextToCompact() may produce "Operation not allowed
> after ResultSet closed"
> ---
>
> Key: HIVE-12725
> URL: https://issues.apache.org/jira/browse/HIVE-12725
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 1.3.0, 2.1.0
>
>
> {noformat}
> 2015-12-21 22:34:55,427 ERROR txn.CompactionTxnHandler
> (CompactionTxnHandler.java:findNextToCompact(210)) -
> [pre-prod-poc-4.novalocal-41] Unable to select next element for \
> compaction, Operation not allowed after ResultSet closed
> 2015-12-21 22:34:55,431 ERROR compactor.Worker (Worker.java:run(181)) -
> [pre-prod-poc-4.novalocal-41] Caught an exception in the main loop of
> compactor worker pre-prod-poc-\
> 4.novalocal-41, MetaException(message:Unable to connect to transaction
> database java.sql.SQLException: Operation not allowed after ResultSet closed
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
> at com.mysql.jdbc.ResultSetImpl.checkClosed(ResultSetImpl.java:794)
> at com.mysql.jdbc.ResultSetImpl.next(ResultSetImpl.java:7139)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findNextToCompact(CompactionTxnHandler.java:206)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:81)
> )
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findNextToCompact(CompactionTxnHandler.java:214)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:81)
> {noformat}
> https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html
> {noformat}
> int updCount = stmt.executeUpdate(s);
> if(updCount == 1) {
> dbConn.commit();
> return info;
> }
> if(updCount == 0) {
> LOG.debug("Another Worker picked up " + info);
> continue;
> }
> {noformat}
> executeUpdate() closes the RS and if competing Worker "steals" the item then
> updCount == 0 is true...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13597) revert HIVE-12892
[ https://issues.apache.org/jira/browse/HIVE-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13597: Attachment: HIVE-13597.01.patch retry for HiveQA > revert HIVE-12892 > - > > Key: HIVE-13597 > URL: https://issues.apache.org/jira/browse/HIVE-13597 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13597.01.patch, HIVE-13597.nogen.patch, > HIVE-13597.patch > > > Updating functions in LLAP via a metastore call is not going to be committed > in the current form due to other design considerations. > For now, we should revert the changes made in HIVE-12892; there's a DB change > involved there, and we don't want that to be included in a released version > before we decide on the final design of the update feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11286) insert values clause should support functions
[ https://issues.apache.org/jira/browse/HIVE-11286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-11286: -- Assignee: (was: Eugene Koifman) > insert values clause should support functions > - > > Key: HIVE-11286 > URL: https://issues.apache.org/jira/browse/HIVE-11286 > Project: Hive > Issue Type: Improvement > Components: SQL, Transactions >Affects Versions: 1.2.0 >Reporter: Eugene Koifman > > insert into T values(1,2) is supported > but > insert into T values(1,current_date()) is not - this would be useful -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13354) Add ability to specify Compaction options per table and per request
[ https://issues.apache.org/jira/browse/HIVE-13354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13354: -- Assignee: Wei Zheng (was: Eugene Koifman) > Add ability to specify Compaction options per table and per request > --- > > Key: HIVE-13354 > URL: https://issues.apache.org/jira/browse/HIVE-13354 > Project: Hive > Issue Type: Improvement >Affects Versions: 1.3.0, 2.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Labels: TODOC2.1 > Attachments: HIVE-13354.1.withoutSchemaChange.patch > > > Currently the are a few options that determine when automatic compaction is > triggered. They are specified once for the warehouse. > This doesn't make sense - some table may be more important and need to be > compacted more often. > We should allow specifying these on per table basis. > Also, compaction is an MR job launched from within the metastore. There is > currently no way to control job parameters (like memory, for example) except > to specify it in hive-site.xml for metastore which means they are site wide. > Should add a way to specify these per table (perhaps even per compaction if > launched via ALTER TABLE) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13676) Tests failing because metastore doesn't come up
[
https://issues.apache.org/jira/browse/HIVE-13676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271638#comment-15271638
]
Hive QA commented on HIVE-13676:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12801931/HIVE-13676.01.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 37 failed/errored test(s), 9978 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_join1.q-schema_evol_text_vec_mapwork_part_all_complex.q-vector_complex_join.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-groupby2.q-tez_dynpart_hashjoin_1.q-custom_input_output_format.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_grouping_sets.q-update_all_partitioned.q-cte_5.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStoreWithEnvironmentContext.testEnvironmentContext
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.insertOverwriteCreate
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testDummyTxnManagerOnAcidTable
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.hcatalog.listener.TestDbNotificationListener.cleanupNotifs
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropDatabase
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.org.apache.hive.minikdc.TestJdbcWithDBTokenStore
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/177/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/177/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-177/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 37 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12801931 - PreCommit-HIVE-MASTER-Build
> Tests failing
[jira] [Updated] (HIVE-11550) ACID queries pollute HiveConf
[ https://issues.apache.org/jira/browse/HIVE-11550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-11550: -- Status: Patch Available (was: Open) > ACID queries pollute HiveConf > - > > Key: HIVE-11550 > URL: https://issues.apache.org/jira/browse/HIVE-11550 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-11550.1.patch, HIVE-11550.patch > > > HiveConf is a SessionState level object. Some ACID related logic makes > changes to it (which are meant to be per query) but become per SessionState. > See SemanticAnalyzer.checkAcidConstraints() > Also note HiveConf.setVar(conf, > HiveConf.ConfVars.DYNAMICPARTITIONINGMODE, "nonstrict"); > in UpdateDeleteSemancitAnalzyer > [~alangates], do you know of other cases or ideas on how to deal with this > differently? > _SortedDynPartitionOptimizer.process()_ is the place to have the logic to do > _conf.setBoolVar(ConfVars.HIVEOPTSORTDYNAMICPARTITION, false);_ on per query > basis -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11550) ACID queries pollute HiveConf
[ https://issues.apache.org/jira/browse/HIVE-11550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-11550: -- Attachment: HIVE-11550.1.patch > ACID queries pollute HiveConf > - > > Key: HIVE-11550 > URL: https://issues.apache.org/jira/browse/HIVE-11550 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-11550.1.patch, HIVE-11550.patch > > > HiveConf is a SessionState level object. Some ACID related logic makes > changes to it (which are meant to be per query) but become per SessionState. > See SemanticAnalyzer.checkAcidConstraints() > Also note HiveConf.setVar(conf, > HiveConf.ConfVars.DYNAMICPARTITIONINGMODE, "nonstrict"); > in UpdateDeleteSemancitAnalzyer > [~alangates], do you know of other cases or ideas on how to deal with this > differently? > _SortedDynPartitionOptimizer.process()_ is the place to have the logic to do > _conf.setBoolVar(ConfVars.HIVEOPTSORTDYNAMICPARTITION, false);_ on per query > basis -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13690) Shade guava in hive-exec fat jar
[ https://issues.apache.org/jira/browse/HIVE-13690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohan Agarwal updated HIVE-13690: - Status: Patch Available (was: Open) > Shade guava in hive-exec fat jar > > > Key: HIVE-13690 > URL: https://issues.apache.org/jira/browse/HIVE-13690 > Project: Hive > Issue Type: Improvement >Reporter: Rohan Agarwal >Assignee: Rohan Agarwal > Attachments: HIVE-13690.patch > > > The hive-exec fat jar bundles the Guava classes. This makes relying on the > hive-exec fat jar difficult (see HIVE-5733 for more discussion). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13690) Shade guava in hive-exec fat jar
[ https://issues.apache.org/jira/browse/HIVE-13690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohan Agarwal updated HIVE-13690: - Attachment: HIVE-13690.patch > Shade guava in hive-exec fat jar > > > Key: HIVE-13690 > URL: https://issues.apache.org/jira/browse/HIVE-13690 > Project: Hive > Issue Type: Improvement >Reporter: Rohan Agarwal >Assignee: Rohan Agarwal > Attachments: HIVE-13690.patch > > > The hive-exec fat jar bundles the Guava classes. This makes relying on the > hive-exec fat jar difficult (see HIVE-5733 for more discussion). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13688) Variable substitutions not working for shell commands in beeline
[
https://issues.apache.org/jira/browse/HIVE-13688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271574#comment-15271574
]
Xuefu Zhang commented on HIVE-13688:
Hi [~ngangam], I think the code you're referring to is for embedded beeline
serving as a replacement for Hive CLI. In that context, there is at most one
connection, so there is no such confusion.
This problem appeared in the past a couple of times. Frankly, I don't have a
good answer for this (other than pointing to the problem). :)
> Variable substitutions not working for shell commands in beeline
>
>
> Key: HIVE-13688
> URL: https://issues.apache.org/jira/browse/HIVE-13688
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>
> Appears that the variable substitutions work fine for DML and DDL commands.
> > set hivevar:v1=databases;
> No rows affected (0.004 seconds)
> > show ${v1};
> ++--+
> | database_name |
> ++--+
> | default|
> ++--+
> 1 row selected (0.243 seconds)
> > set hivevar:v1=tmp;
> No rows affected (0.061 seconds)
> > create table ${v1} (col1 string);
> No rows affected (0.065 seconds)
> > describe ${v1};
> INFO : Compiling
> command(queryId=hive_20160504105151_cb6ab7f1-26ac-4733-80ed-13a5134ba2b5):
> describe tmp
> INFO : Executing
> command(queryId=hive_20160504105151_cb6ab7f1-26ac-4733-80ed-13a5134ba2b5):
> describe tmp
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | col1 | string | |
> +---++--+--+
> 1 row selected (0.121 seconds)
> But it does not appear to work for shell commands (introduced in HIVE-6791 )
> > !sh sudo -u hdfs hdfs dfs -mkdir /user/root/${v1}
> mkdir: `/user/root/${v1}': No such file or directory <== problem
> Command failed with exit code = 1
> > !sh echo ${v1};
> ${v1};
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13660) Vectorizing IN expression with list of columns throws java.lang.ClassCastException ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc
[ https://issues.apache.org/jira/browse/HIVE-13660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13660: Resolution: Fixed Status: Resolved (was: Patch Available) > Vectorizing IN expression with list of columns throws > java.lang.ClassCastException ExprNodeColumnDesc cannot be cast to > ExprNodeConstantDesc > > > Key: HIVE-13660 > URL: https://issues.apache.org/jira/browse/HIVE-13660 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13660.01.patch > > > Example: > SELECT * FROM alltypesorc WHERE cint in (ctinyint, cbigint); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13660) Vectorizing IN expression with list of columns throws java.lang.ClassCastException ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc
[ https://issues.apache.org/jira/browse/HIVE-13660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271532#comment-15271532 ] Matt McCline commented on HIVE-13660: - Committed to master and branch-1. > Vectorizing IN expression with list of columns throws > java.lang.ClassCastException ExprNodeColumnDesc cannot be cast to > ExprNodeConstantDesc > > > Key: HIVE-13660 > URL: https://issues.apache.org/jira/browse/HIVE-13660 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13660.01.patch > > > Example: > SELECT * FROM alltypesorc WHERE cint in (ctinyint, cbigint); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13660) Vectorizing IN expression with list of columns throws java.lang.ClassCastException ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc
[ https://issues.apache.org/jira/browse/HIVE-13660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271534#comment-15271534 ] Matt McCline commented on HIVE-13660: - Thank you [~prasanth_j] for your review. > Vectorizing IN expression with list of columns throws > java.lang.ClassCastException ExprNodeColumnDesc cannot be cast to > ExprNodeConstantDesc > > > Key: HIVE-13660 > URL: https://issues.apache.org/jira/browse/HIVE-13660 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13660.01.patch > > > Example: > SELECT * FROM alltypesorc WHERE cint in (ctinyint, cbigint); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11417) Create shims for the row by row read path that is backed by VectorizedRowBatch
[ https://issues.apache.org/jira/browse/HIVE-11417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271533#comment-15271533 ] Sergey Shelukhin commented on HIVE-11417: - [~owen.omalley] is there a plan to commit this soon? If not, I'd like to commit some code separately for HIVE-13617 > Create shims for the row by row read path that is backed by VectorizedRowBatch > -- > > Key: HIVE-11417 > URL: https://issues.apache.org/jira/browse/HIVE-11417 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 2.1.0 > > Attachments: HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch, > HIVE-11417.patch > > > I'd like to make the default path for reading and writing ORC files to be > vectorized. To ensure that Hive can still read row by row, we'll need shims > to support the old API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13620) Merge llap branch work to master
[ https://issues.apache.org/jira/browse/HIVE-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271517#comment-15271517 ] Jason Dere commented on HIVE-13620: --- Patch v9 fixes TestLlapOutputFormat. The other new failures do not seem to be related. > Merge llap branch work to master > > > Key: HIVE-13620 > URL: https://issues.apache.org/jira/browse/HIVE-13620 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13620.1.patch, HIVE-13620.2.patch, > HIVE-13620.3.patch, HIVE-13620.4.patch, HIVE-13620.5.patch, > HIVE-13620.6.patch, HIVE-13620.8.patch, HIVE-13620.9.patch, > llap_master_diff.txt > > > Would like to try to merge the llap branch work for HIVE-12991 into the > master branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13669) LLAP: io.enabled config is ignored on the server side
[ https://issues.apache.org/jira/browse/HIVE-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13669: Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to master. Will merge into branch-2 if there's another RC for 2.0.1 (or later for 2.0.2) > LLAP: io.enabled config is ignored on the server side > - > > Key: HIVE-13669 > URL: https://issues.apache.org/jira/browse/HIVE-13669 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.1.0 > > Attachments: HIVE-13669.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13159) TxnHandler should support datanucleus.connectionPoolingType = None
[ https://issues.apache.org/jira/browse/HIVE-13159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271499#comment-15271499 ] Alan Gates commented on HIVE-13159: --- I ran the tests locally and got: Test Result (25 failures / -7) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32 org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefaultEmptyString org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_select_read_only_encrypted_tbl org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_selectindate org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avrocountemptytbl org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order_null org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping org.apache.hive.minikdc.TestMiniHiveKdc.testLogin org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver I don't believe any of those are related, and all but four (testCliDriver_join32, testFragmentSpecToTaskSpec, testGetMetaConfDefaultEmptyString, testAddPartitions) failed on a run without this patch. But wow we have a lot of failing tests. Thanks Eugene for the review, I'll make the recommended changes. > TxnHandler should support datanucleus.connectionPoolingType = None > -- > > Key: HIVE-13159 > URL: https://issues.apache.org/jira/browse/HIVE-13159 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Sergey Shelukhin >Assignee: Alan Gates > Attachments: HIVE-13159.patch > > > Right now, one has to choose bonecp or dbcp. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13660) Vectorizing IN expression with list of columns throws java.lang.ClassCastException ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc
[ https://issues.apache.org/jira/browse/HIVE-13660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271495#comment-15271495 ] Matt McCline commented on HIVE-13660: - On internal PTest run on HIVE-13660.01.patch, there were lots of random failures, but none related to this change. > Vectorizing IN expression with list of columns throws > java.lang.ClassCastException ExprNodeColumnDesc cannot be cast to > ExprNodeConstantDesc > > > Key: HIVE-13660 > URL: https://issues.apache.org/jira/browse/HIVE-13660 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13660.01.patch > > > Example: > SELECT * FROM alltypesorc WHERE cint in (ctinyint, cbigint); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-12562) Enabling native fast hash table can cause incorrect results
[ https://issues.apache.org/jira/browse/HIVE-12562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline reassigned HIVE-12562: --- Assignee: Matt McCline > Enabling native fast hash table can cause incorrect results > --- > > Key: HIVE-12562 > URL: https://issues.apache.org/jira/browse/HIVE-12562 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Matt McCline > > Enabling "hive.vectorized.execution.mapjoin.native.fast.hashtable.enabled" > causes incorrect results when running with LLAP. > I believe this does not happen for simple container runs. However, it's > possible that caching of these tables, or using the same table more than once > causes issues - which may be seen with container reuse. > The results vary by a small percentage. > e.g. 82270, 82267 <- Two results for the same query run back to back. > cc [~mmccline] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13620) Merge llap branch work to master
[ https://issues.apache.org/jira/browse/HIVE-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-13620: -- Attachment: HIVE-13620.9.patch > Merge llap branch work to master > > > Key: HIVE-13620 > URL: https://issues.apache.org/jira/browse/HIVE-13620 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13620.1.patch, HIVE-13620.2.patch, > HIVE-13620.3.patch, HIVE-13620.4.patch, HIVE-13620.5.patch, > HIVE-13620.6.patch, HIVE-13620.8.patch, HIVE-13620.9.patch, > llap_master_diff.txt > > > Would like to try to merge the llap branch work for HIVE-12991 into the > master branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13688) Variable substitutions not working for shell commands in beeline
[
https://issues.apache.org/jira/browse/HIVE-13688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271449#comment-15271449
]
Naveen Gangam commented on HIVE-13688:
--
Thanks [~xuefuz] good to hear from you.
Good point. However, looking at the patch from
https://issues.apache.org/jira/secure/attachment/12742071/HIVE-6791.5-beeline-cli.patch
it appears the substitution for the shell commands seems to occurring on the
client side (beeline) in Commands.substituteVariables() which is using the
variables returned by "set" statement (from getConfInternal()) which should
return variables set for that session (specific HS2 instance).
Is my understanding incorrect? Thanks
> Variable substitutions not working for shell commands in beeline
>
>
> Key: HIVE-13688
> URL: https://issues.apache.org/jira/browse/HIVE-13688
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>
> Appears that the variable substitutions work fine for DML and DDL commands.
> > set hivevar:v1=databases;
> No rows affected (0.004 seconds)
> > show ${v1};
> ++--+
> | database_name |
> ++--+
> | default|
> ++--+
> 1 row selected (0.243 seconds)
> > set hivevar:v1=tmp;
> No rows affected (0.061 seconds)
> > create table ${v1} (col1 string);
> No rows affected (0.065 seconds)
> > describe ${v1};
> INFO : Compiling
> command(queryId=hive_20160504105151_cb6ab7f1-26ac-4733-80ed-13a5134ba2b5):
> describe tmp
> INFO : Executing
> command(queryId=hive_20160504105151_cb6ab7f1-26ac-4733-80ed-13a5134ba2b5):
> describe tmp
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | col1 | string | |
> +---++--+--+
> 1 row selected (0.121 seconds)
> But it does not appear to work for shell commands (introduced in HIVE-6791 )
> > !sh sudo -u hdfs hdfs dfs -mkdir /user/root/${v1}
> mkdir: `/user/root/${v1}': No such file or directory <== problem
> Command failed with exit code = 1
> > !sh echo ${v1};
> ${v1};
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13657) Spark driver stderr logs should appear in hive client logs
[
https://issues.apache.org/jira/browse/HIVE-13657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271391#comment-15271391
]
Hive QA commented on HIVE-13657:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12801906/HIVE-13657.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10062 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/176/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/176/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-176/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12801906 - PreCommit-HIVE-MASTER-Build
> Spark driver stderr logs should appear in hive client logs
> --
>
> Key: HIVE-13657
> URL: https://issues.apache.org/jira/browse/HIVE-13657
> Project: Hive
> Issue Type: Bug
>Reporter: Mohit Sabharwal
>Assignee: Mohit Sabharwal
> Attachments: HIVE-13657.patch
>
>
> Currently, spark driver exceptions are not getting logged in beeline.
> Instead, the users sees the not-so-useful:
> {code}
> ERROR : Failed to execute spark task, with exception
> 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark
> client.)'
>
> {code}
> The user has to look at HS2 logs to discover the root cause:
> {code}
> 2015-04-01 11:33:16,048 INFO org.apache.hive.spark.client.SparkClientImpl:
> 15/04/01 11:33:16 WARN UserGroupInformation: PriviledgedActionException
> as:foo (auth:PROXY) via hive (auth:SIMPLE)
> cause:org.apache.hadoop.security.AccessControlException: Permission denied:
> user=foo, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
> ...
> {code}
> We should surface these critical errors in hive client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13688) Variable substitutions not working for shell commands in beeline
[
https://issues.apache.org/jira/browse/HIVE-13688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271386#comment-15271386
]
Xuefu Zhang commented on HIVE-13688:
I don't believe this is a bug. Hive variables are specific for a Hive user
session and variable substitution happens on the server side. Beeline can
establish connections to multiple servers, so there can be multiple sessions,
each having its own variables. Now if you wan to substitute your shell command
at the client side, which variable value would you use? Therefore, doing so can
have semantic difficulty.
> Variable substitutions not working for shell commands in beeline
>
>
> Key: HIVE-13688
> URL: https://issues.apache.org/jira/browse/HIVE-13688
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>
> Appears that the variable substitutions work fine for DML and DDL commands.
> > set hivevar:v1=databases;
> No rows affected (0.004 seconds)
> > show ${v1};
> ++--+
> | database_name |
> ++--+
> | default|
> ++--+
> 1 row selected (0.243 seconds)
> > set hivevar:v1=tmp;
> No rows affected (0.061 seconds)
> > create table ${v1} (col1 string);
> No rows affected (0.065 seconds)
> > describe ${v1};
> INFO : Compiling
> command(queryId=hive_20160504105151_cb6ab7f1-26ac-4733-80ed-13a5134ba2b5):
> describe tmp
> INFO : Executing
> command(queryId=hive_20160504105151_cb6ab7f1-26ac-4733-80ed-13a5134ba2b5):
> describe tmp
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | col1 | string | |
> +---++--+--+
> 1 row selected (0.121 seconds)
> But it does not appear to work for shell commands (introduced in HIVE-6791 )
> > !sh sudo -u hdfs hdfs dfs -mkdir /user/root/${v1}
> mkdir: `/user/root/${v1}': No such file or directory <== problem
> Command failed with exit code = 1
> > !sh echo ${v1};
> ${v1};
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13363) Add hive.metastore.token.signature property to HiveConf
[
https://issues.apache.org/jira/browse/HIVE-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271349#comment-15271349
]
Carl Steinbach commented on HIVE-13363:
---
+1
> Add hive.metastore.token.signature property to HiveConf
> ---
>
> Key: HIVE-13363
> URL: https://issues.apache.org/jira/browse/HIVE-13363
> Project: Hive
> Issue Type: Improvement
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-13363.1.patch, HIVE-13363.2.patch
>
>
> I noticed that the {{hive.metastore.token.signature}} property is not defined
> in HiveConf.java, but hardcoded everywhere it's used in the Hive codebase.
> [HIVE-2963] fixes this but was never committed due to being resolved as a
> duplicate ticket.
> We should add {{hive.metastore.token.signature}} to HiveConf.java to
> centralize its definition and make the property more discoverable (it's
> useful to set it when talking to multiple metastores).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13351) Support drop Primary Key/Foreign Key constraints
[ https://issues.apache.org/jira/browse/HIVE-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13351: - Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to master. > Support drop Primary Key/Foreign Key constraints > > > Key: HIVE-13351 > URL: https://issues.apache.org/jira/browse/HIVE-13351 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Fix For: 2.1.0 > > Attachments: HIVE-13351.1.patch, HIVE-13351.2.patch > > > ALTER TABLE TABLENAME DROP CONSTRAINT CONSTRAINTNAME; > The CONSTRAINTNAME has to be associated with the mentioned table, i.e. there > should be atleast 1 table column of TABLENAME participating in the constraint. > Otherwise, we should throw an error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13351) Support drop Primary Key/Foreign Key constraints
[
https://issues.apache.org/jira/browse/HIVE-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271247#comment-15271247
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-13351:
--
Internal test runs gave the following results. Analyzed the failures below and
it requires a few golden file updates (in the *constraint* files). Other
failures are unrelated to the patch.
Here is the sample output:
{code}
Test Result (24 failures / ±0)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_with_constraints
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestFirstInFirstOutComparator.testWaitQueueComparatorWithinDagPriority
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefaultEmptyString
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_drop_invalid_constraint1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_drop_invalid_constraint2
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_select_read_only_encrypted_tbl
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_selectindate
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avrocountemptytbl
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order_null
org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hive.minikdc.TestMiniHiveKdc.testLogin
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
{code}
I will commit the patch based on the above results.
> Support drop Primary Key/Foreign Key constraints
>
>
> Key: HIVE-13351
> URL: https://issues.apache.org/jira/browse/HIVE-13351
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Logical Optimizer
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13351.1.patch, HIVE-13351.2.patch
>
>
> ALTER TABLE TABLENAME DROP CONSTRAINT CONSTRAINTNAME;
> The CONSTRAINTNAME has to be associated with the mentioned table, i.e. there
> should be atleast 1 table column of TABLENAME participating in the constraint.
> Otherwise, we should throw an error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13608) We should provide better error message while constraints with duplicate names are created
[
https://issues.apache.org/jira/browse/HIVE-13608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271211#comment-15271211
]
Ashutosh Chauhan commented on HIVE-13608:
-
I am not sure even if exception containing
java.sql.SQLIntegrityConstraintViolationException comes from system queries
whether it will be worth retrying, since even on retrial query is sure to fail.
So, if there is Integrity Constraint Violation there is no point in retrying.
> We should provide better error message while constraints with duplicate names
> are created
> -
>
> Key: HIVE-13608
> URL: https://issues.apache.org/jira/browse/HIVE-13608
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13608.1.patch
>
>
> {code}
> PREHOOK: query: create table t1(x int, constraint pk1 primary key (x) disable
> novalidate)
> PREHOOK: type: CREATETABLE
> PREHOOK: Output: database:default
> PREHOOK: Output: default@t1
> POSTHOOK: query: create table t1(x int, constraint pk1 primary key (x)
> disable novalidate)
> POSTHOOK: type: CREATETABLE
> POSTHOOK: Output: database:default
> POSTHOOK: Output: default@t1
> PREHOOK: query: create table t2(x int, constraint pk1 primary key (x) disable
> novalidate)
> PREHOOK: type: CREATETABLE
> PREHOOK: Output: database:default
> PREHOOK: Output: default@t2
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:For direct
> MetaStore DB connections, we don't support retries at the client level.)
> {code}
> In the above case, it seems like useful error message is lost. It looks like
> a generic problem with metastore server/client exception handling and
> message propagation. Seems like exception parsing logic of
> RetryingMetaStoreClient::invoke() needs to be updated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13620) Merge llap branch work to master
[
https://issues.apache.org/jira/browse/HIVE-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271198#comment-15271198
]
Hive QA commented on HIVE-13620:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12802146/HIVE-13620.8.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 48 failed/errored test(s), 10001 tests
executed
*Failed tests:*
{noformat}
TestHBaseAggrStatsCacheIntegration - did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-script_pipe.q-vector_decimal_aggregate.q-acid_globallimit.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-smb_cache.q-transform_ppr2.q-vector_outer_join0.q-and-3-more
- did not produce a TEST-*.xml file
TestSchedulerQueue - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.TestLlapOutputFormat.testValues
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorParallelism
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefaultEmptyString
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfOverridden
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfUnknownPreperty
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testRevokeTimedOutWorkers
org.apache.hadoop.hive.ql.TestTxnCommands2.testInitiatorWithMultipleFailedCompactions
org.apache.hadoop.hive.ql.TestTxnCommands2.updateDeletePartitioned
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.lockConflictDbTable
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
org.apache.hive.s
[jira] [Updated] (HIVE-13395) Lost Update problem in ACID
[ https://issues.apache.org/jira/browse/HIVE-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13395: -- Attachment: HIVE-13395.15.patch patch 15 fixes a couple of tests due to HIVE-13213 - no code changes > Lost Update problem in ACID > --- > > Key: HIVE-13395 > URL: https://issues.apache.org/jira/browse/HIVE-13395 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.2.0, 2.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-13395.11.patch, HIVE-13395.12.patch, > HIVE-13395.13.patch, HIVE-13395.14.patch, HIVE-13395.15.patch, > HIVE-13395.6.patch, HIVE-13395.7.patch, HIVE-13395.8.patch > > > ACID users can run into Lost Update problem. > In Hive 1.2, Driver.recordValidTxns() (which records the snapshot to use for > the query) is called in Driver.compile(). > Now suppose to concurrent "update T set x = x + 1" are executed. (for > simplicity assume there is exactly 1 row in T) > What can happen is that both compile at the same time (more precisely before > acquireLocksAndOpenTxn() in runInternal() is called) and thus will lock in > the same snapshot, say the value of x = 7 in this snapshot. > Now 1 will get the lock on the row, the second will block. > Now 1, makes x = 8 and commits. > Now 2 proceeds and makes x = 8 again since in it's snapshot x is still 7. > This specific issue is solved in Hive 1.3/2.0 (HIVE-11077 which is a large > patch that deals with multi-statement txns) by moving recordValidTxns() after > locks are acquired which reduces the likelihood of this but doesn't eliminate > the problem. > > Even in 1.3 version of the code, you could have the same issue. Assume the > same 2 queries: > Both start a txn, say txnid 9 and 10. Say 10 gets the lock first, 9 blocks. > 10 updates the row (so x = 8) and thus ReaderKey.currentTransactionId=10. > 10 commits. > Now 9 can proceed and it will get a snapshot that includes 10, i.e. it will > see x = 8 and it will write x = 9, but it will set > ReaderKey.currentTransactionId = 9. Thus when merge logic runs, it will see > x = 8 is the later version of this row, i.e. lost update. > The problem is that locks alone are insufficient for MVCC architecture. > > At lower level Row ID has (originalTransactionId, rowid, bucket id, > currentTransactionId) and since on update/delete we do a table scan, we could > check that we are about to write a row with currentTransactionId < > (currentTransactionId of row we've read) and fail the query. Currently, > currentTransactionId is not surfaced at higher level where this check can be > made. > This would not work (efficiently) longer term where we want to support fast > update on user defined PK vis streaming ingest. > Also, this would not work with multi statement txns since in that case we'd > lock in the snapshot at the start of the txn, but then 2nd, 3rd etc queries > would use the same snapshot and the locks for these queries would be acquired > after the snapshot is locked in so this would be the same situation as pre > HIVE-11077. > > > A more robust solution (commonly used with MVCC) is to keep track of start > and commit time (logical counter) or each transaction to detect if two txns > overlap. The 2nd part is to keep track of write-set, i.e. which data (rows, > partitions, whatever appropriate level of granularity is) were modified by > any txn and if 2 txns overlap in time and wrote the same element, abort later > one. This is called first-committer-wins rule. This requires a MS DB schema > change > It would be most convenient to use the same sequence for txnId, start and > commit time (in which case txnid=start time). In this case we'd need to add > 1 filed to TXNS table. The complication here is that we'll be using elements > of the sequence faster and they are used as part of file name of delta and > base dir and currently limited to 7 digits which can be exceeded. So this > would require some thought to handling upgrade/migration. > Also, write-set tracking requires either additional metastore table or > keeping info in HIVE_LOCKS around longer with new state. > > In the short term, on SQL side of things we could (in auto commit mode only) > acquire the locks first and then open the txn AND update these locks with txn > id. > This implies another Thrift change to pass in lockId to openTxn. > The same would not work for Streaming API since it opens several txns at once > and then acquires locks for each. > (Not sure if that's is an issue or not since Streaming only does Insert). > Either way this feels hacky. > > Here is one simple example why we need Write-Set tracking for multi-statement > txns > Consider transactions T ~1~ and T
[jira] [Updated] (HIVE-13664) test patch
[ https://issues.apache.org/jira/browse/HIVE-13664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13664: -- Resolution: Won't Fix Status: Resolved (was: Patch Available) > test patch > -- > > Key: HIVE-13664 > URL: https://issues.apache.org/jira/browse/HIVE-13664 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-13664.2.patch, HIVE-13664.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12634) Add command to kill an ACID transacton
[ https://issues.apache.org/jira/browse/HIVE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271155#comment-15271155 ] Wei Zheng commented on HIVE-12634: -- Test Name Duration Age org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefaultEmptyString 10 sec 1 org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions 3 sec 1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32 6.3 sec 2 org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic 1 min 33 sec2 org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure 3.2 sec 2 org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator 5 sec 3 org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_select_read_only_encrypted_tbl 2 min 15 sec6 org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas 3 sec 10 org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault 10 sec 11 org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static 2 min 11 sec12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_selectindate13 sec 16 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avrocountemptytbl 10 sec 16 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order_null 34 sec 16 org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands 22 sec 16 org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys 1 min 52 sec16 org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore 1 sec 16 org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore 0.31 sec16 org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping 3.2 sec 16 org.apache.hive.minikdc.TestMiniHiveKdc.testLogin 1 min 38 sec16 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 8.6 sec 16 org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver 52 sec 16 Test failures are not related. > Add command to kill an ACID transacton > -- > > Key: HIVE-12634 > URL: https://issues.apache.org/jira/browse/HIVE-12634 > Project: Hive > Issue Type: New Feature > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Labels: TODOC2.1 > Attachments: HIVE-12634.1.patch, HIVE-12634.2.patch, > HIVE-12634.3.patch, HIVE-12634.4.patch, HIVE-12634.5.patch, HIVE-12634.6.patch > > > Should add a CLI command to abort a (runaway) transaction. > This should clean up all state related to this txn. > The initiator of this (if still alive) will get an error trying to > heartbeat/commit, i.e. will become aware that the txn is dead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13458) Heartbeater doesn't fail query when heartbeat fails
[ https://issues.apache.org/jira/browse/HIVE-13458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271151#comment-15271151 ] Wei Zheng commented on HIVE-13458: -- Test Name Duration Age org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32 5.1 sec 1 org.apache.hive.hcatalog.listener.TestDbNotificationListener.cleanupNotifs 1 min 0 sec 1 org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic 1 min 15 sec1 org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure 4.2 sec 1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin3 2 min 23 sec1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_runtime_skewjoin_mapjoin_spark 4.2 sec 1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapjoin_memcheck 0.91 sec1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_temp_table 0.64 sec1 org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator 5 sec 2 org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_select_read_only_encrypted_tbl 1 min 26 sec5 org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas 3 sec 9 org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault 10 sec 10 org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static 1 min 35 sec11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_selectindate12 sec 15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avrocountemptytbl 10 sec 15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order_null 35 sec 15 org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands 22 sec 15 org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys 1 min 34 sec15 org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore 1.6 sec 15 org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore 0.32 sec15 org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping 3.6 sec 15 org.apache.hive.minikdc.TestMiniHiveKdc.testLogin 1 min 25 sec15 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 9.4 sec 15 org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver 1 min 0 sec 15 Test failures are not related, as they all passed locally. > Heartbeater doesn't fail query when heartbeat fails > --- > > Key: HIVE-13458 > URL: https://issues.apache.org/jira/browse/HIVE-13458 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13458.1.patch, HIVE-13458.2.patch, > HIVE-13458.3.patch, HIVE-13458.4.patch, HIVE-13458.5.patch, > HIVE-13458.6.patch, HIVE-13458.7.patch > > > When a heartbeat fails to locate a lock, it should fail the current query. > That doesn't happen, which is a bug. > Another thing is, we need to make sure stopHeartbeat really stops the > heartbeat, i.e. no additional heartbeat will be sent, since that will break > the assumption and cause the query to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13608) We should provide better error message while constraints with duplicate names are created
[
https://issues.apache.org/jira/browse/HIVE-13608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271118#comment-15271118
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-13608:
--
[~ashutoshc] Thinking about it, MITM may not be possible overall because there
are no optional layers anywhere in between the metastore/db, but, if the
message containing 'java.sql.SQLIntegrityConstraintViolationException' did not
originate from the intended source, i.e. in this case from the insert queries
running in metastore db. For e.g., Oracle sometimes internally runs system
queries when a user query is submitted and if the system query fails
temporarily with the final message containing the
java.sql.SQLIntegrityConstraintViolationException string) , we can prevent
retries altogether for the wrong reason.
> We should provide better error message while constraints with duplicate names
> are created
> -
>
> Key: HIVE-13608
> URL: https://issues.apache.org/jira/browse/HIVE-13608
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13608.1.patch
>
>
> {code}
> PREHOOK: query: create table t1(x int, constraint pk1 primary key (x) disable
> novalidate)
> PREHOOK: type: CREATETABLE
> PREHOOK: Output: database:default
> PREHOOK: Output: default@t1
> POSTHOOK: query: create table t1(x int, constraint pk1 primary key (x)
> disable novalidate)
> POSTHOOK: type: CREATETABLE
> POSTHOOK: Output: database:default
> POSTHOOK: Output: default@t1
> PREHOOK: query: create table t2(x int, constraint pk1 primary key (x) disable
> novalidate)
> PREHOOK: type: CREATETABLE
> PREHOOK: Output: database:default
> PREHOOK: Output: default@t2
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:For direct
> MetaStore DB connections, we don't support retries at the client level.)
> {code}
> In the above case, it seems like useful error message is lost. It looks like
> a generic problem with metastore server/client exception handling and
> message propagation. Seems like exception parsing logic of
> RetryingMetaStoreClient::invoke() needs to be updated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13249) Hard upper bound on number of open transactions
[ https://issues.apache.org/jira/browse/HIVE-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271114#comment-15271114 ] Wei Zheng commented on HIVE-13249: -- Above was tests being done. Failures are not related. > Hard upper bound on number of open transactions > --- > > Key: HIVE-13249 > URL: https://issues.apache.org/jira/browse/HIVE-13249 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13249.1.patch, HIVE-13249.2.patch, > HIVE-13249.3.patch, HIVE-13249.4.patch, HIVE-13249.5.patch, HIVE-13249.6.patch > > > We need to have a safeguard by adding an upper bound for open transactions to > avoid huge number of open-transaction requests, usually due to improper > configuration of clients such as Storm. > Once that limit is reached, clients will start failing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13249) Hard upper bound on number of open transactions
[
https://issues.apache.org/jira/browse/HIVE-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271113#comment-15271113
]
Wei Zheng commented on HIVE-13249:
--
{code}
Test Name
Duration
Age
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator
5 sec 1
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
10 sec 2
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
5 sec 3
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_select_read_only_encrypted_tbl
1 min 26 sec4
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
3 sec 8
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
3 sec 8
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
10 sec 9
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
1 min 58 sec10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_selectindate13 sec
14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avrocountemptytbl
10 sec 14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order_null 36 sec
14
org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
21 sec 14
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
1 min 31 sec14
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore
1.8 sec 14
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore
0.34 sec14
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
3.5 sec 14
org.apache.hive.minikdc.TestMiniHiveKdc.testLogin 1 min 20 sec14
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
8.4 sec 14
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
57 sec 14
{code}
> Hard upper bound on number of open transactions
> ---
>
> Key: HIVE-13249
> URL: https://issues.apache.org/jira/browse/HIVE-13249
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-13249.1.patch, HIVE-13249.2.patch,
> HIVE-13249.3.patch, HIVE-13249.4.patch, HIVE-13249.5.patch, HIVE-13249.6.patch
>
>
> We need to have a safeguard by adding an upper bound for open transactions to
> avoid huge number of open-transaction requests, usually due to improper
> configuration of clients such as Storm.
> Once that limit is reached, clients will start failing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13638) CBO rule to pull up constants through Sort/Limit
[ https://issues.apache.org/jira/browse/HIVE-13638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13638: --- Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Pushed to master, thanks for the review [~ashutoshc]! > CBO rule to pull up constants through Sort/Limit > > > Key: HIVE-13638 > URL: https://issues.apache.org/jira/browse/HIVE-13638 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 2.1.0 > > Attachments: HIVE-13638.01.patch, HIVE-13638.patch > > > After HIVE-13068 goes in, we need to pull up constants through Sort/Limit > operator, as it was done previously by ConstantPropagate in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13638) CBO rule to pull up constants through Sort/Limit
[
https://issues.apache.org/jira/browse/HIVE-13638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271101#comment-15271101
]
Jesus Camacho Rodriguez commented on HIVE-13638:
{noformat}
Test Name DurationAge
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefaultEmptyString
10 sec 1
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
10 sec 1
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
5 sec 1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32 6.9 sec 3
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
28 sec 3
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
3.2 sec 3
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
3 sec 7
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_select_read_only_encrypted_tbl
1 min 28 sec12
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
3.2 sec 16
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
10 sec 17
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
1 min 42 sec18
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_selectindate16 sec
22
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avrocountemptytbl
10 sec 22
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order_null 39 sec
22
org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
21 sec 22
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
1 min 32 sec22
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore
1.5 sec 22
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore
0.42 sec22
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
3.3 sec 22
org.apache.hive.minikdc.TestMiniHiveKdc.testLogin 1 min 32 sec22
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
9.9 sec 22
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
47 sec 22
{noformat}
> CBO rule to pull up constants through Sort/Limit
>
>
> Key: HIVE-13638
> URL: https://issues.apache.org/jira/browse/HIVE-13638
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13638.01.patch, HIVE-13638.patch
>
>
> After HIVE-13068 goes in, we need to pull up constants through Sort/Limit
> operator, as it was done previously by ConstantPropagate in Hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13688) Variable substitutions not working for shell commands in beeline
[
https://issues.apache.org/jira/browse/HIVE-13688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-13688:
-
Description:
Appears that the variable substitutions work fine for DML and DDL commands.
> set hivevar:v1=databases;
No rows affected (0.004 seconds)
> show ${v1};
++--+
| database_name |
++--+
| default|
++--+
1 row selected (0.243 seconds)
> set hivevar:v1=tmp;
No rows affected (0.061 seconds)
> create table ${v1} (col1 string);
No rows affected (0.065 seconds)
> describe ${v1};
INFO : Compiling
command(queryId=hive_20160504105151_cb6ab7f1-26ac-4733-80ed-13a5134ba2b5):
describe tmp
INFO : Executing
command(queryId=hive_20160504105151_cb6ab7f1-26ac-4733-80ed-13a5134ba2b5):
describe tmp
+---++--+--+
| col_name | data_type | comment |
+---++--+--+
| col1 | string | |
+---++--+--+
1 row selected (0.121 seconds)
But it does not appear to work for shell commands (introduced in HIVE-6791 )
> !sh sudo -u hdfs hdfs dfs -mkdir /user/root/${v1}
mkdir: `/user/root/${v1}': No such file or directory <== problem
Command failed with exit code = 1
> !sh echo ${v1};
${v1};
was:
Appears that the variable substitutions work fine for DML and DDL commands.
> set hivevar:v1=databases;
No rows affected (0.004 seconds)
> show ${v1};
++--+
| database_name |
++--+
| default|
++--+
1 row selected (0.243 seconds)
> set hivevar:v1=tmp;
No rows affected (0.061 seconds)
> create table ${v1} (col1 string);
No rows affected (0.065 seconds)
> describe ${v1};
INFO : Compiling
command(queryId=hive_20160504105151_cb6ab7f1-26ac-4733-80ed-13a5134ba2b5):
describe tmp
INFO : Executing
command(queryId=hive_20160504105151_cb6ab7f1-26ac-4733-80ed-13a5134ba2b5):
describe tmp
+---++--+--+
| col_name | data_type | comment |
+---++--+--+
| col1 | string | |
+---++--+--+
1 row selected (0.121 seconds)
But it does not appear to work for shell commands (introduced in HIVE-6694 )
> !sh sudo -u hdfs hdfs dfs -mkdir /user/root/${v1}
mkdir: `/user/root/${v1}': No such file or directory <== problem
Command failed with exit code = 1
> !sh echo ${v1};
${v1};
> Variable substitutions not working for shell commands in beeline
>
>
> Key: HIVE-13688
> URL: https://issues.apache.org/jira/browse/HIVE-13688
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>
> Appears that the variable substitutions work fine for DML and DDL commands.
> > set hivevar:v1=databases;
> No rows affected (0.004 seconds)
> > show ${v1};
> ++--+
> | database_name |
> ++--+
> | default|
> ++--+
> 1 row selected (0.243 seconds)
> > set hivevar:v1=tmp;
> No rows affected (0.061 seconds)
> > create table ${v1} (col1 string);
> No rows affected (0.065 seconds)
> > describe ${v1};
> INFO : Compiling
> command(queryId=hive_20160504105151_cb6ab7f1-26ac-4733-80ed-13a5134ba2b5):
> describe tmp
> INFO : Executing
> command(queryId=hive_20160504105151_cb6ab7f1-26ac-4733-80ed-13a5134ba2b5):
> describe tmp
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | col1 | string | |
> +---++--+--+
> 1 row selected (0.121 seconds)
> But it does not appear to work for shell commands (introduced in HIVE-6791 )
> > !sh sudo -u hdfs hdfs dfs -mkdir /user/root/${v1}
> mkdir: `/user/root/${v1}': No such file or directory <== problem
> Command failed with exit code = 1
> > !sh echo ${v1};
> ${v1};
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13363) Add hive.metastore.token.signature property to HiveConf
[
https://issues.apache.org/jira/browse/HIVE-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271014#comment-15271014
]
Ratandeep Ratti commented on HIVE-13363:
The patch looks good to me
> Add hive.metastore.token.signature property to HiveConf
> ---
>
> Key: HIVE-13363
> URL: https://issues.apache.org/jira/browse/HIVE-13363
> Project: Hive
> Issue Type: Improvement
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-13363.1.patch, HIVE-13363.2.patch
>
>
> I noticed that the {{hive.metastore.token.signature}} property is not defined
> in HiveConf.java, but hardcoded everywhere it's used in the Hive codebase.
> [HIVE-2963] fixes this but was never committed due to being resolved as a
> duplicate ticket.
> We should add {{hive.metastore.token.signature}} to HiveConf.java to
> centralize its definition and make the property more discoverable (it's
> useful to set it when talking to multiple metastores).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13342) Improve logging in llap decider and throw exception in case llap mode is all but we cannot run in llap.
[ https://issues.apache.org/jira/browse/HIVE-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15270997#comment-15270997 ] Sergey Shelukhin commented on HIVE-13342: - Test failures appear to be related. > Improve logging in llap decider and throw exception in case llap mode is all > but we cannot run in llap. > --- > > Key: HIVE-13342 > URL: https://issues.apache.org/jira/browse/HIVE-13342 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13342.1.patch, HIVE-13342.2.patch, > HIVE-13342.3.patch, HIVE-13342.4.patch, HIVE-13342.5.patch > > > Currently we do not log our decisions with respect to llap. Are we running > everything in llap mode or only parts of the plan. We need more logging. > Also, if llap mode is all but for some reason, we cannot run the work in llap > mode, fail and throw an exception advise the user to change the mode to auto. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13601) Deadline is not registered at some places in HiveMetastore before the ObjectStore/MetaStoreDirectSql APIs are invoked
[
https://issues.apache.org/jira/browse/HIVE-13601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15270982#comment-15270982
]
Sergey Shelukhin commented on HIVE-13601:
-
cc [~alangates], I think he added the fast path. IIRC it's compatible with
ObjectStore, it's not compatible with either remote metastore or local
metastore, I don't recall
> Deadline is not registered at some places in HiveMetastore before the
> ObjectStore/MetaStoreDirectSql APIs are invoked
> -
>
> Key: HIVE-13601
> URL: https://issues.apache.org/jira/browse/HIVE-13601
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>
> HIVE-13542 exposed this issue as shown below when making Metastore API calls
> via CBO code path.
> {code}
> 16-04-21T11:45:55,201 DEBUG [f2995fd1-e4fc-4783-86cb-9264f715e540 main[]]:
> metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:timingTrace(864)) -
> Direct SQL query in 14.825303ms + 0.086336ms, the query is [select
> "COLUMN_NAME", "COLUMN_TYPE", "LONG_LOW_VALUE", "LONG_HIGH_VALUE",
> "DOUBLE_LOW_VALU\
> E", "DOUBLE_HIGH_VALUE", "BIG_DECIMAL_LOW_VALUE", "BIG_DECIMAL_HIGH_VALUE",
> "NUM_NULLS", "NUM_DISTINCTS", "AVG_COL_LEN", "MAX_COL_LEN", "NUM_TRUES",
> "NUM_FALSES", "LAST_ANALYZED" from "TAB_COL_STATS" where "DB_NAME" = ? and
> "TABLE_NAME" = ? and "COLUMN_NAME" in (...)]
> 2016-04-21T11:45:55,242 WARN [f2995fd1-e4fc-4783-86cb-9264f715e540 main[]]:
> metastore.ObjectStore (ObjectStore.java:handleDirectSqlError(2602)) - Direct
> SQL failed, falling back to ORM
> org.apache.hadoop.hive.metastore.api.MetaException: The threadlocal Deadline
> is null, please register it first.
> at
> org.apache.hadoop.hive.metastore.Deadline.newMetaException(Deadline.java:178)
> ~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Deadline.checkTimeout(Deadline.java:149)
> ~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.makeColumnStats(MetaStoreDirectSql.java:1717)
> ~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getTableStats(MetaStoreDirectSql.java:1201)
> ~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:6849)
> ~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:6846)
> ~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2567)
> [hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:6845)
> [hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatistics(ObjectStore.java:6839)
> [hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_statistics_req(HiveMetaStore.java:4361)
> [hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableColumnStatistics(HiveMetaStoreClient.java:1613)
> [hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTableColumnStatistics(SessionHiveMetaStoreClient.java:347)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getTableColumnStatistics(Hive.java:3317)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:758)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.updateColStats(RelOptHiveTable.java:302)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getColStat(RelOptHiveTable.java:417)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getColStat(RelOptHiveTable.java:410)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan.getColStat(HiveTableScan.java:158)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdDistinctRowCount.getDistinctRowCount(HiveRelMdDistinctRowCount.java:77)
> [hive
[jira] [Commented] (HIVE-13639) CBO rule to pull up constants through Union
[ https://issues.apache.org/jira/browse/HIVE-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15270902#comment-15270902 ] Ashutosh Chauhan commented on HIVE-13639: - +1 pending tests. > CBO rule to pull up constants through Union > --- > > Key: HIVE-13639 > URL: https://issues.apache.org/jira/browse/HIVE-13639 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13639.patch > > > After HIVE-13068 goes in, we need to pull up constants through Union > operator, as it was done previously by ConstantPropagate in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13442) LLAP: refactor submit API to be amenable to signing
[
https://issues.apache.org/jira/browse/HIVE-13442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15270834#comment-15270834
]
Hive QA commented on HIVE-13442:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12801848/HIVE-13442.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/173/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/173/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-173/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-173/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 2d33d09 HIVE-13516: Adding BTEQ .IF, .QUIT, ERRORCODE to HPL/SQL
(Dmitry Tolpeko reviewed by Alan Gates
+ git clean -f -d
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at 2d33d09 HIVE-13516: Adding BTEQ .IF, .QUIT, ERRORCODE to HPL/SQL
(Dmitry Tolpeko reviewed by Alan Gates
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12801848 - PreCommit-HIVE-MASTER-Build
> LLAP: refactor submit API to be amenable to signing
> ---
>
> Key: HIVE-13442
> URL: https://issues.apache.org/jira/browse/HIVE-13442
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 2.1.0
>
> Attachments: HIVE-13442.01.patch, HIVE-13442.nogen.patch,
> HIVE-13442.patch, HIVE-13442.patch, HIVE-13442.protobuf.patch
>
>
> This is going to be a wire compat breaking change.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13445) LLAP: token should encode application and cluster ids
[
https://issues.apache.org/jira/browse/HIVE-13445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15270831#comment-15270831
]
Hive QA commented on HIVE-13445:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12801839/HIVE-13445.05.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/172/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/172/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-172/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-172/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 2d33d09 HIVE-13516: Adding BTEQ .IF, .QUIT, ERRORCODE to HPL/SQL
(Dmitry Tolpeko reviewed by Alan Gates
+ git clean -f -d
Removing common/src/java/org/apache/hadoop/hive/conf/HiveConf.java.orig
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at 2d33d09 HIVE-13516: Adding BTEQ .IF, .QUIT, ERRORCODE to HPL/SQL
(Dmitry Tolpeko reviewed by Alan Gates
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12801839 - PreCommit-HIVE-MASTER-Build
> LLAP: token should encode application and cluster ids
> -
>
> Key: HIVE-13445
> URL: https://issues.apache.org/jira/browse/HIVE-13445
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 2.1.0
>
> Attachments: HIVE-13445.01.patch, HIVE-13445.02.patch,
> HIVE-13445.03.patch, HIVE-13445.04.patch, HIVE-13445.05.patch,
> HIVE-13445.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15270828#comment-15270828
]
Hive QA commented on HIVE-13343:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12801816/HIVE-13343.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 38 failed/errored test(s), 9994 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-bucket_map_join_tez1.q-auto_sortmerge_join_16.q-skewjoin.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorized_parquet.q-orc_merge4.q-orc_merge_incompat1.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_hybridgrace_hashjoin_1
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefaultEmptyString
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfOverridden
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testLockTimeout
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.org.apache.hive.minikdc.TestJdbcWithDBTokenStore
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
org.apache.hive.service.cli.session.TestSessionManagerMetrics.testThreadPoolMetrics
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/171/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/171/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-171/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 38 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12801816 - PreCommit-HIVE-MASTER-Build
> Need to disable hybrid grace hash join in llap mode e
[jira] [Commented] (HIVE-13686) TestRecordReaderImpl is deleting target/tmp causing all the tests after it to fail
[
https://issues.apache.org/jira/browse/HIVE-13686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15270760#comment-15270760
]
Sergio Peña commented on HIVE-13686:
Not sure what happened. The logs were deleted last week.
I will figure out later what is the problem.
Related to the test failure, I've seen that {{hive-site.xml}} is copied from
the root pom.xml to each module target/tmp directory from this line:
https://github.com/apache/hive/blob/master/pom.xml#L914
I've had some issues before when attempting to use test-sources.jar, and I
think the best practice to keep resources is by storing the hive-site.xml on
each module {{src/test/main/resources}} directory. This way we won't have any
issues when some tests delete the temporary directory, as these resources are
copied to target/test-classes.
However, we still have the issue of the metastore db deleted. We can either fix
the ORC test to avoid deleting temp, or use another temp out of target for
metastore.
> TestRecordReaderImpl is deleting target/tmp causing all the tests after it to
> fail
> --
>
> Key: HIVE-13686
> URL: https://issues.apache.org/jira/browse/HIVE-13686
> Project: Hive
> Issue Type: Bug
>Reporter: Rajat Khandelwal
>
> The issue was introduced in HIVE-12159
> (https://github.com/apache/hive/blame/master/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRecordReaderImpl.java).
> This test deletes target/tmp directory. Because of this, the subsequent
> tests don't get to read target/tmp/conf/hive-site.xml, which contains
> test-specific configurations. Also, target/tmp has metastore db directory,
> which also gets deleted causing subsequent tests that use metastore db to
> fail too.
> I'm surprised this issue wasn't caught in pre-commit builds. [~spena] I see
> that even the latest pre-commit jobs aren't reporting any errors, while
> building on local and running hive-exec test cases causes a bunch of tests to
> fail.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7777) Add CSV Serde based on OpenCSV
[
https://issues.apache.org/jira/browse/HIVE-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15270767#comment-15270767
]
chen commented on HIVE-:
any plan to support SERDEPROPERTIES("serialization.encoding"='GBK') function in
this csv server as mentioned in https://issues.apache.org/jira/browse/HIVE-7142?
> Add CSV Serde based on OpenCSV
> --
>
> Key: HIVE-
> URL: https://issues.apache.org/jira/browse/HIVE-
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Ferdinand Xu
>Assignee: Ferdinand Xu
> Fix For: 0.14.0
>
> Attachments: HIVE-.1.patch, HIVE-.2.patch, HIVE-.3.patch,
> HIVE-.patch, csv-serde-master.zip
>
>
> There is no official support for csvSerde for hive while there is an open
> source project in github(https://github.com/ogrodnek/csv-serde). CSV is of
> high frequency in use as a data format.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13686) TestRecordReaderImpl is deleting target/tmp causing all the tests after it to fail
[ https://issues.apache.org/jira/browse/HIVE-13686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15270752#comment-15270752 ] Mohit Sabharwal commented on HIVE-13686: I think this will affect only the subset of tests that (a) get distributed to the same ptest batch as TestRecordReaderImpl (b) get run after TestRecordReaderImpl. My guess is that either this subset doesn't rely on hive-site.xml for some reason or is size zero (i.e. TestRecordReaderImpl runs last). > TestRecordReaderImpl is deleting target/tmp causing all the tests after it to > fail > -- > > Key: HIVE-13686 > URL: https://issues.apache.org/jira/browse/HIVE-13686 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal > > The issue was introduced in HIVE-12159 > (https://github.com/apache/hive/blame/master/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRecordReaderImpl.java). > This test deletes target/tmp directory. Because of this, the subsequent > tests don't get to read target/tmp/conf/hive-site.xml, which contains > test-specific configurations. Also, target/tmp has metastore db directory, > which also gets deleted causing subsequent tests that use metastore db to > fail too. > I'm surprised this issue wasn't caught in pre-commit builds. [~spena] I see > that even the latest pre-commit jobs aren't reporting any errors, while > building on local and running hive-exec test cases causes a bunch of tests to > fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13632) Hive failing on insert empty array into parquet table
[
https://issues.apache.org/jira/browse/HIVE-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15270612#comment-15270612
]
Yongzhi Chen commented on HIVE-13632:
-
Thanks [~spena]
> Hive failing on insert empty array into parquet table
> -
>
> Key: HIVE-13632
> URL: https://issues.apache.org/jira/browse/HIVE-13632
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13632.1.patch, HIVE-13632.2.patch,
> HIVE-13632.3.patch
>
>
> The insert will fail with following stack:
> {noformat}
> by: parquet.io.ParquetEncodingException: empty fields are illegal, the field
> should be ommited completely instead
> at
> parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:271)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$ListDataWriter.write(DataWritableWriter.java:271)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$GroupDataWriter.write(DataWritableWriter.java:199)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$MessageDataWriter.write(DataWritableWriter.java:215)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:88)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31)
> at
> parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:116)
> at
> parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:123)
> at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:42)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:111)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:697)
> {noformat}
> Reproduce:
> {noformat}
> create table test_small (
> key string,
> arrayValues array)
> stored as parquet;
> insert into table test_small select 'abcd', array() from src limit 1;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13639) CBO rule to pull up constants through Union
[ https://issues.apache.org/jira/browse/HIVE-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15270591#comment-15270591 ] Jesus Camacho Rodriguez commented on HIVE-13639: I just did. Thanks > CBO rule to pull up constants through Union > --- > > Key: HIVE-13639 > URL: https://issues.apache.org/jira/browse/HIVE-13639 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13639.patch > > > After HIVE-13068 goes in, we need to pull up constants through Union > operator, as it was done previously by ConstantPropagate in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13639) CBO rule to pull up constants through Union
[ https://issues.apache.org/jira/browse/HIVE-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13639: --- Attachment: HIVE-13639.patch > CBO rule to pull up constants through Union > --- > > Key: HIVE-13639 > URL: https://issues.apache.org/jira/browse/HIVE-13639 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13639.patch > > > After HIVE-13068 goes in, we need to pull up constants through Union > operator, as it was done previously by ConstantPropagate in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13639) CBO rule to pull up constants through Union
[ https://issues.apache.org/jira/browse/HIVE-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13639: --- Attachment: (was: HIVE-13639.patch) > CBO rule to pull up constants through Union > --- > > Key: HIVE-13639 > URL: https://issues.apache.org/jira/browse/HIVE-13639 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > > After HIVE-13068 goes in, we need to pull up constants through Union > operator, as it was done previously by ConstantPropagate in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13639) CBO rule to pull up constants through Union
[ https://issues.apache.org/jira/browse/HIVE-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15270581#comment-15270581 ] Ashutosh Chauhan commented on HIVE-13639: - Can you create a RB for this? > CBO rule to pull up constants through Union > --- > > Key: HIVE-13639 > URL: https://issues.apache.org/jira/browse/HIVE-13639 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13639.patch > > > After HIVE-13068 goes in, we need to pull up constants through Union > operator, as it was done previously by ConstantPropagate in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13490) Change itests to be part of the main Hive build
[ https://issues.apache.org/jira/browse/HIVE-13490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15270579#comment-15270579 ] Ashutosh Chauhan commented on HIVE-13490: - [~kgyrtkirk] There definitely is interest in having ability to run .q tests directly from IDE. Thanks for taking a look. [~spena] IMHO, making it easier for devs to able to run q test out of the box without any hassle far outweighs the risk of folks running mvn test from top dir and waiting forever. We can document that. > Change itests to be part of the main Hive build > --- > > Key: HIVE-13490 > URL: https://issues.apache.org/jira/browse/HIVE-13490 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13490.01.patch, HIVE-13490.02.patch > > > Instead of having to build Hive, and then itests separately. > With IntelliJ, this ends up being loaded as two separate dependencies, and > there's a lot of hops involved to make changes. > Does anyone know why these have been kept separate ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13639) CBO rule to pull up constants through Union
[ https://issues.apache.org/jira/browse/HIVE-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13639: --- Attachment: HIVE-13639.patch There is some overlapping with HIVE-13638 i.e. changes in HiveReduceExpressionsRule are the same. [~ashutoshc], could you take a look? Thanks > CBO rule to pull up constants through Union > --- > > Key: HIVE-13639 > URL: https://issues.apache.org/jira/browse/HIVE-13639 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13639.patch > > > After HIVE-13068 goes in, we need to pull up constants through Union > operator, as it was done previously by ConstantPropagate in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-13639) CBO rule to pull up constants through Union
[ https://issues.apache.org/jira/browse/HIVE-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-13639 started by Jesus Camacho Rodriguez. -- > CBO rule to pull up constants through Union > --- > > Key: HIVE-13639 > URL: https://issues.apache.org/jira/browse/HIVE-13639 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > > After HIVE-13068 goes in, we need to pull up constants through Union > operator, as it was done previously by ConstantPropagate in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13639) CBO rule to pull up constants through Union
[ https://issues.apache.org/jira/browse/HIVE-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13639: --- Status: Patch Available (was: In Progress) > CBO rule to pull up constants through Union > --- > > Key: HIVE-13639 > URL: https://issues.apache.org/jira/browse/HIVE-13639 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > > After HIVE-13068 goes in, we need to pull up constants through Union > operator, as it was done previously by ConstantPropagate in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13671) Add PerfLogger to log4j2.properties logger
[
https://issues.apache.org/jira/browse/HIVE-13671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15270558#comment-15270558
]
Hive QA commented on HIVE-13671:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12801845/HIVE-13671.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 72 failed/errored test(s), 9992 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_auto_smb_mapjoin_14.q-vectorized_distinct_gby.q-union_fast_stats.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_interval_2.q-schema_evol_text_nonvec_mapwork_part_all_primitive.q-tez_fsstat.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_bucket_map_join_tez2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_3
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_5
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_3
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_4
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_5
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llap_nullscan
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llapdecider
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dml
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_hash
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_tests
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_joins_explain
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_smb_main
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefaultEmptyString
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfOverridden
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfUnknownPreperty
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.lock
[jira] [Commented] (HIVE-13687) java.lang.RuntimeException: Reduce operator initialization failed
[ https://issues.apache.org/jira/browse/HIVE-13687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15270544#comment-15270544 ] liuguanghua commented on HIVE-13687: when i run under the sql，it throw upper errors。 set spark.executor.instances=500; add jar ./GroupConcat.jar; create temporary function group_concat as 'com.letv.udaf.GroupConcat'; CREATE TEMPORARY FUNCTION MAXVALUE AS 'com.letv.udf.MaxValue'; select b.id,b.pid,b.vv,b.cv,c.gender,c.age,c.edu,c.purpower from ( select id, pid, sum(case when (init+play+time) >0 then 1 else 0 end) vv, sum(case when (play+time) > 0 then 1 else 0 end) cv from data_sum.sum_user_uuid_play_day where dt>20160318 and dt<=20160417 and pid!='-' and (case when (play+time) > 0 then 1 else 0 end)>0 group by id,pid )b join ( select id, MAXVALUE(gender,0.5) gender, MAXVALUE(age,0) as age, '-' as edu, '-' as purpower from temp.temp_predict_tags where dt='20160417' )c on b.id=c.id; > java.lang.RuntimeException: Reduce operator initialization failed > - > > Key: HIVE-13687 > URL: https://issues.apache.org/jira/browse/HIVE-13687 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0 > Environment: hadoop2.6.4+hive2.0.0+spark1.6.1 > I run sql on the hive on spark model. >Reporter: liuguanghua > > java.lang.RuntimeException: Reduce operator initialization failed > at > org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.init(SparkReduceRecordHandler.java:210) > at > org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunction.call(HiveReduceFunction.java:46) > at > org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunction.call(HiveReduceFunction.java:28) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:192) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:192) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.IndexOutOfBoundsException: Index: 5, Size: 5 > at java.util.ArrayList.rangeCheck(ArrayList.java:635) > at java.util.ArrayList.get(ArrayList.java:411) > at > org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.init(StandardStructObjectInspector.java:118) > at > org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.(StandardStructObjectInspector.java:109) > at > org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:326) > at > org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:311) > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.getJoinOutputObjectInspector(CommonJoinOperator.java:181) > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.initializeOp(CommonJoinOperator.java:319) > at > org.apache.hadoop.hive.ql.exec.JoinOperator.initializeOp(JoinOperator.java:70) > at > org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:355) > at > org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.init(SparkReduceRecordHandler.java:195) > ... 16 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13687) java.lang.RuntimeException: Reduce operator initialization failed
[ https://issues.apache.org/jira/browse/HIVE-13687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuguanghua updated HIVE-13687: --- Assignee: (was: liuguanghua) > java.lang.RuntimeException: Reduce operator initialization failed > - > > Key: HIVE-13687 > URL: https://issues.apache.org/jira/browse/HIVE-13687 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0 > Environment: hadoop2.6.4+hive2.0.0+spark1.6.1 > I run sql on the hive on spark model. >Reporter: liuguanghua > > java.lang.RuntimeException: Reduce operator initialization failed > at > org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.init(SparkReduceRecordHandler.java:210) > at > org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunction.call(HiveReduceFunction.java:46) > at > org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunction.call(HiveReduceFunction.java:28) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:192) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:192) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.IndexOutOfBoundsException: Index: 5, Size: 5 > at java.util.ArrayList.rangeCheck(ArrayList.java:635) > at java.util.ArrayList.get(ArrayList.java:411) > at > org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.init(StandardStructObjectInspector.java:118) > at > org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.(StandardStructObjectInspector.java:109) > at > org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:326) > at > org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:311) > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.getJoinOutputObjectInspector(CommonJoinOperator.java:181) > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.initializeOp(CommonJoinOperator.java:319) > at > org.apache.hadoop.hive.ql.exec.JoinOperator.initializeOp(JoinOperator.java:70) > at > org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:355) > at > org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.init(SparkReduceRecordHandler.java:195) > ... 16 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13687) java.lang.RuntimeException: Reduce operator initialization failed
[ https://issues.apache.org/jira/browse/HIVE-13687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuguanghua reassigned HIVE-13687: -- Assignee: liuguanghua > java.lang.RuntimeException: Reduce operator initialization failed > - > > Key: HIVE-13687 > URL: https://issues.apache.org/jira/browse/HIVE-13687 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0 > Environment: hadoop2.6.4+hive2.0.0+spark1.6.1 > I run sql on the hive on spark model. >Reporter: liuguanghua >Assignee: liuguanghua > > java.lang.RuntimeException: Reduce operator initialization failed > at > org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.init(SparkReduceRecordHandler.java:210) > at > org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunction.call(HiveReduceFunction.java:46) > at > org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunction.call(HiveReduceFunction.java:28) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:192) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:192) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.IndexOutOfBoundsException: Index: 5, Size: 5 > at java.util.ArrayList.rangeCheck(ArrayList.java:635) > at java.util.ArrayList.get(ArrayList.java:411) > at > org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.init(StandardStructObjectInspector.java:118) > at > org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.(StandardStructObjectInspector.java:109) > at > org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:326) > at > org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:311) > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.getJoinOutputObjectInspector(CommonJoinOperator.java:181) > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.initializeOp(CommonJoinOperator.java:319) > at > org.apache.hadoop.hive.ql.exec.JoinOperator.initializeOp(JoinOperator.java:70) > at > org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:355) > at > org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.init(SparkReduceRecordHandler.java:195) > ... 16 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13608) We should provide better error message while constraints with duplicate names are created
[
https://issues.apache.org/jira/browse/HIVE-13608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15270522#comment-15270522
]
Ashutosh Chauhan commented on HIVE-13608:
-
I agree parsing exception messages is hacky, but I didn't follow your MITM
attack description. Can you elaborate on that?
> We should provide better error message while constraints with duplicate names
> are created
> -
>
> Key: HIVE-13608
> URL: https://issues.apache.org/jira/browse/HIVE-13608
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13608.1.patch
>
>
> {code}
> PREHOOK: query: create table t1(x int, constraint pk1 primary key (x) disable
> novalidate)
> PREHOOK: type: CREATETABLE
> PREHOOK: Output: database:default
> PREHOOK: Output: default@t1
> POSTHOOK: query: create table t1(x int, constraint pk1 primary key (x)
> disable novalidate)
> POSTHOOK: type: CREATETABLE
> POSTHOOK: Output: database:default
> POSTHOOK: Output: default@t1
> PREHOOK: query: create table t2(x int, constraint pk1 primary key (x) disable
> novalidate)
> PREHOOK: type: CREATETABLE
> PREHOOK: Output: database:default
> PREHOOK: Output: default@t2
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:For direct
> MetaStore DB connections, we don't support retries at the client level.)
> {code}
> In the above case, it seems like useful error message is lost. It looks like
> a generic problem with metastore server/client exception handling and
> message propagation. Seems like exception parsing logic of
> RetryingMetaStoreClient::invoke() needs to be updated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12866) Allow ReduceSinkDeDuplication to kick in when there are constant keys
[
https://issues.apache.org/jira/browse/HIVE-12866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15270517#comment-15270517
]
Ashutosh Chauhan commented on HIVE-12866:
-
We have entirely disabled RS-dedup in presence of constants, it will be nice if
we can restore this optimization.
> Allow ReduceSinkDeDuplication to kick in when there are constant keys
> --
>
> Key: HIVE-12866
> URL: https://issues.apache.org/jira/browse/HIVE-12866
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>
> Currently in ReduceSinkDeDuplication.checkExprs() we have the following check
> which prevents the optimization from kicking in as soon as we hit constant
> key. We should lift this restriction as much as possible.
> {code}
> private Integer checkExprs(List ckeys, List
> pkeys,
> ReduceSinkOperator cRS, ReduceSinkOperator pRS) throws
> SemanticException {
> // If ckeys or pkeys have constant node expressions avoid the merge.
> for (ExprNodeDesc ck : ckeys) {
> if (ck instanceof ExprNodeConstantDesc) {
> return null;
> }
> }
> for (ExprNodeDesc pk : pkeys) {
> if (pk instanceof ExprNodeConstantDesc) {
> return null;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7166) Vectorization with UDFs returns incorrect results
[
https://issues.apache.org/jira/browse/HIVE-7166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15270514#comment-15270514
]
Ashutosh Chauhan commented on HIVE-7166:
[~mmccline] Since this is a correctness issue can you take a look at see if it
still exists on master ?
> Vectorization with UDFs returns incorrect results
> -
>
> Key: HIVE-7166
> URL: https://issues.apache.org/jira/browse/HIVE-7166
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 0.13.0
> Environment: Hive 0.13 with Hadoop 2.4 on a 3 node cluster
>Reporter: Benjamin Bowman
>Priority: Minor
> Attachments: HIVE-7166.1.patch, HIVE-7166.2.patch
>
>
> Using BETWEEN, a custom UDF, and vectorized query execution yields incorrect
> query results.
> Example Query: SELECT column_1 FROM table_1 WHERE column_1 BETWEEN (UDF_1 -
> X) and UDF_1
> The following test scenario will reproduce the problem:
> TEST UDF (SIMPLE FUNCTION THAT TAKES NO ARGUMENTS AND RETURNS 1):
> package com.test;
> import org.apache.hadoop.hive.ql.exec.Description;
> import org.apache.hadoop.hive.ql.exec.UDF;
> import org.apache.hadoop.io.LongWritable;
> import org.apache.hadoop.io.Text;
> import java.lang.String;
> import java.lang.*;
> public class tenThousand extends UDF {
> private final LongWritable result = new LongWritable();
> public LongWritable evaluate() {
> result.set(1);
> return result;
> }
> }
> TEST DATA (test.input):
> 1|CBCABC|12
> 2|DBCABC|13
> 3|EBCABC|14
> 4|ABCABC|15
> 5|BBCABC|16
> 6|CBCABC|17
> CREATING ORC TABLE:
> 0: jdbc:hive2://server:10002/db> create table testTabOrc (first bigint,
> second varchar(20), third int) partitioned by (range int) clustered by
> (first) sorted by (first) into 8 buckets stored as orc tblproperties
> ("orc.compress" = "SNAPPY", "orc.index" = "true");
> CREATE LOADING TABLE:
> 0: jdbc:hive2://server:10002/db> create table loadingDir (first bigint,
> second varchar(20), third int) partitioned by (range int) row format
> delimited fields terminated by '|' stored as textfile;
> COPY IN DATA:
> [root@server]# hadoop fs -copyFromLocal /tmp/test.input /db/loading/.
> ORC DATA:
> [root@server]# beeline -u jdbc:hive2://server:10002/db -n root --hiveconf
> hive.exec.dynamic.partition.mode=nonstrict --hiveconf
> hive.enforce.sorting=true -e "insert into table testTabOrc partition(range)
> select * from loadingDir;"
> LOAD TEST FUNCTION:
> 0: jdbc:hive2://server:10002/db> add jar /opt/hadoop/lib/testFunction.jar
> 0: jdbc:hive2://server:10002/db> create temporary function ten_thousand as
> 'com.test.tenThousand';
> TURN OFF VECTORIZATION:
> 0: jdbc:hive2://server:10002/db> set hive.vectorized.execution.enabled=false;
> QUERY (RESULTS AS EXPECTED):
> 0: jdbc:hive2://server:10002/db> select first from testTabOrc where first
> between ten_thousand()-1 and ten_thousand()-9995;
> ++
> | first |
> ++
> | 1 |
> | 2 |
> | 3 |
> ++
> 3 rows selected (15.286 seconds)
> TURN ON VECTORIZATION:
> 0: jdbc:hive2://server:10002/db> set hive.vectorized.execution.enabled=true;
> QUERY AGAIN (WRONG RESULTS):
> 0: jdbc:hive2://server:10002/db> select first from testTabOrc where first
> between ten_thousand()-1 and ten_thousand()-9995;
> ++
> | first |
> ++
> ++
> No rows selected (17.763 seconds)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
