[jira] [Commented] (HIVE-12159) Create vectorized readers for the complex types
[ https://issues.apache.org/jira/browse/HIVE-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256477#comment-15256477 ] Owen O'Malley commented on HIVE-12159: -- The TestMiniTezCliDriver-vector_non_string_partition.q-delete_where_non_partitioned.q-auto_sortmerge_join_16.q-and-12-more worked fine on my local box. > Create vectorized readers for the complex types > --- > > Key: HIVE-12159 > URL: https://issues.apache.org/jira/browse/HIVE-12159 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-12159.patch, HIVE-12159.patch, HIVE-12159.patch, > HIVE-12159.patch, HIVE-12159.patch, HIVE-12159.patch, HIVE-12159.patch, > HIVE-12159.patch, HIVE-12159.patch > > > We need vectorized readers for the complex types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13421) Propagate job progress in operation status
[ https://issues.apache.org/jira/browse/HIVE-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256460#comment-15256460 ] Sergio Peña commented on HIVE-13421: Ah, ok. If the patch you were expecting was HIVE-13421.04.patch, then this patch was attached 6 days ago, when Jenkins was not working. It should work now. > Propagate job progress in operation status > -- > > Key: HIVE-13421 > URL: https://issues.apache.org/jira/browse/HIVE-13421 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13421.01.patch, HIVE-13421.02.patch, > HIVE-13421.03.patch, HIVE-13421.04.patch, HIVE-13421.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13533) Remove AST dump
[ https://issues.apache.org/jira/browse/HIVE-13533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13533: --- Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Regenerated q files and pushed to master. Thanks for the review [~ashutoshc]! > Remove AST dump > --- > > Key: HIVE-13533 > URL: https://issues.apache.org/jira/browse/HIVE-13533 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 2.1.0 > > Attachments: HIVE-13533.patch, HIVE-13533.patch > > > For very large queries, dumping the AST can lead to OOM errors. Currently > there are two places where we dump the AST: > - CalcitePlanner if we are running in DEBUG mode (line 300). > - ExplainTask if we use extended explain (line 179). > I guess the original reason to add the dump was to check whether the AST > conversion from CBO was working properly, but I think we are past that stage > now. > We will remove the logic to dump the AST in explain extended. For debug mode > in CalcitePlanner, we will lower the level to LOG.TRACE. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13421) Propagate job progress in operation status
[ https://issues.apache.org/jira/browse/HIVE-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256429#comment-15256429 ] Rajat Khandelwal commented on HIVE-13421: - It wasn't added automatically. I had to manually add it. > Propagate job progress in operation status > -- > > Key: HIVE-13421 > URL: https://issues.apache.org/jira/browse/HIVE-13421 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13421.01.patch, HIVE-13421.02.patch, > HIVE-13421.03.patch, HIVE-13421.04.patch, HIVE-13421.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13421) Propagate job progress in operation status
[ https://issues.apache.org/jira/browse/HIVE-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256416#comment-15256416 ] Sergio Peña commented on HIVE-13421: I see the patch is now on the queue. it might take a while to run your patch as Jenkins is catching up with all untested patches for 1 week. > Propagate job progress in operation status > -- > > Key: HIVE-13421 > URL: https://issues.apache.org/jira/browse/HIVE-13421 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13421.01.patch, HIVE-13421.02.patch, > HIVE-13421.03.patch, HIVE-13421.04.patch, HIVE-13421.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13560) Adding Omid as connection manager for HBase Metastore
[ https://issues.apache.org/jira/browse/HIVE-13560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256384#comment-15256384 ] Hive QA commented on HIVE-13560: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12800344/HIVE-13560.2.patch {color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 358 failed/errored test(s), 9948 tests executed *Failed tests:* {noformat} TestHWISessionManager - did not produce a TEST-*.xml file TestMiniTezCliDriver-explainuser_4.q-update_after_multiple_inserts.q-mapreduce2.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_change_col org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_delete org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_drop_table2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynamic_partition_insert org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join43 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_bitmap_or org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_17 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_auto_sortmerge_join_16 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucketizedhiveinputformat org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucketmapjoin6 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucketmapjoin7 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_semijoin org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_empty_dir_in_table org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_external_table_with_space_in_location_path org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_file_with_header_footer org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_gen_udf_example_add10 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_import_exported_table org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap_auto org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_bucketed_table org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_merge org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_input16_cc org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_leftsemijoin_mr org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_load_fs2 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge3 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge4 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge5 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge6 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge7 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge8 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_incompat1 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_incompat2
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256275#comment-15256275 ] Nemon Lou commented on HIVE-13602: -- The result is right when using hive 1.2.1. After some digging ,finally find HIVE-11104 which causes column missing. But i don't know how it is influenced. [~pxiong] It is easy to reproduce this bug with the latest branch-1 . TPCH tool :https://github.com/hortonworks/hive-testbench > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.3.0, 1.2.2 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > Attachments: calcite_cbo_bad.out, calcite_cbo_good.out, > explain_cbo_bad_part1.out, explain_cbo_bad_part2.out, > explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out, > explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out > > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13412) External table - fields terminated by '\u0044' - 0044 is being interpreted as decimal and not hex
[ https://issues.apache.org/jira/browse/HIVE-13412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256258#comment-15256258 ] Dudu Markovitz commented on HIVE-13412: --- It appears as a general bug (the way '\u' is treated) and not specific for field terminators > select ascii('\u0010'); 10 -- should have been 16 > External table - fields terminated by '\u0044' - 0044 is being interpreted > as decimal and not hex > -- > > Key: HIVE-13412 > URL: https://issues.apache.org/jira/browse/HIVE-13412 > Project: Hive > Issue Type: Bug > Components: Parser >Affects Versions: 0.14.0 >Reporter: Dudu Markovitz >Assignee: Pengcheng Xiong > > , (comma) as the decimal value of '44' and hex value of '2c' > In the following example I'm using '\u0044' as delimiter which is being > interpreted as comma. > hive> create external table test_delimiter_dec_unicode (c1 int,c2 int,c3 int) > row format delimited fields terminated by '\u0044'; > OK > Time taken: 0.035 seconds > hive> show create table test_delimiter_dec_unicode; > OK > CREATE EXTERNAL TABLE `test_delimiter_dec_unicode`( > `c1` int, > `c2` int, > `c3` int) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY ',' > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13391) add an option to LLAP to use keytab to authenticate to read data
[ https://issues.apache.org/jira/browse/HIVE-13391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256223#comment-15256223 ] Siddharth Seth commented on HIVE-13391: --- Comments on the patch. - TaskRunnerCallable has a check on taskUgi being null, at which point it falls back to current behaviour. That'll never happen though since SecurityHelper.createUgiFacatory returns a NoOpFactory if the configs are not setup correctly. - I think we'll need a UGI pool of sorts in this patch itself. Logging in with kerberos credentials for every task will bring down the KDC. - There's some additional changes to MapRecordSource - which appear to more than simple formatting changes. Shouldn't be part of this patch. > add an option to LLAP to use keytab to authenticate to read data > > > Key: HIVE-13391 > URL: https://issues.apache.org/jira/browse/HIVE-13391 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13391.01.patch, HIVE-13391.02.patch, > HIVE-13391.03.patch, HIVE-13391.04.patch, HIVE-13391.05.patch, > HIVE-13391.patch > > > This can be used for non-doAs case to allow access to clients who don't > propagate HDFS tokens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13512) Make initializing dag ids in TezWork thread safe for parallel compilation
[ https://issues.apache.org/jira/browse/HIVE-13512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256215#comment-15256215 ] Hive QA commented on HIVE-13512: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12800330/HIVE-13512.1.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 57 failed/errored test(s), 9949 tests executed *Failed tests:* {noformat} TestHWISessionManager - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler org.apache.hadoop.hive.metastore.hbase.TestHBaseImport.org.apache.hadoop.hive.metastore.hbase.TestHBaseImport org.apache.hadoop.hive.ql.TestTxnCommands2.testInitiatorWithMultipleFailedCompactions org.apache.hadoop.hive.ql.TestTxnCommands2.testOrcNoPPD org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore org.apache.hive.beeline.TestSchemaTool.testSchemaInit org.apache.hive.hcatalog.api.TestHCatClient.testBasicDDLCommands org.apache.hive.hcatalog.api.TestHCatClient.testCreateTableLike org.apache.hive.hcatalog.api.TestHCatClient.testDatabaseLocation org.apache.hive.hcatalog.api.TestHCatClient.testDropPartitionsWithPartialSpec org.apache.hive.hcatalog.api.TestHCatClient.testDropTableException org.apache.hive.hcatalog.api.TestHCatClient.testEmptyTableInstantiation org.apache.hive.hcatalog.api.TestHCatClient.testGetPartitionsWithPartialSpec org.apache.hive.hcatalog.api.TestHCatClient.testObjectNotFoundException org.apache.hive.hcatalog.api.TestHCatClient.testOtherFailure org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema org.apache.hive.hcatalog.api.TestHCatClient.testPartitionsHCatClientImpl org.apache.hive.hcatalog.api.TestHCatClient.testRenameTable org.apache.hive.hcatalog.api.TestHCatClient.testReplicationTaskIter
[jira] [Commented] (HIVE-13445) LLAP: token should encode application and cluster ids
[ https://issues.apache.org/jira/browse/HIVE-13445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256168#comment-15256168 ] Siddharth Seth commented on HIVE-13445: --- bq. Is the yarn option already used somewhere? We could just change the utility method to use it too. Think this should be a separate jira. Will create one. bq. Don't understand. Can you elaborate? A token can be obtained in case of Tez as well, with the hive sessionId passed in, instead of having an alternate path where appId is sent is as null. This would require a lot more work on the LLAP side to associate queries with a sessionId rather than an appId, so it may not be worthwhile right now. bq. Separate JIRA? Think it's worthwhile adding basic tests as part of the patch itself, and a separate jira for more comprehensive system tests. More comments on RB. Thinking on loud on appId in the token... With default and recommended settings post HIVE-13446, only HS2 can obtain delegation tokens or a CLI instance / client which has the hiveserver/llap user kerberos credentials. In this case, users cannot easily fake the appSecret in a token - and llap should be able to trust the appSecret from the token without it being explicitly signed. Also, should we pass in a user in the getDelegationToken request either in place of appSecret or along with it. HS2 can set this user to the actual requesting user, otherwise the token is being issued with the user set to hive. getRealUser does not work afaik without proxy users being setup correctly. On the association of TokenUser / TokenApp on the first request QueryInfo already contains the appIdString and username. The token should be a duplicate of this. If anything we can verify the submitRequest and the token match like you mentioned. Subsequent requests already have the associated username / appId. I don't think the new fields in QueryInfo are required. > LLAP: token should encode application and cluster ids > - > > Key: HIVE-13445 > URL: https://issues.apache.org/jira/browse/HIVE-13445 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13445.01.patch, HIVE-13445.02.patch, > HIVE-13445.03.patch, HIVE-13445.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13447) LLAP: check ZK acls for registry and fail if they are too permissive
[ https://issues.apache.org/jira/browse/HIVE-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256109#comment-15256109 ] Hive QA commented on HIVE-13447: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12800319/HIVE-13447.01.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 41 failed/errored test(s), 9924 tests executed *Failed tests:* {noformat} TestHWISessionManager - did not produce a TEST-*.xml file TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-vectorized_parquet.q-vector_decimal_aggregate.q-tez_self_join.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore org.apache.hive.beeline.TestSchemaTool.testSchemaInit org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropTable org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testConnection org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testIsValid org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testIsValidNeg org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeProxyAuth org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeTokenAuth org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testProxyAuth org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth {noformat} Test results: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/71/testReport Console output: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/71/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-71/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 41 tests failed {noformat} This message is automatically generated.
[jira] [Updated] (HIVE-13421) Propagate job progress in operation status
[ https://issues.apache.org/jira/browse/HIVE-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajat Khandelwal updated HIVE-13421: Status: Patch Available (was: In Progress) > Propagate job progress in operation status > -- > > Key: HIVE-13421 > URL: https://issues.apache.org/jira/browse/HIVE-13421 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13421.01.patch, HIVE-13421.02.patch, > HIVE-13421.03.patch, HIVE-13421.04.patch, HIVE-13421.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13421) Propagate job progress in operation status
[ https://issues.apache.org/jira/browse/HIVE-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255991#comment-15255991 ] Rajat Khandelwal commented on HIVE-13421: - Taking patch from reviewboard and attaching > Propagate job progress in operation status > -- > > Key: HIVE-13421 > URL: https://issues.apache.org/jira/browse/HIVE-13421 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13421.01.patch, HIVE-13421.02.patch, > HIVE-13421.03.patch, HIVE-13421.04.patch, HIVE-13421.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13421) Propagate job progress in operation status
[ https://issues.apache.org/jira/browse/HIVE-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajat Khandelwal updated HIVE-13421: Attachment: HIVE-13421.05.patch > Propagate job progress in operation status > -- > > Key: HIVE-13421 > URL: https://issues.apache.org/jira/browse/HIVE-13421 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13421.01.patch, HIVE-13421.02.patch, > HIVE-13421.03.patch, HIVE-13421.04.patch, HIVE-13421.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-13421) Propagate job progress in operation status
[ https://issues.apache.org/jira/browse/HIVE-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-13421 started by Rajat Khandelwal. --- > Propagate job progress in operation status > -- > > Key: HIVE-13421 > URL: https://issues.apache.org/jira/browse/HIVE-13421 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13421.01.patch, HIVE-13421.02.patch, > HIVE-13421.03.patch, HIVE-13421.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13421) Propagate job progress in operation status
[ https://issues.apache.org/jira/browse/HIVE-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajat Khandelwal updated HIVE-13421: Status: Open (was: Patch Available) Jenkins still hasn't picked. [~spena] maybe there are some other issues too that have this problem. Meanwhile, I'll try cancelling and re-submitting patch. > Propagate job progress in operation status > -- > > Key: HIVE-13421 > URL: https://issues.apache.org/jira/browse/HIVE-13421 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13421.01.patch, HIVE-13421.02.patch, > HIVE-13421.03.patch, HIVE-13421.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255976#comment-15255976 ] Nemon Lou commented on HIVE-13602: -- I just copy it from the result of "explain rewrite q16". {noformat} explain rewrite select p_brand, p_type, p_size, count(distinct ps_suppkey) as supplier_cnt from partsupp, part where p_partkey = ps_partkey and p_brand <> 'Brand#34' and p_type not like 'ECONOMY BRUSHED%' and p_size in (22, 14, 27, 49, 21, 33, 35, 28) and partsupp.ps_suppkey not in ( select s_suppkey from supplier where s_comment like '%Customer%Complaints%' ) group by p_brand, p_type, p_size order by supplier_cnt desc, p_brand, p_type, p_size; {noformat} > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > Attachments: calcite_cbo_bad.out, calcite_cbo_good.out, > explain_cbo_bad_part1.out, explain_cbo_bad_part2.out, > explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out, > explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out > > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13536) LLAP: Add metrics for task scheduler
[ https://issues.apache.org/jira/browse/HIVE-13536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255974#comment-15255974 ] Prasanth Jayachandran commented on HIVE-13536: -- [~sseth] Could you please take another look? > LLAP: Add metrics for task scheduler > > > Key: HIVE-13536 > URL: https://issues.apache.org/jira/browse/HIVE-13536 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13536.1.patch, HIVE-13536.2.patch > > > Currently there are no metrics for task scheduler. It will be useful to > provide one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13097) [Umbrella] Changes dependent on Tez 0.8.3
[ https://issues.apache.org/jira/browse/HIVE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255973#comment-15255973 ] Prasanth Jayachandran commented on HIVE-13097: -- LGMT, +1 > [Umbrella] Changes dependent on Tez 0.8.3 > - > > Key: HIVE-13097 > URL: https://issues.apache.org/jira/browse/HIVE-13097 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth > Attachments: HIVE-13097.01.patch, HIVE-13097.02.patch, > HIVE-13097.03.patch, HIVE-13097.versonChange.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255964#comment-15255964 ] Nemon Lou commented on HIVE-13602: -- >From calcite rel plan ,we can see the p_brand column is declared to fetch. And the value for p_brand column is zero in the final result. {noformat} 0: jdbc:hive2://189.39.151.141:21066/> select * from q16_cbo_bad limit 1; ++---+---+-+--+ | q16_cbo_bad.p_brand | q16_cbo_bad.p_type | q16_cbo_bad.p_size | q16_cbo_bad.supplier_cnt | ++---+---+-+--+ | 0 | MEDIUM BRUSHED TIN| 21| 298 | ++---+---+-+--+ 1 row selected (0.139 seconds) 0: jdbc:hive2://189.39.151.141:21066/> select * from q16_cbo_rewrite limit 1; ++---+---+-+--+ | q16_cbo_rewrite.p_brand | q16_cbo_rewrite.p_type | q16_cbo_rewrite.p_size | q16_cbo_rewrite.supplier_cnt | ++---+---+-+--+ | Brand#31 | PROMO ANODIZED NICKEL | 14 | 40 | ++---+---+-+--+ 1 row selected (0.961 seconds) {noformat} > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > Attachments: calcite_cbo_bad.out, calcite_cbo_good.out, > explain_cbo_bad_part1.out, explain_cbo_bad_part2.out, > explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out, > explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out > > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255953#comment-15255953 ] Pengcheng Xiong commented on HIVE-13602: what is {code} 1 = 1 {code} in the where condition? > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > Attachments: calcite_cbo_bad.out, calcite_cbo_good.out, > explain_cbo_bad_part1.out, explain_cbo_bad_part2.out, > explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out, > explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out > > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255950#comment-15255950 ] Pengcheng Xiong commented on HIVE-13602: [~nemon], if p_brand is missing, i would expect to see that one column is missing, rather than different number of rows? > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > Attachments: calcite_cbo_bad.out, calcite_cbo_good.out, > explain_cbo_bad_part1.out, explain_cbo_bad_part2.out, > explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out, > explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out > > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255937#comment-15255937 ] Nemon Lou commented on HIVE-13602: -- The join order seems to be the same for both queries.The difference is that the bad plan only select three columns for table part at stage-1 (p_brand is missing),the good one select four. > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > Attachments: calcite_cbo_bad.out, calcite_cbo_good.out, > explain_cbo_bad_part1.out, explain_cbo_bad_part2.out, > explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out, > explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out > > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[ https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255934#comment-15255934 ] Hive QA commented on HIVE-13563: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12800332/HIVE-13563.1.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 9992 tests executed *Failed tests:* {noformat} TestHWISessionManager - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit org.apache.hive.beeline.TestSchemaTool.testSchemaInit org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler {noformat} Test results: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/70/testReport Console output: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/70/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-70/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12800332 - PreCommit-HIVE-MASTER-Build > Hive Streaming does not honor orc.compress.size and orc.stripe.size table > properties > > > Key: HIVE-13563 > URL: https://issues.apache.org/jira/browse/HIVE-13563 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC2.1 > Attachments: HIVE-13563.1.patch > > > According to the doc: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax > One should be able to specify tblproperties for many ORC options. > But the settings for orc.compress.size and orc.stripe.size don't take effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nemon Lou updated HIVE-13602: - Attachment: explain_cbo_good(rewrite)_part3.out explain_cbo_good(rewrite)_part2.out explain_cbo_good(rewrite)_part1.out calcite_cbo_good.out calcite_cbo_bad.out > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > Attachments: calcite_cbo_bad.out, calcite_cbo_good.out, > explain_cbo_bad_part1.out, explain_cbo_bad_part2.out, > explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out, > explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out > > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)