[jira] [Commented] (HIVE-13447) LLAP: check ZK acls for registry and fail if they are too permissive
[
https://issues.apache.org/jira/browse/HIVE-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15253534#comment-15253534
]
Hive QA commented on HIVE-13447:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800103/HIVE-13447.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 9995 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/34/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/34/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-34/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12800103 - PreCommit-HIVE-MASTER-Build
> LLAP: check ZK acls for registry and fail if they are too permissive
>
>
> Key: HIVE-13447
> URL: https://issues.apache.org/jira/browse/HIVE-13447
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13447.patch
>
>
> Only the current ("hive") user can have write access.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13535) UPDATE ... SET only working when properties are set globally.
[
https://issues.apache.org/jira/browse/HIVE-13535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin Joseph updated HIVE-13535:
Priority: Major (was: Minor)
> UPDATE ... SET only working when properties are set globally.
> -
>
> Key: HIVE-13535
> URL: https://issues.apache.org/jira/browse/HIVE-13535
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Environment: Operating System : SUSE Linux Enterprise Server 11
> (x86_64)
> Architecture : x86_64
> CPU op-mode(s) : 32-bit, 64-bit
> Byte Order : Little Endian
> Hadoop version : Hadoop 2.7.1.2.3.2.0-2950
> Hive version : Hive 1.2.1.2.3.2.0-2950
>Reporter: Bibin Joseph
>
> h3.Making a hive table transactional
> *Steps followed*
> 1. Entered hive shell.
> 2. Choose database.
> 3. Set hive transaction properties.
> {quote}
> SET hive.support.concurrency=true;
> SET hive.enforce.bucketing=true;
> SET hive.exec.dynamic.partition.mode=nonstrict;
> SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> SET hive.compactor.initiator.on=true;
> SET hive.compactor.worker.threads=1;
> SET hive.support.concurrency=true;
> SET hive.enforce.bucketing=true;
> {quote}
> 4. Created bucketed table with 'transactional'='true'
> {quote}
> create table test(id int, name string) clustered by (id) into 2 buckets
> stored as orc TBLPROPERTIES ('transactional'='true');
> {quote}
> 5. Inserted values.
> {quote}
> insert into table test values(1,'Name');
> {quote}
> 6. Fired update query.
> {quote}
> update test set name='New_Name' where id=1;
> {quote}
> Created following error ;
> {quote}
> FAILED: SemanticException [Error 10294]: Attempt to do update or delete using
> transaction manager that does not support these operations.
> {quote}
> *It works without error when the transaction properties are set, before
> choosing database.*
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12049) Provide an option to write serialized thrift objects in final tasks
[
https://issues.apache.org/jira/browse/HIVE-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15253676#comment-15253676
]
Hive QA commented on HIVE-12049:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800138/HIVE-12049.26.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 9998 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.concurrencyFalse
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testLockTimeout
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testUpdate
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/36/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/36/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-36/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12800138 - PreCommit-HIVE-MASTER-Build
> Provide an option to write serialized thrift objects in final tasks
> ---
>
> Key: HIVE-12049
> URL: https://issues.apache.org/jira/browse/HIVE-12049
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, JDBC
>Affects Versions: 2.0.0
>Reporter: Rohit Dholakia
>Assignee: Rohit Dholakia
> Attachments: HIVE-12049.1.patch, HIVE-12049.11.patch,
> HIVE-12049.12.patch, HIVE-12049.13.patch, HIVE-12049.14.patch,
> HIVE-12049.15.patch, HIVE-12049.16.patch, HIVE-12049.17.patch,
> HIVE-12049.18.patch, HIVE-12049.19.patch, HIVE-12049.2.patch,
> HIVE-12049.25.patch, HIVE-12049.26.patch, HIVE-12049.3.patch,
> HIVE-12049.4.patch, HIVE-12049.5.patch, HIVE-12049.6.patch,
> HIVE-12049.7.patch, HIVE-12049.9.patch, new-driver-profiles.png,
> old-driver-profiles.png
>
>
> For each fetch request to HiveServer2, we pay the penalty of deserializing
> the row objects and translating them into a different representation suitable
> for the RPC transfer. In a moderate to high concurrency scenarios, this can
> result in significant CPU and memory wastage. By having each task write the
> appropriate thrift objects to the output files, HiveServer2 can simply stream
> a batch of rows on the wire without incurring any of the additional cost of
> deserialization and translation.
> This can be implemented by writing a new SerDe, which the FileSinkOperator
> can use to write thrift formatted row batches to the output file. Using the
> pluggable property of the {{hive.query.result.fileformat}}, we can set it to
> use SequenceFile and write a batch of thrift formatted rows as a value blob.
> The FetchTask can now simply read the blob and send it over the wire. On the
> client side, the *DBC driver can read the blob and since it is already
> formatted in the way it expects, it can continue building the ResultSet the
> way it does in the current implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13467) Show llap info on hs2 ui when available
[
https://issues.apache.org/jira/browse/HIVE-13467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15253782#comment-15253782
]
Hive QA commented on HIVE-13467:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800136/HIVE-13467.6.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 9995 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.concurrencyFalse
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDelete
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testLockTimeout
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleReadPartition
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleWriteTable
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testUpdate
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/37/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/37/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-37/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12800136 - PreCommit-HIVE-MASTER-Build
> Show llap info on hs2 ui when available
> ---
>
> Key: HIVE-13467
> URL: https://issues.apache.org/jira/browse/HIVE-13467
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Attachments: HIVE-13467.1.patch, HIVE-13467.2.patch,
> HIVE-13467.3.patch, HIVE-13467.4.patch, HIVE-13467.5.patch,
> HIVE-13467.6.patch, screen-shot-llap.png, screen.png
>
>
> When llap is on and hs2 is configured with access to an llap cluster, HS2 UI
> should show some status of the daemons and provide a mechanism to click
> through to their respective UIs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13570) Some query with Union all fails when CBO is off
[
https://issues.apache.org/jira/browse/HIVE-13570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15253916#comment-15253916
]
Chaoyu Tang commented on HIVE-13570:
[~ychena] I wonder if you can add more tests with combination of
hive.ppd.remove.duplicatefilters, hive.cbo.enable, and hive.optimize.ppd if
possible? Did the issue only happen when cbo is off? How about cbo is enabled
but hive.ppd.remove.duplicatefilters disabled? Will hive.optimize.ppd off work
around the issue?
> Some query with Union all fails when CBO is off
> ---
>
> Key: HIVE-13570
> URL: https://issues.apache.org/jira/browse/HIVE-13570
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13570.1.PATCH
>
>
> Some queries with union all throws IndexOutOfBoundsException
> when:
> set hive.cbo.enable=false;
> set hive.ppd.remove.duplicatefilters=true;
> The stack is as:
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 67, Size: 67
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcCtx.genColLists(ColumnPrunerProcCtx.java:161)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcCtx.handleFilterUnionChildren(ColumnPrunerProcCtx.java:273)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactory$ColumnPrunerFilterProc.process(ColumnPrunerProcFactory.java:108)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPruner$ColumnPrunerWalker.walk(ColumnPruner.java:172)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPruner.transform(ColumnPruner.java:135)
>
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:198)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10327)
>
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:192)
>
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)
>
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:432)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1119)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1167)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1055)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1045)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:305)
> at
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:403)
> at
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:419)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:708)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13585) Add counter metric for direct sql failures
[ https://issues.apache.org/jira/browse/HIVE-13585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15253940#comment-15253940 ] Mohit Sabharwal commented on HIVE-13585: [~sershe], could you please take a look. Thanks. > Add counter metric for direct sql failures > -- > > Key: HIVE-13585 > URL: https://issues.apache.org/jira/browse/HIVE-13585 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-13585.patch > > > In case of direct sql failure, metastore query falls back to DataNucleus. > It'd be good to record how often this happens as a metrics counter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13194) Hive object is not thread safe, is shared via a threadlocal and thus should not be shared via session - part 2
[ https://issues.apache.org/jira/browse/HIVE-13194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15253996#comment-15253996 ] Aihua Xu commented on HIVE-13194: - Yeah. I also noticed that metaCallTimeMap is not thread safe from one test case. We should make it synchronized. I also thought of a pool of Hive objects, but the reason that we can't use that probably is, different connections may have different hiveconf and then require different connections. > Hive object is not thread safe, is shared via a threadlocal and thus should > not be shared via session - part 2 > -- > > Key: HIVE-13194 > URL: https://issues.apache.org/jira/browse/HIVE-13194 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13194.patch > > > Session has session Hive object stored in a field. Given that Hive object is > not thread safe and is also taken from a threadlocal to start with, that is > not a good idea. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13424) Refactoring the code to pass a QueryState object rather than HiveConf object
[ https://issues.apache.org/jira/browse/HIVE-13424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254003#comment-15254003 ] Aihua Xu commented on HIVE-13424: - [~sershe] Can you take another look at the code? The tests above are not related. > Refactoring the code to pass a QueryState object rather than HiveConf object > > > Key: HIVE-13424 > URL: https://issues.apache.org/jira/browse/HIVE-13424 > Project: Hive > Issue Type: Sub-task > Components: Query Processor >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13424.1.patch, HIVE-13424.2.patch, > HIVE-13424.3.patch, HIVE-13424.4.patch, HIVE-13424.5.patch > > > Step1: to refractor the code by creating the QueryState class and moving > query related info from SessionState. Then during compilation, execution > stages, pass single QueryState object for each query. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13527) Using deprecated APIs in HBase client causes zookeeper connection leaks.
[
https://issues.apache.org/jira/browse/HIVE-13527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254013#comment-15254013
]
Hive QA commented on HIVE-13527:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800158/HIVE-13527.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 9932 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-update_orig_table.q-union2.q-bucket4.q-and-12-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.hcatalog.api.TestHCatClient.testBasicDDLCommands
org.apache.hive.hcatalog.api.TestHCatClient.testDatabaseLocation
org.apache.hive.hcatalog.api.TestHCatClient.testDropPartitionsWithPartialSpec
org.apache.hive.hcatalog.api.TestHCatClient.testDropTableException
org.apache.hive.hcatalog.api.TestHCatClient.testGetPartitionsWithPartialSpec
org.apache.hive.hcatalog.api.TestHCatClient.testObjectNotFoundException
org.apache.hive.hcatalog.api.TestHCatClient.testRenameTable
org.apache.hive.hcatalog.api.TestHCatClient.testReplicationTaskIter
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure
org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropTable
org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.org.apache.hive.minikdc.TestJdbcWithDBTokenStore
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/38/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/38/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-38/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12800158 - PreCommit-HIVE-MASTER-Build
> Using deprecated APIs in HBase client causes zookeeper connection leaks.
>
>
> Key: HIVE-13527
> URL: https://issues.apache.org/jira/browse/HIVE-13527
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-13527.2.patch, HIVE-13527.2.patch,
> HIVE-13527.patch, HIVE-13527.patch
>
>
> When running queries against hbase-backed hive tables, the following log
> messages are seen in the HS2 log.
> {code}
> 2016-04-11 07:25:23,657 WARN
> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase: You are using an
> HTable instance that relies on an HBase-managed Connection. This is usually
> due to directly creating an HTable, which is deprecated. Instead, you should
> create a Connection object and then request a Table instance from it. If you
> don't need the Table instance for your own use, you should instead use the
> TableInputFormatBase.initalizeTable method directly.
> 2016-04-11 07:25:23,658 INFO
> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase: Creating an
> additional unmanaged connection because user provided one can't be used for
> administrative actions. We'll close it when we close out the table.
> {code}
> In a HS2 log file, there are 1366 zookeeper connections established but only
>
[jira] [Commented] (HIVE-13585) Add counter metric for direct sql failures
[ https://issues.apache.org/jira/browse/HIVE-13585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254054#comment-15254054 ] Aihua Xu commented on HIVE-13585: - The enhancement looks straightforward. +1. > Add counter metric for direct sql failures > -- > > Key: HIVE-13585 > URL: https://issues.apache.org/jira/browse/HIVE-13585 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-13585.patch > > > In case of direct sql failure, metastore query falls back to DataNucleus. > It'd be good to record how often this happens as a metrics counter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13570) Some query with Union all fails when CBO is off
[
https://issues.apache.org/jira/browse/HIVE-13570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-13570:
Attachment: HIVE-13570.2.patch
[~ctang.ma], The bug can only be reproduced when cbo is false, duplicatefilters
is true.
Attach patch2 with added tests for all the combination values of related
properties.
> Some query with Union all fails when CBO is off
> ---
>
> Key: HIVE-13570
> URL: https://issues.apache.org/jira/browse/HIVE-13570
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13570.1.PATCH, HIVE-13570.2.patch
>
>
> Some queries with union all throws IndexOutOfBoundsException
> when:
> set hive.cbo.enable=false;
> set hive.ppd.remove.duplicatefilters=true;
> The stack is as:
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 67, Size: 67
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcCtx.genColLists(ColumnPrunerProcCtx.java:161)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcCtx.handleFilterUnionChildren(ColumnPrunerProcCtx.java:273)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactory$ColumnPrunerFilterProc.process(ColumnPrunerProcFactory.java:108)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPruner$ColumnPrunerWalker.walk(ColumnPruner.java:172)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPruner.transform(ColumnPruner.java:135)
>
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:198)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10327)
>
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:192)
>
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)
>
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:432)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1119)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1167)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1055)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1045)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:305)
> at
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:403)
> at
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:419)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:708)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12079) Add units tests for HiveServer2 LDAP filters added in HIVE-7193
[ https://issues.apache.org/jira/browse/HIVE-12079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-12079: - Status: Patch Available (was: Open) > Add units tests for HiveServer2 LDAP filters added in HIVE-7193 > --- > > Key: HIVE-12079 > URL: https://issues.apache.org/jira/browse/HIVE-12079 > Project: Hive > Issue Type: Test > Components: HiveServer2 >Affects Versions: 1.1.1 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-12079.patch > > > HIVE-11866 adds a test framework that uses an in-memory ldap server for unit > tests. Need to add unit tests for user and group filtering feature added in > HIVE-7193. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12079) Add units tests for HiveServer2 LDAP filters added in HIVE-7193
[ https://issues.apache.org/jira/browse/HIVE-12079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-12079: - Attachment: HIVE-12079.patch The patch contains 1) New tests for features in HIVE-7193: a) Group filtering functionality b) Custom query functionality HIVE-13295: Use of custom configurable ldap attributes for ldap searches. 2) Refactoring of the current tests to use User objects instead of strings for better readability and maintainability. > Add units tests for HiveServer2 LDAP filters added in HIVE-7193 > --- > > Key: HIVE-12079 > URL: https://issues.apache.org/jira/browse/HIVE-12079 > Project: Hive > Issue Type: Test > Components: HiveServer2 >Affects Versions: 1.1.1 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-12079.patch > > > HIVE-11866 adds a test framework that uses an in-memory ldap server for unit > tests. Need to add unit tests for user and group filtering feature added in > HIVE-7193. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13588) NPE is thrown from MapredLocalTask.executeInChildVM
[
https://issues.apache.org/jira/browse/HIVE-13588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254151#comment-15254151
]
Hive QA commented on HIVE-13588:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800164/HIVE-13588.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 74 failed/errored test(s), 9945 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestHiveMetaStoreWithEnvironmentContext.testEnvironmentContext
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreA
[jira] [Commented] (HIVE-13390) HiveServer2: Add more test to ZK service discovery using MiniHS2
[
https://issues.apache.org/jira/browse/HIVE-13390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254152#comment-15254152
]
Hive QA commented on HIVE-13390:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800173/HIVE-13390.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/40/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/40/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-40/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-40/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at aac9263 HIVE-13488: Restore dag summary when tez exec print
summary enabled and in-place updates disabled (Prasanth Jayachandran reviewed
by Sergey Shelukhin)
+ git clean -f -d
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at aac9263 HIVE-13488: Restore dag summary when tez exec print
summary enabled and in-place updates disabled (Prasanth Jayachandran reviewed
by Sergey Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12800173 - PreCommit-HIVE-MASTER-Build
> HiveServer2: Add more test to ZK service discovery using MiniHS2
>
>
> Key: HIVE-13390
> URL: https://issues.apache.org/jira/browse/HIVE-13390
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, JDBC
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-13390.1.patch, HIVE-13390.1.patch,
> HIVE-13390.2.patch, keystore.jks, keystore_exampledotcom.jks, truststore.jks
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12079) Add units tests for HiveServer2 LDAP filters added in HIVE-7193
[ https://issues.apache.org/jira/browse/HIVE-12079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254161#comment-15254161 ] Aihua Xu commented on HIVE-12079: - The code looks good to me. +1. > Add units tests for HiveServer2 LDAP filters added in HIVE-7193 > --- > > Key: HIVE-12079 > URL: https://issues.apache.org/jira/browse/HIVE-12079 > Project: Hive > Issue Type: Test > Components: HiveServer2 >Affects Versions: 1.1.1 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-12079.patch > > > HIVE-11866 adds a test framework that uses an in-memory ldap server for unit > tests. Need to add unit tests for user and group filtering feature added in > HIVE-7193. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12159) Create vectorized readers for the complex types
[ https://issues.apache.org/jira/browse/HIVE-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-12159: - Attachment: HIVE-12159.patch Reattaching to resubmit to jenkins. > Create vectorized readers for the complex types > --- > > Key: HIVE-12159 > URL: https://issues.apache.org/jira/browse/HIVE-12159 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-12159.patch, HIVE-12159.patch, HIVE-12159.patch, > HIVE-12159.patch, HIVE-12159.patch, HIVE-12159.patch, HIVE-12159.patch, > HIVE-12159.patch > > > We need vectorized readers for the complex types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11550) ACID queries pollute HiveConf
[ https://issues.apache.org/jira/browse/HIVE-11550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254184#comment-15254184 ] Eugene Koifman commented on HIVE-11550: --- Test failures have age > 1 [~alangates] could you review please > ACID queries pollute HiveConf > - > > Key: HIVE-11550 > URL: https://issues.apache.org/jira/browse/HIVE-11550 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-11550.patch > > > HiveConf is a SessionState level object. Some ACID related logic makes > changes to it (which are meant to be per query) but become per SessionState. > See SemanticAnalyzer.checkAcidConstraints() > Also note HiveConf.setVar(conf, > HiveConf.ConfVars.DYNAMICPARTITIONINGMODE, "nonstrict"); > in UpdateDeleteSemancitAnalzyer > [~alangates], do you know of other cases or ideas on how to deal with this > differently? > _SortedDynPartitionOptimizer.process()_ is the place to have the logic to do > _conf.setBoolVar(ConfVars.HIVEOPTSORTDYNAMICPARTITION, false);_ on per query > basis -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13527) Using deprecated APIs in HBase client causes zookeeper connection leaks.
[
https://issues.apache.org/jira/browse/HIVE-13527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254187#comment-15254187
]
Naveen Gangam commented on HIVE-13527:
--
The test failures do not appear to be related to the patch. I have run most of
these test locally and they all pass. I have no failures locally. So +1 for me
on the patch.
{code}
Test set: org.apache.hive.hcatalog.api.TestHCatClient
---
Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 88.006 sec -
in org.apache.hive.hcatalog.api.TestHCatClient
{code}
> Using deprecated APIs in HBase client causes zookeeper connection leaks.
>
>
> Key: HIVE-13527
> URL: https://issues.apache.org/jira/browse/HIVE-13527
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-13527.2.patch, HIVE-13527.2.patch,
> HIVE-13527.patch, HIVE-13527.patch
>
>
> When running queries against hbase-backed hive tables, the following log
> messages are seen in the HS2 log.
> {code}
> 2016-04-11 07:25:23,657 WARN
> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase: You are using an
> HTable instance that relies on an HBase-managed Connection. This is usually
> due to directly creating an HTable, which is deprecated. Instead, you should
> create a Connection object and then request a Table instance from it. If you
> don't need the Table instance for your own use, you should instead use the
> TableInputFormatBase.initalizeTable method directly.
> 2016-04-11 07:25:23,658 INFO
> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase: Creating an
> additional unmanaged connection because user provided one can't be used for
> administrative actions. We'll close it when we close out the table.
> {code}
> In a HS2 log file, there are 1366 zookeeper connections established but only
> a small fraction of them were closed. So lsof would show 1300+ open TCP
> connections to Zookeeper.
> grep "org.apache.zookeeper.ClientCnxn: Session establishment complete on
> server" * |wc -l
> 1366
> grep "INFO org.apache.zookeeper.ZooKeeper: Session:" * |grep closed |wc -l
> 54
> According to the comments in TableInputFormatBase, the recommended means for
> subclasses like HiveHBaseTableInputFormat is to call initializeTable()
> instead of setHTable() that it currently uses.
> "
> Subclasses MUST ensure initializeTable(Connection, TableName) is called for
> an instance to function properly. Each of the entry points to this class used
> by the MapReduce framework, {@link #createRecordReader(InputSplit,
> TaskAttemptContext)} and {@link #getSplits(JobContext)}, will call {@link
> #initialize(JobContext)} as a convenient centralized location to handle
> retrieving the necessary configuration information. If your subclass
> overrides either of these methods, either call the parent version or call
> initialize yourself.
> "
> Currently setHTable() also creates an additional Admin connection, even
> though it is not needed.
> So the use of deprecated APIs are to be replaced.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13493) Fix TransactionBatchImpl.getCurrentTxnId() and mis logging fixes
[ https://issues.apache.org/jira/browse/HIVE-13493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13493: -- Status: Patch Available (was: Open) > Fix TransactionBatchImpl.getCurrentTxnId() and mis logging fixes > > > Key: HIVE-13493 > URL: https://issues.apache.org/jira/browse/HIVE-13493 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.2.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-13493.2.patch, HIVE-13493.patch > > > sort list of transaction IDs deleted by performTimeouts > sort list of "empty aborted" > log the list of lock id removed due to timeout > fix TransactionBatchImpl.getCurrentTxnId() not to look past end of array (see > HIVE-13489) > beginNextTransactionImpl() > if ( currentTxnIndex >= txnIds.size() )//todo: this condition is bogus should > check currentTxnIndex + 1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13493) Fix TransactionBatchImpl.getCurrentTxnId() and mis logging fixes
[ https://issues.apache.org/jira/browse/HIVE-13493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13493: -- Status: Open (was: Patch Available) > Fix TransactionBatchImpl.getCurrentTxnId() and mis logging fixes > > > Key: HIVE-13493 > URL: https://issues.apache.org/jira/browse/HIVE-13493 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.2.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-13493.2.patch, HIVE-13493.patch > > > sort list of transaction IDs deleted by performTimeouts > sort list of "empty aborted" > log the list of lock id removed due to timeout > fix TransactionBatchImpl.getCurrentTxnId() not to look past end of array (see > HIVE-13489) > beginNextTransactionImpl() > if ( currentTxnIndex >= txnIds.size() )//todo: this condition is bogus should > check currentTxnIndex + 1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13588) NPE is thrown from MapredLocalTask.executeInChildVM
[
https://issues.apache.org/jira/browse/HIVE-13588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang updated HIVE-13588:
---
Attachment: HIVE-13588.patch
It looks the failures are not related to patch, resubmit it for testing.
> NPE is thrown from MapredLocalTask.executeInChildVM
> ---
>
> Key: HIVE-13588
> URL: https://issues.apache.org/jira/browse/HIVE-13588
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-13588.patch, HIVE-13588.patch
>
>
> NPE was thrown out from MapredLocalTask.executeInChildVM in running some
> queries with CLI, see error below:
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInChildVM(MapredLocalTask.java:321)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.execute(MapredLocalTask.java:148)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:172)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1868)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1595)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1346)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1117)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1105)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:236)
> [hive-cli-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
> [hive-cli-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> [hive-cli-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:782)
> [hive-cli-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:721)
> [hive-cli-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:648)
> [hive-cli-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.7.0_45]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> ~[?:1.7.0_45]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.7.0_45]
> {code}
> It is because the operationLog is only applicable to HS2 but CLI, therefore
> it might not be set (null)
> It is related to HIVE-13183
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13493) Fix TransactionBatchImpl.getCurrentTxnId() and mis logging fixes
[ https://issues.apache.org/jira/browse/HIVE-13493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13493: -- Attachment: HIVE-13493.2.patch > Fix TransactionBatchImpl.getCurrentTxnId() and mis logging fixes > > > Key: HIVE-13493 > URL: https://issues.apache.org/jira/browse/HIVE-13493 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.2.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-13493.2.patch, HIVE-13493.patch > > > sort list of transaction IDs deleted by performTimeouts > sort list of "empty aborted" > log the list of lock id removed due to timeout > fix TransactionBatchImpl.getCurrentTxnId() not to look past end of array (see > HIVE-13489) > beginNextTransactionImpl() > if ( currentTxnIndex >= txnIds.size() )//todo: this condition is bogus should > check currentTxnIndex + 1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13541) Pass view's ColumnAccessInfo to HiveAuthorizer
[ https://issues.apache.org/jira/browse/HIVE-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13541: --- Status: Open (was: Patch Available) > Pass view's ColumnAccessInfo to HiveAuthorizer > -- > > Key: HIVE-13541 > URL: https://issues.apache.org/jira/browse/HIVE-13541 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-13541.01.patch > > > RIght now, only table's ColumnAccessInfo is passed to HiveAuthorizer -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13553) CTE with upperCase alias throws exception
[ https://issues.apache.org/jira/browse/HIVE-13553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13553: --- Status: Open (was: Patch Available) > CTE with upperCase alias throws exception > - > > Key: HIVE-13553 > URL: https://issues.apache.org/jira/browse/HIVE-13553 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong >Priority: Minor > Attachments: HIVE-13553.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13541) Pass view's ColumnAccessInfo to HiveAuthorizer
[ https://issues.apache.org/jira/browse/HIVE-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13541: --- Status: Patch Available (was: Open) > Pass view's ColumnAccessInfo to HiveAuthorizer > -- > > Key: HIVE-13541 > URL: https://issues.apache.org/jira/browse/HIVE-13541 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-13541.01.patch > > > RIght now, only table's ColumnAccessInfo is passed to HiveAuthorizer -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13341) Stats state is not captured correctly: differentiate load table and create table
[ https://issues.apache.org/jira/browse/HIVE-13341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13341: --- Status: Open (was: Patch Available) > Stats state is not captured correctly: differentiate load table and create > table > > > Key: HIVE-13341 > URL: https://issues.apache.org/jira/browse/HIVE-13341 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer, Statistics >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13341.01.patch, HIVE-13341.02.patch, > HIVE-13341.03.patch, HIVE-13341.04.patch, HIVE-13341.05.patch, > HIVE-13341.06.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13341) Stats state is not captured correctly: differentiate load table and create table
[ https://issues.apache.org/jira/browse/HIVE-13341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13341: --- Status: Patch Available (was: Open) > Stats state is not captured correctly: differentiate load table and create > table > > > Key: HIVE-13341 > URL: https://issues.apache.org/jira/browse/HIVE-13341 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer, Statistics >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13341.01.patch, HIVE-13341.02.patch, > HIVE-13341.03.patch, HIVE-13341.04.patch, HIVE-13341.05.patch, > HIVE-13341.06.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13553) CTE with upperCase alias throws exception
[ https://issues.apache.org/jira/browse/HIVE-13553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13553: --- Status: Patch Available (was: Open) > CTE with upperCase alias throws exception > - > > Key: HIVE-13553 > URL: https://issues.apache.org/jira/browse/HIVE-13553 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong >Priority: Minor > Attachments: HIVE-13553.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13570) Some query with Union all fails when CBO is off
[
https://issues.apache.org/jira/browse/HIVE-13570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254194#comment-15254194
]
Chaoyu Tang commented on HIVE-13570:
Patch looks good to me, +1 pending the testing. It is even better if the test
case could be simplified, but I understand it is hard to reproduce the issue
with some simple queries.
> Some query with Union all fails when CBO is off
> ---
>
> Key: HIVE-13570
> URL: https://issues.apache.org/jira/browse/HIVE-13570
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13570.1.PATCH, HIVE-13570.2.patch
>
>
> Some queries with union all throws IndexOutOfBoundsException
> when:
> set hive.cbo.enable=false;
> set hive.ppd.remove.duplicatefilters=true;
> The stack is as:
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 67, Size: 67
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcCtx.genColLists(ColumnPrunerProcCtx.java:161)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcCtx.handleFilterUnionChildren(ColumnPrunerProcCtx.java:273)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactory$ColumnPrunerFilterProc.process(ColumnPrunerProcFactory.java:108)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPruner$ColumnPrunerWalker.walk(ColumnPruner.java:172)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPruner.transform(ColumnPruner.java:135)
>
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:198)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10327)
>
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:192)
>
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)
>
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:432)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1119)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1167)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1055)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1045)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:305)
> at
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:403)
> at
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:419)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:708)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13541) Pass view's ColumnAccessInfo to HiveAuthorizer
[ https://issues.apache.org/jira/browse/HIVE-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13541: --- Status: Patch Available (was: Open) > Pass view's ColumnAccessInfo to HiveAuthorizer > -- > > Key: HIVE-13541 > URL: https://issues.apache.org/jira/browse/HIVE-13541 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-13541.01.patch > > > RIght now, only table's ColumnAccessInfo is passed to HiveAuthorizer -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13541) Pass view's ColumnAccessInfo to HiveAuthorizer
[ https://issues.apache.org/jira/browse/HIVE-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13541: --- Attachment: HIVE-13541.01.patch > Pass view's ColumnAccessInfo to HiveAuthorizer > -- > > Key: HIVE-13541 > URL: https://issues.apache.org/jira/browse/HIVE-13541 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-13541.01.patch > > > RIght now, only table's ColumnAccessInfo is passed to HiveAuthorizer -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13541) Pass view's ColumnAccessInfo to HiveAuthorizer
[ https://issues.apache.org/jira/browse/HIVE-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13541: --- Attachment: (was: HIVE-13541.01.patch) > Pass view's ColumnAccessInfo to HiveAuthorizer > -- > > Key: HIVE-13541 > URL: https://issues.apache.org/jira/browse/HIVE-13541 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-13541.01.patch > > > RIght now, only table's ColumnAccessInfo is passed to HiveAuthorizer -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13541) Pass view's ColumnAccessInfo to HiveAuthorizer
[ https://issues.apache.org/jira/browse/HIVE-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13541: --- Status: Open (was: Patch Available) > Pass view's ColumnAccessInfo to HiveAuthorizer > -- > > Key: HIVE-13541 > URL: https://issues.apache.org/jira/browse/HIVE-13541 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-13541.01.patch > > > RIght now, only table's ColumnAccessInfo is passed to HiveAuthorizer -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13527) Using deprecated APIs in HBase client causes zookeeper connection leaks.
[
https://issues.apache.org/jira/browse/HIVE-13527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254199#comment-15254199
]
Chaoyu Tang commented on HIVE-13527:
+1
> Using deprecated APIs in HBase client causes zookeeper connection leaks.
>
>
> Key: HIVE-13527
> URL: https://issues.apache.org/jira/browse/HIVE-13527
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-13527.2.patch, HIVE-13527.2.patch,
> HIVE-13527.patch, HIVE-13527.patch
>
>
> When running queries against hbase-backed hive tables, the following log
> messages are seen in the HS2 log.
> {code}
> 2016-04-11 07:25:23,657 WARN
> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase: You are using an
> HTable instance that relies on an HBase-managed Connection. This is usually
> due to directly creating an HTable, which is deprecated. Instead, you should
> create a Connection object and then request a Table instance from it. If you
> don't need the Table instance for your own use, you should instead use the
> TableInputFormatBase.initalizeTable method directly.
> 2016-04-11 07:25:23,658 INFO
> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase: Creating an
> additional unmanaged connection because user provided one can't be used for
> administrative actions. We'll close it when we close out the table.
> {code}
> In a HS2 log file, there are 1366 zookeeper connections established but only
> a small fraction of them were closed. So lsof would show 1300+ open TCP
> connections to Zookeeper.
> grep "org.apache.zookeeper.ClientCnxn: Session establishment complete on
> server" * |wc -l
> 1366
> grep "INFO org.apache.zookeeper.ZooKeeper: Session:" * |grep closed |wc -l
> 54
> According to the comments in TableInputFormatBase, the recommended means for
> subclasses like HiveHBaseTableInputFormat is to call initializeTable()
> instead of setHTable() that it currently uses.
> "
> Subclasses MUST ensure initializeTable(Connection, TableName) is called for
> an instance to function properly. Each of the entry points to this class used
> by the MapReduce framework, {@link #createRecordReader(InputSplit,
> TaskAttemptContext)} and {@link #getSplits(JobContext)}, will call {@link
> #initialize(JobContext)} as a convenient centralized location to handle
> retrieving the necessary configuration information. If your subclass
> overrides either of these methods, either call the parent version or call
> initialize yourself.
> "
> Currently setHTable() also creates an additional Admin connection, even
> though it is not needed.
> So the use of deprecated APIs are to be replaced.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13553) CTE with upperCase alias throws exception
[ https://issues.apache.org/jira/browse/HIVE-13553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13553: --- Status: Patch Available (was: Open) > CTE with upperCase alias throws exception > - > > Key: HIVE-13553 > URL: https://issues.apache.org/jira/browse/HIVE-13553 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong >Priority: Minor > Attachments: HIVE-13553.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13553) CTE with upperCase alias throws exception
[ https://issues.apache.org/jira/browse/HIVE-13553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13553: --- Attachment: HIVE-13553.01.patch > CTE with upperCase alias throws exception > - > > Key: HIVE-13553 > URL: https://issues.apache.org/jira/browse/HIVE-13553 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong >Priority: Minor > Attachments: HIVE-13553.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13553) CTE with upperCase alias throws exception
[ https://issues.apache.org/jira/browse/HIVE-13553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13553: --- Attachment: (was: HIVE-13553.01.patch) > CTE with upperCase alias throws exception > - > > Key: HIVE-13553 > URL: https://issues.apache.org/jira/browse/HIVE-13553 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong >Priority: Minor > Attachments: HIVE-13553.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13553) CTE with upperCase alias throws exception
[ https://issues.apache.org/jira/browse/HIVE-13553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13553: --- Status: Open (was: Patch Available) > CTE with upperCase alias throws exception > - > > Key: HIVE-13553 > URL: https://issues.apache.org/jira/browse/HIVE-13553 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong >Priority: Minor > Attachments: HIVE-13553.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13341) Stats state is not captured correctly: differentiate load table and create table
[ https://issues.apache.org/jira/browse/HIVE-13341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13341: --- Attachment: (was: HIVE-13341.06.patch) > Stats state is not captured correctly: differentiate load table and create > table > > > Key: HIVE-13341 > URL: https://issues.apache.org/jira/browse/HIVE-13341 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer, Statistics >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13341.01.patch, HIVE-13341.02.patch, > HIVE-13341.03.patch, HIVE-13341.04.patch, HIVE-13341.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13341) Stats state is not captured correctly: differentiate load table and create table
[ https://issues.apache.org/jira/browse/HIVE-13341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13341: --- Status: Open (was: Patch Available) > Stats state is not captured correctly: differentiate load table and create > table > > > Key: HIVE-13341 > URL: https://issues.apache.org/jira/browse/HIVE-13341 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer, Statistics >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13341.01.patch, HIVE-13341.02.patch, > HIVE-13341.03.patch, HIVE-13341.04.patch, HIVE-13341.05.patch, > HIVE-13341.06.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13341) Stats state is not captured correctly: differentiate load table and create table
[ https://issues.apache.org/jira/browse/HIVE-13341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13341: --- Attachment: HIVE-13341.06.patch > Stats state is not captured correctly: differentiate load table and create > table > > > Key: HIVE-13341 > URL: https://issues.apache.org/jira/browse/HIVE-13341 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer, Statistics >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13341.01.patch, HIVE-13341.02.patch, > HIVE-13341.03.patch, HIVE-13341.04.patch, HIVE-13341.05.patch, > HIVE-13341.06.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13341) Stats state is not captured correctly: differentiate load table and create table
[ https://issues.apache.org/jira/browse/HIVE-13341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13341: --- Status: Patch Available (was: Open) > Stats state is not captured correctly: differentiate load table and create > table > > > Key: HIVE-13341 > URL: https://issues.apache.org/jira/browse/HIVE-13341 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer, Statistics >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13341.01.patch, HIVE-13341.02.patch, > HIVE-13341.03.patch, HIVE-13341.04.patch, HIVE-13341.05.patch, > HIVE-13341.06.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13559) Pass exception to failure hooks
[ https://issues.apache.org/jira/browse/HIVE-13559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254216#comment-15254216 ] Chaoyu Tang commented on HIVE-13559: LGTM, +1 > Pass exception to failure hooks > --- > > Key: HIVE-13559 > URL: https://issues.apache.org/jira/browse/HIVE-13559 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-13559.1.patch > > > Pass exception to failure hooks so that they know more about the failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13591) TestSchemaTool is failing on master
[ https://issues.apache.org/jira/browse/HIVE-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254247#comment-15254247 ] Ashutosh Chauhan commented on HIVE-13591: - [~thejas] Would you like to take a look? > TestSchemaTool is failing on master > --- > > Key: HIVE-13591 > URL: https://issues.apache.org/jira/browse/HIVE-13591 > Project: Hive > Issue Type: Test > Components: Metastore >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan > > Not sure at what point this started to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-13533) Remove AST dump
[ https://issues.apache.org/jira/browse/HIVE-13533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-13533 started by Jesus Camacho Rodriguez. -- > Remove AST dump > --- > > Key: HIVE-13533 > URL: https://issues.apache.org/jira/browse/HIVE-13533 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13533.patch, HIVE-13533.patch > > > For very large queries, dumping the AST can lead to OOM errors. Currently > there are two places where we dump the AST: > - CalcitePlanner if we are running in DEBUG mode (line 300). > - ExplainTask if we use extended explain (line 179). > I guess the original reason to add the dump was to check whether the AST > conversion from CBO was working properly, but I think we are past that stage > now. > We will remove the logic to dump the AST in explain extended. For debug mode > in CalcitePlanner, we will lower the level to LOG.TRACE. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13533) Remove AST dump
[ https://issues.apache.org/jira/browse/HIVE-13533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13533: --- Status: Open (was: Patch Available) > Remove AST dump > --- > > Key: HIVE-13533 > URL: https://issues.apache.org/jira/browse/HIVE-13533 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13533.patch, HIVE-13533.patch > > > For very large queries, dumping the AST can lead to OOM errors. Currently > there are two places where we dump the AST: > - CalcitePlanner if we are running in DEBUG mode (line 300). > - ExplainTask if we use extended explain (line 179). > I guess the original reason to add the dump was to check whether the AST > conversion from CBO was working properly, but I think we are past that stage > now. > We will remove the logic to dump the AST in explain extended. For debug mode > in CalcitePlanner, we will lower the level to LOG.TRACE. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13533) Remove AST dump
[ https://issues.apache.org/jira/browse/HIVE-13533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13533: --- Status: Patch Available (was: In Progress) > Remove AST dump > --- > > Key: HIVE-13533 > URL: https://issues.apache.org/jira/browse/HIVE-13533 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13533.patch, HIVE-13533.patch > > > For very large queries, dumping the AST can lead to OOM errors. Currently > there are two places where we dump the AST: > - CalcitePlanner if we are running in DEBUG mode (line 300). > - ExplainTask if we use extended explain (line 179). > I guess the original reason to add the dump was to check whether the AST > conversion from CBO was working properly, but I think we are past that stage > now. > We will remove the logic to dump the AST in explain extended. For debug mode > in CalcitePlanner, we will lower the level to LOG.TRACE. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[
https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254281#comment-15254281
]
Ashutosh Chauhan commented on HIVE-13068:
-
Any reason you are introducing {{SimpleConstantReduction}} here? Goal is to
reduce optimizations on Hive operator, not to increase them : )
> Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
> ---
>
> Key: HIVE-13068
> URL: https://issues.apache.org/jira/browse/HIVE-13068
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Logical Optimizer
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch,
> HIVE-13068.patch
>
>
> After HIVE-12543 went in, we need follow-up work to disable the last call to
> ConstantPropagate in Hive. This probably implies work on extending the
> constant folding logic in Calcite.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13494) LLAP: Some metrics from daemon are not exposed to hadoop-metrics2
[ https://issues.apache.org/jira/browse/HIVE-13494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13494: - Attachment: HIVE-13494.1.patch > LLAP: Some metrics from daemon are not exposed to hadoop-metrics2 > - > > Key: HIVE-13494 > URL: https://issues.apache.org/jira/browse/HIVE-13494 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.0.1 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13494.1.patch, HIVE-13494.1.patch > > > LlapDaemonInfo is exposed via JMX but not sent to hadoop metrics. > Async IO metrics also seems incorrect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13068: --- Status: Open (was: Patch Available) > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.01.patch, HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13533) Remove AST dump
[ https://issues.apache.org/jira/browse/HIVE-13533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13533: --- Attachment: HIVE-13533.patch > Remove AST dump > --- > > Key: HIVE-13533 > URL: https://issues.apache.org/jira/browse/HIVE-13533 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13533.patch, HIVE-13533.patch > > > For very large queries, dumping the AST can lead to OOM errors. Currently > there are two places where we dump the AST: > - CalcitePlanner if we are running in DEBUG mode (line 300). > - ExplainTask if we use extended explain (line 179). > I guess the original reason to add the dump was to check whether the AST > conversion from CBO was working properly, but I think we are past that stage > now. > We will remove the logic to dump the AST in explain extended. For debug mode > in CalcitePlanner, we will lower the level to LOG.TRACE. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13494) LLAP: Some metrics from daemon are not exposed to hadoop-metrics2
[ https://issues.apache.org/jira/browse/HIVE-13494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13494: - Status: Patch Available (was: Open) > LLAP: Some metrics from daemon are not exposed to hadoop-metrics2 > - > > Key: HIVE-13494 > URL: https://issues.apache.org/jira/browse/HIVE-13494 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.0.1 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13494.1.patch > > > LlapDaemonInfo is exposed via JMX but not sent to hadoop metrics. > Async IO metrics also seems incorrect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13494) LLAP: Some metrics from daemon are not exposed to hadoop-metrics2
[ https://issues.apache.org/jira/browse/HIVE-13494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13494: - Status: Open (was: Patch Available) > LLAP: Some metrics from daemon are not exposed to hadoop-metrics2 > - > > Key: HIVE-13494 > URL: https://issues.apache.org/jira/browse/HIVE-13494 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.0.1 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13494.1.patch > > > LlapDaemonInfo is exposed via JMX but not sent to hadoop metrics. > Async IO metrics also seems incorrect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13068: --- Status: Patch Available (was: In Progress) > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.01.patch, HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-13068 started by Jesus Camacho Rodriguez. -- > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.01.patch, HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13068: --- Attachment: HIVE-13068.01.patch > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch, > HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13258) LLAP: Add hdfs bytes read and spilled bytes to tez print summary
[ https://issues.apache.org/jira/browse/HIVE-13258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13258: - Status: Patch Available (was: Open) > LLAP: Add hdfs bytes read and spilled bytes to tez print summary > > > Key: HIVE-13258 > URL: https://issues.apache.org/jira/browse/HIVE-13258 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13258.1.patch, llap-fs-counters-full-cache-hit.png, > llap-fs-counters.png > > > When printing counters to console it will be useful to print hdfs bytes read > and spilled bytes which will help with debugging issues faster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13258) LLAP: Add hdfs bytes read and spilled bytes to tez print summary
[ https://issues.apache.org/jira/browse/HIVE-13258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13258: - Status: Open (was: Patch Available) > LLAP: Add hdfs bytes read and spilled bytes to tez print summary > > > Key: HIVE-13258 > URL: https://issues.apache.org/jira/browse/HIVE-13258 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13258.1.patch, HIVE-13258.1.patch, > llap-fs-counters-full-cache-hit.png, llap-fs-counters.png > > > When printing counters to console it will be useful to print hdfs bytes read > and spilled bytes which will help with debugging issues faster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13258) LLAP: Add hdfs bytes read and spilled bytes to tez print summary
[ https://issues.apache.org/jira/browse/HIVE-13258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13258: - Status: Patch Available (was: Open) > LLAP: Add hdfs bytes read and spilled bytes to tez print summary > > > Key: HIVE-13258 > URL: https://issues.apache.org/jira/browse/HIVE-13258 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13258.1.patch, HIVE-13258.1.patch, > llap-fs-counters-full-cache-hit.png, llap-fs-counters.png > > > When printing counters to console it will be useful to print hdfs bytes read > and spilled bytes which will help with debugging issues faster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13258) LLAP: Add hdfs bytes read and spilled bytes to tez print summary
[ https://issues.apache.org/jira/browse/HIVE-13258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13258: - Attachment: HIVE-13258.1.patch > LLAP: Add hdfs bytes read and spilled bytes to tez print summary > > > Key: HIVE-13258 > URL: https://issues.apache.org/jira/browse/HIVE-13258 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13258.1.patch, HIVE-13258.1.patch, > llap-fs-counters-full-cache-hit.png, llap-fs-counters.png > > > When printing counters to console it will be useful to print hdfs bytes read > and spilled bytes which will help with debugging issues faster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13586) Allow UDFs/UDTFs to indicate whether they can run with fetch task conversion
[ https://issues.apache.org/jira/browse/HIVE-13586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-13586: -- Status: Patch Available (was: Open) > Allow UDFs/UDTFs to indicate whether they can run with fetch task conversion > > > Key: HIVE-13586 > URL: https://issues.apache.org/jira/browse/HIVE-13586 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13586.1.patch > > > Some UDFs/UDTFs cannot run properly with fetch task conversion. > For UDFs: > - The UDF may be depending on files/archives added to the distributed cache, > which may not be available in the local fetch task. > - The UDF may be depending on GenericUDF.configure() to be called - this is > not called for locally run UDFs. > For UDTFs, one issue is HIVE-11892. > A bit of a workaround, but for such UDFs/UDTFs we can have a flag to indicate > whether the UDF should be allowed to run in a local fetch task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13586) Allow UDFs/UDTFs to indicate whether they can run with fetch task conversion
[ https://issues.apache.org/jira/browse/HIVE-13586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-13586: -- Attachment: HIVE-13586.1.patch Patch with test (udf_nofetchtask.q). Also update the udtf qfiles that were changed with HIVE-11892 since fetch task optimization is once again enabled for UDTFs. > Allow UDFs/UDTFs to indicate whether they can run with fetch task conversion > > > Key: HIVE-13586 > URL: https://issues.apache.org/jira/browse/HIVE-13586 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13586.1.patch > > > Some UDFs/UDTFs cannot run properly with fetch task conversion. > For UDFs: > - The UDF may be depending on files/archives added to the distributed cache, > which may not be available in the local fetch task. > - The UDF may be depending on GenericUDF.configure() to be called - this is > not called for locally run UDFs. > For UDTFs, one issue is HIVE-11892. > A bit of a workaround, but for such UDFs/UDTFs we can have a flag to indicate > whether the UDF should be allowed to run in a local fetch task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[
https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254299#comment-15254299
]
Jesus Camacho Rodriguez commented on HIVE-13068:
SimpleConstantReduction will only get exercised if CBO has optimized the plan.
For instance, while return path is not on, we might need to use CAST on top of
a constant when we go back through the AST e.g. UDFToDouble(100), and we do not
want to miss that opportunity to fold. Further, we do not fold some expressions
in Calcite yet e.g. power function, which was causing some test regressions.
Concretely, SimpleConstantReduction will only fold a deterministic function
expression with constant parameters; otherwise, it does not even try. Thus, the
overhead should be low.
In addition, if condition for a given e.g. semijoin is just the {{true}}
constant, a filter with condition {{true}} is introduced on its left input by
SemanticAnalyzer. I tried to fix this in SemanticAnalyzer, but the logic to
infer the join conditions should change, and it is not straightforward. Thus, I
took advantage of the fact that we are checking the Filter operators in
SimpleConstantReduction, and we remove the filter if the condition is {{true}}.
> Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
> ---
>
> Key: HIVE-13068
> URL: https://issues.apache.org/jira/browse/HIVE-13068
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Logical Optimizer
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch,
> HIVE-13068.patch
>
>
> After HIVE-12543 went in, we need follow-up work to disable the last call to
> ConstantPropagate in Hive. This probably implies work on extending the
> constant folding logic in Calcite.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13258) LLAP: Add hdfs bytes read and spilled bytes to tez print summary
[ https://issues.apache.org/jira/browse/HIVE-13258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13258: - Status: Open (was: Patch Available) > LLAP: Add hdfs bytes read and spilled bytes to tez print summary > > > Key: HIVE-13258 > URL: https://issues.apache.org/jira/browse/HIVE-13258 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13258.1.patch, llap-fs-counters-full-cache-hit.png, > llap-fs-counters.png > > > When printing counters to console it will be useful to print hdfs bytes read > and spilled bytes which will help with debugging issues faster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13408) Issue appending HIVE_QUERY_ID without checking if the prefix already exists
[
https://issues.apache.org/jira/browse/HIVE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254305#comment-15254305
]
Sergey Shelukhin commented on HIVE-13408:
-
Should this be rebased and committed?
> Issue appending HIVE_QUERY_ID without checking if the prefix already exists
> ---
>
> Key: HIVE-13408
> URL: https://issues.apache.org/jira/browse/HIVE-13408
> Project: Hive
> Issue Type: Bug
> Components: Shims
>Affects Versions: 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13408.1.patch, HIVE-13408.2.patch
>
>
> {code}
> We are resetting the hadoop caller context to HIVE_QUERY_ID:HIVE_QUERY_ID:
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13463) Fix ImportSemanticAnalyzer to allow for different src/dst filesystems
[ https://issues.apache.org/jira/browse/HIVE-13463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13463: Attachment: HIVE-13463-3.patch Again, Jenkins was down > Fix ImportSemanticAnalyzer to allow for different src/dst filesystems > - > > Key: HIVE-13463 > URL: https://issues.apache.org/jira/browse/HIVE-13463 > Project: Hive > Issue Type: Bug > Components: Import/Export >Affects Versions: 2.0.0 >Reporter: Zach York >Assignee: Zach York > Attachments: HIVE-13463-1.patch, HIVE-13463-2.patch, > HIVE-13463-3.patch, HIVE-13463.patch > > > In ImportSemanticAnalyzer, there is an assumption that the src filesystem for > import and the final location are on the same filesystem. Therefore the check > for emptiness and getExternalTmpLocation will be looking on the wrong > filesystem and will cause an error. The output path should be fed into > getExternalTmpLocation to get a temporary file on the correct filesystem. The > check for emptiness should use the output filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13097) [Umbrella] Changes dependent on Tez 0.8.3
[ https://issues.apache.org/jira/browse/HIVE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13097: Attachment: HIVE-13097.03.patch Attaching again > [Umbrella] Changes dependent on Tez 0.8.3 > - > > Key: HIVE-13097 > URL: https://issues.apache.org/jira/browse/HIVE-13097 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth > Attachments: HIVE-13097.01.patch, HIVE-13097.02.patch, > HIVE-13097.03.patch, HIVE-13097.versonChange.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[
https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254309#comment-15254309
]
Jesus Camacho Rodriguez commented on HIVE-13068:
Btw, we already needed to do something similar with PPD: we needed to create
SimplePredicatePushDown (HIVE-11865).
I will add comments to {{SimpleConstantReduction}} to clarify its coverage: I
just realized I did not do that.
> Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
> ---
>
> Key: HIVE-13068
> URL: https://issues.apache.org/jira/browse/HIVE-13068
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Logical Optimizer
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch,
> HIVE-13068.patch
>
>
> After HIVE-12543 went in, we need follow-up work to disable the last call to
> ConstantPropagate in Hive. This probably implies work on extending the
> constant folding logic in Calcite.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13494) LLAP: Some metrics from daemon are not exposed to hadoop-metrics2
[ https://issues.apache.org/jira/browse/HIVE-13494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13494: Attachment: HIVE-13494.2.patch reattaching for HiveQA > LLAP: Some metrics from daemon are not exposed to hadoop-metrics2 > - > > Key: HIVE-13494 > URL: https://issues.apache.org/jira/browse/HIVE-13494 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.0.1 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13494.1.patch, HIVE-13494.1.patch, > HIVE-13494.2.patch > > > LlapDaemonInfo is exposed via JMX but not sent to hadoop metrics. > Async IO metrics also seems incorrect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13408) Issue appending HIVE_QUERY_ID without checking if the prefix already exists
[
https://issues.apache.org/jira/browse/HIVE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254315#comment-15254315
]
Vikram Dixit K commented on HIVE-13408:
---
[~sershe] Unfortunately not. This one depends on updating the hadoop version
used in hive and committing of other patches (HIVE-12254) before we can commit
this to master.
> Issue appending HIVE_QUERY_ID without checking if the prefix already exists
> ---
>
> Key: HIVE-13408
> URL: https://issues.apache.org/jira/browse/HIVE-13408
> Project: Hive
> Issue Type: Bug
> Components: Shims
>Affects Versions: 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13408.1.patch, HIVE-13408.2.patch
>
>
> {code}
> We are resetting the hadoop caller context to HIVE_QUERY_ID:HIVE_QUERY_ID:
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12887) Handle ORC schema on read with fewer columns than file schema (after Schema Evolution changes)
[
https://issues.apache.org/jira/browse/HIVE-12887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254324#comment-15254324
]
Sergey Shelukhin commented on HIVE-12887:
-
[~mmccline] ping?
> Handle ORC schema on read with fewer columns than file schema (after Schema
> Evolution changes)
> --
>
> Key: HIVE-12887
> URL: https://issues.apache.org/jira/browse/HIVE-12887
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-12887.01.patch, HIVE-12887.02.patch
>
>
> Exception caused by reading after column removal.
> {code}
> Caused by: java.lang.IndexOutOfBoundsException: Index: 10, Size: 10
> at java.util.ArrayList.rangeCheck(ArrayList.java:653)
> at java.util.ArrayList.get(ArrayList.java:429)
> at java.util.Collections$UnmodifiableList.get(Collections.java:1309)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcProto$Type.getSubtypes(OrcProto.java:12240)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:2053)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2481)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:216)
> at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:598)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:179)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.(OrcRawRecordMerger.java:222)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:442)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1285)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1165)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:249)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13408) Issue appending HIVE_QUERY_ID without checking if the prefix already exists
[
https://issues.apache.org/jira/browse/HIVE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254326#comment-15254326
]
Sergey Shelukhin commented on HIVE-13408:
-
Should there be a separate fix for 2.0.1 then? If this causes the infinite log
lines, we should just remove the addition for 2.0.1.
> Issue appending HIVE_QUERY_ID without checking if the prefix already exists
> ---
>
> Key: HIVE-13408
> URL: https://issues.apache.org/jira/browse/HIVE-13408
> Project: Hive
> Issue Type: Bug
> Components: Shims
>Affects Versions: 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13408.1.patch, HIVE-13408.2.patch
>
>
> {code}
> We are resetting the hadoop caller context to HIVE_QUERY_ID:HIVE_QUERY_ID:
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13194) Hive object is not thread safe, is shared via a threadlocal and thus should not be shared via session - part 2
[ https://issues.apache.org/jira/browse/HIVE-13194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13194: Resolution: Fixed Fix Version/s: 2.0.1 2.1.0 1.3.0 Status: Resolved (was: Patch Available) Committed to all branches > Hive object is not thread safe, is shared via a threadlocal and thus should > not be shared via session - part 2 > -- > > Key: HIVE-13194 > URL: https://issues.apache.org/jira/browse/HIVE-13194 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 1.3.0, 2.1.0, 2.0.1 > > Attachments: HIVE-13194.patch > > > Session has session Hive object stored in a field. Given that Hive object is > not thread safe and is also taken from a threadlocal to start with, that is > not a good idea. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13194) Hive object is not thread safe, is shared via a threadlocal and thus should not be shared via session - part 2
[ https://issues.apache.org/jira/browse/HIVE-13194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254365#comment-15254365 ] Sergey Shelukhin commented on HIVE-13194: - Er, no I didn't > Hive object is not thread safe, is shared via a threadlocal and thus should > not be shared via session - part 2 > -- > > Key: HIVE-13194 > URL: https://issues.apache.org/jira/browse/HIVE-13194 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13194.patch > > > Session has session Hive object stored in a field. Given that Hive object is > not thread safe and is also taken from a threadlocal to start with, that is > not a good idea. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13194) Hive object is not thread safe, is shared via a threadlocal and thus should not be shared via session - part 2
[ https://issues.apache.org/jira/browse/HIVE-13194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13194: Fix Version/s: (was: 2.0.1) (was: 2.1.0) (was: 1.3.0) > Hive object is not thread safe, is shared via a threadlocal and thus should > not be shared via session - part 2 > -- > > Key: HIVE-13194 > URL: https://issues.apache.org/jira/browse/HIVE-13194 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13194.patch > > > Session has session Hive object stored in a field. Given that Hive object is > not thread safe and is also taken from a threadlocal to start with, that is > not a good idea. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HIVE-13194) Hive object is not thread safe, is shared via a threadlocal and thus should not be shared via session - part 2
[ https://issues.apache.org/jira/browse/HIVE-13194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reopened HIVE-13194: - > Hive object is not thread safe, is shared via a threadlocal and thus should > not be shared via session - part 2 > -- > > Key: HIVE-13194 > URL: https://issues.apache.org/jira/browse/HIVE-13194 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13194.patch > > > Session has session Hive object stored in a field. Given that Hive object is > not thread safe and is also taken from a threadlocal to start with, that is > not a good idea. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-13194) Hive object is not thread safe, is shared via a threadlocal and thus should not be shared via session - part 2
[ https://issues.apache.org/jira/browse/HIVE-13194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin resolved HIVE-13194. - Resolution: Not A Problem I meant to resolve something else as fixed. This is won't fix. I'll create a separate JIRA for making the map threadsafe > Hive object is not thread safe, is shared via a threadlocal and thus should > not be shared via session - part 2 > -- > > Key: HIVE-13194 > URL: https://issues.apache.org/jira/browse/HIVE-13194 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13194.patch > > > Session has session Hive object stored in a field. Given that Hive object is > not thread safe and is also taken from a threadlocal to start with, that is > not a good idea. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13240) GroupByOperator: Drop the hash aggregates when closing operator
[ https://issues.apache.org/jira/browse/HIVE-13240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13240: Resolution: Fixed Fix Version/s: 2.0.1 2.1.0 1.3.0 Status: Resolved (was: Patch Available) Committed to all branches. Thanks for the patch! > GroupByOperator: Drop the hash aggregates when closing operator > --- > > Key: HIVE-13240 > URL: https://issues.apache.org/jira/browse/HIVE-13240 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.3.0, 1.2.1, 2.0.0 >Reporter: Gopal V >Assignee: Gopal V > Fix For: 1.3.0, 2.1.0, 2.0.1 > > Attachments: HIVE-13240.03.patch, HIVE-13240.1.patch, > HIVE-13240.2.patch > > > GroupByOperator holds onto the Hash aggregates accumulated when the plan is > cached. > Drop the hashAggregates in case of error during forwarding to the next > operator. > Added for PTF, TopN and all GroupBy cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13592) metastore calls map is not thread safe
[ https://issues.apache.org/jira/browse/HIVE-13592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13592: Attachment: HIVE-13592.patch This is one of the early patches from HIVE-13002. [~ashutoshc] [~aihuaxu] [~thejas] fyi. We assume now that the rest of the Hive object is thread-safe (see the discussion in HIVE-13194) > metastore calls map is not thread safe > -- > > Key: HIVE-13592 > URL: https://issues.apache.org/jira/browse/HIVE-13592 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13592.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13592) metastore calls map is not thread safe
[ https://issues.apache.org/jira/browse/HIVE-13592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13592: Status: Patch Available (was: Open) > metastore calls map is not thread safe > -- > > Key: HIVE-13592 > URL: https://issues.apache.org/jira/browse/HIVE-13592 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13592.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13463) Fix ImportSemanticAnalyzer to allow for different src/dst filesystems
[ https://issues.apache.org/jira/browse/HIVE-13463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254402#comment-15254402 ] Zach York commented on HIVE-13463: -- Thanks Sergey! I had attached the -1 and -2 patches before Jenkins was down. Thanks for reattaching the patch! > Fix ImportSemanticAnalyzer to allow for different src/dst filesystems > - > > Key: HIVE-13463 > URL: https://issues.apache.org/jira/browse/HIVE-13463 > Project: Hive > Issue Type: Bug > Components: Import/Export >Affects Versions: 2.0.0 >Reporter: Zach York >Assignee: Zach York > Attachments: HIVE-13463-1.patch, HIVE-13463-2.patch, > HIVE-13463-3.patch, HIVE-13463.patch > > > In ImportSemanticAnalyzer, there is an assumption that the src filesystem for > import and the final location are on the same filesystem. Therefore the check > for emptiness and getExternalTmpLocation will be looking on the wrong > filesystem and will cause an error. The output path should be fed into > getExternalTmpLocation to get a temporary file on the correct filesystem. The > check for emptiness should use the output filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13446) LLAP: set default management protocol acls to deny all
[ https://issues.apache.org/jira/browse/HIVE-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254404#comment-15254404 ] Sergey Shelukhin commented on HIVE-13446: - [~vikram.dixit] perhaps you can review? > LLAP: set default management protocol acls to deny all > -- > > Key: HIVE-13446 > URL: https://issues.apache.org/jira/browse/HIVE-13446 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13446.patch > > > The user needs to set the acls. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9660) store end offset of compressed data for RG in RowIndex in ORC
[ https://issues.apache.org/jira/browse/HIVE-9660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-9660: --- Attachment: HIVE-9660.10.patch Rebased the patch > store end offset of compressed data for RG in RowIndex in ORC > - > > Key: HIVE-9660 > URL: https://issues.apache.org/jira/browse/HIVE-9660 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-9660.01.patch, HIVE-9660.02.patch, > HIVE-9660.03.patch, HIVE-9660.04.patch, HIVE-9660.05.patch, > HIVE-9660.06.patch, HIVE-9660.07.patch, HIVE-9660.07.patch, > HIVE-9660.08.patch, HIVE-9660.09.patch, HIVE-9660.10.patch, HIVE-9660.patch, > HIVE-9660.patch > > > Right now the end offset is estimated, which in some cases results in tons of > extra data being read. > We can add a separate array to RowIndex (positions_v2?) that stores number of > compressed buffers for each RG, or end offset, or something, to remove this > estimation magic -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13494) LLAP: Some metrics from daemon are not exposed to hadoop-metrics2
[ https://issues.apache.org/jira/browse/HIVE-13494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13494: - Attachment: HIVE-13494.2.patch Better handling of invalid intervals based on [~sseth]'s review comments in HIVE-13536 > LLAP: Some metrics from daemon are not exposed to hadoop-metrics2 > - > > Key: HIVE-13494 > URL: https://issues.apache.org/jira/browse/HIVE-13494 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.0.1 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13494.1.patch, HIVE-13494.1.patch, > HIVE-13494.2.patch, HIVE-13494.2.patch > > > LlapDaemonInfo is exposed via JMX but not sent to hadoop metrics. > Async IO metrics also seems incorrect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13570) Some query with Union all fails when CBO is off
[
https://issues.apache.org/jira/browse/HIVE-13570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254435#comment-15254435
]
Hive QA commented on HIVE-13570:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800229/HIVE-13570.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 71 failed/errored test(s), 9939 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_bucket_map_join_tez2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_3
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_5
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_3
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_4
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_5
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llap_nullscan
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llapdecider
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dml
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_hash
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_tests
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_joins_explain
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_smb_main
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRemoteUGIHiveMetaStoreIpAddress.testIpAddress
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.lockConflictDbTable
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
o
[jira] [Commented] (HIVE-12049) Provide an option to write serialized thrift objects in final tasks
[
https://issues.apache.org/jira/browse/HIVE-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254434#comment-15254434
]
Vaibhav Gumashta commented on HIVE-12049:
-
Test failures look unrelated - I'll commit shortly.
> Provide an option to write serialized thrift objects in final tasks
> ---
>
> Key: HIVE-12049
> URL: https://issues.apache.org/jira/browse/HIVE-12049
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, JDBC
>Affects Versions: 2.0.0
>Reporter: Rohit Dholakia
>Assignee: Rohit Dholakia
> Attachments: HIVE-12049.1.patch, HIVE-12049.11.patch,
> HIVE-12049.12.patch, HIVE-12049.13.patch, HIVE-12049.14.patch,
> HIVE-12049.15.patch, HIVE-12049.16.patch, HIVE-12049.17.patch,
> HIVE-12049.18.patch, HIVE-12049.19.patch, HIVE-12049.2.patch,
> HIVE-12049.25.patch, HIVE-12049.26.patch, HIVE-12049.3.patch,
> HIVE-12049.4.patch, HIVE-12049.5.patch, HIVE-12049.6.patch,
> HIVE-12049.7.patch, HIVE-12049.9.patch, new-driver-profiles.png,
> old-driver-profiles.png
>
>
> For each fetch request to HiveServer2, we pay the penalty of deserializing
> the row objects and translating them into a different representation suitable
> for the RPC transfer. In a moderate to high concurrency scenarios, this can
> result in significant CPU and memory wastage. By having each task write the
> appropriate thrift objects to the output files, HiveServer2 can simply stream
> a batch of rows on the wire without incurring any of the additional cost of
> deserialization and translation.
> This can be implemented by writing a new SerDe, which the FileSinkOperator
> can use to write thrift formatted row batches to the output file. Using the
> pluggable property of the {{hive.query.result.fileformat}}, we can set it to
> use SequenceFile and write a batch of thrift formatted rows as a value blob.
> The FetchTask can now simply read the blob and send it over the wire. On the
> client side, the *DBC driver can read the blob and since it is already
> formatted in the way it expects, it can continue building the ResultSet the
> way it does in the current implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13572) Redundant setting full file status in Hive::copyFiles
[
https://issues.apache.org/jira/browse/HIVE-13572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254438#comment-15254438
]
Ashutosh Chauhan commented on HIVE-13572:
-
Ideal solution for this problem is to pass desired permissions to
FileSinkOperator and have it create files with desired permission (using mask).
That will avoid this 2-step process of first creating files and then setting
permissions on it.
> Redundant setting full file status in Hive::copyFiles
> -
>
> Key: HIVE-13572
> URL: https://issues.apache.org/jira/browse/HIVE-13572
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-13572.1.patch
>
>
> We set full file status in each copy-file thread. I think it's redundant and
> hurts performance when we have multiple files to copy.
> {code}
> if (inheritPerms) {
> ShimLoader.getHadoopShims().setFullFileStatus(conf,
> fullDestStatus, destFs, destf);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13449) LLAP: HS2 should get the token directly, rather than from LLAP
[ https://issues.apache.org/jira/browse/HIVE-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254439#comment-15254439 ] Sergey Shelukhin commented on HIVE-13449: - [~vikram.dixit] [~hagleitn] perhaps you can review this > LLAP: HS2 should get the token directly, rather than from LLAP > -- > > Key: HIVE-13449 > URL: https://issues.apache.org/jira/browse/HIVE-13449 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13449.patch > > > HS2 doesn't need a roundtrip to LLAP; it can instantiate the SecretManager > directly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13449) LLAP: HS2 should get the token directly, rather than from LLAP
[ https://issues.apache.org/jira/browse/HIVE-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13449: Attachment: HIVE-13449.01.patch Resubmitting the same patch for HiveQA > LLAP: HS2 should get the token directly, rather than from LLAP > -- > > Key: HIVE-13449 > URL: https://issues.apache.org/jira/browse/HIVE-13449 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13449.01.patch, HIVE-13449.patch > > > HS2 doesn't need a roundtrip to LLAP; it can instantiate the SecretManager > directly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13391) add an option to LLAP to use keytab to authenticate to read data
[ https://issues.apache.org/jira/browse/HIVE-13391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13391: Attachment: HIVE-13391.05.patch The same patch for QA... Grrr > add an option to LLAP to use keytab to authenticate to read data > > > Key: HIVE-13391 > URL: https://issues.apache.org/jira/browse/HIVE-13391 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13391.01.patch, HIVE-13391.02.patch, > HIVE-13391.03.patch, HIVE-13391.04.patch, HIVE-13391.05.patch, > HIVE-13391.patch > > > This can be used for non-doAs case to allow access to clients who don't > propagate HDFS tokens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11887) spark tests break the build on a shared machine, can break HiveQA
[ https://issues.apache.org/jira/browse/HIVE-11887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11887: Attachment: HIVE-11887.02.patch Same patch. I never have time to drive this to conclusion... > spark tests break the build on a shared machine, can break HiveQA > - > > Key: HIVE-11887 > URL: https://issues.apache.org/jira/browse/HIVE-11887 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11887.01.patch, HIVE-11887.02.patch, > HIVE-11887.patch > > > Spark download creates UDFExampleAdd jar in /tmp; when building on a shared > machine, someone else's jar from a build prevents this jar from being created > (I have no permissions to this file because it was created by a different > user) and the build fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13221) expose metastore APIs from HS2
[ https://issues.apache.org/jira/browse/HIVE-13221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254461#comment-15254461 ] Sergey Shelukhin commented on HIVE-13221: - [~thejas] [~sushanth] ping? > expose metastore APIs from HS2 > -- > > Key: HIVE-13221 > URL: https://issues.apache.org/jira/browse/HIVE-13221 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13221.01.patch, HIVE-13221.patch > > > I was always wondering why we don't do that, for the people who run HS2 and > also need metastore due to it being used externally; they don't need to run a > standalone metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13523) Fix connection leak in ORC RecordReader and refactor for unit testing
[ https://issues.apache.org/jira/browse/HIVE-13523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254464#comment-15254464 ] Owen O'Malley commented on HIVE-13523: -- I've redone this in my HIVE-12159 patch. The RecordReaderImpl now takes ReaderImpl and Reader.Options. It is simple and direct. > Fix connection leak in ORC RecordReader and refactor for unit testing > - > > Key: HIVE-13523 > URL: https://issues.apache.org/jira/browse/HIVE-13523 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.0.0 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Fix For: 2.1.0, 2.0.1 > > Attachments: HIVE-13523.patch > > > In RecordReaderImpl, a MetadataReaderImpl object was being created (opening a > file), but never closed, causing a leak. This change closes the Metadata > object in RecordReaderImpl, and does substantial refactoring to make > RecordReaderImpl testable: > * Created DataReaderFactory and MetadataReaderFactory (plus default > implementations) so that the create() methods can be mocked to verify that > the objects are actually closed in RecordReaderImpl.close() > * Created MetadataReaderProperties and DataReaderProperties to clean up > argument lists, making code more readable > * Created a builder() for RecordReaderImpl to make the code more readable > * DataReader and MetadataReader now extend closeable (there was no reason > for them not to in the first place) so I can use the guava Closer interface: > http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/io/Closer.html > * Use the Closer interface to guarantee that regardless of if either close() > call fails, both will be attempted (preventing further potential leaks) > * Create builders for MetadataReaderProperties, DataReaderProperties, and > RecordReaderImpl to help with code readability -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254475#comment-15254475 ] Ashutosh Chauhan commented on HIVE-13068: - Historically latency of doing optimization on Hive tree comes from slowness of traversing operator & expression tree, not of doing optimization itself (executing folding logic in this case). So my guess is even if you are executing very little logic latency of doing walk on trees will outnumber any gains. So, my assumption is we will see benefit of migration only if we can totally avoid the optimization on Hive tree. Further, it still leaves the room for bugs in new code, which is avoidable. So my suggestion is to instead try to fix all issues which comes out of it, instead of having this new simplified rule. > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch, > HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13592) metastore calls map is not thread safe
[ https://issues.apache.org/jira/browse/HIVE-13592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254484#comment-15254484 ] Aihua Xu commented on HIVE-13592: - I just wonder why we made such change in addMethodTime(). Should ConcurrentHashMap takes care of the concurrent update? > metastore calls map is not thread safe > -- > > Key: HIVE-13592 > URL: https://issues.apache.org/jira/browse/HIVE-13592 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13592.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12049) HiveServer2: Provide an option to write serialized thrift objects in final tasks
[
https://issues.apache.org/jira/browse/HIVE-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-12049:
Summary: HiveServer2: Provide an option to write serialized thrift objects
in final tasks (was: Provide an option to write serialized thrift objects in
final tasks)
> HiveServer2: Provide an option to write serialized thrift objects in final
> tasks
>
>
> Key: HIVE-12049
> URL: https://issues.apache.org/jira/browse/HIVE-12049
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, JDBC
>Affects Versions: 2.0.0
>Reporter: Rohit Dholakia
>Assignee: Rohit Dholakia
> Attachments: HIVE-12049.1.patch, HIVE-12049.11.patch,
> HIVE-12049.12.patch, HIVE-12049.13.patch, HIVE-12049.14.patch,
> HIVE-12049.15.patch, HIVE-12049.16.patch, HIVE-12049.17.patch,
> HIVE-12049.18.patch, HIVE-12049.19.patch, HIVE-12049.2.patch,
> HIVE-12049.25.patch, HIVE-12049.26.patch, HIVE-12049.3.patch,
> HIVE-12049.4.patch, HIVE-12049.5.patch, HIVE-12049.6.patch,
> HIVE-12049.7.patch, HIVE-12049.9.patch, new-driver-profiles.png,
> old-driver-profiles.png
>
>
> For each fetch request to HiveServer2, we pay the penalty of deserializing
> the row objects and translating them into a different representation suitable
> for the RPC transfer. In a moderate to high concurrency scenarios, this can
> result in significant CPU and memory wastage. By having each task write the
> appropriate thrift objects to the output files, HiveServer2 can simply stream
> a batch of rows on the wire without incurring any of the additional cost of
> deserialization and translation.
> This can be implemented by writing a new SerDe, which the FileSinkOperator
> can use to write thrift formatted row batches to the output file. Using the
> pluggable property of the {{hive.query.result.fileformat}}, we can set it to
> use SequenceFile and write a batch of thrift formatted rows as a value blob.
> The FetchTask can now simply read the blob and send it over the wire. On the
> client side, the *DBC driver can read the blob and since it is already
> formatted in the way it expects, it can continue building the ResultSet the
> way it does in the current implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12049) HiveServer2: Provide an option to write serialized thrift objects in final tasks
[
https://issues.apache.org/jira/browse/HIVE-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-12049:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.1.0
Target Version/s: 2.1.0
Status: Resolved (was: Patch Available)
Patch committed to master. Thanks [~rohitdholakia] for the work!
> HiveServer2: Provide an option to write serialized thrift objects in final
> tasks
>
>
> Key: HIVE-12049
> URL: https://issues.apache.org/jira/browse/HIVE-12049
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, JDBC
>Affects Versions: 2.0.0
>Reporter: Rohit Dholakia
>Assignee: Rohit Dholakia
> Fix For: 2.1.0
>
> Attachments: HIVE-12049.1.patch, HIVE-12049.11.patch,
> HIVE-12049.12.patch, HIVE-12049.13.patch, HIVE-12049.14.patch,
> HIVE-12049.15.patch, HIVE-12049.16.patch, HIVE-12049.17.patch,
> HIVE-12049.18.patch, HIVE-12049.19.patch, HIVE-12049.2.patch,
> HIVE-12049.25.patch, HIVE-12049.26.patch, HIVE-12049.3.patch,
> HIVE-12049.4.patch, HIVE-12049.5.patch, HIVE-12049.6.patch,
> HIVE-12049.7.patch, HIVE-12049.9.patch, new-driver-profiles.png,
> old-driver-profiles.png
>
>
> For each fetch request to HiveServer2, we pay the penalty of deserializing
> the row objects and translating them into a different representation suitable
> for the RPC transfer. In a moderate to high concurrency scenarios, this can
> result in significant CPU and memory wastage. By having each task write the
> appropriate thrift objects to the output files, HiveServer2 can simply stream
> a batch of rows on the wire without incurring any of the additional cost of
> deserialization and translation.
> This can be implemented by writing a new SerDe, which the FileSinkOperator
> can use to write thrift formatted row batches to the output file. Using the
> pluggable property of the {{hive.query.result.fileformat}}, we can set it to
> use SequenceFile and write a batch of thrift formatted rows as a value blob.
> The FetchTask can now simply read the blob and send it over the wire. On the
> client side, the *DBC driver can read the blob and since it is already
> formatted in the way it expects, it can continue building the ResultSet the
> way it does in the current implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[
https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254505#comment-15254505
]
Jesus Camacho Rodriguez commented on HIVE-13068:
I agree with you.
However, I think it is difficult because of the current limitations that I
outlined above.
- For instance, is there a way to specify that a given constant should be of a
certain type in HQL without using cast/udfto? I think there is not, that is why
I came with this solution.
I could try to add the logic to fold the expression to the SemanticAnalyzer,
thus not traversing the tree. But I think it will not make such a big
difference performance wise and we will end up adding more logic to
SemanticAnalyzer, which is already huge. Further, if return path is in place at
some point, it will be more difficult to detect/remove that unnecessary logic
too.
- The problem with {{power}} expression is that it was returning NaN when it
was folded, and I think we cannot convert that currently through AST? It was a
corner case and I did not explore it much further as the patch was so full of
small pieces. But I will explore it further.
- The filter removal for {{true}} predicates comes practically for free if we
need to execute SimpleConstantReduction, but I agree I could try to rewrite
that logic in SemanticAnalyzer.
The fact that we are not carrying constants all over the operator tree and
checking all the expressions to replace them, should make a performance
difference already. I agree with you that we need to take it a step further,
but without further integration work, I did not figure out a better solution
for some of these cases (specially the CAST, that is quite common because of
Calcite tight type system).
> Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
> ---
>
> Key: HIVE-13068
> URL: https://issues.apache.org/jira/browse/HIVE-13068
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Logical Optimizer
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch,
> HIVE-13068.patch
>
>
> After HIVE-12543 went in, we need follow-up work to disable the last call to
> ConstantPropagate in Hive. This probably implies work on extending the
> constant folding logic in Calcite.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13593) HiveServer2: Better performance instrumentation for HIVE-12049 (serializing thrift ResultSets in tasks)
[ https://issues.apache.org/jira/browse/HIVE-13593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-13593: Issue Type: Sub-task (was: Bug) Parent: HIVE-12427 > HiveServer2: Better performance instrumentation for HIVE-12049 (serializing > thrift ResultSets in tasks) > --- > > Key: HIVE-13593 > URL: https://issues.apache.org/jira/browse/HIVE-13593 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Affects Versions: 2.0.0 >Reporter: Vaibhav Gumashta > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
