[jira] [Commented] (HIVE-17350) metrics errors when retrying HS2 startup
[
https://issues.apache.org/jira/browse/HIVE-17350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16966923#comment-16966923
]
Sergey Shelukhin commented on HIVE-17350:
-
I don't recall, I think it wouldn't start anyway in this case because it would
fail again for the same reason it failed for the first time.
I just noticed this was broken so even if retry succeeded, metrics might be
broken.
> metrics errors when retrying HS2 startup
>
>
> Key: HIVE-17350
> URL: https://issues.apache.org/jira/browse/HIVE-17350
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Major
>
> Looks like there are some sort of retries that happen when HS2 init fails.
> When HS2 startup fails for an unrelated reason and is retried, the metrics
> source initialization fails on subsequent attempts.
> {noformat}
> 2017-08-15T23:31:47,650 WARN [main]: impl.MetricsSystemImpl
> (MetricsSystemImpl.java:init(152)) - hiveserver2 metrics system already
> initialized!
> 2017-08-15T23:31:47,650 ERROR [main]: metastore.HiveMetaStore
> (HiveMetaStore.java:init(438)) - error in Metrics init:
> java.lang.reflect.InvocationTargetException null
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hive.common.metrics.common.MetricsFactory.init(MetricsFactory.java:42)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:435)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:79)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:92)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6892)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:140)
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:74)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1653)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:83)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3612)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3664)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3644)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:582)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:545)
> at
> org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:128)
> at org.apache.hive.service.cli.CLIService.init(CLIService.java:113)
> at
> org.apache.hive.service.CompositeService.init(CompositeService.java:59)
> at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:139)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:595)
> at
> org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:97)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:843)
> at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:712)
>
[jira] [Commented] (HIVE-19831) Hiveserver2 should skip doAuth checks for CREATE DATABASE/TABLE if database/table already exists
[ https://issues.apache.org/jira/browse/HIVE-19831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16876426#comment-16876426 ] Sergey Shelukhin commented on HIVE-19831: - Hmm... doesn't this expose the existence of the database by that name to an unauthorized user? > Hiveserver2 should skip doAuth checks for CREATE DATABASE/TABLE if > database/table already exists > > > Key: HIVE-19831 > URL: https://issues.apache.org/jira/browse/HIVE-19831 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 1.2.1, 2.1.0 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-19831.01.patch, HIVE-19831.patch > > > with sqlstdauth on, Create database if exists take TOO LONG if there are too > many objects inside the database directory. Hive should not run the doAuth > checks for all the objects within database if the database already exists. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21676) use a system table as an alternative proc store
[
https://issues.apache.org/jira/browse/HIVE-21676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16831806#comment-16831806
]
Sergey Shelukhin commented on HIVE-21676:
-
Lol, no, it's supposed to be an HBase ticket
> use a system table as an alternative proc store
> ---
>
> Key: HIVE-21676
> URL: https://issues.apache.org/jira/browse/HIVE-21676
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Major
>
> We keep hitting these issues:
> {noformat}
> 2019-04-30 23:41:52,164 INFO [master/master:17000:becomeActiveMaster]
> procedure2.ProcedureExecutor: Starting 16 core workers (bigger of cpus/4 or
> 16) with max (burst) worker count=160
> 2019-04-30 23:41:52,171 INFO [master/master:17000:becomeActiveMaster]
> util.FSHDFSUtils: Recover lease on dfs file
> .../MasterProcWALs/pv2-0481.log
> 2019-04-30 23:41:52,176 INFO [master/master:17000:becomeActiveMaster]
> util.FSHDFSUtils: Recovered lease, attempt=0 on
> file=.../MasterProcWALs/pv2-0481.log after 5ms
> 2019-04-30 23:41:52,288 INFO [master/master:17000:becomeActiveMaster]
> util.FSHDFSUtils: Recover lease on dfs file
> .../MasterProcWALs/pv2-0482.log
> 2019-04-30 23:41:52,289 INFO [master/master:17000:becomeActiveMaster]
> util.FSHDFSUtils: Recovered lease, attempt=0 on
> file=.../MasterProcWALs/pv2-0482.log after 1ms
> 2019-04-30 23:41:52,373 INFO [master/master:17000:becomeActiveMaster]
> wal.WALProcedureStore: Rolled new Procedure Store WAL, id=483
> 2019-04-30 23:41:52,375 INFO [master/master:17000:becomeActiveMaster]
> procedure2.ProcedureExecutor: Recovered WALProcedureStore lease in 206msec
> 2019-04-30 23:41:52,782 INFO [master/master:17000:becomeActiveMaster]
> wal.ProcedureWALFormatReader: Read 1556 entries in
> .../MasterProcWALs/pv2-0482.log
> 2019-04-30 23:41:55,370 INFO [master/master:17000:becomeActiveMaster]
> wal.ProcedureWALFormatReader: Read 28113 entries in
> .../MasterProcWALs/pv2-0481.log
> 2019-04-30 23:41:55,384 ERROR [master/master:17000:becomeActiveMaster]
> wal.WALProcedureTree: Missing stack id 166, max stack id is 181, root
> procedure is Procedure(pid=289380, ppid=-1,
> class=org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure)
> 2019-04-30 23:41:55,384 ERROR [master/master:17000:becomeActiveMaster]
> wal.WALProcedureTree: Missing stack id 178, max stack id is 181, root
> procedure is Procedure(pid=289380, ppid=-1,
> class=org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure)
> 2019-04-30 23:41:55,389 ERROR [master/master:17000:becomeActiveMaster]
> wal.WALProcedureTree: Missing stack id 359, max stack id is 360, root
> procedure is Procedure(pid=285640, ppid=-1,
> class=org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure)
> {noformat}
> After which the procedure(s) is/are lost and cluster is stuck permanently.
> There were no errors writing these files in the log, and no issues reading
> them from HDFS, so it's purely a data loss issue in the structure.
> I was thinking about debugging it, but on 2nd thought what we are trying to
> store is some PB blob, by key.
> Coincidentally, we have an "HBase" facility that we already deploy, that does
> just that... and it even has a WAL implementation. I don't know why we cannot
> use it for procedure state and have to invent another complex implementation
> of a KV store inside a KV store.
> In all/most cases, we don't even support rollback and use the latest state,
> but if we need multiple versions, this HBase product even supports that!
> I think we should add a hbase:proc table that would be maintained similar to
> meta. The latter part esp. given the existing code for meta should be much
> more simple than a separate store impl.
> This should be pluggable and optional via ProcStore interface (made more
> abstract as relevant - update state, scan state, get)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21676) use a system table as an alternative proc store
[
https://issues.apache.org/jira/browse/HIVE-21676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-21676:
Description:
We keep hitting these issues:
{noformat}
2019-04-30 23:41:52,164 INFO [master/master:17000:becomeActiveMaster]
procedure2.ProcedureExecutor: Starting 16 core workers (bigger of cpus/4 or 16)
with max (burst) worker count=160
2019-04-30 23:41:52,171 INFO [master/master:17000:becomeActiveMaster]
util.FSHDFSUtils: Recover lease on dfs file
.../MasterProcWALs/pv2-0481.log
2019-04-30 23:41:52,176 INFO [master/master:17000:becomeActiveMaster]
util.FSHDFSUtils: Recovered lease, attempt=0 on
file=.../MasterProcWALs/pv2-0481.log after 5ms
2019-04-30 23:41:52,288 INFO [master/master:17000:becomeActiveMaster]
util.FSHDFSUtils: Recover lease on dfs file
.../MasterProcWALs/pv2-0482.log
2019-04-30 23:41:52,289 INFO [master/master:17000:becomeActiveMaster]
util.FSHDFSUtils: Recovered lease, attempt=0 on
file=.../MasterProcWALs/pv2-0482.log after 1ms
2019-04-30 23:41:52,373 INFO [master/master:17000:becomeActiveMaster]
wal.WALProcedureStore: Rolled new Procedure Store WAL, id=483
2019-04-30 23:41:52,375 INFO [master/master:17000:becomeActiveMaster]
procedure2.ProcedureExecutor: Recovered WALProcedureStore lease in 206msec
2019-04-30 23:41:52,782 INFO [master/master:17000:becomeActiveMaster]
wal.ProcedureWALFormatReader: Read 1556 entries in
.../MasterProcWALs/pv2-0482.log
2019-04-30 23:41:55,370 INFO [master/master:17000:becomeActiveMaster]
wal.ProcedureWALFormatReader: Read 28113 entries in
.../MasterProcWALs/pv2-0481.log
2019-04-30 23:41:55,384 ERROR [master/master:17000:becomeActiveMaster]
wal.WALProcedureTree: Missing stack id 166, max stack id is 181, root procedure
is Procedure(pid=289380, ppid=-1,
class=org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure)
2019-04-30 23:41:55,384 ERROR [master/master:17000:becomeActiveMaster]
wal.WALProcedureTree: Missing stack id 178, max stack id is 181, root procedure
is Procedure(pid=289380, ppid=-1,
class=org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure)
2019-04-30 23:41:55,389 ERROR [master/master:17000:becomeActiveMaster]
wal.WALProcedureTree: Missing stack id 359, max stack id is 360, root procedure
is Procedure(pid=285640, ppid=-1,
class=org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure)
{noformat}
After which the procedure(s) is/are lost and cluster is stuck permanently.
There were no errors writing these files in the log, and no issues reading them
from HDFS, so it's purely a data loss issue in the structure.
I was thinking about debugging it, but on 2nd thought what we are trying to
store is some PB blob, by key.
Coincidentally, we have an "HBase" facility that we already deploy, that does
just that... and it even has a WAL implementation. I don't know why we cannot
use it for procedure state and have to invent another complex implementation of
a KV store inside a KV store.
In all/most cases, we don't even support rollback and use the latest state, but
if we need multiple versions, this HBase product even supports that!
I think we should add a hbase:proc table that would be maintained similar to
meta. The latter part esp. given the existing code for meta should be much more
simple than a separate store impl.
This should be pluggable and optional via ProcStore interface (made more
abstract as relevant - update state, scan state, get)
was:
We keep hitting these issues:
{noformat}
2019-04-30 23:41:52,164 INFO [master/master:17000:becomeActiveMaster]
procedure2.ProcedureExecutor: Starting 16 core workers (bigger of cpus/4 or 16)
with max (burst) worker count=160
2019-04-30 23:41:52,171 INFO [master/master:17000:becomeActiveMaster]
util.FSHDFSUtils: Recover lease on dfs file
.../MasterProcWALs/pv2-0481.log
2019-04-30 23:41:52,176 INFO [master/master:17000:becomeActiveMaster]
util.FSHDFSUtils: Recovered lease, attempt=0 on
file=.../MasterProcWALs/pv2-0481.log after 5ms
2019-04-30 23:41:52,288 INFO [master/master:17000:becomeActiveMaster]
util.FSHDFSUtils: Recover lease on dfs file
.../MasterProcWALs/pv2-0482.log
2019-04-30 23:41:52,289 INFO [master/master:17000:becomeActiveMaster]
util.FSHDFSUtils: Recovered lease, attempt=0 on
file=.../MasterProcWALs/pv2-0482.log after 1ms
2019-04-30 23:41:52,373 INFO [master/master:17000:becomeActiveMaster]
wal.WALProcedureStore: Rolled new Procedure Store WAL, id=483
2019-04-30 23:41:52,375 INFO [master/master:17000:becomeActiveMaster]
procedure2.ProcedureExecutor: Recovered WALProcedureStore lease in 206msec
2019-04-30 23:41:52,782 INFO [master/master:17000:becomeActiveMaster]
wal.ProcedureWALFormatReader: Read 1556 entries in
.../MasterProcWALs/pv2-000
[jira] [Assigned] (HIVE-17731) add a backward compat option for external users to HIVE-11985
[ https://issues.apache.org/jira/browse/HIVE-17731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-17731: --- Assignee: Sergey Shelukhin (was: Mate Juhasz) > add a backward compat option for external users to HIVE-11985 > - > > Key: HIVE-17731 > URL: https://issues.apache.org/jira/browse/HIVE-17731 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Labels: TODOC2.4 > Fix For: 3.0.0, 2.4.0 > > Attachments: HIVE-17731.patch > > > See HIVE-11985. > Some external callers (e.g. Presto) do not appear to process types from > deserializer correctly, relying on DB types. Ideally, it should be resolved > via HIVE-17714, hiding the custom SerDe logic from users. > For now we can add a backward compatibility config for such cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21563) Improve Table#getEmptyTable performance by disable registerAllFunctionsOnce
[
https://issues.apache.org/jira/browse/HIVE-21563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16808002#comment-16808002
]
Sergey Shelukhin commented on HIVE-21563:
-
Hmm.. what are the side effects of this behavior? It looks like it would affect
other methods.
> Improve Table#getEmptyTable performance by disable registerAllFunctionsOnce
> ---
>
> Key: HIVE-21563
> URL: https://issues.apache.org/jira/browse/HIVE-21563
> Project: Hive
> Issue Type: Improvement
>Reporter: Yuming Wang
>Assignee: Yuming Wang
>Priority: Major
> Attachments: HIVE-21563.001.patch
>
>
> We do not need registerAllFunctionsOnce when {{Table#getEmptyTable}}. The
> stack trace:
> {noformat}
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerGenericUDF(Registry.java:177)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerGenericUDF(Registry.java:170)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.(FunctionRegistry.java:209)
> at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:247)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:231)
> at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:388)
> at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:332)
> at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:312)
> at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:288)
> at
> org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:913)
> at
> org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:877)
> at
> org.apache.hadoop.hive.ql.session.SessionState.getAuthenticator(SessionState.java:1479)
> at
> org.apache.hadoop.hive.ql.session.SessionState.getUserFromAuthenticator(SessionState.java:1150)
> at org.apache.hadoop.hive.ql.metadata.Table.getEmptyTable(Table.java:180)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18935) LLAP: add jmx view of what's in cache
[ https://issues.apache.org/jira/browse/HIVE-18935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-18935: --- Assignee: (was: Sergey Shelukhin) > LLAP: add jmx view of what's in cache > - > > Key: HIVE-18935 > URL: https://issues.apache.org/jira/browse/HIVE-18935 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Priority: Major > > Based on HIVE-16992 work -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17411) LLAP IO may incorrectly release a refcount in some rare cases
[ https://issues.apache.org/jira/browse/HIVE-17411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17411: Description: java.io.IOException: java.lang.AssertionError: Unexpected refCount -1: 0x69a390d5(-1) In a large stream whose buffers are not reused, and that is separated into many CB (e.g. due to a small ORC compression buffer size), it may happen that some, but not all, buffers that are read together as a unit are evicted from cache. If CacheBuffer follows BufferChunk in the buffer list when a stream like this is read, the latter will be converted to ProcCacheChunk; it is possible for early refcount release logic from the former to release the refcount (for a dictionary stream, the initial refCount is always released early), and then backtrack to the latter to see if we can unlock more buffers. It would then to decref an uninitialized MemoryBuffer in ProcCacheChunk because ProcCacheChunk looks like a CacheChunk. PCC initial refcounts are released separately after the data is uncompressed. I'm assuming this would almost never happen with non-stripe-level streams because one would need a large RG to span 2+ CBs, no overlap with next/previous RGs in 2+ buffers for the early release to kick in, and an unfortunate eviction order. However it's possible with large-ish dictionaries. was: In a large stream whose buffers are not reused, and that is separated into many CB (e.g. due to a small ORC compression buffer size), it may happen that some, but not all, buffers that are read together as a unit are evicted from cache. If CacheBuffer follows BufferChunk in the buffer list when a stream like this is read, the latter will be converted to ProcCacheChunk; it is possible for early refcount release logic from the former to release the refcount (for a dictionary stream, the initial refCount is always released early), and then backtrack to the latter to see if we can unlock more buffers. It would then to decref an uninitialized MemoryBuffer in ProcCacheChunk because ProcCacheChunk looks like a CacheChunk. PCC initial refcounts are released separately after the data is uncompressed. I'm assuming this would almost never happen with non-stripe-level streams because one would need a large RG to span 2+ CBs, no overlap with next/previous RGs in 2+ buffers for the early release to kick in, and an unfortunate eviction order. However it's possible with large-ish dictionaries. > LLAP IO may incorrectly release a refcount in some rare cases > - > > Key: HIVE-17411 > URL: https://issues.apache.org/jira/browse/HIVE-17411 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 2.3.0, 3.0.0, 2.4.0 > > Attachments: HIVE-17411.patch > > > java.io.IOException: java.lang.AssertionError: Unexpected refCount -1: > 0x69a390d5(-1) > In a large stream whose buffers are not reused, and that is separated into > many CB (e.g. due to a small ORC compression buffer size), it may happen that > some, but not all, buffers that are read together as a unit are evicted from > cache. > If CacheBuffer follows BufferChunk in the buffer list when a stream like this > is read, the latter will be converted to ProcCacheChunk; it is possible for > early refcount release logic from the former to release the refcount (for a > dictionary stream, the initial refCount is always released early), and then > backtrack to the latter to see if we can unlock more buffers. It would then > to decref an uninitialized MemoryBuffer in ProcCacheChunk because > ProcCacheChunk looks like a CacheChunk. PCC initial refcounts are released > separately after the data is uncompressed. > I'm assuming this would almost never happen with non-stripe-level streams > because one would need a large RG to span 2+ CBs, no overlap with > next/previous RGs in 2+ buffers for the early release to kick in, and an > unfortunate eviction order. However it's possible with large-ish dictionaries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20787) MapJoinBytesTableContainer dummyRow case doesn't handle reuse
[ https://issues.apache.org/jira/browse/HIVE-20787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20787: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Committed to master; thanks for the review > MapJoinBytesTableContainer dummyRow case doesn't handle reuse > - > > Key: HIVE-20787 > URL: https://issues.apache.org/jira/browse/HIVE-20787 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20787.01.patch, HIVE-20787.02.patch, > HIVE-20787.patch > > > Discovered while investigating some (probably) unrelated issue. > MapJoinBytesTableContainer was not intended to be reused, but it looks like > some code might reuse it. If that happens, the dummyRow case will not work > correctly (dummyRow is cleared on first(), so another call to first() will > behave differently). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20807) Refactor LlapStatusServiceDriver
[ https://issues.apache.org/jira/browse/HIVE-20807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16684439#comment-16684439 ] Sergey Shelukhin commented on HIVE-20807: - +1 Nit: some lines are made longer than 100 chars that used to be shorter, and there weren't any logic changes to them. I think your auto-formatting might need to be set to a 100-char limit. > Refactor LlapStatusServiceDriver > > > Key: HIVE-20807 > URL: https://issues.apache.org/jira/browse/HIVE-20807 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20807.01.patch, HIVE-20807.02.patch, > HIVE-20807.03.patch, HIVE-20807.04.patch, HIVE-20807.05.patch, > HIVE-20807.06.patch > > > LlapStatusServiceDriver is the class used to determine if LLAP has started. > The following problems should be solved by refactoring: > 1. The main class is more than 800 lines long,should be cut into multiple > smaller classes. > 2. The current design makes it extremely hard to write unit tests. > 3. There are some overcomplicated, over-engineered parts of the code. > 4. Most of the code is under org.apache.hadoop.hive.llap.cli, but some parts > are under org.apache.hadoop.hive.llap.cli.status. The whole program could be > moved to the latter. > 5. LlapStatusHelpers serves as a class for holding classes, which doesn't > make much sense. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20787) MapJoinBytesTableContainer dummyRow case doesn't handle reuse
[ https://issues.apache.org/jira/browse/HIVE-20787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20787: Attachment: HIVE-20787.02.patch > MapJoinBytesTableContainer dummyRow case doesn't handle reuse > - > > Key: HIVE-20787 > URL: https://issues.apache.org/jira/browse/HIVE-20787 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20787.01.patch, HIVE-20787.02.patch, > HIVE-20787.patch > > > Discovered while investigating some (probably) unrelated issue. > MapJoinBytesTableContainer was not intended to be reused, but it looks like > some code might reuse it. If that happens, the dummyRow case will not work > correctly (dummyRow is cleared on first(), so another call to first() will > behave differently). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20380) LLAP cache should cache small buffers more efficiently
[ https://issues.apache.org/jira/browse/HIVE-20380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20380: Description: The main thing to do is https://issues.apache.org/jira/browse/HIVE-20380?focusedCommentId=16591009&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16591009 There are also minor heap improvements possible. 1) Intern tracking tag. 2) Replace AtomicLong object with a long and unsafe CAS method, we only ever use one method, compareAndSwap. One more idea is making tracking less object oriented, in particular passing around integer indexes instead of objects and storing state in giant arrays somewhere (potentially with some optimizations for less common things), instead of every buffers getting its own object. cc [~gopalv] [~prasanth_j] was: Lately ORC CBs are becoming ridiculously small. First there's the 4Kb minimum (instead of 256Kb), then after we moved metadata cache off-heap, the index streams that are all tiny take up a lot of CBs and waste space. Wasted space can require larger cache and lead to cache OOMs on some workloads. Reducing min.alloc solves this problem, but then there's a lot of heap (and probably compute) overhead to track all these buffers. Arguably even the 4Kb min.alloc is too small. The initial idea was to store multiple CBs per block, however this is a lot of work all over the place (cache mapping, cache lookups, everywhere in the readers, etc.). Consolidating and reducing allocation sizes after we know the "real" size after decompression is the new idea (see comments) that can be confined to the allocator and is also more flexible - no dependence on cache map, so we don't need to make sure stuff is contiguous and such (for example, R_I streams that we want to consolidate are interleaved with large bloom filters, that we don't want to read or consolidate when they are not needed - but cache key structure depends on offsets, so we'd need a new cache map for R_I and separate logic for these streams). Also streams like PRESENT with one small CB cannot be combined with anything realistically speaking, but shrinking the allocation will help them. There are also minor heap improvements possible. 1) Intern tracking tag. 2) Replace AtomicLong object with a long and unsafe CAS method, we only ever use one method, compareAndSwap. One more idea is making tracking less object oriented, in particular passing around integer indexes instead of objects and storing state in giant arrays somewhere (potentially with some optimizations for less common things), instead of every buffers getting its own object. cc [~gopalv] [~prasanth_j] > LLAP cache should cache small buffers more efficiently > -- > > Key: HIVE-20380 > URL: https://issues.apache.org/jira/browse/HIVE-20380 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > > The main thing to do is > https://issues.apache.org/jira/browse/HIVE-20380?focusedCommentId=16591009&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16591009 > There are also minor heap improvements possible. > 1) Intern tracking tag. > 2) Replace AtomicLong object with a long and unsafe CAS method, we only ever > use one method, compareAndSwap. > One more idea is making tracking less object oriented, in particular passing > around integer indexes instead of objects and storing state in giant arrays > somewhere (potentially with some optimizations for less common things), > instead of every buffers getting its own object. > cc [~gopalv] [~prasanth_j] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20117) schema changes for txn stats
[ https://issues.apache.org/jira/browse/HIVE-20117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20117: Resolution: Implemented Status: Resolved (was: Patch Available) > schema changes for txn stats > > > Key: HIVE-20117 > URL: https://issues.apache.org/jira/browse/HIVE-20117 > Project: Hive > Issue Type: Bug > Components: Statistics, Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20117.01.patch, HIVE-20117.02.patch, > HIVE-20117.03.patch, HIVE-20117.04.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19593) CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19593: Resolution: Not A Problem Status: Resolved (was: Patch Available) > CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19593 > URL: https://issues.apache.org/jira/browse/HIVE-19593 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19593.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-19596) NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin resolved HIVE-19596. - Resolution: Not A Problem > NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19596 > URL: https://issues.apache.org/jira/browse/HIVE-19596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19596.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-12014) re-enable most LLAP tests after merge
[ https://issues.apache.org/jira/browse/HIVE-12014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12014: Resolution: Not A Problem Status: Resolved (was: Patch Available) > re-enable most LLAP tests after merge > - > > Key: HIVE-12014 > URL: https://issues.apache.org/jira/browse/HIVE-12014 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-12014.patch > > > see HIVE-12013 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20845) Fix TestJdbcWithDBTokenStoreNoDoAs flakiness
[ https://issues.apache.org/jira/browse/HIVE-20845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16684323#comment-16684323 ] Sergey Shelukhin commented on HIVE-20845: - [~pvary] that is specific to some unit tests that start HS2 in passive mode (HA tests). HS2 that comes up as passive will not accept the session, so starting one is not a valid liveness check. Not sure why it won't repro... I think I hit it on the Mac before fixing. > Fix TestJdbcWithDBTokenStoreNoDoAs flakiness > > > Key: HIVE-20845 > URL: https://issues.apache.org/jira/browse/HIVE-20845 > Project: Hive > Issue Type: Test >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-20845.2.patch, HIVE-20845.patch > > > Previously did a dirty fix for TestJdbcWithDBTokenStoreNoDoAs and > TestJdbcWithDBTokenStore > Found out the issue is that we do not wait enough for HS2 to come up. > Need to fix in MiniHS2.waitForStartup() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20899) Keytab URI for LLAP YARN Service is restrictive to support HDFS only
[ https://issues.apache.org/jira/browse/HIVE-20899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16684291#comment-16684291 ] Sergey Shelukhin commented on HIVE-20899: - Wouldn't this break backward compat for paths specified the old way? We can check the path for being the full URI and append if it's not. +-0 > Keytab URI for LLAP YARN Service is restrictive to support HDFS only > > > Key: HIVE-20899 > URL: https://issues.apache.org/jira/browse/HIVE-20899 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Gour Saha >Assignee: Gour Saha >Priority: Major > Attachments: HIVE-20899.01.patch, HIVE-20899.02.patch, > HIVE-20899.03.patch, HIVE-20899.04.patch > > > llap-server/src/main/resources/package.py restricts the keytab URI to support > HDFS only and hence fails for other FileSystem API conforming FSs like s3a, > wasb, gs, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20853) Expose ShuffleHandler.registerDag in the llap daemon API
[ https://issues.apache.org/jira/browse/HIVE-20853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20853: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the patch! > Expose ShuffleHandler.registerDag in the llap daemon API > > > Key: HIVE-20853 > URL: https://issues.apache.org/jira/browse/HIVE-20853 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.1.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Fix For: 4.0.0 > > Attachments: HIVE-20853.1.patch, HIVE-20853.2.patch, > HIVE-20853.3.patch, HIVE-20853.4.patch, HIVE-20853.5.patch > > > Currently DAGs are only registered when a submitWork is called for that DAG. > At this point the crendentials are added to the ShuffleHandler and it can > start serving. > However Tez might (and will) schedule tasks to fetch from the ShuffleHandler > before anything of this happens and all this tasks will fail which may > results in the query failing. > This happens in the scenario in which a LlapDaemon just comes up and tez > fetchers try to open a connection before a DAG has been registered. > Adding this API will allow to register the DAG against the Daemon when the AM > notices that a new Daemon is up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20853) Expose ShuffleHandler.registerDag in the llap daemon API
[ https://issues.apache.org/jira/browse/HIVE-20853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16678930#comment-16678930 ] Sergey Shelukhin commented on HIVE-20853: - +1 pending tests > Expose ShuffleHandler.registerDag in the llap daemon API > > > Key: HIVE-20853 > URL: https://issues.apache.org/jira/browse/HIVE-20853 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.1.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: HIVE-20853.1.patch, HIVE-20853.2.patch, > HIVE-20853.3.patch, HIVE-20853.4.patch, HIVE-20853.5.patch > > > Currently DAGs are only registered when a submitWork is called for that DAG. > At this point the crendentials are added to the ShuffleHandler and it can > start serving. > However Tez might (and will) schedule tasks to fetch from the ShuffleHandler > before anything of this happens and all this tasks will fail which may > results in the query failing. > This happens in the scenario in which a LlapDaemon just comes up and tez > fetchers try to open a connection before a DAG has been registered. > Adding this API will allow to register the DAG against the Daemon when the AM > notices that a new Daemon is up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20853) Expose ShuffleHandler.registerDag in the llap daemon API
[ https://issues.apache.org/jira/browse/HIVE-20853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16677589#comment-16677589 ] Sergey Shelukhin commented on HIVE-20853: - To fetch something, you first need to write something... currently that always happens in the same DAG I think. So in order to fetch smth from a node current DAG has to have written something there before or after restart. Before should be solved by this patch as far as I can tell, and after is solved because the task that wrote stuff already added credentials. [~gopalv] correct me if I'm wrong about shuffle and fetch :) > Expose ShuffleHandler.registerDag in the llap daemon API > > > Key: HIVE-20853 > URL: https://issues.apache.org/jira/browse/HIVE-20853 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.1.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: HIVE-20853.1.patch, HIVE-20853.2.patch, > HIVE-20853.3.patch, HIVE-20853.4.patch > > > Currently DAGs are only registered when a submitWork is called for that DAG. > At this point the crendentials are added to the ShuffleHandler and it can > start serving. > However Tez might (and will) schedule tasks to fetch from the ShuffleHandler > before anything of this happens and all this tasks will fail which may > results in the query failing. > This happens in the scenario in which a LlapDaemon just comes up and tez > fetchers try to open a connection before a DAG has been registered. > Adding this API will allow to register the DAG against the Daemon when the AM > notices that a new Daemon is up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20853) Expose ShuffleHandler.registerDag in the llap daemon API
[ https://issues.apache.org/jira/browse/HIVE-20853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16677555#comment-16677555 ] Sergey Shelukhin commented on HIVE-20853: - Hm... isn't the problem happening when the recovery happens, i.e. when a new node is added and the shuffle data is fetched from a task that was on the previous instance on the same machine? The latter can only happen if the node was replaced/restarted during the dag execution, right? So by definition we have a DAG id > Expose ShuffleHandler.registerDag in the llap daemon API > > > Key: HIVE-20853 > URL: https://issues.apache.org/jira/browse/HIVE-20853 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.1.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: HIVE-20853.1.patch, HIVE-20853.2.patch, > HIVE-20853.3.patch, HIVE-20853.4.patch > > > Currently DAGs are only registered when a submitWork is called for that DAG. > At this point the crendentials are added to the ShuffleHandler and it can > start serving. > However Tez might (and will) schedule tasks to fetch from the ShuffleHandler > before anything of this happens and all this tasks will fail which may > results in the query failing. > This happens in the scenario in which a LlapDaemon just comes up and tez > fetchers try to open a connection before a DAG has been registered. > Adding this API will allow to register the DAG against the Daemon when the AM > notices that a new Daemon is up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20853) Expose ShuffleHandler.registerDag in the llap daemon API
[ https://issues.apache.org/jira/browse/HIVE-20853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16677398#comment-16677398 ] Sergey Shelukhin commented on HIVE-20853: - Some minor comments. TestLlapDaemonUtils failed/timed out. Has this been tested in a secure cluster? I dunno if tests would cover much w.r.t. credentials. Also for all the new util classes, I'm pretty sure there already some utility classes for LLAP, perhaps some could be merged if they are in the same/visible module. > Expose ShuffleHandler.registerDag in the llap daemon API > > > Key: HIVE-20853 > URL: https://issues.apache.org/jira/browse/HIVE-20853 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.1.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: HIVE-20853.1.patch, HIVE-20853.2.patch, > HIVE-20853.3.patch, HIVE-20853.4.patch > > > Currently DAGs are only registered when a submitWork is called for that DAG. > At this point the crendentials are added to the ShuffleHandler and it can > start serving. > However Tez might (and will) schedule tasks to fetch from the ShuffleHandler > before anything of this happens and all this tasks will fail which may > results in the query failing. > This happens in the scenario in which a LlapDaemon just comes up and tez > fetchers try to open a connection before a DAG has been registered. > Adding this API will allow to register the DAG against the Daemon when the AM > notices that a new Daemon is up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20853) Expose ShuffleHandler.registerDag in the llap daemon API
[ https://issues.apache.org/jira/browse/HIVE-20853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16677345#comment-16677345 ] Sergey Shelukhin commented on HIVE-20853: - Is it possible to post a review board? https://reviews.apache.org/r/new/ > Expose ShuffleHandler.registerDag in the llap daemon API > > > Key: HIVE-20853 > URL: https://issues.apache.org/jira/browse/HIVE-20853 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.1.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: HIVE-20853.1.patch, HIVE-20853.2.patch, > HIVE-20853.3.patch, HIVE-20853.4.patch > > > Currently DAGs are only registered when a submitWork is called for that DAG. > At this point the crendentials are added to the ShuffleHandler and it can > start serving. > However Tez might (and will) schedule tasks to fetch from the ShuffleHandler > before anything of this happens and all this tasks will fail which may > results in the query failing. > This happens in the scenario in which a LlapDaemon just comes up and tez > fetchers try to open a connection before a DAG has been registered. > Adding this API will allow to register the DAG against the Daemon when the AM > notices that a new Daemon is up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20581) Eliminate rename() from full CRUD transactional tables
[
https://issues.apache.org/jira/browse/HIVE-20581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16677333#comment-16677333
]
Sergey Shelukhin commented on HIVE-20581:
-
MM tables were implemented separately with an explicit goal of avoiding all
move operations. Actually there should be no move operations at all for queries
involving MM tables (except for compactions). Writing directly to destination
requires some metadata manipulations to make files visible.
ACID didn't have that requirement so it operates similarly to regular Hive
tables.
I think this JIRA is to extend the MM functionality of not moving stuff to ACID
tables.
> Eliminate rename() from full CRUD transactional tables
> --
>
> Key: HIVE-20581
> URL: https://issues.apache.org/jira/browse/HIVE-20581
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Emily lozano d1751740 551913586
>Priority: Major
>
> The {{MoveTask}} in a query writing to full CRUD transactional table still
> performs a {{FileSystem.rename()}}. Full CRUD should follow the insert-only
> transactional table implementation and write directly to delta_x_x in the
> partition dir. If the txn fails, this delta will be marked aborted and will
> not be read.
> There are several places that rely on this rename. For example, support for
> {{Insert ... select ... Union All ... Select }} which creates multiple dirs,
> 1 for each leg of the union.
> Others?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20581) Eliminate rename() from full CRUD transactional tables
[
https://issues.apache.org/jira/browse/HIVE-20581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16677333#comment-16677333
]
Sergey Shelukhin edited comment on HIVE-20581 at 11/6/18 9:58 PM:
--
MM tables were implemented separately with an explicit goal of avoiding all
move operations. Actually there should be no move operations at all for queries
writing into MM tables (except for compactions). Writing directly to
destination requires some metadata manipulations to make files visible.
ACID didn't have that requirement so it operates similarly to regular Hive
tables.
I think this JIRA is to extend the MM functionality of not moving stuff to ACID
tables.
was (Author: sershe):
MM tables were implemented separately with an explicit goal of avoiding all
move operations. Actually there should be no move operations at all for queries
involving MM tables (except for compactions). Writing directly to destination
requires some metadata manipulations to make files visible.
ACID didn't have that requirement so it operates similarly to regular Hive
tables.
I think this JIRA is to extend the MM functionality of not moving stuff to ACID
tables.
> Eliminate rename() from full CRUD transactional tables
> --
>
> Key: HIVE-20581
> URL: https://issues.apache.org/jira/browse/HIVE-20581
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Emily lozano d1751740 551913586
>Priority: Major
>
> The {{MoveTask}} in a query writing to full CRUD transactional table still
> performs a {{FileSystem.rename()}}. Full CRUD should follow the insert-only
> transactional table implementation and write directly to delta_x_x in the
> partition dir. If the txn fails, this delta will be marked aborted and will
> not be read.
> There are several places that rely on this rename. For example, support for
> {{Insert ... select ... Union All ... Select }} which creates multiple dirs,
> 1 for each leg of the union.
> Others?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20841) LLAP: Make dynamic ports configurable
[ https://issues.apache.org/jira/browse/HIVE-20841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669431#comment-16669431 ] Sergey Shelukhin commented on HIVE-20841: - +1 > LLAP: Make dynamic ports configurable > - > > Key: HIVE-20841 > URL: https://issues.apache.org/jira/browse/HIVE-20841 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20841.1.patch > > > Some ports in llap -> tez interaction code uses dynamic ports, provide an > option to make them configurable to facilitate adding them to iptable rules > in some environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-14535) add insert-only ACID tables to Hive
[ https://issues.apache.org/jira/browse/HIVE-14535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin resolved HIVE-14535. - Resolution: Fixed Fix Version/s: 4.0.0 This was finished a fair amount of time ago > add insert-only ACID tables to Hive > > > Key: HIVE-14535 > URL: https://issues.apache.org/jira/browse/HIVE-14535 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Labels: TODOC3.0 > Fix For: 4.0.0 > > > Design doc: > https://docs.google.com/document/d/1b3t1RywfyRb73-cdvkEzJUyOiekWwkMHdiQ-42zCllY > Feel free to comment. > Update: we ended up going with sequence number based implementation > Update #2: this feature has been partially merged with ACID; the new table > type is insert_only ACID, and the difference from the regular ACID is that it > only supports inserts on one hand; and that it has no restrictions on file > format, table type (bucketing), and much fewer restrictions on other > operations (export/import, list bucketing, etc.) > Currently some features that used to work when it was separated are not > integrated properly; integration of these features is the remaining work in > this JIRA -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-17860) verify CTAS logic for MM tables
[ https://issues.apache.org/jira/browse/HIVE-17860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin resolved HIVE-17860. - Resolution: Fixed This was fixed a long time ago I think > verify CTAS logic for MM tables > --- > > Key: HIVE-17860 > URL: https://issues.apache.org/jira/browse/HIVE-17860 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Priority: Major > Labels: mm-gap-2 > > The mm_all test output matches the output before integration, but we need to > check the internals wrt txns make sense (and perhaps improve the test) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-17861) MM tables - multi-IOW is broken
[ https://issues.apache.org/jira/browse/HIVE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin resolved HIVE-17861. - Resolution: Fixed This was fixed a long time ago I think > MM tables - multi-IOW is broken > --- > > Key: HIVE-17861 > URL: https://issues.apache.org/jira/browse/HIVE-17861 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom >Priority: Major > Labels: mm-gap-2 > > After HIVE-17856, see if multi IOW was commented out because of IOW issues or > because it's broken in addition to IOW being broken. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20787) MapJoinBytesTableContainer dummyRow case doesn't handle reuse
[ https://issues.apache.org/jira/browse/HIVE-20787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669271#comment-16669271 ] Sergey Shelukhin commented on HIVE-20787: - [~jdere] maybe you can take a look? :) > MapJoinBytesTableContainer dummyRow case doesn't handle reuse > - > > Key: HIVE-20787 > URL: https://issues.apache.org/jira/browse/HIVE-20787 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20787.01.patch, HIVE-20787.patch > > > Discovered while investigating some (probably) unrelated issue. > MapJoinBytesTableContainer was not intended to be reused, but it looks like > some code might reuse it. If that happens, the dummyRow case will not work > correctly (dummyRow is cleared on first(), so another call to first() will > behave differently). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20736) Eagerly allocate Containers from Yarn
[ https://issues.apache.org/jira/browse/HIVE-20736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-20736: --- Assignee: (was: Sergey Shelukhin) > Eagerly allocate Containers from Yarn > - > > Key: HIVE-20736 > URL: https://issues.apache.org/jira/browse/HIVE-20736 > Project: Hive > Issue Type: Bug > Components: llap, Tez >Affects Versions: 4.0.0 >Reporter: slim bouguerra >Priority: Major > > According to [~sershe] HS2 interactive at startup-time tries to eagerly > allocated what ever needed to execute queries. But currently this process is > kind of broken. > AS of now HS2I starts and tries to allocate the resources but in cases there > is not enough Yarn resources in the desired queue, HS2I will keep trying in > the background forever and will not bubble this up as an issue. trying > forever to allocate without signaling error defeats the idea of eagerly > allocate in my opinion. I think HS2I has to fail the start if after XX > minutes can not eagerly allocate the minimum space needed to run the max > concurrent query. > CC [~hagleitn]/[~t3rmin4t0r] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20787) MapJoinBytesTableContainer dummyRow case doesn't handle reuse
[ https://issues.apache.org/jira/browse/HIVE-20787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20787: Attachment: HIVE-20787.01.patch > MapJoinBytesTableContainer dummyRow case doesn't handle reuse > - > > Key: HIVE-20787 > URL: https://issues.apache.org/jira/browse/HIVE-20787 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20787.01.patch, HIVE-20787.patch > > > Discovered while investigating some (probably) unrelated issue. > MapJoinBytesTableContainer was not intended to be reused, but it looks like > some code might reuse it. If that happens, the dummyRow case will not work > correctly (dummyRow is cleared on first(), so another call to first() will > behave differently). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20793) add RP namespacing to workload management
[ https://issues.apache.org/jira/browse/HIVE-20793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20793: Resolution: Fixed Status: Resolved (was: Patch Available) Committed to master. Thanks for the review! There's another jira where the sysdb script is getting upgraded, I filed a bug for it and resolved as duplicate. > add RP namespacing to workload management > - > > Key: HIVE-20793 > URL: https://issues.apache.org/jira/browse/HIVE-20793 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20793.01.nogen.patch, HIVE-20793.01.patch, > HIVE-20793.02.nogen.patch, HIVE-20793.02.patch, HIVE-20793.nogen.patch, > HIVE-20793.patch > > > The idea is to be able to use the same warehouse for multiple clusters in the > cloud use cases. This scenario is not currently supported by WM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-20840) sysdb test is not updated and fails on update
[ https://issues.apache.org/jira/browse/HIVE-20840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin resolved HIVE-20840. - Resolution: Duplicate > sysdb test is not updated and fails on update > - > > Key: HIVE-20840 > URL: https://issues.apache.org/jira/browse/HIVE-20840 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20840.test.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20840) sysdb test is not updated and fails on update
[ https://issues.apache.org/jira/browse/HIVE-20840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669230#comment-16669230 ] Sergey Shelukhin edited comment on HIVE-20840 at 10/30/18 7:32 PM: --- [~daijy] [~thejas] looks like when the upgrade changes on master were made the test was not updated to point to the new script. When it's updated, some query results disappear. Attached the changes that happen to the out file when the test is updated, see the end of the diff. was (Author: sershe): [~daijy] [~thejas] looks like when the upgrade changes on master were made the test was not updated to point to the new script. When it's updated, some query results disappear. Attached the changes. > sysdb test is not updated and fails on update > - > > Key: HIVE-20840 > URL: https://issues.apache.org/jira/browse/HIVE-20840 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20840.test.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20840) sysdb test is not updated and fails on update
[ https://issues.apache.org/jira/browse/HIVE-20840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669230#comment-16669230 ] Sergey Shelukhin commented on HIVE-20840: - [~daijy] [~thejas] looks like when the upgrade changes on master were made the test was not updated to point to the new script. When it's updated, some query results disappear. Attached the changes. > sysdb test is not updated and fails on update > - > > Key: HIVE-20840 > URL: https://issues.apache.org/jira/browse/HIVE-20840 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20840.test.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20840) sysdb test is not updated and fails on update
[ https://issues.apache.org/jira/browse/HIVE-20840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20840: Attachment: HIVE-20840.test.patch > sysdb test is not updated and fails on update > - > > Key: HIVE-20840 > URL: https://issues.apache.org/jira/browse/HIVE-20840 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20840.test.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20793) add RP namespacing to workload management
[ https://issues.apache.org/jira/browse/HIVE-20793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665753#comment-16665753 ] Sergey Shelukhin commented on HIVE-20793: - Addressed the feedback, fixed a bug. Also fixed some old issue that I noticed while fixing the bug - having the child pools take the entire capacity of the parent (i.e. 1.0) should be allowed. Fixed that too. > add RP namespacing to workload management > - > > Key: HIVE-20793 > URL: https://issues.apache.org/jira/browse/HIVE-20793 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20793.01.nogen.patch, HIVE-20793.01.patch, > HIVE-20793.02.nogen.patch, HIVE-20793.02.patch, HIVE-20793.nogen.patch, > HIVE-20793.patch > > > The idea is to be able to use the same warehouse for multiple clusters in the > cloud use cases. This scenario is not currently supported by WM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20793) add RP namespacing to workload management
[ https://issues.apache.org/jira/browse/HIVE-20793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20793: Attachment: HIVE-20793.02.patch > add RP namespacing to workload management > - > > Key: HIVE-20793 > URL: https://issues.apache.org/jira/browse/HIVE-20793 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20793.01.nogen.patch, HIVE-20793.01.patch, > HIVE-20793.02.nogen.patch, HIVE-20793.02.patch, HIVE-20793.nogen.patch, > HIVE-20793.patch > > > The idea is to be able to use the same warehouse for multiple clusters in the > cloud use cases. This scenario is not currently supported by WM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20793) add RP namespacing to workload management
[ https://issues.apache.org/jira/browse/HIVE-20793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20793: Attachment: HIVE-20793.02.nogen.patch > add RP namespacing to workload management > - > > Key: HIVE-20793 > URL: https://issues.apache.org/jira/browse/HIVE-20793 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20793.01.nogen.patch, HIVE-20793.01.patch, > HIVE-20793.02.nogen.patch, HIVE-20793.nogen.patch, HIVE-20793.patch > > > The idea is to be able to use the same warehouse for multiple clusters in the > cloud use cases. This scenario is not currently supported by WM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20605) merge master-tez092 branch into master
[ https://issues.apache.org/jira/browse/HIVE-20605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665676#comment-16665676 ] Sergey Shelukhin commented on HIVE-20605: - merged master into branch to make sure we stay on top of conflicts... I will just keep doing this in this JIRA > merge master-tez092 branch into master > -- > > Key: HIVE-20605 > URL: https://issues.apache.org/jira/browse/HIVE-20605 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > > I got tired of waiting for Tez 0.92 release (it's been pending for half a > year) so I created a branch to prevent various patches from conflicting with > each other. > This jira is to merge them into master after Tez 0.92 is finally released. > The jiras here: > https://issues.apache.org/jira/issues/?jql=project%20%3D%20HIVE%20AND%20fixVersion%20%3D%20master-tez092 > should then be updated with the corresponding Hive release version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20419) Vectorization: Prevent mutation of VectorPartitionDesc after being used in a hashmap key
[
https://issues.apache.org/jira/browse/HIVE-20419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664409#comment-16664409
]
Sergey Shelukhin commented on HIVE-20419:
-
Hmm. Cannot this also cause incorrect results? If the key is not found later.
> Vectorization: Prevent mutation of VectorPartitionDesc after being used in a
> hashmap key
>
>
> Key: HIVE-20419
> URL: https://issues.apache.org/jira/browse/HIVE-20419
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Gopal V
>Priority: Major
>
> This is going into the loop because the VectorPartitionDesc is modified after
> it is used in the HashMap key - resulting in a hashcode & equals modification
> after it has been placed in the hashmap.
> {code}
> HiveServer2-Background-Pool: Thread-6049 State: RUNNABLE CPU usage on sample:
> 621ms
> java.util.HashMap$TreeNode.find(int, Object, Class) HashMap.java:1869 <7
> recursive calls>
> java.util.HashMap$TreeNode.putTreeVal(HashMap, HashMap$Node[], int, Object,
> Object) HashMap.java:1989
> java.util.HashMap.putVal(int, Object, Object, boolean, boolean)
> HashMap.java:637
> java.util.HashMap.put(Object, Object) HashMap.java:611
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.addVectorPartitionDesc(PartitionDesc,
> VectorPartitionDesc, Map) Vectorizer.java:1272
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.verifyAndSetVectorPartDesc(PartitionDesc,
> boolean, List, Set, Map, Set, ArrayList, Set) Vectorizer.java:1323
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateInputFormatAndSchemaEvolution(MapWork,
> String, TableScanOperator, Vectorizer$VectorTaskColumnInfo)
> Vectorizer.java:1654
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapWork(MapWork,
> Vectorizer$VectorTaskColumnInfo, boolean) Vectorizer.java:1865
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.convertMapWork(MapWork,
> boolean) Vectorizer.java:1109
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Node,
> Stack, Object[]) Vectorizer.java:961
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(Node, Stack,
> TaskGraphWalker$TaskGraphWalkerContext) TaskGraphWalker.java:111
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(Node)
> TaskGraphWalker.java:180
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(Collection,
> HashMap) TaskGraphWalker.java:125
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.resolve(PhysicalContext)
> Vectorizer.java:2442
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeTaskPlan(List,
> ParseContext, Context) TezCompiler.java:717
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(ParseContext, List,
> HashSet, HashSet) TaskCompiler.java:258
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(ASTNode,
> SemanticAnalyzer$PlannerContextFactory) SemanticAnalyzer.java:12443
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(ASTNode)
> CalcitePlanner.java:358
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20419) Vectorization: Prevent mutation of VectorPartitionDesc after being used in a hashmap key
[
https://issues.apache.org/jira/browse/HIVE-20419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664410#comment-16664410
]
Sergey Shelukhin commented on HIVE-20419:
-
cc [~teddy.choi]
> Vectorization: Prevent mutation of VectorPartitionDesc after being used in a
> hashmap key
>
>
> Key: HIVE-20419
> URL: https://issues.apache.org/jira/browse/HIVE-20419
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Gopal V
>Priority: Major
>
> This is going into the loop because the VectorPartitionDesc is modified after
> it is used in the HashMap key - resulting in a hashcode & equals modification
> after it has been placed in the hashmap.
> {code}
> HiveServer2-Background-Pool: Thread-6049 State: RUNNABLE CPU usage on sample:
> 621ms
> java.util.HashMap$TreeNode.find(int, Object, Class) HashMap.java:1869 <7
> recursive calls>
> java.util.HashMap$TreeNode.putTreeVal(HashMap, HashMap$Node[], int, Object,
> Object) HashMap.java:1989
> java.util.HashMap.putVal(int, Object, Object, boolean, boolean)
> HashMap.java:637
> java.util.HashMap.put(Object, Object) HashMap.java:611
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.addVectorPartitionDesc(PartitionDesc,
> VectorPartitionDesc, Map) Vectorizer.java:1272
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.verifyAndSetVectorPartDesc(PartitionDesc,
> boolean, List, Set, Map, Set, ArrayList, Set) Vectorizer.java:1323
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateInputFormatAndSchemaEvolution(MapWork,
> String, TableScanOperator, Vectorizer$VectorTaskColumnInfo)
> Vectorizer.java:1654
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapWork(MapWork,
> Vectorizer$VectorTaskColumnInfo, boolean) Vectorizer.java:1865
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.convertMapWork(MapWork,
> boolean) Vectorizer.java:1109
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Node,
> Stack, Object[]) Vectorizer.java:961
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(Node, Stack,
> TaskGraphWalker$TaskGraphWalkerContext) TaskGraphWalker.java:111
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(Node)
> TaskGraphWalker.java:180
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(Collection,
> HashMap) TaskGraphWalker.java:125
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.resolve(PhysicalContext)
> Vectorizer.java:2442
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeTaskPlan(List,
> ParseContext, Context) TezCompiler.java:717
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(ParseContext, List,
> HashSet, HashSet) TaskCompiler.java:258
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(ASTNode,
> SemanticAnalyzer$PlannerContextFactory) SemanticAnalyzer.java:12443
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(ASTNode)
> CalcitePlanner.java:358
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20811) Turn on dynamic partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-20811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664346#comment-16664346 ] Sergey Shelukhin commented on HIVE-20811: - Hmm... what correctness issues are those? If without it there can be correctness issues should we remove the config and have it always on? > Turn on dynamic partitioned hash join > - > > Key: HIVE-20811 > URL: https://issues.apache.org/jira/browse/HIVE-20811 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20811.1.patch > > > Currently it is off by default. > Turning if ON by default will help fix correctness and other issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20793) add RP namespacing to workload management
[ https://issues.apache.org/jira/browse/HIVE-20793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20793: Attachment: HIVE-20793.01.patch > add RP namespacing to workload management > - > > Key: HIVE-20793 > URL: https://issues.apache.org/jira/browse/HIVE-20793 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20793.01.nogen.patch, HIVE-20793.01.patch, > HIVE-20793.nogen.patch, HIVE-20793.patch > > > The idea is to be able to use the same warehouse for multiple clusters in the > cloud use cases. This scenario is not currently supported by WM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20793) add RP namespacing to workload management
[ https://issues.apache.org/jira/browse/HIVE-20793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20793: Attachment: HIVE-20793.01.nogen.patch > add RP namespacing to workload management > - > > Key: HIVE-20793 > URL: https://issues.apache.org/jira/browse/HIVE-20793 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20793.01.nogen.patch, HIVE-20793.nogen.patch, > HIVE-20793.patch > > > The idea is to be able to use the same warehouse for multiple clusters in the > cloud use cases. This scenario is not currently supported by WM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20793) add RP namespacing to workload management
[ https://issues.apache.org/jira/browse/HIVE-20793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664318#comment-16664318 ] Sergey Shelukhin commented on HIVE-20793: - Fixed some bugs, updated sysdb views more, fixed the sysdb version in the test file. > add RP namespacing to workload management > - > > Key: HIVE-20793 > URL: https://issues.apache.org/jira/browse/HIVE-20793 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20793.01.nogen.patch, HIVE-20793.nogen.patch, > HIVE-20793.patch > > > The idea is to be able to use the same warehouse for multiple clusters in the > cloud use cases. This scenario is not currently supported by WM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20793) add RP namespacing to workload management
[ https://issues.apache.org/jira/browse/HIVE-20793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663008#comment-16663008 ] Sergey Shelukhin commented on HIVE-20793: - [~prasanth_j] [~jdere] can you please help review? https://reviews.apache.org/r/69148/ This is a very tedious patch... > add RP namespacing to workload management > - > > Key: HIVE-20793 > URL: https://issues.apache.org/jira/browse/HIVE-20793 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20793.nogen.patch, HIVE-20793.patch > > > The idea is to be able to use the same warehouse for multiple clusters in the > cloud use cases. This scenario is not currently supported by WM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20793) add RP namespacing to workload management
[ https://issues.apache.org/jira/browse/HIVE-20793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20793: Status: Patch Available (was: Open) > add RP namespacing to workload management > - > > Key: HIVE-20793 > URL: https://issues.apache.org/jira/browse/HIVE-20793 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20793.nogen.patch, HIVE-20793.patch > > > The idea is to be able to use the same warehouse for multiple clusters in the > cloud use cases. This scenario is not currently supported by WM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20793) add RP namespacing to workload management
[ https://issues.apache.org/jira/browse/HIVE-20793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20793: Attachment: HIVE-20793.patch > add RP namespacing to workload management > - > > Key: HIVE-20793 > URL: https://issues.apache.org/jira/browse/HIVE-20793 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20793.nogen.patch, HIVE-20793.patch > > > The idea is to be able to use the same warehouse for multiple clusters in the > cloud use cases. This scenario is not currently supported by WM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20793) add RP namespacing to workload management
[ https://issues.apache.org/jira/browse/HIVE-20793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20793: Attachment: HIVE-20793.nogen.patch > add RP namespacing to workload management > - > > Key: HIVE-20793 > URL: https://issues.apache.org/jira/browse/HIVE-20793 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20793.nogen.patch, HIVE-20793.patch > > > The idea is to be able to use the same warehouse for multiple clusters in the > cloud use cases. This scenario is not currently supported by WM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20793) add RP namespacing to workload management
[ https://issues.apache.org/jira/browse/HIVE-20793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-20793: --- > add RP namespacing to workload management > - > > Key: HIVE-20793 > URL: https://issues.apache.org/jira/browse/HIVE-20793 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > > The idea is to be able to use the same warehouse for multiple clusters in the > cloud use cases. This scenario is not currently supported by WM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20772) record per-task CPU counters in LLAP
[ https://issues.apache.org/jira/browse/HIVE-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20772: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the review! > record per-task CPU counters in LLAP > > > Key: HIVE-20772 > URL: https://issues.apache.org/jira/browse/HIVE-20772 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20772.01.patch, HIVE-20772.02.patch, > HIVE-20772.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20772) record per-task CPU counters in LLAP
[
https://issues.apache.org/jira/browse/HIVE-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661379#comment-16661379
]
Sergey Shelukhin commented on HIVE-20772:
-
{noformat}
+ java -cp
'/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/hive-ptest-3.0-classes.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/lib/*'
org.apache.hive.ptest.api.client.PTestClient --command testStart --outputDir
/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target
--password '[***]' --testHandle PreCommit-HIVE-Build-14608 --endpoint
http://104.198.109.242:8080/hive-ptest-3.0 --logsEndpoint
http://104.198.109.242/logs/ --profile master-mr2 --patch
https://issues.apache.org/jira/secure/attachment/12945103/HIVE-20772.02.patch
--jira HIVE-20772
Build timed out (after 500 minutes). Marking the build as aborted.
{noformat}
no test reports... not sure what's up with ptest
> record per-task CPU counters in LLAP
>
>
> Key: HIVE-20772
> URL: https://issues.apache.org/jira/browse/HIVE-20772
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20772.01.patch, HIVE-20772.02.patch,
> HIVE-20772.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20787) MapJoinBytesTableContainer dummyRow case doesn't handle reuse
[ https://issues.apache.org/jira/browse/HIVE-20787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20787: Attachment: HIVE-20787.patch > MapJoinBytesTableContainer dummyRow case doesn't handle reuse > - > > Key: HIVE-20787 > URL: https://issues.apache.org/jira/browse/HIVE-20787 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20787.patch > > > Discovered while investigating some (probably) unrelated issue. > MapJoinBytesTableContainer was not intended to be reused, but it looks like > some code might reuse it. If that happens, the dummyRow case will not work > correctly (dummyRow is cleared on first(), so another call to first() will > behave differently). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20787) MapJoinBytesTableContainer dummyRow case doesn't handle reuse
[ https://issues.apache.org/jira/browse/HIVE-20787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20787: Status: Patch Available (was: Open) [~teddy.choi] [~prasanth_j] can you take a look? > MapJoinBytesTableContainer dummyRow case doesn't handle reuse > - > > Key: HIVE-20787 > URL: https://issues.apache.org/jira/browse/HIVE-20787 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20787.patch > > > Discovered while investigating some (probably) unrelated issue. > MapJoinBytesTableContainer was not intended to be reused, but it looks like > some code might reuse it. If that happens, the dummyRow case will not work > correctly (dummyRow is cleared on first(), so another call to first() will > behave differently). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20787) MapJoinBytesTableContainer dummyRow case doesn't handle reuse
[ https://issues.apache.org/jira/browse/HIVE-20787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-20787: --- > MapJoinBytesTableContainer dummyRow case doesn't handle reuse > - > > Key: HIVE-20787 > URL: https://issues.apache.org/jira/browse/HIVE-20787 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > > Discovered while investigating some (probably) unrelated issue. > MapJoinBytesTableContainer was not intended to be reused, but it looks like > some code might reuse it. If that happens, the dummyRow case will not work > correctly (dummyRow is cleared on first(), so another call to first() will > behave differently). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20765) fetch partitions for txn stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20765: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the reviews! [~ashutoshc] the MM tables are not present on branch-3 so this is not an issue. If needed, the new API can indeed just be replaced with List parts = getPartitionsByNames(catName, dbName, tblName, partNames); > fetch partitions for txn stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20765.01.patch, HIVE-20765.01.patch, > HIVE-20765.02.patch, HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20772) record per-task CPU counters in LLAP
[ https://issues.apache.org/jira/browse/HIVE-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20772: Attachment: HIVE-20772.02.patch > record per-task CPU counters in LLAP > > > Key: HIVE-20772 > URL: https://issues.apache.org/jira/browse/HIVE-20772 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20772.01.patch, HIVE-20772.02.patch, > HIVE-20772.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20765) fetch partitions for txn stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657586#comment-16657586 ] Sergey Shelukhin commented on HIVE-20765: - Well we use DN, so presumably the DN's version comes as a package deal. However I'm not sure how it manages to use one version of the interface here and another there (since other tests hitting this code path pass)... might be something messed up with the classpath for this test. Or it might be that the rest of Hive works due to luck because jars are always in the correct order/setup, and so we never run into this problem, and this test exposes it :) > fetch partitions for txn stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.01.patch, HIVE-20765.01.patch, > HIVE-20765.02.patch, HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20765) fetch partitions for txn stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657580#comment-16657580 ] Sergey Shelukhin edited comment on HIVE-20765 at 10/20/18 12:00 AM: According to Oracle docs JDO2 Query does not implement AutoCloseable: https://docs.oracle.com/cd/E24329_01/apirefs.1211/e24397/index.html?javax/jdo/Query.html This is another example for JDO3... https://db.apache.org/jdo/api30/apidocs/javax/jdo/Query.html However, the one we use during compile appears to come from DN javax.jdo 3.2 (?!!), that does (org/datanucleus/javax.jdo/3.2.0-m3/javax.jdo-3.2.0-m3.jar I've no idea why DN uses the same package name or what 3.2 is... seems like that's what we are running into. was (Author: sershe): According to Oracle docs JDO2 Query does not implement AutoCloseable: https://docs.oracle.com/cd/E24329_01/apirefs.1211/e24397/index.html?javax/jdo/Query.html This is another example for JDO3... https://db.apache.org/jdo/api30/apidocs/javax/jdo/Query.html However, the one we use during compile appears to come from DN javax.jdo, that does. I've no idea why DN uses the same package name... seems like that's what we are running into. > fetch partitions for txn stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.01.patch, HIVE-20765.01.patch, > HIVE-20765.02.patch, HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20765) fetch partitions for txn stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657580#comment-16657580 ] Sergey Shelukhin commented on HIVE-20765: - According to Oracle docs JDO2 Query does not implement AutoCloseable: https://docs.oracle.com/cd/E24329_01/apirefs.1211/e24397/index.html?javax/jdo/Query.html This is another example for JDO3... https://db.apache.org/jdo/api30/apidocs/javax/jdo/Query.html However, the one we use during compile appears to come from DN javax.jdo, that does. I've no idea why DN uses the same package name... seems like that's what we are running into. > fetch partitions for txn stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.01.patch, HIVE-20765.01.patch, > HIVE-20765.02.patch, HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20765) fetch partitions for txn stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657557#comment-16657557 ] Sergey Shelukhin edited comment on HIVE-20765 at 10/19/18 11:42 PM: Replacing with closeAll fixes this issue. It should still work with close... this can be investigated later ;) Seems like other codepaths using this call closeAll. was (Author: sershe): Replacing with closeAll fixes this issue > fetch partitions for txn stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.01.patch, HIVE-20765.01.patch, > HIVE-20765.02.patch, HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20765) fetch partitions for txn stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20765: Attachment: HIVE-20765.02.patch > fetch partitions for txn stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.01.patch, HIVE-20765.01.patch, > HIVE-20765.02.patch, HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20765) fetch partitions for txn stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657557#comment-16657557 ] Sergey Shelukhin commented on HIVE-20765: - Replacing with closeAll fixes this issue > fetch partitions for txn stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.01.patch, HIVE-20765.01.patch, > HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20765) fetch partitions for txn stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657553#comment-16657553 ] Sergey Shelukhin commented on HIVE-20765: - Apply this patch and run the test (TestTxnCommands2.testMultiInsertStatement). Yeah it's really bizarre, I tried replacing auto-close with an explicit call and get the same result. Not sure how that is possible... picking up a wrong jar somewhere in this test? I'd assume other tests call this method fine. > fetch partitions for txn stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.01.patch, HIVE-20765.01.patch, > HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20765) fetch partitions for txn stats validation in get_aggr_stats with one call
[
https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657533#comment-16657533
]
Sergey Shelukhin commented on HIVE-20765:
-
Failure is related, but weird... {noformat}
2018-10-19T15:52:41,324 ERROR [main] metastore.RetryingHMSHandler:
java.lang.NoSuchMethodError: javax.jdo.Query.close()V
at
org.apache.hadoop.hive.metastore.PartitionProjectionEvaluator.setSingleValuedFields(PartitionProjectionEvaluator.java:456)
at
org.apache.hadoop.hive.metastore.PartitionProjectionEvaluator.getPartitionsUsingProjectionList(PartitionProjectionEvaluator.java:358)
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql$3.run(MetaStoreDirectSql.java:635)
at
org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:73)
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsUsingProjectionAndFilterSpec(MetaStoreDirectSql.java:632)
at
org.apache.hadoop.hive.metastore.ObjectStore$13.getSqlResult(ObjectStore.java:4115)
at
org.apache.hadoop.hive.metastore.ObjectStore$13.getSqlResult(ObjectStore.java:4080)
at
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3759)
at
org.apache.hadoop.hive.metastore.ObjectStore.getPartitionSpecsByFilterAndProjection(ObjectStore.java:4165)
at
org.apache.hadoop.hive.metastore.ObjectStore.get_aggr_stats_for(ObjectStore.java:9066)
{noformat}
[~vihangk1] have you ever seen this trying to use the new API?
I will look into this a little bit and might just switch to fetching partitions
by names the old way for expediency.
> fetch partitions for txn stats validation in get_aggr_stats with one call
> -
>
> Key: HIVE-20765
> URL: https://issues.apache.org/jira/browse/HIVE-20765
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20765.01.patch, HIVE-20765.01.patch,
> HIVE-20765.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20772) record per-task CPU counters in LLAP
[ https://issues.apache.org/jira/browse/HIVE-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657446#comment-16657446 ] Sergey Shelukhin commented on HIVE-20772: - Fixed an issue where the wrapper callable from the IO is not being handled correctly by Stats thread pool. > record per-task CPU counters in LLAP > > > Key: HIVE-20772 > URL: https://issues.apache.org/jira/browse/HIVE-20772 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20772.01.patch, HIVE-20772.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20772) record per-task CPU counters in LLAP
[ https://issues.apache.org/jira/browse/HIVE-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20772: Attachment: HIVE-20772.01.patch > record per-task CPU counters in LLAP > > > Key: HIVE-20772 > URL: https://issues.apache.org/jira/browse/HIVE-20772 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20772.01.patch, HIVE-20772.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20765) fetch partitions for txn stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20765: Attachment: HIVE-20765.01.patch > fetch partitions for txn stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.01.patch, HIVE-20765.01.patch, > HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20765) fetch partitions for txn stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657300#comment-16657300 ] Sergey Shelukhin commented on HIVE-20765: - Some weird typo made it into the patch.. > fetch partitions for txn stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.01.patch, HIVE-20765.01.patch, > HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[
https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-20321:
Fix Version/s: 4.0.0
> Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1
> CacheLine
> -
>
> Key: HIVE-20321
> URL: https://issues.apache.org/jira/browse/HIVE-20321
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Matt McCline
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20321.01.patch, HIVE-20321.02.patch,
> HIVE-20321.03.patch, HIVE-20321.04.patch, HIVE-20321.05.patch,
> HIVE-20321.06.patch
>
>
> With a full sized LLAP instance, the memory size of the VectorHashKeyWrapper
> is bigger than the low Xmx JVMs.
> {code}
> * 64-bit VM: **
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper object internals:
> OFFSET SIZE
> TYPE DESCRIPTION VALUE
> 016
> (object header) N/A
> 16 4
> int VectorHashKeyWrapper.hashcodeN/A
> 20 4
> (alignment/padding gap)
> 24 8
> long[] VectorHashKeyWrapper.longValues N/A
> 32 8
> double[] VectorHashKeyWrapper.doubleValuesN/A
> 40 8
> byte[][] VectorHashKeyWrapper.byteValues N/A
> 48 8
>int[] VectorHashKeyWrapper.byteStarts N/A
> 56 8
>int[] VectorHashKeyWrapper.byteLengths N/A
> 64 8
> org.apache.hadoop.hive.serde2.io.HiveDecimalWritable[]
> VectorHashKeyWrapper.decimalValues N/A
> 72 8
> java.sql.Timestamp[] VectorHashKeyWrapper.timestampValues N/A
> 80 8
> org.apache.hadoop.hive.common.type.HiveIntervalDayTime[]
> VectorHashKeyWrapper.intervalDayTimeValues N/A
> 88 8
> boolean[] VectorHashKeyWrapper.isNull N/A
> 96 8
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper.HashContext
> VectorHashKeyWrapper.hashCtx N/A
> Instance size: 104 bytes
> Space losses: 4 bytes internal + 0 bytes external = 4 bytes total
> {code}
> Pulling this up to a parent class allows for this to be cut down to 32 bytes
> for the single column case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20765) fetch partitions for txn stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656158#comment-16656158 ] Sergey Shelukhin commented on HIVE-20765: - A tiny fix. Looks like the real failure was obscured by recompilation... > fetch partitions for txn stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.01.patch, HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20765) fetch partitions for txn stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20765: Attachment: HIVE-20765.01.patch > fetch partitions for txn stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.01.patch, HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20765) fetch partitions for txn stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656101#comment-16656101 ] Sergey Shelukhin commented on HIVE-20765: - Looking into test failures... > fetch partitions for txn stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20774) use less calls for GetTables operation
[ https://issues.apache.org/jira/browse/HIVE-20774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656013#comment-16656013 ] Sergey Shelukhin edited comment on HIVE-20774 at 10/18/18 11:22 PM: Actually, nm, this just undoes HIVE-19432. I'm not sure what the logic behind HIVE-19432 though, with many DBs 100s of separate calls would seemingly be much slower than one call. We need to see how this would affect the original case for HIVE-19432 was (Author: sershe): Actually, nm, this just undoes HIVE-19432. I'm not sure what the logic behind HIVE-19432 though, with many DBs 100s of separate calls would seemingly be much slower than one call. > use less calls for GetTables operation > -- > > Key: HIVE-20774 > URL: https://issues.apache.org/jira/browse/HIVE-20774 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20774.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20774) use less calls for GetTables operation
[ https://issues.apache.org/jira/browse/HIVE-20774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656013#comment-16656013 ] Sergey Shelukhin commented on HIVE-20774: - Actually, nm, this just undoes HIVE-19432. I'm not sure what the logic behind HIVE-19432 though, with many DBs 100s of separate calls would seemingly be much slower than one call. > use less calls for GetTables operation > -- > > Key: HIVE-20774 > URL: https://issues.apache.org/jira/browse/HIVE-20774 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20774.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19432) HIVE-7575: GetTablesOperation is too slow if the hive has too many databases and tables
[
https://issues.apache.org/jira/browse/HIVE-19432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656012#comment-16656012
]
Sergey Shelukhin commented on HIVE-19432:
-
Hmm... [~Rajkumar Singh] [~ashutoshc] what kind of testing was done for this?
I'm looking at a log where very large number of DBs is present, and with
network/etc. overhead for each individual call per database, instead of using
the pattern, it takes 10+ minutes to fetch all tables. Seems like a single
patterned call, before this patch, would have been faster.
> HIVE-7575: GetTablesOperation is too slow if the hive has too many databases
> and tables
> ---
>
> Key: HIVE-19432
> URL: https://issues.apache.org/jira/browse/HIVE-19432
> Project: Hive
> Issue Type: Improvement
> Components: Hive, HiveServer2
>Affects Versions: 2.2.0
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-19432.01.patch, HIVE-19432.01.patch,
> HIVE-19432.patch
>
>
> GetTablesOperation is too slow since it does not check for the authorization
> for databases and try pulling all the tables from all the databases using
> getTableMeta. for operation like follows
> {code}
> con.getMetaData().getTables("", "", "%", new String[] \{ "TABLE", "VIEW" });
> {code}
> build the getTableMeta call with wildcard *
> {code}
> metastore.HiveMetaStore: 8: get_table_metas : db=* tbl=*
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20774) use less calls for GetTables operation
[ https://issues.apache.org/jira/browse/HIVE-20774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656000#comment-16656000 ] Sergey Shelukhin edited comment on HIVE-20774 at 10/18/18 10:57 PM: Master patch. The change looks too simple and logical to actually be a real Hive services fix... but it looks like it should work from the examination of the code. We'll see if any tests break. [~thejas] can you take a look ;) I might create a separate patch for branch-1 where the fix might be more involved. was (Author: sershe): Master patch. The change looks too simple and logical to actually be a real Hive services fix... but it looks like it should work from the examination of the code. We'll see if any tests break. [~thejas] can you take a look ;) I might create a separate patch for branch-1 where the fix might be mire involved. > use less calls for GetTables operation > -- > > Key: HIVE-20774 > URL: https://issues.apache.org/jira/browse/HIVE-20774 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20774.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20774) use less calls for GetTables operation
[ https://issues.apache.org/jira/browse/HIVE-20774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656000#comment-16656000 ] Sergey Shelukhin commented on HIVE-20774: - Master patch. The change looks too simple and logical to actually be a real Hive services fix... but it looks like it should work from the examination of the code. We'll see if any tests break. [~thejas] can you take a look ;) I might create a separate patch for branch-1 where the fix might be mire involved. > use less calls for GetTables operation > -- > > Key: HIVE-20774 > URL: https://issues.apache.org/jira/browse/HIVE-20774 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20774.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20774) use less calls for GetTables operation
[ https://issues.apache.org/jira/browse/HIVE-20774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20774: Attachment: HIVE-20774.patch > use less calls for GetTables operation > -- > > Key: HIVE-20774 > URL: https://issues.apache.org/jira/browse/HIVE-20774 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20774.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20774) use less calls for GetTables operation
[ https://issues.apache.org/jira/browse/HIVE-20774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-20774: --- > use less calls for GetTables operation > -- > > Key: HIVE-20774 > URL: https://issues.apache.org/jira/browse/HIVE-20774 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20756) Disable SARG leaf creation for date column until ORC-135
[ https://issues.apache.org/jira/browse/HIVE-20756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20756: Resolution: Fixed Fix Version/s: 1.3.0 Status: Resolved (was: Patch Available) Committed to branch-1. Thanks for the patch! > Disable SARG leaf creation for date column until ORC-135 > > > Key: HIVE-20756 > URL: https://issues.apache.org/jira/browse/HIVE-20756 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Chiran Ravani >Assignee: Chiran Ravani >Priority: Major > Fix For: 1.3.0 > > Attachments: HIVE-20756.01-branch-1.patch, > HIVE-20756.01-branch-1.patch, HIVE-20756.1.patch > > > Until ORC-135 is committed and orc version is updated in hive, disable SARG > creation for date columns in hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20772) record per-task CPU counters in LLAP
[
https://issues.apache.org/jira/browse/HIVE-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655832#comment-16655832
]
Sergey Shelukhin commented on HIVE-20772:
-
Hmm, I was assuming based on other counters there (FS ones) that this object is
scoped per task... e.g. {noformat} tezCounters =
taskRunner2Callable.addAndGetTezCounter(FileSystemCounter.class.getName());
} else if (actualCallable instanceof TezCounterSource) { {noformat}
> record per-task CPU counters in LLAP
>
>
> Key: HIVE-20772
> URL: https://issues.apache.org/jira/browse/HIVE-20772
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20772.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20772) record per-task CPU counters in LLAP
[ https://issues.apache.org/jira/browse/HIVE-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20772: Status: Patch Available (was: Open) > record per-task CPU counters in LLAP > > > Key: HIVE-20772 > URL: https://issues.apache.org/jira/browse/HIVE-20772 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20772.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20772) record per-task CPU counters in LLAP
[ https://issues.apache.org/jira/browse/HIVE-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655820#comment-16655820 ] Sergey Shelukhin commented on HIVE-20772: - A quick patch... I'd probably need to run it in a cluster to make sure it works correctly in a realistic deployment. > record per-task CPU counters in LLAP > > > Key: HIVE-20772 > URL: https://issues.apache.org/jira/browse/HIVE-20772 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20772.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20772) record per-task CPU counters in LLAP
[ https://issues.apache.org/jira/browse/HIVE-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655821#comment-16655821 ] Sergey Shelukhin commented on HIVE-20772: - [~prasanth_j] can you take a look? > record per-task CPU counters in LLAP > > > Key: HIVE-20772 > URL: https://issues.apache.org/jira/browse/HIVE-20772 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20772.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20772) record per-task CPU counters in LLAP
[ https://issues.apache.org/jira/browse/HIVE-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20772: Attachment: HIVE-20772.patch > record per-task CPU counters in LLAP > > > Key: HIVE-20772 > URL: https://issues.apache.org/jira/browse/HIVE-20772 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20772.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20772) record per-task CPU counters in LLAP
[ https://issues.apache.org/jira/browse/HIVE-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-20772: --- > record per-task CPU counters in LLAP > > > Key: HIVE-20772 > URL: https://issues.apache.org/jira/browse/HIVE-20772 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20752) In case of LLAP start failure add info how to find YARN logs
[ https://issues.apache.org/jira/browse/HIVE-20752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655770#comment-16655770 ] Sergey Shelukhin commented on HIVE-20752: - +1 > In case of LLAP start failure add info how to find YARN logs > > > Key: HIVE-20752 > URL: https://issues.apache.org/jira/browse/HIVE-20752 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20752.01.patch, HIVE-20752.02.patch, > HIVE-20752.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18908) FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18908: Fix Version/s: 4.0.0 > FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 4.0.0 > > Attachments: FULL OUTER MapJoin Code Changes.docx, > HIVE-18908.01.patch, HIVE-18908.02.patch, HIVE-18908.03.patch, > HIVE-18908.04.patch, HIVE-18908.05.patch, HIVE-18908.06.patch, > HIVE-18908.08.patch, HIVE-18908.09.patch, HIVE-18908.091.patch, > HIVE-18908.092.patch, HIVE-18908.093.patch, HIVE-18908.096.patch, > HIVE-18908.097.patch, HIVE-18908.098.patch, HIVE-18908.099.patch, > HIVE-18908.0991.patch, HIVE-18908.0992.patch, HIVE-18908.0993.patch, > HIVE-18908.0994.patch, HIVE-18908.0995.patch, HIVE-18908.0996.patch, > HIVE-18908.0997.patch, HIVE-18908.0998.patch, HIVE-18908.0999.patch, > HIVE-18908.09991.patch, HIVE-18908.09992.patch, HIVE-18908.09993.patch, > HIVE-18908.09994.patch, HIVE-18908.09995.patch, HIVE-18908.09996.patch, > HIVE-18908.09997.patch, HIVE-18908.09998.patch, JOIN to MAPJOIN > Transformation.pdf, SHARED-MEMORY FULL OUTER MapJoin.pdf > > > Currently, we do not support FULL OUTER JOIN in MapJoin. > Rough TPC-DS timings run on laptop: > (NOTE: Query 51 has PTF as a bigger serial portion -- Amdahl's law at play) > FULL OUTER MapJoin OFF = MergeJoin > Query 51: > o Vectorization OFF > • FULL OUTER MapJoin OFF: 4:30 minutes > • FULL OUTER MapJoin ON: 4:37 minutes > o Vectorization ON > • FULL OUTER MapJoin OFF: 2:35 minutes > • FULL OUTER MapJoin ON: 1:47 minutes > Query 97: > o Vectorization OFF > • FULL OUTER MapJoin OFF: 2:37 minutes > • FULL OUTER MapJoin ON: 2:42 minutes > o Vectorization ON > • FULL OUTER MapJoin OFF: 1:17 minutes > • FULL OUTER MapJoin ON: 0:06 minutes > FULL OUTER Join 10,000,000 rows against 323,910 small table keys > o Vectorization ON > • FULL OUTER MapJoin OFF: 14:56 minutes > • FULL OUTER MapJoin ON: 1:45 minutes > FULL OUTER Join 10,000,000 rows against 1,000 small table keys > o Vectorization ON > • FULL OUTER MapJoin OFF: 12:37 minutes > • FULL OUTER MapJoin ON: 1:38 minutes > Hopefully, someone will do large scale cluster testing. > [DynamicPartitionedHashJoin] MapJoin should scale dramatically better than > [Sort] MergeJoin reduce-shuffle. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20763) Add google cloud storage (gs) to the exim uri schema whitelist
[ https://issues.apache.org/jira/browse/HIVE-20763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654282#comment-16654282 ] Sergey Shelukhin commented on HIVE-20763: - +1 > Add google cloud storage (gs) to the exim uri schema whitelist > -- > > Key: HIVE-20763 > URL: https://issues.apache.org/jira/browse/HIVE-20763 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20763.01.patch > > > import/export is enabled for s3a by default. Ideally this list should include > other cloud storage options. This Jira adds Google Storage to the list. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20703) Put dynamic sort partition optimization under cost based decision
[ https://issues.apache.org/jira/browse/HIVE-20703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654271#comment-16654271 ] Sergey Shelukhin commented on HIVE-20703: - Can hive.tez.container.size be used for now? Perhaps adding an explicit default in this case if it's not set. > Put dynamic sort partition optimization under cost based decision > - > > Key: HIVE-20703 > URL: https://issues.apache.org/jira/browse/HIVE-20703 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20703.1.patch, HIVE-20703.2.patch, > HIVE-20703.3.patch, HIVE-20703.4.patch, HIVE-20703.5.patch, > HIVE-20703.6.patch, HIVE-20703.7.patch, HIVE-20703.8.patch, HIVE-20703.9.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20765) fetch partitions for txn stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654257#comment-16654257 ] Sergey Shelukhin commented on HIVE-20765: - We can just get full partitions by name on branch-3 > fetch partitions for txn stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20765) fetch partitions for txn stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20765: Summary: fetch partitions for txn stats validation in get_aggr_stats with one call (was: fetch partitions for stats validation in get_aggr_stats with one call) > fetch partitions for txn stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20765) fetch partitions for stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20765: Attachment: (was: HIVE-20765.patch) > fetch partitions for stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20765) fetch partitions for stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20765: Attachment: HIVE-20765.patch > fetch partitions for stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20765) fetch partitions for stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654182#comment-16654182 ] Sergey Shelukhin commented on HIVE-20765: - [~ashutoshc] can you take a look? > fetch partitions for stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20765) fetch partitions for stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20765: Status: Patch Available (was: Open) Small patch... trying to use the new projection mechanism for now. > fetch partitions for stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
