[jira] [Commented] (HIVE-24316) Upgrade ORC from 1.5.6 to 1.5.8 in branch-3.1
[
https://issues.apache.org/jira/browse/HIVE-24316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403893#comment-17403893
]
Sungwoo commented on HIVE-24316:
Hello,
It seems that with ORC-361, the use of MemoryManagerImpl in
LlapAwareMemoryManager is inconsistent.
Before merging ORC-361, LlapAwareMemoryManager sets its own totalMemoryPool and
MemoryManagerImpl accesses totalMemoryPool via getTotalMemoryPool(), so
everything is fine.
With ORC-361 merged, we have the following:
1. LlapAwareMemoryManager sets its own totalMemoryPool as a private field.
2. MemoryManagerImpl sets its own totalMemoryPool as a private field.
3. LlapAwareMemoryManager overrides getTotalMemoryPool() using its own
totalMemoryPool.
Now it is unclear whether or not getTotalMemoryPool() should be overridden.
Here are my thoughts on ORC-361:
1. Is MemoryManagerImpl intended to coordinate all threads writing to ORC files
inside a process (like LLAP Daemon)? Then is it necessary to create
LlapAwareMemoryManager as a ThreadLocal object? Why not just call
OrcFile.getStaticMemoryManager() to obtain the shared MemoryManagerImpl?
3. LlapAwareMemoryManager sets its own totalMemoryPool:
{code:java}
long memPerExecutor = LlapDaemonInfo.INSTANCE.getMemoryPerExecutor();
totalMemoryPool = (long) (memPerExecutor * maxLoad);
{code}
>From my understanding, this has no effect because MemoryManagerImpl sets its
>own totalMemoryPool.
Any comment would be appreciated.
> Upgrade ORC from 1.5.6 to 1.5.8 in branch-3.1
> -
>
> Key: HIVE-24316
> URL: https://issues.apache.org/jira/browse/HIVE-24316
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 3.1.3
>Reporter: Dongjoon Hyun
>Assignee: Dongjoon Hyun
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.1.3
>
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> This will bring eleven bug fixes.
> * ORC 1.5.7: [https://issues.apache.org/jira/projects/ORC/versions/12345702]
> * ORC 1.5.8: [https://issues.apache.org/jira/projects/ORC/versions/12346462]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24316) Upgrade ORC from 1.5.6 to 1.5.8 in branch-3.1
[ https://issues.apache.org/jira/browse/HIVE-24316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403902#comment-17403902 ] Dongjoon Hyun commented on HIVE-24316: -- cc [~omalley] > Upgrade ORC from 1.5.6 to 1.5.8 in branch-3.1 > - > > Key: HIVE-24316 > URL: https://issues.apache.org/jira/browse/HIVE-24316 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 3.1.3 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 3.1.3 > > Time Spent: 4h 50m > Remaining Estimate: 0h > > This will bring eleven bug fixes. > * ORC 1.5.7: [https://issues.apache.org/jira/projects/ORC/versions/12345702] > * ORC 1.5.8: [https://issues.apache.org/jira/projects/ORC/versions/12346462] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24316) Upgrade ORC from 1.5.6 to 1.5.8 in branch-3.1
[ https://issues.apache.org/jira/browse/HIVE-24316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403937#comment-17403937 ] Panagiotis Garefalakis commented on HIVE-24316: --- Hey [~glapark] thanks for bringing this up -- taking a look at MemoryManagerImpl looks like checkMemory() is the new method that determines if the scale has changed and since ORC-361 removed getTotalMemoryPool() calls from multiple places we are loosing the effect of controlling the memory pool. The intention behind LlapAwareMemoryManager was to have memory per executor instead of the entire heap since multiple writers are involved. An idea could be to restore getTotalMemoryPool calls where needed . > Upgrade ORC from 1.5.6 to 1.5.8 in branch-3.1 > - > > Key: HIVE-24316 > URL: https://issues.apache.org/jira/browse/HIVE-24316 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 3.1.3 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 3.1.3 > > Time Spent: 4h 50m > Remaining Estimate: 0h > > This will bring eleven bug fixes. > * ORC 1.5.7: [https://issues.apache.org/jira/projects/ORC/versions/12345702] > * ORC 1.5.8: [https://issues.apache.org/jira/projects/ORC/versions/12346462] -- This message was sent by Atlassian Jira (v8.3.4#803005)
