[jira] [Updated] (HIVE-20203) Arrow SerDe leaks a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HIVE-20203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-20203: -- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) > Arrow SerDe leaks a DirectByteBuffer > > > Key: HIVE-20203 > URL: https://issues.apache.org/jira/browse/HIVE-20203 > Project: Hive > Issue Type: Bug >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Blocker > Fix For: 4.0.0 > > Attachments: HIVE-20203.1.patch, HIVE-20203.2.patch, > HIVE-20203.3.patch, HIVE-20203.4.patch > > > ArrowColumnarBatchSerDe allocates an arrow NullableMapVector for each task > that uses the serde. > The vector is a DirectByteBuffer allocated from Arrow's off-heap buffer pool. > This buffer is never closed and leaks about 1K of physical memory for each > task. > This patch does three things: > # Ensure the buffer is closed when the RecordWriter for the task is closed. > # Adds per-task memory accounting by assigning a ChildAllocator to each task > from the RootAllocator. > # Enforces that the ChildAllocator for a task has released all memory > assigned to it, when the task is completed. > The patch assumes that close() is always called on the RecordWriter when a > task is finished (even if there is a failure during task execution). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20203) Arrow SerDe leaks a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HIVE-20203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-20203: Status: Open (was: Patch Available) > Arrow SerDe leaks a DirectByteBuffer > > > Key: HIVE-20203 > URL: https://issues.apache.org/jira/browse/HIVE-20203 > Project: Hive > Issue Type: Bug >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Blocker > Attachments: HIVE-20203.1.patch, HIVE-20203.2.patch, > HIVE-20203.3.patch, HIVE-20203.4.patch > > > ArrowColumnarBatchSerDe allocates an arrow NullableMapVector for each task > that uses the serde. > The vector is a DirectByteBuffer allocated from Arrow's off-heap buffer pool. > This buffer is never closed and leaks about 1K of physical memory for each > task. > This patch does three things: > # Ensure the buffer is closed when the RecordWriter for the task is closed. > # Adds per-task memory accounting by assigning a ChildAllocator to each task > from the RootAllocator. > # Enforces that the ChildAllocator for a task has released all memory > assigned to it, when the task is completed. > The patch assumes that close() is always called on the RecordWriter when a > task is finished (even if there is a failure during task execution). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20203) Arrow SerDe leaks a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HIVE-20203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-20203: Attachment: HIVE-20203.4.patch > Arrow SerDe leaks a DirectByteBuffer > > > Key: HIVE-20203 > URL: https://issues.apache.org/jira/browse/HIVE-20203 > Project: Hive > Issue Type: Bug >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Blocker > Attachments: HIVE-20203.1.patch, HIVE-20203.2.patch, > HIVE-20203.3.patch, HIVE-20203.4.patch > > > ArrowColumnarBatchSerDe allocates an arrow NullableMapVector for each task > that uses the serde. > The vector is a DirectByteBuffer allocated from Arrow's off-heap buffer pool. > This buffer is never closed and leaks about 1K of physical memory for each > task. > This patch does three things: > # Ensure the buffer is closed when the RecordWriter for the task is closed. > # Adds per-task memory accounting by assigning a ChildAllocator to each task > from the RootAllocator. > # Enforces that the ChildAllocator for a task has released all memory > assigned to it, when the task is completed. > The patch assumes that close() is always called on the RecordWriter when a > task is finished (even if there is a failure during task execution). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20203) Arrow SerDe leaks a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HIVE-20203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-20203: Status: Patch Available (was: Open) > Arrow SerDe leaks a DirectByteBuffer > > > Key: HIVE-20203 > URL: https://issues.apache.org/jira/browse/HIVE-20203 > Project: Hive > Issue Type: Bug >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Blocker > Attachments: HIVE-20203.1.patch, HIVE-20203.2.patch, > HIVE-20203.3.patch, HIVE-20203.4.patch > > > ArrowColumnarBatchSerDe allocates an arrow NullableMapVector for each task > that uses the serde. > The vector is a DirectByteBuffer allocated from Arrow's off-heap buffer pool. > This buffer is never closed and leaks about 1K of physical memory for each > task. > This patch does three things: > # Ensure the buffer is closed when the RecordWriter for the task is closed. > # Adds per-task memory accounting by assigning a ChildAllocator to each task > from the RootAllocator. > # Enforces that the ChildAllocator for a task has released all memory > assigned to it, when the task is completed. > The patch assumes that close() is always called on the RecordWriter when a > task is finished (even if there is a failure during task execution). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20203) Arrow SerDe leaks a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HIVE-20203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-20203: Status: Patch Available (was: Open) > Arrow SerDe leaks a DirectByteBuffer > > > Key: HIVE-20203 > URL: https://issues.apache.org/jira/browse/HIVE-20203 > Project: Hive > Issue Type: Bug >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Blocker > Attachments: HIVE-20203.1.patch, HIVE-20203.2.patch, > HIVE-20203.3.patch > > > ArrowColumnarBatchSerDe allocates an arrow NullableMapVector for each task > that uses the serde. > The vector is a DirectByteBuffer allocated from Arrow's off-heap buffer pool. > This buffer is never closed and leaks about 1K of physical memory for each > task. > This patch does three things: > # Ensure the buffer is closed when the RecordWriter for the task is closed. > # Adds per-task memory accounting by assigning a ChildAllocator to each task > from the RootAllocator. > # Enforces that the ChildAllocator for a task has released all memory > assigned to it, when the task is completed. > The patch assumes that close() is always called on the RecordWriter when a > task is finished (even if there is a failure during task execution). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20203) Arrow SerDe leaks a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HIVE-20203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-20203: Attachment: HIVE-20203.3.patch > Arrow SerDe leaks a DirectByteBuffer > > > Key: HIVE-20203 > URL: https://issues.apache.org/jira/browse/HIVE-20203 > Project: Hive > Issue Type: Bug >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Blocker > Attachments: HIVE-20203.1.patch, HIVE-20203.2.patch, > HIVE-20203.3.patch > > > ArrowColumnarBatchSerDe allocates an arrow NullableMapVector for each task > that uses the serde. > The vector is a DirectByteBuffer allocated from Arrow's off-heap buffer pool. > This buffer is never closed and leaks about 1K of physical memory for each > task. > This patch does three things: > # Ensure the buffer is closed when the RecordWriter for the task is closed. > # Adds per-task memory accounting by assigning a ChildAllocator to each task > from the RootAllocator. > # Enforces that the ChildAllocator for a task has released all memory > assigned to it, when the task is completed. > The patch assumes that close() is always called on the RecordWriter when a > task is finished (even if there is a failure during task execution). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20203) Arrow SerDe leaks a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HIVE-20203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-20203: Status: Open (was: Patch Available) > Arrow SerDe leaks a DirectByteBuffer > > > Key: HIVE-20203 > URL: https://issues.apache.org/jira/browse/HIVE-20203 > Project: Hive > Issue Type: Bug >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Blocker > Attachments: HIVE-20203.1.patch, HIVE-20203.2.patch > > > ArrowColumnarBatchSerDe allocates an arrow NullableMapVector for each task > that uses the serde. > The vector is a DirectByteBuffer allocated from Arrow's off-heap buffer pool. > This buffer is never closed and leaks about 1K of physical memory for each > task. > This patch does three things: > # Ensure the buffer is closed when the RecordWriter for the task is closed. > # Adds per-task memory accounting by assigning a ChildAllocator to each task > from the RootAllocator. > # Enforces that the ChildAllocator for a task has released all memory > assigned to it, when the task is completed. > The patch assumes that close() is always called on the RecordWriter when a > task is finished (even if there is a failure during task execution). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20203) Arrow SerDe leaks a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HIVE-20203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-20203: Status: Open (was: Patch Available) Made an error in the first patch. Line in Serializer.java should have been {code:java} rootVector = NullableMapVector.empty(null, allocator);{code} instead of {code} rootVector = NullableMapVector.empty(null, serDe.rootAllocator); {code} so it uses the ChildAllocator. > Arrow SerDe leaks a DirectByteBuffer > > > Key: HIVE-20203 > URL: https://issues.apache.org/jira/browse/HIVE-20203 > Project: Hive > Issue Type: Bug >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Blocker > Attachments: HIVE-20203.1.patch, HIVE-20203.2.patch > > > ArrowColumnarBatchSerDe allocates an arrow NullableMapVector for each task > that uses the serde. > The vector is a DirectByteBuffer allocated from Arrow's off-heap buffer pool. > This buffer is never closed and leaks about 1K of physical memory for each > task. > This patch does three things: > # Ensure the buffer is closed when the RecordWriter for the task is closed. > # Adds per-task memory accounting by assigning a ChildAllocator to each task > from the RootAllocator. > # Enforces that the ChildAllocator for a task has released all memory > assigned to it, when the task is completed. > The patch assumes that close() is always called on the RecordWriter when a > task is finished (even if there is a failure during task execution). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20203) Arrow SerDe leaks a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HIVE-20203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-20203: Attachment: HIVE-20203.2.patch > Arrow SerDe leaks a DirectByteBuffer > > > Key: HIVE-20203 > URL: https://issues.apache.org/jira/browse/HIVE-20203 > Project: Hive > Issue Type: Bug >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Blocker > Attachments: HIVE-20203.1.patch, HIVE-20203.2.patch > > > ArrowColumnarBatchSerDe allocates an arrow NullableMapVector for each task > that uses the serde. > The vector is a DirectByteBuffer allocated from Arrow's off-heap buffer pool. > This buffer is never closed and leaks about 1K of physical memory for each > task. > This patch does three things: > # Ensure the buffer is closed when the RecordWriter for the task is closed. > # Adds per-task memory accounting by assigning a ChildAllocator to each task > from the RootAllocator. > # Enforces that the ChildAllocator for a task has released all memory > assigned to it, when the task is completed. > The patch assumes that close() is always called on the RecordWriter when a > task is finished (even if there is a failure during task execution). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20203) Arrow SerDe leaks a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HIVE-20203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-20203: Status: Patch Available (was: Open) > Arrow SerDe leaks a DirectByteBuffer > > > Key: HIVE-20203 > URL: https://issues.apache.org/jira/browse/HIVE-20203 > Project: Hive > Issue Type: Bug >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Blocker > Attachments: HIVE-20203.1.patch, HIVE-20203.2.patch > > > ArrowColumnarBatchSerDe allocates an arrow NullableMapVector for each task > that uses the serde. > The vector is a DirectByteBuffer allocated from Arrow's off-heap buffer pool. > This buffer is never closed and leaks about 1K of physical memory for each > task. > This patch does three things: > # Ensure the buffer is closed when the RecordWriter for the task is closed. > # Adds per-task memory accounting by assigning a ChildAllocator to each task > from the RootAllocator. > # Enforces that the ChildAllocator for a task has released all memory > assigned to it, when the task is completed. > The patch assumes that close() is always called on the RecordWriter when a > task is finished (even if there is a failure during task execution). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20203) Arrow SerDe leaks a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HIVE-20203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-20203: Description: ArrowColumnarBatchSerDe allocates an arrow NullableMapVector for each task that uses the serde. The vector is a DirectByteBuffer allocated from Arrow's off-heap buffer pool. This buffer is never closed and leaks about 1K of physical memory for each task. This patch does three things: # Ensure the buffer is closed when the RecordWriter for the task is closed. # Adds per-task memory accounting by assigning a ChildAllocator to each task from the RootAllocator. # Enforces that the ChildAllocator for a task has released all memory assigned to it, when the task is completed. The patch assumes that close() is always called on the RecordWriter when a task is finished (even if there is a failure during task execution). was: ArrowColumnarBatchSerDe allocates an arrow NullableMapVector for each task that uses the serde. The vector is a DirectByteBuffer allocated from Arrow's off-heap buffer pool. This buffer is never closed and leaks about 1K of physical memory for each task. This patch does three things: # Ensure the buffer is closed when the RecordWriter for the task is closed. # Adds per-task memory accounting by assigning a ChildAllocator to each task from the RootAllocator. # Enforces that the ChildAllocator for a task has released all memory assigned to it, when the task is completed. The patch assumes that close() is always called on the RecordWriter when a task is finished (even if their is a failure during task execution). > Arrow SerDe leaks a DirectByteBuffer > > > Key: HIVE-20203 > URL: https://issues.apache.org/jira/browse/HIVE-20203 > Project: Hive > Issue Type: Bug >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Blocker > Attachments: HIVE-20203.1.patch > > > ArrowColumnarBatchSerDe allocates an arrow NullableMapVector for each task > that uses the serde. > The vector is a DirectByteBuffer allocated from Arrow's off-heap buffer pool. > This buffer is never closed and leaks about 1K of physical memory for each > task. > This patch does three things: > # Ensure the buffer is closed when the RecordWriter for the task is closed. > # Adds per-task memory accounting by assigning a ChildAllocator to each task > from the RootAllocator. > # Enforces that the ChildAllocator for a task has released all memory > assigned to it, when the task is completed. > The patch assumes that close() is always called on the RecordWriter when a > task is finished (even if there is a failure during task execution). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20203) Arrow SerDe leaks a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HIVE-20203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-20203: Attachment: HIVE-20203.1.patch > Arrow SerDe leaks a DirectByteBuffer > > > Key: HIVE-20203 > URL: https://issues.apache.org/jira/browse/HIVE-20203 > Project: Hive > Issue Type: Bug >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Blocker > Attachments: HIVE-20203.1.patch > > > ArrowColumnarBatchSerDe allocates an arrow NullableMapVector for each task > that uses the serde. > The vector is a DirectByteBuffer allocated from Arrow's off-heap buffer pool. > This buffer is never closed and leaks about 1K of physical memory for each > task. > This patch does three things: > # Ensure the buffer is closed when the RecordWriter for the task is closed. > # Adds per-task memory accounting by assigning a ChildAllocator to each task > from the RootAllocator. > # Enforces that the ChildAllocator for a task has released all memory > assigned to it, when the task is completed. > The patch assumes that close() is always called on the RecordWriter when a > task is finished (even if their is a failure during task execution). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20203) Arrow SerDe leaks a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HIVE-20203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-20203: Status: Patch Available (was: Open) > Arrow SerDe leaks a DirectByteBuffer > > > Key: HIVE-20203 > URL: https://issues.apache.org/jira/browse/HIVE-20203 > Project: Hive > Issue Type: Bug >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Blocker > Attachments: HIVE-20203.1.patch > > > ArrowColumnarBatchSerDe allocates an arrow NullableMapVector for each task > that uses the serde. > The vector is a DirectByteBuffer allocated from Arrow's off-heap buffer pool. > This buffer is never closed and leaks about 1K of physical memory for each > task. > This patch does three things: > # Ensure the buffer is closed when the RecordWriter for the task is closed. > # Adds per-task memory accounting by assigning a ChildAllocator to each task > from the RootAllocator. > # Enforces that the ChildAllocator for a task has released all memory > assigned to it, when the task is completed. > The patch assumes that close() is always called on the RecordWriter when a > task is finished (even if their is a failure during task execution). -- This message was sent by Atlassian JIRA (v7.6.3#76005)