[jira] [Commented] (HIVE-20979) Fix memory leak in hive streaming
[
https://issues.apache.org/jira/browse/HIVE-20979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037270#comment-17037270
]
Alex Parrill commented on HIVE-20979:
-
This patch isn't in 3.1.1 or 3.1.2, despite what the fix versions says.
It's relatively easy to work around at least; just subclass whatever writer you
want to use, and override the {{close}} method to call {{fs.close()}} after
calling {{super.close()}}.
> Fix memory leak in hive streaming
> -
>
> Key: HIVE-20979
> URL: https://issues.apache.org/jira/browse/HIVE-20979
> Project: Hive
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0, 3.1.1
>
> Attachments: HIVE-20979.1.patch, HIVE-20979.1.patch,
> HIVE-20979.2.patch, HIVE-20979.3.patch, HIVE-20979.4.patch, HIVE-20979.5.patch
>
>
> {{1) HiveStreamingConnection.Builder#init() adds a shutdown hook handler via
> }}{{ShutdownHookManager.addShutdownHook but it is never removed which causes
> all the handlers to accumulate and hence a memory leak.}}
> 2) AbstractRecordWriter creates an instance of FileSystem but does not close
> it which in turn causes a leak due to accumulation in FileSystem$Cache#map
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-20979) Fix memory leak in hive streaming
[
https://issues.apache.org/jira/browse/HIVE-20979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16809868#comment-16809868
]
Gauthier Leonard commented on HIVE-20979:
-
It appears that the fixes are not present on the 3.1.1 release. Can I update
the *Fix Version/s* accordingly? I assume the fix will be in 3.1.2 as it is on
master?
> Fix memory leak in hive streaming
> -
>
> Key: HIVE-20979
> URL: https://issues.apache.org/jira/browse/HIVE-20979
> Project: Hive
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0, 3.1.1
>
> Attachments: HIVE-20979.1.patch, HIVE-20979.1.patch,
> HIVE-20979.2.patch, HIVE-20979.3.patch, HIVE-20979.4.patch, HIVE-20979.5.patch
>
>
> {{1) HiveStreamingConnection.Builder#init() adds a shutdown hook handler via
> }}{{ShutdownHookManager.addShutdownHook but it is never removed which causes
> all the handlers to accumulate and hence a memory leak.}}
> 2) AbstractRecordWriter creates an instance of FileSystem but does not close
> it which in turn causes a leak due to accumulation in FileSystem$Cache#map
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20979) Fix memory leak in hive streaming
[
https://issues.apache.org/jira/browse/HIVE-20979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16704283#comment-16704283
]
Hive QA commented on HIVE-20979:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12950117/HIVE-20979.5.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15644 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15124/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15124/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15124/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12950117 - PreCommit-HIVE-Build
> Fix memory leak in hive streaming
> -
>
> Key: HIVE-20979
> URL: https://issues.apache.org/jira/browse/HIVE-20979
> Project: Hive
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-20979.1.patch, HIVE-20979.1.patch,
> HIVE-20979.2.patch, HIVE-20979.3.patch, HIVE-20979.4.patch, HIVE-20979.5.patch
>
>
> {{1) HiveStreamingConnection.Builder#init() adds a shutdown hook handler via
> }}{{ShutdownHookManager.addShutdownHook but it is never removed which causes
> all the handlers to accumulate and hence a memory leak.}}
> 2) AbstractRecordWriter creates an instance of FileSystem but does not close
> it which in turn causes a leak due to accumulation in FileSystem$Cache#map
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20979) Fix memory leak in hive streaming
[
https://issues.apache.org/jira/browse/HIVE-20979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16704261#comment-16704261
]
Hive QA commented on HIVE-20979:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15124/dev-support/hive-personality.sh
|
| git revision | master / 8411ea5 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: streaming U: streaming |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15124/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix memory leak in hive streaming
> -
>
> Key: HIVE-20979
> URL: https://issues.apache.org/jira/browse/HIVE-20979
> Project: Hive
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-20979.1.patch, HIVE-20979.1.patch,
> HIVE-20979.2.patch, HIVE-20979.3.patch, HIVE-20979.4.patch, HIVE-20979.5.patch
>
>
> {{1) HiveStreamingConnection.Builder#init() adds a shutdown hook handler via
> }}{{ShutdownHookManager.addShutdownHook but it is never removed which causes
> all the handlers to accumulate and hence a memory leak.}}
> 2) AbstractRecordWriter creates an instance of FileSystem but does not close
> it which in turn causes a leak due to accumulation in FileSystem$Cache#map
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20979) Fix memory leak in hive streaming
[
https://issues.apache.org/jira/browse/HIVE-20979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16704218#comment-16704218
]
Shubham Chaurasia commented on HIVE-20979:
--
Thanks for the review [~prasanth_j] [~ewohlstadter] [~ashutoshc]
These test failures do not seem related. However, as you said [~prasanth_j],
will keep uploading patches until I get green.
> Fix memory leak in hive streaming
> -
>
> Key: HIVE-20979
> URL: https://issues.apache.org/jira/browse/HIVE-20979
> Project: Hive
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-20979.1.patch, HIVE-20979.1.patch,
> HIVE-20979.2.patch, HIVE-20979.3.patch, HIVE-20979.4.patch
>
>
> {{1) HiveStreamingConnection.Builder#init() adds a shutdown hook handler via
> }}{{ShutdownHookManager.addShutdownHook but it is never removed which causes
> all the handlers to accumulate and hence a memory leak.}}
> 2) AbstractRecordWriter creates an instance of FileSystem but does not close
> it which in turn causes a leak due to accumulation in FileSystem$Cache#map
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20979) Fix memory leak in hive streaming
[
https://issues.apache.org/jira/browse/HIVE-20979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703999#comment-16703999
]
Ashutosh Chauhan commented on HIVE-20979:
-
+1
> Fix memory leak in hive streaming
> -
>
> Key: HIVE-20979
> URL: https://issues.apache.org/jira/browse/HIVE-20979
> Project: Hive
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-20979.1.patch, HIVE-20979.1.patch,
> HIVE-20979.2.patch, HIVE-20979.3.patch, HIVE-20979.4.patch
>
>
> {{1) HiveStreamingConnection.Builder#init() adds a shutdown hook handler via
> }}{{ShutdownHookManager.addShutdownHook but it is never removed which causes
> all the handlers to accumulate and hence a memory leak.}}
> 2) AbstractRecordWriter creates an instance of FileSystem but does not close
> it which in turn causes a leak due to accumulation in FileSystem$Cache#map
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20979) Fix memory leak in hive streaming
[
https://issues.apache.org/jira/browse/HIVE-20979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703878#comment-16703878
]
Eric Wohlstadter commented on HIVE-20979:
-
[~ShubhamChaurasia]
lgtm
> Fix memory leak in hive streaming
> -
>
> Key: HIVE-20979
> URL: https://issues.apache.org/jira/browse/HIVE-20979
> Project: Hive
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-20979.1.patch, HIVE-20979.1.patch,
> HIVE-20979.2.patch, HIVE-20979.3.patch, HIVE-20979.4.patch
>
>
> {{1) HiveStreamingConnection.Builder#init() adds a shutdown hook handler via
> }}{{ShutdownHookManager.addShutdownHook but it is never removed which causes
> all the handlers to accumulate and hence a memory leak.}}
> 2) AbstractRecordWriter creates an instance of FileSystem but does not close
> it which in turn causes a leak due to accumulation in FileSystem$Cache#map
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20979) Fix memory leak in hive streaming
[
https://issues.apache.org/jira/browse/HIVE-20979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703509#comment-16703509
]
Hive QA commented on HIVE-20979:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12950002/HIVE-20979.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 46 failed/errored test(s), 15643 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=197)
[druidmini_masking.q,druidmini_joins.q,druid_timestamptz.q]
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterPartition
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterTable
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterTableCascade
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterViewParititon
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testColumnStatistics
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testComplexTable
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testComplexTypeApi
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testConcurrentMetastores
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testCreateAndGetTableWithDriver
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testCreateTableSettingId
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBLocationChange
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBOwner
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBOwnerChange
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabase
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabaseLocation
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabaseLocationWithPermissionProblems
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDropDatabaseCascadeMVMultiDB
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDropTable
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFilterLastPartition
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFilterSinglePartition
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFunctionWithResources
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetConfigValue
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetMetastoreUuid
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetPartitionsWithSpec
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetSchemaWithNoClassDefFoundError
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetTableObjects
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetUUIDInParallel
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testJDOPersistanceManagerCleanup
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testListPartitionNames
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testListPartitions
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testListPartitionsWihtLimitEnabled
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testNameMethods
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testPartition
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testPartitionFilter
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testRenamePartition
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testRetriableClientWithConnLifetime
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testSimpleFunction
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testSimpleTable
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testSimpleTypeApi
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testStatsFastTrivial
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testSynchronized
(batchI
[jira] [Commented] (HIVE-20979) Fix memory leak in hive streaming
[
https://issues.apache.org/jira/browse/HIVE-20979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703415#comment-16703415
]
Hive QA commented on HIVE-20979:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15116/dev-support/hive-personality.sh
|
| git revision | master / 3e98143 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: streaming U: streaming |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15116/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix memory leak in hive streaming
> -
>
> Key: HIVE-20979
> URL: https://issues.apache.org/jira/browse/HIVE-20979
> Project: Hive
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-20979.1.patch, HIVE-20979.1.patch,
> HIVE-20979.2.patch, HIVE-20979.3.patch, HIVE-20979.4.patch
>
>
> {{1) HiveStreamingConnection.Builder#init() adds a shutdown hook handler via
> }}{{ShutdownHookManager.addShutdownHook but it is never removed which causes
> all the handlers to accumulate and hence a memory leak.}}
> 2) AbstractRecordWriter creates an instance of FileSystem but does not close
> it which in turn causes a leak due to accumulation in FileSystem$Cache#map
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20979) Fix memory leak in hive streaming
[
https://issues.apache.org/jira/browse/HIVE-20979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703008#comment-16703008
]
Hive QA commented on HIVE-20979:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949940/HIVE-20979.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 46 failed/errored test(s), 15643 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=197)
[druidmini_masking.q,druidmini_joins.q,druid_timestamptz.q]
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testAlterPartition
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testAlterTable
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testAlterTableCascade
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testAlterViewParititon
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testColumnStatistics
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testComplexTable
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testComplexTypeApi
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testConcurrentMetastores
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testCreateAndGetTableWithDriver
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testCreateTableSettingId
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDBLocationChange
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDBOwner
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDBOwnerChange
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDatabase
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDatabaseLocation
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDatabaseLocationWithPermissionProblems
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDropDatabaseCascadeMVMultiDB
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDropTable
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testFilterLastPartition
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testFilterSinglePartition
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testFunctionWithResources
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetConfigValue
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetMetastoreUuid
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetPartitionsWithSpec
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetSchemaWithNoClassDefFoundError
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetTableObjects
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetUUIDInParallel
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testJDOPersistanceManagerCleanup
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testListPartitionNames
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testListPartitions
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testListPartitionsWihtLimitEnabled
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testNameMethods
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testPartition
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testPartitionFilter
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testRenamePartition
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testRetriableClientWithConnLifetime
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testSimpleFunction
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testSimpleTable
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testSimpleTypeApi
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testStatsFastTrivial
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testSynchronized
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testTableDatabase
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testTableFilter
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testUpdatePartitionStat_doesNotUpdateStats
(batchId=230)
org.apache.hadoop.hive
[jira] [Commented] (HIVE-20979) Fix memory leak in hive streaming
[
https://issues.apache.org/jira/browse/HIVE-20979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16702942#comment-16702942
]
Hive QA commented on HIVE-20979:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15111/dev-support/hive-personality.sh
|
| git revision | master / 073c7ab |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: streaming U: streaming |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15111/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix memory leak in hive streaming
> -
>
> Key: HIVE-20979
> URL: https://issues.apache.org/jira/browse/HIVE-20979
> Project: Hive
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-20979.1.patch, HIVE-20979.1.patch,
> HIVE-20979.2.patch, HIVE-20979.3.patch
>
>
> {{1) HiveStreamingConnection.Builder#init() adds a shutdown hook handler via
> }}{{ShutdownHookManager.addShutdownHook but it is never removed which causes
> all the handlers to accumulate and hence a memory leak.}}
> 2) AbstractRecordWriter creates an instance of FileSystem but does not close
> it which in turn causes a leak due to accumulation in FileSystem$Cache#map
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20979) Fix memory leak in hive streaming
[
https://issues.apache.org/jira/browse/HIVE-20979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16702434#comment-16702434
]
Hive QA commented on HIVE-20979:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949869/HIVE-20979.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15628 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=156)
[parquet_complex_types_vectorization.q,cte_4.q,file_with_header_footer.q,explainuser_2.q,rcfile_merge2.q,alter_table_location2.q,bucket5.q,mm_dp.q,orc_struct_type_vectorization.q,parquet_map_type_vectorization.q,reduce_deduplicate.q,load_fs2.q,rcfile_merge3.q,external_table_purge.q,intersect_distinct.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15101/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15101/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15101/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949869 - PreCommit-HIVE-Build
> Fix memory leak in hive streaming
> -
>
> Key: HIVE-20979
> URL: https://issues.apache.org/jira/browse/HIVE-20979
> Project: Hive
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-20979.1.patch, HIVE-20979.1.patch,
> HIVE-20979.2.patch
>
>
> {{1) HiveStreamingConnection.Builder#init() adds a shutdown hook handler via
> }}{{ShutdownHookManager.addShutdownHook but it is never removed which causes
> all the handlers to accumulate and hence a memory leak.}}
> 2) AbstractRecordWriter creates an instance of FileSystem but does not close
> it which in turn causes a leak due to accumulation in FileSystem$Cache#map
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20979) Fix memory leak in hive streaming
[
https://issues.apache.org/jira/browse/HIVE-20979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16702390#comment-16702390
]
Hive QA commented on HIVE-20979:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15101/dev-support/hive-personality.sh
|
| git revision | master / a070a9d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: streaming U: streaming |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15101/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix memory leak in hive streaming
> -
>
> Key: HIVE-20979
> URL: https://issues.apache.org/jira/browse/HIVE-20979
> Project: Hive
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-20979.1.patch, HIVE-20979.1.patch,
> HIVE-20979.2.patch
>
>
> {{1) HiveStreamingConnection.Builder#init() adds a shutdown hook handler via
> }}{{ShutdownHookManager.addShutdownHook but it is never removed which causes
> all the handlers to accumulate and hence a memory leak.}}
> 2) AbstractRecordWriter creates an instance of FileSystem but does not close
> it which in turn causes a leak due to accumulation in FileSystem$Cache#map
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20979) Fix memory leak in hive streaming
[
https://issues.apache.org/jira/browse/HIVE-20979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16702063#comment-16702063
]
Hive QA commented on HIVE-20979:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949836/HIVE-20979.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15643 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
(batchId=182)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15096/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15096/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15096/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949836 - PreCommit-HIVE-Build
> Fix memory leak in hive streaming
> -
>
> Key: HIVE-20979
> URL: https://issues.apache.org/jira/browse/HIVE-20979
> Project: Hive
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-20979.1.patch, HIVE-20979.1.patch
>
>
> {{1) HiveStreamingConnection.Builder#init() adds a shutdown hook handler via
> }}{{ShutdownHookManager.addShutdownHook but it is never removed which causes
> all the handlers to accumulate and hence a memory leak.}}
> 2) AbstractRecordWriter creates an instance of FileSystem but does not close
> it which in turn causes a leak due to accumulation in FileSystem$Cache#map
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20979) Fix memory leak in hive streaming
[
https://issues.apache.org/jira/browse/HIVE-20979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701984#comment-16701984
]
Hive QA commented on HIVE-20979:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15096/dev-support/hive-personality.sh
|
| git revision | master / a070a9d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: streaming U: streaming |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15096/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix memory leak in hive streaming
> -
>
> Key: HIVE-20979
> URL: https://issues.apache.org/jira/browse/HIVE-20979
> Project: Hive
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-20979.1.patch, HIVE-20979.1.patch
>
>
> {{1) HiveStreamingConnection.Builder#init() adds a shutdown hook handler via
> }}{{ShutdownHookManager.addShutdownHook but it is never removed which causes
> all the handlers to accumulate and hence a memory leak.}}
> 2) AbstractRecordWriter creates an instance of FileSystem but does not close
> it which in turn causes a leak due to accumulation in FileSystem$Cache#map
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20979) Fix memory leak in hive streaming
[
https://issues.apache.org/jira/browse/HIVE-20979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701973#comment-16701973
]
Prasanth Jayachandran commented on HIVE-20979:
--
+1, pending tests. [~ShubhamChaurasia] could you please reupload patch (bump up
the version) until you a get a green test run (requirement for commit)?
> Fix memory leak in hive streaming
> -
>
> Key: HIVE-20979
> URL: https://issues.apache.org/jira/browse/HIVE-20979
> Project: Hive
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-20979.1.patch, HIVE-20979.1.patch
>
>
> {{1) HiveStreamingConnection.Builder#init() adds a shutdown hook handler via
> }}{{ShutdownHookManager.addShutdownHook but it is never removed which causes
> all the handlers to accumulate and hence a memory leak.}}
> 2) AbstractRecordWriter creates an instance of FileSystem but does not close
> it which in turn causes a leak due to accumulation in FileSystem$Cache#map
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20979) Fix memory leak in hive streaming
[
https://issues.apache.org/jira/browse/HIVE-20979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701701#comment-16701701
]
Hive QA commented on HIVE-20979:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949807/HIVE-20979.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 46 failed/errored test(s), 15643 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
(batchId=182)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterPartition
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterTable
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterTableCascade
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterViewParititon
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testColumnStatistics
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testComplexTable
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testComplexTypeApi
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testConcurrentMetastores
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testCreateAndGetTableWithDriver
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testCreateTableSettingId
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBLocationChange
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBOwner
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBOwnerChange
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabase
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabaseLocation
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabaseLocationWithPermissionProblems
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDropDatabaseCascadeMVMultiDB
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDropTable
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFilterLastPartition
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFilterSinglePartition
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFunctionWithResources
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetConfigValue
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetMetastoreUuid
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetPartitionsWithSpec
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetSchemaWithNoClassDefFoundError
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetTableObjects
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetUUIDInParallel
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testJDOPersistanceManagerCleanup
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testListPartitionNames
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testListPartitions
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testListPartitionsWihtLimitEnabled
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testNameMethods
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testPartition
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testPartitionFilter
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testRenamePartition
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testRetriableClientWithConnLifetime
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testSimpleFunction
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testSimpleTable
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testSimpleTypeApi
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testStatsFastTrivial
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testSynchronized
(batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBi
[jira] [Commented] (HIVE-20979) Fix memory leak in hive streaming
[
https://issues.apache.org/jira/browse/HIVE-20979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701517#comment-16701517
]
ASF GitHub Bot commented on HIVE-20979:
---
GitHub user ShubhamChaurasia opened a pull request:
https://github.com/apache/hive/pull/495
HIVE-20979: Fix memory leak in hive streaming
Fixes two memory leaks.
-In HiveStreamingConnection due to Shutdown hook not being removed.
-In AbstractRecordWriter due to FileSystem not being closed.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ShubhamChaurasia/hive HIVE-20979
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/hive/pull/495.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #495
commit a6985008713ee731182db774626f0b75277b6172
Author: ShubhamChaurasia
Date: 2018-11-28T08:19:16Z
HIVE-20979: Fix memory leak in hive streaming
> Fix memory leak in hive streaming
> -
>
> Key: HIVE-20979
> URL: https://issues.apache.org/jira/browse/HIVE-20979
> Project: Hive
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Critical
> Labels: pull-request-available
>
> {{1) HiveStreamingConnection.Builder#init() adds a shutdown hook handler via
> }}{{ShutdownHookManager.addShutdownHook but it is never removed which causes
> all the handlers to accumulate and hence a memory leak.}}
> 2) AbstractRecordWriter creates an instance of FileSystem but does not close
> it which in turn causes a leak due to accumulation in FileSystem$Cache#map
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
