[
https://issues.apache.org/jira/browse/SPARK-50961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Jie updated SPARK-50961:
-----------------------------
Description:
After running the test with `build/sbt clean "connect-client-jvm/test" -Phive`,
I see the following content in the `unit-tests.log` file:
{code:java}
25/01/23 16:40:49.903 pool-1-thread-1-ScalaTest-running-ClientE2ETestSuite
ERROR ManagedChannelOrphanWrapper: *~*~*~ Previous channel
ManagedChannelImpl{logId=15, target=localhost:15642} was garbage collected
without being shut down! ~*~*~*
Make sure to call shutdown()/shutdownNow()
java.lang.RuntimeException: ManagedChannel allocation site
at
io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:102)
at
io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:60)
at
io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:51)
at
io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:709)
at
io.grpc.ForwardingChannelBuilder2.build(ForwardingChannelBuilder2.java:272)
at
org.apache.spark.sql.connect.client.SparkConnectClient$Configuration.createChannel(SparkConnectClient.scala:766)
at
org.apache.spark.sql.connect.client.SparkConnectClient$Configuration.toSparkConnectClient(SparkConnectClient.scala:769)
at
org.apache.spark.sql.connect.client.SparkConnectClient$Builder.build(SparkConnectClient.scala:685)
at
org.apache.spark.sql.ClientE2ETestSuite.$anonfun$new$57(ClientE2ETestSuite.scala:423)
at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at
org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
at
org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1564)
at
org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
at
org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
at
org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
at
org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
at org.scalatest.funsuite.AnyFunSuite.runTest(AnyFunSuite.scala:1564)
at
org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
at
org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
at scala.collection.immutable.List.foreach(List.scala:334)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
at
org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
at
org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
{code}
https://github.com/grpc/grpc-java/blob/24085103b926559659ecd3941a3308479876f084/core/src/main/java/io/grpc/internal/ManagedChannelOrphanWrapper.java#L149-L172
!image-2025-01-23-17-04-52-305.png!
>From the code that prints the logs, it seems that only the references of
>un-shutdown sessions are being printed. Is there a risk of resource leak here?
> `SparkSession.sessions` may have a risk of memory leak.
> -------------------------------------------------------
>
> Key: SPARK-50961
> URL: https://issues.apache.org/jira/browse/SPARK-50961
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 4.0.0, 3.5.4, 4.1.0
> Reporter: Yang Jie
> Priority: Major
> Attachments: image-2025-01-23-17-04-52-305.png
>
>
> After running the test with `build/sbt clean "connect-client-jvm/test"
> -Phive`, I see the following content in the `unit-tests.log` file:
> {code:java}
> 25/01/23 16:40:49.903 pool-1-thread-1-ScalaTest-running-ClientE2ETestSuite
> ERROR ManagedChannelOrphanWrapper: *~*~*~ Previous channel
> ManagedChannelImpl{logId=15, target=localhost:15642} was garbage collected
> without being shut down! ~*~*~*
> Make sure to call shutdown()/shutdownNow()
> java.lang.RuntimeException: ManagedChannel allocation site
> at
> io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:102)
> at
> io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:60)
> at
> io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:51)
> at
> io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:709)
> at
> io.grpc.ForwardingChannelBuilder2.build(ForwardingChannelBuilder2.java:272)
> at
> org.apache.spark.sql.connect.client.SparkConnectClient$Configuration.createChannel(SparkConnectClient.scala:766)
> at
> org.apache.spark.sql.connect.client.SparkConnectClient$Configuration.toSparkConnectClient(SparkConnectClient.scala:769)
> at
> org.apache.spark.sql.connect.client.SparkConnectClient$Builder.build(SparkConnectClient.scala:685)
> at
> org.apache.spark.sql.ClientE2ETestSuite.$anonfun$new$57(ClientE2ETestSuite.scala:423)
> at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
> at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> at org.scalatest.Transformer.apply(Transformer.scala:22)
> at org.scalatest.Transformer.apply(Transformer.scala:20)
> at
> org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
> at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
> at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
> at
> org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1564)
> at
> org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
> at
> org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
> at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
> at
> org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
> at
> org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
> at org.scalatest.funsuite.AnyFunSuite.runTest(AnyFunSuite.scala:1564)
> at
> org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
> at
> org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
> at scala.collection.immutable.List.foreach(List.scala:334)
> at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
> at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
> at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
> at
> org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
> at
> org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
> at
> org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564) {code}
> https://github.com/grpc/grpc-java/blob/24085103b926559659ecd3941a3308479876f084/core/src/main/java/io/grpc/internal/ManagedChannelOrphanWrapper.java#L149-L172
> !image-2025-01-23-17-04-52-305.png!
> From the code that prints the logs, it seems that only the references of
> un-shutdown sessions are being printed. Is there a risk of resource leak here?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
