xc-jianghan commented on issue #4814:
URL: https://github.com/apache/kyuubi/issues/4814#issuecomment-3864472115
Hi Cheng Pan,
Thank you for the suggestion about avoiding potential leaks.
So my understanding is: I should initialize the Atlas client earlier, and
also return a flag from getClient() to indicate whether the caller should
explicitly close the client after use (when the shutdown hook cannot be
registered). Would a simple approach like the following make sense?
import org.apache.hadoop.util.ShutdownHookManager object AtlasClient {
@volatile private var cached: AtlasRestClient = _ def
getClient(/* conf / props */): (AtlasRestClient, Boolean) = { if
(cached != null) return (cached, false) this.synchronized {
if (cached != null) return (cached, false)
val tmp = new AtlasRestClient(/* conf / props */)
val hookOk = tryRegisterCleanupShutdownHook(tmp) if
(hookOk) { cached = tmp
(cached, false) } else { (tmp,
true) // caller should close after use } }
} private def tryRegisterCleanupShutdownHook(client:
AtlasRestClient): Boolean = { val shm = ShutdownHookManager.get()
if (shm.isShutdownInProgress) return false try {
 
; shm.addShutdownHook( new
Thread("kyuubi-atlas-client-cleanup") {
override def run(): Unit = client.close() },
/* priority */ 0 )
true } catch { case _:
IllegalStateException => false }
} }
And then update the dispatcher to close the client only when needed:
val (client, shouldClose) = AtlasClient.getClient(...) try {
client.send(...) } finally { if (shouldClose) {
client.close() } }
My question: with this change, can we ensure that in the spark-sql -e
single-statement case, the lineage is sent successfully before Spark exits? Or
is it still possible that the driver shuts down before the listener/dispatcher
has a chance to run?
Thanks again!
Cheng ***@***.***> 在 2026年2月7日 周六 20:29 写道：
pan3793 left a comment (apache/kyuubi#4814)
a simple approach:
object AtlasClient { ... - def getClient(): AtlasClient = { + def
getClient(): (AtlasClient, Boolean) = { ... }
Add an additional flag in the return value to indicate whether the caller
should close it AtlasClient after using it.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
