Dan Burkert has posted comments on this change. Change subject: Fix Spark shutdown ......................................................................
Patch Set 2: (6 comments) http://gerrit.cloudera.org:8080/#/c/2571/2//COMMIT_MSG Commit Message: Line 10: the Spark shell being unable to close properly. This commit also adds a shutdown > Hmm, I don't know about this. I find shutdown hooks to be pretty hack-ish; I agree, but Spark doesn't provide a way to do it properly. See http://apache-spark-developers-list.1001551.n3.nabble.com/graceful-shutdown-in-external-data-sources-td16684.html http://gerrit.cloudera.org:8080/#/c/2571/2/java/kudu-client/src/main/java/org/kududb/client/AsyncKuduClient.java File java/kudu-client/src/main/java/org/kududb/client/AsyncKuduClient.java: Line 201: private final HashedWheelTimer timer; > FWIW, I don't think it's important to use the same timer for our own timeou It's important that both timers use daemon threads. Reusing a single timer isn't necessary, but I do not see a reason not to. Line 1992: new HashedWheelTimer(new ThreadFactoryBuilder().setDaemon(true).build(), 20, MILLISECONDS); > The default is 100 ms; it would be nice to figure out why we're using 20 an I don't know the reason, it's been set to 20ms since the commit introducing the AsyncKuduClient in 2013. Line 2127: new NioWorkerPool(worker, workerCount), > This is just a side-effect of using the constructor with a customized timer Done Line 2128: timer); > So why do we need to use a timer with daemon threads? AsyncKuduClient.shutd Correct, running non-daemon threads will prevent shutdown hooks from running, which means that applications which rely on these shutdown hooks to close the client will hang. http://gerrit.cloudera.org:8080/#/c/2571/2/java/kudu-spark/src/main/scala/org/kududb/spark/KuduContext.scala File java/kudu-spark/src/main/scala/org/kududb/spark/KuduContext.scala: Line 38: println("Creating and registering sync client") > Was this just for debugging? good catch, removed. -- To view, visit http://gerrit.cloudera.org:8080/2571 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I30a64ec5eb30d70361204646523c9947d88c251f Gerrit-PatchSet: 2 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Dan Burkert <[email protected]> Gerrit-Reviewer: Adar Dembo <[email protected]> Gerrit-Reviewer: Binglin Chang <[email protected]> Gerrit-Reviewer: Dan Burkert <[email protected]> Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon <[email protected]> Gerrit-HasComments: Yes
