Grant Henke has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11571 )
Change subject: KUDU-2563: [spark] Use the scanner keepAlive API ...................................................................... KUDU-2563: [spark] Use the scanner keepAlive API Adds scheduled keepAlive calls to the scanner in the KuduRDD RowIterator. The period in which the calls are made is configurable via keepAlivePeriodMs and has a default of 15 seconds (which is 1/4 the default scanner ttl). This implementation is similar to the Impala integration. It checks if a call to the keepAlive API is needed as it processes each row. Compared to a background thread, this has the downside of being less consistently scheduled and susceptible to scenarios in which a single row takes longer to process than the ttl. However, because the scanner is not thread safe, this is the most straightforward solution and has been proven to work. Change-Id: Ia7f26d6ab8deb24982055d247938a11e188c35db Reviewed-on: http://gerrit.cloudera.org:8080/11571 Reviewed-by: Grant Henke <granthe...@apache.org> Tested-by: Grant Henke <granthe...@apache.org> --- M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduScanner.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanner.java M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/DefaultSource.scala M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduReadOptions.scala M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduRDDTest.scala 6 files changed, 139 insertions(+), 11 deletions(-) Approvals: Grant Henke: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11571 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ia7f26d6ab8deb24982055d247938a11e188c35db Gerrit-Change-Number: 11571 Gerrit-PatchSet: 5 Gerrit-Owner: Grant Henke <granthe...@apache.org> Gerrit-Reviewer: Adar Dembo <a...@cloudera.com> Gerrit-Reviewer: Dan Burkert <danburk...@apache.org> Gerrit-Reviewer: Grant Henke <granthe...@apache.org> Gerrit-Reviewer: Hao Hao <hao....@cloudera.com> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy <mpe...@apache.org>