Repository: zeppelin Updated Branches: refs/heads/master 2f03469bc -> 05af3bf4b
[ZEPPELIN-2564] - LivySparkSQLInterpreter throws NullPointerException when getProgress is called ### What is this PR for? When using %livy.sql interpreter following error is logged into interpreter log: `ERROR [2017-05-17 15:01:32,897] ( {pool-1-thread-1} TThreadPoolServer.java[run]:296) - Error occurred during processing of message. java.lang.NullPointerException at org.apache.zeppelin.livy.BaseLivyInterpreter.getProgress(BaseLivyInterpreter.java:178) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.getProgress(LazyOpenInterpreter.java:121) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.getProgress(RemoteInterpreterServer.java:558) at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:1899) at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:1884) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748)` and zeppelin server log: `ERROR [2017-05-17 15:01:44,004] ( {JobProgressPoller, jobId=20170511-200435_1144682471} JobProgressPoller.java[run]:58) - Can not get or update progress org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getProgress(RemoteInterpreter.java:468) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.getProgress(LazyOpenInterpreter.java:121) at org.apache.zeppelin.notebook.Paragraph.progress(Paragraph.java:334) at org.apache.zeppelin.scheduler.JobProgressPoller.run(JobProgressPoller.java:55) Caused by: org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_getProgress(RemoteInterpreterService.java:321) at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.getProgress(RemoteInterpreterService.java:306) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getProgress(RemoteInterpreter.java:465) ... 3 more` This happens because this.livyVersion is never assigned in LivySparkSQLInterpreter: https://github.com/apache/zeppelin/blob/master/livy/src/main/java/org/apache/zeppelin/livy/LivySparkSQLInterpreter.java#L63 ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-2564 ### How should this be tested? New test added to verify that livyVersion is not null ### Screenshots (if appropriate) ![errors](https://cloud.githubusercontent.com/assets/14924427/26164543/9aaa33a6-3b35-11e7-809d-e976277dac44.gif) ### Questions: * Does the licenses files need update? * Is there breaking changes for older versions? * Does this needs documentation? Author: sergeymazin <sergey.ma...@pipedrive.com> Closes #2350 from sergeymazin/master and squashes the following commits: 3b95f5f [sergeymazin] Merge branch 'master' of https://github.com/sergeymazin/zeppelin b506753 [sergeymazin] Check sparkInterpreter for null and move test 630520a [sergeymazin] Check sparkInterpreter for null and move test c29c984 [sergeymazin] LivySparkSQLInterpreter to use getProgress of sparkInterpter 84db2d2 [sergeymazin] LivySparkSQLInterpreter throws NullPointerException when getProgress is called Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/05af3bf4 Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/05af3bf4 Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/05af3bf4 Branch: refs/heads/master Commit: 05af3bf4b7f68c8434a8d1cb576fb6f4571551ab Parents: 2f03469 Author: sergeymazin <sergey.ma...@pipedrive.com> Authored: Thu May 18 01:13:44 2017 +0300 Committer: Jeff Zhang <zjf...@apache.org> Committed: Thu May 18 10:34:08 2017 -0400 ---------------------------------------------------------------------- .../org/apache/zeppelin/livy/LivySparkSQLInterpreter.java | 9 +++++++++ .../java/org/apache/zeppelin/livy/LivyInterpreterIT.java | 2 ++ 2 files changed, 11 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/zeppelin/blob/05af3bf4/livy/src/main/java/org/apache/zeppelin/livy/LivySparkSQLInterpreter.java ---------------------------------------------------------------------- diff --git a/livy/src/main/java/org/apache/zeppelin/livy/LivySparkSQLInterpreter.java b/livy/src/main/java/org/apache/zeppelin/livy/LivySparkSQLInterpreter.java index d132b5b..20d0448 100644 --- a/livy/src/main/java/org/apache/zeppelin/livy/LivySparkSQLInterpreter.java +++ b/livy/src/main/java/org/apache/zeppelin/livy/LivySparkSQLInterpreter.java @@ -240,6 +240,15 @@ public class LivySparkSQLInterpreter extends BaseLivyInterpreter { } @Override + public int getProgress(InterpreterContext context) { + if (this.sparkInterpreter != null) { + return this.sparkInterpreter.getProgress(context); + } else { + return 0; + } + } + + @Override protected String extractAppId() throws LivyException { // it wont' be called because it would delegate to LivySparkInterpreter throw new UnsupportedOperationException(); http://git-wip-us.apache.org/repos/asf/zeppelin/blob/05af3bf4/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java ---------------------------------------------------------------------- diff --git a/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java b/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java index 6db75e9..80965ff 100644 --- a/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java +++ b/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java @@ -308,6 +308,8 @@ public class LivyInterpreterIT { assertEquals(InterpreterResult.Code.SUCCESS, result.code()); assertEquals(InterpreterResult.Type.TABLE, result.message().get(0).getType()); assertTrue(result.message().get(0).getData().contains("tableName")); + int r = sqlInterpreter.getProgress(context); + assertTrue(r == 0); } finally { sqlInterpreter.close(); }