This is an automated email from the ASF dual-hosted git repository. zjffdu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/master by this push: new 65a824e ZEPPELIN-4062. Don't wait ipython kernel if python process failed 65a824e is described below commit 65a824e9315649a4d349c7717f6b990d76239c9b Author: Jeff Zhang <zjf...@apache.org> AuthorDate: Wed Mar 13 10:31:00 2019 +0800 ZEPPELIN-4062. Don't wait ipython kernel if python process failed ### What is this PR for? This PR will exist wait ipython process loop if the python process is failed. This can save time to exit the waiting loop earlier, otherwise it would wait until timeout (30 seconds by default) ### What type of PR is it? [Improvement] ### Todos * [ ] - Task ### What is the Jira issue? * https://jira.apache.org/jira/browse/ZEPPELIN-4062 ### How should this be tested? * Manually tested. Before this PR ``` INFO [2019-03-13 10:06:30,070] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started INFO [2019-03-13 10:06:30,175] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started INFO [2019-03-13 10:06:30,281] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started INFO [2019-03-13 10:06:30,386] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started INFO [2019-03-13 10:06:30,489] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started INFO [2019-03-13 10:06:30,595] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started INFO [2019-03-13 10:06:30,700] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started INFO [2019-03-13 10:06:30,805] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started WARN [2019-03-13 10:06:30,810] ({Exec Default Executor} IPythonInterpreter.java[onProcessFailed]:419) - Exception happens in Python Process org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404) at org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48) at org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200) at java.lang.Thread.run(Thread.java:748) INFO [2019-03-13 10:06:30,910] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to be started INFO [2019-03-13 10:06:31,015] ({FIFOScheduler-Worker-1} ``` After this PR ``` INFO [2019-03-13 10:28:12,805] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to be started INFO [2019-03-13 10:28:12,907] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to be started INFO [2019-03-13 10:28:13,011] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to be started INFO [2019-03-13 10:28:13,116] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to be started INFO [2019-03-13 10:28:13,220] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to be started INFO [2019-03-13 10:28:13,325] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to be started WARN [2019-03-13 10:28:13,347] ({Exec Default Executor} IPythonInterpreter.java[onProcessFailed]:423) - Exception happens in Python Process org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404) at org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48) at org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200) at java.lang.Thread.run(Thread.java:748) INFO [2019-03-13 10:28:13,430] ({FIFOScheduler-Worker-1} IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to be started WARN [2019-03-13 10:28:13,431] ({FIFOScheduler-Worker-1} PythonInterpreter.java[open]:98) - Fail to open IPythonInterpreter java.lang.RuntimeException: Fail to open IPythonInterpreter at org.apache.zeppelin.python.IPythonInterpreter.open(IPythonInterpreter.java:152) at org.apache.zeppelin.python.PythonInterpreter.open(PythonInterpreter.java:93) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:598) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:507) at org.apache.zeppelin.scheduler.Job.run(Job.java:172) at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:121) at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:39) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Fail to launch IPython Kernel as the python process is failed at org.apache.zeppelin.python.IPythonInterpreter.launchIPythonKernel(IPythonInterpreter.java:323) at org.apache.zeppelin.python.IPythonInterpreter.open(IPythonInterpreter.java:149) ... 10 more INFO [2019-03-13 10:28:13,431] ({FIFOScheduler-Worker-1} PythonInterpreter.java[open]:104) - IPython is not available, use the native PythonInterpreter ``` ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjf...@apache.org> Closes #3332 from zjffdu/ZEPPELIN-4062 and squashes the following commits: 130c2a82a [Jeff Zhang] ZEPPELIN-4062. Don't wait ipython kernel if python process failed --- .../main/java/org/apache/zeppelin/python/IPythonInterpreter.java | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/python/src/main/java/org/apache/zeppelin/python/IPythonInterpreter.java b/python/src/main/java/org/apache/zeppelin/python/IPythonInterpreter.java index f5c19be..9e23d04 100644 --- a/python/src/main/java/org/apache/zeppelin/python/IPythonInterpreter.java +++ b/python/src/main/java/org/apache/zeppelin/python/IPythonInterpreter.java @@ -82,6 +82,7 @@ public class IPythonInterpreter extends Interpreter implements ExecuteResultHand private boolean useBuiltinPy4j = true; private boolean usePy4JAuth = true; private String secret; + private volatile boolean pythonProcessFailed = false; private InterpreterOutputStream interpreterOutput = new InterpreterOutputStream(LOGGER); @@ -293,7 +294,7 @@ public class IPythonInterpreter extends Interpreter implements ExecuteResultHand // wait until IPython kernel is started or timeout long startTime = System.currentTimeMillis(); - while (true) { + while (!pythonProcessFailed) { try { Thread.sleep(100); } catch (InterruptedException e) { @@ -318,6 +319,9 @@ public class IPythonInterpreter extends Interpreter implements ExecuteResultHand + " seconds"); } } + if (pythonProcessFailed) { + throw new IOException("Fail to launch IPython Kernel as the python process is failed"); + } } protected Map<String, String> setupIPythonEnv() throws IOException { @@ -417,6 +421,7 @@ public class IPythonInterpreter extends Interpreter implements ExecuteResultHand @Override public void onProcessFailed(ExecuteException e) { LOGGER.warn("Exception happens in Python Process", e); + pythonProcessFailed = true; } static class ProcessLogOutputStream extends LogOutputStream {