[GitHub] [zeppelin] jongyoul closed pull request #3007: zeppelin 0.8 version pyspark interpreter error when using ipython
jongyoul closed pull request #3007: zeppelin 0.8 version pyspark interpreter error when using ipython URL: https://github.com/apache/zeppelin/pull/3007 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@zeppelin.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [zeppelin] jongyoul commented on pull request #3007: zeppelin 0.8 version pyspark interpreter error when using ipython
jongyoul commented on PR #3007: URL: https://github.com/apache/zeppelin/pull/3007#issuecomment-1103814288 Let me close it as it's too outdated to handle. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@zeppelin.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (ZEPPELIN-5691) Code completion for ipython causes exception (sometimes)
jason ogaard created ZEPPELIN-5691: -- Summary: Code completion for ipython causes exception (sometimes) Key: ZEPPELIN-5691 URL: https://issues.apache.org/jira/browse/ZEPPELIN-5691 Project: Zeppelin Issue Type: Bug Components: zeppelin-interpreter Affects Versions: 0.10.1 Reporter: jason ogaard Attachments: image-2022-03-24-11-37-04-519.png The first time a user tries to use code completion for the iPython interpreter the notebook will hang for a long time (I believe it's waiting for the websocket to return) and will get the pop up below (tested on the latest docker image for apache zeppelin found on docker hub) !image-2022-03-24-11-37-04-519.png|width=680,height=181! After this code completion usually works but will sometimes throw this error again. This error will happen again if the user restarts the interpreter. We do not get this error when attempting code completion for the python interpreter. I've traced through the code as well as I can and believe the issue is that code completion is trying to use the PythonInterpreter class to get the completion list, but since we're using the iPython interpreter the PythonInterpreter class is not instantiated, and therefore get the NPE. I could be wrong though. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: ipython interpreter prerequisites
I see this message when I tried to list all installed modules. I assume the IPython.kernel module got brought in by the "pip install jupyter". It seems I can replace it with "pip install jupyter_client"? The `IPython.kernel` package has been deprecated since IPython 4.0.You should import from ipykernel or jupyter_client instead. On Wednesday, October 27, 2021, 11:47:11 PM EDT, Jeff Zhang wrote: I don't think so, at least we need jupyter-client module as we use jupyter-client api to connect with ipython kernel. denny wong 于2021年10月28日周四 上午11:24写道： > Hi > Question about the ipython interpreter prerequisites. The jupyter module > version ( https://pypi.org/project/jupyter/) is pretty old. Can it be > replaced by ipykernel module (https://pypi.org/project/ipykernel/)? > > Prerequisites > - Jupyter `pip install jupyter` > - grpcio `pip install grpcio` > - protobuf `pip install protobuf` > > Will this work instead?- ipykernel `pip install ipykernel` > - grpcio `pip install grpcio` > - protobuf `pip install protobuf` > Thanks > Denny -- Best Regards Jeff Zhang
Re: ipython interpreter prerequisites
I don't think so, at least we need jupyter-client module as we use jupyter-client api to connect with ipython kernel. denny wong 于2021年10月28日周四 上午11:24写道： > Hi > Question about the ipython interpreter prerequisites. The jupyter module > version ( https://pypi.org/project/jupyter/) is pretty old. Can it be > replaced by ipykernel module (https://pypi.org/project/ipykernel/)? > > Prerequisites > - Jupyter `pip install jupyter` > - grpcio `pip install grpcio` > - protobuf `pip install protobuf` > > Will this work instead?- ipykernel `pip install ipykernel` > - grpcio `pip install grpcio` > - protobuf `pip install protobuf` > Thanks > Denny -- Best Regards Jeff Zhang
Re: ipython interpreter prerequisites
Hi Question about the ipython interpreter prerequisites. The jupyter module version ( https://pypi.org/project/jupyter/) is pretty old. Can it be replaced by ipykernel module (https://pypi.org/project/ipykernel/)? Prerequisites - Jupyter `pip install jupyter` - grpcio `pip install grpcio` - protobuf `pip install protobuf` Will this work instead?- ipykernel `pip install ipykernel` - grpcio `pip install grpcio` - protobuf `pip install protobuf` Thanks Denny
[jira] [Created] (ZEPPELIN-5533) Fail to get code completion in IPython interpreter
Jeff Zhang created ZEPPELIN-5533:
Summary: Fail to get code completion in IPython interpreter
Key: ZEPPELIN-5533
URL: https://issues.apache.org/jira/browse/ZEPPELIN-5533
Project: Zeppelin
Issue Type: Bug
Components: python
Affects Versions: 0.10.0
Reporter: Jeff Zhang
{code:java}
ERROR [2021-09-21 19:53:12,756] ({pool-2-thread-1}
ProcessFunction.java[process]:47) - Internal error processing completion
io.grpc.StatusRuntimeException: UNKNOWN: Exception calling application: There
is no current event loop in thread 'ThreadPoolExecutor-0_1'.
at
io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139)
at
org.apache.zeppelin.interpreter.jupyter.proto.JupyterKernelGrpc$JupyterKernelBlockingStub.complete(JupyterKernelGrpc.java:330)
at
org.apache.zeppelin.jupyter.JupyterKernelClient.complete(JupyterKernelClient.java:296)
at
org.apache.zeppelin.jupyter.JupyterKernelInterpreter.completion(JupyterKernelInterpreter.java:347)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.completion(LazyOpenInterpreter.java:145)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.completion(RemoteInterpreterServer.java:974)
at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$completion.getResult(RemoteInterpreterService.java:1932)
at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$completion.getResult(RemoteInterpreterService.java:1911)
at
org.apache.zeppelin.shaded.org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at
org.apache.zeppelin.shaded.org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at
org.apache.zeppelin.shaded.org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:313)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [zeppelin] asfgit closed pull request #4198: [ZEPPELIN-5479] %python.sql doesn't work with ipython interpreter
asfgit closed pull request #4198: URL: https://github.com/apache/zeppelin/pull/4198 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@zeppelin.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4198: [ZEPPELIN-5479] %python.sql doesn't work with ipython interpreter
zjffdu commented on pull request #4198: URL: https://github.com/apache/zeppelin/pull/4198#issuecomment-895688714 Will merge if no more comment -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@zeppelin.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [zeppelin] zjffdu opened a new pull request #4198: [ZEPPELIN-5479] %python.sql doesn't work with ipython interpreter
zjffdu opened a new pull request #4198: URL: https://github.com/apache/zeppelin/pull/4198 ### What is this PR for? This PR make %python.sql work with both vanilla python interpreter and ipython interpreter. It depends on whether ipython prerequisites are met. ### What type of PR is it? [ Improvement ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-5479 ### How should this be tested? * CI pass ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@zeppelin.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (ZEPPELIN-5479) %python.sql doesn't work with ipython interpreter
Jeff Zhang created ZEPPELIN-5479: Summary: %python.sql doesn't work with ipython interpreter Key: ZEPPELIN-5479 URL: https://issues.apache.org/jira/browse/ZEPPELIN-5479 Project: Zeppelin Issue Type: Improvement Components: python Affects Versions: 0.9.0, 0.10.0 Reporter: Jeff Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ZEPPELIN-4684) Unable to use plotly in IPython interpreter
Jeff Zhang created ZEPPELIN-4684:
Summary: Unable to use plotly in IPython interpreter
Key: ZEPPELIN-4684
URL: https://issues.apache.org/jira/browse/ZEPPELIN-4684
Project: Zeppelin
Issue Type: Improvement
Components: python
Affects Versions: 0.9.0
Reporter: Jeff Zhang
{code:java}
import plotly.graph_objects as go
fig = go.Figure(data=go.Bar(y=[2, 3, 1]))
fig.show()
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ZEPPELIN-4569) Zeppelin Ipython interpreter is not closing properly.
Loghi Perinpanayagam created ZEPPELIN-4569:
--
Summary: Zeppelin Ipython interpreter is not closing properly.
Key: ZEPPELIN-4569
URL: https://issues.apache.org/jira/browse/ZEPPELIN-4569
Project: Zeppelin
Issue Type: Bug
Components: zeppelin-interpreter
Affects Versions: 0.8.2
Environment: JDK 1.8.0
Apache Zeppelin 0.8.2
Reporter: Loghi Perinpanayagam
{code:java}
io.grpc.internal.SerializingExecutor run
SEVERE: Exception while executing runnable
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable@619fd7d
java.lang.NullPointerException
at
io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434)
at
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at
io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678)
at
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at
io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:403)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:531)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
The problem must be in io.grpc. Could anyone help?{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ZEPPELIN-4531) Support Dash in IPython Interpreter
Jeff Zhang created ZEPPELIN-4531: Summary: Support Dash in IPython Interpreter Key: ZEPPELIN-4531 URL: https://issues.apache.org/jira/browse/ZEPPELIN-4531 Project: Zeppelin Issue Type: New Feature Affects Versions: 0.9.0 Reporter: Jeff Zhang Just like shiny as a dashboard library in R, dash is a popular dashboard library in python. this ticket is to support dash in IPython Interpreter -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [zeppelin] zjffdu opened a new pull request #3551: [ZEPPELIN-4480]. Move the ipython code into a general jupyter kernel bridge
zjffdu opened a new pull request #3551: [ZEPPELIN-4480]. Move the ipython code into a general jupyter kernel bridge URL: https://github.com/apache/zeppelin/pull/3551 ### What is this PR for? This PR move the ipython code into module zeppelin-jupyter-adapter. And ipython will just depends on this module. zeppelin-jupyter-adapter could be used for connecting any jupyter kernel. ### What type of PR is it? [ Feature | Refactoring] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4480 ### How should this be tested? * Ci pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (ZEPPELIN-4480) Move the ipython code into a general jupyter kernel bridge
Jeff Zhang created ZEPPELIN-4480: Summary: Move the ipython code into a general jupyter kernel bridge Key: ZEPPELIN-4480 URL: https://issues.apache.org/jira/browse/ZEPPELIN-4480 Project: Zeppelin Issue Type: New Feature Affects Versions: 0.9.0 Reporter: Jeff Zhang The ipython code can not only be used by ipython interpreter, but also could bridge any jupyter kernel, such as r kernel. So this ticket is to create a general jupyter kernel so that other interpreter can reuse it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [zeppelin] AyWa edited a comment on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython
AyWa edited a comment on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-478428352 @zjffdu Thank you, the OOM fix the issue ! the test is working well: https://travis-ci.org/AyWa/zeppelin/builds/513921618 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa edited a comment on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython
AyWa edited a comment on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-475549925 I am kind of lost with the test: - I ran my fix, without the new test -> all the test pass on the ci - I ran my fix, with the new test - > always some failure (seems like `GC` or out of memory or timeout) - I ran my fix, with a test, that is not testing the fix (like no concurrent call) -> fail again So i think the problem is not related to the fix itself, but more like an issue with the test: - `ExecutorService pool = Executors.newFixedThreadPool(2);` make the `GC / OOM` ? - running then auto complete make some issue ? I will try to push the same code without executor / thread to see if it is working, then try to thread it again :/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa edited a comment on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython
AyWa edited a comment on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-475549925 I am kind of lost with the test: - I ran my fix, without the new test -> all the test pass on the ci - I ran my fix, with the new test - > always some failure (seems like `GC` or out of memory or timeout) - I ran my fix, with a test, that is not testing the fix (like no concurrent call) -> fail again So i think the problem is not related to the fix itself, but more like an issue with the test: - `ExecutorService pool = Executors.newFixedThreadPool(2); `make the GC / OOM` ? - running then auto complete make some issue ? I will try to push the same code without executor / thread to see if it is working, then try to thread it again :/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] asfgit closed pull request #3337: [ZEPPELIN-4090] Ipython queue performance
asfgit closed pull request #3337: [ZEPPELIN-4090] Ipython queue performance URL: https://github.com/apache/zeppelin/pull/3337 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on issue #3337: [ZEPPELIN-4090] Ipython queue performance
AyWa commented on issue #3337: [ZEPPELIN-4090] Ipython queue performance URL: https://github.com/apache/zeppelin/pull/3337#issuecomment-530291916 Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3337: [ZEPPELIN-4090] Ipython queue performance
zjffdu commented on issue #3337: [ZEPPELIN-4090] Ipython queue performance URL: https://github.com/apache/zeppelin/pull/3337#issuecomment-530291117 That's fine, let's look at this issue next time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa edited a comment on issue #3337: [ZEPPELIN-4090] Ipython queue performance
AyWa edited a comment on issue #3337: [ZEPPELIN-4090] Ipython queue performance URL: https://github.com/apache/zeppelin/pull/3337#issuecomment-530290115 So I double check, it really seems to be correct, next pr I made I will ensure the CI is shown in the pr. does it work for you ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on issue #3337: [ZEPPELIN-4090] Ipython queue performance
AyWa commented on issue #3337: [ZEPPELIN-4090] Ipython queue performance URL: https://github.com/apache/zeppelin/pull/3337#issuecomment-530290115 So I double check, it really seems to be correct, next pr I made I will ensure the CI is shown in the pr. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3337: [ZEPPELIN-4090] Ipython queue performance
zjffdu commented on issue #3337: [ZEPPELIN-4090] Ipython queue performance URL: https://github.com/apache/zeppelin/pull/3337#issuecomment-530288766 Another favor needs from you @AyWa Could you check whether you configure your travis according this guideline ? Because I don't see your travis build link here. https://zeppelin.apache.org/contribution/contributions.html#continuous-integration This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3337: [ZEPPELIN-4090] Ipython queue performance
zjffdu commented on issue #3337: [ZEPPELIN-4090] Ipython queue performance URL: https://github.com/apache/zeppelin/pull/3337#issuecomment-530288322 Awesome, will merge it soon. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on issue #3337: [ZEPPELIN-4090] Ipython queue performance
AyWa commented on issue #3337: [ZEPPELIN-4090] Ipython queue performance URL: https://github.com/apache/zeppelin/pull/3337#issuecomment-530286019 test are successful https://travis-ci.org/AyWa/zeppelin/builds/583492141 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu commented on a change in pull request #3337: [ZEPPELIN-4090] Ipython queue performance
zjffdu commented on a change in pull request #3337: [ZEPPELIN-4090] Ipython queue performance URL: https://github.com/apache/zeppelin/pull/3337#discussion_r323117291 ## File path: python/src/main/resources/grpc/python/ipython_server.py ## @@ -94,69 +105,24 @@ def execute_worker(): t = threading.Thread(name="ConsumerThread", target=execute_worker) with self._lock: t.start() -# We want to ensure that the kernel is alive because in case of OOM or other errors -# Execution might be stuck there: -# https://github.com/jupyter/jupyter_client/blob/master/jupyter_client/blocking/client.py#L32 -while t.is_alive() and self.isKernelAlive(): -while not text_queue.empty(): -output = text_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS, - type=ipython_pb2.TEXT, - output=output) -while not html_queue.empty(): -output = html_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS, - type=ipython_pb2.HTML, - output=output) -while not stderr_queue.empty(): -output = stderr_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.ERROR, - type=ipython_pb2.TEXT, - output=output) -while not png_queue.empty(): -output = png_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS, - type=ipython_pb2.PNG, - output=output) -while not jpeg_queue.empty(): -output = jpeg_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS, - type=ipython_pb2.JPEG, - output=output) - -# if kernel is not alive (should be same as thread is still alive), means that we face -# an unexpected issue. -if not self.isKernelAlive() or t.is_alive(): -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.ERROR, -type=ipython_pb2.TEXT, - output="Ipython kernel has been stopped. Please check logs. It might be because of an out of memory issue.") -return - -while not text_queue.empty(): Review comment: Thanks for the explanation, the changes LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on a change in pull request #3337: [ZEPPELIN-4090] Ipython queue performance
AyWa commented on a change in pull request #3337: [ZEPPELIN-4090] Ipython queue performance URL: https://github.com/apache/zeppelin/pull/3337#discussion_r323114745 ## File path: python/src/main/resources/grpc/python/ipython_server.py ## @@ -94,69 +105,24 @@ def execute_worker(): t = threading.Thread(name="ConsumerThread", target=execute_worker) with self._lock: t.start() -# We want to ensure that the kernel is alive because in case of OOM or other errors -# Execution might be stuck there: -# https://github.com/jupyter/jupyter_client/blob/master/jupyter_client/blocking/client.py#L32 -while t.is_alive() and self.isKernelAlive(): -while not text_queue.empty(): -output = text_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS, - type=ipython_pb2.TEXT, - output=output) -while not html_queue.empty(): -output = html_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS, - type=ipython_pb2.HTML, - output=output) -while not stderr_queue.empty(): -output = stderr_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.ERROR, - type=ipython_pb2.TEXT, - output=output) -while not png_queue.empty(): -output = png_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS, - type=ipython_pb2.PNG, - output=output) -while not jpeg_queue.empty(): -output = jpeg_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS, - type=ipython_pb2.JPEG, - output=output) - -# if kernel is not alive (should be same as thread is still alive), means that we face -# an unexpected issue. -if not self.isKernelAlive() or t.is_alive(): -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.ERROR, -type=ipython_pb2.TEXT, - output="Ipython kernel has been stopped. Please check logs. It might be because of an out of memory issue.") -return - -while not text_queue.empty(): Review comment: So the refactor I did, move all the logic in a single loop condition. So we do not need to have multiple time the similar code (in the main loop and outside of it) It is line https://github.com/apache/zeppelin/pull/3337/files#diff-9abe0e14503875941f3276b5f7743c46R113 until the queue is not empty we will not leave the main loop: `while (t.is_alive() and self.isKernelAlive()) or not stream_reply_queue.empty(): ` Let me know if it makes sense or if I miss something This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on a change in pull request #3337: [ZEPPELIN-4090] Ipython queue performance
AyWa commented on a change in pull request #3337: [ZEPPELIN-4090] Ipython queue performance URL: https://github.com/apache/zeppelin/pull/3337#discussion_r323114745 ## File path: python/src/main/resources/grpc/python/ipython_server.py ## @@ -94,69 +105,24 @@ def execute_worker(): t = threading.Thread(name="ConsumerThread", target=execute_worker) with self._lock: t.start() -# We want to ensure that the kernel is alive because in case of OOM or other errors -# Execution might be stuck there: -# https://github.com/jupyter/jupyter_client/blob/master/jupyter_client/blocking/client.py#L32 -while t.is_alive() and self.isKernelAlive(): -while not text_queue.empty(): -output = text_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS, - type=ipython_pb2.TEXT, - output=output) -while not html_queue.empty(): -output = html_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS, - type=ipython_pb2.HTML, - output=output) -while not stderr_queue.empty(): -output = stderr_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.ERROR, - type=ipython_pb2.TEXT, - output=output) -while not png_queue.empty(): -output = png_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS, - type=ipython_pb2.PNG, - output=output) -while not jpeg_queue.empty(): -output = jpeg_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS, - type=ipython_pb2.JPEG, - output=output) - -# if kernel is not alive (should be same as thread is still alive), means that we face -# an unexpected issue. -if not self.isKernelAlive() or t.is_alive(): -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.ERROR, -type=ipython_pb2.TEXT, - output="Ipython kernel has been stopped. Please check logs. It might be because of an out of memory issue.") -return - -while not text_queue.empty(): Review comment: So the refactor I did, move all the logic in a single loop condition. So we do not need to have multiple time the similar code (in the main loop and outside of it) It is line https://github.com/apache/zeppelin/pull/3337/files#diff-9abe0e14503875941f3276b5f7743c46R113 until the queue is not empty we will not leave the main loop: `while (t.is_alive() and self.isKernelAlive()) or not stream_reply_queue.empty(): ` Let me know if it makes sense This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on a change in pull request #3337: [ZEPPELIN-4090] Ipython queue performance
AyWa commented on a change in pull request #3337: [ZEPPELIN-4090] Ipython queue performance URL: https://github.com/apache/zeppelin/pull/3337#discussion_r323114745 ## File path: python/src/main/resources/grpc/python/ipython_server.py ## @@ -94,69 +105,24 @@ def execute_worker(): t = threading.Thread(name="ConsumerThread", target=execute_worker) with self._lock: t.start() -# We want to ensure that the kernel is alive because in case of OOM or other errors -# Execution might be stuck there: -# https://github.com/jupyter/jupyter_client/blob/master/jupyter_client/blocking/client.py#L32 -while t.is_alive() and self.isKernelAlive(): -while not text_queue.empty(): -output = text_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS, - type=ipython_pb2.TEXT, - output=output) -while not html_queue.empty(): -output = html_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS, - type=ipython_pb2.HTML, - output=output) -while not stderr_queue.empty(): -output = stderr_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.ERROR, - type=ipython_pb2.TEXT, - output=output) -while not png_queue.empty(): -output = png_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS, - type=ipython_pb2.PNG, - output=output) -while not jpeg_queue.empty(): -output = jpeg_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS, - type=ipython_pb2.JPEG, - output=output) - -# if kernel is not alive (should be same as thread is still alive), means that we face -# an unexpected issue. -if not self.isKernelAlive() or t.is_alive(): -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.ERROR, -type=ipython_pb2.TEXT, - output="Ipython kernel has been stopped. Please check logs. It might be because of an out of memory issue.") -return - -while not text_queue.empty(): Review comment: So the refactor I did, move all the logic in a single loop condition. So we do not need to have multiple time the similar code (in the main loop and outside of it) It is line https://github.com/apache/zeppelin/pull/3337/files#diff-9abe0e14503875941f3276b5f7743c46R113 until the queue is not empty we will not leave the main loop: `while (t.is_alive() and self.isKernelAlive()) or not stream_reply_queue.empty(): ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu commented on a change in pull request #3337: [ZEPPELIN-4090] Ipython queue performance
zjffdu commented on a change in pull request #3337: [ZEPPELIN-4090] Ipython queue performance URL: https://github.com/apache/zeppelin/pull/3337#discussion_r323113571 ## File path: python/src/main/resources/grpc/python/ipython_server.py ## @@ -94,69 +105,24 @@ def execute_worker(): t = threading.Thread(name="ConsumerThread", target=execute_worker) with self._lock: t.start() -# We want to ensure that the kernel is alive because in case of OOM or other errors -# Execution might be stuck there: -# https://github.com/jupyter/jupyter_client/blob/master/jupyter_client/blocking/client.py#L32 -while t.is_alive() and self.isKernelAlive(): -while not text_queue.empty(): -output = text_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS, - type=ipython_pb2.TEXT, - output=output) -while not html_queue.empty(): -output = html_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS, - type=ipython_pb2.HTML, - output=output) -while not stderr_queue.empty(): -output = stderr_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.ERROR, - type=ipython_pb2.TEXT, - output=output) -while not png_queue.empty(): -output = png_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS, - type=ipython_pb2.PNG, - output=output) -while not jpeg_queue.empty(): -output = jpeg_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS, - type=ipython_pb2.JPEG, - output=output) - -# if kernel is not alive (should be same as thread is still alive), means that we face -# an unexpected issue. -if not self.isKernelAlive() or t.is_alive(): -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.ERROR, -type=ipython_pb2.TEXT, - output="Ipython kernel has been stopped. Please check logs. It might be because of an out of memory issue.") -return - -while not text_queue.empty(): Review comment: Why removing these ? Will this cause data lose if this queue is not empty ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on issue #3337: [ZEPPELIN-4090] Ipython queue performance
AyWa commented on issue #3337: [ZEPPELIN-4090] Ipython queue performance URL: https://github.com/apache/zeppelin/pull/3337#issuecomment-530264254 @zjffdu it was a bit longer than I expected, because I failed to rebased properly, but now everything is alright and ready to be reviewed. I think all test are passing now https://travis-ci.org/AyWa/zeppelin/builds/583492141 (only one is still in progress but should be alright) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa edited a comment on issue #3337: [ZEPPELIN-4090] Ipython queue performance
AyWa edited a comment on issue #3337: [ZEPPELIN-4090] Ipython queue performance URL: https://github.com/apache/zeppelin/pull/3337#issuecomment-529808863 @zjffdu thank you to ping me I am working on it and I just rebased and fixed the new conflict. I will ping you when test are passing This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on issue #3337: [ZEPPELIN-4090] Ipython queue performance
AyWa commented on issue #3337: [ZEPPELIN-4090] Ipython queue performance URL: https://github.com/apache/zeppelin/pull/3337#issuecomment-529808863 @zjffdu thank you to ping me, working on it and just rebase and fixed the new conflict. I will ping you when test are passing This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3337: [ZEPPELIN-4090] Ipython queue performance
zjffdu commented on issue #3337: [ZEPPELIN-4090] Ipython queue performance URL: https://github.com/apache/zeppelin/pull/3337#issuecomment-529806141 ping @AyWa This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] asfgit closed pull request #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython
asfgit closed pull request #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython
zjffdu commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-529109567 Thanks, I will merge it soon This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython
AyWa commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-529109440 Yeah, I don't know why it is not shown in the pr. It was shown until I fix the conflict yesterday. https://travis-ci.org/AyWa/zeppelin/builds/581966561 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython
zjffdu commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-529109028 @AyWa where is the travis link ? Have you configured your travis ? It is weird that there's on travis CI link here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython
AyWa commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-529085991 I rebased and test succeeded. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython
zjffdu commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-529064509 @AyWa There's some flaky test, let me know if you doubt the failed test is not related This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython
AyWa commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-528911244 Yes, I just could not see the trigger. But I checked and the build fail. I will have a look This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython
zjffdu commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-528759253 Did you configure your travis ? https://zeppelin.apache.org/contribution/contributions.html#continuous-integration This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython
AyWa commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-528757868 No problem :) by the way, the CI changed ? do I need to setup something ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython
zjffdu commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-528757130 Thanks @AyWa , I am sorry that I forget to merge it. I will merge it after the CI completed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython
AyWa commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-528752013 @zjffdu I rebased because it has conflict. i think it was LGTM but not merge. Shall I work on something to make it to master ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (ZEPPELIN-4118) please document all requirements for ipython in 0.8.0 documentation
Charles Hedrick created ZEPPELIN-4118: - Summary: please document all requirements for ipython in 0.8.0 documentation Key: ZEPPELIN-4118 URL: https://issues.apache.org/jira/browse/ZEPPELIN-4118 Project: Zeppelin Issue Type: Bug Components: documentation Affects Versions: 0.8.0 Reporter: Charles Hedrick I'm using 0.8.0, with Hortonworks. I had been unable to get ipython to work. The logs are not helpful. The ipython just exits, status 1. It turns out I was missing protobuf, a requirement that is not documented. The copy in master checks for ipykernel, jupyter-client, ipython, grpcio, and protobuf, and warns if they aren't present. The 0.8.0 version does not, at least not for protobuf. However a prominent warning in the documentation would help a lot. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ZEPPELIN-4114) The Python interpreter crashes when Conda is used in combination with iPython in Zeppelin v0.8.1
Peter Moorthamer created ZEPPELIN-4114:
--
Summary: The Python interpreter crashes when Conda is used in
combination with iPython in Zeppelin v0.8.1
Key: ZEPPELIN-4114
URL: https://issues.apache.org/jira/browse/ZEPPELIN-4114
Project: Zeppelin
Issue Type: Bug
Components: python-interpreter
Affects Versions: 0.8.1
Reporter: Peter Moorthamer
Running a notebook with the following 2 paragraphs will cause a crash of the
Python interpreter:
{{%python.conda}}
{{activate base}}
{{%python.ipython}}
{{import sys}}
{{print(sys.version)}}
Exception:
io.grpc.StatusRuntimeException: UNAVAILABLE at
io.grpc.Status.asRuntimeException(Status.java:543) at
io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:395)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426) at
io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:76) at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:512)
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:429)
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:544)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:52) at
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:117) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) Caused by:
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused:
/127.0.0.1:43307 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at
io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352)
at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496) at
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) at
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
... 1 more
Caused by: java.net.ConnectException: Connection refused ... 11 more
None of the Python notebooks are working after this exception has occurred.
The same error can be reproduced in the standard Zeppelin 0.8.1 installation
and in the Zeppelin 0.8.1 installation with Docker.
The grpcio Python package is causing the trouble.
When installing the iPython dependencies (including grpcio) on Zeppelin 0.8.0,
the same issue is introduced.
We really need this functionality. Can you please try to fix this issue.
Thanks.
We have done some testing with different versions of the grpicio Python package
but none of them are working.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [zeppelin] AyWa commented on issue #3337: [ZEPPELIN-4090] Ipython queue performance
AyWa commented on issue #3337: [ZEPPELIN-4090] Ipython queue performance URL: https://github.com/apache/zeppelin/pull/3337#issuecomment-481575952 Will try to rebase and fix this pr in the coming week :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython
zjffdu commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-479715787 LGTM, will merge if no more comment This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython
AyWa commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-479702958 Should I do something else on this side ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa edited a comment on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython
AyWa edited a comment on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-478428352 @zjffdu Thank you, the OOM fix for the test is working well: https://travis-ci.org/AyWa/zeppelin/builds/513921618 (one test was failed but second time works fine) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython
AyWa commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-478428352 @zjffdu Thank you, the OOM fix for the test is working well: https://travis-ci.org/AyWa/zeppelin/builds/513921618 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu removed a comment on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython
zjffdu removed a comment on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-478341454 @AyWa I have fixed OOM issue in https://github.com/apache/zeppelin/commit/1a8667c655bb848fd2d07e1fe94092190454a5dd You can try rebase it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython
zjffdu commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-478341426 @AyWa I have fixed OOM issue in https://github.com/apache/zeppelin/commit/1a8667c655bb848fd2d07e1fe94092190454a5dd You can try rebase it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython
zjffdu commented on issue #3336: [ZEPPELIN-4091] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-478341454 @AyWa I have fixed OOM issue in https://github.com/apache/zeppelin/commit/1a8667c655bb848fd2d07e1fe94092190454a5dd You can try rebase it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] asfgit closed pull request #3339: [ZEPPELIN-4089] handle ipython kernel crash
asfgit closed pull request #3339: [ZEPPELIN-4089] handle ipython kernel crash URL: https://github.com/apache/zeppelin/pull/3339 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (ZEPPELIN-4091) Ipython hang when concurrent auto complete / run
marc hurabielle created ZEPPELIN-4091: - Summary: Ipython hang when concurrent auto complete / run Key: ZEPPELIN-4091 URL: https://issues.apache.org/jira/browse/ZEPPELIN-4091 Project: Zeppelin Issue Type: Sub-task Reporter: marc hurabielle Assignee: marc hurabielle Ipython / ipython server has a problem when concurrent execution and auto complete are execute in same time / parallel. It can make a paragraph hang forever (until restart of the ipython server). Maybe related to [https://github.com/jupyter/jupyter_client/issues/429] Overall most of those bug might be related also to some jupyter_client bug or wrong usage. However, those are the action item: * synchronize auto complete / paragraph execution for now -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ZEPPELIN-4090) Ipython CPU / queue improvement
marc hurabielle created ZEPPELIN-4090: - Summary: Ipython CPU / queue improvement Key: ZEPPELIN-4090 URL: https://issues.apache.org/jira/browse/ZEPPELIN-4090 Project: Zeppelin Issue Type: Sub-task Reporter: marc hurabielle Assignee: marc hurabielle Ipython / ipython server has currently a problem of High cpu usage. The loop that read from the pub/sub should not try to read everytimes the pub sub. It Needs to be debounce. those are the action item: * sleep time to time when there is no message in pub/sub * Use only one queue instead of 3 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [zeppelin] AyWa commented on issue #3339: [ZEPPELIN-4089] handle ipython kernel crash
AyWa commented on issue #3339: [ZEPPELIN-4089] handle ipython kernel crash URL: https://github.com/apache/zeppelin/pull/3339#issuecomment-476569603 @zjffdu updated the jira for this pr. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (ZEPPELIN-4089) Execution hang after ipython kernel die
marc hurabielle created ZEPPELIN-4089: - Summary: Execution hang after ipython kernel die Key: ZEPPELIN-4089 URL: https://issues.apache.org/jira/browse/ZEPPELIN-4089 Project: Zeppelin Issue Type: Sub-task Reporter: marc hurabielle Assignee: marc hurabielle When an ipython paragraph is running, and the ipython kernel is dying. (like an out of memory), the interpreter will hang and never complete the execution. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [zeppelin] AyWa commented on issue #3339: [ZEPPELIN-4078] handle ipython kernel crash
AyWa commented on issue #3339: [ZEPPELIN-4078] handle ipython kernel crash URL: https://github.com/apache/zeppelin/pull/3339#issuecomment-476549536 Yeah sure. I will create subtask for my 3 pr. (for now they all share the same jira issue ) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu edited a comment on issue #3339: [ZEPPELIN-4078] handle ipython kernel crash
zjffdu edited a comment on issue #3339: [ZEPPELIN-4078] handle ipython kernel crash URL: https://github.com/apache/zeppelin/pull/3339#issuecomment-476545011 @AyWa Before I merge it, could you create a dedicated ticket for this PR ? I found you have 2 PRs for the same ticket ZEPPELIN-4078 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3339: [ZEPPELIN-4078] handle ipython kernel crash
zjffdu commented on issue #3339: [ZEPPELIN-4078] handle ipython kernel crash URL: https://github.com/apache/zeppelin/pull/3339#issuecomment-476545011 @AyWa Before I merge it, could you create a dedicated ticket for this PR. I found you have 2 PRs for the same ticket ZEPPELIN-4078 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on a change in pull request #3337: [ZEPPELIN-4078] Ipython queue performance
AyWa commented on a change in pull request #3337: [ZEPPELIN-4078] Ipython queue
performance
URL: https://github.com/apache/zeppelin/pull/3337#discussion_r268481246
##
File path: python/src/main/resources/grpc/python/ipython_server.py
##
@@ -52,24 +52,19 @@ def execute(self, request, context):
print("execute code:\n")
print(request.code.encode('utf-8'))
sys.stdout.flush()
-stdout_queue = queue.Queue(maxsize = 10)
-stderr_queue = queue.Queue(maxsize = 10)
-image_queue = queue.Queue(maxsize = 5)
-
+stream_reply_queue = queue.Queue(maxsize = 20)
Review comment:
When the queue is full , Add element to the queue is a blocking call, so
message will just be delay.
It could be also related to how the `execute_interactive` from
`jupyter_client` is implemented. But there is no timeout so far:
https://github.com/jupyter/jupyter_client/blob/44980c13680f4e4226cf25f199ce4e4bb6e11296/jupyter_client/blocking/client.py#L338
and they are using queue under the hood too. So even with many message to
handle, it will just bring some slow down.
(relative slow down, because `while not stream_reply_queue.empty():` is
likely to never stop when there is a lot of message to process.) So thats why
there is at most `0.05sc` delay compare to before. (but with less cpu usage, so
likely to be faster at the end)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3339: [ZEPPELIN-4078] handle ipython kernel crash
zjffdu commented on issue #3339: [ZEPPELIN-4078] handle ipython kernel crash URL: https://github.com/apache/zeppelin/pull/3339#issuecomment-475966122 Thanks @AyWa The unit test is very helpful This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on issue #3339: [ZEPPELIN-4078] handle ipython kernel crash
AyWa commented on issue #3339: [ZEPPELIN-4078] handle ipython kernel crash URL: https://github.com/apache/zeppelin/pull/3339#issuecomment-475933364 @zjffdu Hi, I just found a way to test it: https://github.com/apache/zeppelin/pull/3339/commits/f37d5f95dcaab07fdab08d9e1f03265896d02e68 I find the kernel pid and kill it from the "running" paragraph. the test succeed on this branch but hang / fail on master. This is the only way I found (without docker) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] felixcheung commented on a change in pull request #3337: [ZEPPELIN-4078] Ipython queue performance
felixcheung commented on a change in pull request #3337: [ZEPPELIN-4078]
Ipython queue performance
URL: https://github.com/apache/zeppelin/pull/3337#discussion_r268418796
##
File path: python/src/main/resources/grpc/python/ipython_server.py
##
@@ -52,24 +52,19 @@ def execute(self, request, context):
print("execute code:\n")
print(request.code.encode('utf-8'))
sys.stdout.flush()
-stdout_queue = queue.Queue(maxsize = 10)
-stderr_queue = queue.Queue(maxsize = 10)
-image_queue = queue.Queue(maxsize = 5)
-
+stream_reply_queue = queue.Queue(maxsize = 20)
Review comment:
ok, what happens when the queue is full?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] AyWa commented on issue #3339: [ZEPPELIN-4078] handle ipython kernel crash
AyWa commented on issue #3339: [ZEPPELIN-4078] handle ipython kernel crash URL: https://github.com/apache/zeppelin/pull/3339#issuecomment-475551747 Just think that maybe we could just kill the ipython kernel, like finding pid. Might be easier for unit test ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on issue #3336: [ZEPPELIN-4078] Fix concurrent autocomplete and execute for Ipython
AyWa commented on issue #3336: [ZEPPELIN-4078] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-475549925 I am kind of lost with the test...: - I ran my fix, without the new test -> all the test pass on the ci - I ran my fix, with the new test - > always some failure (seems like `GC or out of memory or timeout) - I ran my fix, with a test, that is not testing the fix (like no concurrent call) -> fail again So i think the problem is not related to the fix itself, but more like an issue with the test: - `ExecutorService pool = Executors.newFixedThreadPool(2); `make the GC / OOM` ? - running then auto complete make some issue ? I will try to push the same code without executor / thread to see if it is working, then try to thread it again :/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on issue #3339: [ZEPPELIN-4078] handle ipython kernel crash
AyWa commented on issue #3339: [ZEPPELIN-4078] handle ipython kernel crash URL: https://github.com/apache/zeppelin/pull/3339#issuecomment-475520540 Yeah indeed, it require docker to do so :/ Before using docker I tried to so similar things but without success. I think the problem is that the memory issue is coming from the ipython kernel. (so it will not be the same process as far as I understand and I didnt find option to set memory limit in the kernel) However let me try with the link you shared. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3339: [ZEPPELIN-4078] handle ipython kernel crash
zjffdu commented on issue #3339: [ZEPPELIN-4078] handle ipython kernel crash URL: https://github.com/apache/zeppelin/pull/3339#issuecomment-475519159 Thanks @AyWa , but it seems not easy to do unit test. I find this post which might be helpful. Do you mind to try this ? https://stackoverflow.com/questions/16779497/how-to-set-memory-limit-for-thread-or-process-in-python This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on issue #3339: [ZEPPELIN-4078] handle ipython kernel crash
AyWa commented on issue #3339: [ZEPPELIN-4078] handle ipython kernel crash URL: https://github.com/apache/zeppelin/pull/3339#issuecomment-475516201 @zjffdu Hi, I just push an example with docker. https://github.com/apache/zeppelin/commit/c9ec0335cd2fd93476cee31ba99b6dc157face0d So if you run that: in `zeppelin/python/src/main/resources/grpc` run `docker build -t my-python-app . && docker run -m 1g --memory-swap 1g -p 50053:50053 my-python-app` It will install the dependencies of the zeppelin ipython server, and then run the grpc server on port `50053`. And we put a docker limit of `1g` so we can test the `OOM` Then run in `zeppelin/python/src/main/resources/grpc`: `python3 python/ipython_client_OOM.py` With this pr you should see something like: (both client and server terminate) ``` output:14 output:15 output:16 output:17 output:18 output:Ipython kernel has been stopped. Please check logs. It might be because of an out of memory issue. status: ERROR output: "Ipython kernel has been stopped. Please check logs. It might be because of an out of memory issue." ``` On master you will have: (both client and server hanging) ``` output:18 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on issue #3339: [ZEPPELIN-4078] handle ipython kernel crash
AyWa commented on issue #3339: [ZEPPELIN-4078] handle ipython kernel crash URL: https://github.com/apache/zeppelin/pull/3339#issuecomment-475257022 Yeah an out of memory will do the trick. I Could reproduce with a docker running the ipython server with a memory limit. I will push it tomorrow. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3339: [ZEPPELIN-4078] handle ipython kernel crash
zjffdu commented on issue #3339: [ZEPPELIN-4078] handle ipython kernel crash URL: https://github.com/apache/zeppelin/pull/3339#issuecomment-475236454 @AyWa Do you know how to create such scenario (ipython server not die but ipython kernel die) ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on issue #3339: [ZEPPELIN-4078] handle ipython kernel crash
AyWa commented on issue #3339: [ZEPPELIN-4078] handle ipython kernel crash URL: https://github.com/apache/zeppelin/pull/3339#issuecomment-475169528 I want to bring unit test for that. However the easiest way is to use a docker image with a limit of memory. Wonder if it will be easy to intergrate with the current ci / test pipeline. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on a change in pull request #3339: [ZEPPELIN-4078] handle ipython kernel crash
AyWa commented on a change in pull request #3339: [ZEPPELIN-4078] handle ipython kernel crash URL: https://github.com/apache/zeppelin/pull/3339#discussion_r267684636 ## File path: python/src/main/resources/grpc/python/ipython_server.py ## @@ -146,10 +160,16 @@ def serve(port): server.start() ipython.start() try: -while True: -time.sleep(_ONE_DAY_IN_SECONDS) +while ipython.isKernelAlive(): Review comment: We have multiple options when we figure out that the kernel die: - return an error then kill the process (this pr is doing that) - restart the kernel ? The challenge with 1, is that we maybe need to add a new status to the `status` grpc method. like `shutdown`. We can check this status before running like in the `checkIpythonProcessAndRestartIfDead` The challenge with 2 is that our execute thread will always be stuck. and we might have no way to kill it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa opened a new pull request #3339: [ZEPPELIN-4078] handle ipython kernel crash
AyWa opened a new pull request #3339: [ZEPPELIN-4078] handle ipython kernel crash URL: https://github.com/apache/zeppelin/pull/3339 ### What is this PR for? The pr is the last fix of the `ZEPPELIN-4078` It will aims to notify user when ipython kernel crash. Indeed, right now, execution will be stuck forever when ipython kernel crash. ### What type of PR is it? Bug Fix ### Todos * [x] - logic to check kernel status * [ ] - decide what to do when we spot the failure * [ ] - test ### What is the Jira issue? It is one part of the jira issue. Other fix will come soon https://issues.apache.org/jira/browse/ZEPPELIN-4078 ### How should this be tested? * First time? Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration * Strongly recommended: add automated unit tests for any new or changed behavior * Outline any manual steps to test the PR here. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on issue #3336: [ZEPPELIN-4078] Fix concurrent autocomplete and execute for Ipython
AyWa commented on issue #3336: [ZEPPELIN-4078] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-475155124 Okay thx. I will try to remove my test to see if the instability is coming from my test or the fix. lets see. :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3336: [ZEPPELIN-4078] Fix concurrent autocomplete and execute for Ipython
zjffdu commented on issue #3336: [ZEPPELIN-4078] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-475145728 @AyWa You can rebuild it again, if it still happens, then most likely due to your PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on issue #3336: [ZEPPELIN-4078] Fix concurrent autocomplete and execute for Ipython
AyWa commented on issue #3336: [ZEPPELIN-4078] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-475143484 @Leemoonsoo Seems like my ci is still failing. I got error: ``` ests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 818.986 sec <<< FAILURE! - in org.apache.zeppelin.spark.IPySparkInterpreterTest testZeppelinContext(org.apache.zeppelin.spark.IPySparkInterpreterTest) Time elapsed: 72.083 sec <<< ERROR! java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.jar.Attributes.read(Attributes.java:394) at java.util.jar.Manifest.read(Manifest.java:199) at java.util.jar.Manifest.(Manifest.java:69) at java.util.jar.JarFile.getManifestFromReference(JarFile.java:199) at java.util.jar.JarFile.getManifest(JarFile.java:180) at sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:981) at java.net.URLClassLoader.defineClass(URLClassLoader.java:450) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at scala.tools.nsc.interpreter.IMain.scala$tools$nsc$interpreter$IMain$$_initialize(IMain.scala:138) at scala.tools.nsc.interpreter.IMain.initializeSynchronous(IMain.scala:154) at org.apache.zeppelin.spark.SparkScala211Interpreter.open(SparkScala211Interpreter.scala:85) at org.apache.zeppelin.spark.NewSparkInterpreter.open(NewSparkInterpreter.java:106) at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:66) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69) at org.apache.zeppelin.interpreter.Interpreter.getInterpreterInTheSameSessionByClassName(Interpreter.java:354) at org.apache.zeppelin.interpreter.Interpreter.getInterpreterInTheSameSessionByClassName(Interpreter.java:365) at org.apache.zeppelin.spark.IPySparkInterpreter.open(IPySparkInterpreter.java:52) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69) at org.apache.zeppelin.spark.IPySparkInterpreterTest.startInterpreter(IPySparkInterpreterTest.java:93) ``` Does it happens sometimes ? or is it related to my pr ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa commented on a change in pull request #3337: [ZEPPELIN-4078] Ipython queue performance
AyWa commented on a change in pull request #3337: [ZEPPELIN-4078] Ipython queue
performance
URL: https://github.com/apache/zeppelin/pull/3337#discussion_r267648007
##
File path: python/src/main/resources/grpc/python/ipython_server.py
##
@@ -52,24 +52,19 @@ def execute(self, request, context):
print("execute code:\n")
print(request.code.encode('utf-8'))
sys.stdout.flush()
-stdout_queue = queue.Queue(maxsize = 10)
-stderr_queue = queue.Queue(maxsize = 10)
-image_queue = queue.Queue(maxsize = 5)
-
+stream_reply_queue = queue.Queue(maxsize = 20)
Review comment:
What about 30 ?
The queue is blocking if full, so should be fine isnt ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] AyWa commented on a change in pull request #3337: [ZEPPELIN-4078] Ipython queue performance
AyWa commented on a change in pull request #3337: [ZEPPELIN-4078] Ipython queue
performance
URL: https://github.com/apache/zeppelin/pull/3337#discussion_r267647327
##
File path: python/src/main/resources/grpc/python/ipython_server.py
##
@@ -52,24 +52,19 @@ def execute(self, request, context):
print("execute code:\n")
print(request.code.encode('utf-8'))
sys.stdout.flush()
-stdout_queue = queue.Queue(maxsize = 10)
-stderr_queue = queue.Queue(maxsize = 10)
-image_queue = queue.Queue(maxsize = 5)
-
+stream_reply_queue = queue.Queue(maxsize = 20)
+payload_reply = []
def _output_hook(msg):
msg_type = msg['header']['msg_type']
content = msg['content']
if msg_type == 'stream':
-stdout_queue.put(content['text'])
+
stream_reply_queue.put(ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS,
type=ipython_pb2.TEXT, output=content['text']))
+elif msg_type == 'error':
+
stream_reply_queue.put(ipython_pb2.ExecuteResponse(status=ipython_pb2.ERROR,
type=ipython_pb2.TEXT, output='\n'.join(content['traceback'])))
elif msg_type in ('display_data', 'execute_result'):
-stdout_queue.put(content['data'].get('text/plain', ''))
+
stream_reply_queue.put(ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS,
type=ipython_pb2.TEXT, output=content['data'].get('text/plain', '')))
if 'image/png' in content['data']:
-image_queue.put(content['data']['image/png'])
-elif msg_type == 'error':
-stderr_queue.put('\n'.join(content['traceback']))
-
-
-payload_reply = []
+
stream_reply_queue.put(ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS,
type=ipython_pb2.IMAGE, output=content['data']['image/png']))
Review comment:
Sure let me update that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] AyWa commented on a change in pull request #3336: [ZEPPELIN-4078] Fix concurrent autocomplete and execute for Ipython
AyWa commented on a change in pull request #3336: [ZEPPELIN-4078] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#discussion_r267639690 ## File path: python/src/main/resources/grpc/python/ipython_server.py ## @@ -72,29 +76,30 @@ def _output_hook(msg): payload_reply = [] def execute_worker(): reply = self._kc.execute_interactive(request.code, -output_hook=_output_hook, -timeout=TIMEOUT) +output_hook=_output_hook, +timeout=TIMEOUT) payload_reply.append(reply) t = threading.Thread(name="ConsumerThread", target=execute_worker) -t.start() - -while t.is_alive(): -while not stdout_queue.empty(): -output = stdout_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS, - type=ipython_pb2.TEXT, - output=output) -while not stderr_queue.empty(): -output = stderr_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.ERROR, - type=ipython_pb2.TEXT, - output=output) -while not image_queue.empty(): -output = image_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS, - type=ipython_pb2.IMAGE, - output=output) +with self._lock: Review comment: Indeed, it will. But this is the expected behavior. Right now, if you do an other request to the kernel (auto-complete), the `execute_interactive` will never complete. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] felixcheung commented on a change in pull request #3336: [ZEPPELIN-4078] Fix concurrent autocomplete and execute for Ipython
felixcheung commented on a change in pull request #3336: [ZEPPELIN-4078] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#discussion_r267638285 ## File path: python/src/main/resources/grpc/python/ipython_server.py ## @@ -72,29 +76,30 @@ def _output_hook(msg): payload_reply = [] def execute_worker(): reply = self._kc.execute_interactive(request.code, -output_hook=_output_hook, -timeout=TIMEOUT) +output_hook=_output_hook, +timeout=TIMEOUT) payload_reply.append(reply) t = threading.Thread(name="ConsumerThread", target=execute_worker) -t.start() - -while t.is_alive(): -while not stdout_queue.empty(): -output = stdout_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS, - type=ipython_pb2.TEXT, - output=output) -while not stderr_queue.empty(): -output = stderr_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.ERROR, - type=ipython_pb2.TEXT, - output=output) -while not image_queue.empty(): -output = image_queue.get() -yield ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS, - type=ipython_pb2.IMAGE, - output=output) +with self._lock: Review comment: isn't this going to lock the entire time the thread is running? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] felixcheung commented on a change in pull request #3337: [ZEPPELIN-4078] Ipython queue performance
felixcheung commented on a change in pull request #3337: [ZEPPELIN-4078]
Ipython queue performance
URL: https://github.com/apache/zeppelin/pull/3337#discussion_r267638063
##
File path: python/src/main/resources/grpc/python/ipython_server.py
##
@@ -52,24 +52,19 @@ def execute(self, request, context):
print("execute code:\n")
print(request.code.encode('utf-8'))
sys.stdout.flush()
-stdout_queue = queue.Queue(maxsize = 10)
-stderr_queue = queue.Queue(maxsize = 10)
-image_queue = queue.Queue(maxsize = 5)
-
+stream_reply_queue = queue.Queue(maxsize = 20)
Review comment:
should maxsize be a bit more?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] felixcheung commented on a change in pull request #3337: [ZEPPELIN-4078] Ipython queue performance
felixcheung commented on a change in pull request #3337: [ZEPPELIN-4078]
Ipython queue performance
URL: https://github.com/apache/zeppelin/pull/3337#discussion_r267637879
##
File path: python/src/main/resources/grpc/python/ipython_server.py
##
@@ -52,24 +52,19 @@ def execute(self, request, context):
print("execute code:\n")
print(request.code.encode('utf-8'))
sys.stdout.flush()
-stdout_queue = queue.Queue(maxsize = 10)
-stderr_queue = queue.Queue(maxsize = 10)
-image_queue = queue.Queue(maxsize = 5)
-
+stream_reply_queue = queue.Queue(maxsize = 20)
+payload_reply = []
def _output_hook(msg):
msg_type = msg['header']['msg_type']
content = msg['content']
if msg_type == 'stream':
-stdout_queue.put(content['text'])
+
stream_reply_queue.put(ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS,
type=ipython_pb2.TEXT, output=content['text']))
+elif msg_type == 'error':
+
stream_reply_queue.put(ipython_pb2.ExecuteResponse(status=ipython_pb2.ERROR,
type=ipython_pb2.TEXT, output='\n'.join(content['traceback'])))
elif msg_type in ('display_data', 'execute_result'):
-stdout_queue.put(content['data'].get('text/plain', ''))
+
stream_reply_queue.put(ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS,
type=ipython_pb2.TEXT, output=content['data'].get('text/plain', '')))
if 'image/png' in content['data']:
-image_queue.put(content['data']['image/png'])
-elif msg_type == 'error':
-stderr_queue.put('\n'.join(content['traceback']))
-
-
-payload_reply = []
+
stream_reply_queue.put(ipython_pb2.ExecuteResponse(status=ipython_pb2.SUCCESS,
type=ipython_pb2.IMAGE, output=content['data']['image/png']))
Review comment:
this is a bit long and hard to read? consider refactor
`stream_reply_queue.put` into a separate line
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] AyWa commented on issue #3336: [ZEPPELIN-4078] Fix concurrent autocomplete and execute for Ipython
AyWa commented on issue #3336: [ZEPPELIN-4078] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-475073644 @Leemoonsoo Thx you for the info, I guess it was because of `new Properties()`. i pushed a changed to use `initIntpProperties()` in the test. Let's hope it will pass 爛 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] Leemoonsoo edited a comment on issue #3336: [ZEPPELIN-4078] Fix concurrent autocomplete and execute for Ipython
Leemoonsoo edited a comment on issue #3336: [ZEPPELIN-4078] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-474924351 Thanks @AyWa for the contribution. A test is failing with ``` 13:09:18,331 INFO org.apache.zeppelin.spark.OldSparkInterpreter:338 - -- Create new SparkContext local --- 13:09:18,335 INFO org.apache.spark.SparkContext:58 - Running Spark version 1.6.3 13:09:18,338 ERROR org.apache.spark.SparkContext:95 - Error initializing SparkContext. org.apache.spark.SparkException: An application name must be set in your configuration at org.apache.spark.SparkContext.(SparkContext.scala:404) at org.apache.zeppelin.spark.OldSparkInterpreter.createSparkContext_1(OldSparkInterpreter.java:426) at org.apache.zeppelin.spark.OldSparkInterpreter.createSparkContext(OldSparkInterpreter.java:321) at org.apache.zeppelin.spark.OldSparkInterpreter.getSparkContext(OldSparkInterpreter.java:139) at org.apache.zeppelin.spark.OldSparkInterpreter.open(OldSparkInterpreter.java:696) at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:66) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69) at org.apache.zeppelin.interpreter.Interpreter.getInterpreterInTheSameSessionByClassName(Interpreter.java:354) at org.apache.zeppelin.interpreter.Interpreter.getInterpreterInTheSameSessionByClassName(Interpreter.java:365) at org.apache.zeppelin.spark.IPySparkInterpreter.open(IPySparkInterpreter.java:52) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69) at org.apache.zeppelin.spark.IPySparkInterpreterTest.startInterpreter(IPySparkInterpreterTest.java:93) at org.apache.zeppelin.python.IPythonInterpreterTest.testIpython_shouldNotHang_whenCallingAutoCompleteAndInterpretConcurrently(IPythonInterpreterTest.java:250) ``` Looks like https://github.com/apache/zeppelin/blob/master/spark/interpreter/src/test/java/org/apache/zeppelin/spark/IPySparkInterpreterTest.java#L59 is somehow not applied on the test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] Leemoonsoo commented on issue #3336: [ZEPPELIN-4078] Fix concurrent autocomplete and execute for Ipython
Leemoonsoo commented on issue #3336: [ZEPPELIN-4078] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-474924351 Thanks @AyWa for the contribution. Looks like a test is failing with ``` 13:09:18,331 INFO org.apache.zeppelin.spark.OldSparkInterpreter:338 - -- Create new SparkContext local --- 13:09:18,335 INFO org.apache.spark.SparkContext:58 - Running Spark version 1.6.3 13:09:18,338 ERROR org.apache.spark.SparkContext:95 - Error initializing SparkContext. org.apache.spark.SparkException: An application name must be set in your configuration at org.apache.spark.SparkContext.(SparkContext.scala:404) at org.apache.zeppelin.spark.OldSparkInterpreter.createSparkContext_1(OldSparkInterpreter.java:426) at org.apache.zeppelin.spark.OldSparkInterpreter.createSparkContext(OldSparkInterpreter.java:321) at org.apache.zeppelin.spark.OldSparkInterpreter.getSparkContext(OldSparkInterpreter.java:139) at org.apache.zeppelin.spark.OldSparkInterpreter.open(OldSparkInterpreter.java:696) at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:66) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69) at org.apache.zeppelin.interpreter.Interpreter.getInterpreterInTheSameSessionByClassName(Interpreter.java:354) at org.apache.zeppelin.interpreter.Interpreter.getInterpreterInTheSameSessionByClassName(Interpreter.java:365) at org.apache.zeppelin.spark.IPySparkInterpreter.open(IPySparkInterpreter.java:52) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69) at org.apache.zeppelin.spark.IPySparkInterpreterTest.startInterpreter(IPySparkInterpreterTest.java:93) at org.apache.zeppelin.python.IPythonInterpreterTest.testIpython_shouldNotHang_whenCallingAutoCompleteAndInterpretConcurrently(IPythonInterpreterTest.java:250) ``` Looks like https://github.com/apache/zeppelin/blob/master/spark/interpreter/src/test/java/org/apache/zeppelin/spark/IPySparkInterpreterTest.java#L59 is somehow not applied on the test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3336: [ZEPPELIN-4078] Fix concurrent autocomplete and execute for Ipython
zjffdu commented on issue #3336: [ZEPPELIN-4078] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336#issuecomment-474766783 Thanks @AyWa LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa opened a new pull request #3337: [ZEPPELIN-4078] Ipython queue performance
AyWa opened a new pull request #3337: [ZEPPELIN-4078] Ipython queue performance URL: https://github.com/apache/zeppelin/pull/3337 ### What is this PR for? The pr is to fix a bug that will make the **ipython** queue listener, overuse cpu. After this fix, cpu usage should be way lower. Also there is a bit of refactor to use only one queue to ensure emssage will be order even with a sleep. ### What type of PR is it? Bug Fix / performance improvement ### Todos * [x] - Performance improvement * [ ] - Need to add some performance test ? or other test ? ### What is the Jira issue? It is one part of the jira issue. https://issues.apache.org/jira/browse/ZEPPELIN-4078 ### How should this be tested? * First time? Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration * Strongly recommended: add automated unit tests for any new or changed behavior * Outline any manual steps to test the PR here. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] AyWa opened a new pull request #3336: [ZEPPELIN-4078] Fix concurrent autocomplete and execute for Ipython
AyWa opened a new pull request #3336: [ZEPPELIN-4078] Fix concurrent autocomplete and execute for Ipython URL: https://github.com/apache/zeppelin/pull/3336 ### What is this PR for? The pr is to fix a bug that will make the **ipython** `execute_interactive` hang forever if a auto `complete` call is make at the same time. (see unit test for example that is failing on master). For now the fix is to synchronize those method : `execute` / `complete`. It will not bring regression because anyway, the kernel does not support concurrent execute and auto complete (see https://github.com/jupyter/notebook/issues/3763) ### What type of PR is it? Bug Fix ### Todos * [x] - unit test failing in master / succeed on this branch * [x] - fix with lock ### What is the Jira issue? It is one part of the jira issue. Other fix will come soon https://issues.apache.org/jira/browse/ZEPPELIN-4078 ### How should this be tested? * First time? Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration * Strongly recommended: add automated unit tests for any new or changed behavior * Outline any manual steps to test the PR here. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (ZEPPELIN-4078) Ipython improvement
marc hurabielle created ZEPPELIN-4078: - Summary: Ipython improvement Key: ZEPPELIN-4078 URL: https://issues.apache.org/jira/browse/ZEPPELIN-4078 Project: Zeppelin Issue Type: Bug Components: python-interpreter Affects Versions: 0.8.1 Environment: tested in linux / macos. Tested the ipython server directly. Reporter: marc hurabielle Ipython / ipython server has currently mutliple problem. * Concurrent execution and auto complete can make a paragraph hang forever (until restart of the ipython server). Maybe related to [https://github.com/jupyter/jupyter_client/issues/429] * Out of memory for ipython will make a paragraph hang instead of throw one error. * High cpu usage. The loop that read from the pub/sub should not try to read everytimes the pub sub. Need to be debounce. Overall most of those bug might be related also to some jupyter_client bug or wrong usage. However, those are the action item: * synchronize auto complete / paragraph execution for now * check the kernel status when we are checking for the thread * sleep time to time when there is no message in pub/sub * Use only one queue instead of 3 ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [zeppelin] asfgit closed pull request #3332: ZEPPELIN-4062. Don't wait ipython kernel if python process failed
asfgit closed pull request #3332: ZEPPELIN-4062. Don't wait ipython kernel if python process failed URL: https://github.com/apache/zeppelin/pull/3332 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (ZEPPELIN-4065) Display ipython process launch output when ipython fails to launch
Jeff Zhang created ZEPPELIN-4065: Summary: Display ipython process launch output when ipython fails to launch Key: ZEPPELIN-4065 URL: https://issues.apache.org/jira/browse/ZEPPELIN-4065 Project: Zeppelin Issue Type: Bug Reporter: Jeff Zhang Currently, user have to change log4j.properties to change the log level to debug to see this python output, this is not convenient for users, it would be better to display the launch output in frontend by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [zeppelin] zjffdu opened a new pull request #3332: ZEPPELIN-4062. Don't wait ipython kernel if python process failed
zjffdu opened a new pull request #3332: ZEPPELIN-4062. Don't wait ipython
kernel if python process failed
URL: https://github.com/apache/zeppelin/pull/3332
### What is this PR for?
This PR will exist wait ipython process loop if the python process is
failed. This can save time to exit the waiting loop earlier, otherwise it would
wait until timeout (30 seconds by default)
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://jira.apache.org/jira/browse/ZEPPELIN-4062
### How should this be tested?
* Manually tested.
Before this PR
```
INFO [2019-03-13 10:06:30,070] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:06:30,175] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:06:30,281] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:06:30,386] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:06:30,489] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:06:30,595] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:06:30,700] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:06:30,805] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to
be started
WARN [2019-03-13 10:06:30,810] ({Exec Default Executor}
IPythonInterpreter.java[onProcessFailed]:419) - Exception happens in Python
Process
org.apache.commons.exec.ExecuteException: Process exited with an error: 1
(Exit value: 1)
at
org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
at
org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
at
org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
at java.lang.Thread.run(Thread.java:748)
INFO [2019-03-13 10:06:30,910] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:06:31,015] ({FIFOScheduler-Worker-1}
```
After this PR
```
INFO [2019-03-13 10:28:12,805] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:28:12,907] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:28:13,011] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:28:13,116] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:28:13,220] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:28:13,325] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to
be started
WARN [2019-03-13 10:28:13,347] ({Exec Default Executor}
IPythonInterpreter.java[onProcessFailed]:423) - Exception happens in Python
Process
org.apache.commons.exec.ExecuteException: Process exited with an error: 1
(Exit value: 1)
at
org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
at
org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
at
org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
at java.lang.Thread.run(Thread.java:748)
INFO [2019-03-13 10:28:13,430] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:314) - Wait for IPython Kernel to
be started
WARN [2019-03-13 10:28:13,431] ({FIFOScheduler-Worker-1}
PythonInterpreter.java[open]:98) - Fail to open IPythonInterpreter
java.lang.RuntimeException: Fail to open IPythonInterpreter
at
org.apache.zeppelin.python.IPythonInterpreter.open(IPythonInterpreter.java:152)
at
org.apache.zeppelin.python.PythonInterpreter.open(PythonInterpreter.java:93)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:598)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun
[jira] [Created] (ZEPPELIN-4062) Keep waiting ipython kernel when python process failed
Jeff Zhang created ZEPPELIN-4062:
Summary: Keep waiting ipython kernel when python process failed
Key: ZEPPELIN-4062
URL: https://issues.apache.org/jira/browse/ZEPPELIN-4062
Project: Zeppelin
Issue Type: Bug
Reporter: Jeff Zhang
{code}
INFO [2019-03-13 10:06:30,070] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:06:30,175] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:06:30,281] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:06:30,386] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:06:30,489] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:06:30,595] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:06:30,700] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:06:30,805] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to
be started
WARN [2019-03-13 10:06:30,810] ({Exec Default Executor}
IPythonInterpreter.java[onProcessFailed]:419) - Exception happens in Python
Process
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit
value: 1)
at
org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
at
org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
at org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
at java.lang.Thread.run(Thread.java:748)
INFO [2019-03-13 10:06:30,910] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:06:31,015] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:06:31,119] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:06:31,223] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:06:31,329] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to
be started
INFO [2019-03-13 10:06:31,433] ({FIFOScheduler-Worker-1}
IPythonInterpreter.java[launchIPythonKernel]:313) - Wait for IPython Kernel to
be started
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: ipython
Great question. with Zeppelin-0.8.x, IPython server (in Python interpreter) can run inside its own docker container. You can check http://zeppelin.apache.org/docs/0.8.1/interpreter/python.html#docker to see how it works. with Zeppelin-0.9.x (not released yet, build from source), all interpreter including Zeppelin itself can run in the container on Kubernetes. You can check http://zeppelin.apache.org/docs/0.9.0-SNAPSHOT/quickstart/kubernetes.html to see how it works. Hope this helps. Thanks, moon On Mon, Feb 4, 2019 at 12:52 PM wrote: > Hi > > Can ipython server run inside its own docker container? How does ipython > server run in Zeppelin container? Does it run inside the Zeppelin > container? > > Thanks >
