Hi, Thanks for this detail debugging. At first, notebookserver doesn't have any clue for this symptom because it's used between browser and zeppelin server.
I don't know why R has stoped unexpectedly. Is there any log related to R? I'm not familiar with R actually. BTW, I'll install R and test it in my local On Tue, May 9, 2017 at 8:29 AM, Pietro Pugni <pietro.pu...@gmail.com> wrote: > I repost this because it didn’t appear on the mailing list board. > > These are the step needed to reproduce the error and to track down the log > message. > > 1) I started a brand new instance of zeppelin issuing: > service zeppelin start > > and started a bash script that tracks down R processes activity. > After running a simple R script from Zeppelin, the R interpreter process > was started: > > Mon May 8 11:27:59 CEST 2017 >>> R started > > 2) I left the browser open and at 12:26:15 I closed the browser. Zeppelin > tracked down the connection being closed: > INFO [2017-05-08 12:26:15,879] ({qtp423031029-60} > NotebookServer.java[onClose]:363) - Closed connection to 127.0.0.1 : > 33798. (1001) null > > 3) At 13:08:00 R was closed. My script returned: > Mon May 8 13:08:00 CEST 2017 >>> R stopped > > This is the output from the interpreter log file (deleted non-useful > lines): > INFO [2017-05-08 11:27:43,632] ({Thread-0} > RemoteInterpreterServer.java[run]:95) > - Starting remote interpreter server on port 45227 > INFO [2017-05-08 11:27:44,600] ({pool-1-thread-3} > RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate > interpreter org.apache.zeppelin.spark.SparkInterpreter > INFO [2017-05-08 11:27:44,624] ({pool-1-thread-3} > RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate > interpreter org.apache.zeppelin.spark.SparkSqlInterpreter > INFO [2017-05-08 11:27:44,629] ({pool-1-thread-3} > RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate > interpreter org.apache.zeppelin.spark.DepInterpreter > INFO [2017-05-08 11:27:44,640] ({pool-1-thread-3} > RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate > interpreter org.apache.zeppelin.spark.PySparkInterpreter > INFO [2017-05-08 11:27:44,643] ({pool-1-thread-3} > RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate > interpreter org.apache.zeppelin.spark.SparkRInterpreter > ... > INFO [2017-05-08 11:28:00,188] ({pool-2-thread-2} > SchedulerFactory.java[jobFinished]:137) > - Job remoteInterpretJob_1494235664723 finished by scheduler > org.apache.zeppelin.spark.SparkRInterpreter2097894179 > DEBUG [2017-05-08 11:28:00,819] ({pool-1-thread-3} > RemoteInterpreterServer.java[resourcePoolGetAll]:911) - Request getAll > from ZeppelinServer > *DEBUG [2017-05-08 13:08:00,187] ({Exec Stream Pumper} > InterpreterOutputStream.java[processLine]:72) - Interpreter output:Error in > handleErrors(returnStatus, conn) : * > *DEBUG [2017-05-08 13:08:00,188] ({Exec Stream Pumper} > InterpreterOutputStream.java[processLine]:72) - Interpreter output: No > status is returned. Java SparkR backend might have failed.* > *DEBUG [2017-05-08 13:08:00,188] ({Exec Stream Pumper} > InterpreterOutputStream.java[processLine]:72) - Interpreter output:Calls: > <Anonymous> -> invokeJava -> handleErrors* > *DEBUG [2017-05-08 13:08:00,188] ({Exec Stream Pumper} > InterpreterOutputStream.java[processLine]:72) - Interpreter > output:Execution halted* > > This is the output from zeppelin log file (it didn't track the R > interpreter failure): > INFO [2017-05-08 11:28:00,221] ({pool-2-thread-2} > NotebookServer.java[afterStatusChange]:2056) > - Job 20170506-145151_1585482989 is finished successfully, status: FINISHED > INFO [2017-05-08 11:28:00,675] ({pool-2-thread-2} > SchedulerFactory.java[jobFinished]:137) > - Job paragraph_1494075111996_-1250116940 finished by scheduler > org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_ > session2130846287 > *INFO [2017-05-08 12:26:15,879] ({qtp423031029-60} > NotebookServer.java[onClose]:363) - Closed connection to 127.0.0.1 : 33798. > (1001) null* > INFO [2017-05-08 12:27:12,126] ({Thread-33} > AbstractValidatingSessionManager.java[validateSessions]:271) > - Validating all active sessions... > INFO [2017-05-08 12:27:12,126] ({Thread-33} > AbstractValidatingSessionManager.java[validateSessions]:304) > - Finished session validation. No sessions were stopped. > > Hope this helps. > Any hints? > > Il giorno 08 mag 2017, alle ore 11:08, Pietro Pugni < > pietro.pu...@gmail.com> ha scritto: > > I know for sure that R process gets killed (or quits) but don't know if > its father process (interpreter.sh) gets killed too. > > I noticed that I can always restart the interpreter on 0.7.1 while > sometimes it was impossible to do on 0.7.0 (I had to manually restart > zeppelin service). Probably that JIRA improved the situation a little bit. > > Now I'm running a bash script that tracks start and stop time of R process > in order to shed some light on this issue. I enabled DEBUG logging in log4j > properties file. > > > Il 6 mag 2017 4:43 PM, "Paul Brenner" <pbren...@placeiq.com> ha scritto: > >> Great work documenting repeatable steps for this hard to nail down >> problem. I see similar problems running the spark (scala) interpreter but >> haven’t been as systematic about hunting down the issue as you. >> >> I do wonder if this is related somehow to https://issues.apache.org/j >> ira/browse/ZEPPELIN-1832 >> <https://share.polymail.io/v1/z/b/NTkwZGRlMzNiZmFi/Go00wlomvjABQNciq78PfdeRmR4K6c4M5l8KsTYGlks2sD4oe9jS7NYIkVZ2KKlntmyN0z2ZbiIFSP59SQpYL0hq_V6k3ZjCvIj_gDhLCD5s9K74YEQl1S5xOyCx0TK-xuhEd59t3p3nhZrhs1xXLJxUEM6PoX1EWAcJswdLQj6oNrNLeE-0uF9D4DZjlMlBWs_aYKvi14I21deKenrCDUCPJccm> >> which just seems to have addressed killing off zombie processes but I’m >> not sure it covered where zombie processes are coming from. Perhaps we need >> to open a ticket for this? >> >> In the mean time if you don’t have the ability to restart zeppelin every >> time you run into this process you can probably just kill the interpreter >> process. I find myself doing that multiple times in an normal work day. >> >> <http://www.placeiq.com/> <http://www.placeiq.com/> >> <http://www.placeiq.com/> Paul Brenner <https://twitter.com/placeiq> >> <https://twitter.com/placeiq> <https://twitter.com/placeiq> >> <https://www.facebook.com/PlaceIQ> <https://www.facebook.com/PlaceIQ> >> <https://www.linkedin.com/company/placeiq> >> <https://www.linkedin.com/company/placeiq> >> DATA SCIENTIST >> *(217) 390-3033 * >> >> <http://www.placeiq.com/2015/05/26/placeiq-named-winner-of-prestigious-2015-oracle-data-cloud-activate-award/> >> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/> >> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/> >> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/> >> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/> >> <http://placeiq.com/2016/03/08/measuring-addressable-tv-campaigns-is-now-possible/> >> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/> >> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/> >> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/> >> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/> >> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/> >> <http://pages.placeiq.com/Location-Data-Accuracy-Whitepaper-Download.html?utm_source=Signature&utm_medium=Email&utm_campaign=AccuracyWP> >> <http://placeiq.com/2016/08/03/placeiq-bolsters-location-intelligence-platform-with-mastercard-insights/> >> <http://placeiq.com/2016/10/26/the-making-of-a-location-data-industry-milestone/>[image: >> PlaceIQ:Location Data Accuracy] >> <http://placeiq.com/2016/12/07/placeiq-introduces-landmark-a-groundbreaking-offering-that-delivers-access-to-the-highest-quality-location-data-for-insights-that-fuel-limitless-business-decisions/> >> >> On Sat, May 06, 2017 at 6:47 AM Pietro Pugni <Pietro Pugni >> <pietro+pugni+%3cpietro.pu...@gmail.com%3E>> wrote: >> >>> Hi all, >>> I am facing a strange issue on two different machines that acts like >>> servers. Each of them runs an instance of Zeppelin installed as a system.d >>> service. >>> The configuration is: >>> - Ubuntu Server 16.04.2 LTS >>> - Spark 2.1.0 >>> - Microsoft Open R 3.3.2 >>> - Zeppelin 0.7.1 (0.7.0 gave the same problems) >>> >>> zeppelin-env.sh has the following settings: >>> export SPARK_HOME="/spark/home/directory" >>> >>> spark-env.sh has the following settings: >>> export LANG="en_US" >>> export SPARK_DAEMON_JAVA_OPTS+=" -Dspark.local.dir=/some/dir >>> -Dspark.eventLog.dir=/some/dir/spark-events -Dhadoop.tmp.dir=/some/dir" >>> export _JAVA_OPTIONS+=" -Djava.io.tmpdir=/some/dir" >>> >>> spark-defaults.conf is set as: >>> spark.executor.memory 21g >>> spark.driver.memory 21g >>> spark.python.worker.memory 4g >>> spark.sql.autoBroadcastJoinThreshold 0 >>> >>> I use Spark in stand-alone mode and it works perfectly. It also works >>> correctly with Zeppelin but this is what happens: >>> 1) Start zeppelin on the server using the command service zeppelin start >>> 2) Connect to port 8080 using Mozilla Firefox from client >>> 3) Insert username and password (I enabled Shiro authentication) >>> 4) open a notebook >>> 5) Execute the following code: >>> %spark.r >>> 2+2 >>> 6) The code runs correctly and I can see that R is currently running as >>> a process. >>> 7) Repeat steps 2-5 after some time (let’s say 2 or 3 hours) and >>> Zeppelin remains forever on “Running” or, if the elapsed time is higher >>> (for example 1 day) since the last run, it returns “Error”. The >>> “time-to-be-unresponsive” seems to be random and unpredictable. Also, R is >>> not present in the list of running processes. Spark session remains active >>> because I can access Spark UI from port 4040 and the application name is >>> “Zeppelin”, so it’s the Spark instance created by Zeppelin. >>> >>> I observed that sometimes I can simply restart the interpreter from >>> Zeppelin UI, but many other times it doesn’t work and I have to restart >>> Zeppelin ( service zeppelin restart ). >>> >>> This issue afflicts both 0.7.0 and 0.7.1 but I haven’t tried with >>> previous versions. It also happens if Zeppelin isn’t installed as a service. >>> >>> I can’t provide more detail because I can’t see any error or warning in >>> the logs.. this is really strange. >>> >>> Thank you all. >>> Kind regards >>> Pietro Pugni >>> >> >> > > -- 이종열, Jongyoul Lee, 李宗烈 http://madeng.net