Re: Interpreter zombie processes
I have also seen similar issues even using zeppelin-ddeamon but didn't have much time to investigate the issue when it was happening to me. On Thu, Dec 15, 2016 at 12:15 PM Ruslan Dautkhanov wrote: > Moon, > > > ZeppelinServer try to terminate interpreter process when shutting down > [1]. > > Unfortunatally, this does not happen all the time. > I have seen zombie spark interpreter processes many times. > As a double conirmation - I see spark yarn application was still running. > > > Also bin/zeppelin-deamon.sh script clean up if some processes are left > [2]. > > We were not using that script. We were just running like zeppelin.sh > --config .. > Will try to switch to zeppelin-deamon.sh and see if it makes a difference. > > Thanks! > > > > > -- > Ruslan Dautkhanov > > > > On Thu, Dec 15, 2016 at 12:46 PM, moon wrote: > > Hi, > > > > > > ZeppelinServer try to terminate interpreter process when shutting down [1]. > > > Also bin/zeppelin-deamon.sh script clean up if some processes are left [2]. > > > > > > If some processes are remained after shutting down Zeppelin, that's not an > > > expected result. > > > > > > Thanks, > > > moon > > > > > > [1] > > > > https://github.com/apache/zeppelin/blob/master/zeppelin-server/src/main/java/org/apache/zeppelin/server/ZeppelinServer.java#L159 > > > [2] > > > https://github.com/apache/zeppelin/blob/master/bin/zeppelin-daemon.sh#L218 > > > > > > On Thu, Dec 15, 2016 at 11:25 AM Ruslan Dautkhanov > > > wrote: > > > > > > > How to make Interpreter processes terminate if Zeppelin server exits? > > > > > > > > When we restart Zeppelin server (main process), in many cases, > Interpreter > > > > process keeps running, essential becoming a zombie processes. > > > > > > > > In case of Spark interpreter, it also holds SparkContext - consuming > > > > server-side > > > > resources too. > > > > > > > > How do we configure Zeppelin to kill all interpreters before its main > > > > process terminates? > > > > Or is this a bug? > > > > > > > > We're running ~2 weeks old snapshot of 0.7.0 Zeppelin. > > > > > > > > > > > > Thank you, > > > > Ruslan Dautkhanov > > > > > > > > -- Sent from my Mobile device
Re: Zeppelin with Separate Spark Connection
On Wed, Nov 9, 2016 at 1:34 PM, Tseytlin, Keren < keren.tseyt...@capitalone.com> wrote: > Hi All, > > > > I’ve just set up Zeppelin, and I’ve also set up my own Spark with > connection to Alluxio. I installed Zeppelin using the binary. When I use > Zeppelin, it seems to be using some internal Spark, not the one that I set > up. What configurations should I set in order to make the notebooks and > Spark jobs execute on my own Spark? > > > > I edited zeppelin-env.sh and added SPARK_HOME, but that caused anything I > tried to run in my notebook just shoot back “ERROR” with no output. > > > > Any help would be much appreciated! Thanks!! > > > > Best, > > Keren > > Try updating the following in zeppelin-env.sh export MASTER="spark://spark-02.softlayer.com:7077" export SPARK_HOME=/opt/spark-1.6.2-bin-hadoop2.6 And then on the ui, update the spark interpreter configuration to have master url properly configured. -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
Re: missing scala libraries in flink interpreter ? (binary)
I have created a jira to track the issue https://issues.apache.org/jira/browse/ZEPPELIN-1379 On Friday, August 26, 2016, Frank Dekervel wrote: > Hello, > > I tried the latest zeppelin release (binary with all interpreters), and in > order to make the flink interpreter work in non-local mode, i had to copy > all scala libraries from another interpreter to the flink interpreter. > > After trying to connect to an existing flink cluster i got > > zeppelin_1 | Exception in thread "pool-1-thread-2" > java.lang.NoClassDefFoundError: scala/collection/Seq > zeppelin_1 |at java.lang.Class.forName0(Native Method) > zeppelin_1 |at java.lang.Class.forName(Class.java:264) > zeppelin_1 |at org.apache.zeppelin.interprete > r.remote.RemoteInterpreterServer.createInterpreter(RemoteInt > erpreterServer.java:148) > zeppelin_1 |at org.apache.zeppelin.interprete > r.thrift.RemoteInterpreterService$Processor$createInterpreter.getResult(Re > moteInterpreterService.java:1409) > zeppelin_1 |at org.apache.zeppelin.interprete > r.thrift.RemoteInterpreterService$Processor$createInterpreter.getResult(Re > moteInterpreterService.java:1394) > zeppelin_1 |at org.apache.thrift.ProcessFunct > ion.process(ProcessFunction.java:39) > zeppelin_1 |at org.apache.thrift.TBaseProcess > or.process(TBaseProcessor.java:39) > zeppelin_1 |at org.apache.thrift.server.TThre > adPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) > zeppelin_1 |at java.util.concurrent.ThreadPoo > lExecutor.runWorker(ThreadPoolExecutor.java:1142) > zeppelin_1 |at java.util.concurrent.ThreadPoo > lExecutor$Worker.run(ThreadPoolExecutor.java:617) > zeppelin_1 |at java.lang.Thread.run(Thread.java:745) > zeppelin_1 | Caused by: java.lang.ClassNotFoundException: > scala.collection.Seq > zeppelin_1 |at java.net.URLClassLoader.findCl > ass(URLClassLoader.java:381) > zeppelin_1 |at java.lang.ClassLoader.loadClas > s(ClassLoader.java:424) > zeppelin_1 |at sun.misc.Launcher$AppClassLoad > er.loadClass(Launcher.java:331) > zeppelin_1 |at java.lang.ClassLoader.loadClas > s(ClassLoader.java:357) > zeppelin_1 |... 11 more > > to get it working i did > > RUN cp /opt/zeppelin/interpreter/ignite/scala*jar > /opt/zeppelin/interpreter/flink/ > > which would copy: > > scala-compiler-2.11.7.jar > scala-library-2.11.7.jar > scala-parser-combinators_2.11-1.0.4.jar > scala-reflect-2.11.7.jar > scala-xml_2.11-1.0.4.jar > > According to Trevor Grant (who helped me out on flink-user), that is a > regression of upgrading Zeppelin to spark 2.0/Scala 2.11. > > Thanks, > greetings, > Frank > > > -- Sent from my Mobile device
Re: Scratching my head around Zeppelin/Spark & Docker
On Fri, Aug 5, 2016 at 11:18 PM, Luciano Resende wrote: > Not exactly what you want, but I have an example here : > https://github.com/lresende/docker-systemml-notebook > > You should be able to accomplish what you want playing with --link which I > did in the example below (but just with Yarn and HDFS) > https://github.com/lresende/docker-yarn-cluster > > BTW, you might have to use Levy to access the remote Spark. -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
Re: Scratching my head around Zeppelin/Spark & Docker
Not exactly what you want, but I have an example here : https://github.com/lresende/docker-systemml-notebook You should be able to accomplish what you want playing with --link which I did in the example below (but just with Yarn and HDFS) https://github.com/lresende/docker-yarn-cluster On Fri, Aug 5, 2016 at 11:07 PM, DuyHai Doan wrote: > No, using Docker compose is easy, what I want is: > > 1) Zeppelin running inside a Docker container > 2) Spark deployed in stand-alone mode, running somewhere on bare-metal / > cloud / Docker but in another network > > In this scenario, it's very hard to make the Zeppelin client that is > living inside the Docker container to communicate with the external Spark > cluster > > On Fri, Aug 5, 2016 at 9:00 PM, George Webster > wrote: > >> can you share your docker file? also, are you using docker-compose or >> just docker-machine >> >> On Fri, Aug 5, 2016 at 8:46 PM, DuyHai Doan wrote: >> >>> Hello guys >>> >>> Has anyone attempted to run Zeppelin inside Docker that connect to a >>> real Spark cluster running on the host machine ? >>> >>> I've spend a day trying to make it work but unsuccessfully, the job >>> never completed because the driver program (Zeppelin Spark shell) is >>> listening on an Internal IP address (the internal IP address of the >>> container). >>> >>> I've tried to run the Docker container with host network (--net=host) >>> but in this case, I cannot access to Zeppelin through localhost:8080 or >>> 127.0.0.1:8080 >>> >>> Anyone has an idea to unblock my use-case ? >>> >> >> > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
Re: Dockerfile?
Or you could provide a PR and enhance what is already there :) On Sun, Jun 5, 2016 at 11:09 AM, Darren Govoni wrote: > Thanks. I will share my dockerfile once I get it working too. > > > > Sent from my Verizon Wireless 4G LTE smartphone > > > Original message ---- > From: Luciano Resende > Date: 06/05/2016 1:48 PM (GMT-05:00) > To: users@zeppelin.apache.org > Subject: Re: Dockerfile? > > This is not Ubuntu based, but can give you some help: > > Spark base: > https://github.com/lresende/docker-spark > > Zeppelin (using the above spark base): > https://github.com/lresende/docker-systemml-notebook > > This is pre-r dependencies, so I still need to update with R. > > > > On Sun, Jun 5, 2016 at 10:07 AM, Darren Govoni > wrote: > >> Hi >> >> Does anyone know of an updated docker file that builds latest zeppelin, >> spark, hadoop etc. Ubuntu based? >> >> Thanks >> Darren >> >> >> >> Sent from my Verizon Wireless 4G LTE smartphone >> > > > > -- > Luciano Resende > http://twitter.com/lresende1975 > http://lresende.blogspot.com/ > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
Re: Dockerfile?
This is not Ubuntu based, but can give you some help: Spark base: https://github.com/lresende/docker-spark Zeppelin (using the above spark base): https://github.com/lresende/docker-systemml-notebook This is pre-r dependencies, so I still need to update with R. On Sun, Jun 5, 2016 at 10:07 AM, Darren Govoni wrote: > Hi > > Does anyone know of an updated docker file that builds latest zeppelin, > spark, hadoop etc. Ubuntu based? > > Thanks > Darren > > > > Sent from my Verizon Wireless 4G LTE smartphone > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
Re: Failure in building web package
This seems to be happening in multiple areas, and starting to be more
frequent... I am having the same issue in my scala-2.11 PR...
Someone that is more familiar with the npm installation/integration should
take a look at this...
On Fri, Jun 3, 2016 at 8:57 AM, atreya90 wrote:
> I am facing the following issue while compiling the source for zeppelin
> (0.5.6) on Windows 10 64-bit machine.
>
> --- frontend-maven-plugin:0.0.23:npm (npm install) @ zeppelin-web ---
>
> [ERROR] npm http GET http://registry.npmjs.org/grunt-concurrent
> [ERROR] npm http GET http://registry.npmjs.org/grunt-contrib-clean
> [ERROR] npm http GET http://registry.npmjs.org/grunt-contrib-copy
> [ERROR] npm http GET http://registry.npmjs.org/grunt
> [ERROR] npm http GET http://registry.npmjs.org/grunt-contrib-concat
> [ERROR] npm http GET http://registry.npmjs.org/grunt-contrib-connect
> [ERROR] npm ERR! Error: connect ECONNREFUSED
> [ERROR] npm ERR! at errnoException (net.js:901:11)
> [ERROR] npm ERR! at Object.afterConnect [as oncomplete] (net.js:892:19)
> [ERROR] npm ERR! { [Error: connect ECONNREFUSED]
> [ERROR] npm ERR! code: 'ECONNREFUSED',
> [ERROR] npm ERR! errno: 'ECONNREFUSED',
> [ERROR] npm ERR! syscall: 'connect' }
> [ERROR] npm ERR!
> [ERROR] npm ERR! If you are behind a proxy, please make sure that the
> [ERROR] npm ERR! 'proxy' config is set properly. See: 'npm help config'
> [ERROR]
> [ERROR] npm ERR! System Windows_NT 6.2.9200
> [ERROR] npm ERR! command
>
> "C:\\Users\\***\\Downloads\\zeppelin-0.5.6-incubating\\zeppelin-web\\node\\node"
>
> "C:\\Users\\\\Downloads\\zeppelin-0.5.6-incubating\\zeppelin-web\\node\\npm\\bin\\npm-cli.js"
> "install" "--color=false"
> [ERROR] npm ERR! cwd
> C:\Users\*\Downloads\zeppelin-0.5.6-incubating\zeppelin-web
> [ERROR] npm ERR! node -v v0.10.22
> [ERROR] npm ERR! npm -v 1.3.8
> [ERROR] npm ERR! syscall connect
> [ERROR] npm ERR! code ECONNREFUSED
> [ERROR] npm ERR! errno ECONNREFUSED
> [ERROR] npm ERR! stack Error: connect ECONNREFUSED
> [ERROR] npm ERR! stack at errnoException (net.js:901:11)
> [ERROR] npm ERR! stack at Object.afterConnect [as oncomplete]
> (net.js:892:19)
> [ERROR] npm http GET http://registry.npmjs.org/grunt-contrib-uglify
> [ERROR] npm http GET http://registry.npmjs.org/grunt-contrib-watch
> [ERROR] npm http GET http://registry.npmjs.org/grunt-contrib-cssmin
> [ERROR] npm http GET http://registry.npmjs.org/grunt-contrib-htmlmin
> [ERROR] npm http GET http://registry.npmjs.org/grunt-contrib-jshint
> [ERROR] npm http GET http://registry.npmjs.org/grunt-newer
> [ERROR] npm http GET http://registry.npmjs.org/grunt-postcss
> [ERROR] npm http GET http://registry.npmjs.org/grunt-filerev
> [ERROR] npm http GET http://registry.npmjs.org/grunt-svgmin
> [ERROR] npm http GET http://registry.npmjs.org/grunt-ng-annotate
> [ERROR] npm http GET http://registry.npmjs.org/grunt-karma
> [ERROR] npm http GET http://registry.npmjs.org/karma-phantomjs-launcher
> [ERROR] npm http GET http://registry.npmjs.org/grunt-wiredep
> [ERROR] npm http GET http://registry.npmjs.org/load-grunt-tasks
> [ERROR] npm http GET http://registry.npmjs.org/time-grunt
> [ERROR] npm http GET http://registry.npmjs.org/karma-coverage
> [ERROR] npm http GET http://registry.npmjs.org/grunt-cli
> [ERROR] npm http GET http://registry.npmjs.org/jshint-stylish
> [ERROR] npm http GET http://registry.npmjs.org/karma
> [ERROR] npm http GET http://registry.npmjs.org/karma-jasmine
> [ERROR] npm http GET http://registry.npmjs.org/grunt-usemin
> [ERROR] npm
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 01:22 min
> [INFO] Finished at: 2016-06-03T10:18:22-05:00
> [INFO] Final Memory: 17M/204M
> [INFO]
>
> [ERROR] Failed to execute goal
> com.github.eirslett:frontend-maven-plugin:0.0.23:npm (npm install) on
> project zeppelin-web: Failed to run task: 'npm install --color=false'
> failed. (error code 1) -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions,
> please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
>
>
> Could someone please help me resolve this issue.
> (Note- I am not behind any proxy)
>
>
>
>
> --
> View this message in context:
> http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Failure-in-building-web-package-tp3173.html
> Sent from the Apache Zeppelin Users (incubating) mailing list mailing list
> archive at Nabble.com.
>
--
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/
Re: spark summit 2016 (San Francisco, CA)
Count me in, as I will be there as well. On Mon, May 30, 2016 at 9:04 PM, Zhong Wang wrote: > Hi zeppelin users/developers, > > Are any of you going to attend the spark summit in June? > > Zeppelin is an awesome project, and we had a great experience using > Zeppelin and working with the community this year. I would like to meet > with anyone who is interested in Zeppelin in person and chat about > interactive data analysis during the summit. > > Please let me know if you are going to join. > > Thanks! > > Zhong > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
Re: The Apache Software Foundation Announces Apache(R) Zeppelin™ as a Top-Level Project
, Founder and CEO of Memcore.io. "I am very excited > to see Apache Zeppelin graduating as an ASF Top Level Project. This shows > that more people are joining the community, bringing the project to a new > level, and adding more integration points with existing data analytics and > transactional software systems. This directly benefits the community > at-large." > > Apache Zeppelin originated in 2013 at NFLabs as Peloton, a commercial data > analytics product. Since entering the Apache Incubator in December 2014, > the project has had three releases, and twice participated in Google Summer > of Code under the Apache umbrella. > > "It was an honor to help with the incubation of Zeppelin," said Ted > Dunning, Vice President of the Apache Incubator. "I have been very > impressed with the Zeppelin community and the software they have built. I > see Apache Zeppelin being adopted all over the place where people need to > apply a notebook style to a wide variety of kinds of computing." > > Catch Apache Zeppelin in action during Berlin Buzzwords, 7 June 2016 > https://s.apache.org/mV8E > > Availability and Oversight > Apache Zeppelin software is released under the Apache License v2.0 and is > overseen by a self-selected team of active contributors to the project. A > Project Management Committee (PMC) guides the Project's day-to-day > operations, including community development and product releases. For > downloads, documentation, and ways to become involved with Apache Zeppelin, > visit http://zeppelin.apache.org/ and https://twitter.com/ApacheZeppelin > > About the Apache Incubator > The Apache Incubator is the entry path for projects and codebases wishing > to become part of the efforts at The Apache Software Foundation. All code > donations from external organizations and existing external projects > wishing to join the ASF enter through the Incubator to: 1) ensure all > donations are in accordance with the ASF legal standards; and 2) develop > new communities that adhere to our guiding principles. Incubation is > required of all newly accepted projects until a further review indicates > that the infrastructure, communications, and decision making process have > stabilized in a manner consistent with other successful ASF projects. While > incubation status is not necessarily a reflection of the completeness or > stability of the code, it does indicate that the project has yet to be > fully endorsed by the ASF. For more information, visit > http://incubator.apache.org/ > > About The Apache Software Foundation (ASF) > Established in 1999, the all-volunteer Foundation oversees more than 350 > leading Open Source projects, including Apache HTTP Server --the world's > most popular Web server software. Through the ASF's meritocratic process > known as "The Apache Way," more than 550 individual Members and 5,300 > Committers successfully collaborate to develop freely available > enterprise-grade software, benefiting millions of users worldwide: > thousands of software solutions are distributed under the Apache License; > and the community actively participates in ASF mailing lists, mentoring > initiatives, and ApacheCon, the Foundation's official user conference, > trainings, and expo. The ASF is a US 501(c)(3) charitable organization, > funded by individual donations and corporate sponsors including Alibaba > Cloud Computing, ARM, Bloomberg, Budget Direct, Cerner, Cloudera, Comcast, > Confluent, Facebook, Google, Hortonworks, HP, Huawei, IBM, InMotion > Hosting, iSigma, LeaseWeb, Microsoft, OPDi, PhoenixNAP, Pivotal, Private > Internet Access, Produban, Red Hat, Serenata Flowers, WANdisco, and Yahoo. > For more information, visit http://www.apache.org/ and > https://twitter.com/TheASF > > © The Apache Software Foundation. "Apache", "Zeppelin", "Apache Zeppelin", > "Ambari", "Apache Ambari", "Flink", "Apache Flink", "Hadoop", "Apache > Hadoop", "Hive", "Apache Hive", "Spark", "Apache Spark", and "ApacheCon" > are registered trademarks or trademarks of the Apache Software Foundation > in the United States and/or other countries. All other brands and > trademarks are the property of their respective owners. > > # # # > > NOTE: you are receiving this message because you are subscribed to the > annou...@apache.org distribution list. To unsubscribe, send email from the > recipient account to announce-unsubscr...@apache.org with the word > "Unsubscribe" in the subject line. > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
