visualize sql generator
Hi, I am beginner in Zeppelin, but i am getting impressed by this marvelous front end software. And to introduce it into us, i am searching a sort of plugin that createsql code by visual tool, like making ER figure in DB. I appreciate If you have some candidates one, whichever plugin type or hard coded. Thx Keiji
Re: Zeppelin unable to respond after some time
I am facing on the same issue now. 2017-02-17 18:25 GMT+09:00 RUSHIKESH RAUT : > Hi all, > > I am facing a issue while using Zeppelin. I am trying to load some > data(not that big data) into Zeppelin and then build some visualization on > it. The problem is that when I try to run the code first time it's working > but after some time the same code doesn't work. It remains in running state > on gui, but no logs are generated in Zeppelin logs. Also all further tasks > are hanging in pending state. > As soon as I restart Zeppelin it works. So I am guessing it's some memory > issue. I have read that Zeppelin stores the data in memory so it is > possible that it runs out of memory after some time. > How do I debug this issue? How much is the default memory that Zeppelin > takes at start? Also is there any way that I can run Zeppelin with > specified memory so that I can start the process with more memory. Because > it doesn't make sense to restart Zeppelin after every half hour > > Thanks, > Rushikesh Raut >
Re: Zeppelin unable to respond after some time
yes, almost every time. There are not any special operations. Just run the tutorial demos. >From my feeling, it happens in R demo frequently. 2017-02-17 18:50 GMT+09:00 Jeff Zhang : > > Is it easy to reproduce it ? > > 小野圭二 于2017年2月17日周五 下午5:47写道： > >> I am facing on the same issue now. >> >> 2017-02-17 18:25 GMT+09:00 RUSHIKESH RAUT : >> >> Hi all, >> >> I am facing a issue while using Zeppelin. I am trying to load some >> data(not that big data) into Zeppelin and then build some visualization on >> it. The problem is that when I try to run the code first time it's working >> but after some time the same code doesn't work. It remains in running state >> on gui, but no logs are generated in Zeppelin logs. Also all further tasks >> are hanging in pending state. >> As soon as I restart Zeppelin it works. So I am guessing it's some >> memory issue. I have read that Zeppelin stores the data in memory so it is >> possible that it runs out of memory after some time. >> How do I debug this issue? How much is the default memory that Zeppelin >> takes at start? Also is there any way that I can run Zeppelin with >> specified memory so that I can start the process with more memory. Because >> it doesn't make sense to restart Zeppelin after every half hour >> >> Thanks, >> Rushikesh Raut >> >> >>
How to write a style of printout in a application?
Hi, I am super beginner on here, but this is an amazing project, thx. And not sure all of them, so am sorry if this is a little bit silly question. I saw how to write some code on Zeppelin, at the same time, the same code can run on it as a application program like this. [in notebook] % hogehoge.scala But in the case of this type, how should i write the code in the application to use variety figures and so on? I could not find out the relation between a application / code and display procedure in the docs. :-b Any info and advices thank. -Keiji
Re: Zeppelin can let users write Java code?
I am not sure but could not Zeppelin run a binary program on notebook? You know Spark support Java program and it could run on it. I wonder Zeppelin simulate Spark console environment or not. -Keiji 2017-06-25 5:11 GMT+09:00 Alexander Bezzubov : > It's not very well known, but Apache Zeppelin already has one Java > interpreter. > If you look at Apache Beam one [1] - underneath [2] it has quite generic > facilities to interpreter java code, so you can use it for anything, not > just Java API of Beam. > > But Felix's suggestion on using JShell in Java9 as soon as it's released > definitely sounds as a more mature solution. > > > 1. http://zeppelin.apache.org/docs/0.7.2/interpreter/beam.html > 2. https://github.com/apache/zeppelin/tree/master/beam > > -- > Alex > > On Sat, Jun 24, 2017 at 9:26 PM, Felix Cheung > wrote: > >> There has been discussions on leveraging a Java REPL project to create >> such Java interpreter. >> >> Also Java 9 has JShell which seems to target a release in the next few >> months. >> >> -- >> *From:* Yi Lyu >> *Sent:* Saturday, June 24, 2017 9:17:32 AM >> *To:* users >> *Subject:* Zeppelin can let users write Java code? >> >> hi Team >> In our company, there are a lot of Java engineers, they are not good >> at Scala, may I ask, Zeppelin can let users write Java code, and then run >> them? Currently, I have tried Scala, Python, and can run it. >> and, Is there a plan to develop Java Interpreter ? >> -- >> Yi Lyu >> >> >
Re: How to write a style of printout in a application?
Thank you for your rapid and kindly reply, Park. :-) To make clear a little bit. Let's look in conf/interpreter.json on ver.0.7.0, focusing on Spark setting to make simple now, they have 'spark,sql, pyspark,r, scala' in it. I guess they are for executing those kinds programs as script on notebook. And i wonder a binary program could run like script on notebook? [for example] >%spark main<- this is main class that is described in hogehoge.scala or >%spark hogehoge.jar <-this jar is java binary program Any advance great. -Keiji 2017-06-26 12:55 GMT+09:00 Park Hoon <1am...@gmail.com>: > Hi, > > `%something` is called *interpreter* in Zeppelin. > > Here are few docs you can refer to write new interpreters. > > - http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/development/ > writingzeppelininterpreter.html > > You can write interpreters work on the browser using javascript. > > It's Spell which has few limitations but also provides good features. (e.g > can be display system as well) > > - http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/development/ > writingzeppelinspell.html > > > > Thanks. > > On Fri, Jun 23, 2017 at 8:27 AM, 小野圭二 wrote: > >> Hi, >> >> I am super beginner on here, but this is an amazing project, thx. >> And not sure all of them, so am sorry if this is a little bit silly >> question. >> >> I saw how to write some code on Zeppelin, at the same time, the same code >> can run on it as a application program like this. >> >> [in notebook] >> % hogehoge.scala >> >> But in the case of this type, how should i write the code in the >> application to use variety figures and so on? >> I could not find out the relation between a application / code and >> display procedure in the docs. :-b >> >> Any info and advices thank. >> >> -Keiji >> >> >> >
Are there any way a notebook to be black box?
Hi, I wonder if this mail title would be realize. I mean, when I do not want to be changed my notebook programing code, but just give a way to change the query of programing by dynamic forms. 1. are there way to hide all code in the notebook? 2. are there rock system to prevent changing the code in notebook? If No.1 would be available, it makes easy to provide commercial codes and/or secret codes to widely. In fact, from the view of code writers, especially in the case of commercial products, they do not want to provide any source code to their customers definitely. Another hand, it is costly to force R or Python programer to write Java or Scala program to change their naked codes to binary code. I have a image Script programers write and test their R/Python/Spark-Shell/SQL codes on notebooks and be it to BLACKBOX-NOTEBOOK, then upload it to github/svn where endusers who want to use those notebooks could access by themselves. The endusers down load those BLACKBOX-NOTEBOOK and import into their Zeppelin and it works fine. Sound like happy the both. No.2 seems like easier than No.1, because the issue is depend on authorization. But it is incomplete, because the codes reveal to the endusers. I saw zepl.com as well. Seems like Zeppelin and ZELP are aimed to help collaboration between analysts. That is good. And if adding the above NO.1 feature into it, they will be the best, i think. Do not You think so? -Ono
Re: Are there any way a notebook to be black box?
Hey, how do you think about my idea of BLACKBOX-NOTEBOOK? -Ono. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> ウイルス フリー｡ www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> 2017-06-30 10:38 GMT+09:00 小野圭二 : > Hi, > > I wonder if this mail title would be realize. > I mean, when I do not want to be changed my notebook programing code, but > just give a way to change the query of programing by dynamic forms. > > 1. are there way to hide all code in the notebook? > 2. are there rock system to prevent changing the code in notebook? > > If No.1 would be available, it makes easy to provide commercial codes > and/or secret codes to widely. > In fact, from the view of code writers, especially in the case of > commercial products, they do not want to provide any source code to their > customers definitely. Another hand, it is costly to force R or Python > programer to write Java or Scala program to change their naked codes to > binary code. > > I have a image Script programers write and test their > R/Python/Spark-Shell/SQL codes on notebooks and be it to > BLACKBOX-NOTEBOOK, then upload it to github/svn where endusers who want > to use those notebooks could access by themselves. The endusers down load > those BLACKBOX-NOTEBOOK and import into their Zeppelin and it works fine. > Sound like happy the both. > > No.2 seems like easier than No.1, because the issue is depend on > authorization. But it is incomplete, because the codes reveal to the > endusers. > > I saw zepl.com as well. > Seems like Zeppelin and ZELP are aimed to help collaboration between > analysts. That is good. And if adding the above NO.1 feature into it, they > will be the best, i think. > > Do not You think so? > > -Ono >
Re: Are there any way a notebook to be black box?
Thx for your reply, Hyung. Thank you for introducing some docs, but i knew them already.And my point was each notebook execute but hide/invisible the code. The zeppelin auth system manage every notebooks at once. That feature effect in the case of all user share same zeppelin system, like cloud computing. Against it, how about in the case of just sharing notebook? For example i set strict roles into my notebook, and share it with someone who has their zeppelin. After they download and import my strict auth notebook into their zeppelin, they can arrange my roles setting by themselves. In this case, they can see my hided code on their zeppelin. I do not misunderstand about the zeppelin auth system, do i ? To prevent this issue, may need another authenticate/management feature on Zeppelin, this is my point. :^) What for it, imagine such a case. In the case of selling my big data analyze application through zeppelin eco-system, i write and test my code on my zeppelin. Then export it and upload to sharing site. May someone see my sharing notebook on the site, then they download and import it into their zeppelin, and execute it, Ok that is good, fantastic, viva Zeppelin, if i am a volunteer programmer. But i would like to get some revenue from my code if someone go down my analyzing, hopefully. In this case, i just want to provide the result of analyzing, not my algorithm. Even my client share my notebook on his zeppelin, my code is protected by independent authentication, encryption and so on. This is my point. :-) Sorry for getting longer. -Ono 2017-07-04 23:35 GMT+09:00 Hyung Sung Shim : > Hello onokeiji. > > You can refer http://zeppelin.apache.org/docs/0.7.2/quickstart/ > explorezeppelinui.html#note-toolbar if you want to hide code in the > notebook. > and http://zeppelin.apache.org/docs/0.7.2/security/ > notebook_authorization.html#authorization-setting is relate to lock code > your note. > > Hope this helps. > > > 2017-07-04 22:05 GMT+09:00 小野圭二 : > >> Hey, how do you think about my idea of BLACKBOX-NOTEBOOK? >> >> -Ono. >> >> >> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> >> ウイルス >> フリー｡ www.avast.com >> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> >> <#m_3961825290808015369_m_-7798815867934407536_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >> >> 2017-06-30 10:38 GMT+09:00 小野圭二 : >> >>> Hi, >>> >>> I wonder if this mail title would be realize. >>> I mean, when I do not want to be changed my notebook programing code, >>> but just give a way to change the query of programing by dynamic forms. >>> >>> 1. are there way to hide all code in the notebook? >>> 2. are there rock system to prevent changing the code in notebook? >>> >>> If No.1 would be available, it makes easy to provide commercial codes >>> and/or secret codes to widely. >>> In fact, from the view of code writers, especially in the case of >>> commercial products, they do not want to provide any source code to their >>> customers definitely. Another hand, it is costly to force R or Python >>> programer to write Java or Scala program to change their naked codes to >>> binary code. >>> >>> I have a image Script programers write and test their >>> R/Python/Spark-Shell/SQL codes on notebooks and be it to >>> BLACKBOX-NOTEBOOK, then upload it to github/svn where endusers who want >>> to use those notebooks could access by themselves. The endusers down load >>> those BLACKBOX-NOTEBOOK and import into their Zeppelin and it works fine. >>> Sound like happy the both. >>> >>> No.2 seems like easier than No.1, because the issue is depend on >>> authorization. But it is incomplete, because the codes reveal to the >>> endusers. >>> >>> I saw zepl.com as well. >>> Seems like Zeppelin and ZELP are aimed to help collaboration between >>> analysts. That is good. And if adding the above NO.1 feature into it, they >>> will be the best, i think. >>> >>> Do not You think so? >>> >>> -Ono >>> >> >> >
Re: Are there any way a notebook to be black box?
Hi there, I created this issue on JIRA as ZEPPELIN-2721. -Ono <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> ウイルス フリー｡ www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> 2017-07-05 16:10 GMT+09:00 Khalid Huseynov : > Hello Ono, > > Thanks for reporting your use case scenario and it's indeed valid that > users would like to share their notes across Zeppelin instances with their > original permissions. It definitely deserves an issue and should be handled > toward maturity of Zeppelin (e.g. in v. 0.9~1.0). Possible solutions could > be including permission info inside of note, or some hash identifying the > original creator. Also as you mentioned, some of scenarios possibly could > be handled in zepl.com. > > On Wed, Jul 5, 2017 at 1:36 PM, Hyung Sung Shim wrote: > >> Thank you very much to describe in detail. >> I totally understand your question now. >> There is no way to solve your question for now but it's really >> interesting issue. >> so Could you make a Jira issue for this with your experience? >> Thanks! >> >> 2017-07-05 10:16 GMT+09:00 小野圭二 : >> >>> Thx for your reply, Hyung. >>> >>> Thank you for introducing some docs, but i knew them already.And my >>> point was each notebook execute but hide/invisible the code. >>> The zeppelin auth system manage every notebooks at once. That feature >>> effect in the case of all user share same zeppelin system, like cloud >>> computing. >>> >>> Against it, how about in the case of just sharing notebook? >>> For example i set strict roles into my notebook, and share it with >>> someone who has their zeppelin. After they download and import my strict >>> auth notebook into their zeppelin, they can arrange my roles setting by >>> themselves. In this case, they can see my hided code on their zeppelin. >>> >>> I do not misunderstand about the zeppelin auth system, do i ? >>> >>> To prevent this issue, may need another authenticate/management feature >>> on Zeppelin, this is my point. :^) >>> >>> What for it, imagine such a case. >>> In the case of selling my big data analyze application through zeppelin >>> eco-system, i write and test my code on my zeppelin. Then export it and >>> upload to sharing site. >>> May someone see my sharing notebook on the site, then they download and >>> import it into their zeppelin, and execute it, >>> >>> Ok that is good, fantastic, viva Zeppelin, if i am a volunteer >>> programmer. >>> >>> But i would like to get some revenue from my code if someone go down my >>> analyzing, hopefully. >>> In this case, i just want to provide the result of analyzing, not my >>> algorithm. >>> Even my client share my notebook on his zeppelin, my code is protected >>> by independent authentication, encryption and so on. >>> >>> This is my point. :-) >>> Sorry for getting longer. >>> >>> -Ono >>> >>> >>> 2017-07-04 23:35 GMT+09:00 Hyung Sung Shim : >>> >>>> Hello onokeiji. >>>> >>>> You can refer http://zeppelin.apache.org/docs/0.7.2/quickstart/explo >>>> rezeppelinui.html#note-toolbar if you want to hide code in the >>>> notebook. >>>> and http://zeppelin.apache.org/docs/0.7.2/security/notebook_ >>>> authorization.html#authorization-setting is relate to lock code your >>>> note. >>>> >>>> Hope this helps. >>>> >>>> >>>> 2017-07-04 22:05 GMT+09:00 小野圭二 : >>>> >>>>> Hey, how do you think about my idea of BLACKBOX-NOTEBOOK? >>>>> >>>>> -Ono. >>>>> >>>>> >>>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> >>>>> ウイルス >>>>> フリー｡ www.avast.com >>>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> >>>>> <#m_-2740405082895278623_m_7229949389285741499_m_-8962903669315512691_m_3961825290808015369_m_-7798815867934407536_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >>>>> >>>>> 2017-06-30 10:38 GMT+09:00 小野圭二 : >>>>> >>>>>> H
python is not responding on ver.0.7.2
Hi, This might be an old and popular issue, but i could not find the solution on Jira, mail archive and so on. The python demo does not work. [Env]: CentOS 6.2 Python 2.6.6 Zeppelin 0.7.2 netinst beneath 'interpreter': jdbc,lib,md,python,shell,spark [notebook]: from demo %python import matplotlib.pyplot as plt plt.plot([1,2,3]) Traceback (most recent call last): File "/tmp/zeppelin_python-2865335277309763273.py", line 20, in from py4j.java_gateway import java_import, JavaGateway, GatewayClient ImportError: No module named py4j.java_gateway Traceback (most recent call last): File "/tmp/zeppelin_python-2865335277309763273.py", line 20, in from py4j.java_gateway import java_import, JavaGateway, GatewayClient ImportError: No module named py4j.java_gateway python is not responding Any advice would be appreciated. -Keiji
Re: python is not responding on ver.0.7.2
Hi Park, Thank you for response to this. I thought that was caused in your pointing at the first, but the py4j had already been installed in it. I guessed the issue of depending on PATH or CLASSPATH environment parameters as well. They were not. Indeed, it worked fine on other our Linux machine. I am not sure yet what was wrong on the first Linux that was my best dev-machine. :-b Now i have been starting it to be clean-install on the new environment. I would like to share this issue if it will be clear. -Keiji 2017-09-01 12:35 GMT+09:00 Park Hoon <1am...@gmail.com>: > Hi, I haven't used net installation version. But it seems that u need to > install `py4j` python library as the error says > > > No module named py4j.java_gateway > > You can do easily `pip install py4j` with your python bin which related > to the python interpreter in Zeppelin. > > On Tue, Aug 22, 2017 at 2:18 PM, 小野圭二 wrote: > >> Hi, >> >> This might be an old and popular issue, but i could not find the solution >> on Jira, mail archive and so on. >> The python demo does not work. >> >> [Env]: CentOS 6.2 Python 2.6.6 >> Zeppelin 0.7.2 netinst >> beneath 'interpreter': jdbc,lib,md,python,shell,spark >> [notebook]: from demo >> %python >> import matplotlib.pyplot as plt >> plt.plot([1,2,3]) >> >> Traceback (most recent call last): >> File "/tmp/zeppelin_python-2865335277309763273.py", line 20, in >> from py4j.java_gateway import java_import, JavaGateway, GatewayClient >> ImportError: No module named py4j.java_gateway >> Traceback (most recent call last): >> File "/tmp/zeppelin_python-2865335277309763273.py", line 20, in >> from py4j.java_gateway import java_import, JavaGateway, GatewayClient >> ImportError: No module named py4j.java_gateway >> >> python is not responding >> >> >> Any advice would be appreciated. >> >> -Keiji >> > >
How to execute spark-submit on Note
Hi all, I searched this topic on the archive of ml, but still could not find out the solution clearly. So i have tried to post this again(maybe). I am using ver 0.8.0, and have installed spark 2.2 on the other path, just for checking my test program. Then i wrote a quite simple sample python code to check the how to. 1. the code works fine on a note in Zeppelin 2. the same code but added the initialize code for SparkContext in it works fine on the Spark by using 'spark-submit'. 3. tried to execute "2" from a note in Zeppelin with the following script. yes, "spark" interpreter has been implemented in the note. then on the note, %spark-submit -> interpreter not found error 4.I have arranged 'SPARK_SUBMIT_OPTIONS' in zeppelin-env.sh order by the doc ex. export SPARK_SUBMIT_OPTIONS='--packages com.databricks:spark-csv_2.10:1.2.0' 5. then running %spark-submit -> interpreter not found error (as same as "3") How can i use spark-submit from a note? Any advice thanks. -Keiji
Re: How to execute spark-submit on Note
Thank you for your reply, Jeff "%sh" ? "sh" seems like request something execution code. I tried "%sh", then %sh %sh bash: : no permission I made binary file from .py to .pyc, but the answer was as same. I am sorry seems like doubting you, but Is "%sh" the resolution? -Keiji 2017-10-03 17:35 GMT+09:00 Jianfeng (Jeff) Zhang : > > I am surprised why would you use %spark-submit, there’s no document about > %spark-submit. If you want to use spark-submit in zeppelin, then you > could use %sh > > > Best Regard, > Jeff Zhang > > > From: 小野圭二 > Reply-To: "users@zeppelin.apache.org" > Date: Tuesday, October 3, 2017 at 12:49 PM > To: "users@zeppelin.apache.org" > Subject: How to execute spark-submit on Note > > Hi all, > > I searched this topic on the archive of ml, but still could not find out > the solution clearly. > So i have tried to post this again(maybe). > > I am using ver 0.8.0, and have installed spark 2.2 on the other path, just > for checking my test program. > Then i wrote a quite simple sample python code to check the how to. > > 1. the code works fine on a note in Zeppelin > 2. the same code but added the initialize code for SparkContext in it > works fine on the Spark by using 'spark-submit'. > 3. tried to execute "2" from a note in Zeppelin with the following script. > yes, "spark" interpreter has been implemented in the note. > then on the note, > %spark-submit > -> interpreter not found error > 4.I have arranged 'SPARK_SUBMIT_OPTIONS' in zeppelin-env.sh order by the > doc > ex. export SPARK_SUBMIT_OPTIONS='--packages > com.databricks:spark-csv_2.10:1.2.0' > 5. then running > %spark-submit > -> interpreter not found error (as same as "3") > > How can i use spark-submit from a note? > Any advice thanks. > > -Keiji >
Re: How to execute spark-submit on Note
Thank you for your quick reply again, Jeff. Yes i know the difference of the execution environment between "%sh" and ">spark-submit". And my question was "how to execute spark-submit as shell interpreter". That mean, i am searching how to execute a binary program from a note of zeppelin. This time it has been limited on Spark. Seems like Zeppelin have several procedure to execute Spark shell, like spark.pyspark, spark.sql So how to do "spark-submit" was my wondering. I am sorry for bothering Your time, but at the same time, i am appreciated if You get my wondering clearly, and show me some tips. -Keiji 2017-10-03 18:30 GMT+09:00 Jeff Zhang : > %sh is shell interpreter, you can run spark-submit just as you run it in > shell terminal. > > 小野圭二 于2017年10月3日周二 下午4:58写道： > >> Thank you for your reply, Jeff >> >> "%sh" ? >> "sh" seems like request something execution code. >> I tried "%sh", then >> >> %sh >> %sh bash: : no permission >> >> I made binary file from .py to .pyc, but the answer was as same. >> I am sorry seems like doubting you, but Is "%sh" the resolution? >> >> -Keiji >> >> 2017-10-03 17:35 GMT+09:00 Jianfeng (Jeff) Zhang >> : >> >>> >>> I am surprised why would you use %spark-submit, there’s no document >>> about %spark-submit. If you want to use spark-submit in zeppelin, then >>> you could use %sh >>> >>> >>> Best Regard, >>> Jeff Zhang >>> >>> >>> From: 小野圭二 >>> Reply-To: "users@zeppelin.apache.org" >>> Date: Tuesday, October 3, 2017 at 12:49 PM >>> To: "users@zeppelin.apache.org" >>> Subject: How to execute spark-submit on Note >>> >>> Hi all, >>> >>> I searched this topic on the archive of ml, but still could not find out >>> the solution clearly. >>> So i have tried to post this again(maybe). >>> >>> I am using ver 0.8.0, and have installed spark 2.2 on the other path, >>> just for checking my test program. >>> Then i wrote a quite simple sample python code to check the how to. >>> >>> 1. the code works fine on a note in Zeppelin >>> 2. the same code but added the initialize code for SparkContext in it >>> works fine on the Spark by using 'spark-submit'. >>> 3. tried to execute "2" from a note in Zeppelin with the following >>> script. >>> yes, "spark" interpreter has been implemented in the note. >>> then on the note, >>> %spark-submit >>> -> interpreter not found error >>> 4.I have arranged 'SPARK_SUBMIT_OPTIONS' in zeppelin-env.sh order by the >>> doc >>> ex. export SPARK_SUBMIT_OPTIONS='--packages >>> com.databricks:spark-csv_2.10:1.2.0' >>> 5. then running >>> %spark-submit >>> -> interpreter not found error (as same as "3") >>> >>> How can i use spark-submit from a note? >>> Any advice thanks. >>> >>> -Keiji >>> >> >>
Re: How to execute spark-submit on Note
Hi Dave, Thank You for your suggestion. It worked fine order by my expectation so far. I did not know "%sh" could use like that. Anyhow, i would like to explain why i would like to execute "spark-submit" in a note, to be clear your wondering. Yes, i know the basic instruction of Zeppelin as You explained to me in your reply, Dave. So, now, i tried to find the prospect of the environment of execution in Zeppelin. That mean, we were considering how to deliver our programs to users widely after we made a program with collaboration on Zeppelin. In this case, we might do not want to disclose our source code to them, but want to keep the execution environment for rejecting any unnecessary issues. Now i succeeded with a script code. Next will try to run a binary one. That was the reason why, i posted this question into ML. And i asked similar but another solution into JIRA,(#2721) Once again, thank You Dave. -Keiji 2017-10-03 19:12 GMT+09:00 David Howell : > Hi Keiji, > > > > In the paragraph you would write: > > %sh > > spark-submit myapp.jar ... > > > > The %sh interpreter is a shell, and runs as the zeppelin service user with > whatever permissions it has. You can run any shell commands in it. > > > > Although, this is a fairly strange way to run zeppelin so I’m not really > sure that is what you want. > > > > You can just use the %spark.pyspark interpreter and write your python > spark code in there. The spark interpreters in Zeppelin already create the > Spark Context for you, as well as sqlContext and spark session. These are > available as sc, sqlContext and spark. If you have a program that is ready > for spark submit, I would use some other tool to schedule and run it, like > cron, oozie, NiFi, Luigi, Airflow etc. Or if you want to run manually just > use spark submit from the shell directly or ssh. > > > > > > Dave > > > > *From: *小野圭二 > *Sent: *Tuesday, 3 October 2017 8:43 PM > *To: *users@zeppelin.apache.org > *Subject: *Re: How to execute spark-submit on Note > > > Thank you for your quick reply again, Jeff. > > Yes i know the difference of the execution environment between "%sh" and > ">spark-submit". > And my question was "how to execute spark-submit as shell interpreter". > That mean, i am searching how to execute a binary program from a note of > zeppelin. > This time it has been limited on Spark. > > Seems like Zeppelin have several procedure to execute Spark shell, like > spark.pyspark, spark.sql > So how to do "spark-submit" was my wondering. > > I am sorry for bothering Your time, but at the same time, i am appreciated > if You get my wondering clearly, and show me some tips. > > -Keiji > > > 2017-10-03 18:30 GMT+09:00 Jeff Zhang : > >> %sh is shell interpreter, you can run spark-submit just as you run it in >> shell terminal. >> >> 小野圭二 于2017年10月3日周二 下午4:58写道： >> >>> Thank you for your reply, Jeff >>> >>> "%sh" ? >>> "sh" seems like request something execution code. >>> I tried "%sh", then >>> >>> %sh >>> %sh bash: : no permission >>> >>> I made binary file from .py to .pyc, but the answer was as same. >>> I am sorry seems like doubting you, but Is "%sh" the resolution? >>> >>> -Keiji >>> >>> 2017-10-03 17:35 GMT+09:00 Jianfeng (Jeff) Zhang >> >: >>> >>>> >>>> I am surprised why would you use %spark-submit, there’s no document >>>> about %spark-submit. If you want to use spark-submit in zeppelin, then >>>> you could use %sh >>>> >>>> >>>> Best Regard, >>>> Jeff Zhang >>>> >>>> >>>> From: 小野圭二 >>>> Reply-To: "users@zeppelin.apache.org" >>>> Date: Tuesday, October 3, 2017 at 12:49 PM >>>> To: "users@zeppelin.apache.org" >>>> Subject: How to execute spark-submit on Note >>>> >>>> Hi all, >>>> >>>> I searched this topic on the archive of ml, but still could not find >>>> out the solution clearly. >>>> So i have tried to post this again(maybe). >>>> >>>> I am using ver 0.8.0, and have installed spark 2.2 on the other path, >>>> just for checking my test program. >>>> Then i wrote a quite simple sample python code to check the how to. >>>> >>>> 1. the code works fine on a note in Zeppelin >>>> 2. the same code but added the initialize code for SparkContext in it >>>> works fine on the Spark by using 'spark-submit'. >>>> 3. tried to execute "2" from a note in Zeppelin with the following >>>> script. >>>> yes, "spark" interpreter has been implemented in the note. >>>> then on the note, >>>> %spark-submit >>>> -> interpreter not found error >>>> 4.I have arranged 'SPARK_SUBMIT_OPTIONS' in zeppelin-env.sh order by >>>> the doc >>>> ex. export SPARK_SUBMIT_OPTIONS='--packages >>>> com.databricks:spark-csv_2.10:1.2.0' >>>> 5. then running >>>> %spark-submit >>>> -> interpreter not found error (as same as "3") >>>> >>>> How can i use spark-submit from a note? >>>> Any advice thanks. >>>> >>>> -Keiji >>>> >>> >>> >
Re: How to execute spark-submit on Note
Thank You for your information, Lucas. Sound interesting and good tip for how to deliver it. I think this tip should be up to Zeppelin wiki if there is it. :-) And i should check the Databricks notebook anyhow. -Keiji 2017-10-04 17:20 GMT+09:00 Partridge, Lucas (GE Aviation) < lucas.partri...@ge.com>: > “we were considering how to deliver our programs to users widely after we > made a program with collaboration on Zeppelin” > > > > - This is a common question/use case in my experience with Zeppelin: “How > do we roll out code to everyone once it’s been prototyped in Zeppelin?”. > Our approach is to package it up in jars or Python packages and make them > available on the environment. Then users can just import it like any other > code in their own Zeppelin %spark or %pyspark paragraphs. No %sh is > required. Other notebook-based environments like Databricks make this > packaging and importing of libraries part of their UI. > > > > Thanks, Lucas. > > > > *From:* 小野圭二 [mailto:onoke...@gmail.com] > *Sent:* 04 October 2017 02:24 > *To:* users@zeppelin.apache.org > *Subject:* EXT: Re: How to execute spark-submit on Note > > > > Hi Dave, > > > > Thank You for your suggestion. > > It worked fine order by my expectation so far. > > I did not know "%sh" could use like that. > > > > Anyhow, i would like to explain why i would like to execute "spark-submit" > in a note, to be clear your wondering. > > Yes, i know the basic instruction of Zeppelin as You explained to me in > your reply, Dave. > > So, now, i tried to find the prospect of the environment of execution in > Zeppelin. > > That mean, we were considering how to deliver our programs to users widely > after we made a program with collaboration on Zeppelin. In this case, we > might do not want to disclose our source code to them, but want to keep the > execution environment for rejecting any unnecessary issues. > > Now i succeeded with a script code. Next will try to run a binary one. > > > > That was the reason why, i posted this question into ML. > > And i asked similar but another solution into JIRA,(#2721) > > > > Once again, thank You Dave. > > > > -Keiji > > > > > > 2017-10-03 19:12 GMT+09:00 David Howell : > > Hi Keiji, > > > > In the paragraph you would write: > > %sh > > spark-submit myapp.jar ... > > > > The %sh interpreter is a shell, and runs as the zeppelin service user with > whatever permissions it has. You can run any shell commands in it. > > > > Although, this is a fairly strange way to run zeppelin so I’m not really > sure that is what you want. > > > > You can just use the %spark.pyspark interpreter and write your python > spark code in there. The spark interpreters in Zeppelin already create the > Spark Context for you, as well as sqlContext and spark session. These are > available as sc, sqlContext and spark. If you have a program that is ready > for spark submit, I would use some other tool to schedule and run it, like > cron, oozie, NiFi, Luigi, Airflow etc. Or if you want to run manually just > use spark submit from the shell directly or ssh. > > > > > > Dave > > > > *From: *小野圭二 > *Sent: *Tuesday, 3 October 2017 8:43 PM > *To: *users@zeppelin.apache.org > *Subject: *Re: How to execute spark-submit on Note > > > > Thank you for your quick reply again, Jeff. > > > > Yes i know the difference of the execution environment between "%sh" and > ">spark-submit". > > And my question was "how to execute spark-submit as shell interpreter". > > That mean, i am searching how to execute a binary program from a note of > zeppelin. > > This time it has been limited on Spark. > > > > Seems like Zeppelin have several procedure to execute Spark shell, like > spark.pyspark, spark.sql > > So how to do "spark-submit" was my wondering. > > > > I am sorry for bothering Your time, but at the same time, i am appreciated > if You get my wondering clearly, and show me some tips. > > > > -Keiji > > > > > > 2017-10-03 18:30 GMT+09:00 Jeff Zhang : > > %sh is shell interpreter, you can run spark-submit just as you run it in > shell terminal. > > > > 小野圭二 于2017年10月3日周二 下午4:58写道： > > Thank you for your reply, Jeff > > > > "%sh" ? > > "sh" seems like request something execution code. > > I tried "%sh", then > > > > %sh > > %sh bash: : no permission > > > > I made binary file from .py to .pyc, but the answer was as same. &
How to initialize ZeppelinContext in a stand alone program
Hi,
I am sure Zeppelin manage printouts of a program to a note.
Then i would like to know how to initialize ZeppelinContext in python/scala
code like Spark?
In the case of SparkContext, it is clear like that,
[python]
from pyspark import SparkConf, SparkContext
conf = SparkConf().setAppName("...").setMaster("...")
sc = SparkContext(conf=conf)
So i wonder ZeppelinContext could be called like above?
Any advice is welcome
-Keiji
How to get the data url
Hi,
I am searching a useful API for getting a data URL that is accessed in a
notebook.
For example, when this URL is in a application
new URL("https://s3.amazonaws.com/apache-zeppelin/tutorial/bank/
bank.csv")
How to get this url from using Zeppelin API?
I guess it is usually and easy to execute by a interpreter API like Spark,
but that way would be forced to implement the logic in all interpreters, I
do not want to do it.
It is convenient If Zeppelin provide this function.
Any advice are welcome.
-Keiji
Re: How to apache zeppelin front ui?
Hi Steve, I am not sure the front ui can replace with the bootstrap theme, but I have been customizing my Zep by changing the files below 'zeppelin-web/src/app/home' and/or 'zeppelin-web/src/components'. You know after changing those files You need recompile to reflect it to yours. Hey guys, am I correct? -keiji 2017-11-21 15:24 GMT+09:00 Steve Fox : > Hi zeppelin users! > > I am trying to change the front ui of apache zeppelin. > > Can I change to the bootstrap theme when I change front ui? > > ex https://bootswatch.com/ > > I can not seem to find it, but where is the document to refer to? > > Thanks. >
Re: Importing notes
interest question, i also want to know how to do it. :-) 2018-01-31 9:25 GMT+09:00 Ricardo Martinelli de Oliveira < rmart...@redhat.com>: > Hello, > > Is there a way to import notes from a command line? I am trying to write a > script that imports all my notes in a zeppelin docker container, but the > only ways to import is from the web UI or REST. In the script, zeppelin is > not started (there is no reason to be) so I'm trying to find a way to > import the note in a disconnected way. > > Any ideas to do this? >
Thrift server for call back will start. Port:
Hi there,
I am so confusing now, let me know why if You know this, please.
The env:
Mac: High Sierra ver.10.13.3
Zep: 0.7.3
Spark: 2.2.0
Java: 1.8
notebook script: %pyspark print 'aaa' (just it for test)
In this env, i got the error in log as
ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused)
Indeed, before running notebook, RemoteInterpreterManagedProcess.java
prepares a Port for callback to Thrift.
INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130) -
Thrift server for callback will start. Port: 63039
I guess, the reason why the error was Zeppelin could not attache to the
Thrift PORT.
And my confusion is, when i run the same module on CentOS, it worked fine.
CentOS: 7
Zep: same
Spark: same
Java: same
notebook script: same
And there was no the Thift port description in the log file. From this
consequence, the fine working was cause of did not use Thrift port on
CentOS.
Do You know actually there is different behavior between Mac and CentOS?
And i am so appreciated if You know the workaround on Mac.
Best,
Ono
Re: Thrift server for call back will start. Port:
Hi Jeff,
No one graved 63039 so far on Mac.
Yes, this issue has been reproduced as changing the port NO randomly.
Ono
2018-03-02 14:49 GMT+09:00 Jeff Zhang :
>
> It is possible that 63039 is used by other processes, as Mac can not
> detect whether this port is used or not from my experience.
>
> And can you reproduce this issue consistently ? Because each time zeppelin
> will chose a random port as long as you don't specify the portRange, so if
> you reproduce it consistently, it might due to other issues.
>
>
>
>
> 小野圭二 于2018年3月2日周五 下午1:32写道：
>
>> Hi there,
>>
>> I am so confusing now, let me know why if You know this, please.
>>
>> The env:
>>Mac: High Sierra ver.10.13.3
>>Zep: 0.7.3
>>Spark: 2.2.0
>>Java: 1.8
>>notebook script: %pyspark print 'aaa' (just it for
>> test)
>>
>> In this env, i got the error in log as
>>
>> ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed
>>
>> org.apache.zeppelin.interpreter.InterpreterException:
>> org.apache.zeppelin.interpreter.InterpreterException:
>> org.apache.thrift.transport.TTransportException:
>> java.net.ConnectException: Connection refused (Connection refused)
>>
>>
>> Indeed, before running notebook, RemoteInterpreterManagedProcess.java
>> prepares a Port for callback to Thrift.
>>
>>
>> INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130)
>> - Thrift server for callback will start. Port: 63039
>>
>>
>> I guess, the reason why the error was Zeppelin could not attache to the
>> Thrift PORT.
>>
>>
>> And my confusion is, when i run the same module on CentOS, it worked fine.
>> CentOS: 7
>> Zep: same
>> Spark: same
>> Java: same
>> notebook script: same
>>
>> And there was no the Thift port description in the log file. From this
>> consequence, the fine working was cause of did not use Thrift port on
>> CentOS.
>>
>>
>> Do You know actually there is different behavior between Mac and CentOS?
>> And i am so appreciated if You know the workaround on Mac.
>>
>>
>> Best,
>> Ono
>>
>
Re: Thrift server for call back will start. Port:
va:152)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at
org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at
org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at
org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection
refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
WARN [2018-03-05 13:15:32,885] ({pool-2-thread-2}
NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973
is finished, status: ERROR, exception:
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused), result:
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused)
INFO [2018-03-05 13:15:32,910] ({pool-2-thread-2}
VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
INFO [2018-03-05 13:15:32,914] ({pool-2-thread-2}
SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973
finished by scheduler
org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
2018-03-02 15:07 GMT+09:00 小野圭二 :
> Hi Jeff,
>
> No one graved 63039 so far on Mac.
> Yes, this issue has been reproduced as changing the port NO randomly.
>
> Ono
>
>
> 2018-03-02 14:49 GMT+09:00 Jeff Zhang :
>
>>
>> It is possible that 63039 is used by other processes, as Mac can not
>> detect whether this port is used or not from my experience.
>>
>> And can you reproduce this issue consistently ? Because each time
>> zeppelin will chose a random port as long as you don't specify the
>> portRange, so if you reproduce it consistently, it might due to other
>> issues.
>>
>>
>>
>>
>> 小野圭二 于2018年3月2日周五 下午1:32写道：
>>
>>> Hi there,
>>>
>>> I am so confusing now, let me know why if You know this, please.
>>>
>>> The env:
>>>Mac: High Sierra ver.10.13.3
>>>Zep: 0.7.3
>>>Spark: 2.2.0
>>>Java: 1.8
>>>notebook script: %pyspark print 'aaa' (just it for
>>> test)
>>>
>>> In this env, i got the error in log as
>>>
>>> ERROR ({pool-2-thread-2}
Re: Thrift server for call back will start. Port:
Actually, there was not created the interpreter log file as said 'Cannot
run interpreter' in the log. And it was run on local machine as MAC, i mean
stand alone. i stopped every firewall to make it easy. And again,this
happened just on Mac. On CentOS, it works fine.
That makes me confusion. :-b
2018-03-05 14:06 GMT+09:00 Jeff Zhang :
>
> Could you check the interpreter log ? The issue of not be able to
> connecting to thrift port could happen due to network issue, like firewall
> or you use VPN, etc.
>
>
>
> 小野圭二 于2018年3月5日周一 下午12:32写道：
>
>> Hi,
>>
>> I show the all log on this error.There are 2 type error. The 1 is at the
>> first run. The 2 is at the second run. Seems like could not established to
>> Thrift callback port 51348 that was set at the first run.
>> First of all, does it need to establish the callback port of Thrift
>> server?
>> The notebook script was just
>> %pyspark
>> print 'aaa'
>> Any advice appreciate.
>>
>> 1.
>> INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13}
>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05
>> 13:11:13,934] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109)
>> - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.
>> interpreter.remote.RemoteInterpreter-shared_session INFO [2018-03-05
>> 13:11:13,936] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run
>> paragraph 20180302-135047_99102973 using pyspark
>> RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
>> INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
>> ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) - Create
>> InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process WARN
>> [2018-03-05 13:11:14,010] ({pool-2-thread-2}
>> InterpreterSetting.java[setupPropertiesForSparkR]:796)
>> - sparkr.zip is not found, SparkR may not work. INFO [2018-03-05
>> 13:11:14,017] ({pool-2-thread-2}
>> RemoteInterpreterManagedProcess.java[start]:123)
>> - Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05
>> 13:11:14,023] ({pool-2-thread-2}
>> RemoteInterpreterManagedProcess.java[start]:130)
>> - Thrift server for callback will start. Port: 51348 INFO [2018-03-05
>> 13:11:14,049] ({pool-2-thread-2}
>> RemoteInterpreterManagedProcess.java[start]:205)
>> - Run interpreter process
>> [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh,
>> -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c,
>> 172.16.20.10, -p, 51348, -l,
>> /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX,
>> -g, spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor}
>> RemoteInterpreterManagedProcess.java[onProcessComplete]:258) -
>> Interpreter process exited 0 ERROR [2018-03-05 13:12:14,058]
>> ({pool-2-thread-2} Job.java[run]:184) - Job failed org.apache.zeppelin.
>> interpreter.InterpreterException: Cannot run interpreter at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProces
>> s.start(RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.
>> interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_
>> create(RemoteInterpreter.java:153) at org.apache.zeppelin.
>> interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126) at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at
>> org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
>> java.util.concurrent.ScheduledThreadPoolExecutor$
>> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(
>> ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.
>> ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748) ERROR [2018-03-05 13:12:14,075]
>> ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error
>> org.apache.zeppelin.interpreter.InterpreterException: Cannot run
>> interpreter at org.apache.zeppelin.interpreter.remote.
>> RemoteInterpreterManagedProcess.
Re: Thrift server for call back will start. Port:
Nothing changed. I guess this property does not effect cause of failing to
call the interpreter.
i challenged it as log4j...zeppelin=DEBUG, but no way.
2018-03-05 18:24 GMT+09:00 Jeff Zhang :
>
> Try to add the following to log4j.properties for more log
>
> log4j.logger.org.apache.zeppelin.interpreter=DEBUG
>
>
>
> 小野圭二 于2018年3月5日周一 下午5:13写道：
>
>> Actually, there was not created the interpreter log file as said 'Cannot
>> run interpreter' in the log. And it was run on local machine as MAC, i mean
>> stand alone. i stopped every firewall to make it easy. And again,this
>> happened just on Mac. On CentOS, it works fine.
>> That makes me confusion. :-b
>>
>> 2018-03-05 14:06 GMT+09:00 Jeff Zhang :
>>
>>>
>>> Could you check the interpreter log ? The issue of not be able to
>>> connecting to thrift port could happen due to network issue, like firewall
>>> or you use VPN, etc.
>>>
>>>
>>>
>>> 小野圭二 于2018年3月5日周一 下午12:32写道：
>>>
>>>> Hi,
>>>>
>>>> I show the all log on this error.There are 2 type error. The 1 is at
>>>> the first run. The 2 is at the second run. Seems like could not established
>>>> to Thrift callback port 51348 that was set at the first run.
>>>> First of all, does it need to establish the callback port of Thrift
>>>> server?
>>>> The notebook script was just
>>>> %pyspark
>>>> print 'aaa'
>>>> Any advice appreciate.
>>>>
>>>> 1.
>>>> INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13}
>>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO
>>>> [2018-03-05 13:11:13,934] ({pool-2-thread-2}
>>>> SchedulerFactory.java[jobStarted]:109)
>>>> - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.
>>>> interpreter.remote.RemoteInterpreter-shared_session INFO [2018-03-05
>>>> 13:11:13,936] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run
>>>> paragraph 20180302-135047_99102973 using pyspark
>>>> RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
>>>> INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
>>>> ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) -
>>>> Create InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process
>>>> WARN [2018-03-05 13:11:14,010] ({pool-2-thread-2}
>>>> InterpreterSetting.java[setupPropertiesForSparkR]:796)
>>>> - sparkr.zip is not found, SparkR may not work. INFO [2018-03-05
>>>> 13:11:14,017] ({pool-2-thread-2}
>>>> RemoteInterpreterManagedProcess.java[start]:123)
>>>> - Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05
>>>> 13:11:14,023] ({pool-2-thread-2}
>>>> RemoteInterpreterManagedProcess.java[start]:130)
>>>> - Thrift server for callback will start. Port: 51348 INFO [2018-03-05
>>>> 13:11:14,049] ({pool-2-thread-2}
>>>> RemoteInterpreterManagedProcess.java[start]:205)
>>>> - Run interpreter process
>>>> [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh,
>>>> -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c,
>>>> 172.16.20.10, -p, 51348, -l,
>>>> /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX,
>>>> -g, spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor}
>>>> RemoteInterpreterManagedProcess.java[onProcessComplete]:258) -
>>>> Interpreter process exited 0 ERROR [2018-03-05 13:12:14,058]
>>>> ({pool-2-thread-2} Job.java[run]:184) - Job failed org.apache.zeppelin.
>>>> interpreter.InterpreterException: Cannot run interpreter at
>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProces
>>>> s.start(RemoteInterpreterManagedProcess.java:220) at
>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.
>>>> getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at
>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_
>>>> create(RemoteInterpreter.java:153) at org.apache.zeppelin.
>>>> interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
>>>> at org.apache.zeppelin.interpreter.remote.
>>>> RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at
>>>> org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at
>>>> org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
>>>> org.apache.zeppelin.sche
Re: Thrift server for call back will start. Port:
I wonder why, but it seems work fine, and would like to share how to fix it.
First of all, this issue happened just on MAC.
[THE ENV WAS]
macOS High Sierra v 10.13.3
cpu core i5
mem 8GB
Zeppelin 0.8.0 snapshot
Spark 2.2.0
[ISSUE]
see this previous mails
[SOLUTION]
ref.
http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Zeppelin-0-6-2-Error-Interpreter-process-failed-td4743.html
According to the above ref, i removed 'SPARK_HOME',
'SPARK_SUBMIT_OPTIONS' from my environment parameters and
conf/zeppelin-env.sh(set it as same like CentOS). Then restart zeppelin,
that's all.
[TO DO]
Seems like it is not good chemistry between Spark and Zeppelin on Mac.
>From the reason that the consequence of removing the environment
parameters, Zeppelin headed to own spark lib.
Maybe should write this in the document as caution.
Cheers,
Ono Keiji
2018-03-06 9:15 GMT+09:00 小野圭二 :
> Nothing changed. I guess this property does not effect cause of failing to
> call the interpreter.
> i challenged it as log4j...zeppelin=DEBUG, but no way.
>
> 2018-03-05 18:24 GMT+09:00 Jeff Zhang :
>
>>
>> Try to add the following to log4j.properties for more log
>>
>> log4j.logger.org.apache.zeppelin.interpreter=DEBUG
>>
>>
>>
>> 小野圭二 于2018年3月5日周一 下午5:13写道：
>>
>>> Actually, there was not created the interpreter log file as said 'Cannot
>>> run interpreter' in the log. And it was run on local machine as MAC, i mean
>>> stand alone. i stopped every firewall to make it easy. And again,this
>>> happened just on Mac. On CentOS, it works fine.
>>> That makes me confusion. :-b
>>>
>>> 2018-03-05 14:06 GMT+09:00 Jeff Zhang :
>>>
>>>>
>>>> Could you check the interpreter log ? The issue of not be able to
>>>> connecting to thrift port could happen due to network issue, like firewall
>>>> or you use VPN, etc.
>>>>
>>>>
>>>>
>>>> 小野圭二 于2018年3月5日周一 下午12:32写道：
>>>>
>>>>> Hi,
>>>>>
>>>>> I show the all log on this error.There are 2 type error. The 1 is at
>>>>> the first run. The 2 is at the second run. Seems like could not
>>>>> established
>>>>> to Thrift callback port 51348 that was set at the first run.
>>>>> First of all, does it need to establish the callback port of Thrift
>>>>> server?
>>>>> The notebook script was just
>>>>> %pyspark
>>>>> print 'aaa'
>>>>> Any advice appreciate.
>>>>>
>>>>> 1.
>>>>> INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13}
>>>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO
>>>>> [2018-03-05 13:11:13,934] ({pool-2-thread-2}
>>>>> SchedulerFactory.java[jobStarted]:109)
>>>>> - Job 20180302-135047_99102973 started by scheduler
>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
>>>>> INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
>>>>> Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using
>>>>> pyspark
>>>>> RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
>>>>> INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
>>>>> ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) -
>>>>> Create InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process
>>>>> WARN [2018-03-05 13:11:14,010] ({pool-2-thread-2}
>>>>> InterpreterSetting.java[setupPropertiesForSparkR]:796) - sparkr.zip
>>>>> is not found, SparkR may not work. INFO [2018-03-05 13:11:14,017]
>>>>> ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:123) -
>>>>> Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05
>>>>> 13:11:14,023] ({pool-2-thread-2}
>>>>> RemoteInterpreterManagedProcess.java[start]:130)
>>>>> - Thrift server for callback will start. Port: 51348 INFO [2018-03-05
>>>>> 13:11:14,049] ({pool-2-thread-2}
>>>>> RemoteInterpreterManagedProcess.java[start]:205)
>>>>> - Run interpreter process
>>>>> [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh,
>>>>> -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c,
>>>>> 172.16.20.10, -p, 51348, -l,
>>>>> /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX,
>>>>&
Re: Thrift server for call back will start. Port:
Hi Jeff,
I did set SPARK_HOME to use external spark at first. And that was fine work
on CentOS, but not on Mac.
Do you mean try it again?
2018-03-06 11:12 GMT+09:00 Jeff Zhang :
> >>> Seems like it is not good chemistry between Spark and Zeppelin on
> Mac. From the reason that the consequence of removing the environment
> parameters, Zeppelin headed to own spark lib.
>
> Could you specify SPARK_HOME in your spark interpreter to use external
> spark instead of zeppelin's own spark lib ?
>
>
>
>
> 小野圭二 于2018年3月6日周二 上午10:06写道：
>
>> I wonder why, but it seems work fine, and would like to share how to fix
>> it.
>>
>> First of all, this issue happened just on MAC.
>> [THE ENV WAS]
>> macOS High Sierra v 10.13.3
>> cpu core i5
>> mem 8GB
>> Zeppelin 0.8.0 snapshot
>> Spark 2.2.0
>> [ISSUE]
>>see this previous mails
>> [SOLUTION]
>>ref. http://apache-zeppelin-users-incubating-mailing-list.
>> 75479.x6.nabble.com/Zeppelin-0-6-2-Error-Interpreter-
>> process-failed-td4743.html
>>According to the above ref, i removed 'SPARK_HOME',
>> 'SPARK_SUBMIT_OPTIONS' from my environment parameters and
>> conf/zeppelin-env.sh(set it as same like CentOS). Then restart zeppelin,
>> that's all.
>> [TO DO]
>>Seems like it is not good chemistry between Spark and Zeppelin on Mac.
>> From the reason that the consequence of removing the environment
>> parameters, Zeppelin headed to own spark lib.
>>Maybe should write this in the document as caution.
>>
>>
>> Cheers,
>>
>> Ono Keiji
>>
>>
>>
>>
>> 2018-03-06 9:15 GMT+09:00 小野圭二 :
>>
>>> Nothing changed. I guess this property does not effect cause of failing
>>> to call the interpreter.
>>> i challenged it as log4j...zeppelin=DEBUG, but no way.
>>>
>>> 2018-03-05 18:24 GMT+09:00 Jeff Zhang :
>>>
>>>>
>>>> Try to add the following to log4j.properties for more log
>>>>
>>>> log4j.logger.org.apache.zeppelin.interpreter=DEBUG
>>>>
>>>>
>>>>
>>>> 小野圭二 于2018年3月5日周一 下午5:13写道：
>>>>
>>>>> Actually, there was not created the interpreter log file as said
>>>>> 'Cannot run interpreter' in the log. And it was run on local machine as
>>>>> MAC, i mean stand alone. i stopped every firewall to make it easy. And
>>>>> again,this happened just on Mac. On CentOS, it works fine.
>>>>> That makes me confusion. :-b
>>>>>
>>>>> 2018-03-05 14:06 GMT+09:00 Jeff Zhang :
>>>>>
>>>>>>
>>>>>> Could you check the interpreter log ? The issue of not be able to
>>>>>> connecting to thrift port could happen due to network issue, like
>>>>>> firewall
>>>>>> or you use VPN, etc.
>>>>>>
>>>>>>
>>>>>>
>>>>>> 小野圭二 于2018年3月5日周一 下午12:32写道：
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I show the all log on this error.There are 2 type error. The 1 is at
>>>>>>> the first run. The 2 is at the second run. Seems like could not
>>>>>>> established
>>>>>>> to Thrift callback port 51348 that was set at the first run.
>>>>>>> First of all, does it need to establish the callback port of Thrift
>>>>>>> server?
>>>>>>> The notebook script was just
>>>>>>> %pyspark
>>>>>>> print 'aaa'
>>>>>>> Any advice appreciate.
>>>>>>>
>>>>>>> 1.
>>>>>>> INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13}
>>>>>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO
>>>>>>> [2018-03-05 13:11:13,934] ({pool-2-thread-2}
>>>>>>> SchedulerFactory.java[jobStarted]:109)
>>>>>>> - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.
>>>>>>> interpreter.remote.RemoteInterpreter-shared_session INFO
>>>>>>> [2018-03-05 13:11:13,936] ({pool-2-thread-2}
>>>>>>> Paragraph.java[jobRun]:397) -
>>>>>>> run paragraph 20180302-135047_99102973 using pyspark
>>>>>>> RemoteInterpreter_org.apache.zeppelin.spark.
>>>>>>> PySparkInterpr
Re: Thrift server for call back will start. Port:
Combinations: SPARK_HOME set in 'O'->point to external spark , no '-' ->head to internal spark lib spark interpreter setting 〇 〇 - - 〇 - - zeppelin -env.sh - - - - 〇 〇 〇 environment parameters - 〇 〇 - 〇 〇 - Work Fine:〇 NG:× 〇 〇 〇 〇 × × × The consequences are zeppelin-env.sh has a top priority. And if you want to use external Spark on Mac, you should not set SPARK_HOME in zeppelin-env.sh. Set it as environment parameters or in spark interpreter. Cheers, Ono Keiji 2018-03-06 12:23 GMT+09:00 Jeff Zhang : > > If you set SPARK_HOME in spark interpreter setting, then please remove > SPARK_HOME in spark-env.sh, otherwise you do will hit weird issue. > > > > > 小野圭二 于2018年3月6日周二 上午10:56写道： > >> Hi Jeff, >> >> I did set SPARK_HOME to use external spark at first. And that was fine >> work on CentOS, but not on Mac. >> Do you mean try it again? >> >> 2018-03-06 11:12 GMT+09:00 Jeff Zhang : >> >>> >>> Seems like it is not good chemistry between Spark and Zeppelin on >>> Mac. From the reason that the consequence of removing the environment >>> parameters, Zeppelin headed to own spark lib. >>> >>> Could you specify SPARK_HOME in your spark interpreter to use external >>> spark instead of zeppelin's own spark lib ? >>> >>> >>> >>> >>> 小野圭二 于2018年3月6日周二 上午10:06写道： >>> >>>> I wonder why, but it seems work fine, and would like to share how to >>>> fix it. >>>> >>>> First of all, this issue happened just on MAC. >>>> [THE ENV WAS] >>>> macOS High Sierra v 10.13.3 >>>> cpu core i5 >>>> mem 8GB >>>> Zeppelin 0.8.0 snapshot >>>> Spark 2.2.0 >>>> [ISSUE] >>>>see this previous mails >>>> [SOLUTION] >>>>ref. http://apache-zeppelin-users-incubating-mailing-list. >>>> 75479.x6.nabble.com/Zeppelin-0-6-2-Error-Interpreter- >>>> process-failed-td4743.html >>>>According to the above ref, i removed 'SPARK_HOME', >>>> 'SPARK_SUBMIT_OPTIONS' from my environment parameters and >>>> conf/zeppelin-env.sh(set it as same like CentOS). Then restart zeppelin, >>>> that's all. >>>> [TO DO] >>>>Seems like it is not good chemistry between Spark and Zeppelin on >>>> Mac. From the reason that the consequence of removing the environment >>>> parameters, Zeppelin headed to own spark lib. >>>>Maybe should write this in the document as caution. >>>> >>>> >>>> Cheers, >>>> >>>> Ono Keiji >>>> >>>> >>>> >>>> >>>> 2018-03-06 9:15 GMT+09:00 小野圭二 : >>>> >>>>> Nothing changed. I guess this property does not effect cause of >>>>> failing to call the interpreter. >>>>> i challenged it as log4j...zeppelin=DEBUG, but no way. >>>>> >>>>> 2018-03-05 18:24 GMT+09:00 Jeff Zhang : >>>>> >>>>>> >>>>>> Try to add the following to log4j.properties for more log >>>>>> >>>>>> log4j.logger.org.apache.zeppelin.interpreter=DEBUG >>>>>> >>>>>> >>>>>> >>>>>> 小野圭二 于2018年3月5日周一 下午5:13写道： >>>>>> >>>>>>> Actually, there was not created the interpreter log file as said >>>>>>> 'Cannot run interpreter' in the log. And it was run on local machine as >>>>>>> MAC, i mean stand alone. i stopped every firewall to make it easy. And >>>>>>> again,this happened just on Mac. On CentOS, it works fine. >>>>>>> That makes me confusion. :-b >>>>>>> >>>>>>> 2018-03-05 14:06 GMT+09:00 Jeff Zhang : >>>>>>> >>>>>>>> >>>>>>>> Could you check the interpreter log ? The issue of not be able to >>>>>>>> connecting to thrift port could happen due to network issue, like >>>>>>>> firewall >>>>>>>> or you use VPN, etc. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> 小野圭二 于2018年3月5日周一 下午12:32写道： >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I show the all log o
