Using pyvis and other javascript wrappers in Zeppelin
All:
Googled this problem but have not found a solution that works.
I am trying to use pyvis to generate interactive network graphs.
The pyvis functions output an HTML file. I have tried several approaches
but not found anything that works for something that should be simple.
Here is my sample code:
%ipython
from pyvis.network import Network
from IPython.display import HTML
import networkx as nx
import os
nx_graph = nx.cycle_graph(10)
nx_graph.nodes[1]['title'] = 'Number 1'
nx_graph.nodes[1]['group'] = 1
nx_graph.nodes[3]['title'] = 'I belong to a different group!'
nx_graph.nodes[3]['group'] = 10
nx_graph.add_node(20, size=20, title='couple', group=2)
nx_graph.add_node(21, size=15, title='couple', group=2)
nx_graph.add_edge(20, 21, weight=5)
nx_graph.add_node(25, size=25, label='lonely', title='lonely node',
group=3)
nt = Network("500px", "500px",notebook=True)
# populates the nodes and edges data structures
nt.from_nx(nx_graph)
nt.prep_notebook()
nt.show("/tmp/nx.html")
file = open("/tmp/nx.html")
htmldata=file.read()
HTML(htmldata)
So I was able to get this to work.
But if I have a subsequent note paragraph to display a different
graph the graph in the first note is overwritten.
Surely there is a better approach and I am looking for suggestions
on approaches and ways to get multiple graphs to display.
--
= mailto:db...@incadencecorp.com
David W. Boyd
VP, Data Solutions
10432 Balls Ford, Suite 240
Manassas, VA 20109
office: +1-703-552-2862
cell: +1-703-402-7908
== http://www.incadencecorp.com/
ISO/IEC JTC1 SC42/WG2, editor ISO/IEC 20546, ISO/IEC 20547-1
Chair INCITS TG Big Data
Co-chair NIST Big Data Public Working Group Reference Architecture
First Robotic Mentor - FRC, FTC - www.iliterobotics.org
Board Member- USSTEM Foundation - www.usstem.org
The information contained in this message may be privileged
and/or confidential and protected from disclosure.
If the reader of this message is not the intended recipient
or an employee or agent responsible for delivering this message
to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication
is strictly prohibited. If you have received this communication
in error, please notify the sender immediately by replying to
this message and deleting the material from any computer.
Re: 0.9.0 - R tutorial - google vis - no output
Jeff:
Good to know.
Thank you so much for the prompt responses and patience with me.
On 7/2/2020 8:16 PM, Jeff Zhang wrote:
It looks like a bug, I can reproduce this issue.
David Boyd mailto:db...@incadencecorp.com>>
于2020年7月3日周五 上午7:43写道：
All:
Sorry for dumping so much on the list lately. Making progress
but it is slow.
So I finally got the %ir interpreter to work.
Got pretty much every paragraph to work with either %r or %ir
But I am not able to get any of the google vis outputs to generate
any output.
First question - is this because flash has basically been disabled
across the board?
I found
If I look at say the bubble object from that paragrah it looks
like it has real data:
$type
'BubbleChart'
$chartid
'BubbleChartID30605c24963'
$html
$header
'https://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd;
<https://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd>>\nhttps://www.w3.org/1999/xhtml;
<https://www.w3.org/1999/xhtml>>\n\nBubbleChartID30605c24963\n\n\nbody {\n color: #44;\n font-family:
Arial,Helvetica,sans-serif;\n font-size: 75%;\n }\n a {\n
color: #4D87C7;\n text-decoration:
none;\n}\n\n\n\n'
$chart
jsHeader
'\n\n\n\n\n
Re: %r.ir vs %ir
Jeff: mvn clean package -DskipTests -Pspark-2.4 -Phadoop2 -Pscala-2.11 -Dcheckstyle.skip -Drat.skip=true On 7/2/2020 7:22 PM, Jeff Zhang wrote: What's your command of building zeppelin. ? David Boyd mailto:db...@incadencecorp.com>> 于2020年7月3日周五 上午6:59写道： All: So with some wailing and knashing of teeth, I was able to build the IRKernel from source and install that into my R environment. So the tutorial uses r.ir <http://r.ir> - If I run a paragraph with that I get the below stack trace. If I run with just %ir it works (for some paragraphs). It also works for some paragraphs with %spark.ir <http://spark.ir> Why the difference? Running 0.9.0 built from source 2-3 days ago. org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: Fail to init SparkRBackend at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:760) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:668) at org.apache.zeppelin.scheduler.Job.run(Job.java:172) at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:130) at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:39) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.zeppelin.interpreter.InterpreterException: Fail to init SparkRBackend at org.apache.zeppelin.r.IRInterpreter.open(IRInterpreter.java:99) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70) ... 8 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.zeppelin.r.SparkRBackend.init(SparkRBackend.java:56) at org.apache.zeppelin.r.IRInterpreter.open(IRInterpreter.java:97) ... 9 more Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream at org.apache.spark.SparkConf.loadFromSystemProperties(SparkConf.scala:76) at org.apache.spark.SparkConf.(SparkConf.scala:71) at org.apache.spark.SparkConf.(SparkConf.scala:58) at org.apache.spark.api.r.RBackend.init(RBackend.scala:51) ... 15 more Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 19 more -- =mailto:db...@incadencecorp.com <mailto:db...@incadencecorp.com> David W. Boyd VP, Data Solutions 10432 Balls Ford, Suite 240 Manassas, VA 20109 office: +1-703-552-2862 cell: +1-703-402-7908 ==http://www.incadencecorp.com/ <http://www.incadencecorp.com/> ISO/IEC JTC1 SC42/WG2, editor ISO/IEC 20546, ISO/IEC 20547-1 Chair INCITS TG Big Data Co-chair NIST Big Data Public Working Group Reference Architecture First Robotic Mentor - FRC, FTC -www.iliterobotics.org <http://www.iliterobotics.org> Board Member- USSTEM Foundation -www.usstem.org <http://www.usstem.org> The information contained in this message may be privileged and/or confidential and protected from disclosure. If the reader of this message is not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and deleting the material from any computer. -- Best Regards Jeff Zhang -- = mailto:db...@incadencecorp.com David W. Boyd VP, Data Solutions 10432 Balls Ford, Suite 240 Manassas, VA 20109 office: +1-703-552-2862 cell: +1-703-402-7908 == http://www.incadencecorp.com/ ISO/IEC JTC1 SC42/WG2, editor ISO/IEC 20546, ISO/IEC 20547-1 Chair INCITS TG Big Data Co-chair NIST Big Data Public Working Group Reference Ar
0.9.0 - R tutorial - google vis - no output
All:
Sorry for dumping so much on the list lately. Making progress but it
is slow.
So I finally got the %ir interpreter to work.
Got pretty much every paragraph to work with either %r or %ir
But I am not able to get any of the google vis outputs to generate
any output.
First question - is this because flash has basically been disabled
across the board?
I found
If I look at say the bubble object from that paragrah it looks
like it has real data:
$type
'BubbleChart'
$chartid
'BubbleChartID30605c24963'
$html
$header
'https://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd;>\nhttps://www.w3.org/1999/xhtml;>\n\nBubbleChartID30605c24963\n\n\nbody {\n color: #44;\n
font-family: Arial,Helvetica,sans-serif;\n font-size: 75%;\n
}\n a {\n color: #4D87C7;\n text-decoration:
none;\n}\n\n\n\n'
$chart
jsHeader
%r.ir vs %ir
All: So with some wailing and knashing of teeth, I was able to build the IRKernel from source and install that into my R environment. So the tutorial uses r.ir - If I run a paragraph with that I get the below stack trace. If I run with just %ir it works (for some paragraphs). It also works for some paragraphs with %spark.ir Why the difference? Running 0.9.0 built from source 2-3 days ago. org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: Fail to init SparkRBackend at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:760) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:668) at org.apache.zeppelin.scheduler.Job.run(Job.java:172) at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:130) at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:39) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.zeppelin.interpreter.InterpreterException: Fail to init SparkRBackend at org.apache.zeppelin.r.IRInterpreter.open(IRInterpreter.java:99) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70) ... 8 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.zeppelin.r.SparkRBackend.init(SparkRBackend.java:56) at org.apache.zeppelin.r.IRInterpreter.open(IRInterpreter.java:97) ... 9 more Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream at org.apache.spark.SparkConf.loadFromSystemProperties(SparkConf.scala:76) at org.apache.spark.SparkConf.(SparkConf.scala:71) at org.apache.spark.SparkConf.(SparkConf.scala:58) at org.apache.spark.api.r.RBackend.init(RBackend.scala:51) ... 15 more Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 19 more -- = mailto:db...@incadencecorp.com David W. Boyd VP, Data Solutions 10432 Balls Ford, Suite 240 Manassas, VA 20109 office: +1-703-552-2862 cell: +1-703-402-7908 == http://www.incadencecorp.com/ ISO/IEC JTC1 SC42/WG2, editor ISO/IEC 20546, ISO/IEC 20547-1 Chair INCITS TG Big Data Co-chair NIST Big Data Public Working Group Reference Architecture First Robotic Mentor - FRC, FTC - www.iliterobotics.org Board Member- USSTEM Foundation - www.usstem.org The information contained in this message may be privileged and/or confidential and protected from disclosure. If the reader of this message is not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and deleting the material from any computer.
Plotting in plain R interpreter generates no output or error
All:
So this is a followup to a prior question. I am running 0.9.0 built
2 days ago from source.
I can not run the IR interpreter because there is no version of it for R
3.6 and I can not downgrade R.
So in the straight %r interpreter I am trying to get plotting to work.
I am running on a secure server as a daemon under a zeppelin user .
The user has a home directory and permissions.
In the R tutorial the google vis paragraphs do not produce
any output:
%r
library(googleVis)
df=data.frame(country=c("US", "GB", "BR"),
val1=c(10,13,14),
val2=c(23,12,32))
Bar <- gvisBarChart(df)
print(Bar, tag = 'chart')
The straight plotting paragrahs:
%r
pairs(iris)
Product this error:
Error in value[[3L]](cond): could not open file
'figure/unnamed-chunk-1-1.png'
I have tried setting the working directory to the home dir and I can
view the
contents but get the same error.
I have looked through the source code and simply have no idea where this
process is running or where it is trying to write out the image
My end customer for this effort is really into R so the ability to
produce graphs in Zeppelin in R is critical.
I welcome ANY suggestion on how to demonstrate graphs/plots.
--
= mailto:db...@incadencecorp.com
David W. Boyd
VP, Data Solutions
10432 Balls Ford, Suite 240
Manassas, VA 20109
office: +1-703-552-2862
cell: +1-703-402-7908
== http://www.incadencecorp.com/
ISO/IEC JTC1 SC42/WG2, editor ISO/IEC 20546, ISO/IEC 20547-1
Chair INCITS TG Big Data
Co-chair NIST Big Data Public Working Group Reference Architecture
First Robotic Mentor - FRC, FTC - www.iliterobotics.org
Board Member- USSTEM Foundation - www.usstem.org
The information contained in this message may be privileged
and/or confidential and protected from disclosure.
If the reader of this message is not the intended recipient
or an employee or agent responsible for delivering this message
to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication
is strictly prohibited. If you have received this communication
in error, please notify the sender immediately by replying to
this message and deleting the material from any computer.
spark.jars.packages not working in spark interpreter tutorial
All:
Trying to run the Spark Interpreter tutorial note.
The spark.conf paragraph which specifies spark.jars.packages runs clean.
But the next paragraph which tries to use the avro jar fails with a
class not found for
org.apache.spark.sql.avro.AvroFileFormat.DefaultSource
Spark is set to run Per Note in scoped process.
There are no errors in the
zeppelin-interpreter-spark-shared_process-zeppelin-dspcnode11.dspc.incadencecorp.com.log
Any thoughts would be appreciated.
Note Spark Basic Features tutorial works fine.
The base zeppelin log has this error:
INFO [2020-07-02 15:57:58,339] ({qtp923219673-142}
VFSNotebookRepo.java[save]:145) - Saving note 2F8KN6TKK to Spark
Tutorial/1. Spark Interpreter Introduction_2F8KN6TKK.zpln
INFO [2020-07-02 15:57:58,343] ({SchedulerFactory3}
AbstractScheduler.java[runJob]:125) - Job 20180530-222838_1995256600
started by scheduler RemoteInterpreter-spark-shared_process-2F8KN6TKK
INFO [2020-07-02 15:57:58,343] ({SchedulerFactory3}
Paragraph.java[jobRun]:388) - Run paragraph [paragraph_id:
20180530-222838_1995256600, interpreter:
org.apache.zeppelin.spark.SparkInterpreter, note_id: 2F8KN6TKK, user:
dspc_demo]
INFO [2020-07-02 15:57:58,444]
({JobStatusPoller-20180530-222838_1995256600}
NotebookServer.java[onStatusChange]:1927) - Job
20180530-222838_1995256600 starts to RUNNING
INFO [2020-07-02 15:57:58,445]
({JobStatusPoller-20180530-222838_1995256600}
VFSNotebookRepo.java[save]:145) - Saving note 2F8KN6TKK to Spark
Tutorial/1. Spark Interpreter Introduction_2F8KN6TKK.zpln
WARN [2020-07-02 15:57:58,734] ({SchedulerFactory3}
NotebookServer.java[onStatusChange]:1924) - Job
20180530-222838_1995256600 is finished, status: ERROR, exception:
null, result: %text java.lang.ClassNotFoundException: Failed to find
data source: org.apache.spark.sql.avro.AvroFileFormat. Please find
packages at http://spark.apache.org/third-party-projects.html
at
org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:657)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:194)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
... 45 elided
Caused by: java.lang.ClassNotFoundException:
org.apache.spark.sql.avro.AvroFileFormat.DefaultSource
at
scala.reflect.internal.util.AbstractFileClassLoader.findClass(AbstractFileClassLoader.scala:62)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at
org.apache.spark.sql.execution.datasources.DataSource$$anonfun$20$$anonfun$apply$12.apply(DataSource.scala:634)
at
org.apache.spark.sql.execution.datasources.DataSource$$anonfun$20$$anonfun$apply$12.apply(DataSource.scala:634)
at scala.util.Try$.apply(Try.scala:192)
at
org.apache.spark.sql.execution.datasources.DataSource$$anonfun$20.apply(DataSource.scala:634)
at
org.apache.spark.sql.execution.datasources.DataSource$$anonfun$20.apply(DataSource.scala:634)
at scala.util.Try.orElse(Try.scala:84)
at
org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:634)
... 47 more
INFO [2020-07-02 15:57:58,735] ({SchedulerFactory3}
VFSNotebookRepo.java[save]:145) - Saving note 2F8KN6TKK to Spark
Tutorial/1. Spark Interpreter Introduction_2F8KN6TKK.zpln
--
= mailto:db...@incadencecorp.com
David W. Boyd
VP, Data Solutions
10432 Balls Ford, Suite 240
Manassas, VA 20109
office: +1-703-552-2862
cell: +1-703-402-7908
== http://www.incadencecorp.com/
ISO/IEC JTC1 SC42/WG2, editor ISO/IEC 20546, ISO/IEC 20547-1
Chair INCITS TG Big Data
Co-chair NIST Big Data Public Working Group Reference Architecture
First Robotic Mentor - FRC, FTC - www.iliterobotics.org
Board Member- USSTEM Foundation - www.usstem.org
The information contained in this message may be privileged
and/or confidential and protected from disclosure.
If the reader of this message is not the intended recipient
or an employee or agent responsible for delivering this message
to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication
is strictly prohibited. If you have received this communication
in error, please notify the sender immediately by replying to
this message and deleting the material from any computer.
Re: 0.9.0 - R tutorial - no such kernel named ir
All:
So it turns out that the IRKernel is not available for
R 3.6 (actually one of its dependencies)
** testing if installed package keeps a record of temporary
installation path
* DONE (IRdisplay)
Making 'packages.html' ... done
ERROR: dependency ‘pbdZMQ’ is not available for package ‘IRkernel’
* removing ‘/usr/lib64/R/library/IRkernel’
I can not downgrade R because that is inherited by the environment.
So I went back to a straight %r interpreter.
Which takes me back to the original issue I had with Zeppelin 0.8.2 .
When I run the googlevis tests I get no errors and not output.
When I run the plot tests I get:
Error in value[[3L]](cond): could not open file
'figure/unnamed-chunk-1-1.png'
Need to find where zeppelin thinks the directory is located.
On 7/1/2020 9:56 AM, Jeff Zhang wrote:
If you are using %r.ir <http://r.ir>, then you need to install
irkernel https://github.com/IRkernel/IRkernel
<https://github.com/IRkernel/IRkernel>
here's full doc about R interpreter in Zeppelin
http://zeppelin.apache.org/docs/0.9.0-preview1/interpreter/r.html
<http://zeppelin.apache.org/docs/0.9.0-preview1/interpreter/r.html>
David Boyd mailto:db...@incadencecorp.com>>
于2020年7月1日周三 下午7:45写道：
Sorry to be such a pain but I am under a tight deadline to get
this operational.
In running the r tutorial I get this message that says there is no
such kernel named ir
I can run the same paragraph with a straight %r interpreter just
fine.
Where would this get set up?
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.zeppelin.interpreter.InterpreterException: Fail to
open JupyterKernelInterpreter: java.io.IOException: Fail to
launch Jupyter Kernel as the python process is failed.
starting...E0701 11:28:38.350657744 25080
socket_utils_common_posix.cc:222] check for SO_REUSEPORT:
{"created":"@1593602918.350637104","description":"SO_REUSEPORT
unavailable on compiling
system","file":"src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":190}
starting...E0701 11:28:38.350657744 25080
socket_utils_common_posix.cc:222] check for SO_REUSEPORT:
{"created":"@1593602918.350637104","description":"SO_REUSEPORT
unavailable on compiling
system","file":"src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":190}
Traceback (most recent call last): File
"/tmp/zeppelin_jupyter_kernel_ir3856431290391083629/kernel_server.py",
line 186, in serve(sys.argv[1], sys.argv[2]) File
"/tmp/zeppelin_jupyter_kernel_ir3856431290391083629/kernel_server.py",
line 172, in serve kernel.start() File
"/tmp/zeppelin_jupyter_kernel_ir3856431290391083629/kernel_server.py",
line 50, in start self._km, self._kc =
jupyter_client.manager.start_new_kernel(kernel_name=self._kernel_name)
File
"/data01/zeppelin/miniconda/envs/py37/lib/python3.8/site-packages/jupyter_client/manager.py",
line 745, in start_new_kernel km.start_kernel(**kwargs) File
"/data01/zeppelin/miniconda/envs/py37/lib/python3.8/site-packages/jupyter_client/manager.py",
line 301, in start_kernel kernel_cmd, kw =
self.pre_start_kernel(**kw) File
"/data01/zeppelin/miniconda/envs/py37/lib/python3.8/site-packages/jupyter_client/manager.py",
line 254, in pre_start_kernel kernel_cmd =
self.format_kernel_cmd(extra_arguments=extra_arguments) File
"/data01/zeppelin/miniconda/envs/py37/lib/python3.8/site-packages/jupyter_client/manager.py",
line 178, in format_kernel_cmd cmd = self.kernel_spec.argv +
extra_arguments File
"/data01/zeppelin/miniconda/envs/py37/lib/python3.8/site-packages/jupyter_client/manager.py",
line 84, in kernel_spec self._kernel_spec =
self.kernel_spec_manager.get_kernel_spec(self.kernel_name) File
"/data01/zeppelin/miniconda/envs/py37/lib/python3.8/site-packages/jupyter_client/kernelspec.py",line
235, in get_kernel_spec raise NoSuchKernel(kernel_name)
jupyter_client.kernelspec.NoSuchKernel: No such kernel named ir
at
org.apache.zeppelin.jupyter.JupyterKernelInterpreter.launchJupyterKernel(JupyterKernelInterpreter.java:207)
at
org.apache.zeppelin.jupyter.JupyterKernelInterpreter.open(JupyterKernelInterpreter.java:129)
at
org.apache.zeppelin.r.IRInterpreter.open(IRInterpreter.java:90)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:760)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:668)
Re: ipython interpreter - ipython not installed
So it looks like bokeh 2.1.1 which is what gets installed by default has
removed builtin support for zeppelin.
If I take out the bkzep I now get:
RuntimeError: no display hook installed for notebook type 'zeppelin'
Does anyone know the last version of Bokeh to work?
On 7/1/2020 10:35 AM, Jeff Zhang wrote:
The latest bokeh doesn't need bkzep, sorry, We should update the
tutorial.
https://github.com/zjffdu/bkzep/issues/12
<https://github.com/zjffdu/bkzep/issues/12>
David Boyd mailto:db...@incadencecorp.com>>
于2020年7月1日周三 下午10:28写道：
Jeff:
So I got the worst of this fixed by blowing away my python
virtual env and rebuilding it
but using pip instead of conda for most installs.
Attached is my current environment.yml file that works. This may
be something
to think about building and including in the distribution.
I am down to an error with bkzep now on some of the visualizations.
The error is bundle not defined.
Any suggestions?
On 7/1/2020 9:57 AM, Jeff Zhang wrote:
What error do you see now ?
David Boyd mailto:db...@incadencecorp.com>> 于2020年7月1日周三 下午7:24写道：
Jeff:
Checked that. And tried and explicit path to the python.
In the same notebook I created the following shell note.
Not sure why the ipython note is failing.
I added all the settings for the environment to
zeppelin-env.sh which should
get them into the interpreter correct?
%sh
which python
python -V
conda info --envs
conda list -n py37 ipython
conda list -n py37 jupyter
which produces the output and shows the python environment is
correct and ipython is installed:
/data01/zeppelin/miniconda/envs/py37/bin/python
Python 3.8.3
# conda environments:
#
base /data01/zeppelin/miniconda
py37 * /data01/zeppelin/miniconda/envs/py37
# packages in environment at
/data01/zeppelin/miniconda/envs/py37:
#
# Name Version Build Channel
ipython 7.16.1 py38h5ca1d4c_0
ipython_genutils 0.2.0 py38_0
# packages in environment at
/data01/zeppelin/miniconda/envs/py37:
#
# Name Version Build Channel
jupyter 1.0.0 py38_7
jupyter_client 6.1.3 py_0 conda-forge
jupyter_console 6.1.0 py_0
jupyter_core 4.6.3 py38_0 conda-forge
On 7/1/2020 1:25 AM, Jeff Zhang wrote:
Do you have multiple versions of python installed? You need
to set zeppelin.python to the right python that has ipython
installed.
David Boyd mailto:db...@incadencecorp.com>> 于2020年7月1日周三
下午12:16写道：
All:
I am trying to run the ipython example in the tutorials.
I have ipython installed. From the conda environment:
- ipython=7.16.1=py38h5ca1d4c_0
- ipython_genutils=0.2.0=py38_0
I am getting the following error:
> org.apache.zeppelin.interpreter.InterpreterException:
> org.apache.zeppelin.interpreter.InterpreterException:
Fail to open
> JupyterKernelInterpreter:
> org.apache.zeppelin.interpreter.InterpreterException:
Kernel
> prerequisite is not meet: ipython is not installed. at
>
org.apache.zeppelin.jupyter.JupyterKernelInterpreter.open(JupyterKernelInterpreter.java:116)
> at
>
org.apache.zeppelin.python.IPythonInterpreter.open(IPythonInterpreter.java:109)
> at
>
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
> at
>
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:760)
> at
>
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:668)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:172) at
>
org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:130)
> at
>
org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:39)
> at
>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) at
>
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76)
> at
Re: ipython interpreter - ipython not installed
Jeff:
So I got the worst of this fixed by blowing away my python virtual
env and rebuilding it
but using pip instead of conda for most installs.
Attached is my current environment.yml file that works. This may be
something
to think about building and including in the distribution.
I am down to an error with bkzep now on some of the visualizations.
The error is bundle not defined.
Any suggestions?
On 7/1/2020 9:57 AM, Jeff Zhang wrote:
What error do you see now ?
David Boyd mailto:db...@incadencecorp.com>>
于2020年7月1日周三 下午7:24写道：
Jeff:
Checked that. And tried and explicit path to the python.
In the same notebook I created the following shell note.
Not sure why the ipython note is failing.
I added all the settings for the environment to zeppelin-env.sh
which should
get them into the interpreter correct?
%sh
which python
python -V
conda info --envs
conda list -n py37 ipython
conda list -n py37 jupyter
which produces the output and shows the python environment is
correct and ipython is installed:
/data01/zeppelin/miniconda/envs/py37/bin/python
Python 3.8.3
# conda environments:
#
base /data01/zeppelin/miniconda
py37 * /data01/zeppelin/miniconda/envs/py37
# packages in environment at /data01/zeppelin/miniconda/envs/py37:
#
# Name Version Build Channel
ipython 7.16.1 py38h5ca1d4c_0
ipython_genutils 0.2.0 py38_0
# packages in environment at /data01/zeppelin/miniconda/envs/py37:
#
# Name Version Build Channel
jupyter 1.0.0 py38_7
jupyter_client 6.1.3 py_0 conda-forge
jupyter_console 6.1.0 py_0
jupyter_core 4.6.3 py38_0 conda-forge
On 7/1/2020 1:25 AM, Jeff Zhang wrote:
Do you have multiple versions of python installed? You need to
set zeppelin.python to the right python that has ipython installed.
David Boyd mailto:db...@incadencecorp.com>> 于2020年7月1日周三 下午12:16写道：
All:
I am trying to run the ipython example in the tutorials.
I have ipython installed. From the conda environment:
- ipython=7.16.1=py38h5ca1d4c_0
- ipython_genutils=0.2.0=py38_0
I am getting the following error:
> org.apache.zeppelin.interpreter.InterpreterException:
> org.apache.zeppelin.interpreter.InterpreterException: Fail
to open
> JupyterKernelInterpreter:
> org.apache.zeppelin.interpreter.InterpreterException: Kernel
> prerequisite is not meet: ipython is not installed. at
>
org.apache.zeppelin.jupyter.JupyterKernelInterpreter.open(JupyterKernelInterpreter.java:116)
> at
>
org.apache.zeppelin.python.IPythonInterpreter.open(IPythonInterpreter.java:109)
> at
>
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
> at
>
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:760)
> at
>
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:668)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:172) at
>
org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:130)
> at
>
org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:39)
> at
>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) at
>
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76)
> at
>
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:760)
> at
>
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:668)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:172) at
>
org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:130)
> at
>
org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:39)
> at
>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) Caused by:
> org.apache.zeppelin.interpreter.InterpreterException: Fail
to open
0.9.0 - R tutorial - no such kernel named ir
Sorry to be such a pain but I am under a tight deadline to get this
operational.
In running the r tutorial I get this message that says there is no such
kernel named ir
I can run the same paragraph with a straight %r interpreter just fine.
Where would this get set up?
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.zeppelin.interpreter.InterpreterException: Fail to open
JupyterKernelInterpreter: java.io.IOException: Fail to launch Jupyter
Kernel as the python process is failed. starting...E0701
11:28:38.350657744 25080 socket_utils_common_posix.cc:222] check for
SO_REUSEPORT:
{"created":"@1593602918.350637104","description":"SO_REUSEPORT
unavailable on compiling
system","file":"src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":190}
starting...E0701 11:28:38.350657744 25080
socket_utils_common_posix.cc:222] check for SO_REUSEPORT:
{"created":"@1593602918.350637104","description":"SO_REUSEPORT
unavailable on compiling
system","file":"src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":190}
Traceback (most recent call last): File
"/tmp/zeppelin_jupyter_kernel_ir3856431290391083629/kernel_server.py",
line 186, in serve(sys.argv[1], sys.argv[2]) File
"/tmp/zeppelin_jupyter_kernel_ir3856431290391083629/kernel_server.py",
line 172, in serve kernel.start() File
"/tmp/zeppelin_jupyter_kernel_ir3856431290391083629/kernel_server.py",
line 50, in start self._km, self._kc =
jupyter_client.manager.start_new_kernel(kernel_name=self._kernel_name)
File
"/data01/zeppelin/miniconda/envs/py37/lib/python3.8/site-packages/jupyter_client/manager.py",
line 745, in start_new_kernel km.start_kernel(**kwargs) File
"/data01/zeppelin/miniconda/envs/py37/lib/python3.8/site-packages/jupyter_client/manager.py",
line 301, in start_kernel kernel_cmd, kw = self.pre_start_kernel(**kw)
File
"/data01/zeppelin/miniconda/envs/py37/lib/python3.8/site-packages/jupyter_client/manager.py",
line 254, in pre_start_kernel kernel_cmd =
self.format_kernel_cmd(extra_arguments=extra_arguments) File
"/data01/zeppelin/miniconda/envs/py37/lib/python3.8/site-packages/jupyter_client/manager.py",
line 178, in format_kernel_cmd cmd = self.kernel_spec.argv +
extra_arguments File
"/data01/zeppelin/miniconda/envs/py37/lib/python3.8/site-packages/jupyter_client/manager.py",
line 84, in kernel_spec self._kernel_spec =
self.kernel_spec_manager.get_kernel_spec(self.kernel_name) File
"/data01/zeppelin/miniconda/envs/py37/lib/python3.8/site-packages/jupyter_client/kernelspec.py",line
235, in get_kernel_spec raise NoSuchKernel(kernel_name)
jupyter_client.kernelspec.NoSuchKernel: No such kernel named ir at
org.apache.zeppelin.jupyter.JupyterKernelInterpreter.launchJupyterKernel(JupyterKernelInterpreter.java:207)
at
org.apache.zeppelin.jupyter.JupyterKernelInterpreter.open(JupyterKernelInterpreter.java:129)
at org.apache.zeppelin.r.IRInterpreter.open(IRInterpreter.java:90) at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:760)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:668)at
org.apache.zeppelin.scheduler.Job.run(Job.java:172) at
org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:130)
at
org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:39)at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76)at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:760)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:668)at
org.apache.zeppelin.scheduler.Job.run(Job.java:172) at
org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:130)
at
org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:39)at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) Caused by:
org.apache.zeppelin.interpreter.InterpreterException: Fail to open
JupyterKernelInterpreter: java.io.IOException: Fail to launch Jupyter
Kernel as the python process is failed. starting...E0701
11:28:38.350657744 25080 socket_utils_common_posix.cc:222] check for
SO_REUSEPORT:
{"created":"@1593602918.350637104","description":"SO_REUSEPORT
unavailable on compiling
system","file":"src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":190}
starting...E0701
Re: ipython interpreter - ipython not installed
Jeff: Checked that. And tried and explicit path to the python. In the same notebook I created the following shell note. Not sure why the ipython note is failing. I added all the settings for the environment to zeppelin-env.sh which should get them into the interpreter correct? %sh which python python -V conda info --envs conda list -n py37 ipython conda list -n py37 jupyter which produces the output and shows the python environment is correct and ipython is installed: /data01/zeppelin/miniconda/envs/py37/bin/python Python 3.8.3 # conda environments: # base /data01/zeppelin/miniconda py37 * /data01/zeppelin/miniconda/envs/py37 # packages in environment at /data01/zeppelin/miniconda/envs/py37: # # Name Version Build Channel ipython 7.16.1 py38h5ca1d4c_0 ipython_genutils 0.2.0 py38_0 # packages in environment at /data01/zeppelin/miniconda/envs/py37: # # Name Version Build Channel jupyter 1.0.0 py38_7 jupyter_client 6.1.3 py_0 conda-forge jupyter_console 6.1.0 py_0 jupyter_core 4.6.3 py38_0 conda-forge On 7/1/2020 1:25 AM, Jeff Zhang wrote: Do you have multiple versions of python installed? You need to set zeppelin.python to the right python that has ipython installed. David Boyd mailto:db...@incadencecorp.com>> 于2020年7月1日周三 下午12:16写道： All: I am trying to run the ipython example in the tutorials. I have ipython installed. From the conda environment: - ipython=7.16.1=py38h5ca1d4c_0 - ipython_genutils=0.2.0=py38_0 I am getting the following error: > org.apache.zeppelin.interpreter.InterpreterException: > org.apache.zeppelin.interpreter.InterpreterException: Fail to open > JupyterKernelInterpreter: > org.apache.zeppelin.interpreter.InterpreterException: Kernel > prerequisite is not meet: ipython is not installed. at > org.apache.zeppelin.jupyter.JupyterKernelInterpreter.open(JupyterKernelInterpreter.java:116) > at > org.apache.zeppelin.python.IPythonInterpreter.open(IPythonInterpreter.java:109) > at > org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70) > at > org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:760) > at > org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:668) > at org.apache.zeppelin.scheduler.Job.run(Job.java:172) at > org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:130) > at > org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:39) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) at > org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76) > at > org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:760) > at > org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:668) > at org.apache.zeppelin.scheduler.Job.run(Job.java:172) at > org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:130) > at > org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:39) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) Caused by: > org.apache.zeppelin.interpreter.InterpreterException: Fail to open > JupyterKernelInterpreter: > org.apache.zeppelin.interpreter.InterpreterException: Kernel > prerequisite is not meet: ipython is not installed. at > org.apache.zeppelin.jupyter.JupyterKernelInterpreter.open(JupyterKernelInterpreter.java:116) > at > org.apache.zeppelin.python.IPythonInterpreter.open(IPythonInterpreter.java:109) > at > org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70) > at > org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:760) > at > org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:668) > at org.apache.zeppelin.scheduler.Job.run(Job.java:172) at > org.apache.zeppelin.scheduler.Abst
Re: Installing python packages to support tutorial
All: I got around the problem by going onto my server. Installing each package one at a time, then exporting the environment to a file. I then used that to perform my installs via puppet. On 6/30/2020 7:23 PM, David Boyd wrote: All: So I am setting up a python based virtual environment to support zeppelin. Has anyone sucessfully set up a virtual environment with all the packages for the python tutorial? If so how? An hour plus after I ran conda to set up the env, I got massive conflict errors. I created an environment.yml file with all the packages that are referenced in the tutorial as shown below: name: py37 channels: - conda-forge dependencies: - python=3.7 - numpy - pandas - jupyter - grpcio - protobuf - matplotlib - seaborn - bokeh - holoviews - altair - keras - ggplot - plotnine Using conda In attempted to create the environment: conda env create -f environment.yml An hour plus later I get back the following list of incompatibilities: UnsatisfiableError: The following specifications were found to be incompatible with each other: Output in format: Requested package -> Available versions Package python conflicts for: pandas -> python-dateutil[version='>=2.6.1'] -> python[version='3.7.*|3.8.*'] plotnine -> python[version='2.7.*|3.5.*|3.6.*|>=3.5.0|>=3.6.0'] jupyter -> ipykernel -> python[version='3.4.*|>=3|>=3.4|>=3.5'] holoviews -> bokeh[version='>=1.1.0'] -> python[version='>=2.7|>=3|>=3.8,<3.9.0a0'] protobuf -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0|3.4.*'] ggplot -> cycler -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0'] protobuf -> python_abi=3.8[build=*_cp38] -> python[version='3.7.*|3.8.*'] keras -> h5py -> python[version='3.7.*|>=3.8,<3.9.0a0|3.8.*'] bokeh -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0'] altair -> python[version='2.7.*|3.4.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.5|>=3.6|>=3.8,<3.9.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.5,<3.6.0a0'] pandas -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0|3.4.*'] keras -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.6|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.5,<3.6.0a0|3.4.*'] grpcio -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0'] matplotlib -> pyqt -> python[version='3.6.*|<3'] matplotlib -> python[version='2.7.*|3.4.*|3.5.*|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0'] seaborn -> statsmodels[version='>=0.8.0'] -> python[version='>=3.8,<3.9.0a0'] python=3.7 seaborn -> python[version='2.7.*|3.5.*|3.6.*|>=3.6|3.4.*|>=3.5,<3.6.0a0|>=3.7,<3.8.0a0|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0'] jupyter -> python[version='2.7.*|3.5.*|3.6.*|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0|>=3.6,<3.7.0a0|>=2.7,<2.8.0a0|>=3.7,<3.8.0a0'] grpcio -> python_abi=3.7[build=*_cp37m] -> python[version='2.7.*|3.5.*|3.6.*|3.7.*|3.4.*|3.8.*'] ggplot -> python[version='2.7.*|3.5.*|3.6.*|3.4.*'] plotnine -> descartes[version='>=1.1.0'] -> python[version='3.4.*|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.8,<3.9.0a0|>=3.7,<3.8.0a0|>=3.5,<3.6.0a0|>=3.5'] holoviews -> python[version='2.7.*|3.5.*|3.6.*|3.4.*|>=3.5,<3.6.0a0|>=3.7,<3.8.0a0|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0'] bokeh -> jinja2[version='>=2.7'] -> python[version='3.4.*|3.7.*|>=3.5|3.8.*'] Package setuptools conflicts for: altair -> jinja2 -> setuptools[version='>=18.5'] keras -> setuptools ggplot -> brewer2mpl -> setuptools holoviews -> ipython[version='>=5.4.0'] -> setuptools[version='>=18.5'] grpcio -> setuptools matplotlib -> setuptools protobuf -> setuptools plotnine -> matplotlib-base[version='>=3.1.1'] -> setuptools python=3.7 -> pip -> setuptools bokeh -> jinja2[version='>=2.7'] -> setuptools seaborn -> matplotlib-base[version='>=2.1.2'] -> setuptools Package enum34 conflicts for: keras -> tensorflow -> enum34[version='>=1.1.6'] matplotlib -> pyqt -> enum34 altair -> traitlets -> enum34 grpcio -> enum34[version='>=1.0.4'] Package python-dateutil conflicts for: plotnine -> matplotlib-base[version='>=3.1.1'] -> python-dateutil[version='>=2.5.*|>=2.6.1'] holoviews -> bokeh[version='>=1.1.0'] -> pyth
Installing python packages to support tutorial
All: So I am setting up a python based virtual environment to support zeppelin. Has anyone sucessfully set up a virtual environment with all the packages for the python tutorial? If so how? An hour plus after I ran conda to set up the env, I got massive conflict errors. I created an environment.yml file with all the packages that are referenced in the tutorial as shown below: name: py37 channels: - conda-forge dependencies: - python=3.7 - numpy - pandas - jupyter - grpcio - protobuf - matplotlib - seaborn - bokeh - holoviews - altair - keras - ggplot - plotnine Using conda In attempted to create the environment: conda env create -f environment.yml An hour plus later I get back the following list of incompatibilities: UnsatisfiableError: The following specifications were found to be incompatible with each other: Output in format: Requested package -> Available versions Package python conflicts for: pandas -> python-dateutil[version='>=2.6.1'] -> python[version='3.7.*|3.8.*'] plotnine -> python[version='2.7.*|3.5.*|3.6.*|>=3.5.0|>=3.6.0'] jupyter -> ipykernel -> python[version='3.4.*|>=3|>=3.4|>=3.5'] holoviews -> bokeh[version='>=1.1.0'] -> python[version='>=2.7|>=3|>=3.8,<3.9.0a0'] protobuf -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0|3.4.*'] ggplot -> cycler -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0'] protobuf -> python_abi=3.8[build=*_cp38] -> python[version='3.7.*|3.8.*'] keras -> h5py -> python[version='3.7.*|>=3.8,<3.9.0a0|3.8.*'] bokeh -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0'] altair -> python[version='2.7.*|3.4.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.5|>=3.6|>=3.8,<3.9.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.5,<3.6.0a0'] pandas -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0|3.4.*'] keras -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.6|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.5,<3.6.0a0|3.4.*'] grpcio -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0'] matplotlib -> pyqt -> python[version='3.6.*|<3'] matplotlib -> python[version='2.7.*|3.4.*|3.5.*|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0'] seaborn -> statsmodels[version='>=0.8.0'] -> python[version='>=3.8,<3.9.0a0'] python=3.7 seaborn -> python[version='2.7.*|3.5.*|3.6.*|>=3.6|3.4.*|>=3.5,<3.6.0a0|>=3.7,<3.8.0a0|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0'] jupyter -> python[version='2.7.*|3.5.*|3.6.*|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0|>=3.6,<3.7.0a0|>=2.7,<2.8.0a0|>=3.7,<3.8.0a0'] grpcio -> python_abi=3.7[build=*_cp37m] -> python[version='2.7.*|3.5.*|3.6.*|3.7.*|3.4.*|3.8.*'] ggplot -> python[version='2.7.*|3.5.*|3.6.*|3.4.*'] plotnine -> descartes[version='>=1.1.0'] -> python[version='3.4.*|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.8,<3.9.0a0|>=3.7,<3.8.0a0|>=3.5,<3.6.0a0|>=3.5'] holoviews -> python[version='2.7.*|3.5.*|3.6.*|3.4.*|>=3.5,<3.6.0a0|>=3.7,<3.8.0a0|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0'] bokeh -> jinja2[version='>=2.7'] -> python[version='3.4.*|3.7.*|>=3.5|3.8.*'] Package setuptools conflicts for: altair -> jinja2 -> setuptools[version='>=18.5'] keras -> setuptools ggplot -> brewer2mpl -> setuptools holoviews -> ipython[version='>=5.4.0'] -> setuptools[version='>=18.5'] grpcio -> setuptools matplotlib -> setuptools protobuf -> setuptools plotnine -> matplotlib-base[version='>=3.1.1'] -> setuptools python=3.7 -> pip -> setuptools bokeh -> jinja2[version='>=2.7'] -> setuptools seaborn -> matplotlib-base[version='>=2.1.2'] -> setuptools Package enum34 conflicts for: keras -> tensorflow -> enum34[version='>=1.1.6'] matplotlib -> pyqt -> enum34 altair -> traitlets -> enum34 grpcio -> enum34[version='>=1.0.4'] Package python-dateutil conflicts for: plotnine -> matplotlib-base[version='>=3.1.1'] -> python-dateutil[version='>=2.5.*|>=2.6.1'] holoviews -> bokeh[version='>=1.1.0'] -> python-dateutil[version='>=2.1|>=2.6.1|>=2.5.*'] matplotlib -> python-dateutil pandas -> python-dateutil[version='>=2.5.*|>=2.6.1'] bokeh -> python-dateutil[version='>=2.1'] ggplot -> matplotlib-base -> python-dateutil[version='>=2.5.*|>=2.6.1'] seaborn -> matplotlib-base[version='>=2.1.2'] -> python-dateutil[version='>=2.5.*|>=2.6.1'] bokeh -> matplotlib -> python-dateutil[version='>=2.5.*|>=2.6.1'] altair -> pandas -> python-dateutil[version='>=2.5.*|>=2.6.1'] Package functools32 conflicts for: plotnine -> matplotlib[version='>=2.1.0'] -> functools32 seaborn -> matplotlib-base[version='>=2.1.2'] -> functools32 ggplot -> matplotlib-base -> functools32 holoviews -> matplotlib-base[version='>=2.2'] -> functools32 matplotlib -> functools32 bokeh -> matplotlib -> functools32 altair -> jsonschema -> functools32 Package expat conflicts for: pandas -> pypy3.6[version='>=7.3.1'] ->
Error starting spark interpreter with 0.9.0
All: Just trying to get 0.9.0 to work and running into all sorts of issues. Previously I had set SPARK_MASTER to be yarn-client so it would use my existing yarn cluster. That threw an error about yarn-client being deprecated in 2.0. So I switched it to local. I now get the error about the interpreter not starting and the following output in the note: org.apache.zeppelin.interpreter.InterpreterException: java.io.IOException: Fail to launch interpreter process: Interpreter launch command: /opt/spark/spark-current/bin/spark-submit --class org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer --driver-class-path ":/opt/zeppelin/zeppelin-current/interpreter/spark/*::/opt/hadoop/hadoop-current/share/hadoop/common/sources/:/opt/hadoop/hadoop-current/share/hadoop/common/sources/:/opt/zeppelin/zeppelin-current/interpreter/zeppelin-interpreter-shaded-0.9.0-SNAPSHOT-shaded.jar /opt/zeppelin/zeppelin-current/interpreter/zeppelin-interpreter-shaded-0.9.0-SNAPSHOT.jar:/opt/zeppelin/zeppelin-current/interpreter/spark/spark-interpreter-0.9.0-SNAPSHOT.jar:/opt/hadoop/hadoop-current/etc/hadoop" --driver-java-options " -Dfile.encoding=UTF-8 -Dlog4j.configuration='file:///opt/zeppelin/zeppelin-current/conf/log4j.properties' -Dlog4j.configurationFile='file:///opt/zeppelin/zeppelin-current/conf/log4j2.properties' -Dzeppelin.log.file='/opt/zeppelin/zeppelin-current/logs/zeppelin-interpreter-spark-dspc_demo-zeppelin-dspcnode11.dspc.incadencecorp.com.log'" --driver-memory 4G --executor-memory 6G --conf spark\.serializer\=org\.apache\.spark\.serializer\.KryoSerializer --conf spark\.executor\.memory\=1G --conf spark\.app\.name\=Zeppelin --conf spark\.executor\.instances\=5 --conf spark\.master\=local\[\*\] --conf spark\.sql\.crossJoin\.enabled\=true --conf spark\.cores\.max\=10 /opt/zeppelin/zeppelin-current/interpreter/spark/spark-interpreter-0.9.0-SNAPSHOT.jar 10.1.50.111 33591 "spark-dspc_demo" : SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/zeppelin/zeppelin-0.9.0-SNAPSHOT/interpreter/spark/spark-interpreter-0.9.0-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/spark/spark-2.4.3.bdp-1-bin-hadoop2.7/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:134) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:281) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:412) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:72) at org.apache.zeppelin.scheduler.Job.run(Job.java:172) at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:130) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:180) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Fail to launch interpreter process: Interpreter launch command: /opt/spark/spark-current/bin/spark-submit --class org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer --driver-class-path ":/opt/zeppelin/zeppelin-current/interpreter/spark/*::/opt/hadoop/hadoop-current/share/hadoop/common/sources/:/opt/hadoop/hadoop-current/share/hadoop/common/sources/:/opt/zeppelin/zeppelin-current/interpreter/zeppelin-interpreter-shaded-0.9.0-SNAPSHOT-shaded.jar /opt/zeppelin/zeppelin-current/interpreter/zeppelin-interpreter-shaded-0.9.0-SNAPSHOT.jar:/opt/zeppelin/zeppelin-current/interpreter/spark/spark-interpreter-0.9.0-SNAPSHOT.jar:/opt/hadoop/hadoop-current/etc/hadoop" --driver-java-options " -Dfile.encoding=UTF-8 -Dlog4j.configuration='file:///opt/zeppelin/zeppelin-current/conf/log4j.properties' -Dlog4j.configurationFile='file:///opt/zeppelin/zeppelin-current/conf/log4j2.properties' -Dzeppelin.log.file='/opt/zeppelin/zeppelin-current/logs/zeppelin-interpreter-spark-dspc_demo-zeppelin-dspcnode11.dspc.incadencecorp.com.log'" --driver-memory 4G --executor-memory 6G --conf spark\.serializer\=org\.apache\.spark\.serializer\.KryoSerializer --conf spark\.executor\.memory\=1G --conf spark\.app\.name\=Zeppelin --conf spark\.executor\.instances\=5 --conf spark\.master\=local\[\*\] --conf
Question about 0.9.0 - Interpreters downloading dependencies
All: Very much a newbie question but one which I have not encountered before. I am working with a build from source of 0.9.0 from the branch-0.9 branch on github. I am running this on AWS on an EC2 node. In the log file when I start up I get a bunch of messages about interpreters downloading dependencies. But I can not see any of them change state to READY in the log. How do I tell if the downloads are taking place? I assume this only happens on the first start up after a clean install? I assume it is downloading the files from the maven URL specified in the zeppelin-site.xml? Where are the downloaded dependencies stored? (I have limited space on some file systems in production). Is there documentation on how to "Pre-download" these and have them installed with the software? -- = mailto:db...@incadencecorp.com David W. Boyd VP, Data Solutions 10432 Balls Ford, Suite 240 Manassas, VA 20109 office: +1-703-552-2862 cell: +1-703-402-7908 == http://www.incadencecorp.com/ ISO/IEC JTC1 SC42/WG2, editor ISO/IEC 20546, ISO/IEC 20547-1 Chair INCITS TG Big Data Co-chair NIST Big Data Public Working Group Reference Architecture First Robotic Mentor - FRC, FTC - www.iliterobotics.org Board Member- USSTEM Foundation - www.usstem.org The information contained in this message may be privileged and/or confidential and protected from disclosure. If the reader of this message is not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and deleting the material from any computer.
Issue with r and plotting in 0.8.2
All: I am testing out a new Zeppelin installation on RHEL 7. It is running under systemd as a zeppelin user. Running through the tutorial tests. In the R tutorial all of the plot and ggplot2 paragraphs give this error: Error in value[[3L]](cond): could not open file 'figure/unnamed-chunk-1-1.png' None of the googlevis paragraphs produce any output and no errors. I am sure this is an installation problem of some sort but I was not able to find any help with google. Also, knitr is not available as an interpreter, what do I need to do to enable that? -- = mailto:db...@incadencecorp.com David W. Boyd VP, Data Solutions 10432 Balls Ford, Suite 240 Manassas, VA 20109 office: +1-703-552-2862 cell: +1-703-402-7908 == http://www.incadencecorp.com/ ISO/IEC JTC1 SC42/WG2, editor ISO/IEC 20546, ISO/IEC 20547-1 Chair INCITS TG Big Data Co-chair NIST Big Data Public Working Group Reference Architecture First Robotic Mentor - FRC, FTC - www.iliterobotics.org Board Member- USSTEM Foundation - www.usstem.org The information contained in this message may be privileged and/or confidential and protected from disclosure. If the reader of this message is not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and deleting the material from any computer.
Issue with r and plotting in 0.8.2
All: I am testing out a new Zeppelin installation on RHEL 7. It is running under systemd as a zeppelin user. Running through the tutorial tests. In the R tutorial all of the plot and ggplot2 paragraphs give this error: Also, none of the googlevis paragraphs produce any output and not errors. I am sure this is an installation problem of some sort but I was not able to find any help with google. Also, knitr is not available as an interpreter, what do I need to do to enable that? -- = mailto:db...@incadencecorp.com David W. Boyd VP, Data Solutions 10432 Balls Ford, Suite 240 Manassas, VA 20109 office: +1-703-552-2862 cell: +1-703-402-7908 == http://www.incadencecorp.com/ ISO/IEC JTC1 SC42/WG2, editor ISO/IEC 20546, ISO/IEC 20547-1 Chair INCITS TG Big Data Co-chair NIST Big Data Public Working Group Reference Architecture First Robotic Mentor - FRC, FTC - www.iliterobotics.org Board Member- USSTEM Foundation - www.usstem.org The information contained in this message may be privileged and/or confidential and protected from disclosure. If the reader of this message is not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and deleting the material from any computer.
Questions on GIT integration
All: I am running 0.8.2 in a docker image. I have all my external directories set up and mounted to the right places. Now I have a GIT repository with several directories, one of which is my notebook directory like follows: repodir docker python shiny notebook I have checked out the repo, the repodir is where the .git directory exists. Zeppelin has its notebook dir pointed to the notebook directory If I change a node a git status command shows that note changed. However, if I use the zeppelin gui and say commit a note Zeppelin creates a new .git directory inside the notebook directory and my git status shows all new notes. How can I configure zeppelin to use a notebook dir that is part of a larger git repository? I also, need to figure out how to have Zeppelin support a push and creating a branch for different users. Any help would be appreciated. -- = mailto:db...@incadencecorp.com David W. Boyd VP, Data Solutions 10432 Balls Ford, Suite 240 Manassas, VA 20109 office: +1-703-552-2862 cell: +1-703-402-7908 == http://www.incadencecorp.com/ ISO/IEC JTC1 SC42/WG2, editor ISO/IEC 20546, ISO/IEC 20547-1 Chair ANSI/INCITS TG Big Data Co-chair NIST Big Data Public Working Group Reference Architecture First Robotic Mentor - FRC, FTC - www.iliterobotics.org Board Member- USSTEM Foundation - www.usstem.org The information contained in this message may be privileged and/or confidential and protected from disclosure. If the reader of this message is not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and deleting the material from any computer.
Re: Should max-results = 0 (or max-rows = 0) mean unlimited results?
Sanjay: The other convention is that -1 is unlimited. However, it is always possible to set it to MAXINT. I would think most folks can work around this by setting the value to a very high number. As long as the constraint only limit printed output (e.g. z.show()) and not what data is pulled into the code. On 5/11/18 10:01 AM, Sanjay Dasgupta wrote: Many of the interpreters have a parameter named maxResults, max_no_of_rows, max_count, etc whose purpose is to limit the number of output rows displayed (for example from z.show(...)). In most (or perhaps all) of these implementations, setting this parameter to 0 causes no output to be displayed at all as 0 is taken literally. In certain other contexts, the value 0 in a configuration parameter is often used as a special indicator meaning "unlimited". We have at least one recent request for such an interpretation of the value "0" in the maximum output rows parameter (see https://issues.apache.org/jira/browse/ZEPPELIN-3446). I would like to ask the user community what they think of making such a change. How common would such use be? are there any downsides? Thanks for your ideas. -- = mailto:db...@incadencecorp.com David W. Boyd VP, Data Solutions 10432 Balls Ford, Suite 240 Manassas, VA 20109 office: +1-703-552-2862 cell: +1-703-402-7908 == http://www.incadencecorp.com/ ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture Chair ANSI/INCITS TC Big Data Co-chair NIST Big Data Public Working Group Reference Architecture First Robotic Mentor - FRC, FTC - www.iliterobotics.org Board Member- USSTEM Foundation - www.usstem.org The information contained in this message may be privileged and/or confidential and protected from disclosure. If the reader of this message is not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and deleting the material from any computer.
Re: Artifact dependency for geomesa causes NPE
All: Ok more details after sifting through logs. This appears to be a
dependency issue. In the interpreter log I
am seeing:
ERROR [2018-05-08 23:27:27,331] ({SparkListenerBus}
Logging.scala[logError]:91) - uncaught error in thread
SparkListenerBus, stopping SparkContext
java.lang.NoSuchMethodError:
org.json4s.Formats.emptyValueStrategy()Lorg/json4s/prefs/EmptyValueStrategy;
at org.json4s.jackson.JsonMethods$class.render(JsonMethods.scala:32)
at org.json4s.jackson.JsonMethods$.render(JsonMethods.scala:50)
at
org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$1.apply(EventLoggingListener.scala:136)
at
org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$1.apply(EventLoggingListener.scala:136)
at scala.Option.foreach(Option.scala:257)
at
org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:136)
at
org.apache.spark.scheduler.EventLoggingListener.onExecutorAdded(EventLoggingListener.scala:187)
at
org.apache.spark.scheduler.SparkListenerBus$class.doPostEvent(SparkListenerBus.scala:61)
at
org.apache.spark.scheduler.LiveListenerBus.doPostEvent(LiveListenerBus.scala:36)
at
org.apache.spark.scheduler.LiveListenerBus.doPostEvent(LiveListenerBus.scala:36)
at
org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:63)
at
org.apache.spark.scheduler.LiveListenerBus.postToAll(LiveListenerBus.scala:36)
at
org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(LiveListenerBus.scala:94)
at
org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79)
at
org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at
org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(LiveListenerBus.scala:78)
at
org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1245)
at
org.apache.spark.scheduler.LiveListenerBus$$anon$1.run(LiveListenerBus.scala:77)
Based on that and after chasing down things in Maven I added the
following excludes to my dependency:
org.json4s:json4s-core_2.11, org.json4s:json4s-jackson_2.11,
org.json4s:json4s-native_2.11
That did not make a difference.
Googling it appears to be an incompatibility between 3.2.10 and 3.2.11
of json4s.
On 5/8/18 6:46 PM, David Boyd wrote:
All:
I am following the instructions here:
http://www.geomesa.org/documentation/current/user/spark/zeppelin.html
To use geomesa spark with zeppelin.
Whenever I add the artifact dependency I get the following error on
any code I try to run (this includes the basic features -spark
tutorial).
java.lang.NullPointerException
at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:38)
at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:33)
at
org.apache.zeppelin.spark.SparkInterpreter.createSparkContext_2(SparkInterpreter.java:398)
at
org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:387)
at
org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:146)
at
org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:843)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:491)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at
org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
I have tried specifying the jar as a maven artifact and as an absolute
path. I have tried multiple versions (1.3.4, 1.3.5, and 2.0.0) of
the artifact. As soon as I remove the dependency the code works again.
Is there another external dependency I can try to see if that is the
problem.
I have tried this with both the 0.7.3 Zeppelin binary distribution,
and also with a 0.7.4 distribution I built specifically.
I am running spark 2.1 on my cluster. Like I said without this
dependency the example code works just fine.
Here is the build command I used for the distribution I am running:
mvn clean package -DskipTests -Pspark-2.1 -Phadoop-2.7 -Pyarn
-Ppyspark -Psparkr -Pr -Pscala-2.11
Artifact dependency for geomesa causes NPE
All: I am following the instructions here: http://www.geomesa.org/documentation/current/user/spark/zeppelin.html To use geomesa spark with zeppelin. Whenever I add the artifact dependency I get the following error on any code I try to run (this includes the basic features -spark tutorial). java.lang.NullPointerException at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:38) at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:33) at org.apache.zeppelin.spark.SparkInterpreter.createSparkContext_2(SparkInterpreter.java:398) at org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:387) at org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:146) at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:843) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:491) at org.apache.zeppelin.scheduler.Job.run(Job.java:175) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) I have tried specifying the jar as a maven artifact and as an absolute path. I have tried multiple versions (1.3.4, 1.3.5, and 2.0.0) of the artifact. As soon as I remove the dependency the code works again. Is there another external dependency I can try to see if that is the problem. I have tried this with both the 0.7.3 Zeppelin binary distribution, and also with a 0.7.4 distribution I built specifically. I am running spark 2.1 on my cluster. Like I said without this dependency the example code works just fine. Here is the build command I used for the distribution I am running: mvn clean package -DskipTests -Pspark-2.1 -Phadoop-2.7 -Pyarn -Ppyspark -Psparkr -Pr -Pscala-2.11 -Pexamples -Pbuild-distr From looking at the code around the trace it appears either a class is not found or something is getting dorked with SparkContext. Any help would be appreciated. -- = mailto:db...@incadencecorp.com David W. Boyd VP, Data Solutions 10432 Balls Ford, Suite 240 Manassas, VA 20109 office: +1-703-552-2862 cell: +1-703-402-7908 == http://www.incadencecorp.com/ ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture Chair ANSI/INCITS TC Big Data Co-chair NIST Big Data Public Working Group Reference Architecture First Robotic Mentor - FRC, FTC - www.iliterobotics.org Board Member- USSTEM Foundation - www.usstem.org The information contained in this message may be privileged and/or confidential and protected from disclosure. If the reader of this message is not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and deleting the material from any computer.
