Re: Cannot fetch dependencies for org.apache.flink:flink-json:1.12.2

2021-06-23 Thread Jeff Zhang 
Why do you use the insecure maven repo
http://insecure.repo1.maven.org/maven2/ ? The default repo should work


Chenyu Zheng  于2021年6月22日周二 下午10:39写道：

> Yes. I tried inside and outside docker container, both could connect to
> the maven repo.
>
>
> Get Outlook for Android 
> --
> *From:* Jeff Zhang 
> *Sent:* Tuesday, June 22, 2021 10:23:17 PM
> *To:* users 
> *Subject:* Re: Cannot fetch dependencies for
> org.apache.flink:flink-json:1.12.2
>
> From the logs, it seems due to the network issue. Have you tried to run it
> outside the docker container ?
>
> Chenyu Zheng  于2021年6月22日周二 下午9:50写道：
>
> Hi contributors,
>
>
>
> I’m trying to download jar dependencies using “flink.execution.packages”
> in “%flink.conf”, but I got an error like below. I have changed my “
> zeppelin.interpreter.dep.mvnRepo” to
> http://insecure.repo1.maven.org/maven2/
> 
> , and I could connect to this maven repo in my zeppelin host/container.
>
>
>
> My Zeppelin version: *0.9.0*
>
>
>
> org.apache.zeppelin.interpreter.InterpreterException:
> org.apache.zeppelin.interpreter.InterpreterException:
> org.apache.zeppelin.shaded.org.eclipse.aether.RepositoryException: Cannot
> fetch dependencies for org.apache.flink:flink-json:1.12.2
>
>  at
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76)
>
>  at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:836)
>
>  at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:744)
>
>  at org.apache.zeppelin.scheduler.Job.run(Job.java:172)
>
>  at
> org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:132)
>
>  at
> org.apache.zeppelin.scheduler.ParallelScheduler.lambda$runJobInScheduler$0(ParallelScheduler.java:46)
>
>  at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
>  at java.lang.Thread.run(Thread.java:748)
>
> Caused by: org.apache.zeppelin.interpreter.InterpreterException:
> org.apache.zeppelin.shaded.org.eclipse.aether.RepositoryException: Cannot
> fetch dependencies for org.apache.flink:flink-json:1.12.2
>
>  at
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76)
>
>  at
> org.apache.zeppelin.interpreter.Interpreter.getInterpreterInTheSameSessionByClassName(Interpreter.java:355)
>
>  at
> org.apache.zeppelin.interpreter.Interpreter.getInterpreterInTheSameSessionByClassName(Interpreter.java:366)
>
>  at
> org.apache.zeppelin.flink.FlinkStreamSqlInterpreter.open(FlinkStreamSqlInterpreter.java:47)
>
>  at
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
>
>  ... 8 more
>
> Caused by:
> org.apache.zeppelin.shaded.org.eclipse.aether.RepositoryException: Cannot
> fetch dependencies for org.apache.flink:flink-json:1.12.2
>
>  at
> org.apache.zeppelin.dep.DependencyResolver.getArtifactsWithDep(DependencyResolver.java:185)
>
>  at
> org.apache.zeppelin.dep.DependencyResolver.loadFromMvn(DependencyResolver.java:133)
>
>  at
> org.apache.zeppelin.dep.DependencyResolver.load(DependencyResolver.java:81)
>
>  at
> org.apache.zeppelin.dep.DependencyResolver.load(DependencyResolver.java:68)
>
>  at
> org.apache.zeppelin.flink.FlinkScalaInterpreter$$anonfun$10.apply(FlinkScalaInterpreter.scala:790)
>
>  at
> org.apache.zeppelin.flink.FlinkScalaInterpreter$$anonfun$10.apply(FlinkScalaInterpreter.scala:790)
>
>  at
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
>
>  at
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
>
>  at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>
>  at
> scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
>
>  at
> scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
>
>  at
> scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:186)
>
>  at
> org.apache.zeppelin.flink.FlinkScalaInterpreter.getUserJarsExceptUdfJars(FlinkScalaInterpreter.scala:790)
>
>  at
> org.apache.zeppelin.flink.FlinkScalaInterpreter.initFlinkConfig(FlinkScalaInterpreter.scala:191)
>
>  at
> org.apache.zeppelin.flink.FlinkScalaInterpreter.open(FlinkScalaInterpreter.scala:112)
>
>  at
> org.apache.zeppelin.flink.FlinkInterpreter.open(FlinkInterpreter.java:65)
>
>  at
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(L

Running spark interpreter with zeppelin on k8s

2021-06-23 Thread Lior Chaga 
I'm trying to deploy zeppelin 0.10 on k8s, using following manual build:

mvn clean package -DskipTests -Pspark-scala-2.12 -Pinclude-hadoop
-Pspark-3.0 -Phadoop2  -Pbuild-distr  -pl
zeppelin-interpreter,zeppelin-zengine,spark/interpreter,spark/spark-dependencies,zeppelin-web,zeppelin-server,zeppelin-distribion,jdbc,zeppelin-plugins/notebookrepo/filesystem,zeppelin-plugins/launcher/k8s-standard
-am


Spark itself is configured to use mesos as resource manager.
It seems as if when trying to start the spark
interpreter, K8sRemoteInterpreterProcess tries to find a sidecar pod for
spark interpreter:

Pod pod = client.pods().inNamespace(namespace).withName(podName).get();

Is there any option not to have spark interpreter as a separate pod, and
instead just create the spark context within the zeppelin process? I'm
trying to understand if I could make zeppelin
use K8sStandardInterpreterLauncher instead (I assume it's an alternative to
the remote interpreter?)

Thanks,
Lior

Re: Running spark interpreter with zeppelin on k8s

2021-06-23 Thread Jeff Zhang 
set zeppelin.run.mode in zeppelin-site.xml to be local

Lior Chaga  于2021年6月23日周三 下午4:35写道：

> I'm trying to deploy zeppelin 0.10 on k8s, using following manual build:
>
> mvn clean package -DskipTests -Pspark-scala-2.12 -Pinclude-hadoop -Pspark-3.0 
> -Phadoop2  -Pbuild-distr  -pl 
> zeppelin-interpreter,zeppelin-zengine,spark/interpreter,spark/spark-dependencies,zeppelin-web,zeppelin-server,zeppelin-distribion,jdbc,zeppelin-plugins/notebookrepo/filesystem,zeppelin-plugins/launcher/k8s-standard
>  -am
>
>
> Spark itself is configured to use mesos as resource manager.
> It seems as if when trying to start the spark
> interpreter, K8sRemoteInterpreterProcess tries to find a sidecar pod for
> spark interpreter:
>
> Pod pod = client.pods().inNamespace(namespace).withName(podName).get();
>
> Is there any option not to have spark interpreter as a separate pod, and
> instead just create the spark context within the zeppelin process? I'm
> trying to understand if I could make zeppelin
> use K8sStandardInterpreterLauncher instead (I assume it's an alternative to
> the remote interpreter?)
>
> Thanks,
> Lior
>


-- 
Best Regards

Jeff Zhang

Re: Running spark interpreter with zeppelin on k8s

2021-06-23 Thread Lior Chaga 
Ok, I've tried it.
Indeed it doesn't look for a spark pod. Other issues though, but if I
wouldn't overcome I'll open a new thread.
Thanks Jeff!

On Wed, Jun 23, 2021 at 11:40 AM Jeff Zhang  wrote:

> set zeppelin.run.mode in zeppelin-site.xml to be local
>
> Lior Chaga  于2021年6月23日周三 下午4:35写道：
>
>> I'm trying to deploy zeppelin 0.10 on k8s, using following manual build:
>>
>> mvn clean package -DskipTests -Pspark-scala-2.12 -Pinclude-hadoop 
>> -Pspark-3.0 -Phadoop2  -Pbuild-distr  -pl 
>> zeppelin-interpreter,zeppelin-zengine,spark/interpreter,spark/spark-dependencies,zeppelin-web,zeppelin-server,zeppelin-distribion,jdbc,zeppelin-plugins/notebookrepo/filesystem,zeppelin-plugins/launcher/k8s-standard
>>  -am
>>
>>
>> Spark itself is configured to use mesos as resource manager.
>> It seems as if when trying to start the spark
>> interpreter, K8sRemoteInterpreterProcess tries to find a sidecar pod for
>> spark interpreter:
>>
>> Pod pod = client.pods().inNamespace(namespace).withName(podName).get();
>>
>> Is there any option not to have spark interpreter as a separate pod, and
>> instead just create the spark context within the zeppelin process? I'm
>> trying to understand if I could make zeppelin
>> use K8sStandardInterpreterLauncher instead (I assume it's an alternative to
>> the remote interpreter?)
>>
>> Thanks,
>> Lior
>>
>
>
> --
> Best Regards
>
> Jeff Zhang
>