Andy created ZEPPELIN-5127:
------------------------------
Summary: Livy Interpreter Failing to Connect to Remote EMR
Key: ZEPPELIN-5127
URL: https://issues.apache.org/jira/browse/ZEPPELIN-5127
Project: Zeppelin
Issue Type: Bug
Reporter: Andy
Attachments: Screen Shot 2020-11-09 at 5.55.20 PM.png
Hi,
We have an EMR cluster in AWS running the Livy server. I installed the Zeppelin
binary to my local, and added the master node's IP as the URL to the
interpreter settings. Screenshot is attached.
Livy server settings ({{livy.conf}}) on the cluster are set to defaults.
The issue is, whenever I try to run a {{%livy.spark}} cell, it always times out
when trying to check the Livy server version on the remote cluster. This is the
error message outputted:
{noformat}
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.zeppelin.interpreter.InterpreterException: Fail to create session,
please check livy interpreter log and livy server log
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:760)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:668)
at org.apache.zeppelin.scheduler.Job.run(Job.java:172)
at
org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:130)
at
org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:39)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: Fail to create
session, please check livy interpreter log and livy server log
at
org.apache.zeppelin.livy.BaseLivyInterpreter.open(BaseLivyInterpreter.java:169)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
... 8 more
Caused by: org.apache.zeppelin.livy.LivyException:
org.springframework.web.client.ResourceAccessException: I/O error on GET
request for "http://REDACTED:8998/version": Operation timed out; nested
exception is java.net.ConnectException: Operation timed out
at
org.apache.zeppelin.livy.BaseLivyInterpreter.callRestAPI(BaseLivyInterpreter.java:743)
at
org.apache.zeppelin.livy.BaseLivyInterpreter.callRestAPI(BaseLivyInterpreter.java:696)
at
org.apache.zeppelin.livy.BaseLivyInterpreter.getLivyVersion(BaseLivyInterpreter.java:450)
at
org.apache.zeppelin.livy.BaseLivyInterpreter.open(BaseLivyInterpreter.java:159)
... 9 more
Caused by: org.springframework.web.client.ResourceAccessException: I/O error on
GET request for "http://REDACTED:8998/version": Operation timed out; nested
exception is java.net.ConnectException: Operation timed out
at
org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:633)
at
org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580)
at
org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498)
at
org.apache.zeppelin.livy.BaseLivyInterpreter.callRestAPI(BaseLivyInterpreter.java:716)
... 12 more
Caused by: java.net.ConnectException: Operation timed out
at java.base/sun.nio.ch.Net.connect0(Native Method)
at java.base/sun.nio.ch.Net.connect(Net.java:574)
at java.base/sun.nio.ch.Net.connect(Net.java:563)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:588)
at java.base/java.net.Socket.connect(Socket.java:648)
at java.base/java.net.Socket.connect(Socket.java:597)
at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:182)
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
at java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:341)
at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:362)
at
java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1261)
at
java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1194)
at
java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1082)
at
java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1016)
at
org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:78)
at
org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
at
org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
at
org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:619)
... 15 more{noformat}
One thing to note is that I'm behind a corporate proxy, but I set the env vars
before starting Zeppelin, and when I try curling the URL in a {{%sh}} cell, it
works fine.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
