Thanks!
I think rather than "sudo /etc/init.d/zeppelin start" the usual way would be
"sudo service zeppelin start". However, on EMR 4.x, we actually use upstart
(http://upstart.ubuntu.com) to manage the service processes and ensure that
they are restarted automatically if they die, so the corresponding command on
EMR is "sudo start zeppelin".
> On Jan 14, 2016, at 11:13 PM, Ophir Cohen <oph...@gmail.com> wrote:
>
> Yes, I just checked that out and find out it does not work on 0.6.
> Ok, good luck, you are doing great job in emr!
>
> BTW
> If I may say, you need to add service support to Zeppelin.
> I.e. to allow start of Zeppelin this way:
> sudo /etc/init.d/zeppelin start
>
> *that will start zeppelin under Zeppelin user.
>> On Jan 15, 2016 9:03 AM, "Jonathan Kelly" <jonathaka...@gmail.com> wrote:
>> Yes, there are a lot of changes since 0.5.5, of course, but we're not going
>> to support an unreleased version on EMR. We did that for emr-4.1.0 (the
>> first version to support Zeppelin), but it's much less confusing for us to
>> support only released versions on EMR. As I mentioned earlier, we will
>> upgrade to 0.5.6 once it's released, but as I also mentioned earlier, while
>> the Notebook API appears fixed on 0.5.6, the Job API is not. The Job API,
>> btw, is what allows you to run notebooks via the REST API. The Notebook API
>> only lets you do CRUD on notebooks/paragraphs/crons.
>>
>> FWIW, I also tried building Zeppelin from the tip of the master branch but
>> still hit the same error when using the Job API, but that's not too
>> surprising because v0.5.6 isn't too different from the tip of the master
>> branch, particularly in the REST API code (other than the new Shiro Security
>> API).
>>
>>> On Thu, Jan 14, 2016 at 10:45 PM Ophir Cohen <oph...@gmail.com> wrote:
>>> I havn't try the job api, in fact what is the differences between notebook
>>> and job api?
>>> Anyway, why not jumping to 0.6.0?
>>> Looks like there is MANY improvments and bug fixes there....
>>> Sounds to me that you are debugging last year snow 😉
>>>> On Jan 15, 2016 3:07 AM, "Jonathan Kelly" <jonathaka...@gmail.com> wrote:
>>>> But now some bad news. I was playing around with the REST API a little
>>>> more, and I noticed that I get the same AbstractMethodError on Zeppelin
>>>> 0.5.6 for a POST to /api/job/2A94M5J1Z. :( Does anybody have any idea how
>>>> this error could possibly have been fixed for /api/notebook in one
>>>> release, for /api/notebook/* in the next release, and yet still fail for
>>>> /api/job/* in that same release?
>>>>
>>>>> On Thu, Jan 14, 2016 at 4:13 PM Jonathan Kelly <jonathaka...@gmail.com>
>>>>> wrote:
>>>>> Oh, hey, good news! I just tried Zeppelin 0.5.6 on EMR, and the Notebook
>>>>> API works fine. So now we just need Zeppelin 0.5.6 to be officially
>>>>> released, and then we can support it on a future version of EMR.
>>>>>
>>>>>> On Thu, Jan 14, 2016 at 2:01 AM Ophir Cohen <oph...@gmail.com> wrote:
>>>>>> OK, I've made it work.
>>>>>> As Jonathan stated above, on emr 4.1 the notebook APIs does not work at
>>>>>> all. On both, emr 4.2 and emr 4.3 the /api/notebook works but the query
>>>>>> for a specific notebook not.
>>>>>>
>>>>>> Deploying vanilla Zeppelin (master branch) on the same cluster worked as
>>>>>> a charm.
>>>>>>
>>>>>> It probably origin by Zeppelin versions (0.5.5 on emr vs. 0.6.0 the
>>>>>> latest).
>>>>>>
>>>>>> Now I'm encountering different error that, as far as I can see, does not
>>>>>> interfere Zeppelin work:
>>>>>> ERROR [2016-01-14 09:55:38,499] ({qtp1505370540-32}
>>>>>> NotebookServer.java[onMessage]:176) - Can't handle message
>>>>>> java.lang.NullPointerException
>>>>>>
>>>>>> Many of those.
>>>>>> If anyone knows what does it mean....
>>>>>>
>>>>>>> On Thu, Jan 14, 2016 at 12:23 AM, Jonathan Kelly
>>>>>>> <jonathaka...@gmail.com> wrote:
>>>>>>> Interesting... I confirmed that removing all of the extra classpath
>>>>>>> entries added by EMR in zeppelin-env.sh does not fix this issue. This
>>>>>>> must mean that the issue is caused by the custom EMR builds of either
>>>>>>> Spark or Hadoop, which we use when building Zeppelin.
>>>>>>>
>>>>>>>> On Wed, Jan 13, 2016 at 2:15 PM Jonathan Kelly
>>>>>>>> <jonathaka...@gmail.com> wrote:
>>>>>>>> Ah, never mind. I had incorrectly been excluding the jars in
>>>>>>>> /usr/lib/zeppelin/interpreter, and I just noticed that
>>>>>>>> jersey-core-1.9.jar appears in two places underneath that directory
>>>>>>>> (in the hive and phoenix interpreter libs), and those jars are
>>>>>>>> included in the Zeppelin server classpath. This older version of
>>>>>>>> jersey-core conflicts with jersey-core-1.13.jar from
>>>>>>>> /usr/lib/zeppelin/lib, so this probably doesn't help. However,
>>>>>>>> removing just these two jars did not help by itself.
>>>>>>>>
>>>>>>>> Haven't tried removing classpath entries from zeppelin-env.sh yet
>>>>>>>> though.
>>>>>>>>
>>>>>>>>> On Wed, Jan 13, 2016 at 2:07 PM Jonathan Kelly
>>>>>>>>> <jonathaka...@gmail.com> wrote:
>>>>>>>>> Yeah, I'm sure you're right that it must be due to something that EMR
>>>>>>>>> is putting in the classpath, but the thing is that I didn't see
>>>>>>>>> anything else in the classpath that includes the
>>>>>>>>> javax.ws.rs.core.Response class.
>>>>>>>>>
>>>>>>>>> And yes, I agree that trying to remove these two jars from the
>>>>>>>>> Zeppelin classpath was a futile attempt and would only have helped if
>>>>>>>>> somehow these jars were near duplicates of each other but different
>>>>>>>>> versions of the same thing. Besides, if removing one of the jars had
>>>>>>>>> helped, it would have shown that this was probably not a problem with
>>>>>>>>> EMR specifically but with Zeppelin itself, which of course would have
>>>>>>>>> been doubtful.
>>>>>>>>>
>>>>>>>>> Anyway, I'll try removing the EMR-supplied extra classpath entries
>>>>>>>>> from zeppelin-env.sh.
>>>>>>>>>
>>>>>>>>>> On Wed, Jan 13, 2016 at 1:48 PM Ophir Cohen <oph...@gmail.com> wrote:
>>>>>>>>>> Interesting
>>>>>>>>>>
>>>>>>>>>> I'm using emr 4.1.0, I'll try 4.2.0 tomorrow.
>>>>>>>>>>
>>>>>>>>>> If I need to guess, I don't think that the mis-version dependency
>>>>>>>>>> comes from the zeppelin jars as Zeppelin api works when not in emr.
>>>>>>>>>> It most likely that it comes from the emr classpath.
>>>>>>>>>> I would have removed all the additional paths (set in
>>>>>>>>>> zeppelin-env.sh) and check again.
>>>>>>>>>>
>>>>>>>>>> BTW
>>>>>>>>>> Removing the one of the jars from Zeppelin as you did won't work as
>>>>>>>>>> Zeppelin needs these jars.
>>>>>>>>>> If you really want to test it you need to build zeppelin and
>>>>>>>>>> instructe maven to exclude the problematic dependency fron one of
>>>>>>>>>> these jars.
>>>>>>>>>> In other words, maven should bring the jar, but without the
>>>>>>>>>> duplicate dependency.
>>>>>>>>>> Having said that, and as I stated above, I think that the
>>>>>>>>>> problematic dependency comes from the additional classpath and not
>>>>>>>>>> from Zeppelin core jars.
>>>>>>>>>>
>>>>>>>>>>> On Jan 13, 2016 11:08 PM, "Jonathan Kelly" <jonathaka...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>> Hi, Ophir, (Jonathan from EMR here)
>>>>>>>>>>>
>>>>>>>>>>> Are you using emr-4.1.0 or emr-4.2.0? I just tried this with
>>>>>>>>>>> emr-4.2.0 and found that http://<my-server>:8890/api/notebook works
>>>>>>>>>>> for me, but http://<my-server>:8890/api/notebook/2A94M5J1Z fails.
>>>>>>>>>>> For some reason it doesn't quite fail with the same exception that
>>>>>>>>>>> you're seeing, but it definitely seems like a similar cause anyway.
>>>>>>>>>>> Here's the exception I get:
>>>>>>>>>>>
>>>>>>>>>>> WARN [2016-01-13 20:34:52,279] ({qtp508683864-37}
>>>>>>>>>>> ServletHandler.java[doHandle]:590) - Error for
>>>>>>>>>>> /api/notebook/2A94M5J1Z
>>>>>>>>>>> java.lang.AbstractMethodError:
>>>>>>>>>>> javax.ws.rs.core.Response.getStatusInfo()Ljavax/ws/rs/core/Response$StatusType;
>>>>>>>>>>> at
>>>>>>>>>>> javax.ws.rs.WebApplicationException.validate(WebApplicationException.java:186)
>>>>>>>>>>> at
>>>>>>>>>>> javax.ws.rs.ClientErrorException.<init>(ClientErrorException.java:88)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.cxf.jaxrs.utils.JAXRSUtils.findTargetMethod(JAXRSUtils.java:503)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.processRequest(JAXRSInInterceptor.java:207)
>>>>>>>>>>>
>>>>>>>>>>> I found that the Zeppelin classpath includes two different jars
>>>>>>>>>>> that contain the javax/ws/rs/core/Response class:
>>>>>>>>>>> /usr/lib/zeppelin/lib/javax.ws.rs-api-2.0-m10.jar and
>>>>>>>>>>> /usr/lib/zeppelin/lib/jersey-core-1.13.jar
>>>>>>>>>>>
>>>>>>>>>>> What's weird though is that if I remove either of these from the
>>>>>>>>>>> classpath, the Zeppelin server fails to start up for different
>>>>>>>>>>> reasons. If I remove /usr/lib/zeppelin/lib/jersey-core-1.13.jar, I
>>>>>>>>>>> get a ClassNotFoundException for
>>>>>>>>>>> com.sun.jersey.core.util.FeaturesAndProperties, and if I remove
>>>>>>>>>>> /usr/lib/zeppelin/lib/javax.ws.rs-api-2.0-m10.jar, I get a
>>>>>>>>>>> ClassNotFoundException for javax.ws.rs.NotFoundException.
>>>>>>>>>>>
>>>>>>>>>>> According to a `mvn ... dependency:tree` in the zeppelin-server
>>>>>>>>>>> submodule, /usr/lib/zeppelin/lib/jersey-core-1.13.jar comes
>>>>>>>>>>> indirectly from zeppelin-server's direct dependency on
>>>>>>>>>>> com.sun.jersey:jersey-servlet:jar:1.13:compile, and
>>>>>>>>>>> /usr/lib/zeppelin/lib/javax.ws.rs-api-2.0-m10.jar comes from
>>>>>>>>>>> zeppelin-server's direct dependency on
>>>>>>>>>>> javax.ws.rs:javax.ws.rs-api:jar:2.0-m10:compile. Both of these
>>>>>>>>>>> dependencies seem to have been added a long time ago, and both seem
>>>>>>>>>>> to be used by the websocket API rather than the REST API (at least
>>>>>>>>>>> based upon the Git commit descriptions where they were added), so
>>>>>>>>>>> this confuses me even more.
>>>>>>>>>>>
>>>>>>>>>>> Does anybody from the Zeppelin side have any idea what is going on
>>>>>>>>>>> here?
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Jonathan
>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jan 13, 2016 at 5:34 AM Ophir Cohen <oph...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> Hi All,
>>>>>>>>>>>> We migrated our Zeppelin to use EMR Zeppelin. It's straight
>>>>>>>>>>>> forward and we were happy with the migration till we found out
>>>>>>>>>>>> that something isn't working well with the rest API.
>>>>>>>>>>>>
>>>>>>>>>>>> Calling to the interpreter API:
>>>>>>>>>>>> http://<my-server>:8890/api/interpreter
>>>>>>>>>>>> and this:
>>>>>>>>>>>> http://<my-server>:8890/api/interpreter/setting
>>>>>>>>>>>> Returns results as expected.
>>>>>>>>>>>>
>>>>>>>>>>>> When trying to access the notebooks:
>>>>>>>>>>>> http://<my-server>:8890/api/notebook
>>>>>>>>>>>> Or a specific notebook:
>>>>>>>>>>>> http://<my-server>:8890/api/notebook/2A94M5J1Z
>>>>>>>>>>>> It fails with HTTP 500 error.
>>>>>>>>>>>>
>>>>>>>>>>>> The error I can see in the logs is NoSuchMethodError (see below)
>>>>>>>>>>>> which suggests we might have here 'jar-hell' and the wrong
>>>>>>>>>>>> (probably old) jar loaded instead of the need one - but I can't
>>>>>>>>>>>> figure that out.
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>
>>>>>>>>>>>> The exception:
>>>>>>>>>>>> WARN [2016-01-13 07:47:11,690] ({qtp716961517-38}
>>>>>>>>>>>> ServletHandler.java[doHandle]:590) - Error for
>>>>>>>>>>>> /api/notebook/2A94M5J1Z
>>>>>>>>>>>> java.lang.NoSuchMethodError:
>>>>>>>>>>>> javax.ws.rs.ClientErrorException.validate(Ljavax/ws/rs/core/Response;Ljavax/ws/rs/core/Response$Status$Family;)Ljavax/ws/rs/core/Response;
>>>>>>>>>>>> at
>>>>>>>>>>>> javax.ws.rs.ClientErrorException.<init>(ClientErrorException.java:88)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.cxf.jaxrs.utils.JAXRSUtils.findTargetMethod(JAXRSUtils.java:503)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.processRequest(JAXRSInInterceptor.java:207)
