Re: [VOTE] Release Apache Spark 1.0.0 (RC11)

[Ankur Dave]

0

OK, I withdraw my downvote.

Ankur 

Kafka + Spark Streaming and NoSuchMethodError, related to Manifest / reflection?

[Sean Owen]

I'd like to resurrect this thread:
http://mail-archives.apache.org/mod_mbox/spark-user/201403.mbox/%3c6d657d19-1ecf-4e92-bf15-cc4762ef9...@thekratos.com%3E

Basically when you call this particular Java-flavored overloading of
KafkaUtils.createStream:
https://github.com/apache/spark/blob/master/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala#L133

... you get

java.lang.NoSuchMethodException:
java.lang.Object.(kafka.utils.VerifiableProperties)
at java.lang.Class.getConstructor0(Class.java:2763)
at java.lang.Class.getConstructor(Class.java:1693)
at 
org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:108)

This doesn't appear to be a version issue. It doesn't appear when
calling other versions of this method. Other overloadings work (well,
have other issues).

Something is making it try to instantiate java.lang.Object as if it's
a Decoder class.

I am wondering about this code at
https://github.com/apache/spark/blob/master/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala#L148

implicit val keyCmd: Manifest[U] =
implicitly[Manifest[AnyRef]].asInstanceOf[Manifest[U]]
implicit val valueCmd: Manifest[T] =
implicitly[Manifest[AnyRef]].asInstanceOf[Manifest[T]]

... where U and T are key/value Decoder types. I don't know enough
Scala to fully understand this, but is it possible this causes the
reflective call later to lose the type and try to instantiate Object?
The AnyRef made me wonder.

@tdas I'm hoping you might have some insight as it came in this commit
in January:
https://github.com/apache/spark/commit/aa99f226a691ddcb4442d60f4cd4908f434cc4ce

I'll file a JIRA if it's legitimate; just asking first.

Clearspring Analytics Version

[Surendranauth Hiraman]

Hi,

It looks like the version of Clearspring's stream analytics class in 1.0
branch and master is 2.5

There were some significant bug fixes in 2.6 and version 2.7 is just out
now as well.

Are there any plans to upgrade?

The QDigest deserialization code in 2.5 seems to have bugs that are fixed
in 2.6 and higher.


  
com.clearspring.analytics
stream
2.5.1-
  



SUREN HIRAMAN, VP TECHNOLOGY
Velos
Accelerating Machine Learning

440 NINTH AVENUE, 11TH FLOOR
NEW YORK, NY 10001
O: (917) 525-2466 ext. 105
F: 646.349.4063
E: suren.hiraman@v elos.io
W: www.velos.io

Re: Clearspring Analytics Version

[Reynold Xin]

Would you like to submit a pull request to update it?

Also in the latest version HyperLogLog is serializable. That means we can
get rid of the SerializableHyperLogLog class. (and move to use
HyperLogLogPlus).






On Tue, May 27, 2014 at 3:01 PM, Surendranauth Hiraman <
suren.hira...@velos.io> wrote:

> Hi,
>
> It looks like the version of Clearspring's stream analytics class in 1.0
> branch and master is 2.5
>
> There were some significant bug fixes in 2.6 and version 2.7 is just out
> now as well.
>
> Are there any plans to upgrade?
>
> The QDigest deserialization code in 2.5 seems to have bugs that are fixed
> in 2.6 and higher.
>
>
>   
> com.clearspring.analytics
> stream
> 2.5.1-
>   
>
>
>
> SUREN HIRAMAN, VP TECHNOLOGY
> Velos
> Accelerating Machine Learning
>
> 440 NINTH AVENUE, 11TH FLOOR
> NEW YORK, NY 10001
> O: (917) 525-2466 ext. 105
> F: 646.349.4063
> E: suren.hiraman@v elos.io
> W: www.velos.io
>

Re: Clearspring Analytics Version

[Surendranauth Hiraman]

Great, I will submit the pull request.

Any objection to version 2.7?

-Suren



On Tue, May 27, 2014 at 6:02 PM, Reynold Xin  wrote:

> Would you like to submit a pull request to update it?
>
> Also in the latest version HyperLogLog is serializable. That means we can
> get rid of the SerializableHyperLogLog class. (and move to use
> HyperLogLogPlus).
>
>
>
>
>
>
> On Tue, May 27, 2014 at 3:01 PM, Surendranauth Hiraman <
> suren.hira...@velos.io> wrote:
>
> > Hi,
> >
> > It looks like the version of Clearspring's stream analytics class in 1.0
> > branch and master is 2.5
> >
> > There were some significant bug fixes in 2.6 and version 2.7 is just out
> > now as well.
> >
> > Are there any plans to upgrade?
> >
> > The QDigest deserialization code in 2.5 seems to have bugs that are fixed
> > in 2.6 and higher.
> >
> >
> >   
> > com.clearspring.analytics
> > stream
> > 2.5.1-
> >   
> >
> >
> >
> > SUREN HIRAMAN, VP TECHNOLOGY
> > Velos
> > Accelerating Machine Learning
> >
> > 440 NINTH AVENUE, 11TH FLOOR
> > NEW YORK, NY 10001
> > O: (917) 525-2466 ext. 105
> > F: 646.349.4063
> > E: suren.hiraman@v elos.io
> > W: www.velos.io
> >
>



-- 

SUREN HIRAMAN, VP TECHNOLOGY
Velos
Accelerating Machine Learning

440 NINTH AVENUE, 11TH FLOOR
NEW YORK, NY 10001
O: (917) 525-2466 ext. 105
F: 646.349.4063
E: suren.hiraman@v elos.io
W: www.velos.io

Re: Clearspring Analytics Version

[Reynold Xin]

2.7 sounds good. I was actually waiting for 2.7 to come out to post a JIRA
(mainly for the serializable HyperLogLogPlus class).


On Tue, May 27, 2014 at 3:11 PM, Surendranauth Hiraman <
suren.hira...@velos.io> wrote:

> Great, I will submit the pull request.
>
> Any objection to version 2.7?
>
> -Suren
>
>
>
> On Tue, May 27, 2014 at 6:02 PM, Reynold Xin  wrote:
>
> > Would you like to submit a pull request to update it?
> >
> > Also in the latest version HyperLogLog is serializable. That means we can
> > get rid of the SerializableHyperLogLog class. (and move to use
> > HyperLogLogPlus).
> >
> >
> >
> >
> >
> >
> > On Tue, May 27, 2014 at 3:01 PM, Surendranauth Hiraman <
> > suren.hira...@velos.io> wrote:
> >
> > > Hi,
> > >
> > > It looks like the version of Clearspring's stream analytics class in
> 1.0
> > > branch and master is 2.5
> > >
> > > There were some significant bug fixes in 2.6 and version 2.7 is just
> out
> > > now as well.
> > >
> > > Are there any plans to upgrade?
> > >
> > > The QDigest deserialization code in 2.5 seems to have bugs that are
> fixed
> > > in 2.6 and higher.
> > >
> > >
> > >   
> > > com.clearspring.analytics
> > > stream
> > > 2.5.1-
> > >   
> > >
> > >
> > >
> > > SUREN HIRAMAN, VP TECHNOLOGY
> > > Velos
> > > Accelerating Machine Learning
> > >
> > > 440 NINTH AVENUE, 11TH FLOOR
> > > NEW YORK, NY 10001
> > > O: (917) 525-2466 ext. 105
> > > F: 646.349.4063
> > > E: suren.hiraman@v elos.io
> > > W: www.velos.io
> > >
> >
>
>
>
> --
>
> SUREN HIRAMAN, VP TECHNOLOGY
> Velos
> Accelerating Machine Learning
>
> 440 NINTH AVENUE, 11TH FLOOR
> NEW YORK, NY 10001
> O: (917) 525-2466 ext. 105
> F: 646.349.4063
> E: suren.hiraman@v elos.io
> W: www.velos.io
>

Re: Clearspring Analytics Version

[Surendranauth Hiraman]

Reynold,

My environment is not quite set up to build Spark core properly and I'm
having various dependency issues running sbt/sbt assembly.

Any change you could go ahead and submit a pull request for this if it's
easy for you? :-)

-Suren



On Tue, May 27, 2014 at 6:13 PM, Reynold Xin  wrote:

> 2.7 sounds good. I was actually waiting for 2.7 to come out to post a JIRA
> (mainly for the serializable HyperLogLogPlus class).
>
>
> On Tue, May 27, 2014 at 3:11 PM, Surendranauth Hiraman <
> suren.hira...@velos.io> wrote:
>
> > Great, I will submit the pull request.
> >
> > Any objection to version 2.7?
> >
> > -Suren
> >
> >
> >
> > On Tue, May 27, 2014 at 6:02 PM, Reynold Xin 
> wrote:
> >
> > > Would you like to submit a pull request to update it?
> > >
> > > Also in the latest version HyperLogLog is serializable. That means we
> can
> > > get rid of the SerializableHyperLogLog class. (and move to use
> > > HyperLogLogPlus).
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Tue, May 27, 2014 at 3:01 PM, Surendranauth Hiraman <
> > > suren.hira...@velos.io> wrote:
> > >
> > > > Hi,
> > > >
> > > > It looks like the version of Clearspring's stream analytics class in
> > 1.0
> > > > branch and master is 2.5
> > > >
> > > > There were some significant bug fixes in 2.6 and version 2.7 is just
> > out
> > > > now as well.
> > > >
> > > > Are there any plans to upgrade?
> > > >
> > > > The QDigest deserialization code in 2.5 seems to have bugs that are
> > fixed
> > > > in 2.6 and higher.
> > > >
> > > >
> > > >   
> > > > com.clearspring.analytics
> > > > stream
> > > > 2.5.1-
> > > >   
> > > >
> > > >
> > > >
> > > > SUREN HIRAMAN, VP TECHNOLOGY
> > > > Velos
> > > > Accelerating Machine Learning
> > > >
> > > > 440 NINTH AVENUE, 11TH FLOOR
> > > > NEW YORK, NY 10001
> > > > O: (917) 525-2466 ext. 105
> > > > F: 646.349.4063
> > > > E: suren.hiraman@v elos.io
> > > > W: www.velos.io
> > > >
> > >
> >
> >
> >
> > --
> >
> > SUREN HIRAMAN, VP TECHNOLOGY
> > Velos
> > Accelerating Machine Learning
> >
> > 440 NINTH AVENUE, 11TH FLOOR
> > NEW YORK, NY 10001
> > O: (917) 525-2466 ext. 105
> > F: 646.349.4063
> > E: suren.hiraman@v elos.io
> > W: www.velos.io
> >
>



-- 

SUREN HIRAMAN, VP TECHNOLOGY
Velos
Accelerating Machine Learning

440 NINTH AVENUE, 11TH FLOOR
NEW YORK, NY 10001
O: (917) 525-2466 ext. 105
F: 646.349.4063
E: suren.hiraman@v elos.io
W: www.velos.io

Re: [VOTE] Release Apache Spark 1.0.0 (RC11)

[Mark Hamstra]

+1


On Tue, May 27, 2014 at 9:26 AM, Ankur Dave  wrote:

> 0
>
> OK, I withdraw my downvote.
>
> Ankur 
>

Re: [VOTE] Release Apache Spark 1.0.0 (RC11)

[Holden Karau]

+1 (I did some very basic testing with PySpark & Pandas on rc11)


On Tue, May 27, 2014 at 3:53 PM, Mark Hamstra wrote:

> +1
>
>
> On Tue, May 27, 2014 at 9:26 AM, Ankur Dave  wrote:
>
> > 0
> >
> > OK, I withdraw my downvote.
> >
> > Ankur 
> >
>



-- 
Cell : 425-233-8271

Re: Clearspring Analytics Version

[Reynold Xin]

Posted a question on streamlib user group about API compatibility:

https://groups.google.com/forum/#!topic/stream-lib-user/4VDeKcPiTJU


On Tue, May 27, 2014 at 3:36 PM, Surendranauth Hiraman <
suren.hira...@velos.io> wrote:

> Reynold,
>
> My environment is not quite set up to build Spark core properly and I'm
> having various dependency issues running sbt/sbt assembly.
>
> Any change you could go ahead and submit a pull request for this if it's
> easy for you? :-)
>
> -Suren
>
>
>
> On Tue, May 27, 2014 at 6:13 PM, Reynold Xin  wrote:
>
> > 2.7 sounds good. I was actually waiting for 2.7 to come out to post a
> JIRA
> > (mainly for the serializable HyperLogLogPlus class).
> >
> >
> > On Tue, May 27, 2014 at 3:11 PM, Surendranauth Hiraman <
> > suren.hira...@velos.io> wrote:
> >
> > > Great, I will submit the pull request.
> > >
> > > Any objection to version 2.7?
> > >
> > > -Suren
> > >
> > >
> > >
> > > On Tue, May 27, 2014 at 6:02 PM, Reynold Xin 
> > wrote:
> > >
> > > > Would you like to submit a pull request to update it?
> > > >
> > > > Also in the latest version HyperLogLog is serializable. That means we
> > can
> > > > get rid of the SerializableHyperLogLog class. (and move to use
> > > > HyperLogLogPlus).
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, May 27, 2014 at 3:01 PM, Surendranauth Hiraman <
> > > > suren.hira...@velos.io> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > It looks like the version of Clearspring's stream analytics class
> in
> > > 1.0
> > > > > branch and master is 2.5
> > > > >
> > > > > There were some significant bug fixes in 2.6 and version 2.7 is
> just
> > > out
> > > > > now as well.
> > > > >
> > > > > Are there any plans to upgrade?
> > > > >
> > > > > The QDigest deserialization code in 2.5 seems to have bugs that are
> > > fixed
> > > > > in 2.6 and higher.
> > > > >
> > > > >
> > > > >   
> > > > > com.clearspring.analytics
> > > > > stream
> > > > > 2.5.1-
> > > > >   
> > > > >
> > > > >
> > > > >
> > > > > SUREN HIRAMAN, VP TECHNOLOGY
> > > > > Velos
> > > > > Accelerating Machine Learning
> > > > >
> > > > > 440 NINTH AVENUE, 11TH FLOOR
> > > > > NEW YORK, NY 10001
> > > > > O: (917) 525-2466 ext. 105
> > > > > F: 646.349.4063
> > > > > E: suren.hiraman@v elos.io
> > > > > W: www.velos.io
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > SUREN HIRAMAN, VP TECHNOLOGY
> > > Velos
> > > Accelerating Machine Learning
> > >
> > > 440 NINTH AVENUE, 11TH FLOOR
> > > NEW YORK, NY 10001
> > > O: (917) 525-2466 ext. 105
> > > F: 646.349.4063
> > > E: suren.hiraman@v elos.io
> > > W: www.velos.io
> > >
> >
>
>
>
> --
>
> SUREN HIRAMAN, VP TECHNOLOGY
> Velos
> Accelerating Machine Learning
>
> 440 NINTH AVENUE, 11TH FLOOR
> NEW YORK, NY 10001
> O: (917) 525-2466 ext. 105
> F: 646.349.4063
> E: suren.hiraman@v elos.io
> W: www.velos.io
>

About JIRA SPARK-1825

[innowireless TaeYun Kim]

Could somebody please review and fix
https://issues.apache.org/jira/browse/SPARK-1825 ?

It's a cross-platform issue.

I've fixed the Spark source code based on rc5 and it's working for me now.
but totally not sure whether I've done correctly, since I'm almost new to
Spark and don't know much about the source code or API of both Spark and
YARN.

 

It would be nice that the issue be resolved before the 1.0.0 release.

Without the fix, developing a Spark application on Windows will be
inconvenient.

 

I have the source code I've fixed, but I don't' know how to upload it to be
shown by other, more Spark-capable developers. (And I'm not sure that my fix
is correct)

 

Thanks in advance. 

 

Re: About JIRA SPARK-1825

[Matei Zaharia]

Hei Taeyun, have you sent a pull request for this fix? We can review it there.

It’s too late to merge anything but blockers for 1.0.0 but we can do it for 
1.0.1 or 1.1, depending how big the patch is.

Matei

On May 27, 2014, at 5:25 PM, innowireless TaeYun Kim 
 wrote:

> Could somebody please review and fix
> https://issues.apache.org/jira/browse/SPARK-1825 ?
> 
> It's a cross-platform issue.
> 
> I've fixed the Spark source code based on rc5 and it's working for me now.
> but totally not sure whether I've done correctly, since I'm almost new to
> Spark and don't know much about the source code or API of both Spark and
> YARN.
> 
> 
> 
> It would be nice that the issue be resolved before the 1.0.0 release.
> 
> Without the fix, developing a Spark application on Windows will be
> inconvenient.
> 
> 
> 
> I have the source code I've fixed, but I don't' know how to upload it to be
> shown by other, more Spark-capable developers. (And I'm not sure that my fix
> is correct)
> 
> 
> 
> Thanks in advance. 
> 
> 
> 

RE: About JIRA SPARK-1825

[innowireless TaeYun Kim]

I'm afraid I don't know how to send a 'pull request'. (Sorry for my
ignorance)
I've modified the source code on my PC.
I can learn how to send a pull request, or I can attach the modified source
code on the mail right now(only 2 scala files).
Which way is preferable?


-Original Message-
From: Matei Zaharia [mailto:matei.zaha...@gmail.com] 
Sent: Wednesday, May 28, 2014 9:53 AM
To: dev@spark.apache.org
Subject: Re: About JIRA SPARK-1825

Hei Taeyun, have you sent a pull request for this fix? We can review it
there.

It's too late to merge anything but blockers for 1.0.0 but we can do it for
1.0.1 or 1.1, depending how big the patch is.

Matei

On May 27, 2014, at 5:25 PM, innowireless TaeYun Kim
 wrote:

> Could somebody please review and fix
> https://issues.apache.org/jira/browse/SPARK-1825 ?
> 
> It's a cross-platform issue.
> 
> I've fixed the Spark source code based on rc5 and it's working for me now.
> but totally not sure whether I've done correctly, since I'm almost new 
> to Spark and don't know much about the source code or API of both 
> Spark and YARN.
> 
> 
> 
> It would be nice that the issue be resolved before the 1.0.0 release.
> 
> Without the fix, developing a Spark application on Windows will be 
> inconvenient.
> 
> 
> 
> I have the source code I've fixed, but I don't' know how to upload it 
> to be shown by other, more Spark-capable developers. (And I'm not sure 
> that my fix is correct)
> 
> 
> 
> Thanks in advance. 
> 
> 
> 

Re: About JIRA SPARK-1825

[Reynold Xin]

It is actually pretty simple. You will first need to fork Spark on github,
and then push your changes to it, and then follow:
https://help.github.com/articles/using-pull-requests



On Tue, May 27, 2014 at 6:10 PM, innowireless TaeYun Kim <
taeyun@innowireless.co.kr> wrote:

> I'm afraid I don't know how to send a 'pull request'. (Sorry for my
> ignorance)
> I've modified the source code on my PC.
> I can learn how to send a pull request, or I can attach the modified source
> code on the mail right now(only 2 scala files).
> Which way is preferable?
>
>
> -Original Message-
> From: Matei Zaharia [mailto:matei.zaha...@gmail.com]
> Sent: Wednesday, May 28, 2014 9:53 AM
> To: dev@spark.apache.org
> Subject: Re: About JIRA SPARK-1825
>
> Hei Taeyun, have you sent a pull request for this fix? We can review it
> there.
>
> It's too late to merge anything but blockers for 1.0.0 but we can do it for
> 1.0.1 or 1.1, depending how big the patch is.
>
> Matei
>
> On May 27, 2014, at 5:25 PM, innowireless TaeYun Kim
>  wrote:
>
> > Could somebody please review and fix
> > https://issues.apache.org/jira/browse/SPARK-1825 ?
> >
> > It's a cross-platform issue.
> >
> > I've fixed the Spark source code based on rc5 and it's working for me
> now.
> > but totally not sure whether I've done correctly, since I'm almost new
> > to Spark and don't know much about the source code or API of both
> > Spark and YARN.
> >
> >
> >
> > It would be nice that the issue be resolved before the 1.0.0 release.
> >
> > Without the fix, developing a Spark application on Windows will be
> > inconvenient.
> >
> >
> >
> > I have the source code I've fixed, but I don't' know how to upload it
> > to be shown by other, more Spark-capable developers. (And I'm not sure
> > that my fix is correct)
> >
> >
> >
> > Thanks in advance.
> >
> >
> >
>
>

RE: About JIRA SPARK-1825

[innowireless TaeYun Kim]

Sent a pull request. (https://github.com/apache/spark/pull/899)
Thanks.

-Original Message-
From: Reynold Xin [mailto:r...@databricks.com] 
Sent: Wednesday, May 28, 2014 10:12 AM
To: dev@spark.apache.org
Subject: Re: About JIRA SPARK-1825

It is actually pretty simple. You will first need to fork Spark on github, and 
then push your changes to it, and then follow:
https://help.github.com/articles/using-pull-requests



On Tue, May 27, 2014 at 6:10 PM, innowireless TaeYun Kim < 
taeyun@innowireless.co.kr> wrote:

> I'm afraid I don't know how to send a 'pull request'. (Sorry for my
> ignorance)
> I've modified the source code on my PC.
> I can learn how to send a pull request, or I can attach the modified 
> source code on the mail right now(only 2 scala files).
> Which way is preferable?
>
>
> -Original Message-
> From: Matei Zaharia [mailto:matei.zaha...@gmail.com]
> Sent: Wednesday, May 28, 2014 9:53 AM
> To: dev@spark.apache.org
> Subject: Re: About JIRA SPARK-1825
>
> Hei Taeyun, have you sent a pull request for this fix? We can review 
> it there.
>
> It's too late to merge anything but blockers for 1.0.0 but we can do 
> it for
> 1.0.1 or 1.1, depending how big the patch is.
>
> Matei
>
> On May 27, 2014, at 5:25 PM, innowireless TaeYun Kim 
>  wrote:
>
> > Could somebody please review and fix
> > https://issues.apache.org/jira/browse/SPARK-1825 ?
> >
> > It's a cross-platform issue.
> >
> > I've fixed the Spark source code based on rc5 and it's working for 
> > me
> now.
> > but totally not sure whether I've done correctly, since I'm almost 
> > new to Spark and don't know much about the source code or API of 
> > both Spark and YARN.
> >
> >
> >
> > It would be nice that the issue be resolved before the 1.0.0 release.
> >
> > Without the fix, developing a Spark application on Windows will be 
> > inconvenient.
> >
> >
> >
> > I have the source code I've fixed, but I don't' know how to upload 
> > it to be shown by other, more Spark-capable developers. (And I'm not 
> > sure that my fix is correct)
> >
> >
> >
> > Thanks in advance.
> >
> >
> >
>
>

FYI -- javax.servlet dependency issue workaround

[Kay Ousterhout]

Hi all,

I had some trouble compiling an application (Shark) against Spark 1.0,
where Shark had a runtime exception (at the bottom of this message) because
it couldn't find the javax.servlet classes.  SBT seemed to have trouble
downloading the servlet APIs that are dependencies of Jetty (used by the
Spark web UI), so I had to manually add them to the application's build
file:

libraryDependencies += "org.mortbay.jetty" % "servlet-api" % "3.0.20100224"

Not exactly sure why this happens but thought it might be useful in case
others run into the same problem.

-Kay

-

Exception in thread "main" java.lang.NoClassDefFoundError:
javax/servlet/FilterRegistration

at
org.eclipse.jetty.servlet.ServletContextHandler.(ServletContextHandler.java:136)

at
org.eclipse.jetty.servlet.ServletContextHandler.(ServletContextHandler.java:129)

at
org.eclipse.jetty.servlet.ServletContextHandler.(ServletContextHandler.java:98)

at org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:98)

at org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:89)

at org.apache.spark.ui.WebUI.attachPage(WebUI.scala:64)

at org.apache.spark.ui.WebUI$anonfun$attachTab$1.apply(WebUI.scala:57)

at org.apache.spark.ui.WebUI$anonfun$attachTab$1.apply(WebUI.scala:57)

at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)

at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)

at org.apache.spark.ui.WebUI.attachTab(WebUI.scala:57)

at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:66)

at org.apache.spark.ui.SparkUI.(SparkUI.scala:60)

at org.apache.spark.ui.SparkUI.(SparkUI.scala:42)

at org.apache.spark.SparkContext.(SparkContext.scala:222)

at org.apache.spark.SparkContext.(SparkContext.scala:85)

at shark.SharkContext.(SharkContext.scala:42)

at shark.SharkContext.(SharkContext.scala:61)

at shark.SharkEnv$.initWithSharkContext(SharkEnv.scala:78)

at shark.SharkEnv$.init(SharkEnv.scala:38)

at shark.SharkCliDriver.(SharkCliDriver.scala:280)

at shark.SharkCliDriver$.main(SharkCliDriver.scala:162)

at shark.SharkCliDriver.main(SharkCliDriver.scala)

Caused by: java.lang.ClassNotFoundException:
javax.servlet.FilterRegistration

at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

... 23 more

Re: FYI -- javax.servlet dependency issue workaround

[Prashant Sharma]

Also just for sake of completeness, sometimes the desired dependency might
just be an older version in that case even if you include it like above it
may get evicted (Sbt's default strategy for conflict manager is to choose
the latest version).

So to further ensure that it does include it. We can
libraryDependencies += "org.mortbay.jetty" % "servlet-api" % "3.0.20100224"
*force()*

force it.

Thanks

Prashant Sharma


On Wed, May 28, 2014 at 10:38 AM, Kay Ousterhout wrote:

> Hi all,
>
> I had some trouble compiling an application (Shark) against Spark 1.0,
> where Shark had a runtime exception (at the bottom of this message) because
> it couldn't find the javax.servlet classes.  SBT seemed to have trouble
> downloading the servlet APIs that are dependencies of Jetty (used by the
> Spark web UI), so I had to manually add them to the application's build
> file:
>
> libraryDependencies += "org.mortbay.jetty" % "servlet-api" % "3.0.20100224"
>
> Not exactly sure why this happens but thought it might be useful in case
> others run into the same problem.
>
> -Kay
>
> -
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> javax/servlet/FilterRegistration
>
> at
>
> org.eclipse.jetty.servlet.ServletContextHandler.(ServletContextHandler.java:136)
>
> at
>
> org.eclipse.jetty.servlet.ServletContextHandler.(ServletContextHandler.java:129)
>
> at
>
> org.eclipse.jetty.servlet.ServletContextHandler.(ServletContextHandler.java:98)
>
> at
> org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:98)
>
> at
> org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:89)
>
> at org.apache.spark.ui.WebUI.attachPage(WebUI.scala:64)
>
> at org.apache.spark.ui.WebUI$anonfun$attachTab$1.apply(WebUI.scala:57)
>
> at org.apache.spark.ui.WebUI$anonfun$attachTab$1.apply(WebUI.scala:57)
>
> at
>
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>
> at org.apache.spark.ui.WebUI.attachTab(WebUI.scala:57)
>
> at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:66)
>
> at org.apache.spark.ui.SparkUI.(SparkUI.scala:60)
>
> at org.apache.spark.ui.SparkUI.(SparkUI.scala:42)
>
> at org.apache.spark.SparkContext.(SparkContext.scala:222)
>
> at org.apache.spark.SparkContext.(SparkContext.scala:85)
>
> at shark.SharkContext.(SharkContext.scala:42)
>
> at shark.SharkContext.(SharkContext.scala:61)
>
> at shark.SharkEnv$.initWithSharkContext(SharkEnv.scala:78)
>
> at shark.SharkEnv$.init(SharkEnv.scala:38)
>
> at shark.SharkCliDriver.(SharkCliDriver.scala:280)
>
> at shark.SharkCliDriver$.main(SharkCliDriver.scala:162)
>
> at shark.SharkCliDriver.main(SharkCliDriver.scala)
>
> Caused by: java.lang.ClassNotFoundException:
> javax.servlet.FilterRegistration
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>
> ... 23 more
>