Re: Unsubscribe

[Charles Allen]

Can we get a bot that auto-subscribes folks to cat-facts! when they email
the user list instead of user-unsubscr...@spark.apache.org ?

On Mon, Nov 28, 2016 at 1:28 PM R. Revert  wrote:

> Unsubscribe
>
> El 28 nov. 2016 5:22 p. m.,  escribió:
>
> Unsubscribe
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>

Re: Spark 2.0.0-preview ... problem with jackson core version

[Charles Allen]

I'm having the same difficulty porting
https://github.com/metamx/druid-spark-batch/tree/spark2 over to spark2.x,
where I have to go track down who is pulling in bad jackson versions.



On Fri, Jul 1, 2016 at 11:59 AM Sean Owen  wrote:

> Are you just asking why you can't use 2.5.3 in your app? because
> Jackson isn't shaded, which is sort of the bad news. But just use
> 2.6.5 too, ideally. I don't know where 2.6.1 is coming from, but Spark
> doesn't use it.
>
> On Fri, Jul 1, 2016 at 5:48 PM,   wrote:
> > In my project I found the library which brings Jackson core 2.6.5 and it
> is
> > used in conjunction with the requested Jackson scala module 2.5.3 wanted
> by
> > spark 2.0.0 preview. At runtime it's the cause of exception.
> >
> > Now I have excluded 2.6.5 using sbt but it could be dangerous for the
> other
> > library.
> >
> > Why this regression to Jackson 2.5.3 switching from 2.0.0 snapshot to
> > preview ?
> >
> > Thanks
> > Paolo
> >
> > Get Outlook for Android
> >
> >
> >
> >
> > On Fri, Jul 1, 2016 at 4:24 PM +0200, "Paolo Patierno" <
> ppatie...@live.com>
> > wrote:
> >
> > Hi,
> >
> > developing a custom receiver up today I used spark version
> "2.0.0-SNAPSHOT"
> > and scala version 2.11.7.
> > With these version all tests work fine.
> >
> > I have just switching to "2.0.0-preview" as spark version but not I have
> > following error :
> >
> > An exception or error caused a run to abort: class
> > com.fasterxml.jackson.module.scala.ser.ScalaIteratorSerializer overrides
> > final method
> >
> withResolved.(Lcom/fasterxml/jackson/databind/BeanProperty;Lcom/fasterxml/jackson/databind/jsontype/TypeSerializer;Lcom/fasterxml/jackson/databind/JsonSerializer;)Lcom/fasterxml/jackson/databind/ser/std/AsArraySerializerBase;
> > java.lang.VerifyError: class
> > com.fasterxml.jackson.module.scala.ser.ScalaIteratorSerializer overrides
> > final method
> >
> withResolved.(Lcom/fasterxml/jackson/databind/BeanProperty;Lcom/fasterxml/jackson/databind/jsontype/TypeSerializer;Lcom/fasterxml/jackson/databind/JsonSerializer;)Lcom/fasterxml/jackson/databind/ser/std/AsArraySerializerBase;
> > at java.lang.ClassLoader.defineClass1(Native Method)
> >
> > I see that using 2.0.0-SNAPSHOT the jackson core version is 2.6.5 ... now
> > 2.6.1 with module scala 2.5.3.
> >
> > Thanks,
> > Paolo.
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno
> > Linkedin : paolopatierno
> > Blog : DevExperience
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>

Re: Spark 1.5 on Mesos

[Charles Allen]

Re: Spark on Mesos Warning regarding disk space:
https://issues.apache.org/jira/browse/SPARK-12330

That's a spark flaw I encountered on a very regular basis on mesos. That
and a few other annoyances are fixed in
https://github.com/metamx/spark/tree/v1.5.2-mmx

Here's another mild annoyance I've encountered:
https://issues.apache.org/jira/browse/SPARK-11714

On Wed, Mar 2, 2016 at 1:31 PM Ashish Soni  wrote:

> I have no luck and i would to ask the question to spark committers will
> this be ever designed to run on mesos ?
>
> spark app as a docker container not working at all on mesos  ,if any one
> would like the code i can send it over to have a look.
>
> Ashish
>
> On Wed, Mar 2, 2016 at 12:23 PM, Sathish Kumaran Vairavelu <
> vsathishkuma...@gmail.com> wrote:
>
>> Try passing jar using --jars option
>>
>> On Wed, Mar 2, 2016 at 10:17 AM Ashish Soni 
>> wrote:
>>
>>> I made some progress but now i am stuck at this point , Please help as
>>> looks like i am close to get it working
>>>
>>> I have everything running in docker container including mesos slave and
>>> master
>>>
>>> When i try to submit the pi example i get below error
>>> *Error: Cannot load main class from JAR file:/opt/spark/Example*
>>>
>>> Below is the command i use to submit as a docker container
>>>
>>> docker run -it --rm -e SPARK_MASTER="mesos://10.0.2.15:7077"  -e
>>> SPARK_IMAGE="spark_driver:latest" spark_driver:latest ./bin/spark-submit
>>> --deploy-mode cluster --name "PI Example" --class
>>> org.apache.spark.examples.SparkPi --driver-memory 512m --executor-memory
>>> 512m --executor-cores 1
>>> http://10.0.2.15/spark-examples-1.6.0-hadoop2.6.0.jar
>>>
>>>
>>> On Tue, Mar 1, 2016 at 2:59 PM, Timothy Chen  wrote:
>>>
 Can you go through the Mesos UI and look at the driver/executor log
 from steer file and see what the problem is?

 Tim

 On Mar 1, 2016, at 8:05 AM, Ashish Soni  wrote:

 Not sure what is the issue but i am getting below error  when i try to
 run spark PI example

 Blacklisting Mesos slave value: "5345asdasdasdkas234234asdasdasdasd"
due to too many failures; is Spark installed on it?
 WARN TaskSchedulerImpl: Initial job has not accepted any resources; 
 check your cluster UI to ensure that workers are registered and have 
 sufficient resources


 On Mon, Feb 29, 2016 at 1:39 PM, Sathish Kumaran Vairavelu <
 vsathishkuma...@gmail.com> wrote:

> May be the Mesos executor couldn't find spark image or the constraints
> are not satisfied. Check your Mesos UI if you see Spark application in the
> Frameworks tab
>
> On Mon, Feb 29, 2016 at 12:23 PM Ashish Soni 
> wrote:
>
>> What is the Best practice , I have everything running as docker
>> container in single host ( mesos and marathon also as docker container )
>>  and everything comes up fine but when i try to launch the spark shell i
>> get below error
>>
>>
>> SQL context available as sqlContext.
>>
>> scala> val data = sc.parallelize(1 to 100)
>> data: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at
>> parallelize at :27
>>
>> scala> data.count
>> [Stage 0:>
>>  (0 + 0) / 2]16/02/29 18:21:12 WARN TaskSchedulerImpl: Initial job has 
>> not
>> accepted any resources; check your cluster UI to ensure that workers are
>> registered and have sufficient resources
>> 16/02/29 18:21:27 WARN TaskSchedulerImpl: Initial job has not
>> accepted any resources; check your cluster UI to ensure that workers are
>> registered and have sufficient resources
>>
>>
>>
>> On Mon, Feb 29, 2016 at 12:04 PM, Tim Chen  wrote:
>>
>>> No you don't have to run Mesos in docker containers to run Spark in
>>> docker containers.
>>>
>>> Once you have Mesos cluster running you can then specfiy the Spark
>>> configurations in your Spark job (i.e: 
>>> spark.mesos.executor.docker.image=mesosphere/spark:1.6)
>>> and Mesos will automatically launch docker containers for you.
>>>
>>> Tim
>>>
>>> On Mon, Feb 29, 2016 at 7:36 AM, Ashish Soni 
>>> wrote:
>>>
 Yes i read that and not much details here.

 Is it true that we need to have spark installed on each mesos
 docker container ( master and slave ) ...

 Ashish

 On Fri, Feb 26, 2016 at 2:14 PM, Tim Chen 
 wrote:

> https://spark.apache.org/docs/latest/running-on-mesos.html should
> be the best source, what problems were you running into?
>
> Tim
>
> On Fri, Feb 26, 2016 at 11:06 AM, Yin Yang 
> wrote:
>
>> Have you read this ?

Re: Mesos scheduler obeying limit of tasks / executor

[Charles Allen]

That answers it thanks!

On Fri, Dec 11, 2015 at 6:37 AM Iulian Dragoș <iulian.dra...@typesafe.com>
wrote:

> Hi Charles,
>
> I am not sure I totally understand your issues, but the spark.task.cpus
> limit is imposed at a higher level, for all cluster managers. The code is
> in TaskSchedulerImpl
> <https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala#L250>
> .
>
> There is a pending PR to implement spark.executor.cores (and launching
> multiple executors on a single worker), but it wasn’t yet merged:
> https://github.com/apache/spark/pull/4027
>
> iulian
> ​
>
> On Wed, Dec 9, 2015 at 7:23 PM, Charles Allen <
> charles.al...@metamarkets.com> wrote:
>
>> I have a spark app in development which has relatively strict cpu/mem
>> ratios that are required. As such, I cannot arbitrarily add CPUs to a
>> limited memory size.
>>
>> The general spark cluster behaves as expected, where tasks are launched
>> with a specified memory/cpu ratio, but the mesos scheduler seems to ignore
>> this.
>>
>> Specifically, I cannot find where in the code the limit of number of
>> tasks per executor of "spark.executor.cores" / "spark.task.cpus" is
>> enforced in the MesosBackendScheduler.
>>
>> The Spark App in question has some JVM heap heavy activities inside a
>> RDD.mapPartitionsWithIndex, so having more tasks per limited JVM memory
>> resource is bad. The workaround planned handling of this is to limit the
>> number of tasks per JVM, which does not seem possible in mesos mode, where
>> it seems to just keep stacking on CPUs as tasks come in without adjusting
>> any memory constraints, or looking for limits of tasks per executor.
>>
>> How can I limit the tasks per executor (or per memory pool) in the Mesos
>> backend scheduler?
>>
>> Thanks,
>> Charles Allen
>>
>
>
>
> --
>
> --
> Iulian Dragos
>
> --
> Reactive Apps on the JVM
> www.typesafe.com
>
>

Mesos scheduler obeying limit of tasks / executor

[Charles Allen]

I have a spark app in development which has relatively strict cpu/mem
ratios that are required. As such, I cannot arbitrarily add CPUs to a
limited memory size.

The general spark cluster behaves as expected, where tasks are launched
with a specified memory/cpu ratio, but the mesos scheduler seems to ignore
this.

Specifically, I cannot find where in the code the limit of number of tasks
per executor of "spark.executor.cores" / "spark.task.cpus" is enforced in
the MesosBackendScheduler.

The Spark App in question has some JVM heap heavy activities inside a
RDD.mapPartitionsWithIndex, so having more tasks per limited JVM memory
resource is bad. The workaround planned handling of this is to limit the
number of tasks per JVM, which does not seem possible in mesos mode, where
it seems to just keep stacking on CPUs as tasks come in without adjusting
any memory constraints, or looking for limits of tasks per executor.

How can I limit the tasks per executor (or per memory pool) in the Mesos
backend scheduler?

Thanks,
Charles Allen

Re: ClassLoader resources on executor

[Charles Allen]

I still have to propagate the file into the directory somehow, and also
that's marked as only for legacy jobs (deprecated?), so no, I have not
experimented with it yet.

On Wed, Dec 2, 2015 at 12:53 AM Rishi Mishra <rmis...@snappydata.io> wrote:

> Did you try to use *spark.executor.extraClassPath*. The classpath
> resources will be accessible through the executors class loader which
> executes your job.
>
> On Wed, Dec 2, 2015 at 2:15 AM, Charles Allen <
> charles.al...@metamarkets.com> wrote:
>
>> Is there a way to pass configuration file resources to be resolvable
>> through the classloader?
>>
>> For example, if I'm using a library (non-spark) that can use a
>> some-lib.properties file in the classpath/classLoader, can I pass that file
>> so that when it tries to get the resource from the classloader it is able
>> to find it?
>>
>> One potential solution is to take the files and package them as resources
>> in a jar, and include the jar as part of the spark job, but that feels like
>> a hack instead of an actual solution.
>>
>> Is there any support or planned support for such a thing?
>>
>> https://github.com/apache/spark/pull/9118 seems to tackle a similar
>> problem in a hard-coded kind of way.
>>
>> Thank you,
>> Charles Allen
>>
>
>
>
> --
> Regards,
> Rishitesh Mishra,
> SnappyData . (http://www.snappydata.io/)
>
> https://in.linkedin.com/in/rishiteshmishra
>