Re: Use cases for kafka direct stream messageHandler
I'd probably prefer to keep it the way it is, unless it's becoming more
like the function without the messageHandler argument.
Right now I have code like this, but I wish it were more similar looking:
if (parsed.partitions.isEmpty()) {
JavaPairInputDStream kvstream = KafkaUtils
.createDirectStream(jssc, String.class, MessageWrapper.class,
StringDecoder.class,
MessageDecoder.class, kafkaArgs(parsed), topicSet);
requests = kvstream.map((Function,
MessageWrapper>) Tuple2::_2);
} else {
requests = KafkaUtils.createDirectStream(jssc, String.class,
MessageWrapper.class, StringDecoder.class, MessageDecoder.class,
MessageWrapper.class,
kafkaArgs(parsed), parsed.partitions,
(Function,
MessageWrapper>) MessageAndMetadata::message);
}
Of course, this is in the Java API so it may not have relevance to what
you're talking about.
Perhaps if both functions (the one with partitions arg and the one without)
returned just ConsumerRecord, I would like that more.
- Alan
On Tue, Mar 8, 2016 at 6:49 AM, Cody Koeninger wrote:
> No, looks like you'd have to catch them in the serializer and have the
> serializer return option or something. The new consumer builds a buffer
> full of records, not one at a time.
> On Mar 8, 2016 4:43 AM, "Marius Soutier" wrote:
>
>>
>> > On 04.03.2016, at 22:39, Cody Koeninger wrote:
>> >
>> > The only other valid use of messageHandler that I can think of is
>> > catching serialization problems on a per-message basis. But with the
>> > new Kafka consumer library, that doesn't seem feasible anyway, and
>> > could be handled with a custom (de)serializer.
>>
>> What do you mean, that doesn't seem feasible? You mean when using a
>> custom deserializer? Right now I'm catching serialization problems in the
>> message handler, after your proposed change I'd catch them in `map()`.
>>
>>
Re: Spark Streaming Specify Kafka Partition
One quick newbie question since I got another chance to look at this today. We're using java for our spark applications. The createDirectStream we were using previously [1] returns a JavaPairInputDStream, but the createDirectStream with fromOffsets expects an argument recordClass to pass into the generic constructor for createDirectStream. In the code for the first function signature (without fromOffsets) it's being constructed in Scala as just a tuple (K, V). How do I pass this same class/type information from java as the record class to get a JavaPairInputDStream<K, V>? I understand this might be a question more fit for a scala mailing list but google is failing me at the moment for hints on the interoperability of scala and java generics. [1] The original createDirectStream: https://github.com/apache/spark/blob/branch-1.5/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala#L395-L423 Thanks, - Alan On Tue, Dec 1, 2015 at 8:12 AM, Cody Koeninger <c...@koeninger.org> wrote: > I actually haven't tried that, since I tend to do the offset lookups if > necessary. > > It's possible that it will work, try it and let me know. > > Be aware that if you're doing a count() or take() operation directly on > the rdd it'll definitely give you the wrong result if you're using -1 for > one of the offsets. > > > > On Tue, Dec 1, 2015 at 9:58 AM, Alan Braithwaite <a...@cloudflare.com> > wrote: > >> Neat, thanks. If I specify something like -1 as the offset, will it >> consume from the latest offset or do I have to instrument that manually? >> >> - Alan >> >> On Tue, Dec 1, 2015 at 6:43 AM, Cody Koeninger <c...@koeninger.org> >> wrote: >> >>> Yes, there is a version of createDirectStream that lets you specify >>> fromOffsets: Map[TopicAndPartition, Long] >>> >>> On Mon, Nov 30, 2015 at 7:43 PM, Alan Braithwaite <a...@cloudflare.com> >>> wrote: >>> >>>> Is there any mechanism in the kafka streaming source to specify the >>>> exact partition id that we want a streaming job to consume from? >>>> >>>> If not, is there a workaround besides writing our a custom receiver? >>>> >>>> Thanks, >>>> - Alan >>>> >>> >>> >> >
Re: Spark Streaming Specify Kafka Partition
Neat, thanks. If I specify something like -1 as the offset, will it consume from the latest offset or do I have to instrument that manually? - Alan On Tue, Dec 1, 2015 at 6:43 AM, Cody Koeninger <c...@koeninger.org> wrote: > Yes, there is a version of createDirectStream that lets you specify > fromOffsets: Map[TopicAndPartition, Long] > > On Mon, Nov 30, 2015 at 7:43 PM, Alan Braithwaite <a...@cloudflare.com> > wrote: > >> Is there any mechanism in the kafka streaming source to specify the exact >> partition id that we want a streaming job to consume from? >> >> If not, is there a workaround besides writing our a custom receiver? >> >> Thanks, >> - Alan >> > >
Spark Streaming Specify Kafka Partition
Is there any mechanism in the kafka streaming source to specify the exact partition id that we want a streaming job to consume from? If not, is there a workaround besides writing our a custom receiver? Thanks, - Alan
Weird Spark Dispatcher Offers?
Hey All, Using spark with mesos and docker. I'm wondering if anybody's seen the behavior of spark dispatcher where it just continually requests resources and immediately declines the offer. https://gist.github.com/anonymous/41e7c91899b0122b91a7 I'm trying to debug some issues with spark and I'm having trouble figuring out if this is part of the problem or if it's safe to ignore it. Any help or pointers would be appreciated. Thanks! - Alan
Re: Weird Spark Dispatcher Offers?
> > So if there is no jobs to run the dispatcher will decline all offers by > default. > So would this be a bug in mesos then? I'm not sure I understand how this offer is appearing in the first place. It only shows up in the master logs when I start the dispatcher. > Also we list all the jobs enqueued and it's specifications in the Spark > dispatcher UI, you should see the port in the dispatcher logs itself. Yes, this job is not listed under that UI. Hence my confusion. Thanks, - Alan On Fri, Oct 2, 2015 at 11:49 AM, Tim Chen <t...@mesosphere.io> wrote: > So if there is no jobs to run the dispatcher will decline all offers by > default. > > Also we list all the jobs enqueued and it's specifications in the Spark > dispatcher UI, you should see the port in the dispatcher logs itself. > > Tim > > On Fri, Oct 2, 2015 at 11:46 AM, Alan Braithwaite <a...@cloudflare.com> > wrote: > >> This happened right after blowing away /var/lib/mesos zk://mesos and >> zk://spark_mesos_dispatcher and before I've submitted anything new to it so >> I _shouldn't_ have anything enqueued. Unless there's state being stored >> somewhere besides those places that I don't know about. >> >> I'm not sure what the resource specifications are for this one because I >> didn't submit it directly. If you have a way for me to grab a specific >> offer configuration, I'd be delighted to provide it. I just can't seem to >> figure out how to get that information after digging through the mesos docs >> :-( >> >> Also, I can't read the docker logs because: >> >> Oct 02 11:39:59 sparky docker[556]: >> time="2015-10-02T11:39:59.165474049-07:00" level=error msg="Error streaming >> logs: invalid character '\\x00' looking for beginning of value" >> >> (that's coming from the spark-dispatcher docker). >> >> Thanks! >> - Alan >> >> On Fri, Oct 2, 2015 at 11:36 AM, Tim Chen <t...@mesosphere.io> wrote: >> >>> Do you have jobs enqueued? And if none of the jobs matches any offer it >>> will just decline it. >>> >>> What's your job resource specifications? >>> >>> Tim >>> >>> On Fri, Oct 2, 2015 at 11:34 AM, Alan Braithwaite <a...@cloudflare.com> >>> wrote: >>> >>>> Hey All, >>>> >>>> Using spark with mesos and docker. >>>> >>>> I'm wondering if anybody's seen the behavior of spark dispatcher where >>>> it just continually requests resources and immediately declines the offer. >>>> >>>> https://gist.github.com/anonymous/41e7c91899b0122b91a7 >>>> >>>> I'm trying to debug some issues with spark and I'm having trouble >>>> figuring out if this is part of the problem or if it's safe to ignore it. >>>> >>>> Any help or pointers would be appreciated. >>>> >>>> Thanks! >>>> - Alan >>>> >>> >>> >> >
Re: Weird Spark Dispatcher Offers?
This happened right after blowing away /var/lib/mesos zk://mesos and zk://spark_mesos_dispatcher and before I've submitted anything new to it so I _shouldn't_ have anything enqueued. Unless there's state being stored somewhere besides those places that I don't know about. I'm not sure what the resource specifications are for this one because I didn't submit it directly. If you have a way for me to grab a specific offer configuration, I'd be delighted to provide it. I just can't seem to figure out how to get that information after digging through the mesos docs :-( Also, I can't read the docker logs because: Oct 02 11:39:59 sparky docker[556]: time="2015-10-02T11:39:59.165474049-07:00" level=error msg="Error streaming logs: invalid character '\\x00' looking for beginning of value" (that's coming from the spark-dispatcher docker). Thanks! - Alan On Fri, Oct 2, 2015 at 11:36 AM, Tim Chen <t...@mesosphere.io> wrote: > Do you have jobs enqueued? And if none of the jobs matches any offer it > will just decline it. > > What's your job resource specifications? > > Tim > > On Fri, Oct 2, 2015 at 11:34 AM, Alan Braithwaite <a...@cloudflare.com> > wrote: > >> Hey All, >> >> Using spark with mesos and docker. >> >> I'm wondering if anybody's seen the behavior of spark dispatcher where it >> just continually requests resources and immediately declines the offer. >> >> https://gist.github.com/anonymous/41e7c91899b0122b91a7 >> >> I'm trying to debug some issues with spark and I'm having trouble >> figuring out if this is part of the problem or if it's safe to ignore it. >> >> Any help or pointers would be appreciated. >> >> Thanks! >> - Alan >> > >
Re: Spark on Mesos with Jobs in Cluster Mode Documentation
That could be the behavior but spark.mesos.executor.home being unset still raises an exception inside the dispatcher preventing a docker from even being started. I can see if other properties are inherited from the default environment when that's set, if you'd like. I think the main problem is just that premature validation is being done on the dispatcher and the dispatcher crashing in the event of bad config. - Alan On Sat, Sep 19, 2015 at 11:03 AM, Timothy Chen <t...@mesosphere.io> wrote: > You can still provide properties through the docker container by putting > configuration in the conf directory, but we try to pass all properties > submitted from the driver spark-submit through which I believe will > override the defaults. > > This is not what you are seeing? > > Tim > > > On Sep 19, 2015, at 9:01 AM, Alan Braithwaite <a...@cloudflare.com> wrote: > > The assumption that the executor has no default properties set in it's > environment through the docker container. Correct me if I'm wrong, but any > properties which are unset in the SparkContext will come from the > environment of the executor will it not? > > Thanks, > - Alan > > On Sat, Sep 19, 2015 at 1:09 AM, Tim Chen <t...@mesosphere.io> wrote: > >> I guess I need a bit more clarification, what kind of assumptions was the >> dispatcher making? >> >> Tim >> >> >> On Thu, Sep 17, 2015 at 10:18 PM, Alan Braithwaite <a...@cloudflare.com> >> wrote: >> >>> Hi Tim, >>> >>> Thanks for the follow up. It's not so much that I expect the executor >>> to inherit the configuration of the dispatcher as I* don't *expect the >>> dispatcher to make assumptions about the system environment of the executor >>> (since it lives in a docker). I could potentially see a case where you >>> might want to explicitly forbid the defaults, but I can't think of any >>> right now. >>> >>> Otherwise, I'm confused as to why the defaults in the docker image for >>> the executor are just ignored. I suppose that it's the dispatchers job to >>> ensure the *exact* configuration of the executor, regardless of the >>> defaults set on the executors machine? Is that the assumption being made? >>> I can understand that in contexts which aren't docker driven since jobs >>> could be rolling out in the middle of a config update. Trying to think of >>> this outside the terms of just mesos/docker (since I'm fully aware that >>> docker doesn't rule the world yet). >>> >>> So I can see this from both perspectives now and passing in the >>> properties file will probably work just fine for me, but for my better >>> understanding: When the executor starts, will it read any of the >>> environment that it's executing in or will it just take only the properties >>> given to it by the dispatcher and nothing more? >>> >>> Lemme know if anything needs more clarification and thanks for your >>> mesos contribution to spark! >>> >>> - Alan >>> >>> On Thu, Sep 17, 2015 at 5:03 PM, Timothy Chen <t...@mesosphere.io> wrote: >>> >>>> Hi Alan, >>>> >>>> If I understand correctly, you are setting executor home when you >>>> launch the dispatcher and not on the configuration when you submit job, and >>>> expect it to inherit that configuration? >>>> >>>> When I worked on the dispatcher I was assuming all configuration is >>>> passed to the dispatcher to launch the job exactly how you will need to >>>> launch it with client mode. >>>> >>>> But indeed it shouldn't crash dispatcher, I'll take a closer look when >>>> I get a chance. >>>> >>>> Can you recommend changes on the documentation, either in email or a PR? >>>> >>>> Thanks! >>>> >>>> Tim >>>> >>>> Sent from my iPhone >>>> >>>> On Sep 17, 2015, at 12:29 PM, Alan Braithwaite <a...@cloudflare.com> >>>> wrote: >>>> >>>> Hey All, >>>> >>>> To bump this thread once again, I'm having some trouble using the >>>> dispatcher as well. >>>> >>>> I'm using Mesos Cluster Manager with Docker Executors. I've deployed >>>> the dispatcher as Marathon job. When I submit a job using spark submit, >>>> the dispatcher writes back that the submission was successful and then >>>> promptly dies in marathon. Looking at the logs reveals it was hitting the >&
Re: Spark on Mesos with Jobs in Cluster Mode Documentation
One other piece of information:
We're using zookeeper for persistence and when we brought the dispatcher
back online, it crashed on the same exception after loading the config from
zookeeper.
Cheers,
- Alan
On Thu, Sep 17, 2015 at 12:29 PM, Alan Braithwaite <a...@cloudflare.com>
wrote:
> Hey All,
>
> To bump this thread once again, I'm having some trouble using the
> dispatcher as well.
>
> I'm using Mesos Cluster Manager with Docker Executors. I've deployed the
> dispatcher as Marathon job. When I submit a job using spark submit, the
> dispatcher writes back that the submission was successful and then promptly
> dies in marathon. Looking at the logs reveals it was hitting the following
> line:
>
> 398: throw new SparkException("Executor Spark home
> `spark.mesos.executor.home` is not set!")
>
> Which is odd because it's set in multiple places (SPARK_HOME,
> spark.mesos.executor.home, spark.home, etc). Reading the code, it
> appears that the driver desc pulls only from the request and disregards any
> other properties that may be configured. Testing by passing --conf
> spark.mesos.executor.home=/usr/local/spark on the command line to
> spark-submit confirms this. We're trying to isolate the number of places
> where we have to set properties within spark and were hoping that it will
> be possible to have this pull in the spark-defaults.conf from somewhere, or
> at least allow the user to inform the dispatcher through spark-submit that
> those properties will be available once the job starts.
>
> Finally, I don't think the dispatcher should crash in this event. It
> seems not exceptional that a job is misconfigured when submitted.
>
> Please direct me on the right path if I'm headed in the wrong direction.
> Also let me know if I should open some tickets for these issues.
>
> Thanks,
> - Alan
>
> On Fri, Sep 11, 2015 at 1:05 PM, Tim Chen <t...@mesosphere.io> wrote:
>
>> Yes you can create an issue, or actually contribute a patch to update it
>> :)
>>
>> Sorry the docs is a bit light, I'm going to make it more complete along
>> the way.
>>
>> Tim
>>
>>
>> On Fri, Sep 11, 2015 at 11:11 AM, Tom Waterhouse (tomwater) <
>> tomwa...@cisco.com> wrote:
>>
>>> Tim,
>>>
>>> Thank you for the explanation. You are correct, my Mesos experience is
>>> very light, and I haven’t deployed anything via Marathon yet. What you
>>> have stated here makes sense, I will look into doing this.
>>>
>>> Adding this info to the docs would be great. Is the appropriate action
>>> to create an issue regarding improvement of the docs? For those of us who
>>> are gaining the experience having such a pointer is very helpful.
>>>
>>> Tom
>>>
>>> From: Tim Chen <t...@mesosphere.io>
>>> Date: Thursday, September 10, 2015 at 10:25 AM
>>> To: Tom Waterhouse <tomwa...@cisco.com>
>>> Cc: "user@spark.apache.org" <user@spark.apache.org>
>>> Subject: Re: Spark on Mesos with Jobs in Cluster Mode Documentation
>>>
>>> Hi Tom,
>>>
>>> Sorry the documentation isn't really rich, since it's probably assuming
>>> users understands how Mesos and framework works.
>>>
>>> First I need explain the rationale of why create the dispatcher. If
>>> you're not familiar with Mesos yet, each node in your datacenter is
>>> installed a Mesos slave where it's responsible for publishing resources and
>>> running/watching tasks, and Mesos master is responsible for taking the
>>> aggregated resources and scheduling them among frameworks.
>>>
>>> Frameworks are not managed by Mesos, as Mesos master/slave doesn't
>>> launch and maintain framework but assume they're launched and kept running
>>> on its own. All the existing frameworks in the ecosystem therefore all have
>>> their own ways to deploy, HA and persist state (e.g: Aurora, Marathon, etc).
>>>
>>> Therefore, to introduce cluster mode with Mesos, we must create a
>>> framework that is long running that can be running in your datacenter, and
>>> can handle launching spark drivers on demand and handle HA, etc. This is
>>> what the dispatcher is all about.
>>>
>>> So the idea is that you should launch the dispatcher not on the client,
>>> but on a machine in your datacenter. In Mesosphere's DCOS we launch all
>>> frameworks and long running services with Marathon, and you can use
>>> Marathon to launch the Spark dispatcher.
>>>
>>> Then all clients instead of speci
Re: Spark on Mesos with Jobs in Cluster Mode Documentation
Hey All,
To bump this thread once again, I'm having some trouble using the
dispatcher as well.
I'm using Mesos Cluster Manager with Docker Executors. I've deployed the
dispatcher as Marathon job. When I submit a job using spark submit, the
dispatcher writes back that the submission was successful and then promptly
dies in marathon. Looking at the logs reveals it was hitting the following
line:
398: throw new SparkException("Executor Spark home
`spark.mesos.executor.home` is not set!")
Which is odd because it's set in multiple places (SPARK_HOME,
spark.mesos.executor.home, spark.home, etc). Reading the code, it appears
that the driver desc pulls only from the request and disregards any other
properties that may be configured. Testing by passing --conf
spark.mesos.executor.home=/usr/local/spark on the command line to
spark-submit confirms this. We're trying to isolate the number of places
where we have to set properties within spark and were hoping that it will
be possible to have this pull in the spark-defaults.conf from somewhere, or
at least allow the user to inform the dispatcher through spark-submit that
those properties will be available once the job starts.
Finally, I don't think the dispatcher should crash in this event. It seems
not exceptional that a job is misconfigured when submitted.
Please direct me on the right path if I'm headed in the wrong direction.
Also let me know if I should open some tickets for these issues.
Thanks,
- Alan
On Fri, Sep 11, 2015 at 1:05 PM, Tim Chen wrote:
> Yes you can create an issue, or actually contribute a patch to update it :)
>
> Sorry the docs is a bit light, I'm going to make it more complete along
> the way.
>
> Tim
>
>
> On Fri, Sep 11, 2015 at 11:11 AM, Tom Waterhouse (tomwater) <
> tomwa...@cisco.com> wrote:
>
>> Tim,
>>
>> Thank you for the explanation. You are correct, my Mesos experience is
>> very light, and I haven’t deployed anything via Marathon yet. What you
>> have stated here makes sense, I will look into doing this.
>>
>> Adding this info to the docs would be great. Is the appropriate action
>> to create an issue regarding improvement of the docs? For those of us who
>> are gaining the experience having such a pointer is very helpful.
>>
>> Tom
>>
>> From: Tim Chen
>> Date: Thursday, September 10, 2015 at 10:25 AM
>> To: Tom Waterhouse
>> Cc: "user@spark.apache.org"
>> Subject: Re: Spark on Mesos with Jobs in Cluster Mode Documentation
>>
>> Hi Tom,
>>
>> Sorry the documentation isn't really rich, since it's probably assuming
>> users understands how Mesos and framework works.
>>
>> First I need explain the rationale of why create the dispatcher. If
>> you're not familiar with Mesos yet, each node in your datacenter is
>> installed a Mesos slave where it's responsible for publishing resources and
>> running/watching tasks, and Mesos master is responsible for taking the
>> aggregated resources and scheduling them among frameworks.
>>
>> Frameworks are not managed by Mesos, as Mesos master/slave doesn't launch
>> and maintain framework but assume they're launched and kept running on its
>> own. All the existing frameworks in the ecosystem therefore all have their
>> own ways to deploy, HA and persist state (e.g: Aurora, Marathon, etc).
>>
>> Therefore, to introduce cluster mode with Mesos, we must create a
>> framework that is long running that can be running in your datacenter, and
>> can handle launching spark drivers on demand and handle HA, etc. This is
>> what the dispatcher is all about.
>>
>> So the idea is that you should launch the dispatcher not on the client,
>> but on a machine in your datacenter. In Mesosphere's DCOS we launch all
>> frameworks and long running services with Marathon, and you can use
>> Marathon to launch the Spark dispatcher.
>>
>> Then all clients instead of specifying the Mesos master URL (e.g:
>> mesos://mesos.master:2181), then just talks to the dispatcher only
>> (mesos://spark-dispatcher.mesos:7077), and the dispatcher will then start
>> and watch the driver for you.
>>
>> Tim
>>
>>
>>
>> On Thu, Sep 10, 2015 at 10:13 AM, Tom Waterhouse (tomwater) <
>> tomwa...@cisco.com> wrote:
>>
>>> After spending most of yesterday scouring the Internet for sources of
>>> documentation for submitting Spark jobs in cluster mode to a Spark cluster
>>> managed by Mesos I was able to do just that, but I am not convinced that
>>> how I have things setup is correct.
>>>
>>> I used the Mesos published
>>>
Re: Spark on Mesos with Jobs in Cluster Mode Documentation
Small update: I found properties-file spark-submit parameter by reading
the code and that seems to work, but appears to be undocumented in the
submitting applications doc page.
- Alan
On Thu, Sep 17, 2015 at 12:39 PM, Alan Braithwaite <a...@cloudflare.com>
wrote:
> One other piece of information:
>
> We're using zookeeper for persistence and when we brought the dispatcher
> back online, it crashed on the same exception after loading the config from
> zookeeper.
>
> Cheers,
> - Alan
>
> On Thu, Sep 17, 2015 at 12:29 PM, Alan Braithwaite <a...@cloudflare.com>
> wrote:
>
>> Hey All,
>>
>> To bump this thread once again, I'm having some trouble using the
>> dispatcher as well.
>>
>> I'm using Mesos Cluster Manager with Docker Executors. I've deployed the
>> dispatcher as Marathon job. When I submit a job using spark submit, the
>> dispatcher writes back that the submission was successful and then promptly
>> dies in marathon. Looking at the logs reveals it was hitting the following
>> line:
>>
>> 398: throw new SparkException("Executor Spark home
>> `spark.mesos.executor.home` is not set!")
>>
>> Which is odd because it's set in multiple places (SPARK_HOME,
>> spark.mesos.executor.home, spark.home, etc). Reading the code, it
>> appears that the driver desc pulls only from the request and disregards any
>> other properties that may be configured. Testing by passing --conf
>> spark.mesos.executor.home=/usr/local/spark on the command line to
>> spark-submit confirms this. We're trying to isolate the number of places
>> where we have to set properties within spark and were hoping that it will
>> be possible to have this pull in the spark-defaults.conf from somewhere, or
>> at least allow the user to inform the dispatcher through spark-submit that
>> those properties will be available once the job starts.
>>
>> Finally, I don't think the dispatcher should crash in this event. It
>> seems not exceptional that a job is misconfigured when submitted.
>>
>> Please direct me on the right path if I'm headed in the wrong direction.
>> Also let me know if I should open some tickets for these issues.
>>
>> Thanks,
>> - Alan
>>
>> On Fri, Sep 11, 2015 at 1:05 PM, Tim Chen <t...@mesosphere.io> wrote:
>>
>>> Yes you can create an issue, or actually contribute a patch to update it
>>> :)
>>>
>>> Sorry the docs is a bit light, I'm going to make it more complete along
>>> the way.
>>>
>>> Tim
>>>
>>>
>>> On Fri, Sep 11, 2015 at 11:11 AM, Tom Waterhouse (tomwater) <
>>> tomwa...@cisco.com> wrote:
>>>
>>>> Tim,
>>>>
>>>> Thank you for the explanation. You are correct, my Mesos experience is
>>>> very light, and I haven’t deployed anything via Marathon yet. What you
>>>> have stated here makes sense, I will look into doing this.
>>>>
>>>> Adding this info to the docs would be great. Is the appropriate action
>>>> to create an issue regarding improvement of the docs? For those of us who
>>>> are gaining the experience having such a pointer is very helpful.
>>>>
>>>> Tom
>>>>
>>>> From: Tim Chen <t...@mesosphere.io>
>>>> Date: Thursday, September 10, 2015 at 10:25 AM
>>>> To: Tom Waterhouse <tomwa...@cisco.com>
>>>> Cc: "user@spark.apache.org" <user@spark.apache.org>
>>>> Subject: Re: Spark on Mesos with Jobs in Cluster Mode Documentation
>>>>
>>>> Hi Tom,
>>>>
>>>> Sorry the documentation isn't really rich, since it's probably assuming
>>>> users understands how Mesos and framework works.
>>>>
>>>> First I need explain the rationale of why create the dispatcher. If
>>>> you're not familiar with Mesos yet, each node in your datacenter is
>>>> installed a Mesos slave where it's responsible for publishing resources and
>>>> running/watching tasks, and Mesos master is responsible for taking the
>>>> aggregated resources and scheduling them among frameworks.
>>>>
>>>> Frameworks are not managed by Mesos, as Mesos master/slave doesn't
>>>> launch and maintain framework but assume they're launched and kept running
>>>> on its own. All the existing frameworks in the ecosystem therefore all have
>>>> their own ways to deploy, HA and persist state (e.g: Aurora, Marathon,
>>>> etc).
>>>>
>&g
Re: Spark on Mesos with Jobs in Cluster Mode Documentation
Hi Tim,
Thanks for the follow up. It's not so much that I expect the executor to
inherit the configuration of the dispatcher as I* don't *expect the
dispatcher to make assumptions about the system environment of the executor
(since it lives in a docker). I could potentially see a case where you
might want to explicitly forbid the defaults, but I can't think of any
right now.
Otherwise, I'm confused as to why the defaults in the docker image for the
executor are just ignored. I suppose that it's the dispatchers job to
ensure the *exact* configuration of the executor, regardless of the
defaults set on the executors machine? Is that the assumption being made?
I can understand that in contexts which aren't docker driven since jobs
could be rolling out in the middle of a config update. Trying to think of
this outside the terms of just mesos/docker (since I'm fully aware that
docker doesn't rule the world yet).
So I can see this from both perspectives now and passing in the properties
file will probably work just fine for me, but for my better understanding:
When the executor starts, will it read any of the environment that it's
executing in or will it just take only the properties given to it by the
dispatcher and nothing more?
Lemme know if anything needs more clarification and thanks for your mesos
contribution to spark!
- Alan
On Thu, Sep 17, 2015 at 5:03 PM, Timothy Chen <t...@mesosphere.io> wrote:
> Hi Alan,
>
> If I understand correctly, you are setting executor home when you launch
> the dispatcher and not on the configuration when you submit job, and expect
> it to inherit that configuration?
>
> When I worked on the dispatcher I was assuming all configuration is passed
> to the dispatcher to launch the job exactly how you will need to launch it
> with client mode.
>
> But indeed it shouldn't crash dispatcher, I'll take a closer look when I
> get a chance.
>
> Can you recommend changes on the documentation, either in email or a PR?
>
> Thanks!
>
> Tim
>
> Sent from my iPhone
>
> On Sep 17, 2015, at 12:29 PM, Alan Braithwaite <a...@cloudflare.com>
> wrote:
>
> Hey All,
>
> To bump this thread once again, I'm having some trouble using the
> dispatcher as well.
>
> I'm using Mesos Cluster Manager with Docker Executors. I've deployed the
> dispatcher as Marathon job. When I submit a job using spark submit, the
> dispatcher writes back that the submission was successful and then promptly
> dies in marathon. Looking at the logs reveals it was hitting the following
> line:
>
> 398: throw new SparkException("Executor Spark home
> `spark.mesos.executor.home` is not set!")
>
> Which is odd because it's set in multiple places (SPARK_HOME,
> spark.mesos.executor.home, spark.home, etc). Reading the code, it
> appears that the driver desc pulls only from the request and disregards any
> other properties that may be configured. Testing by passing --conf
> spark.mesos.executor.home=/usr/local/spark on the command line to
> spark-submit confirms this. We're trying to isolate the number of places
> where we have to set properties within spark and were hoping that it will
> be possible to have this pull in the spark-defaults.conf from somewhere, or
> at least allow the user to inform the dispatcher through spark-submit that
> those properties will be available once the job starts.
>
> Finally, I don't think the dispatcher should crash in this event. It
> seems not exceptional that a job is misconfigured when submitted.
>
> Please direct me on the right path if I'm headed in the wrong direction.
> Also let me know if I should open some tickets for these issues.
>
> Thanks,
> - Alan
>
> On Fri, Sep 11, 2015 at 1:05 PM, Tim Chen <t...@mesosphere.io> wrote:
>
>> Yes you can create an issue, or actually contribute a patch to update it
>> :)
>>
>> Sorry the docs is a bit light, I'm going to make it more complete along
>> the way.
>>
>> Tim
>>
>>
>> On Fri, Sep 11, 2015 at 11:11 AM, Tom Waterhouse (tomwater) <
>> tomwa...@cisco.com> wrote:
>>
>>> Tim,
>>>
>>> Thank you for the explanation. You are correct, my Mesos experience is
>>> very light, and I haven’t deployed anything via Marathon yet. What you
>>> have stated here makes sense, I will look into doing this.
>>>
>>> Adding this info to the docs would be great. Is the appropriate action
>>> to create an issue regarding improvement of the docs? For those of us who
>>> are gaining the experience having such a pointer is very helpful.
>>>
>>> Tom
>>>
>>> From: Tim Chen <t...@mesosphere.io>
>>> Date: Thursday, September 10, 2015 a
Re: Spark-submit fails when jar is in HDFS
Did you try this way? /usr/local/spark/bin/spark-submit --master mesos://mesos.master:5050 --conf spark.mesos.executor.docker.image=docker.repo/spark:latest --class org.apache.spark.examples.SparkPi --jars hdfs://hdfs1/tmp/spark-examples-1.4.1-hadoop2.6.0-cdh5.4.4.jar 100 I did, and got the same error (I verified again right now too). Also, Spark on Mesos supports cluster mode: http://spark.apache.org/docs/latest/running-on-mesos.html#cluster-mode Oh cool! Looks like this page needs to be updated then: http://spark.apache.org/docs/latest/submitting-applications.html Thanks! - Alan
