Thank you Matei, it's running for me with Spark 0.3 + pre-protobuf, and I'm testing with provided example "run spark.examples.SparkPi mesos://[email protected]:5050 3". However, it seems I keep losing TID after starting the task, and then job aborted with 4 tries. It might be the problem on my side too, did this happen before? I could go for trying masters for spark + mesos. 11/08/22 15:48:27 INFO spark.SimpleJob: Lost TID 13 (task 0:2) 11/08/22 15:48:27 ERROR spark.SimpleJob: Task 0:2 failed more than 4 times; aborting job 11/08/22 15:48:28 INFO spark.MesosScheduler: Ignoring update from TID 12 because its job is gone 11/08/22 15:48:28 INFO spark.MesosScheduler: Ignoring update from TID 14 because its job is gone
Thanks! Joe On 8/22/11 3:38 PM, "Joe Xie" <[email protected]> wrote: >Thank you Matei! I can go for Spark 0.3 + pre-protobuf. I understand there >might be lots of new updates on the master branch. Is there any >significant difference in current master with Spark 0.3 + pre-protobuf? I >could also try to keep a separate branch with stable masters if that's the >case. >Best, >Joe >On 8/22/11 3:21 PM, "Matei Zaharia" <[email protected]> wrote: > >>If you want the most stable versions right now, I recommend Spark 0.3 >>(download it from the website or the 0.3-scala-2.9 tag in Git) and >>pre-protobuf Mesos. However, the current master Spark + master Mesos >>should also work OK; it's just that further commits may break things >>temporarily. >> >>Matei >> >>On Aug 22, 2011, at 3:17 PM, Joe Xie wrote: >> >>> Hi Matei, >>> Thanks for your feedback. I bet I'm using the master brach of Spark and >>> pre-protobuf version of Mesos. Do you mean I should go for Spark >>>(master) >>> + protobuf (master). Let's me know the most stable combination of >>>braches >>> from Spark and pre-protobuf, I can try them out. >>> >>> Best, >>> Joe >>> >>> >>> On 8/22/11 3:09 PM, "Matei Zaharia" <[email protected]> wrote: >>> >>>> Are you using the master branch of Spark, or one of the releases (e.g. >>>> 0.3)? The releases work with the pre-protobuf version of Mesos but the >>>> master branch of Spark actually requires the master branch of Mesos >>>>now. >>>> That could cause the kind of error you're seeing. >>>> >>>> Matei >>>> >>>> On Aug 22, 2011, at 3:06 PM, Joe Xie wrote: >>>> >>>>> Sorry this should have been sent to user mail list. >>>>> >>>>> On 8/22/11 2:19 PM, "Joe Xie" <[email protected]> wrote: >>>>> >>>>>> Hi there: >>>>>> >>>>>> I has build and installed mesos (pre-protobuf branch for spark) and >>>>>> spark, >>>>>> and deployed them to a cluster. However, I have following error when >>>>>>I >>>>>> run >>>>>> spark task on the mesos URL of my master, the spark task runs fine >>>>>>in >>>>>> local mode. Any suggestion or help? Thanks a lot! >>>>>> Joe >>>>>> >>>>>> My os: CentOS 5.2 >>>>>> Python: 2.6 >>>>>> Gcc: 4.2.1 >>>>>> Swig:1.3.40 >>>>>> ./run spark.examples.SparkPi mesos://[email protected]:5050 1 >>>>>> 11/08/22 13:31:11 INFO spark.CacheTrackerActor: Registered actor on >>>>>> port >>>>>> 50501 >>>>>> 11/08/22 13:31:11 INFO spark.MapOutputTrackerActor: Registered actor >>>>>>on >>>>>> port 50501 >>>>>> 11/08/22 13:31:11 INFO spark.SparkContext: Starting job... >>>>>> 11/08/22 13:31:11 INFO spark.CacheTracker: Registering RDD ID 0 with >>>>>> cache >>>>>> 11/08/22 13:31:11 INFO spark.CacheTrackerActor: Registering RDD 0 >>>>>>with >>>>>> 1 >>>>>> partitions >>>>>> 11/08/22 13:31:11 INFO spark.CacheTrackerActor: Asked for current >>>>>>cache >>>>>> locations >>>>>> 11/08/22 13:31:12 INFO spark.MesosScheduler: Final stage: Stage 0 >>>>>> 11/08/22 13:31:12 INFO spark.MesosScheduler: Parents of final stage: >>>>>> List() >>>>>> 11/08/22 13:31:12 INFO spark.MesosScheduler: Missing parents: List() >>>>>> 11/08/22 13:31:12 INFO spark.MesosScheduler: Submitting Stage 0, >>>>>>which >>>>>> has >>>>>> no missing parents >>>>>> 11/08/22 13:31:12 INFO spark.MesosScheduler: Got a job with 1 tasks >>>>>> Exception in thread "Spark scheduler" >>>>>>java.lang.UnsatisfiedLinkError: >>>>>> org.apache.mesos.MesosSchedulerDriver.initialize()V >>>>>> at org.apache.mesos.MesosSchedulerDriver.initialize(Native Method) >>>>>> at >>>>>> >>>>>> >>>>>>org.apache.mesos.MesosSchedulerDriver.<init>(MesosSchedulerDriver.jav >>>>>>a >>>>>>:4 >>>>>> 2) >>>>>> at >>>>>> >>>>>> >>>>>>org.apache.mesos.MesosSchedulerDriver.<init>(MesosSchedulerDriver.jav >>>>>>a >>>>>>:4 >>>>>> 6) >>>>>> at spark.MesosScheduler$$anon$1.run(MesosScheduler.scala:94) >>>>>> >>>>>> Sent to wrong address previously >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> On 8/21/11 4:30 PM, "[email protected]" >>>>>>> <[email protected]> wrote: >>>>>>> >>>>>>>> Hi! This is the ezmlm program. I'm managing the >>>>>>>> [email protected] mailing list. >>>>>>>> >>>>>>>> I'm working for my owner, who can be reached >>>>>>>> at [email protected]. >>>>>>>> >>>>>>>> Acknowledgment: I have added the address >>>>>>>> >>>>>>>> [email protected] >>>>>>>> >>>>>>>> to the mesos-dev mailing list. >>>>>>>> >>>>>>>> Welcome to [email protected]! >>>>>>>> >>>>>>>> Please save this message so that you know the address you are >>>>>>>> subscribed under, in case you later want to unsubscribe or change >>>>>>>> your >>>>>>>> subscription address. >>>>>>>> >>>>>>>> >>>>>>>> --- Administrative commands for the mesos-dev list --- >>>>>>>> >>>>>>>> I can handle administrative requests automatically. Please >>>>>>>> do not send them to the list address! Instead, send >>>>>>>> your message to the correct command address: >>>>>>>> >>>>>>>> To subscribe to the list, send a message to: >>>>>>>> <[email protected]> >>>>>>>> >>>>>>>> To remove your address from the list, send a message to: >>>>>>>> <[email protected]> >>>>>>>> >>>>>>>> Send mail to the following for info and FAQ for this list: >>>>>>>> <[email protected]> >>>>>>>> <[email protected]> >>>>>>>> >>>>>>>> Similar addresses exist for the digest list: >>>>>>>> <[email protected]> >>>>>>>> <[email protected]> >>>>>>>> >>>>>>>> To get messages 123 through 145 (a maximum of 100 per request), >>>>>>>>mail: >>>>>>>> <[email protected]> >>>>>>>> >>>>>>>> To get an index with subject and author for messages 123-456 , >>>>>>>>mail: >>>>>>>> <[email protected]> >>>>>>>> >>>>>>>> They are always returned as sets of 100, max 2000 per request, >>>>>>>> so you'll actually get 100-499. >>>>>>>> >>>>>>>> To receive all messages with the same subject as message 12345, >>>>>>>> send a short message to: >>>>>>>> <[email protected]> >>>>>>>> >>>>>>>> The messages should contain one line or word of text to avoid >>>>>>>>being >>>>>>>> treated as sp@m, but I will ignore their content. >>>>>>>> Only the ADDRESS you send to is important. >>>>>>>> >>>>>>>> You can start a subscription for an alternate address, >>>>>>>> for example "[email protected]", just add a hyphen and your >>>>>>>> address (with '=' instead of '@') after the command word: >>>>>>>> <[email protected]> >>>>>>>> >>>>>>>> To stop subscription for this address, mail: >>>>>>>> <[email protected]> >>>>>>>> >>>>>>>> In both cases, I'll send a confirmation message to that address. >>>>>>>>When >>>>>>>> you receive it, simply reply to it to complete your subscription. >>>>>>>> >>>>>>>> If despite following these instructions, you do not get the >>>>>>>> desired results, please contact my owner at >>>>>>>> [email protected]. Please be patient, my owner >>>>>>>>is >>>>>>>> a >>>>>>>> lot slower than I am ;-) >>>>>>>> >>>>>>>> --- Enclosed is a copy of the request I received. >>>>>>>> >>>>>>>> Return-Path: <[email protected]> >>>>>>>> Received: (qmail 39962 invoked by uid 99); 21 Aug 2011 23:30:07 >>>>>>>>-0000 >>>>>>>> Received: from nike.apache.org (HELO nike.apache.org) >>>>>>>> (192.87.106.230) >>>>>>>> by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Aug 2011 >>>>>>>>23:30:07 >>>>>>>> +0000 >>>>>>>> X-ASF-Spam-Status: No, hits=-0.0 required=5.0 >>>>>>>> tests=SPF_PASS >>>>>>>> X-Spam-Check-By: apache.org >>>>>>>> Received-SPF: pass (nike.apache.org: domain of [email protected] >>>>>>>> designates >>>>>>>> 67.231.145.42 as permitted sender) >>>>>>>> Received: from [67.231.145.42] (HELO mx0a-00082601.pphosted.com) >>>>>>>> (67.231.145.42) >>>>>>>> by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Aug 2011 >>>>>>>>23:30:00 >>>>>>>> +0000 >>>>>>>> Received: from pps.filterd (m0004348 [127.0.0.1]) >>>>>>>> by m0004348.ppops.net (8.14.4/8.14.4) with SMTP id >>>>>>>> p7LNTI6g015638 >>>>>>>> for >>>>>>>> >>>>>>>> >>>>>>>><mesos-dev-sc.1313966607.anahdnaofbfbnfgijcem-zhiyong=fb.com@incuba >>>>>>>>t >>>>>>>>or >>>>>>>> .a >>>>>>>> p >>>>>>>> a >>>>>>>> che.org>; Sun, 21 Aug 2011 16:29:18 -0700 >>>>>>>> Received: from mail.thefacebook.com (corpout1.snc1.tfbnw.net >>>>>>>> [66.220.144.38]) >>>>>>>> by m0004348.ppops.net with ESMTP id yb5cerc88-2 >>>>>>>> (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT) >>>>>>>> for >>>>>>>> >>>>>>>> >>>>>>>><mesos-dev-sc.1313966607.anahdnaofbfbnfgijcem-zhiyong=fb.com@incuba >>>>>>>>t >>>>>>>>or >>>>>>>> .a >>>>>>>> p >>>>>>>> a >>>>>>>> che.org>; Sun, 21 Aug 2011 16:29:18 -0700 >>>>>>>> Received: from SC-MBX01-5.TheFacebook.com >>>>>>>> ([fe80::f05e:c7f1:c7dd:e640]) >>>>>>>> by >>>>>>>> sc-hub04.TheFacebook.com ([192.168.18.212]) with mapi id >>>>>>>> 14.01.0289.001; >>>>>>>> Sun, >>>>>>>> 21 Aug 2011 16:29:14 -0700 >>>>>>>> From: Joe Xie <[email protected]> >>>>>>>> To: >>>>>>>> >>>>>>>> >>>>>>>>"mesos-dev-sc.1313966607.anahdnaofbfbnfgijcem-zhiyong=fb.com@incuba >>>>>>>>t >>>>>>>>or >>>>>>>> .a >>>>>>>> p >>>>>>>> a >>>>>>>> che.org" >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> <mesos-dev-sc.1313966607.anahdnaofbfbnfgijcem-zhiyong=fb.com@incub >>>>>>>>a >>>>>>>>to >>>>>>>> r. >>>>>>>> a >>>>>>>> p >>>>>>>> ache.org> >>>>>>>> Subject: Re: confirm subscribe to [email protected] >>>>>>>> Thread-Topic: confirm subscribe to [email protected] >>>>>>>> Thread-Index: AQHMYFO/ABVA64TXDkGEVkIetlsT6JUn85UA >>>>>>>> Date: Sun, 21 Aug 2011 23:29:13 +0000 >>>>>>>> Message-ID: <ca76e2a4.4259%[email protected]> >>>>>>>> In-Reply-To: <[email protected]> >>>>>>>> Accept-Language: en-US >>>>>>>> Content-Language: en-US >>>>>>>> X-MS-Has-Attach: >>>>>>>> X-MS-TNEF-Correlator: >>>>>>>> x-originating-ip: [192.168.18.252] >>>>>>>> Content-Type: text/plain; charset="us-ascii" >>>>>>>> Content-ID: <[email protected]> >>>>>>>> Content-Transfer-Encoding: quoted-printable >>>>>>>> MIME-Version: 1.0 >>>>>>>> X-Virus-Checked: Checked by ClamAV on apache.org >>>>>>>> >>>>>>>> Thanks! >>>>>>>> Joe >>>>>>>> On 8/21/11 3:43 PM, "[email protected]" >>>>>>>> <[email protected]> wrote: >>>>>>>> >>>>>>>>> Hi! This is the ezmlm program. I'm managing the >>>>>>>>> [email protected] mailing list. >>>>>>>>> >>>>>>>>> I'm working for my owner, who can be reached >>>>>>>>> at [email protected]. >>>>>>>>> >>>>>>>>> To confirm that you would like >>>>>>>>> >>>>>>>>> [email protected] >>>>>>>>> >>>>>>>>> added to the mesos-dev mailing list, please send >>>>>>>>> a short reply to this address: >>>>>>>>> >>>>>>>>> =20 >>>>>>>>> >>>>>>>>> >>>>>>>>>mesos-dev-sc.1313966607.anahdnaofbfbnfgijcem-zhiyong=3Dfb.com@incu >>>>>>>>>b >>>>>>>>>at >>>>>>>>> or >>>>>>>>> . >>>>>>>>> a >>>>>>>>> p= >>>>>>>> ac >>>>>>>>> he.org >>>>>>>>> >>>>>>>>> Usually, this happens when you just hit the "reply" button. >>>>>>>>> If this does not work, simply copy the address and paste it into >>>>>>>>> the "To:" field of a new message. >>>>>>>>> >>>>>>>>> or click here: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> mailto:mesos-dev-sc.1313966607.anahdnaofbfbnfgijcem-zhiyong=3Dfb. >>>>>>>>>c >>>>>>>>>om >>>>>>>>> @i >>>>>>>>> n >>>>>>>>> c >>>>>>>>> u= >>>>>>>> ba >>>>>>>>> tor.apache.org >>>>>>>>> >>>>>>>>> This confirmation serves two purposes. First, it verifies that I >>>>>>>>>am >>>>>>>>> able >>>>>>>>> to get mail through to you. Second, it protects you in case >>>>>>>>>someone >>>>>>>>> forges a subscription request in your name. >>>>>>>>> >>>>>>>>> Please note that ALL Apache dev- and user- mailing lists are >>>>>>>>> publicly >>>>>>>>> archived. Do familiarize yourself with Apache's public archive >>>>>>>>> policy >>>>>>>>> at >>>>>>>>> >>>>>>>>> http://www.apache.org/foundation/public-archives.html >>>>>>>>> >>>>>>>>> prior to subscribing and posting messages to >>>>>>>>> [email protected]. >>>>>>>>> If you're not sure whether or not the policy applies to this >>>>>>>>>mailing >>>>>>>>> list, >>>>>>>>> assume it does unless the list name contains the word "private" >>>>>>>>>in >>>>>>>>> it. >>>>>>>>> >>>>>>>>> Some mail programs are broken and cannot handle long addresses. >>>>>>>>>If >>>>>>>>> you >>>>>>>>> cannot reply to this request, instead send a message to >>>>>>>>> <[email protected]> and put the >>>>>>>>> entire address listed above into the "Subject:" line. >>>>>>>>> >>>>>>>>> >>>>>>>>> --- Administrative commands for the mesos-dev list --- >>>>>>>>> >>>>>>>>> I can handle administrative requests automatically. Please >>>>>>>>> do not send them to the list address! Instead, send >>>>>>>>> your message to the correct command address: >>>>>>>>> >>>>>>>>> To subscribe to the list, send a message to: >>>>>>>>> <[email protected]> >>>>>>>>> >>>>>>>>> To remove your address from the list, send a message to: >>>>>>>>> <[email protected]> >>>>>>>>> >>>>>>>>> Send mail to the following for info and FAQ for this list: >>>>>>>>> <[email protected]> >>>>>>>>> <[email protected]> >>>>>>>>> >>>>>>>>> Similar addresses exist for the digest list: >>>>>>>>> <[email protected]> >>>>>>>>> <[email protected]> >>>>>>>>> >>>>>>>>> To get messages 123 through 145 (a maximum of 100 per request), >>>>>>>>> mail: >>>>>>>>> <[email protected]> >>>>>>>>> >>>>>>>>> To get an index with subject and author for messages 123-456 , >>>>>>>>>mail: >>>>>>>>> <[email protected]> >>>>>>>>> >>>>>>>>> They are always returned as sets of 100, max 2000 per request, >>>>>>>>> so you'll actually get 100-499. >>>>>>>>> >>>>>>>>> To receive all messages with the same subject as message 12345, >>>>>>>>> send a short message to: >>>>>>>>> <[email protected]> >>>>>>>>> >>>>>>>>> The messages should contain one line or word of text to avoid >>>>>>>>>being >>>>>>>>> treated as sp@m, but I will ignore their content. >>>>>>>>> Only the ADDRESS you send to is important. >>>>>>>>> >>>>>>>>> You can start a subscription for an alternate address, >>>>>>>>> for example "[email protected]", just add a hyphen and your >>>>>>>>> address (with '=3D' instead of '@') after the command word: >>>>>>>>> <[email protected]> >>>>>>>>> >>>>>>>>> To stop subscription for this address, mail: >>>>>>>>> <[email protected]> >>>>>>>>> >>>>>>>>> In both cases, I'll send a confirmation message to that address. >>>>>>>>> When >>>>>>>>> you receive it, simply reply to it to complete your subscription. >>>>>>>>> >>>>>>>>> If despite following these instructions, you do not get the >>>>>>>>> desired results, please contact my owner at >>>>>>>>> [email protected]. Please be patient, my owner >>>>>>>>> is a >>>>>>>>> lot slower than I am ;-) >>>>>>>>> >>>>>>>>> --- Enclosed is a copy of the request I received. >>>>>>>>> >>>>>>>>> Return-Path: <[email protected]> >>>>>>>>> Received: (qmail 19986 invoked by uid 99); 21 Aug 2011 22:43:26 >>>>>>>>> -0000 >>>>>>>>> Received: from athena.apache.org (HELO athena.apache.org) >>>>>>>>> (140.211.11.136) >>>>>>>>> by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Aug 2011 >>>>>>>>>22:43:26 >>>>>>>>> +0000 >>>>>>>>> X-ASF-Spam-Status: No, hits=3D0.2 required=3D10.0 >>>>>>>>> tests=3DASF_LIST_OPS,HTML_MESSAGE,SPF_PASS >>>>>>>>> X-Spam-Check-By: apache.org >>>>>>>>> Received-SPF: pass (athena.apache.org: domain of [email protected] >>>>>>>>> designates 67.231.145.42 as permitted sender) >>>>>>>>> Received: from [67.231.145.42] (HELO mx0a-00082601.pphosted.com) >>>>>>>>> (67.231.145.42) >>>>>>>>> by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Aug 2011 >>>>>>>>>22:43:19 >>>>>>>>> +0000 >>>>>>>>> Received: from pps.filterd (m0004347 [127.0.0.1]) >>>>>>>>> by m0004347.ppops.net (8.14.4/8.14.4) with SMTP id >>>>>>>>>p7LMcbSq017336 >>>>>>>>> for <[email protected]>; Sun, 21 Aug 2011 >>>>>>>>> 15:42:58 -0700 >>>>>>>>> Received: from mail.thefacebook.com (corpout1.snc1.tfbnw.net >>>>>>>>> [66.220.144.38]) >>>>>>>>> by m0004347.ppops.net with ESMTP id yb0jhrkt9-1 >>>>>>>>> (version=3DTLSv1/SSLv3 cipher=3DAES128-SHA bits=3D128 >>>>>>>>>verify=3DNOT) >>>>>>>>> for <[email protected]>; Sun, 21 Aug 2011 >>>>>>>>> 15:42:58 -0700 >>>>>>>>> Received: from SC-MBX01-5.TheFacebook.com >>>>>>>>> ([fe80::f05e:c7f1:c7dd:e640]) >>>>>>>>> by >>>>>>>>> sc-hub03.TheFacebook.com ([192.168.18.198]) with mapi id >>>>>>>>> 14.01.0289.001; >>>>>>>>> Sun, >>>>>>>>> 21 Aug 2011 15:42:53 -0700 >>>>>>>>> From: Joe Xie <[email protected]> >>>>>>>>> To: "[email protected]" >>>>>>>>> <[email protected]> >>>>>>>>> Subject: can i join this >>>>>>>>> Thread-Topic: can i join this >>>>>>>>> Thread-Index: AQHMYFOp36obbNljKE2n3mexQg9sQQ=3D=3D >>>>>>>>> Date: Sun, 21 Aug 2011 22:42:53 +0000 >>>>>>>>> Message-ID: <ca76d7f7.4256%[email protected]> >>>>>>>>> Accept-Language: en-US >>>>>>>>> Content-Language: en-US >>>>>>>>> X-MS-Has-Attach:=20 >>>>>>>>> X-MS-TNEF-Correlator: >>>>>>>>> x-originating-ip: [192.168.18.252] >>>>>>>>> Content-Type: multipart/alternative; >>>>>>>>> boundary=3D"_000_CA76D7F74256zhiyongfbcom_" >>>>>>>>> MIME-Version: 1.0 >>>>>>>>> >>>>>>>>> --_000_CA76D7F74256zhiyongfbcom_ >>>>>>>>> Content-Type: text/plain; charset=3D"us-ascii" >>>>>>>>> Content-Transfer-Encoding: quoted-printable >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Joe >>>>>>>>> >>>>>>>>> --_000_CA76D7F74256zhiyongfbcom_ >>>>>>>>> Content-Type: text/html; charset=3D"us-ascii" >>>>>>>>> Content-ID: <[email protected]> >>>>>>>>> Content-Transfer-Encoding: quoted-printable >>>>>>>>> >>>>>>>>> <html> >>>>>>>>> <head> >>>>>>>>> <meta http-equiv=3D3D"Content-Type" content=3D3D"text/html; >>>>>>>>> charset=3D3Dus-ascii"=3D >>>>>>>>>> >>>>>>>>> </head> >>>>>>>>> <body style=3D3D"word-wrap: break-word; -webkit-nbsp-mode: space; >>>>>>>>> -webkit-lin=3D >>>>>>>>> e-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; >>>>>>>>> font-fami=3D >>>>>>>>> ly: Calibri, sans-serif; "> >>>>>>>>> <div>Best,</div> >>>>>>>>> <div>Joe</div> >>>>>>>>> </body> >>>>>>>>> </html> >>>>>>>>> >>>>>>>>> --_000_CA76D7F74256zhiyongfbcom_-- >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
