Multiple drivers, same worker
Hi everyone, We've been getting an issue with spark lately where multiple drivers are assigned to a same worker but resources are never assigned to them and get "stuck" forever. If I login in the worker machine I see that the driver processes aren't really running and the worker's log don't show any error or anything related to the driver. The master UI does show the drivers as submitted and in RUNNING state. Not sure where else to look for clues, any ideas? -- Andrés Blanco Morales
Re: Multiple drivers, same worker
Forgot to mention that it doesn't happen every time, it's pretty random so far. We've have complete days when it behaves just fine and others when it gets crazy. We're using spark 1.5.2 2015-12-09 13:33 GMT-06:00 andresb...@gmail.com : > Hi everyone, > > We've been getting an issue with spark lately where multiple drivers are > assigned to a same worker but resources are never assigned to them and get > "stuck" forever. > > If I login in the worker machine I see that the driver processes aren't > really running and the worker's log don't show any error or anything > related to the driver. The master UI does show the drivers as submitted and > in RUNNING state. > > > Not sure where else to look for clues, any ideas? > > -- > Andrés Blanco Morales > -- Andrés Blanco Morales
Re: Multiple drivers, same worker
Sadly, no. The only evidence I have is the master's log which shows that the Driver was requested: 15/12/09 18:25:06 INFO Master: Driver submitted org.apache.spark.deploy.worker.DriverWrapper 15/12/09 18:25:06 INFO Master: Launching driver driver-20151209182506-0164 on worker worker-20151209181534-172.31.31.159-7077 2015-12-09 14:19 GMT-06:00 Ted Yu : > When this happened, did you have a chance to take jstack of the stuck > driver process ? > > Thanks > > On Wed, Dec 9, 2015 at 11:38 AM, andresb...@gmail.com < > andresb...@gmail.com> wrote: > >> Forgot to mention that it doesn't happen every time, it's pretty random >> so far. We've have complete days when it behaves just fine and others when >> it gets crazy. We're using spark 1.5.2 >> >> 2015-12-09 13:33 GMT-06:00 andresb...@gmail.com : >> >>> Hi everyone, >>> >>> We've been getting an issue with spark lately where multiple drivers are >>> assigned to a same worker but resources are never assigned to them and get >>> "stuck" forever. >>> >>> If I login in the worker machine I see that the driver processes aren't >>> really running and the worker's log don't show any error or anything >>> related to the driver. The master UI does show the drivers as submitted and >>> in RUNNING state. >>> >>> >>> Not sure where else to look for clues, any ideas? >>> >>> -- >>> Andrés Blanco Morales >>> >> >> >> >> -- >> Andrés Blanco Morales >> > > -- Andrés Blanco Morales
Re: Multiple drivers, same worker
Ok, attached you can see the jstack 2015-12-09 14:22 GMT-06:00 andresb...@gmail.com : > Sadly, no. > > The only evidence I have is the master's log which shows that the Driver > was requested: > > 15/12/09 18:25:06 INFO Master: Driver submitted > org.apache.spark.deploy.worker.DriverWrapper > 15/12/09 18:25:06 INFO Master: Launching driver driver-20151209182506-0164 > on worker worker-20151209181534-172.31.31.159-7077 > > > > 2015-12-09 14:19 GMT-06:00 Ted Yu : > >> When this happened, did you have a chance to take jstack of the stuck >> driver process ? >> >> Thanks >> >> On Wed, Dec 9, 2015 at 11:38 AM, andresb...@gmail.com < >> andresb...@gmail.com> wrote: >> >>> Forgot to mention that it doesn't happen every time, it's pretty random >>> so far. We've have complete days when it behaves just fine and others when >>> it gets crazy. We're using spark 1.5.2 >>> >>> 2015-12-09 13:33 GMT-06:00 andresb...@gmail.com : >>> >>>> Hi everyone, >>>> >>>> We've been getting an issue with spark lately where multiple drivers >>>> are assigned to a same worker but resources are never assigned to them and >>>> get "stuck" forever. >>>> >>>> If I login in the worker machine I see that the driver processes aren't >>>> really running and the worker's log don't show any error or anything >>>> related to the driver. The master UI does show the drivers as submitted and >>>> in RUNNING state. >>>> >>>> >>>> Not sure where else to look for clues, any ideas? >>>> >>>> -- >>>> Andrés Blanco Morales >>>> >>> >>> >>> >>> -- >>> Andrés Blanco Morales >>> >> >> > > > -- > Andrés Blanco Morales > -- Andrés Blanco Morales jstack Description: Binary data - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Storing DecisionTreeModel
Ok, thanks for your reply! On Tue, Jan 27, 2015 at 2:32 PM, Joseph Bradley wrote: > Hi Andres, > > Currently, serializing the object is probably the best way to do it. > However, there are efforts to support actual model import/export: > https://issues.apache.org/jira/browse/SPARK-4587 > https://issues.apache.org/jira/browse/SPARK-1406 > > I'm hoping to have the PR for the first JIRA ready soon. > > Joseph > > On Tue, Jan 27, 2015 at 7:45 AM, andresbm84 wrote: > >> Hi everyone, >> >> Is there a way to save on disk the model to reuse it later? I could >> serialize the object and save the bytes, but I'm guessing there might be a >> better way to do so. >> >> Has anyone tried that? >> >> >> Andres. >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Storing-DecisionTreeModel-tp21393.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> - >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> > -- Andrés Blanco Morales
