Re: 答复: bug in Worker.scala, ExecutorRunner is not serializable
Sounds good. On Fri, Sep 18, 2015 at 8:50 AM, Shixiong Zhu wrote: > I'm wondering if we should create a tag trait (e.g., LocalMessage) for > messages like this and add the comment in the trait. Looks better than > adding inline comments for all these messages. > > Best Regards, > Shixiong Zhu > > 2015-09-18 15:10 GMT+08:00 Reynold Xin : > >> Maybe we should add some inline comment explaining why it is ok for that >> message to be not serializable. >> >> >> On Thu, Sep 17, 2015 at 4:08 AM, Huangguowei >> wrote: >> >>> Thanks for your reply. I just want to do some monitors, never mind! >>> >>> >>> >>> *发件人:* Shixiong Zhu [mailto:zsxw...@gmail.com] >>> *发送时间:* 2015年9月17日 17:23 >>> *收件人:* Huangguowei; dev@spark.apache.org >>> *主题:* Re: bug in Worker.scala, ExecutorRunner is not serializable >>> >>> >>> >>> RequestWorkerState is an internal message between Worker >>> and WorkerWebUI. Since they are in the same process, that's fine. Actually, >>> these are not public APIs. Could you elaborate your use case? >>> >>> >>> Best Regards, >>> >>> Shixiong Zhu >>> >>> >>> >>> 2015-09-17 16:36 GMT+08:00 Huangguowei : >>> >>> >>> >>> Is it possible to get Executors status when running an application? >>> >>> >>> >>> *发件人:* Sean Owen [mailto:so...@cloudera.com] >>> *发送时间:* 2015年9月17日 15:54 >>> *收件人:* Huangguowei; Dev >>> *主题:* Re: bug in Worker.scala, ExecutorRunner is not serializable >>> >>> >>> >>> Did this cause an error for you? >>> >>> >>> >>> On Thu, Sep 17, 2015, 8:51 AM Huangguowei >>> wrote: >>> >>> >>> >>> In Worker.scala line 480: >>> >>> >>> >>> case RequestWorkerState => >>> >>> sender ! WorkerStateResponse(host, port, workerId, >>> executors.values.toList, >>> >>> finishedExecutors.values.toList, drivers.values.toList, >>> >>> finishedDrivers.values.toList, activeMasterUrl, cores, memory, >>> >>> coresUsed, memoryUsed, activeMasterWebUiUrl) >>> >>> >>> >>> The executors’s type is: >>> >>> val executors = new HashMap[String, ExecutorRunner] >>> >>> >>> >>> but ExecutorRunner cannot be Serialized, so if ask RequestWorkerState >>> will cause java.io.NotSerializableException. >>> >>> >>> >>> >>> >>> >>> >> >> >
Re: 答复: bug in Worker.scala, ExecutorRunner is not serializable
I'm wondering if we should create a tag trait (e.g., LocalMessage) for messages like this and add the comment in the trait. Looks better than adding inline comments for all these messages. Best Regards, Shixiong Zhu 2015-09-18 15:10 GMT+08:00 Reynold Xin : > Maybe we should add some inline comment explaining why it is ok for that > message to be not serializable. > > > On Thu, Sep 17, 2015 at 4:08 AM, Huangguowei > wrote: > >> Thanks for your reply. I just want to do some monitors, never mind! >> >> >> >> *发件人:* Shixiong Zhu [mailto:zsxw...@gmail.com] >> *发送时间:* 2015年9月17日 17:23 >> *收件人:* Huangguowei; dev@spark.apache.org >> *主题:* Re: bug in Worker.scala, ExecutorRunner is not serializable >> >> >> >> RequestWorkerState is an internal message between Worker and WorkerWebUI. >> Since they are in the same process, that's fine. Actually, these are not >> public APIs. Could you elaborate your use case? >> >> >> Best Regards, >> >> Shixiong Zhu >> >> >> >> 2015-09-17 16:36 GMT+08:00 Huangguowei : >> >> >> >> Is it possible to get Executors status when running an application? >> >> >> >> *发件人:* Sean Owen [mailto:so...@cloudera.com] >> *发送时间:* 2015年9月17日 15:54 >> *收件人:* Huangguowei; Dev >> *主题:* Re: bug in Worker.scala, ExecutorRunner is not serializable >> >> >> >> Did this cause an error for you? >> >> >> >> On Thu, Sep 17, 2015, 8:51 AM Huangguowei wrote: >> >> >> >> In Worker.scala line 480: >> >> >> >> case RequestWorkerState => >> >> sender ! WorkerStateResponse(host, port, workerId, >> executors.values.toList, >> >> finishedExecutors.values.toList, drivers.values.toList, >> >> finishedDrivers.values.toList, activeMasterUrl, cores, memory, >> >> coresUsed, memoryUsed, activeMasterWebUiUrl) >> >> >> >> The executors’s type is: >> >> val executors = new HashMap[String, ExecutorRunner] >> >> >> >> but ExecutorRunner cannot be Serialized, so if ask RequestWorkerState >> will cause java.io.NotSerializableException. >> >> >> >> >> >> >> > >
Re: 答复: bug in Worker.scala, ExecutorRunner is not serializable
Maybe we should add some inline comment explaining why it is ok for that message to be not serializable. On Thu, Sep 17, 2015 at 4:08 AM, Huangguowei wrote: > Thanks for your reply. I just want to do some monitors, never mind! > > > > *发件人:* Shixiong Zhu [mailto:zsxw...@gmail.com] > *发送时间:* 2015年9月17日 17:23 > *收件人:* Huangguowei; dev@spark.apache.org > *主题:* Re: bug in Worker.scala, ExecutorRunner is not serializable > > > > RequestWorkerState is an internal message between Worker and WorkerWebUI. > Since they are in the same process, that's fine. Actually, these are not > public APIs. Could you elaborate your use case? > > > Best Regards, > > Shixiong Zhu > > > > 2015-09-17 16:36 GMT+08:00 Huangguowei : > > > > Is it possible to get Executors status when running an application? > > > > *发件人:* Sean Owen [mailto:so...@cloudera.com] > *发送时间:* 2015年9月17日 15:54 > *收件人:* Huangguowei; Dev > *主题:* Re: bug in Worker.scala, ExecutorRunner is not serializable > > > > Did this cause an error for you? > > > > On Thu, Sep 17, 2015, 8:51 AM Huangguowei wrote: > > > > In Worker.scala line 480: > > > > case RequestWorkerState => > > sender ! WorkerStateResponse(host, port, workerId, > executors.values.toList, > > finishedExecutors.values.toList, drivers.values.toList, > > finishedDrivers.values.toList, activeMasterUrl, cores, memory, > > coresUsed, memoryUsed, activeMasterWebUiUrl) > > > > The executors’s type is: > > val executors = new HashMap[String, ExecutorRunner] > > > > but ExecutorRunner cannot be Serialized, so if ask RequestWorkerState will > cause java.io.NotSerializableException. > > > > > > >
答复: bug in Worker.scala, ExecutorRunner is not serializable
Thanks for your reply. I just want to do some monitors, never mind! 发件人: Shixiong Zhu [mailto:zsxw...@gmail.com] 发送时间: 2015年9月17日 17:23 收件人: Huangguowei; dev@spark.apache.org 主题: Re: bug in Worker.scala, ExecutorRunner is not serializable RequestWorkerState is an internal message between Worker and WorkerWebUI. Since they are in the same process, that's fine. Actually, these are not public APIs. Could you elaborate your use case? Best Regards, Shixiong Zhu 2015-09-17 16:36 GMT+08:00 Huangguowei mailto:huangguo...@huawei.com>>: Is it possible to get Executors status when running an application? 发件人: Sean Owen [mailto:so...@cloudera.com<mailto:so...@cloudera.com>] 发送时间: 2015年9月17日 15:54 收件人: Huangguowei; Dev 主题: Re: bug in Worker.scala, ExecutorRunner is not serializable Did this cause an error for you? On Thu, Sep 17, 2015, 8:51 AM Huangguowei mailto:huangguo...@huawei.com>> wrote: In Worker.scala line 480: case RequestWorkerState => sender ! WorkerStateResponse(host, port, workerId, executors.values.toList, finishedExecutors.values.toList, drivers.values.toList, finishedDrivers.values.toList, activeMasterUrl, cores, memory, coresUsed, memoryUsed, activeMasterWebUiUrl) The executors’s type is: val executors = new HashMap[String, ExecutorRunner] but ExecutorRunner cannot be Serialized, so if ask RequestWorkerState will cause java.io.NotSerializableException.
Re: bug in Worker.scala, ExecutorRunner is not serializable
RequestWorkerState is an internal message between Worker and WorkerWebUI. Since they are in the same process, that's fine. Actually, these are not public APIs. Could you elaborate your use case? Best Regards, Shixiong Zhu 2015-09-17 16:36 GMT+08:00 Huangguowei : > > > Is it possible to get Executors status when running an application? > > > > *发件人:* Sean Owen [mailto:so...@cloudera.com] > *发送时间:* 2015年9月17日 15:54 > *收件人:* Huangguowei; Dev > *主题:* Re: bug in Worker.scala, ExecutorRunner is not serializable > > > > Did this cause an error for you? > > > > On Thu, Sep 17, 2015, 8:51 AM Huangguowei wrote: > > > > In Worker.scala line 480: > > > > case RequestWorkerState => > > sender ! WorkerStateResponse(host, port, workerId, > executors.values.toList, > > finishedExecutors.values.toList, drivers.values.toList, > > finishedDrivers.values.toList, activeMasterUrl, cores, memory, > > coresUsed, memoryUsed, activeMasterWebUiUrl) > > > > The executors’s type is: > > val executors = new HashMap[String, ExecutorRunner] > > > > but ExecutorRunner cannot be Serialized, so if ask RequestWorkerState will > cause java.io.NotSerializableException. > > > > > >
re: bug in Worker.scala, ExecutorRunner is not serializable
Is it possible to get Executors status when running an application? 发件人: Sean Owen [mailto:so...@cloudera.com] 发送时间: 2015年9月17日 15:54 收件人: Huangguowei; Dev 主题: Re: bug in Worker.scala, ExecutorRunner is not serializable Did this cause an error for you? On Thu, Sep 17, 2015, 8:51 AM Huangguowei mailto:huangguo...@huawei.com>> wrote: In Worker.scala line 480: case RequestWorkerState => sender ! WorkerStateResponse(host, port, workerId, executors.values.toList, finishedExecutors.values.toList, drivers.values.toList, finishedDrivers.values.toList, activeMasterUrl, cores, memory, coresUsed, memoryUsed, activeMasterWebUiUrl) The executors’s type is: val executors = new HashMap[String, ExecutorRunner] but ExecutorRunner cannot be Serialized, so if ask RequestWorkerState will cause java.io.NotSerializableException.
答复: bug in Worker.scala, ExecutorRunner is not serializable
Not error in normal case. But if I want to ask Worker through akkaUrl to get executors status, it will cause Exception. 发件人: Sean Owen [mailto:so...@cloudera.com] 发送时间: 2015年9月17日 15:54 收件人: Huangguowei; Dev 主题: Re: bug in Worker.scala, ExecutorRunner is not serializable Did this cause an error for you? On Thu, Sep 17, 2015, 8:51 AM Huangguowei mailto:huangguo...@huawei.com>> wrote: In Worker.scala line 480: case RequestWorkerState => sender ! WorkerStateResponse(host, port, workerId, executors.values.toList, finishedExecutors.values.toList, drivers.values.toList, finishedDrivers.values.toList, activeMasterUrl, cores, memory, coresUsed, memoryUsed, activeMasterWebUiUrl) The executors’s type is: val executors = new HashMap[String, ExecutorRunner] but ExecutorRunner cannot be Serialized, so if ask RequestWorkerState will cause java.io.NotSerializableException.
Re: bug in Worker.scala, ExecutorRunner is not serializable
Did this cause an error for you? On Thu, Sep 17, 2015, 8:51 AM Huangguowei wrote: > > > In Worker.scala line 480: > > > > case RequestWorkerState => > > sender ! WorkerStateResponse(host, port, workerId, > executors.values.toList, > > finishedExecutors.values.toList, drivers.values.toList, > > finishedDrivers.values.toList, activeMasterUrl, cores, memory, > > coresUsed, memoryUsed, activeMasterWebUiUrl) > > > > The executors’s type is: > > val executors = new HashMap[String, ExecutorRunner] > > > > but ExecutorRunner cannot be Serialized, so if ask RequestWorkerState will > cause java.io.NotSerializableException. > > > > >
bug in Worker.scala, ExecutorRunner is not serializable
In Worker.scala line 480: case RequestWorkerState => sender ! WorkerStateResponse(host, port, workerId, executors.values.toList, finishedExecutors.values.toList, drivers.values.toList, finishedDrivers.values.toList, activeMasterUrl, cores, memory, coresUsed, memoryUsed, activeMasterWebUiUrl) The executors's type is: val executors = new HashMap[String, ExecutorRunner] but ExecutorRunner cannot be Serialized, so if ask RequestWorkerState will cause java.io.NotSerializableException.