strange behavior of spark 2.1.0
Hello, Guys
I am running the spark streaming in 2.1.0, the scala version is tried on
2.11.7 and 2.11.4. And it is consuming from JMS. Recently, I have get the
following error
*"ERROR scheduler.ReceiverTracker: Deregistered receiver for stream 0:
Stopped by driver"*
*This error can be occurred randomly, it might be couple hours or couple
days. besides this error, everything is perfect.*
When the error happens, my job is stopped completely. There is no any other
error can be found.
I am running on top of yarn, and tried to look up the error through yarn
logs, container, no any further information appears there. The job is just
stopped from driver gracefully. BTW I have customized receiver, I either do
not think it is happened from receiver, there is no any error exception
from receiver, and I can also track the stop command is sent from "onStop"
function in receiver.
FYI, the driver is not consuming any large memory, there is no any RDD
"collect" command in the driver. I have also checked container log for each
executor, and cannot find any further error.
The following is my conf for the spark context
val conf = new SparkConf().setAppName(jobName).setMaster(master)
.set("spark.hadoop.validateOutputSpecs", "false")
.set("spark.driver.allowMultipleContexts", "true")
.set("spark.streaming.receiver.maxRate", "500")
.set("spark.streaming.backpressure.enabled", "true")
.set("spark.streaming.stopGracefullyOnShutdown", "true")
.set("spark.eventLog.enabled", "true");
If you have any idea or suggestion, please let me know. Appreciate on the
solution.
Thank you so much
Re: strange behavior of spark 2.1.0
Thank you for replying.
Actually there is no any message coming during the exception. And there is no
OOME in any executor. What I am suspecting it might be caused by AWL.
> On Apr 2, 2017, at 5:22 AM, Timur Shenkao wrote:
>
> Hello,
> It's difficult to tell without details.
> I believe one of the executors dies because of OOM or some Runtime Exception
> (some unforeseen dirty data row).
> Less probable is GC stop-the-world pause when incoming message rate increases
> drastically.
>
>
>> On Saturday, April 1, 2017, Jiang Jacky wrote:
>> Hello, Guys
>> I am running the spark streaming in 2.1.0, the scala version is tried on
>> 2.11.7 and 2.11.4. And it is consuming from JMS. Recently, I have get the
>> following error
>> "ERROR scheduler.ReceiverTracker: Deregistered receiver for stream 0:
>> Stopped by driver"
>>
>> This error can be occurred randomly, it might be couple hours or couple
>> days. besides this error, everything is perfect.
>> When the error happens, my job is stopped completely. There is no any other
>> error can be found.
>> I am running on top of yarn, and tried to look up the error through yarn
>> logs, container, no any further information appears there. The job is just
>> stopped from driver gracefully. BTW I have customized receiver, I either do
>> not think it is happened from receiver, there is no any error exception from
>> receiver, and I can also track the stop command is sent from "onStop"
>> function in receiver.
>>
>> FYI, the driver is not consuming any large memory, there is no any RDD
>> "collect" command in the driver. I have also checked container log for each
>> executor, and cannot find any further error.
>>
>>
>>
>>
>> The following is my conf for the spark context
>> val conf = new SparkConf().setAppName(jobName).setMaster(master)
>> .set("spark.hadoop.validateOutputSpecs", "false")
>> .set("spark.driver.allowMultipleContexts", "true")
>> .set("spark.streaming.receiver.maxRate", "500")
>> .set("spark.streaming.backpressure.enabled", "true")
>> .set("spark.streaming.stopGracefullyOnShutdown", "true")
>> .set("spark.eventLog.enabled", "true");
>>
>> If you have any idea or suggestion, please let me know. Appreciate on the
>> solution.
>>
>> Thank you so much
>>
Re: convert JavaRDD> to JavaRDD
There is no flattop just flatMap > On Apr 5, 2017, at 12:24 PM, Vinay Parekar wrote: > > I think flattop() will be helpful in this case . Correct me if I am wrong. > > From: Hamza HACHANI > Date: Wednesday, April 5, 2017 at 3:43 AM > To: "user@spark.apache.org" > Subject: convert JavaRDD> to JavaRDD > > I want to convert a JavaRDD> to JavaRDD. For example if > there is 3 elment in List 3 Object would be created in my new > JavaRDD. > > Does any one have an idea ? > > > > CONFIDENTIALITY NOTICE: This message is intended only for the use and review > of the individual or entity to which it is addressed and may contain > information that is privileged and confidential. If the reader of this > message is not the intended recipient, or the employee or agent responsible > for delivering the message solely to the intended recipient, you are hereby > notified that any dissemination, distribution or copying of this > communication is strictly prohibited. If you have received this communication > in error, please notify sender immediately by telephone or return email. > Thank you.
