Re: Data lost in spark streaming

Maybe the driver got restarted. See the log4j logs of the driver before it restarted. On Thu, Sep 10, 2015 at 11:32 PM, Bin Wang wrote: > I'm using spark streaming 1.4.0 and have a DStream that have all the data > it received. But today the history data in the DStream seems to

Re: Data lost in spark streaming

There is some error logs in the executor and I don't know if it is related: 15/09/11 10:54:05 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$Inv alidToken): Invalid AMRMToken from

Re: Data lost in spark streaming

Can you retrieve log for appattempt_1440495451668_0258_01 and see if there is some clue there ? Cheers On Sun, Sep 13, 2015 at 3:28 AM, Bin Wang wrote: > There is some error logs in the executor and I don't know if it is related: > > 15/09/11 10:54:05 WARN ipc.Client:

Data lost in spark streaming

I'm using spark streaming 1.4.0 and have a DStream that have all the data it received. But today the history data in the DStream seems to be lost suddenly. And the application UI also lost the streaming process time and all the related data. Could any give some hint to debug this? Thanks.