Re: Spark Streaming using Flume body size limitation

Hi, I think I found the problem. In SparkFlumeEvent the readExternal method use in.read(bodyBuff) which read the first 1020 bytes, but no more. The code should make sure to read everything. The following change will fix the problem: in.read(bodyBuff) to: in.readFully(bodyBuff) I attached a

Spark Streaming using Flume body size limitation

Hi, I am trying to send events to spark streaming via flume. It's working fine up to a certain point. I have problems when the size of the body is over 1020 characters. Basically, up to 1020 it works 1021 through 1024, the event will be accepted and there is no exception, but the channel seems