Re: any way to control memory usage when streaming input's speed is faster than the speed of handled by spark streaming ?

Unfortunately, there is no API support for this right now. You could implement it yourself by implementing your own receiver and controlling the rate at which objects are received. If you are using any of the standard receivers (Flume, Kafka, etc.), I recommended looking at the source code of the

Re: any way to control memory usage when streaming input's speed is faster than the speed of handled by spark streaming ?

Apologies for the premature send. Unfortunately, there is no API support for this right now. You could implement it yourself by implementing your own receiver and controlling the rate at which objects are received. If you are using any of the standard receivers (Flume, Kafka, etc.), I

any way to control memory usage when streaming input's speed is faster than the speed of handled by spark streaming ?

sparkers, Is there a better way to control memory usage when streaming input's speed is faster than the speed of handled by spark streaming ? Thanks, Francis.Hu