Re: Max number of streams supported ?

Thanks Michael, TD for quick reply. It was helpful. I will let you know the numbers(limit) based on my experiments. On Wed, Jan 31, 2018 at 3:10 PM, Tathagata Das wrote: > Just to clarify a subtle difference between DStreams and Structured > Streaming. Multiple

Re: Max number of streams supported ?

Just to clarify a subtle difference between DStreams and Structured Streaming. Multiple input streams in a DStreamGraph is likely to mean they are all being processed/computed in the same way as there can be only one streaming query / context active in the StreamingContext. However, in the case of

Re: Max number of streams supported ?

-dev +user > Similarly for structured streaming, Would there be any limit on number of > of streaming sources I can have ? > There is no fundamental limit, but each stream will have a thread on the driver that is doing coordination of execution. We comfortably run 20+ streams on a single