Structured streaming is stable in production! At Databricks, we and our customers collectively process almost 100s of billions of records per day using SS. However, we are not using kubernetes :)
Though I don't think it will matter too much as long as kubes are correctly provisioned+configured and you are checkpointing to HDFS (for fault-tolerance guarantees). TD On Fri, Apr 13, 2018, 12:28 AM Krishna Kalyan <krishnakaly...@gmail.com> wrote: > Hello All, > We were evaluating Spark Structured Streaming on Kubernetes (Running on > GCP). It would be awesome if the spark community could share their > experience around this. I would like to know more about you production > experience and the monitoring tools you are using. > > Since spark on kubernetes is a relatively new addition to spark, I was > wondering if structured streaming is stable in production. We were also > evaluating Apache Beam with Flink. > > Regards, > Krishna > > >
