Hi, I've got a case where the same structured query (it's union) gives 1 stage for a run and 5 stages for another. I could not find any pattern yet (and it's hard to reproduce it due to the volume and the application), but I'm pretty certain that it's *never* possible that Spark 2.3 could come up with 1 vs 5 stages for the very same query plan (even if I changed number of executors or number of cores or anything execution-related).
So my question is, is this possible that Spark SQL could give 1-stage execution plan and 5-stage execution plan for the very same query? (I am not saying that I'm 100% sure that the query is indeed the same since I'm working on a reproducible test case and only when I got it I'll really be). Sorry for the vague description, but I've got nothing more to share yet. Pozdrawiam, Jacek Laskowski ---- https://about.me/JacekLaskowski Mastering Spark SQL https://bit.ly/mastering-spark-sql Spark Structured Streaming https://bit.ly/spark-structured-streaming Mastering Kafka Streams https://bit.ly/mastering-kafka-streams Follow me at https://twitter.com/jaceklaskowski
