That should not be the case. See https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#using-foreach-and-foreachbatch Maybe you are calling .foreach on some Scala object inadvertently.
On Tue, Mar 9, 2021 at 4:41 PM Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Hi, > > When I use *foreachBatch *is Spark structured streaming, yarn mode works > fine. > > When one switches to *foreach* mode (row by row processing), this > effectively runs in local mode on a single JVM. It seems to crash when > running in a distributed mode. That is my experience. > > Can someone else please verify this independently? > > Cheers > > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > >
