subject:"\[Spark streaming\]\: Microbatch id in logs"

Re: [Spark streaming]: Microbatch id in logs

2023-06-26 Thread Mich Talebzadeh

In SSS writeStream. \ outputMode('append'). \ option("truncate", "false"). \ * foreachBatch(SendToBigQuery). \* option('checkpointLocation', checkpoint_path). \ so this writeStream will call

[Spark streaming]: Microbatch id in logs

2023-06-25 Thread Anil Dasari

Hi, I am using spark 3.3.1 distribution and spark stream in my application. Is there a way to add a microbatch id to all logs generated by spark and spark applications ? Thanks.