Spark batch job chaining

Hi, I want to write a batch job which would call another batch job based on condition. Can I call one batch job through another in scala or I can do it just by python script. Example would be really helpful. Thanks Amit

[SPARK-STRUCTURED-STREAMING] IllegalStateException: Race while writing batch 4

Hi, I have 2spark structure streaming queries writing to the same outpath in object storage. Once in a while I am getting the "IllegalStateException: Race while writing batch 4". I found that this error is because there are two writers writing to the output path. The file streaming sink doesn't su