lets say i have 2 dataframe jobs that write to /somedir/a=1 and somedir/a=2. these can run at same time without issues.
but now i get excited about dynamic partitioning. so i add "a" as a column to my 2 dataframes, set the option partitionOverwriteMode=dynamic, add partitionBy("a": _*) to the writing, and write both to /somedir. this works fine, and the result is the same, but i can no longer safely run both jobs at same time, because they both try to create and delete /somedir/_temporary, and i end up with exceptions about files not found in /somedir/_temporary. this makes me wonder why _temporary is hardcoded. why not _temporary_<uuid>?