This seems like a very expensive operation. Why do you want to write out all the exploded values? If you just want all combinations of values, could you instead do it at read-time with a UDF or something?
On Sat, Aug 1, 2020 at 8:34 PM hesouol <heso...@gmail.com> wrote: > I forgot to add an information. By "can't write" I mean it keeps processing > and nothing happens. The job runs for hours even with a very small file and > I have to force the stoppage. > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > -- *Patrick McCarthy * Senior Data Scientist, Machine Learning Engineering Dstillery 470 Park Ave South, 17th Floor, NYC 10016
