Hey Panos,
our solution allows us to analyze the full path and modify the file name.
so for multiple partitions, we can extract the values of the partitions and
then inject them into the file name.
for example,
for the following file:
Hey Stefan,
Thank you for your replay.
May I ask for a use-case or an example of how you would use this ability.
I want to make sure our solution would work for you.
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
May I ask how do you handle multiple partitions? Can't two files have the
same name with this approach, or am I missing something?
BR,
Panos
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
-
To unsubscribe
Yes, I think so.
Stefan Panayotov, PhD
spanayo...@outlook.com
spanayo...@comcast.net
spanayo...@gmail.com
-Original Message-
From: ilaimalka
Sent: Monday, June 8, 2020 9:17 AM
To: user@spark.apache.org
Subject: we control spark file names before we write them - should we
opensource
