In trying to sort some largish datasets, we came across the
spark.shuffle.consolidateFiles property, and I found in the source code
that it is set, by default, to false, with a note to default it to true
when the feature is stable.
Does anyone know what is unstable about this? If we set it true, what
problems should we anticipate?
Thanks,
-Nathan Kronenfeld
--
Nathan Kronenfeld
Senior Visualization Developer
Oculus Info Inc
2 Berkeley Street, Suite 600,
Toronto, Ontario M5A 4J5
Phone: +1-416-203-3003 x 238
Email: [email protected]
