Is there a way how to change the default storage level? If not, how can I properly change the storage level wherever necessary, if my input and intermediate results do not fit into memory?
In this example: context.wholeTextFiles(...) .flatMap(s -> ...) .flatMap(s -> ...) Does persist() need to be called after every transformation? context.wholeTextFiles(...) .persist(StorageLevel.MEMORY_AND_DISK) .flatMap(s -> ...) .persist(StorageLevel.MEMORY_AND_DISK) .flatMap(s -> ...) .persist(StorageLevel.MEMORY_AND_DISK) Thanks!