The reads are from a non-IGFS source, but the writes are through IGFS. Spark uses Hadoop's FileOutputCommitter to write the output to IGFS. I think what happens is essentially:
- During processing, temporary files are written by each of n executors running on different nodes to some /data/path/output/_temporary/part-n... - When the job completes, each of the executor performs the final "commit" by renaming the files under /data/path/output/_temporary/part-n... to /data/path/output/part-n... and deletes the _temporary directory. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/igfs-meta-behavior-when-node-restarts-tp13155p13322.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
