Hi Kevin, On Thu, May 22, 2014 at 9:49 AM, Kevin Markey <kevin.mar...@oracle.com> wrote: > The FS closed exception only effects the cleanup of the staging directory, > not the final success or failure. I've not yet tested the effect of > changing my application's initialization, use, or closing of FileSystem.
Without going and reading more of the Spark code, if your app is explicitly close()'ing the FileSystem instance, it may be causing the exception. If Spark is caching the FileSystem instance, your app is probably closing that same instance (which it got from the HDFS library's internal cache). It would be nice if you could test that theory; it might be worth knowing that's the case so that we can tell people not to do that. -- Marcelo
