Hi Niels,
I assume the directoryName you are passing doesn't have the file system
prefix (hdfs:// or s3://, ...) specified.
In those cases, Path.getFileSystem() is looking up the default file system
prefix from the configuration.
Probably the environment where you are submitting the job from doesn't have
the correct Flink configuration.
How are you starting the Flink job? Does the conf/flink-conf.yaml have the
correct hdfs configuration?
Regards,
Robert
On Thu, Aug 18, 2016 at 2:02 PM, Niels Basjes <ni...@basjes.nl> wrote:
> Hi,
>
> I have a batch job that I run on yarn that creates files in HDFS.
> I want to avoid running this job at all if the output already exists.
>
> So in my code (before submitting the job into yarn-session) I do this:
>
> String directory = "foo";
>
> Path directory = new Path(directoryName); FileSystem fs =
> directory.getFileSystem();
> if (!fs.exists(directory)) {
>
> // run the job
>
> }
>
> What I found is that this code apparently checks the 'wrong' file system. (I
> always get 'false' even if it exists in hdfs)
>
> I checked the API of the execution environment yet I was unable to get the
> 'correct' filesystem from there.
>
> What is the proper way to check this?
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>
