TextIO - writing to directory, possible problem

[Borisa Zivkovic]

Hi everyone,

lost some time figuring out why TextIO is not writing data for me.

I have this:

stringsPCollection.apply(Window.<String>into(FixedWindows.of(Duration.standardSeconds(5))))
        .apply("WritingToOutput",
TextIO.write().withWindowedWrites().withFilenamePolicy(new
WindowedFilenamePolicy("my_pref")).to("/tmp/").withNumShards(1));

Where WindowedFilenamePolicy is the one found
in org.apache.beam.sdk.io.AvroIOTest

I expected output files to be written to /tmp/ directory but they are not.
They are written to root directory ... Is this expected behaviour?

I did some analysis and here are few findings:

I think the problem
is org.apache.beam.sdk.io.FileBasedSink.ExtractDirectory ...

The following small main method shows the problem:

import org.apache.beam.sdk.io.FileBasedSink;
import org.apache.beam.sdk.io.fs.ResourceId;
import org.apache.beam.sdk.options.ValueProvider;
import org.apache.beam.sdk.options.ValueProvider.StaticValueProvider;

public class Test {

  public static void main(String[] args) {
    ResourceId res = FileBasedSink.convertToFileResourceIfPossible("/tmp/");
    System.out.println("Resource is " + res + ", current directory is " +
res.getCurrentDirectory() + ", filename is " + res.getFilename());
    FileBasedSink<String> mockFBS = new
FileBasedSink<String>(StaticValueProvider.of(res), null) {

      @Override
      public org.apache.beam.sdk.io.FileBasedSink.WriteOperation<String>
createWriteOperation() {
        return null;
      }
    };

    final ValueProvider<ResourceId> provider =
mockFBS.getBaseOutputDirectoryProvider();

    System.out.println("BaseOutputProvider is " + provider + ",
isAccessible=" + provider.isAccessible() + ", getValue=" + provider.get());
  }

}

The output is

Resource is /tmp, current directory is //, filename is tmp
BaseOutputProvider is
NestedValueProvider{value=StaticValueProvider{value=/tmp}},
isAccessible=true, getValue=//

But I think this is wrong - or at least confusing.

Thoughts?

cheers
Borisa