Dear developers.
I would like to ask question regarding to artifacts directory for job
server used in portable runner.
Recently, I had a chat with other beam user and just realized that
"--artifacts-directory" supports both local filesystem path and gcs
path .
How does beam handle both local filesystem and gcs for artifacts directory ?
In "FlinkJobServerDriver.java", it uses GcsOptions for implementing
PipelineOptions interface so I thought gcs bucket is the only option
but its not.
---------------------------------------------------------------------
// TODO: Expose the fileSystem related options.
PipelineOptions options = PipelineOptionsFactory.create();
// Limiting gcs upload buffer to reduce memory usage while doing
parallel artifact uploads.
options.as(GcsOptions.class).setGcsUploadBufferSizeBytes(1024 * 1024);
// Register standard file systems.
FileSystems.setDefaultPipelineOptions(options);
fromParams(args).run();
---------------------------------------------------------------------
I appreciate if I could get some clarification with my question.
Best Regards,
Yu Watanabe
--
Yu Watanabe
[email protected]
