Hi,
I'm writing Java Beam code to run with both Dataflow and Spark. The input
files are tfrecord format and are from multiple directories. Java
TFRecordIO doesn't have readAll from list of files so what I'm doing is:
for (String dir: listOfDirs) {
p.apply(TFRecordIO.read().from(dir))
.apply(ParDo.of(new BatchElements()))
.apply(ParDo.of(new Process()))
.apply(Combine.globally(new CombineResult()))
.apply(TextIO.write().to(dir))
}
These directories are fairly independent and I only need result of each
directory. When running on Dataflow, processing of these directories happen
concurrently. But when running with Spark, I saw the spark jobs and stages
are sequential. It needs finish all steps in one directory before moving to
next one. What's the way to make multiple transforms run concurrently with
SparkRunner?
