Hi Hans, Thanks for the quick reply! The "Supported Engines" box in the documentation is much appreciated.
I had read about the SINGLE_BEAM option but forgotten about it since. I just tried it on the "Avro file output" and the "Serialize to file" transforms, but still get empty files on GCS when running with BeamDirect. I configured the SINGLE_BEAM option via the "Specify copies" on the output, is that correct? When SINGLE_BEAM is configured, it looks like the transform is being ignored by the local runner? cheers Fabian > Am 25.08.2022 um 11:43 schrieb Hans Van Akelyen <[email protected]>: > > Hi Fabian, > > Did you try running those transforms with the "SINGLE_BEAM" option in the > number of copies? (for more info here > <https://hop.apache.org/manual/latest/pipeline/beam/getting-started-with-beam.html#_all_others> > in the Non-Beam output transforms section) > > This being said, we are working on getting this tested on all runners, I have > written textfiles on Flink in the past and that worked but DataFlow is > another beast and might produce other results. To improve transparency we are > adding indicators in the next version of our docs on each and every transform > on what we have tested (you can already see this by switching to our > pre-release docs example > <https://hop.apache.org/manual/next/pipeline/transforms/dummy.html>). These > docs are still very much alive so expect changes in these flags up until > release. > > For the Avro and Parquet transforms we can implement the Beam equivalent in > the backend so they should definitely start working once that work is done > (tickets HOP-4168 and HOP-4169). > > Once we have tested everything the plan is to include an advisory or warnings > in the application that some transforms do not work on the specified engine. > > Cheers, > Hans > > On Thu, 25 Aug 2022 at 11:00, Fabian Peters <[email protected] > <mailto:[email protected]>> wrote: > Hi all, > > During development I used the "Serialize to file" output to share data among > pipelines > <https://hop.apache.org/manual/latest/best-practices/index.html#_size_matters>. > Unfortunately that transform only creates empty files when running on Beam, > as do the Parquet <https://issues.apache.org/jira/browse/HOP-3557>, Avro and > Text file outputs. The Beam output on the other hand only works on Beam. > > Is there any output that works with the local runner and Beam/Dataflow? > > cheers > > Fabian
