Hi Fabian, Did you try running those transforms with the "SINGLE_BEAM" option in the number of copies? (for more info here <https://hop.apache.org/manual/latest/pipeline/beam/getting-started-with-beam.html#_all_others> in the Non-Beam output transforms section)
This being said, we are working on getting this tested on all runners, I have written textfiles on Flink in the past and that worked but DataFlow is another beast and might produce other results. To improve transparency we are adding indicators in the next version of our docs on each and every transform on what we have tested (you can already see this by switching to our pre-release docs example <https://hop.apache.org/manual/next/pipeline/transforms/dummy.html>). These docs are still very much alive so expect changes in these flags up until release. For the Avro and Parquet transforms we can implement the Beam equivalent in the backend so they should definitely start working once that work is done (tickets HOP-4168 and HOP-4169). Once we have tested everything the plan is to include an advisory or warnings in the application that some transforms do not work on the specified engine. Cheers, Hans On Thu, 25 Aug 2022 at 11:00, Fabian Peters <[email protected]> wrote: > Hi all, > > During development I used the "Serialize to file" output to share data > among pipelines > <https://hop.apache.org/manual/latest/best-practices/index.html#_size_matters>. > Unfortunately that transform only creates empty files when running on Beam, > as do the Parquet <https://issues.apache.org/jira/browse/HOP-3557>, Avro > and Text file outputs. The Beam output on the other hand only works on Beam. > > Is there any output that works with the local runner and Beam/Dataflow? > > cheers > > Fabian >
