razvanculea commented on issue #32113:
URL: https://github.com/apache/beam/issues/32113#issuecomment-3164000481
@damccorm pre my IObase work, all IOs derived from IObased and
FileBasedSink don't work in streaming (i should have logged an issue). They do
work in batch.
My PR to add streaming support:
PR - [(apache/beam#35137)] adds support in IObase (fyi, most of the
implementation needs to be in FileBasedSink for most used IOs :
Text,Avro,Parquet,TFRecords)
PR - [(apache/beam#35253)] adds surpport in FileBasedSink and TextIO
PR - [(apache/beam#35418)] , i added support for all the others IOs using
IObase and FileBasedSink in apache_beam : AvroIO, ParquetIO, TFRecordsIO
Your fist 2 tests are ok for batch, they would fail in streaming (with my
PRs as well) because window is Global and no triggering frequency specified in
the Write.
Your last test looks correct and works with my PRs, code (in the attached
achive as test_periodimpulse.py)
See job id in my internal test project
2025-08-07_05_14_41-9611060739629941282
```
#Danny McCormick test
elements =(
p
| "Create elements" >>
beam.transforms.periodicsequence.PeriodicImpulse(
start_timestamp=1,
stop_timestamp=100,
fire_interval=10,
apply_windowing=False )
| 'ApplyWindowing' >> beam.WindowInto(FixedWindows(20))
| beam.io.WriteToText("./test")
)
```
I think i'll add thses examples under
sdks/python/apache_beam/examples/unbouded_sinks/
[unbounded_sinks.zip](https://github.com/user-attachments/files/21665148/unbounded_sinks.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@beam.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
