This does appear to be a significant missing feature. I'll try to make sure something easier gets in by the next release. See also below.
On Mon, Jan 8, 2024 at 11:30 AM Ferran Fernández Garrido <ffernandez....@gmail.com> wrote: > > Hi Yarden, > > Since it's a bounded source you could try with Sql transformation > grouping by the timestamp column. Here are some examples of grouping: > > https://github.com/apache/beam/tree/master/sdks/python/apache_beam/yaml > > However, if you want to add a timestamp column in addition to the > original CSV records then, there are multiple ways to achieve that. > > 1) MapToFields: > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/yaml_mapping.md > [Your timestamp column could be a callable to get the current > timestamp on each record] > > 2) If you need an extra layer of transformation complexity I would > recommend creating a custom transformation: > > # - type: MyCustomTransform > # name: AddDateTimeColumn > # config: > # prefix: 'whatever' > > providers: > - type: 'javaJar' > config: > jar: 'gs://path/of/the/java.jar' > transforms: > MyCustomTransform: 'beam:transform:org.apache.beam:javatransformation:v1' Alternatively you can use PyTransform, if you're more comfortable with that by invoking it via its fully qualified name. pipeline: transforms: ... - type: MyAssignTimestamps config: kwarg1: ... kwarg2: ... providers: type:python config: packages: ['py_py_package_identifier'] transforms: MyAssignTimestamps: fully_qualified_package.module.AssignTimestampsPTransform > Best, > Ferran > > El lun, 8 ene 2024 a las 19:53, Yarden BenMoshe (<yarde...@gmail.com>) > escribió: > > > > Hi all, > > Im quite new to using beam yaml. I am working with a CSV file and want to > > implement some windowing logic to it. > > Was wondering what is the right way to add timestamps to each element, > > assuming I have a column including a timestamp. > > > > I am aware of Beam Programming Guide (apache.org) part but not sure how > > this can be implemented and used from yaml prespective. > > > > Thanks > > Yarden
