+1 (binding) On Fri, Aug 2, 2024 at 10:43 PM Kaxil Naik <kaxiln...@gmail.com> wrote:
> Yup, I am fine removing that language to make it explicit but leave it up > to TP. > > On Fri, 2 Aug 2024 at 19:56, Daniel Standish > <daniel.stand...@astronomer.io.invalid> wrote: > > > My concern with the AIP is the talk of support for incremental data > > pipelines. In an incremental data pipeline, you don't think of a delta > > load (let's say a collection of updated rows) as a partition. A > partition > > in data is defined by a partition key, which should be an immutable field > > or fields in a record. You can't use an "updated at" field as a > partition > > key because then the same record can be in multiple partitions. And it > > doesn't make sense either when you think about what it would mean to > > "reprocess a partition" -- the rows that were in that partition now might > > not be there anymore. So I think this AIP needs to not brand itself as > any > > kind of solution for incremental loads. > > If your processing hive partitions (by time), and those data can be > > updated, you might need to reprocess the last N partitions each time. > > That's a common way to handle updates. (And maybe something that we > should > > consider supporting in this AIP.) If you're doing some kind of change > > tracking, you're just processing rows or new files, and it doesn't make > > sense to consider those a partition. > > My suggestion would be to remove the language talking about incremental > > loads from this AIP. > > >
