Re: ETL with changing reference data

2017-09-11 Thread Peter Lappo
eline capture those events with late arriving > event handling might interact with external storage and amend results. > > Thanks, > Chen > > From: Peter Lappo <mailto:pe...@systematicmethods.com> > Sent: Sunday, September 10, 2017 3:00 PM > To: user@flink.apache.org <

ETL with changing reference data

2017-09-10 Thread Peter Lappo
hi, We are building an ETL style application in Flink that consumes records from a file or a message bus as a DataStream. We are transforming records using SQL and UDFs. The UDF loads reference data in the open method and currently the data loaded remains in memory until the job is cancelled. Th