Re: ETL with changing reference data

2017-09-17 Thread Chen Qin
as a Jira talk about retraction. >> In other cases, results are derived from long history which makes not >> rationale to keep. A side pipeline capture those events with late arriving >> event handling might interact with external storage and amend results. >> >> Thanks

Re: ETL with changing reference data

2017-09-14 Thread Fabian Hueske
e events with late arriving > event handling might interact with external storage and amend results. > > Thanks, > Chen > > > *From: *Peter Lappo <pe...@systematicmethods.com> > *Sent: *Sunday, September 10, 2017 3:00 PM > *To: *user@flink.apache.org > *Subject:

Re: ETL with changing reference data

2017-09-11 Thread Peter Lappo
eline capture those events with late arriving > event handling might interact with external storage and amend results. > > Thanks, > Chen > > From: Peter Lappo <mailto:pe...@systematicmethods.com> > Sent: Sunday, September 10, 2017 3:00 PM > To: user@flink.apache.org <

RE: ETL with changing reference data

2017-09-10 Thread qinnchen
, September 10, 2017 3:00 PM To: user@flink.apache.org Subject: ETL with changing reference data hi, We are building an ETL style application in Flink that consumes records from a file or a message bus as a DataStream. We are transforming records using SQL and UDFs. The UDF loads reference data

ETL with changing reference data

2017-09-10 Thread Peter Lappo
hi, We are building an ETL style application in Flink that consumes records from a file or a message bus as a DataStream. We are transforming records using SQL and UDFs. The UDF loads reference data in the open method and currently the data loaded remains in memory until the job is cancelled.