Re: Stream conversion

2016-02-05 Thread Ufuk Celebi
> On 05 Feb 2016, at 08:56, Jeyhun Karimov wrote: > > For example, I will do aggregate operations with other windows (n-window > aggregations) that are already outputted. > I tried your suggestion and used filesystem sink, outputted to HDFS. > I got k files in HDFS

Re: Stream conversion

2016-02-04 Thread Stephan Ewen
Hi! If I understand you correctly, what you are looking for is a kind of periodic batch job, where the input data for each batch is a large window. We have actually thought about this kind of application before. It is not on the short term road map that we shared a few weeks ago, but I think it

Re: Stream conversion

2016-02-04 Thread Jeyhun Karimov
For example, I will do aggregate operations with other windows (n-window aggregations) that are already outputted. I tried your suggestion and used filesystem sink, outputted to HDFS. I got k files in HDFS directory where k is the number of parallelism (I used single machine). These files get

Re: Stream conversion

2016-02-04 Thread Robert Metzger
I'm wondering which kind of transformations you want to apply to the window you cannot apply with the DataStream API? Would it be sufficient for you to have the windows as files in HDFS and then run batch jobs against the windows on disk? If so, you could use our filesystem sink, which creates

Re: Stream conversion

2016-02-04 Thread Matthias J. Sax
Hi Sane, Currently, DataSet and DataStream API a strictly separated. Thus, this is not possible at the moment. What kind of operation do you want to perform on the data of a window? Why do you want to convert the data into a data set? -Matthias On 02/04/2016 10:11 AM, Sane Lee wrote: > Dear

Re: Stream conversion

2016-02-04 Thread Jeyhun Karimov
Hi Matthias, This need not to be necessarily in api functions. I just want to get a roadmap to add this functionality. Should I save each window's data into disk and create a new dataset environment in parallel? Or change trigger functionality maybe? I have large windows. As I asked in previous

Re: Stream conversion

2016-02-04 Thread Sane Lee
I have also, similar scenario. Any suggestion would be appreciated. On Thu, Feb 4, 2016 at 10:29 AM Jeyhun Karimov wrote: > Hi Matthias, > > This need not to be necessarily in api functions. I just want to get a > roadmap to add this functionality. Should I save each