[jira] [Created] (FLINK-10201) The batchTestUtil was mistakenly used in some stream sql tests

2018-08-22 Thread Xingcan Cui (JIRA)
Xingcan Cui created FLINK-10201: --- Summary: The batchTestUtil was mistakenly used in some stream sql tests Key: FLINK-10201 URL: https://issues.apache.org/jira/browse/FLINK-10201 Project: Flink

Re: [Proposal] Utilities for reading, transforming and creating Streaming savepoints

2018-08-22 Thread Hequn Cheng
Hi Gyula, +1 for this feature. State bootstrapping and state analytics will be very helpful. On Thu, Aug 23, 2018 at 4:09 AM Gyula Fóra wrote: > Hi Shuyi, > > The tool allows you to convert a Flink DataSet containing the individual > state rows, (key, state) pairs, into a state for a streaming

Re: [Proposal] Utilities for reading, transforming and creating Streaming savepoints

2018-08-22 Thread Gyula Fóra
Hi Shuyi, The tool allows you to convert a Flink DataSet containing the individual state rows, (key, state) pairs, into a state for a streaming operator. So if you want to bootstrap your state from data on HDFS, you would read the file in Flink, convert it to the required DataSet format, then

Re: [Proposal] Utilities for reading, transforming and creating Streaming savepoints

2018-08-22 Thread Shuyi Chen
+1 on the tooling. Also, you mentioned about state bootstrapping problem. Could you please elaborate on how we can leverage the tooling to solve state bootstrapping? I think this is a common problem to stream processing, and it will be great the community can work on it. Thanks. Shuyi On Wed,

Re: [Proposal] Utilities for reading, transforming and creating Streaming savepoints

2018-08-22 Thread Gyula Fóra
Thanks, I guess the first thing that would be great help from anyone interested in helping is to try it for some streaming state :) We have tested these tools at King to analyze, transform and perform some aggregations on our user-states. The major limitation is that it requires RocksDB

[jira] [Created] (FLINK-10200) Consolidate FileBasedStateOutputStream and FsCheckpointStateOutputStream

2018-08-22 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-10200: -- Summary: Consolidate FileBasedStateOutputStream and FsCheckpointStateOutputStream Key: FLINK-10200 URL: https://issues.apache.org/jira/browse/FLINK-10200

[jira] [Created] (FLINK-10199) StandaloneSessionClusterEntrypoint does not respect host/ui-port given from jobmanager.sh

2018-08-22 Thread Dawid Wysakowicz (JIRA)
Dawid Wysakowicz created FLINK-10199: Summary: StandaloneSessionClusterEntrypoint does not respect host/ui-port given from jobmanager.sh Key: FLINK-10199 URL: https://issues.apache.org/jira/browse/FLINK-10199

Re: [DISCUSS][TABLE] How to handle empty delete for UpsertSource

2018-08-22 Thread Hequn Cheng
Hi all, Thanks a lot for your replies, @Piotr Nowojski @Fabian Hueske @Xingcan Cui Option 3 will send empty deletes to downstream operators, but downstream operators seems don't know how to deal with it. Assume that upsert source forward deletes by default. Let's see what problems we have.

Re: [DISCUSS][TABLE] How to handle empty delete for UpsertSource

2018-08-22 Thread Piotr Nowojski
For the record I was also in favour of option 3. regarding empty deletes. About stateful Filters on upsert stream. Hequn, I don’t fully understand this: > However, if UpsertSource can also output empty deletes, these > empty deletes will be more difficult to control. We don't know where these >

[jira] [Created] (FLINK-10197) flink CEP error when use sql

2018-08-22 Thread sean.miao (JIRA)
sean.miao created FLINK-10197: - Summary: flink CEP error when use sql Key: FLINK-10197 URL: https://issues.apache.org/jira/browse/FLINK-10197 Project: Flink Issue Type: Bug Components:

[jira] [Created] (FLINK-10196) API to test streaming pipelines with windows

2018-08-22 Thread Nicos Maris (JIRA)
Nicos Maris created FLINK-10196: --- Summary: API to test streaming pipelines with windows Key: FLINK-10196 URL: https://issues.apache.org/jira/browse/FLINK-10196 Project: Flink Issue Type: New