Re: [DISCUSS] Create a Flink ecosystem website

Thanks for the update, Robert. Looking forward to the website. If there is already a list of software we need to run the website, we can ask Apache infra team to prepare the VM for us, as that may also take some time. On Wed, Apr 24, 2019 at 11:57 PM Robert Metzger wrote: > Hey all, > > quick

Re: Flink Stream SQL group by TUMBLE(rowtime,)

您好,请问 【没有offset从earliest开始，有则从offset开始】这个是怎么实现的啊 On 4/24/2019 16:46，邵志鹏 wrote： 大家好，问题求助： 事件时间使用EventTime，默认Checkpoint【没有offset从earliest开始，有则从offset开始】。 assignTimestampsAndWatermarks，1、不使用Watermark，即默认eventtime时间戳；2、使用官方max取最大值；两种情况问题基本相同： 问题描述：

Re: QueryableState startup regression in 1.8.0 ( migration from 1.7.2 )

You could try to set queryable-state.enable to true. And check again. Vishal Santoshi 于2019年4月25日 周四上午1:40写道： > Any one ? > > On Wed, Apr 24, 2019 at 12:02 PM Vishal Santoshi < > vishal.santo...@gmail.com> wrote: > >> Hello folks, >> >> Following >>

Job Startup Arguments

Hello! Is there a way (via the REST API) to see the parameters used to start a job? -Steve

Re: AskTimeoutException

I found the issue was a hard-coded timeout value in MiniCluster class, which is used for stand alone execution: public MiniCluster(MiniClusterConfiguration miniClusterConfiguration) { this.miniClusterConfiguration = checkNotNull(miniClusterConfiguration, "config may not

Re: Watermark for each key?

Thanks Till What about this workaround. If I after the watermark assignment split the stream in elements that fits in the watermark (s1) and those that don’t (s2). The s1 I process with the table api with a window aggregate using watermark and s2 I handle with an unbounded non-windows

Re: QueryableState startup regression in 1.8.0 ( migration from 1.7.2 )

Any one ? On Wed, Apr 24, 2019 at 12:02 PM Vishal Santoshi wrote: > Hello folks, > > Following > https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html#querying-state > . > for setting up the Queryable Server and proxy, I have my classpath ( the >

Re: AskTimeoutException

Thanks Abdul for the help. So I added this: cfg.setString(AkkaOptions.LOOKUP_TIMEOUT, "2 min"); But I am still I am getting the same error: Caused by: akka.pattern.AskTimeoutException: Ask timed out on

Re: Apache Flink - Question about dynamically changing window end time at run time

Hi Mans, Sameer is correct. if you would like to control window triggering based on other elements that does not belong to this window (in a keyed stream context) then this is probably the best way to approach. I think you've also posted in another thread that describes what will be left after

Re: Fast restart of a job with a large state

Hi Till, Thanks for the info! It's good to know. Regards, Sergey On Wed, Apr 24, 2019, 13:08 Till Rohrmann wrote: > Hi Sergey, > > at the moment neither local nor incremental savepoints are supported in > Flink afaik. There were some ideas wrt incremental savepoints floating > around in the

Re: Watermark for each key?

Hi Lasse, at the moment this is not supported out of the box by Flink. The community thought about this feature but so far did not implement it. Unfortunately, I'm also not aware of an easy workaround one could do in the user code space. Cheers, Till On Wed, Apr 24, 2019 at 3:26 PM Lasse

QueryableState startup regression in 1.8.0 ( migration from 1.7.2 )

Hello folks, Following https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html#querying-state . for setting up the Queryable Server and proxy, I have my classpath ( the lib directory ) that has the required jar, But I do not see the mentioned log and

Re: [DISCUSS] Create a Flink ecosystem website

Hey all, quick update on this project: The frontend and backend code have been put together into this repository: https://github.com/sorahn/flink-ecosystem We also just agreed on an API specification, and will now work on finishing the backend. It will probably take a few more weeks for this to

Re: Flink CLI

Hi Steven, As much as I am aware, 1) no update call. our build flow feels a little weird to us as well. definitely requires scripting. 2) we are using Flink management API remotely in our build flow to 1) get jobs, 2) savepoint them, 3) cancel them etc. I am going to release a Python script for

Re: Flink CLI

Hi Steve, I recently solved this problem using the REST api and some python scripts. The script has a function "upgrade_job" which will cancel with savepoint, optionally upload a new jar from the local filestystem or S3, and start the job from the savepoint including any changes in parallelism.

Re: No zero ( 2 ) exit code on k8s StandaloneJobClusterEntryPoint when save point with cancel...

Good to hear. Could you create a documentation JIRA issue for this problem? Thanks a lot. Cheers, Till On Wed, Apr 24, 2019 at 4:58 PM Vishal Santoshi wrote: > Verified, I think we just need to make sure that it is documented :) > > On Wed, Apr 24, 2019 at 9:47 AM Vishal Santoshi > wrote: >

Flink CLI

Hello! I am working on automating our deployments to our Flink cluster. I had a couple questions about the flink cli. 1) I thought there was an "update" command that would internally manage the cancel with savepoint, upload new jar, restart from savepoint process. 2) Is there a way to get the

Re: No zero ( 2 ) exit code on k8s StandaloneJobClusterEntryPoint when save point with cancel...

Verified, I think we just need to make sure that it is documented :) On Wed, Apr 24, 2019 at 9:47 AM Vishal Santoshi wrote: > This makes total sense and actually is smart ( defensive ). Will test and > report. I think though that this needs to be documented :) > > On Wed, Apr 24, 2019 at 6:03

Re: No zero ( 2 ) exit code on k8s StandaloneJobClusterEntryPoint when save point with cancel...

This makes total sense and actually is smart ( defensive ). Will test and report. I think though that this needs to be documented :) On Wed, Apr 24, 2019 at 6:03 AM Till Rohrmann wrote: > Hi Vishal, > > it seems that the following is happening: You triggered the cancel with > savepoint command

Re: [DISCUSS] Temporarily remove support for job rescaling via CLI action "modify"

The idea is to also remove the rescaling code in the JobMaster. This will make it easier to remove the ExecutionGraph reference from the JobMaster which is needed for the scheduling rework [1]. [1] https://issues.apache.org/jira/browse/FLINK-12231 On Wed, Apr 24, 2019 at 12:14 PM Shuai Xu

Watermark for each key?

Hi. We work with IoT data and we have cases where the IoT-device delay data transfer if it can't get network access. We would like to use table windows aggregate function over each device to calculate some statistics, but for windows aggregate functions to work we need to assign a watermark. This

RE: Flink 状态使用问题咨询

主要是考虑是在从异常恢复场景下，业务state是否需要重新加载。如果不需要重新加载，就不要记录这么多时间用来判断了。 Sent from Mail for Windows 10 From: zhang yue Sent: Wednesday, April 24, 2019 8:29:07 PM To: user-zh@flink.apache.org Subject: Re: Flink 状态使用问题咨询

Re: [EXTERNAL] Re: Looking for help in configuring Swift as State Backend

I think you also need to specify a path for the checkpoint directory. Try to set state.checkpoints.dir: swift://spout-checkpoints.magellan/flink/checkpoints Cheers, Till On Wed, Apr 24, 2019 at 2:58 PM PoolakkalMukkath, Shakir < shakir_poolakkalmukk...@comcast.com> wrote: > Hi Till, Thanks for

Re: [EXTERNAL] Re: Looking for help in configuring Swift as State Backend

Hi Till, Thanks for the response. Yes, I looks at the document. But still trying to figure out Let me summaries my config and what I did 1. Copied flink-swift-fs-hadoop-1.6.2.jar to lib 2. flink-conf.yaml #==

Re: May be useful: our reference document for "Understanding State in Flink"

Thank you all! @David and @Fabian can guide me (or Deepak as well) to maintain this document if they'd like. I can export HTML from this that we can easily play with and put in docs. --- Oytun Tez *M O T A W O R D* The World's Fastest Human Translation Platform. oy...@motaword.com —

Re: Flink 状态使用问题咨询

嗯，明白你的意思，initTime < openTime是做何考虑，什么情况下initTime < openTime会满足 > 在 2019年4月24日，下午8:16，Shi Quan 写道： > > initTime < openTime

RE: Flink 状态使用问题咨询

有做过类似的事情，不用侵入flink的源码。记录几个关键信息： 1. Function open的时间，openTime； 2. 数据初始化的时间，initTime，可以用State保存； 3. 真正的业务State 当有数据过来时，iff (null == initTime) || (initTime < openTime) 进行初始数据加载动作。 Sent from Mail for Windows 10

Re: Missing state in RocksDB checkpoints

Till, Thank you for escalating this to blocker. I agree that data loss is always a serious issue. For reference, the workaround is to unchain the stateful operators. To make the new job be able to recover from previous checkpoint, we also had to change the UID of the operator that was missing

Re: Apache Flink - Question about dynamically changing window end time at run time

Global Windows is fine for this use case. I have used the same strategy. You just define custom evictors and triggers and you are all good. Windows are managed by keys, so as such as long as events are evicted from the window, that counts towards reclaiming memory for the key+window combination.

Re: Flink Control Stream

Hi Dominik, I think it is not possible to use Flink's AsyncFunction together with a ConnectedStream or when you use BroadcastState. Therefore, it would be necessary that you inject the control messages into your normal stream and then filter them out in the AsyncFunction#asyncInvoke call.

Re: HA lock nodes, Checkpoints, and JobGraphs not being removed after failure

Cross linking the dev ML thread [1]. Let us continue the discussion there. [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/HA-lock-nodes-Checkpoints-and-JobGraphs-after-failure-td28432.html Cheers, Till On Tue, Apr 23, 2019 at 9:52 AM dyana.rose wrote: > originally posted

Re: Apache Flink - Question about dynamically changing window end time at run time

Hi Rong: Thanks for your answer. >From what I understand the dynamic gap session windows are also created when >the event is encountered.  I need to be able to change the window end time at >a later time based on what other events are in that window.  One way to do >this is to use

Re: State Migration with RocksDB MapState

Hi Gordon, I noticed there has been no movement on this issue and I'm wondering if I can find some way to work around this. My MapState value is a case class container of Avro-generated SpecificRecords. If one SpecificRecord changes I am stuck. >From the issue It seems like the blocker is around

Re: May be useful: our reference document for "Understanding State in Flink"

I want to volunteer for maintaining or adding to this kind of document. Please do let me know if i can. Thanks Deepak On Wed, Apr 24, 2019 at 6:33 AM Deepak Sharma wrote: > > > On Wed, Apr 24, 2019 at 5:14 AM Till Rohrmann > wrote: > >> Thanks for sharing this resource with the community

Re: May be useful: our reference document for "Understanding State in Flink"

On Wed, Apr 24, 2019 at 5:14 AM Till Rohrmann wrote: > Thanks for sharing this resource with the community Oytun. It looks really > helpful. > > I'm pulling in David and Fabian who work a lot on documentation. Maybe > it's interesting for them to take a look at. The community had once the > idea

Re: Flink 状态使用问题咨询

这种情况我需要改flink源码吗，还是自己实现一个自定义的state类就好了，还有在这个state类中怎么能获取到key呢 > 在 2019年4月17日，上午11:24，wenlong.lwl 写道： > > 可以封装一下state 的访问，从state get不到数据的时候，去数据库里取下，更新到state里 > > On Tue, 16 Apr 2019 at 20:53, zhang yue wrote: > >> 是的，我希望从mysql加载初始的状态，因为我的kafka消息是从某个时间点开始的，在这个时间点之前的数据需要先加载到flink state >>

Re: May be useful: our reference document for "Understanding State in Flink"

Thanks for sharing this resource with the community Oytun. It looks really helpful. I'm pulling in David and Fabian who work a lot on documentation. Maybe it's interesting for them to take a look at. The community had once the idea to set up a cook book with common Flink recipes but we never

Re: Fast restart of a job with a large state

Hi Sergey, at the moment neither local nor incremental savepoints are supported in Flink afaik. There were some ideas wrt incremental savepoints floating around in the community but nothing concrete yet. Cheers, Till On Tue, Apr 23, 2019 at 6:58 PM Sergey Zhemzhitsky wrote: > Hi Stefan, Paul,

Re: Looking for help in configuring Swift as State Backend

Hi Shakir, have you checked out Flink's documentation for Filesystems [1]? What is the problem you are observing? [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems.html Cheers, Till On Tue, Apr 23, 2019 at 9:30 PM PoolakkalMukkath, Shakir <

Re: No zero ( 2 ) exit code on k8s StandaloneJobClusterEntryPoint when save point with cancel...

Hi Vishal, it seems that the following is happening: You triggered the cancel with savepoint command from via the REST call. This command is an asynchronous operation which produces a result (the savepoint path). In order to deliver asynchronous results to the caller, Flink waits before shutting

Re: metric does not display on web

"No metrics available" indicates that either a) metrics have not been queried yet or b) metrics can not be transferred from the TaskManagers to the JobManager. Regarding the first option, how long have you waited for metrics to show up? It may take a bit for metrics to be available (around 10

Flink Control Stream

Hey, I wanted to use the control stream to dynamically adjust parameters of the tasks. I know that it is possible to use *connect()* and *BroadcastState *to obtain such a thing. But I would like to have the possibility to control the parameters inside the *AsyncFunction. *Like specific timeout for

Re: Error restoring from checkpoint on Flink 1.8

For future reference here is a cross link to the referred ML thread discussion [1]. [1] http://mail-archives.apache.org/mod_mbox/flink-user/201904.mbox/%3cm2ef5tpfwy.wl-nings...@gmail.com%3E Cheers, Till On Wed, Apr 24, 2019 at 4:00 AM Ning Shi wrote: > Hi Congxian, > > I think I have figured

Re: get custom gauge metric from WebMonitorEndpoint

You should be able to get the value of your custom metric. You'll have to set the log level to DEBUG and scan the logs for metric-related errors. On 23/04/2019 22:25, Georgi Stoyanov wrote: I've got custom metric -> || And I'm using them as suggested in the documentation -> || |

Re: Missing state in RocksDB checkpoints

Thanks for reporting this issue Ning. I think this is actually a blocker for the next release and should be fixed right away. For future reference here is the issue [1]. I've also pulled in Stefan who knows these components very well. [1] https://issues.apache.org/jira/browse/FLINK-12296

Flink Stream SQL group by TUMBLE(rowtime,)

大家好，问题求助： 事件时间使用EventTime，默认Checkpoint【没有offset从earliest开始，有则从offset开始】。 assignTimestampsAndWatermarks，1、不使用Watermark，即默认eventtime时间戳；2、使用官方max取最大值；两种情况问题基本相同： 问题描述： 比如，发送40条数据，窗口消费33条。另外7条，需要继续发送新的数据，才会被消费掉，即便重启程序-也要发送新的数据，才会消费上次"未及时"消费的数据，而不是自动从上一次的offset+1开始。 SQL： SELECT astyle,

Re: How to implement custom stream operator over a window? And after the Count-Min Sketch?

Hi Rong, thanks for your reply. I guess I already did something regarding what you have told to me. I have one example on this application [1], which uses this state [2] and computes a CountMinSketch [3]. I am seeking how to implement my own operator over a window in order to have more

Re: [DISCUSS] Temporarily remove support for job rescaling via CLI action "modify"

+1 for temporarily removing support for the modify command. Eventually, we have to add it again in order to support auto scaling. The next time we add it, we should address the known limitations. Cheers, Till On Wed, Apr 24, 2019 at 9:06 AM Paul Lam wrote: > Hi Gary, > > + 1 to remove it for

Re: [DISCUSS] Temporarily remove support for job rescaling via CLI action "modify"

Hi Gary, + 1 to remove it for now. Actually some users are not aware of that it’s still experimental, and ask quite a lot about the problem it causes. Best, Paul Lam > 在 2019年4月24日，14:49，Stephan Ewen 写道： > > Sounds reasonable to me. If it is a broken feature, then there is not much > value

Re: [DISCUSS] Temporarily remove support for job rescaling via CLI action "modify"

Sounds reasonable to me. If it is a broken feature, then there is not much value in it. On Tue, Apr 23, 2019 at 7:50 PM Gary Yao wrote: > Hi all, > > As the subject states, I am proposing to temporarily remove support for > changing the parallelism of a job via the following syntax [1]: > >

Re: [Discuss] Add JobListener (hook) in flink job lifecycle

Hi Till, IMHO, allow adding hooks involves 2 steps. 1. Provide hook interface, and call these hook in flink (ClusterClient) at the right place. This should be done by framework (flink) 2. Implement new hook implementation and add/register them into framework(flink) What I am doing is step 1