?????? how to simulate the scene "back pressure" in flink?Thanks~!

Thanks for both of your help... but... I can not understand both: Dear David Anderson: Is the whole command like this? flink run --backpressure  -c wordcount_increstate datastream_api-1.0-SNAPSHOT.ja

checkpoint fail

Summary I'm hitting an error when running a job, it happens several times, and I dont know why. Any help would be appreciated. Thanks! Details flink version: 1.4.2-1700 java.lang.Exception: Could not complete snapshot 158 for operator asyncio_by_transform -> flatmap_by_action_list_flat -

Re: Any testing issues when using StreamTableEnvironment.createTemporaryView?

I figured out my issue. I needed to assign watermarks (e.g. assignTimestampsAndWatermarks) after the fromElements. I could not figure out how the auto-generated code worked. I hooked up a debugger and guessed at the issue. On Thu, Oct 8, 2020 at 11:09 PM Dan Hill wrote: > *Summary* > I'm hitt

Re: NPE when checkpointing

Hi Binh, Could you try upgrading Flink's Java runtime? It was previously reported that upgrading to jdk1.8.0_251 was solving the problem. Piotrek pt., 9 paź 2020 o 19:41 Binh Nguyen Van napisał(a): > Hi, > > Thank you for helping me! > The code is compiled on > > java version "1.8.0_161" > Jav

Re: NPE when checkpointing

Hi, Thank you for helping me! The code is compiled on java version "1.8.0_161" Java(TM) SE Runtime Environment (build 1.8.0_161-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode) But I just checked our Hadoop and its Java version is java version "1.8.0_77" Java(TM) SE Runtime

Re: NPE when checkpointing

Hi, One more thing. It looks like it's not a Flink issue, but some JDK bug. Others reported that upgrading JDK version (for example to jdk1.8.0_251) seemed to be solving this problem. What JDK version are you using? Piotrek pt., 9 paź 2020 o 17:59 Piotr Nowojski napisał(a): > Hi, > > Thanks f

[PyFlink] register udf functions with different versions of the same library in the same job

Hi mates ! I've just read an amazing article about PyFlink and I'm absolutely delighted. I got some questions about udf registration, and it seems that it's possible to specify the list of libraries that

Re: Network issue leading to "No pooled slot available"

Quick update: it appears to work outside my test case too. I have not encountered this issue post update at all. On Thu, Oct 8, 2020 at 11:15 PM Khachatryan Roman < khachatryan.ro...@gmail.com> wrote: > Thanks for checking this workaround! > > I've created a jira issue [1] to check if AWS SDK ver

[Announce] Flink Forward Global Is Just Around the Corner

Hi Everyone, Flink Forward Global is just around the corner - October 19th to the 22nd - and our amazing line up of keynotes has been announced. - Peter Chalif from Citi will explore how Apache Flink has been fundamentally changing Big Data in the financial services industry - Ricky Sa

Re: NPE when checkpointing

Hi, Thanks for reporting the problem. I think this is a known issue [1] on which we are working to fix. Piotrek [1] https://issues.apache.org/jira/browse/FLINK-18196 pon., 5 paź 2020 o 08:54 Binh Nguyen Van napisał(a): > Hi, > > I have a streaming job that is written in Apache Beam and uses F

Re: Streaming SQL Job Switches to FINISHED before all records processed

Hey Timo, Hah, that's a fair point about using time. I guess I should update my statement to "as a user, I don't want to worry about *manually managing* time". That's a nice suggestion with the KeyedProcessFunction and no windows, I'll give that a shot. If I don't want to emit any duplicates, I'd

Re: autoCommit for postgres jdbc streaming in Table/SQL API

Thanks for the explanation. I will review the pull request. Let's move the discussion to the PR. Best, Jark On Fri, 9 Oct 2020 at 21:06, Dylan Forciea wrote: > Jark, > > > > Thank you! I had actually mistyped the JIRA issue; autoCommit needs to be > set to false for streaming to work. The defau

Re: Best way to test Table API and SQL

Hi Rex, let me copy paste my answer from a similar thread 2 months ago: Hi, this might be helpful as well: https://lists.apache.org/thread.html/rfe3b45a10fc58cf19d2f71c6841515eb7175ba731d5055b06f236b3f%40%3Cuser.flink.apache.org%3E First of all, it is important to know if you are interested i

Re: sql/table configuration naming guide/style/spec

Hi Luan, we haven't updated all config parameters to string-based options. This is still on going. The idle state retention will be configurable in 1.12: https://issues.apache.org/jira/browse/FLINK-18555 I hope this helps. Regards, Timo On 09.10.20 15:33, Luan Cooper wrote: Hi I've read

Re: Streaming SQL Job Switches to FINISHED before all records processed

Hi Austin, if you don't want to worry about time at all, you should probably not use any windows because those are a time-based operation. A solution that would look a bit nicer could be to use a pure KeyedProcessFunction and implement the deduplication logic without reusing windows. In Proc

Re: Flink Kuberntes Libraries

Hi Saksham, if you want to extend the Flink Docker image you can find here more details [1]. If you want to include the library in your user jar, then you have to add the library as a dependency to your pom.xml file and enable the shade plugin for building an uber jar [2]. [1] https://ci.apache.

Re: flink session job retention time

Hi Richard, If you mean the retention of completed jobs, there are following options: jobstore.cache-size [1] jobstore.expiration-time [2] jobstore.max-capacity [3] Best, Andrey [1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/config.html#jobstore-cache-size [2] https://ci.ap

sql/table configuration naming guide/style/spec

Hi I've read Sql Configuration guide https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/config.html#execution-options *which is very helpful to write sql like hive* *example on our flink sql ide: * set table.exec.async-lookup.timeout = 3s; insert into xxx select * from xxx *w

Re: autoCommit for postgres jdbc streaming in Table/SQL API

Jark, Thank you! I had actually mistyped the JIRA issue; autoCommit needs to be set to false for streaming to work. The default on the driver is true when the option isn’t specified. I’ve updated the issue accordingly. Setting this to false automatically on the read path would fix my issue. Ho

Re: Native State in Python Stateful Functions?

Hi Gordan, Glad to hear this is all inline with the patterns for StateFun! I think one thing that still trips me up in understanding the relationship between Flink and StateFun is how masters and workers come into play. In the case of remote functions, what are the scaling requirements for th

Re: how to simulate the scene "back pressure" in flink?Thanks~!

The Flink Operations Playground includes an optional backpressure simulation you can experiment with. It is described at the end of [1]. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/try-flink/flink-operations-playground.html#variants On Fri, Oct 9, 2020 at 10:02 AM Arvid Heis

Re: [Discuss] Add JobListener (hook) in flink job lifecycle

Reviving this thread again after I came across FLINK-12214 [1] since there are use cases which might benefit from this feature. Was there some conclusion on public APIs in the meantime? Should we proceed with the discussion here? Best, Matthias [1] https://issues.apache.org/jira/browse/FLINK-1221

Re: Native State in Python Stateful Functions?

Hi, On Fri, Oct 9, 2020, 4:20 PM Clements, Danial C wrote: > Hi, > > > > This makes sense and I can appreciate the stateless aspect for the remote > functions. We have a number of components that need access to quite a bit > of data, the idea was to key the incoming stream in a way that would >

Re: Native State in Python Stateful Functions?

Hi, This makes sense and I can appreciate the stateless aspect for the remote functions. We have a number of components that need access to quite a bit of data, the idea was to key the incoming stream in a way that would minimize calls to a reference DB and then store that result set in the st

Re: how to simulate the scene "back pressure" in flink?Thanks~!

You can add conf.setInteger(RestOptions.PORT, RestOptions.PORT.defaultValue()); at any place before creating the environment [1]. Default value is 8081, so you can access web ui through http://localhost:8081, but you can really choose any other free port. [1] https://github.com/apache/flink/blob

Re: How can I increase the parallelism on the Table API for Streaming Aggregation?

thanks! I will test -- -- Felipe Gutierrez -- skype: felipe.o.gutierrez -- https://felipeogutierrez.blogspot.com On Thu, Oct 8, 2020 at 6:19 PM Khachatryan Roman wrote: > > Hi Felipe, > > Your source is not parallel so it doesn't make sense to make local group > operator parallel. > If the sourc

?????? how to simulate the scene "back pressure" in flink?Thanks~!

 Thanks for your repies, could you tell me where to set RestOption.POPT?in configuration  what's the value should I set for RestOption.PORT? Thanks. --  -- ??:

Re: how to simulate the scene "back pressure" in flink?Thanks~!

The easiest way to see backpressure is to add some sleep to your sink, check [1] for an example. If you execute that unit test with a RestOption.PORT set in the configuration, you can even load the Web UI and watch the backpressure accumulate and finally go away at the end of the test. [1] https:/