Re: Flink SQL Async UDF

Thanks, Aitozi. Giannis Polyzos 于2023年5月8日周一 15:04写道： > > What im curious is about the completable future in the function signature > (eval) > public final void eval(CompletableFuture> future, > Object... keys) > > is is injected automatically? > > Best > > On

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.0.0 released

Thanks Yang and Nice to see it happen. Best, Aitozi. Yang Wang 于2022年6月5日周日 16:14写道： > The Apache Flink community is very happy to announce the release of Apache > Flink Kubernetes Operator 1.0.0. > > The Flink Kubernetes Operator allows users to manage their Apache Flink &g

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.0.0 released

Thanks Yang and Nice to see it happen. Best, Aitozi. Yang Wang 于2022年6月5日周日 16:14写道： > The Apache Flink community is very happy to announce the release of Apache > Flink Kubernetes Operator 1.0.0. > > The Flink Kubernetes Operator allows users to manage their Apache Flink &g

Re: State migration for sql job

Thanks for JING & Kurt's reply. I think we prefer to choose the option (a) that will not take the history data into account. IMO, if we want to process all the historical data, we have to store the original data, which may be a big overhead to backend. But if we just aggregate after the new

State migration for sql job

When use flink sql, we encounter a big problem to deal with sql state compatibility.Think we have a group agg sql like ```sqlselect sum(`a`) from source_t group by `uid But if i want to add a new agg column to ```sqlselect sum(`a`), max(`a`) from source_t group by `uidThen sql state will

Strange stack when job encounter back pressure

Hi, I notice the job encounter a strange case: the upstream operator is underpressure, all task in back pressure sample shows HIGH, but when i jstack the downstream task, I only see the stack below: java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native

Re: A little doubt about the blog A Deep To Flink's NetworkStack

Hi Zhijiang, Thanks for your explanation, I have understood it. My former thoughts was wrong, the real TCP channel was established when createPartitionRequestClient. Thanks, Aitozi -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

A little doubt about the blog A Deep To Flink's NetworkStack

ection no matter there is different subtasks of the same task are scheduled onto the same TaskManager or not. Please correct me if my thoughts was wrong. Thanks Aitozi -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Timeout about local test case

va:242) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) Caused by: akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka.tcp://flink@151.101.16.133:50619/), Path(/user/jobmanager_0)]] Thanks, Aitozi -- Sent from: http://apache-flink-user-mailing-list-archive.2

Re: RocksDB backend with deferred writes?

/tpsservice/1df9ccb8a7b6b2782a558d3c32d40c19.pdf Thanks, Aitozi 发件人: "David J. C. Beach" 日期: 2019年4月29日 星期一 上午11:52 收件人: aitozi 抄送: 主题: Re: RocksDB backend with deferred writes? Thanks Aitozi. Your answer makes good sense and I'm trying to implement this now. My code i

Re: RocksDB backend with deferred writes?

by setCurrentKey before flush to rocksdb or make the prefix key (keygroup + key) yourself put/get value to/from rocksdb. Thanks, Aitozi 发件人: "David J. C. Beach" 日期: 2019年4月29日 星期一 上午6:43 收件人: 主题: RocksDB backend with deferred writes? I have a stateful operator in a task which

Re: [Discuss] Semantics of event time for state TTL

this capacity. So in this scenario, the event-time-accesstime/watermark-expiration-check will be the proper combination. I think if can add the interface to allow user to custom will be flexible. Thanks, Aitozi -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Production readiness

Hi, Andrey I have another question that if i do not set the maximum parallelism first(which be set to 128 by default), and then rescale to a parallelism bigger than 128. In this scenario，will the state lost? Thanks, Aitozi -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4

Re: [SURVEY] Custom RocksDB branch

+1 from my side, since we rely on this feature to implement the real state ttl . -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: IndexOutOfBoundsException on deserialization after updating to 1.6.1

Hi,Bruno Aranda Could you provide an complete example to reproduce the exception? Thanks, Aitozi -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Trying to figure out why a slot takes a long time to checkpoint

Thanks, aitozi -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: 回复：InpoolUsage & InpoolBuffers inconsistence

nel buffer is not enough, at least we should see this value to be 1. Mainly reason I care about this is i want to find a metric to monitor this. you just mention the autoread flag. Do you think monitor this flag in inputchannel is a good choice ? Thanks, aitozi. Zhijiang(wangzhijiang999)

Re: InpoolUsage & InpoolBuffers inconsistence

And my doubt for that comes from the debug of problem of checkpoint expiration. I encountered the checkpoint expiration with no backpressure shown in web ui. But after i add many log, i found that the barrier send to the downstream, And the downstream may be set to autoread = false , and block the

InpoolUsage & InpoolBuffers inconsistence

Hi, I found that these two metric is inconsistent, the inpoolQueueLength is positive, but the inpoolUsage is always zero. Is this a bug? cc @Chesnay Schepler -- Sent from:

Re: Aggregator in CEP

Hi, Now that it still not support the aggregator function in cep iterativeCondition. Now may be you need to check the condition by yourself to get the aggregator result. I will work for this these day, you can take a look on this issue:

A litte doubt about the usage of checkpointLock

Hi, community I see the checkpointLock is used in StreamTask to ensure that we don't have concurrent method calls that void consistent checkpoints. As i known, it is used in the data consume , state interactive and the timerService, But I am doubt that, if an application don't enable the

Re: Streaming

or the SQL above? Thanks, Aitozi -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Streaming

is no need to do the keyBy previous, how does the window deal with this. Thanks Aitozi. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: [Flink-Forward]Why cant get video of 2018 forward in youtube?

Hi, Kl0 Got it. Thanks, Aitozi -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Flink-Forward]Why cant get video of 2018 forward in youtube?

Hi, community, Why cant we get the talk of 2018 Flink forward conference from youtube? thanks, Aitozi -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: TaskManager use more memory than Xmx

Hi, Szczypiński Do you have noticed the size of the maxDirectMemorySize in your taskamanger jvm options， AFAIK the total memory one Taskmanager used is the (heap memory + directMemory) which may larger than the heap size specified by -Xmx? Best wishes Aitozi -- Sent from: http://apache-flink

Re: ConcurrentModificationException while accessing managed keyed state

Hi ,Garvit Sharma Just read your code snippet, I think it look ok. May need your reproduce senior and exception stackstrace then. Best wishes Aitozi Garvit Sharma wrote > Hi, > > Sorry guys for the delay. I was trying to reproduce the complete error on > my local machine but c

Re: ConcurrentModificationException while accessing managed keyed state

Hi Garvit Sharma, Flink run with per parallel with a single thread. Can you show a little code about how you use the keyed state in processFunction Garvit Sharma wrote > Hi, > > I have a use case where I am keeping the keyed state in ProcessFunction. > > Key: Integer personId; > > /** > *

Re: sharebuffer prune code

Can you explain it more explictly? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: sharebuffer prune code

OK, i will post a jira later. and i am foucs on cep library recently, and run into a little bug, and can you share some progress about the community of cep library and what feature you are working on thanks -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

sharebuffer prune code

Hi， i am running into a cep bug : it always running into failed to find previous sharebufferEntry, i think it may be caused by prune the sharebufferEntry wrongly, but when i read the code, i cant understand this : https://gist.github.com/Aitozi/007210bc7ade01a81f8d0fc4ba5a2c99 why when encounted

Visual CEP pattern & State transformation

Hi, When i read about the CEP code, i am feeling that it is a bit difficult to understand the whole transformation, I hava a idea that we can have a draw of the pattern and State transformation like the DAG plan picture in web UI. May be it can be the "ignore, take, processd" like this

Re: cep code

Then what is STOP state in NFA, i haven't seen this state in event pattern match paper ? Does each Not pattern will be transformed to stop state? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

cep code

Hi, i am reading flink-cep source code based on release-1.3.2 . I cant understand here , can anyone help me on this in NFACompiler? private List, String>> getCurrentNotCondition() { List, String>> notConditions = new

MergingWindow

Hi, i cant unserstand usage of this snippest of the code in MergingWindowSet.java, can anyone explain this for me ? if (!(mergedWindows.contains(mergeResult) && mergedWindows.size() == 1)) { mergeFunction.merge(mergeResult,

Mergingwindow

Hi, i cant understand the usage of this snippest of this code , can anyone explain foe me ? thanks -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

merging window

Hi, i cant understand the usage of this snippest of this code , can anyone explain foe me ? thanks -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Telling if a job has caught up with Kafka

Hi, rmetzger0 Sorry to reply to this old question, i found that we use the kafka client 0.9 in class kafkaThread which lead to the lose of many other detail metrics add in kafka client 10 like per partition consumer lag mentioned by this issuse https://issues.apache.org/jira/browse/FLINK-7945. i

Re: ResultPartitionMetrics

hi, i have understood it. thanks, aitozi -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Watermark on connected stream

Hi, You can see the field in AbstractStreamOperator // We keep track of watermarks from both inputs, the combined input is the minimum // Once the minimum advances we emit a new watermark for downstream operators private long combinedWatermark = Long.MIN_VALUE; it will chose the Min watermark

Re: Using latency markers

Hi, Aljoscha, the dashboard shown NAN is just because the value of the latencyGague is not numerical， so it can't be shown in dashboard, i removed the other latencydescprition except the sink, so i can see the latency in dashboard, do i need to post a pr? thanks, Aitozi -- Sent from: http

Re: LatencyMarker

. https://github.com/apache/flink/pull/4665 about the num of lost data, can you take a look at this, and give me some suggestion, thank you! Aitozi Tzu-Li (Gordon) Tai wrote > Hi, > > There is actually latency metrics in the Web UI, but I think there was > also some previou

HeapInternalTimerService#advanceWatermark

Hi, i have read this snip of code again and again, but i cant understand the usage of it , can anyone explain it for me. thank you very much. public void advanceWatermark(long time) throws Exception { currentWatermark = time; InternalTimer timer;

Re: LatencyMarker

Hi, You are right, i have proved that the backpressure will increase the latency,and in have another question, now the value of the latency is not Visualization in the dashboard，is there any plan to do this, i have add this by remove the other operator and only keep the source -> end latency and

LatencyMarker

i am doubt that whether the backpressure will increase the latency of the LatencyMarker? does the latency number can be used to monitor the backpressure -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: WaterMark & Eventwindow not fired correctly

} } and when i changed the Parallelism(300) of assigntimestampandwatermarks , the window can be fired. thanks, aitozi Aljoscha Krettek wrote > Hi, > > So I understood that you have roughly this pipeline: > > Input 1 --\ >|- CoFlatMap - TimestampAndWate

Re: WaterMark & Eventwindow not fired correctly

t very good , i dont know can you understand me :) thanks, aitozi -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/WaterMark-Eventwindow-not-fired-correctly-tp14668p14727.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

WaterMark & Eventwindow not fired correctly

why is the windowapply Function can not be triggerd only by the event time and watermark thanks, aitozi -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/WaterMark-Eventwindow-not-fired-correctly-tp14668.html Sent from the Apache Flink User Mailing List arch

Re: KafkaConsumerBase

rtitions for each local subtask, using the `assignTopicPartitions` method" Thanks aitozi -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/KafkaConsumerBase-tp14636p14642.html Sent from the Apache Flink User Mailing List archive. mailing

Re: About KafkaConsumerBase

Hi， thanks，you explained clearly! -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/About-KafkaConsumerBase-tp14601p14621.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: About KafkaConsumerBase

Hi,Piotr Nowojski i think you are right, but i think it is executed in parallel, but in each parallel , it maintain itself a individual instance of FlinkKafkaConsumerBase, and it contains a individual pendingOffsetsToCommit , am right ? thanks, aitozi -- View this message in context

About KafkaConsumerBase

Hello: i am new to Flink, ijust read the source code . i am doubt that , why in FlinkKafkaConsumerBase.java (version1.2), like method : notifyCheckpointComplete may change the pendingOffsetsToCommit in parallel , why dont need to be surrouned with "synchronized" thanks -- View this