Question about concurrentRequests in ElasticsearchWriter

Hi, I'm currently working with the ElasticsearchSink class in the Flink Elasticsearch connector. I noticed that in the createBulkProcessor method, setConcurrentRequests(0) is used, which makes the flush() operation blocking. From my understanding, it seems that even if we set

Operator/Autoscaler/Autotuner tuning behavior question

Hello. I have some questions about memory autotuning in the Operator. 1. Does the autotuner try to upgrade the job with more memory allocated if it intercepts OutOfMemoryError? Say I initially provided too little memory for TM `resource` - will the job fail and stop on initializing or will the

Re: Question around manually setting Flink jobId

Thanks for the response, Asimansu. I should have been a bit more clearer and shared some additional context on our internal deployment. Currently, we are running *Flink in YARN application mode* for *batch* execution purposes (we also run it for stream execution as well). In the YARN application

Re: Question around manually setting Flink jobId

Hello Venkat, There are few ways to get the JobID from the client side. JobID is alpha numeric as 9eec4d17246b5ff965a43082818a3336. When you submit the job using flink command line client , Job is returned as Job has been submitted with JobID 9eec4d17246b5ff965a43082818a3336 1. using below

Re: Question around manually setting Flink jobId

Junrui, Thanks for your answer for the above questions. Allison and I work together on Flink. One of the main questions is, is there an easy way to get the Flink "JobID" from the Flink client side? Without the "JobID", users have no way to access Flink HistoryServer other than searching through

Re: Question around manually setting Flink jobId

Hi Allison, The PIPELINE_FIXED_JOB_ID configuration option is not intended for public use. IIUC, the only way to manually specify the jobId is submitting a job through the JAR RUN REST API, where you can provide the jobId in the request body (

Question around manually setting Flink jobId

Hi, I was wondering if there is any way to manually set the jobID for the jobGraph. I noticed that there is a configuration for PIPELINE_FIXED_JOB_ID, but there doesn't seem to be a way to set it via config with the StreamingJobGraphGenerator.java. Would appreciate any assistance if anyone has

Re: Question about time-based operators with RocksDB backend

t; abstractions such as ValueState, ListState, etc... as reported here > <https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/fault-tolerance/state/> > . > > My question is related to the utilization of time-based window operators > with the RocksDB

Re: Question about time-based operators with RocksDB backend

on pattern, which is important for large windows. >> >> Best, >> Zhanghao Chen >> -- >> *From:* Gabriele Mencagli >> *Sent:* Monday, March 4, 2024 19:38 >> *To:* user@flink.apache.org >> *Subject:* Question about time-based operators with

Re: Question about time-based operators with RocksDB backend

s important for large windows. > > Best, > Zhanghao Chen > -- > *From:* Gabriele Mencagli > *Sent:* Monday, March 4, 2024 19:38 > *To:* user@flink.apache.org > *Subject:* Question about time-based operators with RocksDB backend >

Re: Question about time-based operators with RocksDB backend

be satisfied with the reduce/aggregate function pattern, which is important for large windows. Best, Zhanghao Chen From: Gabriele Mencagli Sent: Monday, March 4, 2024 19:38 To: user@flink.apache.org Subject: Question about time-based operators with RocksDB backend

Question about time-based operators with RocksDB backend

a proper API to access the state. For example, the Keyed State API with different abstractions such as ValueState, ListState, etc... as reported here <https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/fault-tolerance/state/>. My question is related to the utili

[Meta question] Sharing blog posts

Hi, I have a netiquette question - is it ok to share blog posts here that are specific to Apache Flink and not vendor-focussed? thanks, Robin.

Re: keyby mapState use question

Hi, This should not happen. I guess the `onTimer` and `processElement` you are testing are triggered under different keyby keys. Note that the keyed states are partitioned by the keyby key first, so if querying or setting the state, you are only manipulating the specific partition which does not

keyby mapState use question

Hi all: KeyBy process function EventKeyedBroadcastProcessFunction { private transient mapstate = null; public void open(Configuration parameters) throws Exception { // initial map state } public void processElement() { // can't get onTimer() function set state key value }

Re: [Question] How to scale application based on 'reactive' mode

larity and allows us to identify >>>bottleneck tasks. >>>3. Autoscaler feature currently only works for K8s opeartor + native >>>K8s mode. >>> >>> >>> Best, >>> Zhanghao Chen >>> -- &

pyflink aggfunction in tvf question

hi flink community~ I came across a problem I didn't understand，I can't use pyflink aggfuction function properly in window tvf, The following are available： java aggfuntion flink system aggfunction window (not window tvf) I want to know if this is a bug or if I'm using it the wrong

Re: [Question] How to scale application based on 'reactive' mode

task granularity and allows us to identify >>bottleneck tasks. >>3. Autoscaler feature currently only works for K8s opeartor + native >>K8s mode. >> >> >> Best, >> Zhanghao Chen >> -- >> *发件人:* Dennis Jung

Re: [Question] How to scale application based on 'reactive' mode

caler feature currently only works for K8s opeartor + native >K8s mode. > > > Best, > Zhanghao Chen > -- > *发件人:* Dennis Jung > *发送时间:* 2023年9月2日 12:58 > *收件人:* Gyula Fóra > *抄送:* user@flink.apache.org > *主题:* Re: [Question] How to sca

Re: Question regarding asyncIO timeout

Hi, Leon > Besides that, Do you know if the async timeout is actually a global timeout? meaning it accounts for the time of each attempt call plus any interval time in between. Yes, the timeout is total timeout, you can see [1][2] for more detail. [1]

Re: Question regarding asyncIO timeout

Hi Ken, Thanks for the suggestion. Definitely a good call to just wrap the retry inside the client code. I'll give it a try. Besides that, Do you know if the async timeout is actually a global timeout? meaning it accounts for the time of each attempt call plus any interval time in between. I

Re: Question regarding asyncIO timeout

Hi Leon, Normally I try to handle retrying in the client being used to call the server, as you have more control/context. If that’s not an option for you, then normally (un)orderedWaitWithRetry() should work - when you say “it doesn’t seem to help much”, are you saying that even with retry

Question regarding asyncIO timeout

Hi Flink users, We are using Flink AsyncIO to call a grpc-based service in our Flink job. And from time to time we are experiencing Async function timeout issues, here's the exception. ``` java.lang.Exception: Could not complete the stream element: Record @ 169393916 : [B@cadc5b3. Caused by:

回复: [Question] How to scale application based on 'reactive' mode

to identify bottleneck tasks. 3. Autoscaler feature currently only works for K8s opeartor + native K8s mode. Best, Zhanghao Chen 发件人: Dennis Jung 发送时间: 2023年9月2日 12:58 收件人: Gyula Fóra 抄送: user@flink.apache.org 主题: Re: [Question] How to scale application based

Re: [Question] How to scale application based on 'reactive' mode

ecent feature and is >>>>>> specific to the k8s operator at the moment. >>>>>> >>>>>> Gyula >>>>>> >>>>>> On Fri, 1 Sep 2023 at 04:50, Dennis Jung wrote: >>>>>> >>>>>>>

Re: [Question] How to scale application based on 'reactive' mode

;>>>>> Thanks for your notice. >>>>>> >>>>>> Than what is the purpose of using 'reactive', if this doesn't do >>>>>> anything itself? >>>>>> What is the difference if I use auto-scaler without 'reactive' mode? >

Re: [Question] How to scale application based on 'reactive' mode

t; >>>>> >>>>> >>>>> 2023년 8월 18일 (금) 오후 7:51, Gyula Fóra 님이 작성: >>>>> >>>>>> Hi! >>>>>> >>>>>> I think what you need is probably not the reactive mode but a proper >>>>>&g

Re: [Question] How to scale application based on 'reactive' mode

>>>> autoscaler. The reactive mode as you say doesn't do anything in itself, >>>>> you >>>>> need to build a lot of logic around it. >>>>> >>>>> Check this instead: >>>>> https://nightlies.apache.org/flink/flink-

Re: [Question] How to scale application based on 'reactive' mode

, you >>>> need to build a lot of logic around it. >>>> >>>> Check this instead: >>>> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/autoscaler/ >>>> >>>> The Kubernetes Operat

Re: [Question] How to scale application based on 'reactive' mode

sed on kafka data rate / processing throughput. It also doesn't rely on >>> the reactive mode. >>> >>> Cheers, >>> Gyula >>> >>> On Fri, Aug 18, 2023 at 12:43 PM Dennis Jung >>> wrote: >>> >>>> Hello,

Re: [Question] How to scale application based on 'reactive' mode

It also doesn't rely on >> the reactive mode. >> >> Cheers, >> Gyula >> >> On Fri, Aug 18, 2023 at 12:43 PM Dennis Jung wrote: >> >>> Hello, >>> Sorry for frequent questions. This is a question about 'reactive' mode. >>>

Re: [Question] How to scale application based on 'reactive' mode

aler/ > > The Kubernetes Operator has a built in autoscaler that can scale jobs > based on kafka data rate / processing throughput. It also doesn't rely on > the reactive mode. > > Cheers, > Gyula > > On Fri, Aug 18, 2023 at 12:43 PM Dennis Jung wrote: > >> Hell

Re: [Question] How to scale application based on 'reactive' mode

stions. This is a question about 'reactive' mode. > > 1. As far as I understand, though I've setup `scheduler-mode: reactive`, > it will not change parallelism automatically by itself, by CPU usage or > Kafka consumer rate. It needs additional resource monitor features (such as > Horizontal Po

[Question] How to scale application based on 'reactive' mode

Hello, Sorry for frequent questions. This is a question about 'reactive' mode. 1. As far as I understand, though I've setup `scheduler-mode: reactive`, it will not change parallelism automatically by itself, by CPU usage or Kafka consumer rate. It needs additional resource monitor features

Re: [Question] Good way to monitor data skewness

Hello, Thanks for feedback. I'll try to add setup in the internal Grafana. BR, JUNG 2023년 8월 16일 (수) 오후 6:29, Hang Ruan 님이 작성: > Hi, Dennis. > > As Ron said, we could judge this situation by the metrics. > We are usually reporting the metrics to the external system like > Prometheus by the

Re: [Question] Good way to monitor data skewness

Hi, Dennis. As Ron said, we could judge this situation by the metrics. We are usually reporting the metrics to the external system like Prometheus by the metric reporter[1]. And these metrics could be shown by some other tools like grafana[2]. Best, Hang [1]

Re: [Question] Good way to monitor data skewness

Hi, Dennis, Although all operators are chained together, each operator metrics is there, you can view the metrcis related to the corresponding operator's input and output records through the UI, as following: [image: image.png] Best, Ron Dennis Jung 于2023年8月16日周三 14:13写道： > Hello people, >

[Question] Good way to monitor data skewness

Hello people, I'm trying to monitor data skewness with 'web-ui', between TaskManagers. Currently all operators has been chained, so I cannot find how data has been skewed to TaskManagers (or subtasks). But if I disable chaining, AFAIK, it can degrade performance.

Re: Question about serialization of java.util classes

41695117f6eaf4c798cc183cf1e36/flink-core/src/test/java/org/apache/flink/api/java/typeutils/PojoTypeExtractionTest.java#L192 >>> >>> Best, >>> Ron >>> >>> 于2023年8月13日周日 22:50写道： >>> >>>> Greetings, >>>> >>>> I

Re: Question about serialization of java.util classes

s using >>> a `java.util` class, either `ArrayList`, `HashSet` or `SortedSet` etc. This >>> forces Flink to use Kyro and throw these warnings: >>> >>> ``` >>> class java.util.ArrayList does not contain a setter for field size >>> Class class ja

Re: Question about serialization of java.util classes

l` class, either `ArrayList`, `HashSet` or `SortedSet` >>> etc. This forces Flink to use Kyro and throw these warnings: >>> >>> ``` >>> class java.util.ArrayList does not contain a setter for field size >>> Class class java.util.ArrayList cannot be used as a POJO type

Re: Question about serialization of java.util classes

sed as GenericType. >> Please read the Flink documentation on "Data Types & Serialization" for >> details of the effect on performance and schema evolution. >> ``` >> >> ``` >> No fields were detected for class java.util.HashSet so it cannot be used as

Re: Question about serialization of java.util classes

Types & Serialization" for details of the effect on >> performance and schema evolution. >> I read through the documentation and stackoverflow and the conclusion is >> that I need to make a TypeInfoFactory and use it inside a TypeInfo >> annotation over my POJO. >&

Re: Question about serialization of java.util classes

ation on "Data Types & Serialization" for details of the effect on > performance and schema evolution. > I read through the documentation and stackoverflow and the conclusion is that > I need to make a TypeInfoFactory and use it inside a TypeInfo annotation over > my POJO.

Re: Question about serialization of java.util classes

mentation and stackoverflow and the conclusion is > that I need to make a TypeInfoFactory and use it inside a TypeInfo > annotation over my POJO. > ``` > > My question is what do I need to do to get Flink to recognize my classes > as POJOs and use the POJO serializer for b

Question about serialization of java.util classes

pes & Serialization" for details of the effect on performance and schema evolution. I read through the documentation and stackoverflow and the conclusion is that I need to make a TypeInfoFactory and use it inside a TypeInfo annotation over my POJO. ``` My question is what do I need to do

RE: Re: flink configuration in flink kubernetes operator question about password

Hi tian tian, I think we can use podTemplate to mount kubernetes secrets as file or environment variables. Then we can access the secrets in our flink program. Please refers to https://github.com/apache/flink-kubernetes-operator/blob/main/examples/pod-template.yaml

Re: flink configuration in flink kubernetes operator question about password

Like s3.secret-key, the plaintext password cannot be directly written in the configuration. Is there a template language like jinja that can be replaced after mounting to the pod? >

flink configuration in flink kubernetes operator question about password

hi all, How to specify the password and other information that needs to be encrypted in the configuration file?

Re: apache-flink java question

Hi Chris, not yet and we are working on it[1]. best regards, Jing [1] https://issues.apache.org/jira/browse/FLINK-15736?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel=17697544#comment-17697544 On Thu, Jun 1, 2023 at 11:40 PM Joseph, Chris S wrote: > Hi, > > > > Does

apache-flink java question

Hi, Does Apache flink table api work with java 17? Thanks, Chris Joseph

Re: Flink Kubernetes Operator lifecycle state count metrics question

Hi Andrew! I think you are completely right, this is a bug. The per namespace metrics do not seem to filter per namespace and show the aggregated global count for each namespace: I opened a ticket: https://issues.apache.org/jira/browse/FLINK-32164 Thanks for reporting this! Gyula On Mon, May

Re: Question about Flink exception handling

Thanks for the clarification. On Tue, May 23, 2023 at 7:07 PM Weihua Hu wrote: > Hi Sharif, > > You could not catch exceptions globally. > > For exceptions that can be explicitly ignored for your business, you need > to add a try-catch in the operators. > For exceptions that are not catched,

Re: Question about Flink exception handling

Hi Sharif, You could not catch exceptions globally. For exceptions that can be explicitly ignored for your business, you need to add a try-catch in the operators. For exceptions that are not catched, Flink will trigger a recovery from failure automatically[1]. [1]

Re: Question about Flink exception handling

Thanks for your response. For simplicity, I want to capture exceptions in a centralized manner and log them for further analysis, without interrupting the job's execution or causing it to restart. On Tue, May 23, 2023 at 6:31 AM Shammon FY wrote: > Hi Sharif, > > I would like to know what do

Re: Question about Flink exception handling

Hi Sharif, I would like to know what do you want to do with the exception after catching it? There are different ways for different requirements, for example, Flink has already reported these exceptions. Best, Shammon FY On Mon, May 22, 2023 at 4:45 PM Sharif Khan via user wrote: > Hi,

Re: Flink Kubernetes Operator lifecycle state count metrics question

Also! I do have 2 FlinkDeployments deployed with this operator, but they are in different namespaces, and each of the per namespace metrics reports that it has 2 Deployments in them, even though there is only one according to kubectl. Actually...we just tried to deploy a change (enabling some

Re: Flink Kubernetes Operator lifecycle state count metrics question

Oh, FWIW, I do have operator HA enabled with 2 replicas running, but in my examples there, I am curl-ing the leader flink operator pod. On Mon, May 22, 2023 at 2:47 PM Andrew Otto wrote: > Hello! > > I'm doing some grafana+prometheus dashboarding for > flink-kubernetes-operator. Reading

Flink Kubernetes Operator lifecycle state count metrics question

Hello! I'm doing some grafana+prometheus dashboarding for flink-kubernetes-operator. Reading metrics docs , I see that I have nice per k8s namespace lifecycle current count gauge metrics in Prometheus. Using kubectl, I can see that I have one

Question about Flink exception handling

Hi, community. Can anyone please let me know? 1. What is the best practice in terms of handling exceptions in Flink jobs? 2. Is there any way to catch exceptions globally in Flink jobs? Basically, I want to catch exceptions from any operators in one place (globally). my expectation is let's say

Question about RexNodeExtractor formatting UDF names

Hi Flink Team: I have a question about RexNodeExtractor in Flink 1.16.0. I am trying to push down UDFs (with function names in the format ST_XXX, including underscores, e.g. ST_Contains) into TableSourceScan, and I have implemented applyFilters and handling of pushdown functions based

Re: Question about Flink metrics

! [1] https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/metric_reporters/#scope-variables-additional Best, Mason On Thu, May 4, 2023 at 11:35 PM neha goyal wrote: > Hello, > I have a question about the Prometheus metrics. I am able to fetch the > metrics from the

Question about Flink metrics

Hello, I have a question about the Prometheus metrics. I am able to fetch the metrics from the following expression. sum(flink_jobmanager_job_numRestarts{job_name="$job_name"}) by (job_name) Now I am interested in only a few jobs and I want to give them a label. How to achieve this? H

Re: Quick question about flink document.

23 PM Dongwoo Kim > wrote: > >> Hi community, I’m new to flink and trying to learn about the concepts of >> flink to prepare migrating heron application to flink. >> I have a quick question about this flink document. >> ( >> https://nightlies.apache.org/flink/flink-docs

Re: Quick question about flink document.

23 at 10:23 PM Dongwoo Kim > wrote: > >> Hi community, I’m new to flink and trying to learn about the concepts of >> flink to prepare migrating heron application to flink. >> I have a quick question about this flink document. >> ( >> https://nightlies.apache.org/

Re: Quick question about flink document.

new to flink and trying to learn about the concepts of > flink to prepare migrating heron application to flink. > I have a quick question about this flink document. > ( > https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/concepts/stateful-stream-processing/#snapshotting

Quick question about flink document.

Hi community, I’m new to flink and trying to learn about the concepts of flink to prepare migrating heron application to flink. I have a quick question about this flink document. (https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/concepts/stateful-stream-processing/#snapshotting

Question regarding Flink-CDC for postgres

Hi Flink Users, We are looking to use Flink Postgres CDC but then we noticed it only supports single thread reading during the snapshot reading phase. Is there any plan to support parallel snapshot reading for postgres in the near future? Looks like this is not an issue for mysql. Thanks Leon

Re: Pyflink Side Output Question and/or suggested documentation change

Thank you! On Mon, Feb 13, 2023 at 5:55 AM Dian Fu wrote: > Thanks Andrew, I think this is a valid advice. I will update the > documentation~ > > Regards, > Dian > > , > > On Fri, Feb 10, 2023 at 10:08 PM Andrew Otto wrote: > >> Question about side outputs

Re: Pyflink Side Output Question and/or suggested documentation change

Thanks Andrew, I think this is a valid advice. I will update the documentation~ Regards, Dian , On Fri, Feb 10, 2023 at 10:08 PM Andrew Otto wrote: > Question about side outputs and OutputTags in pyflink. The docs > <https://nightlies.apache.org/flink/flink-docs-master/docs/dev/d

Pyflink Side Output Question and/or suggested documentation change

Question about side outputs and OutputTags in pyflink. The docs <https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/side_output/> say we are supposed to yield output_tag, value Docs then say: > For retrieving the side output stream you use getSideOutput(

Question regarding java.lang.NoSuchMethodError in flink-s3-fs-hadoop.jar

Hi Flink Users, I am getting this exception after upgrading flink-s3-fs-hadoop from 1.15.3 to 1.16.0. *java.lang.NoSuchMethodError: 'com.google.protobuf.Descriptors$FileDescriptor org.apache.hadoop.security.proto.SecurityProtos.getDescriptor()'* After looking into the content of the

Re: Flink Forward Session Question

Hi Rion, Unlike the previous Flink Forwards to the best of my knowledge the latest edition was not uploaded to YouTube. It might make sense to reach out to the authors directly. On Sat, Dec 31, 2022 at 5:35 PM Rion Williams wrote: > Hey Flinkers, > > Firstly, early Happy New Year’s to everyone

Flink Forward Session Question

Hey Flinkers, Firstly, early Happy New Year’s to everyone in the community. I’ve been digging a bit into exactly-once processing with Flink and Pinot and I came across this session from Flink Foward last year: -

Re: Question about match_recognize clause in Flink

Hi Marjan, That's rather weird, because PyFlink uses the same implementation. Could you file a Jira ticket? If not, let me know and I'll create one for you. Best regards, Martijn On Thu, Dec 22, 2022 at 11:37 AM Marjan Jordanovski wrote: > Hello, > > I am using custom made connector to

Question about match_recognize clause in Flink

Hello, I am using custom made connector to create Source table in this way: create table Source ( ts TIMESTAMP(3), instance STRING, sservice STRING, logdatetime STRING, threadid STRING, level STRING, log_line STRING ) with (

Re: question about Async IO

key extractor, then original order might not be >>> preserved since keyBy will execute re-shuffle/re-balance. >>> >>> We were also using reinterpretAsKeyedStream feature [1] after async >>> operators to avoid calling ".keyBay" multiple times in pipeline

Re: Question about UDF randomly processed input row twice

in the StreamPhysicalCalc, as of result of which, it seems the one row will be processed for twice. Best regards, Yuxia 发件人: "Xinyi Yan" 收件人: "yuxia" 抄送: "User" 发送时间: 星期五, 2022年 11 月 04日 上午 5:28:30 主题: Re: Question about UDF randomly processed input row twice Ok. Th

Re: Question about UDF randomly processed input row twice

egards, > Yuxia > > -- > *发件人: *"Xinyi Yan" > *收件人: *"User" > *发送时间: *星期四, 2022年 11 月 03日 上午 6:59:20 > *主题: *Question about UDF randomly processed input row twice > > Hi all, > I found a weird UDF behavior, and it's a

Re: Question about UDF randomly processed input row twice

The dategen may produce rows with same values. >From my side, in Flink, the udf shouldn't process one row for twice, >otherwise, it should be a critical bug. Best regards, Yuxia 发件人: "Xinyi Yan" 收件人: "User" 发送时间: 星期四, 2022年 11 月 03日 上午 6:59:20 主题: Q

Question about UDF randomly processed input row twice

Hi all, I found a weird UDF behavior, and it's a single thread that processes UDF twice, see FLINK-29855 for more details. Basically, I created a datagen table with a random integer (1 row per second) and passed this value into the UDF. Inside

Re: question about Async IO

using reinterpretAsKeyedStream feature [1] after async >> operators to avoid calling ".keyBay" multiple times in pipeline. Calling >> .keyBy always has negative impact on performance. >> With reinterpretAsKeyedStream we were able to use keyed operators with >> acces

Re: question about Async IO

rs. > > Hope that helped. > > [1] > https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/experimental/ > > Regards, > Krzysztof Chmielewski > > > > > > > > pt., 14 paź 2022 o 19:11 Galen Warren > napisał(a): > >

Re: question about Async IO

e were able to use keyed operators with access to keyed state after Async operators. Hope that helped. [1] https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/experimental/ Regards, Krzysztof Chmielewski pt., 14 paź 2022 o 19:11 Galen Warren napisał(a): > I have a

question about Async IO

I have a question about Flink's Async IO support: Async I/O | Apache Flink <https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/asyncio/> . I understand that access to state is not supported in an AsyncFunction. However, if the input to an AsyncFunction is a

Re: Question about SQL gateway

Thanks Xuyang. Jack > On Oct 12, 2022, at 8:46 AM, Xuyang wrote: > > Hi, currently I think there is no ha about gateway. When the gateway crashes, > the job about being submitted sync will be cancelled, and the async job will > continue running. When the gateway restarts, the async job could

Question about SQL gateway

Hi, I submit a stream job from the SQL gateway. The stream job keeps outputting results to the SQL gateway. If the SQL gateway restarts or crashes, the stream job will continue running. After the SQL gateway restarts, how to get the results of the steam job? Thanks. Jack

Question about Flink Broadcast State event ordering

Hi Flink user group, I have a question around broadcast. Reading the docs https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/#important-considerations, it says the following: > Order of events in Broadcast State may differ across ta

Re: Question regarding to debezium format

Hello Edwin, Would you mind sharing a simple FlinkSQL DDL for the table you are creating with the kafka connector and dthe debezium-avro-confluent format? Also, can you elaborate on the mechanism who publishes initially to the schema registry and share the corresponding schema? In a nutshell,

Re: Question regarding to debezium format

Hi Edwin, I'm suspecting that's because those fields are considered metadata which are treated separately. There's https://issues.apache.org/jira/browse/FLINK-20454 for adding the metadata support for the Debezium format with a PR provided, but not yet reviewed. If you could have a look at the PR

Question regarding to debezium format

Hi guys, I was trying to use flink sql to consume data from kafka source, the format of which is debezium-avro-confluent. And I encountered a AvroTypeException saying that "Found something, expecting union", where something is not a type but a field that I defined in the schema registery. So

Re: A question about restoring state with an additional variable with kryo

t; >>> >>> Thias >>> >>> >>> >>> >>> >>> >>> >>> [1] >>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/schema_evolution/#kryo-cannot-be-used-for

Re: A question about restoring state with an additional variable with kryo

://nightlies.apache.org/flink/flink-docs-master/docs/libs/state_processor_api/ >> >> [3] >> https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html >> >> >> >> >> >> >> >> *From:* Vishal Santoshi >> *Sent:* Fr

Re: A question about restoring state with an additional variable with kryo

e-used-for-schema-evolution > > [2] > https://nightlies.apache.org/flink/flink-docs-master/docs/libs/state_processor_api/ > > [3] > https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html > > > > > > > > *From:* Vishal Santoshi >

RE: A question about restoring state with an additional variable with kryo

16, 2022 1:17 AM To: user Subject: Re: A question about restoring state with an additional variable with kryo ⚠EXTERNAL MESSAGE – CAUTION: Think Before You Click ⚠ The exception thrown is as follows. I realize that it is trying to read the long value. How do I signal to kryo that it is OK

Re: A question about restoring state with an additional variable with kryo

The exception thrown is as follows. I realize that it is trying to read the long value. How do I signal to kryo that it is OK and that he object can have a default value Caused by: java.io.EOFException: No more bytes left. at

Re: A question about restoring state with an additional variable with kryo

<< How do I make sure that when reconstituting the state, kryo does not complain? It tries to map the previous state to the new definition of Class A and complains that it cannot read the value for `String b`. >> How do I make sure that when reconstituting the state, kryo does not complain? It

A question about restoring state with an additional variable with kryo

I have state in rocksDB that represents say class A { String a } I now change my class and add another variable Class A { String a; long b = 0; } How do I make sure that when reconstituting the state, kryo does not complain? It tries to map the previous state to the new definition of

Re: A classloading question about submitting Flink jobs on Yarn-Perjob Mode

r 13, 2022 11:24 > To: zhanghao.chen ; user > Subject: Re: A classloading question about submitting Flink jobs on > Yarn-Perjob Mode > > Hi, > > The yarn.classpath.include-user-jar parameter is shown as > yarn.per-job-cluster.include-user-jar parameter in Flink 1.14. &

Re: A classloading question about submitting Flink jobs on Yarn-Perjob Mode

@h.yuan...@gmail.com<mailto:h.yuan...@gmail.com> Any thoughts on this? Best, Zhanghao Chen From: hjw <1010445...@qq.com> Sent: Tuesday, September 13, 2022 11:24 To: zhanghao.chen ; user Subject: Re: A classloading question about submitting Flink

  1   2   3   4   5   6   7   8   9   10   >