Re: Question regarding asyncIO timeout

2023-09-05 Thread Leon Xu
} > > } > > }, executor) > > .completeOnTimeout(timeoutResult, *REQUEST_TIMEOUT_SECONDS*, > TimeUnit.*SECONDS*) > > .thenAccept((ServerResult result) -> { > > ServerRequestResult requestResult = *new* >

Question regarding asyncIO timeout

2023-09-05 Thread Leon Xu
Hi Flink users, We are using Flink AsyncIO to call a grpc-based service in our Flink job. And from time to time we are experiencing Async function timeout issues, here's the exception. ``` java.lang.Exception: Could not complete the stream element: Record @ 169393916 : [B@cadc5b3. Caused by:

Re: 回复: Questions regarding adaptive scheduler with YARN and application mode

2023-06-28 Thread Leon Xu
/confluence/display/FLINK/FLIP-291%3A+Externalized+Declarative+Resource+Management> > [3] Autoscaler | Apache Flink Kubernetes Operator > <https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.5/docs/custom-resource/autoscaler/> > > Best, > Zhanghao Chen >

Questions regarding adaptive scheduler with YARN and application mode

2023-06-26 Thread Leon Xu
Hi Flink users, I am trying to use Adaptive Scheduler to auto scale our Flink streaming jobs (NOT batch job). Our jobs are running on YARN with application mode. There isn't much doc around how adaptive scheduler works. So I have some questions: 1. How does Adaptive Scheduler work with

Question regarding Flink-CDC for postgres

2023-03-03 Thread Leon Xu
Hi Flink Users, We are looking to use Flink Postgres CDC but then we noticed it only supports single thread reading during the snapshot reading phase. Is there any plan to support parallel snapshot reading for postgres in the near future? Looks like this is not an issue for mysql. Thanks Leon

Re: How to process mini-batch events in Flink with Datastream API

2023-02-10 Thread Leon Xu
t; Austin > > [1]: > https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/datastream/operators/asyncio/ > > On Fri, Feb 10, 2023 at 3:22 PM Leon Xu wrote: > >> I wonder if windows will be the solution when it comes to datastream API. >> >> On Fri,

Re: How to process mini-batch events in Flink with Datastream API

2023-02-10 Thread Leon Xu
I wonder if windows will be the solution when it comes to datastream API. On Fri, Feb 10, 2023 at 12:07 PM Leon Xu wrote: > Hi Flink Users, > > We wanted to use Flink to run a decoration pipeline, where we would like > to make calls to some external service to fetch data and alt

How to process mini-batch events in Flink with Datastream API

2023-02-10 Thread Leon Xu
Hi Flink Users, We wanted to use Flink to run a decoration pipeline, where we would like to make calls to some external service to fetch data and alter the event in the Flink pipeline. Since there's external service call involved so we want to do batch calls so that it can reduce the load on the

Question regarding java.lang.NoSuchMethodError in flink-s3-fs-hadoop.jar

2023-01-29 Thread Leon Xu
Hi Flink Users, I am getting this exception after upgrading flink-s3-fs-hadoop from 1.15.3 to 1.16.0. *java.lang.NoSuchMethodError: 'com.google.protobuf.Descriptors$FileDescriptor org.apache.hadoop.security.proto.SecurityProtos.getDescriptor()'* After looking into the content of the

Re: Flink application mode/S3A Exception after upgrading from to Flink 1.16.0

2023-01-27 Thread Leon Xu
Thank you Gabor. I will test with a downgraded version to see how it goes. On Fri, Jan 27, 2023 at 11:51 AM Gabor Somogyi wrote: > The min supported version was 2.8.5 but in 1.17 it's gonna be 2.10.1 so > one can downgrade. > > G > > On Fri, Jan 27, 2023, 20:42 Leon Xu wro

Re: Flink application mode/S3A Exception after upgrading from to Flink 1.16.0

2023-01-27 Thread Leon Xu
m.copyFromLocalFile(source, s3FileSystem.getHomeDirectory()); > // will fail > s3FileSystem.copyFromLocalFile(source2, s3FileSystem.getHomeDirectory()); > // works fine > > I don't have a JIRA account yet, but once I do and no one has any > objections I'll create a bug ticket and try to resolve this. > &

Flink application mode/S3A Exception after upgrading from to Flink 1.16.0

2023-01-26 Thread Leon Xu
Hi Flink Users, We are trying to upgrade Flink from 1.12.7 to 1.16.0. But we run into the following issue: We are running Flink job through application mode. After the upgrade, when we submit the job and now it gets this exception: *org.apache.flink.client.deployment.ClusterDeploymentException:

Re: java.lang.ClassCastException on flink application mode

2022-11-30 Thread Leon Xu
I also tried different settings for *yarn.per-job-cluster.include-user-jar* and *classloader.resolve-order* But none of them worked. Leon On Wed, Nov 30, 2022 at 11:17 PM Leon Xu wrote: > Hi Biao, > > Thanks for getting back to me. > Here's the command I used: > >

Re: java.lang.ClassCastException on flink application mode

2022-11-30 Thread Leon Xu
: parent-first Thanks Leon On Wed, Nov 30, 2022 at 10:14 PM Biao Geng wrote: > Hi Leon, > > Can you share your full command for submission? > > > Best, > Biao Geng > > Leon Xu 于2022年12月1日周四 06:27写道: > >> Hi Flink Users, >> >> We ran into java.lang.Cl

java.lang.ClassCastException on flink application mode

2022-11-30 Thread Leon Xu
Hi Flink Users, We ran into java.lang.ClassCastException after moving the flink job from session mode to application mode. *java.lang.ClassCastException: class [B cannot be cast to class java.lang.String ([B and java.lang.String are in module java.base of loader 'bootstrap') at

Re: Questions regarding classpath loading order in YarnClusterDescriptor

2022-06-05 Thread Leon Xu
is managing > the directory name, which is not so elegant. It can be useful to add the > ability to customize loading orders of jars in classpath while it is also > important to package the jars more carefully to avoid the conflicts. > > Best, > Biao Geng > &

Questions regarding classpath loading order in YarnClusterDescriptor

2022-06-04 Thread Leon Xu
Hi Flink Community, We are building on top of *org.apache.flink.yarn.YarnClusterDescriptor *to submit a flink application from Java code to YARN cluster, in the application mode. We are setting the classpath as the value of *the yarn.provided.lib.dirs *property under the yarn configuration. By

Re: Application mode deployment through API call

2022-05-22 Thread Leon Xu
PM Leon Xu wrote: > Thanks Geng, I will take a look at the YARNApplicationITCase > implementation. > And yeah I am switching from session mode to application mode, as > application mode can offer better resource isolation. > Besides that with application mode we don't need to worry

Re: Application mode deployment through API call

2022-05-22 Thread Leon Xu
k correctly and so on. All > details shaded by flink have to be handled by hand. > > > > Best, > > Biao Geng > > > > *发件人**:* Leon Xu > *日期**:* 星期六, 2022年5月21日 下午2:24 > *收件人**:* user > *主题**:* Application mode deployment through API call > > Hi Flink commun

Application mode deployment through API call

2022-05-21 Thread Leon Xu
Hi Flink community, I am looking to deploy my flink job through *Application Mode *in my Java program. Ideally I'd like my java code to just call an API to achieve this. Does flink provide a rest API to support this? I don't seem to find any documentation or code on that. If I need to build this

Re: IOException/StacklessClosedChannelException on flink-connector-kinesis trigger job to restart

2021-12-15 Thread Leon Xu
just increase the max retries. Of course, that > means Flink takes longer to restart if something is really off. > > If you have more information on the circumstances under which you see > these exceptions, please share. I'm CCing Danny in case he has more ideas. > > On Mon, Dec 13,

IOException or StacklessClosedChannelException on flink-connector-kinesis triggers job to restart

2021-12-13 Thread Leon Xu
Hi Flink users, I use flink-1.12.5 kinesis connector to consume data from kinesis. >From time to time I am getting IOException or StacklessClosedChannelException, which will fail the Flink operator when it by default reaches 10 times and trigger the entire job to restart. I have two questions:

IOException/StacklessClosedChannelException on flink-connector-kinesis trigger job to restart

2021-12-12 Thread Leon Xu
Hi Flink users, I used flink-1.12.5 kinesis connector to consume data from kinesis. >From time to time I am getting IOException or StacklessClosedChannelException, which will fail the Flink operator when it by default reaches 10 times and trigger the entire job to restart. I have two questions: