Re: Batch mode execution
Thank you both! I'll try to switch the scheduler to "AdaptiveBatchScheduler". Best, Irakli From: Junrui Lee Sent: 05 March 2024 03:50 To: user Subject: Re: Batch mode execution Hello Irakli, The error is due to the fact that the Adaptive Scheduler doesn’t support batch jobs, as detailed in the Flink documentation[1]. When operating in reactive mode, Flink automatically decides the type of scheduler to use. For batch execution, the default scheduler is AdaptiveBatchScheduler, not AdaptiveScheduler as in the streaming case. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/elastic_scaling/#limitations<https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/elastic_scaling/#limitations> Best regards, Junrui lorenzo.affetti.ververica.com<http://lorenzo.affetti.ververica.com> via user mailto:user@flink.apache.org>> 于2024年3月4日周一 23:24写道： Hello Irakli and thank you for your question. I guess that somehow Flink enters the "reactive" mode while the adaptive scheduler is not configured. I would go with 2 options to isolate your issue: * Try with forcing the scheduling mode (https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/deployment/elastic_scaling/#adaptive-scheduler<https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/deployment/elastic_scaling/#adaptive-scheduler>) in your configuration: `jobmanager.scheduler: adaptive` *Let Flink decide for the runtime mode: if the source is bounded, you don't need `env.setRuntimeMode(RuntimeExecutionMode.BATCH)`, as Flink will understand that correctly. Can you try one of the two and see if that helps? For the rest: "running it in the "BATCH" mode was better as I don't have to deal with the Watermark Strategy". Still, you could opt for a simple watermarking strategy and stay with the streaming mode (for example, 'BoundedOutOfOrcerness': https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#watermark-strategies-and-the-kafka-connector<https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#watermark-strategies-and-the-kafka-connector>). On Mar 4, 2024 at 15:54 +0100, irakli.keshel...@sony.com<mailto:irakli.keshel...@sony.com> mailto:irakli.keshel...@sony.com>>, wrote: Hello, I have a Flink job which is processing bounded number of events. Initially, I was running the job in the "STREAMING" mode, but I realized that running it in the "BATCH" mode was better as I don't have to deal with the Watermark Strategy. The job is reading the data from the Kafka topic and was running fine in the "STREAMING" mode. I switched the job to the "BATCH" mode by setting "env.setRuntimeMode(RuntimeExecutionMode.BATCH)". I changed the Kafka Source to be bounded by setting ".setBounded(OffsetsInitializer.latest())" to the source builder and I updated the Watermark strategy to be "WatermarkStrategy.noWatermarks()". After making these changes and deploying the job I end up with following error: "java.lang.IllegalStateException","error.message":"Adaptive Scheduler is required for reactive mode". I couldn't find any documentation online which is connecting "Adaptive Scheduler" to the "BATCH" processing. Does anyone know where this error is coming from and how I can deal with it? Cheers, Irakli
Re: Batch mode execution
Hello Irakli, The error is due to the fact that the Adaptive Scheduler doesn’t support batch jobs, as detailed in the Flink documentation[1]. When operating in reactive mode, Flink automatically decides the type of scheduler to use. For batch execution, the default scheduler is AdaptiveBatchScheduler, not AdaptiveScheduler as in the streaming case. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/elastic_scaling/#limitations Best regards, Junrui lorenzo.affetti.ververica.com via user 于2024年3月4日周一 23:24写道： > Hello Irakli and thank you for your question. > > I guess that somehow Flink enters the "reactive" mode while the adaptive > scheduler is not configured. > > I would go with 2 options to isolate your issue: > >- Try with forcing the scheduling mode ( > > https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/deployment/elastic_scaling/#adaptive-scheduler) >in your configuration: `jobmanager.scheduler: adaptive` >- Let Flink decide for the runtime mode: if the source is bounded, >you don't need `env.setRuntimeMode(RuntimeExecutionMode.BATCH)`, as Flink >will understand that correctly. > > > Can you try one of the two and see if that helps? > > For the rest: "running it in the "BATCH" mode was better as I don't have > to deal with the Watermark Strategy". Still, you could opt for a simple > watermarking strategy and stay with the streaming mode (for example, > 'BoundedOutOfOrcerness': > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#watermark-strategies-and-the-kafka-connector > ). > On Mar 4, 2024 at 15:54 +0100, irakli.keshel...@sony.com < > irakli.keshel...@sony.com>, wrote: > > Hello, > > I have a Flink job which is processing bounded number of events. > Initially, I was running the job in the "STREAMING" mode, but I realized > that running it in the "BATCH" mode was better as I don't have to deal with > the Watermark Strategy. The job is reading the data from the Kafka topic > and was running fine in the "STREAMING" mode. > I switched the job to the "BATCH" mode by setting > "env.setRuntimeMode(RuntimeExecutionMode.BATCH)". I changed the Kafka > Source to be bounded by setting ".setBounded(OffsetsInitializer.latest())" > to the source builder and I updated the Watermark strategy to be > "WatermarkStrategy.noWatermarks()". > After making these changes and deploying the job I end up with following > error: "java.lang.IllegalStateException","error.message":"Adaptive > Scheduler is required for reactive mode". I couldn't find any documentation > online which is connecting "Adaptive Scheduler" to the "BATCH" processing. > Does anyone know where this error is coming from and how I can deal with it? > > Cheers, > Irakli > >
Re: Batch mode execution
Hello Irakli and thank you for your question. I guess that somehow Flink enters the "reactive" mode while the adaptive scheduler is not configured. I would go with 2 options to isolate your issue: • Try with forcing the scheduling mode (https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/deployment/elastic_scaling/#adaptive-scheduler) in your configuration: `jobmanager.scheduler: adaptive` • Let Flink decide for the runtime mode: if the source is bounded, you don't need `env.setRuntimeMode(RuntimeExecutionMode.BATCH)`, as Flink will understand that correctly. Can you try one of the two and see if that helps? For the rest: "running it in the "BATCH" mode was better as I don't have to deal with the Watermark Strategy". Still, you could opt for a simple watermarking strategy and stay with the streaming mode (for example, 'BoundedOutOfOrcerness': https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#watermark-strategies-and-the-kafka-connector). On Mar 4, 2024 at 15:54 +0100, irakli.keshel...@sony.com , wrote: > Hello, > > I have a Flink job which is processing bounded number of events. Initially, I > was running the job in the "STREAMING" mode, but I realized that running it > in the "BATCH" mode was better as I don't have to deal with the Watermark > Strategy. The job is reading the data from the Kafka topic and was running > fine in the "STREAMING" mode. > I switched the job to the "BATCH" mode by setting > "env.setRuntimeMode(RuntimeExecutionMode.BATCH)". I changed the Kafka Source > to be bounded by setting ".setBounded(OffsetsInitializer.latest())" to the > source builder and I updated the Watermark strategy to be > "WatermarkStrategy.noWatermarks()". > After making these changes and deploying the job I end up with following > error: "java.lang.IllegalStateException","error.message":"Adaptive Scheduler > is required for reactive mode". I couldn't find any documentation online > which is connecting "Adaptive Scheduler" to the "BATCH" processing. Does > anyone know where this error is coming from and how I can deal with it? > > Cheers, > Irakli
Batch mode execution
Hello, I have a Flink job which is processing bounded number of events. Initially, I was running the job in the "STREAMING" mode, but I realized that running it in the "BATCH" mode was better as I don't have to deal with the Watermark Strategy. The job is reading the data from the Kafka topic and was running fine in the "STREAMING" mode. I switched the job to the "BATCH" mode by setting "env.setRuntimeMode(RuntimeExecutionMode.BATCH)". I changed the Kafka Source to be bounded by setting ".setBounded(OffsetsInitializer.latest())" to the source builder and I updated the Watermark strategy to be "WatermarkStrategy.noWatermarks()". After making these changes and deploying the job I end up with following error: "java.lang.IllegalStateException","error.message":"Adaptive Scheduler is required for reactive mode". I couldn't find any documentation online which is connecting "Adaptive Scheduler" to the "BATCH" processing. Does anyone know where this error is coming from and how I can deal with it? Cheers, Irakli
