Adding flink user group On Tue, 28 Nov 2023 at 13:39, Tauseef Janvekar <tauseefjanve...@gmail.com> wrote:
> Did you set some specific job manager or task manager deployment > parameters ? - No > > Did you test without the basic ingress auth ? to be sure this is not > related to that. - yes we did. And the problem persists. > > Please let me know if I can share anything else that might be useful. > > > On Tue, 28 Nov 2023 at 12:52, Benoit Tailhades <benoit.tailha...@gmail.com> > wrote: > >> Did you set some specific job manager or task manager deployment >> parameters ? >> >> Did you test without the basic ingress auth ? to be sure this is not >> related to that. >> >> Le mar. 28 nov. 2023 à 06:58, Tauseef Janvekar <tauseefjanve...@gmail.com> >> a écrit : >> >>> Hi Benoit, >>> >>> Are your task manager and job manager on the same vm ? >>> >>> We have deployed it on kubernetes cluster with helm chart - >>> https://github.com/bitnami/charts/blob/main/bitnami/flink/values.yaml >>> So we cannot confirm if it is on the same vm/node. >>> One more thing is that we have enabled authentication using basic >>> ingress auth. - >>> https://kubernetes.github.io/ingress-nginx/examples/auth/basic/ >>> >>> How did you configure the Job manager address in the task manager conf >>> file ? >>> Did you modify the binding in configuration files ? >>> It got auto configured using helm chart. We did not modify anything on >>> top of basic helm chart installation. >>> >>> Thanks, >>> Tauseef >>> >>> On Mon, 27 Nov 2023 at 19:29, Benoit Tailhades < >>> benoit.tailha...@gmail.com> wrote: >>> >>>> Hello, Tauseef, >>>> >>>> Can you give more details ? Are your task manager and job manager on >>>> the same vm ? >>>> >>>> How did you configure the Job manager address in the task manager conf >>>> file ? >>>> Did you modify the binding in configuration files ? >>>> >>>> Benoit >>>> >>>> Le lun. 27 nov. 2023 à 14:29, Tauseef Janvekar < >>>> tauseefjanve...@gmail.com> a écrit : >>>> >>>>> Dear Team, >>>>> >>>>> We are getting below error messages in our logs. >>>>> Any help on how to resolve would be greatly appreciated. >>>>> >>>>> 2023-11-27 08:14:29,712 INFO org.apache.pekko.remote.transport. >>>>> ProtocolStateActor [] - No response from remote for outbound >>>>> association. Associate timed out after [20000 ms]. >>>>> 2023-11-27 08:14:29,713 WARN org.apache.pekko.remote. >>>>> ReliableDeliverySupervisor [] - Association with remote >>>>> system [pekko.tcp://flink-metrics@flink-taskmanager:34309] has >>>>> failed, address is now gated for [50] ms. Reason: [Association failed >>>>> with [pekko.tcp://flink-metrics@flink-taskmanager:34309]] Caused by: [ >>>>> No response from remote for outbound association. Associate timed out >>>>> after [20000 ms].] >>>>> 2023-11-27 08:14:29,730 WARN org.apache.pekko.remote.transport.netty. >>>>> NettyTransport [] - Remote connection to [null] failed with >>>>> org.jboss.netty.channel.ConnectTimeoutException: connection timed >>>>> out: flink-taskmanager/172.20.237.127:34309 >>>>> 2023-11-27 08:14:58,401 INFO org.apache.pekko.remote.transport. >>>>> ProtocolStateActor [] - No response from remote for outbound >>>>> association. Associate timed out after [20000 ms]. >>>>> 2023-11-27 08:14:58,402 WARN org.apache.pekko.remote. >>>>> ReliableDeliverySupervisor [] - Association with remote >>>>> system [pekko.tcp://flink-metrics@flink-taskmanager:34309] has >>>>> failed, address is now gated for [50] ms. Reason: [Association failed >>>>> with [pekko.tcp://flink-metrics@flink-taskmanager:34309]] Caused by: [ >>>>> No response from remote for outbound association. Associate timed out >>>>> after [20000 ms].] >>>>> 2023-11-27 08:14:58,426 WARN org.apache.pekko.remote.transport.netty. >>>>> NettyTransport [] - Remote connection to [null] failed with >>>>> org.jboss.netty.channel.ConnectTimeoutException: connection timed >>>>> out: flink-taskmanager/172.20.237.127:34309 >>>>> 2023-11-27 08:15:22,402 INFO org.apache.pekko.remote.transport. >>>>> ProtocolStateActor [] - No response from remote for outbound >>>>> association. Associate timed out after [20000 ms]. >>>>> 2023-11-27 08:15:22,403 WARN org.apache.pekko.remote. >>>>> ReliableDeliverySupervisor [] - Association with remote >>>>> system [pekko.tcp://flink-metrics@flink-taskmanager:34309] has >>>>> failed, address is now gated for [50] ms. Reason: [Association failed >>>>> with [pekko.tcp://flink-metrics@flink-taskmanager:34309]] Caused by: [ >>>>> No response from remote for outbound association. Associate timed out >>>>> after [20000 ms].] >>>>> 2023-11-27 08:15:22,434 WARN org.apache.pekko.remote.transport.netty. >>>>> NettyTransport [] - Remote connection to [null] failed with >>>>> org.jboss.netty.channel.ConnectTimeoutException: connection timed >>>>> out: flink-taskmanager/172.20.237.127:34309 >>>>> 2023-11-27 08:15:46,411 INFO org.apache.pekko.remote.transport. >>>>> ProtocolStateActor [] - No response from remote for outbound >>>>> association. Associate timed out after [20000 ms]. >>>>> 2023-11-27 08:15:46,412 WARN org.apache.pekko.remote. >>>>> ReliableDeliverySupervisor [] - Association with remote >>>>> system [pekko.tcp://flink-metrics@flink-taskmanager:34309] has >>>>> failed, address is now gated for [50] ms. Reason: [Association failed >>>>> with [pekko.tcp://flink-metrics@flink-taskmanager:34309]] Caused by: [ >>>>> No response from remote for outbound association. Associate timed out >>>>> after [20000 ms].] >>>>> 2023-11-27 08:15:46,436 WARN org.apache.pekko.remote.transport.netty. >>>>> NettyTransport [] - Remote connection to [null] failed with >>>>> org.jboss.netty.channel.ConnectTimeoutException: connection timed >>>>> out: flink-taskmanager/172.20.237.127:34309 >>>>> 2023-11-27 08:16:10,434 INFO org.apache.pekko.remote.transport. >>>>> ProtocolStateActor [] - No response from remote for outbound >>>>> association. Associate timed out after [20000 ms]. >>>>> 2023-11-27 08:16:10,435 WARN org.apache.pekko.remote. >>>>> ReliableDeliverySupervisor [] - Association with remote >>>>> system [pekko.tcp://flink-metrics@flink-taskmanager:34309] has >>>>> failed, address is now gated for [50] ms. Reason: [Association failed >>>>> with [pekko.tcp://flink-metrics@flink-taskmanager:34309]] Caused by: [ >>>>> No response from remote for outbound association. Associate timed out >>>>> after [20000 ms].] >>>>> 2023-11-27 08:16:10,477 WARN org.apache.pekko.remote.transport.netty. >>>>> NettyTransport [] - Remote connection to [null] failed with >>>>> org.jboss.netty.channel.ConnectTimeoutException: connection timed >>>>> out: flink-taskmanager/172.20.237.127:34309 >>>>> 2023-11-27 08:16:34,402 WARN org.apache.pekko.remote. >>>>> ReliableDeliverySupervisor [] - Association with remote >>>>> system [pekko.tcp://flink-metrics@flink-taskmanager:34309] has >>>>> failed, address is now gated for [50] ms. Reason: [Association failed >>>>> with [pekko.tcp://flink-metrics@flink-taskmanager:34309]] Caused by: [ >>>>> No response from remote for outbound association. Associate timed out >>>>> after [20000 ms].] >>>>> 2023-11-27 08:16:34,402 INFO org.apache.pekko.remote.transport. >>>>> ProtocolStateActor [] - No response from remote for outbound >>>>> association. Associate timed out after [20000 ms]. >>>>> 2023-11-27 08:16:34,415 WARN org.apache.pekko.remote.transport.netty. >>>>> NettyTransport [] - Remote connection to [null] failed with >>>>> org.jboss.netty.channel.ConnectTimeoutException: connection timed >>>>> out: flink-taskmanager/172.20.237.127:34309 >>>>> 2023-11-27 08:16:58,401 INFO org.apache.pekko.remote.transport. >>>>> ProtocolStateActor [] - No response from remote for outbound >>>>> association. Associate timed out after [20000 ms]. >>>>> 2023-11-27 08:16:58,405 WARN org.apache.pekko.remote. >>>>> ReliableDeliverySupervisor [] - Association with remote >>>>> system [pekko.tcp://flink-metrics@flink-taskmanager:34309] has >>>>> failed, address is now gated for [50] ms. Reason: [Association failed >>>>> with [pekko.tcp://flink-metrics@flink-taskmanager:34309]] Caused by: [ >>>>> No response from remote for outbound association. Associate timed out >>>>> after [20000 ms].] >>>>> 2023-11-27 08:16:58,443 WARN org.apache.pekko.remote.transport.netty. >>>>> NettyTransport [] - Remote connection to [null] failed with >>>>> org.jboss.netty.channel.ConnectTimeoutException: connection timed >>>>> out: flink-taskmanager/172.20.237.127:34309 >>>>> 2023-11-27 08:17:22,412 INFO org.apache.pekko.remote.transport. >>>>> ProtocolStateActor [] - No response from remote for outbound >>>>> association. Associate timed out after [20000 ms]. >>>>> 2023-11-27 08:17:22,412 WARN org.apache.pekko.remote. >>>>> ReliableDeliverySupervisor [] - Association with remote >>>>> system [pekko.tcp://flink-metrics@flink-taskmanager:34309] has >>>>> failed, address is now gated for [50] ms. Reason: [Association failed >>>>> with [pekko.tcp://flink-metrics@flink-taskmanager:34309]] Caused by: [ >>>>> No response from remote for outbound association. Associate timed out >>>>> after [20000 ms].] >>>>> 2023-11-27 08:17:22,425 WARN org.apache.pekko.remote.transport.netty. >>>>> NettyTransport [] - Remote connection to [null] failed with >>>>> org.jboss.netty.channel.ConnectTimeoutException: connection timed >>>>> out: flink-taskmanager/172.20.237.127:34309 >>>>> 2023-11-27 08:17:46,401 INFO org.apache.pekko.remote.transport. >>>>> ProtocolStateActor [] - No response from remote for outbound >>>>> association. Associate timed out after [20000 ms]. >>>>> 2023-11-27 08:17:46,402 WARN org.apache.pekko.remote. >>>>> ReliableDeliverySupervisor [] - Association with remote >>>>> system [pekko.tcp://flink-metrics@flink-taskmanager:34309] has >>>>> failed, address is now gated for [50] ms. Reason: [Association failed >>>>> with [pekko.tcp://flink-metrics@flink-taskmanager:34309]] Caused by: [ >>>>> No response from remote for outbound association. Associate timed out >>>>> after [20000 ms].] >>>>> 2023-11-27 08:17:46,413 WARN org.apache.pekko.remote.transport.netty. >>>>> NettyTransport [] - Remote connection to [null] failed with >>>>> org.jboss.netty.channel.ConnectTimeoutException: connection timed >>>>> out: flink-taskmanager/172.20.237.127:34309 >>>>> 2023-11-27 08:18:11,711 INFO org.apache.pekko.remote.transport. >>>>> ProtocolStateActor [] - No response from remote for outbound >>>>> association. Associate timed out after [20000 ms]. >>>>> 2023-11-27 08:18:11,711 WARN org.apache.pekko.remote. >>>>> ReliableDeliverySupervisor [] - Association with remote >>>>> system [pekko.tcp://flink-metrics@flink-taskmanager:34309] has >>>>> failed, address is now gated for [50] ms. Reason: [Association failed >>>>> with [pekko.tcp://flink-metrics@flink-taskmanager:34309]] Caused by: [ >>>>> No response from remote for outbound association. Associate timed out >>>>> after [20000 ms].] >>>>> 2023-11-27 08:18:11,719 WARN org.apache.pekko.remote.transport.netty. >>>>> NettyTransport [] - Remote connection to [null] failed with >>>>> org.jboss.netty.channel.ConnectTimeoutException: connection timed >>>>> out: flink-taskmanager/172.20.237.127:34309 >>>>> 2023-11-27 08:18:35,933 INFO org.apache.pekko.remote.transport. >>>>> ProtocolStateActor [] - No response from remote for outbound >>>>> association. Associate timed out after [20000 ms]. >>>>> 2023-11-27 08:18:35,933 WARN org.apache.pekko.remote. >>>>> ReliableDeliverySupervisor [] - Association with remote >>>>> system [pekko.tcp://flink-metrics@flink-taskmanager:34309] has >>>>> failed, address is now gated for [50] ms. Reason: [Association failed >>>>> with [pekko.tcp://flink-metrics@flink-taskmanager:34309]] Caused by: [ >>>>> No response from remote for outbound association. Associate timed out >>>>> after [20000 ms].] >>>>> 2023-11-27 08:18:35,941 WARN org.apache.pekko.remote.transport.netty. >>>>> NettyTransport [] - Remote connection to [null] failed with >>>>> org.jboss.netty.channel.ConnectTimeoutException: connection timed >>>>> out: flink-taskmanager/172.20.237.127:34309 >>>>> >>>>> Thanks, >>>>> Tauseef >>>>> >>>>
