Ok I see. So I guess in 1.4.3, one was fixed but broke something else? Because in 1.4.2 it feels like it worked after a refresh but with 1.4.3 it always fails looks like.
On Fri., Jan. 21, 2022, 3:29 a.m. Chesnay Schepler, <ches...@apache.org> wrote: > What you are seeing in 1.14.3 is a separate bug, that behaves very > similarly. https://issues.apache.org/jira/browse/FLINK-25732 > > On 20/01/2022 22:11, John Smith wrote: > > As per another recent thread. This is still an issue. > > On Wed, 19 Jan 2022 at 06:36, Chesnay Schepler <ches...@apache.org> wrote: > >> This is a serialization bug in Flink, see >> https://issues.apache.org/jira/browse/FLINK-24550. >> It will be fixed in the upcoming 1.14.3 release. >> >> On 19/01/2022 09:01, Caizhi Weng wrote: >> >> Hi! >> >> To print out gc logs of job manager you can add this configuration to >> flink-conf.yaml >> >> env.java.opts.jobmanager: -Xloggc:/tmp/jobmanager-gc.log >> -XX:+PrintGCDetails -XX:+PrintGCDateStamps >> >> This will print gc logs to /tmp/jobmanager-gc.log. >> >> I'm not familiar with the garbage collection metrics page. If the unit of >> time is ms then gc does not seem to be heavy. However I would still >> recommend to print out gc logs for a double check. >> >> John Smith <java.dev....@gmail.com> 于2022年1月19日周三 06:43写道： >> >>> I think I may know what is causing the issue... So I have 3 job managers. >>> >>> 1- I Navigated to a non leader UI and submitted a new job... >>> 2- The UI timed out with grey lines >>> 3- Some Internal Server error messages appeared. >>> 4- Going back to the leader UI checking the running jobs, the job seems >>> to have been submitted and running. >>> 5- Going back to the job manager UI that failed, now shows ok. >>> >>> And the logs are as follows... And below are the GC metrics from the UI. >>> >>> 2022-01-18 22:33:24,574 INFO >>> org.apache.flink.client.deployment.application.executors. >>> EmbeddedExecutor [] - Job 2297fdac52fa7191afee9ec4ff11c805 is submitted. >>> 2022-01-18 22:33:24,574 INFO >>> org.apache.flink.client.deployment.application.executors. >>> EmbeddedExecutor [] - Submitting Job with JobId=2297f >>> dac52fa7191afee9ec4ff11c805. >>> 2022-01-18 22:34:00,618 ERROR org.apache.flink.runtime.rest.handler.job. >>> JobDetailsHandler [] - Unhandled exception. >>> java.util.concurrent.CancellationException: null >>> at java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java: >>> 2276) ~[?:1.8.0_312] >>> at org.apache.flink.runtime.rest.handler.legacy. >>> DefaultExecutionGraphCache.getExecutionGraphInternal( >>> DefaultExecutionGraphCache.java:98) ~[flink-dist_2.12-1.14.2.jar:1.14.2] >>> at org.apache.flink.runtime.rest.handler.legacy. >>> DefaultExecutionGraphCache.getExecutionGraphInfo( >>> DefaultExecutionGraphCache.java:67) ~[flink-dist_2.12-1.14.2.jar:1.14.2] >>> at org.apache.flink.runtime.rest.handler.job. >>> AbstractExecutionGraphHandler.handleRequest( >>> AbstractExecutionGraphHandler.java:81) ~[flink-dist_2.12-1.14.2.jar:1.14 >>> .2] >>> at org.apache.flink.runtime.rest.handler.AbstractRestHandler >>> .respondToRequest(AbstractRestHandler.java:83) ~[flink-dist_2.12-1.14.2 >>> .jar:1.14.2] >>> at org.apache.flink.runtime.rest.handler.AbstractHandler >>> .respondAsLeader(AbstractHandler.java:195) ~[flink-dist_2.12-1.14.2.jar: >>> 1.14.2] >>> at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler >>> .lambda$channelRead0$0(LeaderRetrievalHandler.java:83) ~[flink-dist_2.12 >>> -1.14.2.jar:1.14.2] >>> at java.util.Optional.ifPresent(Optional.java:159) [?:1.8.0_312] >>> at org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer >>> .java:45) [flink-dist_2.12-1.14.2.jar:1.14.2] >>> at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler >>> .channelRead0(LeaderRetrievalHandler.java:80) [flink-dist_2.12-1.14.2 >>> .jar:1.14.2] >>> at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler >>> .channelRead0(LeaderRetrievalHandler.java:49) [flink-dist_2.12-1.14.2 >>> .jar:1.14.2] >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler >>> .java:99) [flink-dist_2.12-1.14.2.jar:1.14.2] >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.invokeChannelRead( >>> AbstractChannelHandlerContext.java:379) [flink-dist_2.12-1.14.2.jar:1.14 >>> .2] >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.invokeChannelRead( >>> AbstractChannelHandlerContext.java:365) [flink-dist_2.12-1.14.2.jar:1.14 >>> .2] >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.fireChannelRead( >>> AbstractChannelHandlerContext.java:357) [flink-dist_2.12-1.14.2.jar:1.14 >>> .2] >>> at org.apache.flink.runtime.rest.handler.router.RouterHandler.routed( >>> RouterHandler.java:115) [flink-dist_2.12-1.14.2.jar:1.14.2] >>> at org.apache.flink.runtime.rest.handler.router.RouterHandler >>> .channelRead0(RouterHandler.java:94) [flink-dist_2.12-1.14.2.jar:1.14.2] >>> at org.apache.flink.runtime.rest.handler.router.RouterHandler >>> .channelRead0(RouterHandler.java:55) [flink-dist_2.12-1.14.2.jar:1.14.2] >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler >>> .java:99) [flink-dist_2.12-1.14.2.jar:1.14.2] >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.invokeChannelRead( >>> AbstractChannelHandlerContext.java:379) [flink-dist_2.12-1.14.2.jar:1.14 >>> .2] >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.invokeChannelRead( >>> AbstractChannelHandlerContext.java:365) [flink-dist_2.12-1.14.2.jar:1.14 >>> .2] >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.fireChannelRead( >>> AbstractChannelHandlerContext.java:357) [flink-dist_2.12-1.14.2.jar:1.14 >>> .2] >>> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >>> MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) >>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.invokeChannelRead( >>> AbstractChannelHandlerContext.java:379) [flink-dist_2.12-1.14.2.jar:1.14 >>> .2] >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.invokeChannelRead( >>> AbstractChannelHandlerContext.java:365) [flink-dist_2.12-1.14.2.jar:1.14 >>> .2] >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.fireChannelRead( >>> AbstractChannelHandlerContext.java:357) [flink-dist_2.12-1.14.2.jar:1.14 >>> .2] >>> at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0( >>> FileUploadHandler.java:238) [flink-dist_2.12-1.14.2.jar:1.14.2] >>> at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0( >>> FileUploadHandler.java:71) [flink-dist_2.12-1.14.2.jar:1.14.2] >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler >>> .java:99) [flink-dist_2.12-1.14.2.jar:1.14.2] >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.invokeChannelRead( >>> AbstractChannelHandlerContext.java:379) [flink-dist_2.12-1.14.2.jar:1.14 >>> .2] >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.invokeChannelRead( >>> AbstractChannelHandlerContext.java:365) [flink-dist_2.12-1.14.2.jar:1.14 >>> .2] >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.fireChannelRead( >>> AbstractChannelHandlerContext.java:357) [flink-dist_2.12-1.14.2.jar:1.14 >>> .2] >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> CombinedChannelDuplexHandler$DelegatingChannelHandlerContext >>> .fireChannelRead(CombinedChannelDuplexHandler.java:436) [flink-dist_2.12 >>> -1.14.2.jar:1.14.2] >>> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >>> ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) >>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>> at org.apache.flink.shaded.netty4.io.netty.handler.codec. >>> ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) >>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler >>> .java:251) [flink-dist_2.12-1.14.2.jar:1.14.2] >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.invokeChannelRead( >>> AbstractChannelHandlerContext.java:379) [flink-dist_2.12-1.14.2.jar:1.14 >>> .2] >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.invokeChannelRead( >>> AbstractChannelHandlerContext.java:365) [flink-dist_2.12-1.14.2.jar:1.14 >>> .2] >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.fireChannelRead( >>> AbstractChannelHandlerContext.java:357) [flink-dist_2.12-1.14.2.jar:1.14 >>> .2] >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline >>> .java:1410) [flink-dist_2.12-1.14.2.jar:1.14.2] >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.invokeChannelRead( >>> AbstractChannelHandlerContext.java:379) [flink-dist_2.12-1.14.2.jar:1.14 >>> .2] >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> AbstractChannelHandlerContext.invokeChannelRead( >>> AbstractChannelHandlerContext.java:365) [flink-dist_2.12-1.14.2.jar:1.14 >>> .2] >>> at org.apache.flink.shaded.netty4.io.netty.channel. >>> DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) >>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>> at org.apache.flink.shaded.netty4.io.netty.channel.nio. >>> AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java: >>> 166) [flink-dist_2.12-1.14.2.jar:1.14.2] >>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop >>> .processSelectedKey(NioEventLoop.java:719) [flink-dist_2.12-1.14.2.jar: >>> 1.14.2] >>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop >>> .processSelectedKeysOptimized(NioEventLoop.java:655) [flink-dist_2.12- >>> 1.14.2.jar:1.14.2] >>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop >>> .processSelectedKeys(NioEventLoop.java:581) [flink-dist_2.12-1.14.2.jar: >>> 1.14.2] >>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run( >>> NioEventLoop.java:493) [flink-dist_2.12-1.14.2.jar:1.14.2] >>> at org.apache.flink.shaded.netty4.io.netty.util.concurrent. >>> SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) >>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>> at org.apache.flink.shaded.netty4.io.netty.util.internal. >>> ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [flink-dist_2.12-1.14 >>> .2.jar:1.14.2] >>> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312] >>> >>> These are the stats displayed on the UI for the garbage collector... >>> >>> Advanced >>> JVM (Heap/Non-Heap) Memory >>> Type Committed Used Maximum >>> Heap 2.87 GB 571 MB 2.87 GB >>> Non-Heap 136 MB 128 MB 744 MB >>> Outside JVM Memory >>> Type Count Used Capacity >>> Direct 33 642 KB 642 KB >>> Mapped 0 0 B 0 B >>> Garbage Collection >>> Collector Count Time >>> PS_MarkSweep 3 346 >>> PS_Scavenge 83 2772 >>> >>> >>> On Tue, 18 Jan 2022 at 17:15, John Smith <java.dev....@gmail.com> wrote: >>> >>>> Sorry, unhandled exception. >>>> >>>> On Tue, 18 Jan 2022 at 15:16, John Smith <java.dev....@gmail.com> >>>> wrote: >>>> >>>>> Also that I handled exception is printed in job managers. >>>>> >>>>> On Tue., Jan. 18, 2022, 10:11 a.m. John Smith, <java.dev....@gmail.com> >>>>> wrote: >>>>> >>>>>> I actually mean the job manager. I run a total of three job managers >>>>>> for HA. >>>>>> >>>>>> For example of I click running jobs, it displays light grey boxes for >>>>>> a while and then top right corner throws internal server error. But after >>>>>> if I refresh it's ok I see the list. It seems to happen on the non leader >>>>>> job manager UIs. Looking at the UI metric for the job manager doesn't >>>>>> seem >>>>>> to indicate anything out of the ordinary. >>>>>> >>>>>> Is there a java 8 command tool that can hook into the live PID and >>>>>> display GC stats? >>>>>> >>>>>> On Mon., Jan. 17, 2022, 12:06 a.m. Caizhi Weng, <tsreape...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi! >>>>>>> >>>>>>> "When I use the UI and just navigate from one jobmanager ui to the >>>>>>> other", I guess you mean you're talking about task managers instead of >>>>>>> job >>>>>>> managers (there is only one job manager per web ui). >>>>>>> >>>>>>> This long pause usually indicates that the component (no matter task >>>>>>> managers or job manager) you're jumping to is very busy so that they >>>>>>> can't >>>>>>> handle web requests. This is not a healthy state and you should look >>>>>>> into >>>>>>> it. The most probable cause is heavy GC. To determine this you can >>>>>>> print GC >>>>>>> details to log and see if there are long GC pauses. >>>>>>> >>>>>>> John Smith <java.dev....@gmail.com> 于2022年1月15日周六 09:53写道： >>>>>>> >>>>>>>> Hi using 1.14.2, running 3 job nodes. >>>>>>>> >>>>>>>> Everything seems to work ok. >>>>>>>> >>>>>>>> When I use the UI and just navigate from one jobmanager ui to the >>>>>>>> other, it sometimes seems to take long or timeout and I see an >>>>>>>> "Internal >>>>>>>> Server Error" popup on the top right messages. >>>>>>>> >>>>>>>> Looking at the logs I see this, but not sure it's related... >>>>>>>> >>>>>>>> 2022-01-15 00:04:44,925 ERROR >>>>>>>> org.apache.flink.runtime.rest.handler.job.JobExceptionsHandler [] - >>>>>>>> Unhandled exception. >>>>>>>> java.util.concurrent.CancellationException: null >>>>>>>> at >>>>>>>> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276) >>>>>>>> ~[?:1.8.0_312] >>>>>>>> at >>>>>>>> org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInternal(DefaultExecutionGraphCache.java:98) >>>>>>>> ~[flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInfo(DefaultExecutionGraphCache.java:67) >>>>>>>> ~[flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.handleRequest(AbstractExecutionGraphHandler.java:81) >>>>>>>> ~[flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:83) >>>>>>>> ~[flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:195) >>>>>>>> ~[flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:83) >>>>>>>> ~[flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at java.util.Optional.ifPresent(Optional.java:159) [?:1.8.0_312] >>>>>>>> at >>>>>>>> org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:45) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:80) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:115) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:94) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:55) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:238) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:71) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at >>>>>>>> org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) >>>>>>>> [flink-dist_2.12-1.14.2.jar:1.14.2] >>>>>>>> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312] >>>>>>>> >>>>>>>> >> >
