flink on yarn 指定节点开启 yarn session 报错
各位大佬好！ 请教各位个问题：使用的flink 版本1.9.0。测试flink on yarn 指定节点 开启yarn session 会话。 希望yarnsession开启的集群 jobmanager开启在我指定的节点上。 更改的配置如下： 报错信息如下： === 2019-11-27 10:35:45,640 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperati 2019-11-27 10:35:45,650 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Session: 0x36ea133e2b51435 closed 2019-11-27 10:35:45,650 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x3 2019-11-27 10:35:45,651 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopping Akka RPC service. 2019-11-27 10:35:45,659 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopping Akka RPC service. 2019-11-27 10:35:45,666 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 2019-11-27 10:35:45,669 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with fl 2019-11-27 10:35:45,676 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 2019-11-27 10:35:45,677 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with fl 2019-11-27 10:35:45,696 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Remoting shut down. 2019-11-27 10:35:45,696 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Remoting shut down. 2019-11-27 10:35:45,709 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopped Akka RPC service. 2019-11-27 10:35:45,713 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopped Akka RPC service. 2019-11-27 10:35:45,714 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Could not start cluster entrypoint YarnSess org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to initialize the cluster entrypoint YarnSessionClusterEntrypoint. at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:182) at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:501) at org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint.main(YarnSessionClusterEntrypoint.java:93) Caused by: org.apache.flink.util.FlinkException: Could not create the DispatcherResourceManagerComponent. at org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResou at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:210) at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:163) ... 2 more Caused by: java.net.BindException: Could not start rest endpoint on any port in port range 8082 at org.apache.flink.runtime.rest.RestServerEndpoint.start(RestServerEndpoint.java:219) at org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResou ... 9 more End of LogType:jobmanager.log === 请问这种问题要怎么解决？多谢各位大佬！
Re: flink on yarn 指定节点开启 yarn session 报错
8082 端口被占用了吧？ Best, tison. 杨浩程 于2019年11月27日周三 上午10:56写道： > 各位大佬好！ >请教各位个问题：使用的flink 版本1.9.0。测试flink on yarn 指定节点 开启yarn session 会话。 >希望yarnsession开启的集群 jobmanager开启在我指定的节点上。 >更改的配置如下： > 报错信息如下： > > === > 2019-11-27 10:35:45,640 INFO > org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl > - backgroundOperati > 2019-11-27 10:35:45,650 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - > Session: 0x36ea133e2b51435 closed > 2019-11-27 10:35:45,650 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - > EventThread shut down for session: 0x3 > 2019-11-27 10:35:45,651 INFO > org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopping > Akka RPC service. > 2019-11-27 10:35:45,659 INFO > org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopping > Akka RPC service. > 2019-11-27 10:35:45,666 INFO > akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting > down remote daemon. > 2019-11-27 10:35:45,669 INFO > akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote > daemon shut down; proceeding with fl > 2019-11-27 10:35:45,676 INFO > akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting > down remote daemon. > 2019-11-27 10:35:45,677 INFO > akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote > daemon shut down; proceeding with fl > 2019-11-27 10:35:45,696 INFO > akka.remote.RemoteActorRefProvider$RemotingTerminator - Remoting > shut down. > 2019-11-27 10:35:45,696 INFO > akka.remote.RemoteActorRefProvider$RemotingTerminator - Remoting > shut down. > 2019-11-27 10:35:45,709 INFO > org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopped > Akka RPC service. > 2019-11-27 10:35:45,713 INFO > org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopped > Akka RPC service. > 2019-11-27 10:35:45,714 ERROR > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Could not > start cluster entrypoint YarnSess > org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to > initialize the cluster entrypoint YarnSessionClusterEntrypoint. > at > org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:182) > at > org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:501) > at > org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint.main(YarnSessionClusterEntrypoint.java:93) > Caused by: org.apache.flink.util.FlinkException: Could not create the > DispatcherResourceManagerComponent. > at > org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResou > at > org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:210) > at > org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) > at > org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) > at > org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:163) > ... 2 more > Caused by: java.net.BindException: Could not start rest endpoint on any > port in port range 8082 > at > org.apache.flink.runtime.rest.RestServerEndpoint.start(RestServerEndpoint.java:219) > at > org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResou > ... 9 more > > End of LogType:jobmanager.log > > === > 请问这种问题要怎么解决？多谢各位大佬！ > > > >
Re: flink on yarn 指定节点开启 yarn session 报错
您好，目前ML不支持图片，能将配置贴上来么？ 另外看报错信息像是8082端口被占用导致rest服务起不起来 Best, Yangze Guo On Wed, Nov 27, 2019 at 10:56 AM 杨浩程 wrote: > 各位大佬好！ >请教各位个问题：使用的flink 版本1.9.0。测试flink on yarn 指定节点 开启yarn session 会话。 >希望yarnsession开启的集群 jobmanager开启在我指定的节点上。 >更改的配置如下： > 报错信息如下： > > === > 2019-11-27 10:35:45,640 INFO > org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl > - backgroundOperati > 2019-11-27 10:35:45,650 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - > Session: 0x36ea133e2b51435 closed > 2019-11-27 10:35:45,650 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - > EventThread shut down for session: 0x3 > 2019-11-27 10:35:45,651 INFO > org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopping > Akka RPC service. > 2019-11-27 10:35:45,659 INFO > org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopping > Akka RPC service. > 2019-11-27 10:35:45,666 INFO > akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting > down remote daemon. > 2019-11-27 10:35:45,669 INFO > akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote > daemon shut down; proceeding with fl > 2019-11-27 10:35:45,676 INFO > akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting > down remote daemon. > 2019-11-27 10:35:45,677 INFO > akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote > daemon shut down; proceeding with fl > 2019-11-27 10:35:45,696 INFO > akka.remote.RemoteActorRefProvider$RemotingTerminator - Remoting > shut down. > 2019-11-27 10:35:45,696 INFO > akka.remote.RemoteActorRefProvider$RemotingTerminator - Remoting > shut down. > 2019-11-27 10:35:45,709 INFO > org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopped > Akka RPC service. > 2019-11-27 10:35:45,713 INFO > org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopped > Akka RPC service. > 2019-11-27 10:35:45,714 ERROR > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Could not > start cluster entrypoint YarnSess > org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to > initialize the cluster entrypoint YarnSessionClusterEntrypoint. > at > org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:182) > at > org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:501) > at > org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint.main(YarnSessionClusterEntrypoint.java:93) > Caused by: org.apache.flink.util.FlinkException: Could not create the > DispatcherResourceManagerComponent. > at > org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResou > at > org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:210) > at > org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) > at > org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) > at > org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:163) > ... 2 more > Caused by: java.net.BindException: Could not start rest endpoint on any > port in port range 8082 > at > org.apache.flink.runtime.rest.RestServerEndpoint.start(RestServerEndpoint.java:219) > at > org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResou > ... 9 more > > End of LogType:jobmanager.log > > === > 请问这种问题要怎么解决？多谢各位大佬！ > > > >
Re:Re: flink on yarn 指定节点开启 yarn session 报错
tison 你好， 我netstat查看了想要指定的节点的端口情况，8082没有被占用。所以有点困惑这个报错。 在 2019-11-27 10:59:52，"tison" 写道： >8082 端口被占用了吧？ > >Best, >tison. > > >杨浩程 于2019年11月27日周三 上午10:56写道： > >> 各位大佬好！ >>请教各位个问题：使用的flink 版本1.9.0。测试flink on yarn 指定节点 开启yarn session 会话。 >>希望yarnsession开启的集群 jobmanager开启在我指定的节点上。 >>更改的配置如下： >> 报错信息如下： >> >> === >> 2019-11-27 10:35:45,640 INFO >> org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl >> - backgroundOperati >> 2019-11-27 10:35:45,650 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - >> Session: 0x36ea133e2b51435 closed >> 2019-11-27 10:35:45,650 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - >> EventThread shut down for session: 0x3 >> 2019-11-27 10:35:45,651 INFO >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopping >> Akka RPC service. >> 2019-11-27 10:35:45,659 INFO >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopping >> Akka RPC service. >> 2019-11-27 10:35:45,666 INFO >> akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting >> down remote daemon. >> 2019-11-27 10:35:45,669 INFO >> akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote >> daemon shut down; proceeding with fl >> 2019-11-27 10:35:45,676 INFO >> akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting >> down remote daemon. >> 2019-11-27 10:35:45,677 INFO >> akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote >> daemon shut down; proceeding with fl >> 2019-11-27 10:35:45,696 INFO >> akka.remote.RemoteActorRefProvider$RemotingTerminator - Remoting >> shut down. >> 2019-11-27 10:35:45,696 INFO >> akka.remote.RemoteActorRefProvider$RemotingTerminator - Remoting >> shut down. >> 2019-11-27 10:35:45,709 INFO >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopped >> Akka RPC service. >> 2019-11-27 10:35:45,713 INFO >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopped >> Akka RPC service. >> 2019-11-27 10:35:45,714 ERROR >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Could not >> start cluster entrypoint YarnSess >> org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to >> initialize the cluster entrypoint YarnSessionClusterEntrypoint. >> at >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:182) >> at >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:501) >> at >> org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint.main(YarnSessionClusterEntrypoint.java:93) >> Caused by: org.apache.flink.util.FlinkException: Could not create the >> DispatcherResourceManagerComponent. >> at >> org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResou >> at >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:210) >> at >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:164) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:422) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) >> at >> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) >> at >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:163) >> ... 2 more >> Caused by: java.net.BindException: Could not start rest endpoint on any >> port in port range 8082 >> at >> org.apache.flink.runtime.rest.RestServerEndpoint.start(RestServerEndpoint.java:219) >> at >> org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResou >> ... 9 more >> >> End of LogType:jobmanager.log >> >> === >> 请问这种问题要怎么解决？多谢各位大佬！ >> >> >> >>
Re:Re: flink on yarn 指定节点开启 yarn session 报错
好的，更改过的配置如下： #== # Rest & web frontend #== # The port to which the REST client connects to. If rest.bind-port has # not been specified, then the server will bind to this port as well. # rest.port: 8082 # The address to which the REST client will connect to # rest.address: bigdata-test-8 # Port range for the REST and web server to bind to. # #rest.bind-port: 8080-8090 rest.bind-port: 8082 # The address that the REST & web server binds to # rest.bind-address: bigdata-test-8 # Flag to specify whether job submission is enabled from the web-based # runtime monitor. Uncomment to disable. #web.submit.enable: false #web.upload.dir: /data/flink-1.9.0/upload_jars/ 在 2019-11-27 11:01:47，"Yangze Guo" 写道： >您好，目前ML不支持图片，能将配置贴上来么？ >另外看报错信息像是8082端口被占用导致rest服务起不起来 > >Best, >Yangze Guo > > >On Wed, Nov 27, 2019 at 10:56 AM 杨浩程 wrote: > >> 各位大佬好！ >>请教各位个问题：使用的flink 版本1.9.0。测试flink on yarn 指定节点 开启yarn session 会话。 >>希望yarnsession开启的集群 jobmanager开启在我指定的节点上。 >>更改的配置如下： >> 报错信息如下： >> >> === >> 2019-11-27 10:35:45,640 INFO >> org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl >> - backgroundOperati >> 2019-11-27 10:35:45,650 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - >> Session: 0x36ea133e2b51435 closed >> 2019-11-27 10:35:45,650 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - >> EventThread shut down for session: 0x3 >> 2019-11-27 10:35:45,651 INFO >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopping >> Akka RPC service. >> 2019-11-27 10:35:45,659 INFO >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopping >> Akka RPC service. >> 2019-11-27 10:35:45,666 INFO >> akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting >> down remote daemon. >> 2019-11-27 10:35:45,669 INFO >> akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote >> daemon shut down; proceeding with fl >> 2019-11-27 10:35:45,676 INFO >> akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting >> down remote daemon. >> 2019-11-27 10:35:45,677 INFO >> akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote >> daemon shut down; proceeding with fl >> 2019-11-27 10:35:45,696 INFO >> akka.remote.RemoteActorRefProvider$RemotingTerminator - Remoting >> shut down. >> 2019-11-27 10:35:45,696 INFO >> akka.remote.RemoteActorRefProvider$RemotingTerminator - Remoting >> shut down. >> 2019-11-27 10:35:45,709 INFO >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopped >> Akka RPC service. >> 2019-11-27 10:35:45,713 INFO >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopped >> Akka RPC service. >> 2019-11-27 10:35:45,714 ERROR >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Could not >> start cluster entrypoint YarnSess >> org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to >> initialize the cluster entrypoint YarnSessionClusterEntrypoint. >> at >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:182) >> at >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:501) >> at >> org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint.main(YarnSessionClusterEntrypoint.java:93) >> Caused by: org.apache.flink.util.FlinkException: Could not create the >> DispatcherResourceManagerComponent. >> at >> org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResou >> at >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:210) >> at >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:164) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:422) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) >> at >> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) >> at >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:163) >> ... 2 more >> Caused by: java.net.BindException: Could not start rest endpoint on any >> port in port range 8082 >> at >> org.apache.flink.runtime.rest.RestServerEndpoint.start(RestServerEndpoint.java:219) >> at >> org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManager
Re: Re: flink on yarn 指定节点开启 yarn session 报错
8082我记得是historyserver.web.port的默认值，很可能是冲突了改成8081呢？ Best, Yangze Guo On Wed, Nov 27, 2019 at 11:13 AM 杨浩程 wrote: > > > 好的，更改过的配置如下： > > #== > # Rest & web frontend > #== > > > # The port to which the REST client connects to. If rest.bind-port has > # not been specified, then the server will bind to this port as well. > # > rest.port: 8082 > > > # The address to which the REST client will connect to > # > rest.address: bigdata-test-8 > > > # Port range for the REST and web server to bind to. > # > #rest.bind-port: 8080-8090 > rest.bind-port: 8082 > > > # The address that the REST & web server binds to > # > rest.bind-address: bigdata-test-8 > > > # Flag to specify whether job submission is enabled from the web-based > # runtime monitor. Uncomment to disable. > > > #web.submit.enable: false > #web.upload.dir: /data/flink-1.9.0/upload_jars/ > > > > > > > 在 2019-11-27 11:01:47，"Yangze Guo" 写道： > >您好，目前ML不支持图片，能将配置贴上来么？ > >另外看报错信息像是8082端口被占用导致rest服务起不起来 > > > >Best, > >Yangze Guo > > > > > >On Wed, Nov 27, 2019 at 10:56 AM 杨浩程 wrote: > > > >> 各位大佬好！ > >>请教各位个问题：使用的flink 版本1.9.0。测试flink on yarn 指定节点 开启yarn session 会话。 > >>希望yarnsession开启的集群 jobmanager开启在我指定的节点上。 > >>更改的配置如下： > >> 报错信息如下： > >> > >> === > >> 2019-11-27 10:35:45,640 INFO > >> org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl > >> - backgroundOperati > >> 2019-11-27 10:35:45,650 INFO > >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - > >> Session: 0x36ea133e2b51435 closed > >> 2019-11-27 10:35:45,650 INFO > >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - > >> EventThread shut down for session: 0x3 > >> 2019-11-27 10:35:45,651 INFO > >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopping > >> Akka RPC service. > >> 2019-11-27 10:35:45,659 INFO > >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopping > >> Akka RPC service. > >> 2019-11-27 10:35:45,666 INFO > >> akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting > >> down remote daemon. > >> 2019-11-27 10:35:45,669 INFO > >> akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote > >> daemon shut down; proceeding with fl > >> 2019-11-27 10:35:45,676 INFO > >> akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting > >> down remote daemon. > >> 2019-11-27 10:35:45,677 INFO > >> akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote > >> daemon shut down; proceeding with fl > >> 2019-11-27 10:35:45,696 INFO > >> akka.remote.RemoteActorRefProvider$RemotingTerminator - Remoting > >> shut down. > >> 2019-11-27 10:35:45,696 INFO > >> akka.remote.RemoteActorRefProvider$RemotingTerminator - Remoting > >> shut down. > >> 2019-11-27 10:35:45,709 INFO > >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopped > >> Akka RPC service. > >> 2019-11-27 10:35:45,713 INFO > >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - Stopped > >> Akka RPC service. > >> 2019-11-27 10:35:45,714 ERROR > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Could not > >> start cluster entrypoint YarnSess > >> org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to > >> initialize the cluster entrypoint YarnSessionClusterEntrypoint. > >> at > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:182) > >> at > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:501) > >> at > >> org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint.main(YarnSessionClusterEntrypoint.java:93) > >> Caused by: org.apache.flink.util.FlinkException: Could not create the > >> DispatcherResourceManagerComponent. > >> at > >> org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResou > >> at > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:210) > >> at > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:164) > >> at java.security.AccessController.doPrivileged(Native Method) > >> at javax.security.auth.Subject.doAs(Subject.java:422) > >> at > >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) > >> at > >> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) > >> at > >> org.apache.flink.runtime.entrypoint.Cluster
Re: Re: flink on yarn 指定节点开启 yarn session 报错
确实是 historyserver.web.port 的默认值，但我记得 HistoryServer 默认是不启动的 总之可以确认下问题能否稳定复现，失败的瞬间端口有无被占用，以及换个端口能不能起来 如果还不行再找找其他原因 Best, tison. Yangze Guo 于2019年11月27日周三 上午11:23写道： > 8082我记得是historyserver.web.port的默认值，很可能是冲突了改成8081呢？ > > Best, > Yangze Guo > > On Wed, Nov 27, 2019 at 11:13 AM 杨浩程 wrote: > > > > > > > 好的，更改过的配置如下： > > > > > #== > > # Rest & web frontend > > > #== > > > > > > # The port to which the REST client connects to. If rest.bind-port has > > # not been specified, then the server will bind to this port as well. > > # > > rest.port: 8082 > > > > > > # The address to which the REST client will connect to > > # > > rest.address: bigdata-test-8 > > > > > > # Port range for the REST and web server to bind to. > > # > > #rest.bind-port: 8080-8090 > > rest.bind-port: 8082 > > > > > > # The address that the REST & web server binds to > > # > > rest.bind-address: bigdata-test-8 > > > > > > # Flag to specify whether job submission is enabled from the web-based > > # runtime monitor. Uncomment to disable. > > > > > > #web.submit.enable: false > > #web.upload.dir: /data/flink-1.9.0/upload_jars/ > > > > > > > > > > > > > > > 在 2019-11-27 11:01:47，"Yangze Guo" 写道： > > >您好，目前ML不支持图片，能将配置贴上来么？ > > >另外看报错信息像是8082端口被占用导致rest服务起不起来 > > > > > >Best, > > >Yangze Guo > > > > > > > > >On Wed, Nov 27, 2019 at 10:56 AM 杨浩程 wrote: > > > > > >> 各位大佬好！ > > >>请教各位个问题：使用的flink 版本1.9.0。测试flink on yarn 指定节点 开启yarn session > 会话。 > > >>希望yarnsession开启的集群 jobmanager开启在我指定的节点上。 > > >>更改的配置如下： > > >> 报错信息如下： > > >> > > >> > === > > >> 2019-11-27 10:35:45,640 INFO > > >> > org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl > > >> - backgroundOperati > > >> 2019-11-27 10:35:45,650 INFO > > >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - > > >> Session: 0x36ea133e2b51435 closed > > >> 2019-11-27 10:35:45,650 INFO > > >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - > > >> EventThread shut down for session: 0x3 > > >> 2019-11-27 10:35:45,651 INFO > > >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - > Stopping > > >> Akka RPC service. > > >> 2019-11-27 10:35:45,659 INFO > > >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - > Stopping > > >> Akka RPC service. > > >> 2019-11-27 10:35:45,666 INFO > > >> akka.remote.RemoteActorRefProvider$RemotingTerminator - > Shutting > > >> down remote daemon. > > >> 2019-11-27 10:35:45,669 INFO > > >> akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote > > >> daemon shut down; proceeding with fl > > >> 2019-11-27 10:35:45,676 INFO > > >> akka.remote.RemoteActorRefProvider$RemotingTerminator - > Shutting > > >> down remote daemon. > > >> 2019-11-27 10:35:45,677 INFO > > >> akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote > > >> daemon shut down; proceeding with fl > > >> 2019-11-27 10:35:45,696 INFO > > >> akka.remote.RemoteActorRefProvider$RemotingTerminator - > Remoting > > >> shut down. > > >> 2019-11-27 10:35:45,696 INFO > > >> akka.remote.RemoteActorRefProvider$RemotingTerminator - > Remoting > > >> shut down. > > >> 2019-11-27 10:35:45,709 INFO > > >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - > Stopped > > >> Akka RPC service. > > >> 2019-11-27 10:35:45,713 INFO > > >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - > Stopped > > >> Akka RPC service. > > >> 2019-11-27 10:35:45,714 ERROR > > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Could > not > > >> start cluster entrypoint YarnSess > > >> org.apache.flink.runtime.entrypoint.ClusterEntrypointException: > Failed to > > >> initialize the cluster entrypoint YarnSessionClusterEntrypoint. > > >> at > > >> > org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:182) > > >> at > > >> > org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:501) > > >> at > > >> > org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint.main(YarnSessionClusterEntrypoint.java:93) > > >> Caused by: org.apache.flink.util.FlinkException: Could not create the > > >> DispatcherResourceManagerComponent. > > >> at > > >> > org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResou > > >> at > > >> > org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:210) > > >> at > > >> > org.apache.flink.runtime.entrypoi
Re:Re: Re: flink on yarn 指定节点开启 yarn session 报错
hi，我确保，端口没占用的情况下更换rest.port: 8072，且更改了配置： #== # HistoryServer #== # The HistoryServer is started and stopped via bin/historyserver.sh (start|stop) # Directory to upload completed jobs to. Add this directory to the list of # monitored directories of the HistoryServer as well (see below). jobmanager.archive.fs.dir: hdfs://bigdata-test-8:18082/completed-jobs/ # The address under which the web-based HistoryServer listens. historyserver.web.address: bigdata-test-8 # The port under which the web-based HistoryServer listens. historyserver.web.port: 18082 # Comma separated list of directories to monitor for completed jobs. historyserver.archive.fs.dir: hdfs://bigdata-test-8:18082/completed-jobs/ # Interval in milliseconds for refreshing the monitored directories. historyserver.archive.fs.refresh-interval: 1 但在这种情况下启动yarn session 报错信息如下： = 2019-11-27 14:50:40,981 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Could not start cluster entrypoint YarnSessionClusterEntrypoint. org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to initialize the cluster entrypoint YarnSessionClusterEntrypoint. at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:182) at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:501) at org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint.main(YarnSessionClusterEntrypoint.java:93) Caused by: org.apache.flink.util.FlinkException: Could not create the DispatcherResourceManagerComponent. at org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResourceManagerComponentFactory.java:257) at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:210) at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:163) ... 2 more Caused by: java.net.BindException: Could not start rest endpoint on any port in port range 8070-8090 at org.apache.flink.runtime.rest.RestServerEndpoint.start(RestServerEndpoint.java:219) at org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResourceManagerComponentFactory.java:161) ... 9 more 在 2019-11-27 11:28:06，"tison" 写道： >确实是 historyserver.web.port 的默认值，但我记得 HistoryServer 默认是不启动的 > >总之可以确认下问题能否稳定复现，失败的瞬间端口有无被占用，以及换个端口能不能起来 > >如果还不行再找找其他原因 > >Best, >tison. > > >Yangze Guo 于2019年11月27日周三 上午11:23写道： > >> 8082我记得是historyserver.web.port的默认值，很可能是冲突了改成8081呢？ >> >> Best, >> Yangze Guo >> >> On Wed, Nov 27, 2019 at 11:13 AM 杨浩程 wrote: >> > >> > >> >> > 好的，更改过的配置如下： >> > >> > >> #== >> > # Rest & web frontend >> > >> #== >> > >> > >> > # The port to which the REST client connects to. If rest.bind-port has >> > # not been specified, then the server will bind to this port as well. >> > # >> > rest.port: 8082 >> > >> > >> > # The address to which the REST client will connect to >> > # >> > rest.address: bigdata-test-8 >> > >> > >> > # Port range for the REST and web server to bind to. >> > # >> > #rest.bind-port: 8080-8090 >> > rest.bind-port: 8082 >> > >> > >> > # The address that the REST & web server binds to >> > # >> > rest.bind-address: bigdata-test-8 >> > >> > >> > # Flag to specify whether job submission is enabled from the web-based >> > # runtime monitor. Uncomment to disable. >> > >> > >> > #web.submit.enable: false >> > #web.upload.dir: /data/flink-1.9.0/upload_jars/ >> > >> >> > >> > >> > >> > >> > >> > 在 2019-11-27 11:01:47，"Yangze Guo" 写道： >> > >您好，目前ML不支持图片，能将配置贴上来么？ >> > >另外看报错信息像是8082端口被占用导致rest服务起不起来 >> > > >> > >Best, >> > >Yangze Guo >> > > >> > > >> > >On Wed, Nov 27, 2019 at 10:56 AM 杨浩程 wrote: >> > > >> > >> 各位大佬好！ >> > >>请教各位个问题：使用的flink 版本1.9.0。测试flink on yarn 指定节点 开启yarn session >> 会话。 >> >
