没有人遇到这类问题吗？
[docker@master flink-1.13.2]$ ./bin/yarn-session.sh SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/docker/flink-1.13.2/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/docker/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 2021-11-08 14:36:43,651 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.address, master 2021-11-08 14:36:43,657 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.port, 6123 2021-11-08 14:36:43,657 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.process.size, 1600m 2021-11-08 14:36:43,657 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.process.size, 1728m 2021-11-08 14:36:43,657 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.numberOfTaskSlots, 8 2021-11-08 14:36:43,657 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: parallelism.default, 1 2021-11-08 14:36:43,658 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.checkpointing.interval, 30s 2021-11-08 14:36:43,658 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.checkpointing.unaligned, true 2021-11-08 14:36:43,658 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.checkpointing.timeout, 1200s 2021-11-08 14:36:43,658 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: state.backend, filesystem 2021-11-08 14:36:43,658 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: state.checkpoints.dir, oss://datalake-huifu/hudi/flink/checkpoints 2021-11-08 14:36:43,659 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: state.savepoints.dir, oss://datalake-huifu/hudi/flink/savepoints 2021-11-08 14:36:43,659 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: state.backend.incremental, true 2021-11-08 14:36:43,659 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.execution.failover-strategy, region 2021-11-08 14:36:43,659 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: rest.port, 18081 2021-11-08 14:36:43,660 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: classloader.resolve-order, parent-first 2021-11-08 14:36:43,660 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: web.submit.enabled, false 2021-11-08 14:36:43,660 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: fs.oss.endpoint, oss-cn-shanghai.aliyuncs.com 2021-11-08 14:36:43,660 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: fs.oss.accessKeyId, LTAI5tJ4k9pk1KwZVsLd8NHd 2021-11-08 14:36:43,661 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: fs.oss.accessKeySecret, ****** 2021-11-08 14:36:44,042 WARN org.apache.hadoop.util.NativeCodeLoader [] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2021-11-08 14:36:44,155 INFO org.apache.flink.runtime.security.modules.HadoopModule [] - Hadoop user set to docker (auth:SIMPLE) 2021-11-08 14:36:44,171 INFO org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file will be created as /tmp/jaas-5414790866026859677.conf. 2021-11-08 14:36:44,213 WARN org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The configuration directory ('/home/docker/flink-1.13.2/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file. 2021-11-08 14:36:44,340 INFO org.apache.hadoop.yarn.client.RMProxy [] - Connecting to ResourceManager at master/192.168.16.191:8032 2021-11-08 14:36:44,711 INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (160.000mb (167772162 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead 2021-11-08 14:36:44,734 INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (172.800mb (181193935 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead 2021-11-08 14:36:45,025 INFO org.apache.hadoop.conf.Configuration [] - resource-types.xml not found 2021-11-08 14:36:45,026 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils [] - Unable to find 'resource-types.xml'. 2021-11-08 14:36:45,091 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - The configured JobManager memory is 1600 MB. YARN will allocate 2048 MB to make up an integer multiple of its minimum allocation memory (1024 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 448 MB may not be used by Flink. 2021-11-08 14:36:45,092 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - The configured TaskManager memory is 1728 MB. YARN will allocate 2048 MB to make up an integer multiple of its minimum allocation memory (1024 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 320 MB may not be used by Flink. 2021-11-08 14:36:45,092 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster specification: ClusterSpecification{masterMemoryMB=1600, taskManagerMemoryMB=1728, slotsPerTaskManager=8} 2021-11-08 14:36:45,847 INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2021-11-08 14:36:45,979 INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2021-11-08 14:36:46,004 INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2021-11-08 14:36:46,032 INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2021-11-08 14:36:46,191 INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2021-11-08 14:36:46,248 INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2021-11-08 14:36:46,287 INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2021-11-08 14:36:46,325 INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2021-11-08 14:36:46,368 INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2021-11-08 14:36:46,546 INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false ............. 一直输出 INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 查看yarn控台没有flink session集群创建 At 2021-11-04 21:18:03, "casel.chen" <casel_c...@126.com> wrote: >flink-conf.yaml 内容如下 > > >################################################################################ ># Licensed to the Apache Software Foundation (ASF) under one ># or more contributor license agreements. See the NOTICE file ># distributed with this work for additional information ># regarding copyright ownership. The ASF licenses this file ># to you under the Apache License, Version 2.0 (the ># "License"); you may not use this file except in compliance ># with the License. You may obtain a copy of the License at ># ># http://www.apache.org/licenses/LICENSE-2.0 ># ># Unless required by applicable law or agreed to in writing, software ># distributed under the License is distributed on an "AS IS" BASIS, ># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ># See the License for the specific language governing permissions and ># limitations under the License. >################################################################################ > > > > >#============================================================================== ># Common >#============================================================================== > > ># The external address of the host on which the JobManager runs and can be ># reached by the TaskManagers and any clients which want to connect. This >setting ># is only used in Standalone mode and may be overwritten on the JobManager side ># by specifying the --host <hostname> parameter of the bin/jobmanager.sh >executable. ># In high availability mode, if you use the bin/start-cluster.sh script and >setup ># the conf/masters file, this will be taken care of automatically. Yarn/Mesos ># automatically configure the host name based on the hostname of the node >where the ># JobManager runs. > > >jobmanager.rpc.address: master > > ># The RPC port where the JobManager is reachable. > > >jobmanager.rpc.port: 6123 > > > > ># The total process memory size for the JobManager. ># ># Note this accounts for all memory usage within the JobManager process, >including JVM metaspace and other overhead. > > >jobmanager.memory.process.size: 1600m > > > > ># The total process memory size for the TaskManager. ># ># Note this accounts for all memory usage within the TaskManager process, >including JVM metaspace and other overhead. > > >taskmanager.memory.process.size: 1728m > > ># To exclude JVM metaspace and overhead, please, use total Flink memory size >instead of 'taskmanager.memory.process.size'. ># It is not recommended to set both 'taskmanager.memory.process.size' and >Flink memory. ># ># taskmanager.memory.flink.size: 1280m > > ># The number of task slots that each TaskManager offers. Each slot runs one >parallel pipeline. > > >taskmanager.numberOfTaskSlots: 8 > > ># The parallelism used for programs that did not specify and other parallelism. > > >parallelism.default: 1 > > ># The default file system scheme and authority. ># ># By default file paths without scheme are interpreted relative to the local ># root file system 'file:///'. Use this to override the default and interpret ># relative paths relative to a different file system, ># for example 'hdfs://mynamenode:12345' ># ># fs.default-scheme > > >#============================================================================== ># High Availability >#============================================================================== > > ># The high-availability mode. Possible options are 'NONE' or 'zookeeper'. ># ># high-availability: zookeeper > > ># The path where metadata for master recovery is persisted. While ZooKeeper >stores ># the small ground truth for checkpoint and leader election, this location >stores ># the larger objects, like persisted dataflow graphs. ># ># Must be a durable file system that is accessible from all nodes ># (like HDFS, S3, Ceph, nfs, ...) ># ># high-availability.storageDir: hdfs:///flink/ha/ > > ># The list of ZooKeeper quorum peers that coordinate the high-availability ># setup. This must be a list of the form: ># "host1:clientPort,host2:clientPort,..." (default clientPort: 2181) ># ># high-availability.zookeeper.quorum: localhost:2181 > > > > ># ACL options are based on >https://zookeeper.apache.org/doc/r3.1.2/zookeeperProgrammers.html#sc_BuiltinACLSchemes ># It can be either "creator" (ZOO_CREATE_ALL_ACL) or "open" >(ZOO_OPEN_ACL_UNSAFE) ># The default value is "open" and it can be changed to "creator" if ZK >security is enabled ># ># high-availability.zookeeper.client.acl: open > > >#============================================================================== ># Fault tolerance and checkpointing >#============================================================================== > > >execution.checkpointing.interval: 60s > > >execution.checkpointing.unaligned: true > > >execution.checkpointing.timeout: 1200s > > ># The backend that will be used to store operator state checkpoints if ># checkpointing is enabled. ># ># Supported backends are 'jobmanager', 'filesystem', 'rocksdb', or the ># <class-name-of-factory>. ># >state.backend: filesystem > > ># Directory for checkpoints filesystem, when using any of the default bundled ># state backends. ># >state.checkpoints.dir: oss://datalake-huifu/hudi/flink/checkpoints > > ># Default target directory for savepoints, optional. ># >state.savepoints.dir: oss://datalake-huifu/hudi/flink/savepoints > > ># Flag to enable/disable incremental checkpoints for backends that ># support incremental checkpoints (like the RocksDB state backend). ># >state.backend.incremental: true > > ># The failover strategy, i.e., how the job computation recovers from task >failures. ># Only restart tasks that may have been affected by the task failure, which >typically includes ># downstream tasks and potentially upstream tasks if their produced data is no >longer available for consumption. > > >jobmanager.execution.failover-strategy: region > > >#============================================================================== ># Rest & web frontend >#============================================================================== > > ># The port to which the REST client connects to. If rest.bind-port has ># not been specified, then the server will bind to this port as well. ># >rest.port: 18081 > > ># The address to which the REST client will connect to ># >#rest.address: 0.0.0.0 > > ># Port range for the REST and web server to bind to. ># >#rest.bind-port: 8080-8090 > > ># The address that the REST & web server binds to ># >#rest.bind-address: 0.0.0.0 > > ># Flag to specify whether job submission is enabled from the web-based ># runtime monitor. Uncomment to disable. > > >#web.submit.enable: false > > >#============================================================================== ># Advanced >#============================================================================== > > ># Override the directories for temporary files. If not specified, the ># system-specific Java temporary directory (java.io.tmpdir property) is taken. ># ># For framework setups on Yarn or Mesos, Flink will automatically pick up the ># containers' temp directories without any need for configuration. ># ># Add a delimited list for multiple directories, using the system directory ># delimiter (colon ':' on unix) or a comma, e.g.: ># /data1/tmp:/data2/tmp:/data3/tmp ># ># Note: Each directory entry is read from and written to by a different I/O ># thread. You can include the same directory multiple times in order to create ># multiple I/O threads against that directory. This is for example relevant for ># high-throughput RAIDs. ># ># io.tmp.dirs: /tmp > > ># The classloading resolve order. Possible values are 'child-first' (Flink's >default) ># and 'parent-first' (Java's default). ># ># Child first classloading allows users to use different dependency/library ># versions in their application than those in the classpath. Switching back ># to 'parent-first' may help with debugging dependency issues. ># >classloader.resolve-order: parent-first > > ># The amount of memory going to the network stack. These numbers usually need ># no tuning. Adjusting them may be necessary in case of an "Insufficient number ># of network buffers" error. The default min is 64MB, the default max is 1GB. ># ># taskmanager.memory.network.fraction: 0.1 ># taskmanager.memory.network.min: 64mb ># taskmanager.memory.network.max: 1gb > > >#============================================================================== ># Flink Cluster Security Configuration >#============================================================================== > > ># Kerberos authentication for various components - Hadoop, ZooKeeper, and >connectors - ># may be enabled in four steps: ># 1. configure the local krb5.conf file ># 2. provide Kerberos credentials (either a keytab or a ticket cache w/ kinit) ># 3. make the credentials available to various JAAS login contexts ># 4. configure the connector to use JAAS/SASL > > ># The below configure how Kerberos credentials are provided. A keytab will be >used instead of ># a ticket cache if the keytab path and principal are set. > > ># security.kerberos.login.use-ticket-cache: true ># security.kerberos.login.keytab: /path/to/kerberos/keytab ># security.kerberos.login.principal: flink-user > > ># The configuration below defines which JAAS login contexts > > ># security.kerberos.login.contexts: Client,KafkaClient > > >#============================================================================== ># ZK Security Configuration >#============================================================================== > > ># Below configurations are applicable if ZK ensemble is configured for security > > ># Override below configuration to provide custom ZK service name if configured ># zookeeper.sasl.service-name: zookeeper > > ># The configuration below must match one of the values set in >"security.kerberos.login.contexts" ># zookeeper.sasl.login-context-name: Client > > >#============================================================================== ># HistoryServer >#============================================================================== > > ># The HistoryServer is started and stopped via bin/historyserver.sh >(start|stop) > > ># Directory to upload completed jobs to. Add this directory to the list of ># monitored directories of the HistoryServer as well (see below). >#jobmanager.archive.fs.dir: hdfs:///completed-jobs/ > > ># The address under which the web-based HistoryServer listens. >#historyserver.web.address: 0.0.0.0 > > ># The port under which the web-based HistoryServer listens. >#historyserver.web.port: 8082 > > ># Comma separated list of directories to monitor for completed jobs. >#historyserver.archive.fs.dir: hdfs:///completed-jobs/ > > ># Interval in milliseconds for refreshing the monitored directories. >#historyserver.archive.fs.refresh-interval: 10000 > > >fs.oss.endpoint: oss-cn-shanghai.aliyuncs.com >fs.oss.accessKeyId: xxx >fs.oss.accessKeySecret: xxx > > >metrics.reporter.promgateway.class: >org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter >metrics.reporter.promgateway.host: 192.168.24.41 >metrics.reporter.promgateway.port: 9091 >metrics.reporter.promgateway.jobName: PerfTest >metrics.reporter.promgateway.randomJobNameSuffix: true >metrics.reporter.promgateway.deleteOnShutdown: false > > > > > > > > > > > > > > > > > > > > > > > > > >在 2021-11-04 19:32:45，"Caizhi Weng" <tsreape...@gmail.com> 写道： >>Hi！ >> >>没有在邮件里发现附件，可以考虑把 flink-conf.yaml 的内容贴在邮件里，或者外部剪贴板。 >> >>casel.chen <casel_c...@126.com> 于2021年11月4日周四 下午6:42写道： >> >>> flink 1.13.2 + hadoop 3.2.1 >>> yarn上已经成功跑了hive和spark作业 >>> flink上通过运行 bin/yarn-session.sh 启动yarn session集群的时候一直报如下INFO日志，查看yarn web >>> console发现并没有启flink-session集群，我的flink-conf.yaml配置如附件，hadoop集群并没有开启认证SSL之类的，改用standalone模式是可以启动3节点集群的，请问这会是什么原因造成的？要怎么修复？谢谢！ >>> >>> >>> >>> 2021-11-04 16:51:39,964 INFO >>> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] >>> - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted >>> = false >>> >>> 2021-11-04 16:51:39,986 INFO >>> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] >>> - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted >>> = false >>> >>> 2021-11-04 16:51:40,004 INFO >>> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] >>> - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted >>> = false >>> >>> 2021-11-04 16:51:40,020 INFO >>> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] >>> - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted >>> = false >>> >>> 2021-11-04 16:51:40,041 INFO >>> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] >>> - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted >>> = false >>> >>> 2021-11-04 16:51:40,059 INFO >>> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] >>> - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted >>> = false >>> >>> 2021-11-04 16:51:40,078 INFO >>> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] >>> - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted >>> = false >>> >>> 2021-11-04 16:51:40,097 INFO >>> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] >>> - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted >>> = false >>> >>> 2021-11-04 16:51:40,114 INFO >>> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] >>> - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted >>> = false >>> >>> 2021-11-04 16:51:40,134 INFO >>> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] >>> - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted >>> = false >>> >>> 2021-11-04 16:51:40,155 INFO >>> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] >>> - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted >>> = false >>> >>> 2021-11-04 16:51:40,175 INFO >>> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] >>> - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted >>> = false >>> >>> 2021-11-04 16:51:40,193 INFO >>> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] >>> - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted >>> = false >>> >>> 2021-11-04 16:51:40,212 INFO >>> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient [] >>> - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted >>> = false >>> >>> >>> >>> >>> >>>
