取消订阅

2021-11-07 文章 tanggen...@163.com 
取消订阅


tanggen...@163.com

退订

2021-10-14 文章 tanggen...@163.com 

退订


tanggen...@163.com

退订

2021-10-14 文章 tanggen...@163.com 
退订



tanggen...@163.com

flink Kafka producer执行出错

2021-04-30 文章 tanggen...@163.com 
我在flink处理消费kafka的一个topic，将迟到数据通过侧流发送到一个新的topic，基本上是一直报这个错，每次提交checkpoint时都会报这个错，然后就会重启
还请指导一下，需要做些其它的设置吗
2021-04-30 17:00:51
org.apache.flink.streaming.connectors.kafka.FlinkKafkaException: Failed to send 
data to Kafka: Producer attempted an operation with an old epoch. Either there 
is a newer producer with the same transactionalId, or the producer's 
transaction has been expired by the broker.
at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.checkErroneous(FlinkKafkaProducer.java:1282)
at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.invoke(FlinkKafkaProducer.java:816)
at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.invoke(FlinkKafkaProducer.java:99)
at 
org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.invoke(TwoPhaseCommitSinkFunction.java:235)
at 
org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:56)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:717)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:703)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$BroadcastingOutputCollector.collect(OperatorChain.java:794)
at 
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:58)
at 
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.sideOutput(WindowOperator.java:558)
at 
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processElement(WindowOperator.java:422)
at 
org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:161)
at 
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:178)
at 
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:153)
at 
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:67)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:351)
at 
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:185)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:569)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:534)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.common.errors.ProducerFencedException: Producer 
attempted an operation with an old epoch. Either there is a newer producer with 
the same transactionalId, or the producer's transaction has been expired by the 
broker.




tanggen...@163.com

flink在yarn集群上启动的问题

2021-04-21 文章 tanggen...@163.com 
:
 Start request for container_1618931441017_0004_03_01 by user root
2021-04-20 23:34:12,570 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
 TimelineService V2.0 is not enabled. Skipping updating flowContext for 
application application_1618931441017_0004
2021-04-20 23:34:12,571 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
 Adding container_1618931441017_0004_03_01 to application 
application_1618931441017_0004
2021-04-20 23:34:12,572 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
 Container container_1618931441017_0004_03_01 transitioned from NEW to 
LOCALIZING
2021-04-20 23:34:12,572 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got 
event CONTAINER_INIT for appId application_1618931441017_0004
2021-04-20 23:34:12,574 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
 Container container_1618931441017_0004_03_01 transitioned from LOCALIZING 
to SCHEDULED
2021-04-20 23:34:12,574 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler.ContainerScheduler:
 Starting container [container_1618931441017_0004_03_01]
2021-04-20 23:34:12,600 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
 Container container_1618931441017_0004_03_01 transitioned from SCHEDULED 
to RUNNING
2021-04-20 23:34:12,600 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
 Starting resource-monitoring for container_1618931441017_0004_03_01
2021-04-20 23:34:12,603 INFO 
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: 
launchContainer: [bash, 
/opt/hadoop/hadoopdata/nm-local-dir/usercache/root/appcache/application_1618931441017_0004/container_1618931441017_0004_03_01/default_container_executor.sh]
2021-04-20 23:34:12,905 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
 container_1618931441017_0004_03_01's ip = 10.100.8.108, and hostname = 
node108
2021-04-20 23:34:12,911 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
 Skipping monitoring container container_1618931441017_0004_03_01 since CPU 
usage is not yet available.
2021-04-20 23:34:16,067 WARN 
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code 
from container container_1618931441017_0004_03_01 is : 1
2021-04-20 23:34:16,067 WARN 
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception 
from container-launch with container ID: container_1618931441017_0004_03_01 
and exit code: 1
ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:1009)
at org.apache.hadoop.util.Shell.run(Shell.java:902)
at 
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1227)
at 
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:294)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:501)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:311)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:106)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-04-20 23:34:16,067 INFO 
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from 
container-launch.
2021-04-20 23:34:16,067 INFO 
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: 
container_1618931441017_0004_03_01
2021-04-20 23:34:16,067 INFO 
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1
2021-04-20 23:34:16,067 WARN 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
 Container launch failed : Container exited with a non-zero exit code 1.
2021-04-20 23:34:16,069 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
 Container container_1618931441017_0004_03_01 transitioned from RUNNING to 
EXITED_WITH_FAILURE
2021-04-20 23:34:16,069 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
 Cleaning up container container_1618931441017_0004_03_01



tanggen...@163.com

Application application_1618931441017_0004

2021-04-20 文章 tanggen...@163.com 
:
 Start request for container_1618931441017_0004_03_01 by user root
2021-04-20 23:34:12,570 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
 TimelineService V2.0 is not enabled. Skipping updating flowContext for 
application application_1618931441017_0004
2021-04-20 23:34:12,571 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
 Adding container_1618931441017_0004_03_01 to application 
application_1618931441017_0004
2021-04-20 23:34:12,572 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
 Container container_1618931441017_0004_03_01 transitioned from NEW to 
LOCALIZING
2021-04-20 23:34:12,572 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got 
event CONTAINER_INIT for appId application_1618931441017_0004
2021-04-20 23:34:12,574 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
 Container container_1618931441017_0004_03_01 transitioned from LOCALIZING 
to SCHEDULED
2021-04-20 23:34:12,574 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler.ContainerScheduler:
 Starting container [container_1618931441017_0004_03_01]
2021-04-20 23:34:12,600 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
 Container container_1618931441017_0004_03_01 transitioned from SCHEDULED 
to RUNNING
2021-04-20 23:34:12,600 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
 Starting resource-monitoring for container_1618931441017_0004_03_01
2021-04-20 23:34:12,603 INFO 
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: 
launchContainer: [bash, 
/opt/hadoop/hadoopdata/nm-local-dir/usercache/root/appcache/application_1618931441017_0004/container_1618931441017_0004_03_01/default_container_executor.sh]
2021-04-20 23:34:12,905 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
 container_1618931441017_0004_03_01's ip = 10.100.8.108, and hostname = 
node108
2021-04-20 23:34:12,911 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
 Skipping monitoring container container_1618931441017_0004_03_01 since CPU 
usage is not yet available.
2021-04-20 23:34:16,067 WARN 
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code 
from container container_1618931441017_0004_03_01 is : 1
2021-04-20 23:34:16,067 WARN 
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception 
from container-launch with container ID: container_1618931441017_0004_03_01 
and exit code: 1
ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:1009)
at org.apache.hadoop.util.Shell.run(Shell.java:902)
at 
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1227)
at 
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:294)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:501)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:311)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:106)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-04-20 23:34:16,067 INFO 
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from 
container-launch.
2021-04-20 23:34:16,067 INFO 
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: 
container_1618931441017_0004_03_01
2021-04-20 23:34:16,067 INFO 
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1
2021-04-20 23:34:16,067 WARN 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
 Container launch failed : Container exited with a non-zero exit code 1.
2021-04-20 23:34:16,069 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
 Container container_1618931441017_0004_03_01 transitioned from RUNNING to 
EXITED_WITH_FAILURE
2021-04-20 23:34:16,069 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
 Cleaning up container container_1618931441017_0004_03_01



tanggen...@163.com