FlinkKafkaProducer好像每次事务提交都会重连producer导致打印了大量log

2020-11-27 Thread Wz


下面是addSink的代码:
result.addSink(new FlinkKafkaProducer(DataSourceConfig.ResultTopic,new
MyKafkaSerializationSchema(DataSourceConfig.ResultTopic),ConnectToKafka.getKafKaProducerProperties(),FlinkKafkaProducer.Semantic.EXACTLY_ONCE,
3)).setParallelism(1);

KafkaProducer配置信息:
props_Producer.put("bootstrap.servers",
DataSourceConfig.bootstrapServersIPAddress);
props_Producer.put("acks","all");
props_Producer.put("request.timeout.ms", 3000);


总之我也不太清楚为什么会反复打印下面的链接时才会打印的log,推测是一直在重新连接,几乎无间断的打印下面的log给磁盘撑爆了。请教各位大佬可能的原因?


2020-11-20 15:55:56,672 INFO 
org.apache.kafka.clients.producer.KafkaProducer  [] - [Producer
clientId=producer-CepOperator -> Sink:
Unnamed-1848139bf30d999062379bb9e1d14fd8-2, transactionalId=CepOperator ->
Sink: Unnamed-1848139bf30d999062379bb9e1d14fd8-2] Instantiated a
transactional producer.
2020-11-20 15:55:56,672 INFO 
org.apache.kafka.clients.producer.KafkaProducer  [] - [Producer
clientId=producer-CepOperator -> Sink:
Unnamed-1848139bf30d999062379bb9e1d14fd8-2, transactionalId=CepOperator ->
Sink: Unnamed-1848139bf30d999062379bb9e1d14fd8-2] Overriding the default
retries config to the recommended value of 2147483647 since the idempotent
producer is enabled.
2020-11-20 15:55:56,676 INFO  org.apache.kafka.common.utils.AppInfoParser   
  
[] - Kafka version: 2.4.1
2020-11-20 15:55:56,676 INFO  org.apache.kafka.common.utils.AppInfoParser   
  
[] - Kafka commitId: c57222ae8cd7866b
2020-11-20 15:55:56,676 INFO  org.apache.kafka.common.utils.AppInfoParser   
  
[] - Kafka startTimeMs: 1605858956676
2020-11-20 15:55:56,676 INFO 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer [] - Starting
FlinkKafkaInternalProducer (1/1) to produce into default topic
spc_testResult
2020-11-20 15:55:56,676 INFO 
org.apache.kafka.clients.producer.internals.TransactionManager [] -
[Producer clientId=producer-CepOperator -> Sink:
Unnamed-1848139bf30d999062379bb9e1d14fd8-2, transactionalId=CepOperator ->
Sink: Unnamed-1848139bf30d999062379bb9e1d14fd8-2] ProducerId set to -1 with
epoch -1
2020-11-20 15:55:56,678 INFO  org.apache.kafka.clients.Metadata 
  
[] - [Producer clientId=producer-CepOperator -> Sink:
Unnamed-1848139bf30d999062379bb9e1d14fd8-2, transactionalId=CepOperator ->
Sink: Unnamed-1848139bf30d999062379bb9e1d14fd8-2] Cluster ID:
8IUUMEvGQLKWsQRfKWc9Hw
2020-11-20 15:55:56,779 INFO 
org.apache.kafka.clients.producer.internals.TransactionManager [] -
[Producer clientId=producer-CepOperator -> Sink:
Unnamed-1848139bf30d999062379bb9e1d14fd8-2, transactionalId=CepOperator ->
Sink: Unnamed-1848139bf30d999062379bb9e1d14fd8-2] ProducerId set to 50001
with epoch 1753
2020-11-20 15:55:56,793 INFO 
org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction []
- FlinkKafkaProducer 1/1 - checkpoint 5383 complete, committing transaction
TransactionHolder{handle=KafkaTransactionState [transactionalId=CepOperator
-> Sink: Unnamed-1848139bf30d999062379bb9e1d14fd8-0, producerId=50002,
epoch=1752], transactionStartTime=1605858953780} from checkpoint 5383
2020-11-20 15:55:56,793 INFO 
org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer
[] - Flushing new partitions
2020-11-20 15:55:56,793 INFO 
org.apache.kafka.clients.producer.KafkaProducer  [] - [Producer
clientId=producer-CepOperator -> Sink:
Unnamed-1848139bf30d999062379bb9e1d14fd8-0, transactionalId=CepOperator ->
Sink: Unnamed-1848139bf30d999062379bb9e1d14fd8-0] Closing the Kafka producer
with timeoutMillis = 0 ms.
2020-11-20 15:55:56,793 INFO 
org.apache.kafka.clients.producer.KafkaProducer  [] - [Producer
clientId=producer-CepOperator -> Sink:
Unnamed-1848139bf30d999062379bb9e1d14fd8-0, transactionalId=CepOperator ->
Sink: Unnamed-1848139bf30d999062379bb9e1d14fd8-0] Proceeding to force close
the producer since pending requests could not be completed within timeout 0
ms.
2020-11-20 15:55:59,670 INFO 
org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer
[] - Flushing new partitions
2020-11-20 15:55:59,671 INFO 
org.apache.kafka.clients.producer.ProducerConfig [] -
ProducerConfig values: 
acks = all
batch.size = 16384
bootstrap.servers = [192.168.81.128:9092]
buffer.memory = 33554432
client.dns.lookup = default
client.id = 
compression.type = none
connections.max.idle.ms = 54
delivery.timeout.ms = 12
enable.idempotence = false
interceptor.classes = []
key.serializer = class
org.apache.kafka.common.serialization.ByteArraySerializer
linger.ms = 0
max.block.ms = 6
max.in.flight.requests.per.connection = 5
max.request.size = 1048576
metadata.max.age.ms = 30
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 3

cep调用pattern前使用keyby报错:getBufferBuilder越界

2020-10-20 Thread Wz
在使用CEP.pattern 、
patternStream.process前。对input输入流不使用keyby,就不会报错,检查了key应该不为null。而且如果不使用下游的cep,keyby后直接print也不会报错。
查了一些资料好像和内存buffer有关系,请教具体可能是由什么原因会造成的这个数组越界错误。谢谢~







Caused by: java.lang.RuntimeException: 3
at
org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:110)
at
org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:89)
at
org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:45)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$BroadcastingOutputCollector.collect(OperatorChain.java:787)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$BroadcastingOutputCollector.collect(OperatorChain.java:740)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30)
at
org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41)
at
org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:161)
at
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:178)
at
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:153)
at
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:67)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:345)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxStep(MailboxProcessor.java:191)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:181)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:558)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:530)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
at java.lang.Thread.run(Thread.java:748)



Caused by: java.lang.ArrayIndexOutOfBoundsException: 3
at
org.apache.flink.runtime.io.network.api.writer.ChannelSelectorRecordWriter.getBufferBuilder(ChannelSelectorRecordWriter.java:92)
at
org.apache.flink.runtime.io.network.api.writer.RecordWriter.copyFromSerializerToTargetChannel(RecordWriter.java:135)
at
org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:120)
at
org.apache.flink.runtime.io.network.api.writer.ChannelSelectorRecordWriter.emit(ChannelSelectorRecordWriter.java:60)
at
org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:107)
... 19 more

Process finished with exit code 1




--
Sent from: http://apache-flink.147419.n8.nabble.com/