Hi Ted, Sorry for my big font earlier…was not intended ☺
I am on flink 1.2.0
I built flink-connector-kinesis_2.10-1.2.0.jar from source and included in the
fatjar I am running.
Followed this http://www.kidder.io/2017/02/15/flink-kinesis-streaming-connector/
From code I read a kinesis stream using
consumerConfig.setProperty(ConsumerConfigConstants.AWS_REGION, region);
consumerConfig.setProperty(ConsumerConfigConstants.DEFAULT_STREAM_INITIAL_POSITION,
"LATEST");
consumerConfig.setProperty(ConsumerConfigConstants.AWS_CREDENTIALS_PROVIDER,
"AUTO");
consumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_MAX, "10");
consumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_RETRIES,
"200");
consumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS,
"2000");
DataStream<Map<String, Object>> stream = env.addSource(new
FlinkKinesisConsumer<>(inputStream, new MyJsonDeserializationSchema(),
consumerConfig));
While I push the json event to the Kinesis stream intermittently I see this NPE
and flink job fails
2017-04-14 00:31:54,672 WARN
org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer - Error
while closing Kinesis data fetcher
java.lang.NullPointerException
at
org.apache.flink.streaming.connectors.kinesis.internals.KinesisDataFetcher.shutdownFetcher(KinesisDataFetcher.java:472)
at
org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.cancel(FlinkKinesisConsumer.java:246)
at
org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.close(FlinkKinesisConsumer.java:256)
at
org.apache.flink.api.common.functions.util.FunctionUtils.closeFunction(FunctionUtils.java:43)
at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.dispose(AbstractUdfStreamOperator.java:127)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.disposeAllOperators(StreamTask.java:442)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:343)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
at java.lang.Thread.run(Thread.java:745)
Thanks
Sathi
From: Ted Yu <[email protected]>
Date: Thursday, April 13, 2017 at 5:02 PM
To: Sathi Chowdhury <[email protected]>
Cc: "[email protected]" <[email protected]>
Subject: Re: Flink errors out and job fails--IOException from CollectSink.open()
Can you give us a bit more information ?
release of flink
snippet of your code
Thanks
=============Notice to Recipient: This e-mail transmission, and any documents,
files or previous e-mail messages attached to it may contain information that
is confidential or legally privileged, and intended for the use of the
individual or entity named above. If you are not the intended recipient, or a
person responsible for delivering it to the intended recipient, you are hereby
notified that you must not read this transmission and that any disclosure,
copying, printing, distribution or use of any of the information contained in
or attached to this transmission is STRICTLY PROHIBITED. If you have received
this transmission in error, please immediately notify the sender by telephone
or return e-mail and delete the original transmission and its attachments
without reading or saving in any manner. Thank you. =============
