[
https://issues.apache.org/jira/browse/SPARK-31413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082040#comment-17082040
]
Hyukjin Kwon commented on SPARK-31413:
--------------------------------------
Questions should go to the mailing list. You would have a better answer there.
> Accessing the sequence number and partition id for records in Kinesis adapter
> -----------------------------------------------------------------------------
>
> Key: SPARK-31413
> URL: https://issues.apache.org/jira/browse/SPARK-31413
> Project: Spark
> Issue Type: Question
> Components: DStreams
> Affects Versions: 2.4.3
> Reporter: varun senthilnathan
> Priority: Critical
>
> We are using spark 2.4.3 in java. We would like to log the partition key and
> the sequence number of every event. The overloaded create stream function of
> the kinesis utils always throws a compilation error.
>
> Function<Record,Record> printSeq = s -> s;
> KinesisUtils.createStream(
> jssc,
> appName,
> streamName,
> endPointUrl,
> regionName,
> InitialPositionInStream.TRIM_HORIZON,
> kinesisCheckpointInterval,
> StorageLevel.MEMORY_AND_DISK_SER(),
> printSeq,
> Record.class);
> *{color:#172b4d}Record is of type
> com.amazonaws.services.kinesis.model.Record{color}*
>
> The exception is as follows:
> {quote}no suitable method found for
> createStream(org.apache.spark.streaming.api.java.JavaStreamingContext,java.lang.String,java.lang.String,java.lang.String,java.lang.String,com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream,org.apache.spark.streaming.Duration,org.apache.spark.storage.StorageLevel,java.util.function.Function,java.lang.Class)
> {quote}
> JAVA DOCS :
> [https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/kinesis/KinesisUtils.html#createStream-org.apache.spark.streaming.api.java.JavaStreamingContext-java.lang.String-java.lang.String-java.lang.String-java.lang.String-com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream-org.apache.spark.streaming.Duration-org.apache.spark.storage.StorageLevel-org.apache.spark.api.java.function.Function-java.lang.Class-]
> Is there a way out?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
