[ https://issues.apache.org/jira/browse/SPARK-31413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
varun senthilnathan updated SPARK-31413: ---------------------------------------- Description: We are using spark 2.4.3 in java. We would like to log the partition key and the sequence number of every event. The overloaded create stream function of the kinesis utils always throws a compilation error. {{Function<Record,Record> printSeq = s -> s; KinesisUtils.createStream( jssc, appName, streamName, endPointUrl, regionName, InitialPositionInStream.TRIM_HORIZON, kinesisCheckpointInterval, StorageLevel.MEMORY_AND_DISK_SER(), printSeq, Record.class);}} The exception is as follows: {quote}no suitable method found for createStream(org.apache.spark.streaming.api.java.JavaStreamingContext,java.lang.String,java.lang.String,java.lang.String,java.lang.String,com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream,org.apache.spark.streaming.Duration,org.apache.spark.storage.StorageLevel,java.util.function.Function,java.lang.Class) {quote} JAVA DOCS : [https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/kinesis/KinesisUtils.html#createStream-org.apache.spark.streaming.api.java.JavaStreamingContext-java.lang.String-java.lang.String-java.lang.String-java.lang.String-com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream-org.apache.spark.streaming.Duration-org.apache.spark.storage.StorageLevel-org.apache.spark.api.java.function.Function-java.lang.Class-] Is there a way out? was:We > Accessing the sequence number and partition id for records in Kinesis adapter > ----------------------------------------------------------------------------- > > Key: SPARK-31413 > URL: https://issues.apache.org/jira/browse/SPARK-31413 > Project: Spark > Issue Type: Question > Components: DStreams > Affects Versions: 2.4.3 > Environment: We are using spark 2.4.3 in java. We would like to log > the partition key and the sequence number of every event. The overloaded > create stream function of the kinesis utils always throws a compilation error. > > {{Function<Record,Record> printSeq = s -> s;}} > {{KinesisUtils.createStream(}}jssc, appName, streamName, endPointUrl, > regionName, InitialPositionInStream.TRIM_HORIZON, kinesisCheckpointInterval, > StorageLevel.MEMORY_AND_DISK_SER(), printSeq, Record.class); > The exception is as follows: > {quote}no suitable method found for > createStream(org.apache.spark.streaming.api.java.JavaStreamingContext,java.lang.String,java.lang.String,java.lang.String,java.lang.String,com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream,org.apache.spark.streaming.Duration,org.apache.spark.storage.StorageLevel,java.util.function.Function,java.lang.Class) > {quote} > JAVA DOCS: > [https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/kinesis/KinesisUtils.html#createStream-org.apache.spark.streaming.api.java.JavaStreamingContext-java.lang.String-java.lang.String-java.lang.String-java.lang.String-com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream-org.apache.spark.streaming.Duration-org.apache.spark.storage.StorageLevel-org.apache.spark.api.java.function.Function-java.lang.Class-] > Is there a way out? > Reporter: varun senthilnathan > Priority: Critical > > We are using spark 2.4.3 in java. We would like to log the partition key and > the sequence number of every event. The overloaded create stream function of > the kinesis utils always throws a compilation error. > > {{Function<Record,Record> printSeq = s -> s; > KinesisUtils.createStream( > jssc, > appName, > streamName, > endPointUrl, > regionName, > InitialPositionInStream.TRIM_HORIZON, > kinesisCheckpointInterval, > StorageLevel.MEMORY_AND_DISK_SER(), > printSeq, > Record.class);}} > The exception is as follows: > {quote}no suitable method found for > createStream(org.apache.spark.streaming.api.java.JavaStreamingContext,java.lang.String,java.lang.String,java.lang.String,java.lang.String,com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream,org.apache.spark.streaming.Duration,org.apache.spark.storage.StorageLevel,java.util.function.Function,java.lang.Class) > {quote} > JAVA DOCS : > [https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/kinesis/KinesisUtils.html#createStream-org.apache.spark.streaming.api.java.JavaStreamingContext-java.lang.String-java.lang.String-java.lang.String-java.lang.String-com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream-org.apache.spark.streaming.Duration-org.apache.spark.storage.StorageLevel-org.apache.spark.api.java.function.Function-java.lang.Class-] > Is there a way out? -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org