[jira] [Commented] (BEAM-5060) Issues with aws KPL while writing to kinesis using beam
[ https://issues.apache.org/jira/browse/BEAM-5060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589924#comment-16589924 ] Alexey Romanenko commented on BEAM-5060: I changed priority to "Critical" since it's a release blocker > Issues with aws KPL while writing to kinesis using beam > --- > > Key: BEAM-5060 > URL: https://issues.apache.org/jira/browse/BEAM-5060 > Project: Beam > Issue Type: Bug > Components: io-java-aws >Affects Versions: 2.5.0 >Reporter: Varsha Thanooj >Assignee: Alexey Romanenko >Priority: Critical > Attachments: pom.xml > > Time Spent: 20m > Remaining Estimate: 0h > > I am trying to write data to kinesis using apache beam kinesis IO. But I am > having some issues. > PS: I am using aws sts. > > The console output shows > > {code:java} > Exception in thread "main" > org.apache.beam.sdk.Pipeline$PipelineExecutionException: > java.lang.NoSuchMethodError: > com.amazonaws.services.kinesis.producer.IKinesisProducer.addUserRecord(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/nio/ByteBuffer;)Lorg/apache/beam/repackaged/beam_sdks_java_io_kinesis/com/google/common/util/concurrent/ListenableFuture; > at > org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:349) > at > org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:319) > at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:210) > at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:66) > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311) > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297) > at com.nestaway.beam_demo.KinesisSql.main(KinesisSql.java:153) > Caused by: java.lang.NoSuchMethodError: > com.amazonaws.services.kinesis.producer.IKinesisProducer.addUserRecord(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/nio/ByteBuffer;)Lorg/apache/beam/repackaged/beam_sdks_java_io_kinesis/com/google/common/util/concurrent/ListenableFuture; > at > org.apache.beam.sdk.io.kinesis.KinesisIO$Write$KinesisWriterFn.processElement(KinesisIO.java:568) > {code} > > > Code... > data is a Pcollection in byte[ ] format. > > {code:java} > data.apply(KinesisIO.write() > .withStreamName("stagWatchBallEventStream") > .withPartitionKey("a") > .withAWSClientsProvider(new CustomKinesisClientProvider())); > {code} > > Custom Kinesis Client : > {code:java} > public class CustomKinesisClientProvider implements AWSClientsProvider { > private static final long serialVersionUID = 1L; > private static String ID = "X"; > private static String SECRET = "X"; > private static String TOKEN = "X"; > private static BasicSessionCredentials sessionCredentials = new > BasicSessionCredentials( > ID, > SECRET, > TOKEN); > private static KinesisProducerConfiguration config = new > KinesisProducerConfiguration() > .setRecordMaxBufferedTime(3000) > .setMaxConnections(1) > .setRequestTimeout(6) > .setRegion("us-west-2") > .setCredentialsProvider(new > AWSStaticCredentialsProvider(sessionCredentials)); > @Override > public IKinesisProducer createKinesisProducer(KinesisProducerConfiguration > config) { > return new KinesisProducer(config); > } > }{code} > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5060) Issues with aws KPL while writing to kinesis using beam
[ https://issues.apache.org/jira/browse/BEAM-5060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16587170#comment-16587170 ] Varsha Thanooj commented on BEAM-5060: -- [^pom.xml] > Issues with aws KPL while writing to kinesis using beam > --- > > Key: BEAM-5060 > URL: https://issues.apache.org/jira/browse/BEAM-5060 > Project: Beam > Issue Type: Bug > Components: io-java-aws >Affects Versions: 2.5.0 >Reporter: Varsha Thanooj >Assignee: Alexey Romanenko >Priority: Major > Attachments: pom.xml > > > I am trying to write data to kinesis using apache beam kinesis IO. But I am > having some issues. > PS: I am using aws sts. > > The console output shows > > {code:java} > Exception in thread "main" > org.apache.beam.sdk.Pipeline$PipelineExecutionException: > java.lang.NoSuchMethodError: > com.amazonaws.services.kinesis.producer.IKinesisProducer.addUserRecord(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/nio/ByteBuffer;)Lorg/apache/beam/repackaged/beam_sdks_java_io_kinesis/com/google/common/util/concurrent/ListenableFuture; > at > org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:349) > at > org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:319) > at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:210) > at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:66) > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311) > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297) > at com.nestaway.beam_demo.KinesisSql.main(KinesisSql.java:153) > Caused by: java.lang.NoSuchMethodError: > com.amazonaws.services.kinesis.producer.IKinesisProducer.addUserRecord(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/nio/ByteBuffer;)Lorg/apache/beam/repackaged/beam_sdks_java_io_kinesis/com/google/common/util/concurrent/ListenableFuture; > at > org.apache.beam.sdk.io.kinesis.KinesisIO$Write$KinesisWriterFn.processElement(KinesisIO.java:568) > {code} > > > Code... > data is a Pcollection in byte[ ] format. > > {code:java} > data.apply(KinesisIO.write() > .withStreamName("stagWatchBallEventStream") > .withPartitionKey("a") > .withAWSClientsProvider(new CustomKinesisClientProvider())); > {code} > > Custom Kinesis Client : > {code:java} > public class CustomKinesisClientProvider implements AWSClientsProvider { > private static final long serialVersionUID = 1L; > private static String ID = "X"; > private static String SECRET = "X"; > private static String TOKEN = "X"; > private static BasicSessionCredentials sessionCredentials = new > BasicSessionCredentials( > ID, > SECRET, > TOKEN); > private static KinesisProducerConfiguration config = new > KinesisProducerConfiguration() > .setRecordMaxBufferedTime(3000) > .setMaxConnections(1) > .setRequestTimeout(6) > .setRegion("us-west-2") > .setCredentialsProvider(new > AWSStaticCredentialsProvider(sessionCredentials)); > @Override > public IKinesisProducer createKinesisProducer(KinesisProducerConfiguration > config) { > return new KinesisProducer(config); > } > }{code} > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5060) Issues with aws KPL while writing to kinesis using beam
[ https://issues.apache.org/jira/browse/BEAM-5060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16586158#comment-16586158 ] Alexey Romanenko commented on BEAM-5060: [~bvt279] Thank you for reporting this. Could you attach pom file which you use to build your Beam application? > Issues with aws KPL while writing to kinesis using beam > --- > > Key: BEAM-5060 > URL: https://issues.apache.org/jira/browse/BEAM-5060 > Project: Beam > Issue Type: Bug > Components: io-java-aws >Affects Versions: 2.5.0 >Reporter: Varsha Thanooj >Assignee: Alexey Romanenko >Priority: Major > > I am trying to write data to kinesis using apache beam kinesis IO. But I am > having some issues. > PS: I am using aws sts. > > The console output shows > > {code:java} > Exception in thread "main" > org.apache.beam.sdk.Pipeline$PipelineExecutionException: > java.lang.NoSuchMethodError: > com.amazonaws.services.kinesis.producer.IKinesisProducer.addUserRecord(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/nio/ByteBuffer;)Lorg/apache/beam/repackaged/beam_sdks_java_io_kinesis/com/google/common/util/concurrent/ListenableFuture; > at > org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:349) > at > org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:319) > at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:210) > at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:66) > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311) > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297) > at com.nestaway.beam_demo.KinesisSql.main(KinesisSql.java:153) > Caused by: java.lang.NoSuchMethodError: > com.amazonaws.services.kinesis.producer.IKinesisProducer.addUserRecord(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/nio/ByteBuffer;)Lorg/apache/beam/repackaged/beam_sdks_java_io_kinesis/com/google/common/util/concurrent/ListenableFuture; > at > org.apache.beam.sdk.io.kinesis.KinesisIO$Write$KinesisWriterFn.processElement(KinesisIO.java:568) > {code} > > > Code... > data is a Pcollection in byte[ ] format. > > {code:java} > data.apply(KinesisIO.write() > .withStreamName("stagWatchBallEventStream") > .withPartitionKey("a") > .withAWSClientsProvider(new CustomKinesisClientProvider())); > {code} > > Custom Kinesis Client : > {code:java} > public class CustomKinesisClientProvider implements AWSClientsProvider { > private static final long serialVersionUID = 1L; > private static String ID = "X"; > private static String SECRET = "X"; > private static String TOKEN = "X"; > private static BasicSessionCredentials sessionCredentials = new > BasicSessionCredentials( > ID, > SECRET, > TOKEN); > private static KinesisProducerConfiguration config = new > KinesisProducerConfiguration() > .setRecordMaxBufferedTime(3000) > .setMaxConnections(1) > .setRequestTimeout(6) > .setRegion("us-west-2") > .setCredentialsProvider(new > AWSStaticCredentialsProvider(sessionCredentials)); > @Override > public IKinesisProducer createKinesisProducer(KinesisProducerConfiguration > config) { > return new KinesisProducer(config); > } > }{code} > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)