[jira] [Updated] (BEAM-5060) Issues with aws KPL while writing to kinesis using beam
[ https://issues.apache.org/jira/browse/BEAM-5060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko updated BEAM-5060: --- Priority: Critical (was: Major) > Issues with aws KPL while writing to kinesis using beam > --- > > Key: BEAM-5060 > URL: https://issues.apache.org/jira/browse/BEAM-5060 > Project: Beam > Issue Type: Bug > Components: io-java-aws >Affects Versions: 2.5.0 >Reporter: Varsha Thanooj >Assignee: Alexey Romanenko >Priority: Critical > Attachments: pom.xml > > Time Spent: 20m > Remaining Estimate: 0h > > I am trying to write data to kinesis using apache beam kinesis IO. But I am > having some issues. > PS: I am using aws sts. > > The console output shows > > {code:java} > Exception in thread "main" > org.apache.beam.sdk.Pipeline$PipelineExecutionException: > java.lang.NoSuchMethodError: > com.amazonaws.services.kinesis.producer.IKinesisProducer.addUserRecord(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/nio/ByteBuffer;)Lorg/apache/beam/repackaged/beam_sdks_java_io_kinesis/com/google/common/util/concurrent/ListenableFuture; > at > org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:349) > at > org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:319) > at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:210) > at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:66) > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311) > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297) > at com.nestaway.beam_demo.KinesisSql.main(KinesisSql.java:153) > Caused by: java.lang.NoSuchMethodError: > com.amazonaws.services.kinesis.producer.IKinesisProducer.addUserRecord(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/nio/ByteBuffer;)Lorg/apache/beam/repackaged/beam_sdks_java_io_kinesis/com/google/common/util/concurrent/ListenableFuture; > at > org.apache.beam.sdk.io.kinesis.KinesisIO$Write$KinesisWriterFn.processElement(KinesisIO.java:568) > {code} > > > Code... > data is a Pcollection in byte[ ] format. > > {code:java} > data.apply(KinesisIO.write() > .withStreamName("stagWatchBallEventStream") > .withPartitionKey("a") > .withAWSClientsProvider(new CustomKinesisClientProvider())); > {code} > > Custom Kinesis Client : > {code:java} > public class CustomKinesisClientProvider implements AWSClientsProvider { > private static final long serialVersionUID = 1L; > private static String ID = "X"; > private static String SECRET = "X"; > private static String TOKEN = "X"; > private static BasicSessionCredentials sessionCredentials = new > BasicSessionCredentials( > ID, > SECRET, > TOKEN); > private static KinesisProducerConfiguration config = new > KinesisProducerConfiguration() > .setRecordMaxBufferedTime(3000) > .setMaxConnections(1) > .setRequestTimeout(6) > .setRegion("us-west-2") > .setCredentialsProvider(new > AWSStaticCredentialsProvider(sessionCredentials)); > @Override > public IKinesisProducer createKinesisProducer(KinesisProducerConfiguration > config) { > return new KinesisProducer(config); > } > }{code} > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-5060) Issues with aws KPL while writing to kinesis using beam
[ https://issues.apache.org/jira/browse/BEAM-5060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varsha Thanooj updated BEAM-5060: - Attachment: pom.xml > Issues with aws KPL while writing to kinesis using beam > --- > > Key: BEAM-5060 > URL: https://issues.apache.org/jira/browse/BEAM-5060 > Project: Beam > Issue Type: Bug > Components: io-java-aws >Affects Versions: 2.5.0 >Reporter: Varsha Thanooj >Assignee: Alexey Romanenko >Priority: Major > Attachments: pom.xml > > > I am trying to write data to kinesis using apache beam kinesis IO. But I am > having some issues. > PS: I am using aws sts. > > The console output shows > > {code:java} > Exception in thread "main" > org.apache.beam.sdk.Pipeline$PipelineExecutionException: > java.lang.NoSuchMethodError: > com.amazonaws.services.kinesis.producer.IKinesisProducer.addUserRecord(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/nio/ByteBuffer;)Lorg/apache/beam/repackaged/beam_sdks_java_io_kinesis/com/google/common/util/concurrent/ListenableFuture; > at > org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:349) > at > org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:319) > at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:210) > at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:66) > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311) > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297) > at com.nestaway.beam_demo.KinesisSql.main(KinesisSql.java:153) > Caused by: java.lang.NoSuchMethodError: > com.amazonaws.services.kinesis.producer.IKinesisProducer.addUserRecord(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/nio/ByteBuffer;)Lorg/apache/beam/repackaged/beam_sdks_java_io_kinesis/com/google/common/util/concurrent/ListenableFuture; > at > org.apache.beam.sdk.io.kinesis.KinesisIO$Write$KinesisWriterFn.processElement(KinesisIO.java:568) > {code} > > > Code... > data is a Pcollection in byte[ ] format. > > {code:java} > data.apply(KinesisIO.write() > .withStreamName("stagWatchBallEventStream") > .withPartitionKey("a") > .withAWSClientsProvider(new CustomKinesisClientProvider())); > {code} > > Custom Kinesis Client : > {code:java} > public class CustomKinesisClientProvider implements AWSClientsProvider { > private static final long serialVersionUID = 1L; > private static String ID = "X"; > private static String SECRET = "X"; > private static String TOKEN = "X"; > private static BasicSessionCredentials sessionCredentials = new > BasicSessionCredentials( > ID, > SECRET, > TOKEN); > private static KinesisProducerConfiguration config = new > KinesisProducerConfiguration() > .setRecordMaxBufferedTime(3000) > .setMaxConnections(1) > .setRequestTimeout(6) > .setRegion("us-west-2") > .setCredentialsProvider(new > AWSStaticCredentialsProvider(sessionCredentials)); > @Override > public IKinesisProducer createKinesisProducer(KinesisProducerConfiguration > config) { > return new KinesisProducer(config); > } > }{code} > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-5060) Issues with aws KPL while writing to kinesis using beam
[ https://issues.apache.org/jira/browse/BEAM-5060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-5060: --- Labels: (was: newbie) > Issues with aws KPL while writing to kinesis using beam > --- > > Key: BEAM-5060 > URL: https://issues.apache.org/jira/browse/BEAM-5060 > Project: Beam > Issue Type: Bug > Components: io-java-aws >Affects Versions: 2.5.0 >Reporter: Varsha Thanooj >Assignee: Alexey Romanenko >Priority: Major > > I am trying to write data to kinesis using apache beam kinesis IO. But I am > having some issues. > PS: I am using aws sts. > > The console output shows > > {code:java} > Exception in thread "main" > org.apache.beam.sdk.Pipeline$PipelineExecutionException: > java.lang.NoSuchMethodError: > com.amazonaws.services.kinesis.producer.IKinesisProducer.addUserRecord(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/nio/ByteBuffer;)Lorg/apache/beam/repackaged/beam_sdks_java_io_kinesis/com/google/common/util/concurrent/ListenableFuture; > at > org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:349) > at > org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:319) > at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:210) > at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:66) > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311) > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297) > at com.nestaway.beam_demo.KinesisSql.main(KinesisSql.java:153) > Caused by: java.lang.NoSuchMethodError: > com.amazonaws.services.kinesis.producer.IKinesisProducer.addUserRecord(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/nio/ByteBuffer;)Lorg/apache/beam/repackaged/beam_sdks_java_io_kinesis/com/google/common/util/concurrent/ListenableFuture; > at > org.apache.beam.sdk.io.kinesis.KinesisIO$Write$KinesisWriterFn.processElement(KinesisIO.java:568) > {code} > > > Code... > data is a Pcollection in byte[ ] format. > > {code:java} > data.apply(KinesisIO.write() > .withStreamName("stagWatchBallEventStream") > .withPartitionKey("a") > .withAWSClientsProvider(new CustomKinesisClientProvider())); > {code} > > Custom Kinesis Client : > {code:java} > public class CustomKinesisClientProvider implements AWSClientsProvider { > private static final long serialVersionUID = 1L; > private static String ID = "X"; > private static String SECRET = "X"; > private static String TOKEN = "X"; > private static BasicSessionCredentials sessionCredentials = new > BasicSessionCredentials( > ID, > SECRET, > TOKEN); > private static KinesisProducerConfiguration config = new > KinesisProducerConfiguration() > .setRecordMaxBufferedTime(3000) > .setMaxConnections(1) > .setRequestTimeout(6) > .setRegion("us-west-2") > .setCredentialsProvider(new > AWSStaticCredentialsProvider(sessionCredentials)); > @Override > public IKinesisProducer createKinesisProducer(KinesisProducerConfiguration > config) { > return new KinesisProducer(config); > } > }{code} > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)