Re: KinesisIO - aggregation

2020-09-03 Thread Jonothan Farr
till the way to go? > > Regards, > Mani > > From: Jonothan Farr > Sent: Thursday, September 3, 2020 1:02 AM > To: dev@beam.apache.org > Subject: Re: KinesisIO - aggregation > > CAUTION: This email originated from outside of D Please do not click links > or ope

Re: KinesisIO - aggregation

2020-09-02 Thread Jonothan Farr
KinesisIO works fine for me with aggegated records. Here's where deaggregate() is called in v1: https://github.com/apache/beam/blob/9f4d54a2e60ec45150437c0b050f4c73cca91f36/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/SimplifiedKinesisClient.java#L170 And v2:

Re: Using KinesisIO to put records in Firehose

2020-06-08 Thread Jonothan Farr
I believe KinesisIO only works with Kinesis Data Streams. > On Jun 8, 2020, at 9:42 AM, Brahmachaitanya Wajapey > wrote: > > Hi, > > Apache beam supports KinesisIO connector. Is this only for Kinesis streams or > it can be used to put records in Kinesis firehose as well? > > I have a

Support for AWS SDK v2 and enhanced fanout in KinesisIO

2020-04-17 Thread Jonothan Farr
Hi, I wanted to try out KinesisIO with enhanced fanout for a PoC I was working on, so I ended up submitting that as https://github.com/apache/beam/pull/9899. But since that still needs work to be fully functional (right now it does not handle resharding), I figured I could at least contribute the

Re: ** Configurable FluentBackoff for IO's **

2020-04-16 Thread Jonothan Farr
Maybe this is a separate conversation, but for AWS IOs specifically wouldn't it be better to use the AWS client's retry policy? Something similar to this: ``` @Override public AmazonS3ClientBuilder createBuilder(S3Options s3Options) { RetryPolicy retryPolicy = new RetryPolicy(

Re: contributor permission for Beam Jira tickets

2019-10-09 Thread Jonothan Farr
Done, thank you! On Wed, Oct 9, 2019 at 9:24 PM Kenneth Knowles wrote: > Done & welcome & thanks in advance for the fix! You should be able to > assign the issue to yourself now. > > Kenn > > On Wed, Oct 9, 2019 at 9:05 PM Jonothan Farr > wrote: > >> Hi,

contributor permission for Beam Jira tickets

2019-10-09 Thread Jonothan Farr
Hi, this is Jonothan from GoDaddy. I work with Ankit Jhalaria who contributed SnsIO and I would like to submit a bug fix. Can someone please add me as a contributor for Beam's Jira issue tracker? The issue is https://issues.apache.org/jira/browse/BEAM-8374. Thank you, Jonothan