[spark-kinesis] [SPARK-20168] Requesting some attention for a review

2017-11-14 Thread Yash Sharma
Hi Team, Could I please pull some attention towards the pull request on Spark-Kinesis operability. We have iterated over the patch for past few months, and it would be great to have some final review of the patch. I think its very close now. I would love to work on improvements if any. This patch

[Streaming] Requesting more Committers for Spark-Kinesis integration

2017-09-28 Thread Yash Sharma
Hi Fellow Spark developers/ PMC Members, I am a new member of the community and have started my tiny contributions to Spark-Kinesis Integration. I am trying to fill in the gaps in making spark operate with Kinesis as nicely as Kafka. I am writing this mail to highlight an issue with the kinesis

[Spark][Kinesis] Could I get some committer review on the pull request

2017-09-05 Thread Yash Sharma
Hi All, I've been working on a pull request [1] to allow Spark read from a specific timestamp from Kinesis. I have iterated the patch with the help of other contributors and we think that its in a good state now. This patch would save hours of crash recovery time for Spark while reading off

[DStream][Kinesis] Requesting review for spark-kinesis retries

2017-04-18 Thread Yash Sharma
Hi Fellow Devs, Please share your thoughts on the pull request that allows spark to have more graceful retries with kinesis streaming. The patch removes simple hard codings in the code and allows user to pass the values in config. This will help users to cope up with kinesis throttling errors and

Spark - Kinesis integration needs improvements

2017-03-30 Thread Yash Sharma
Hello fellow spark devs, hope you are doing fabulous, Dropping a brain dump here about the Spark kinesis integration. I am able to get spark kinesis to work perfectly under ideal conditions, but see a lot of open ends when things are not so ideal. I feel there are lot of open ends

Re: Spark + Kinesis

2015-05-09 Thread Chris Fregly
! -chris On Tue, Apr 7, 2015 at 6:17 PM, Vadim Bichutskiy vadim.bichuts...@gmail.com wrote: Hey y'all, While I haven't been able to get Spark + Kinesis integration working, I pivoted to plan B: I now push data to S3 where I set up a DStream to monitor an S3 bucket with textFileStream

Re: Spark + Kinesis

2015-04-07 Thread Vadim Bichutskiy
Hey y'all, While I haven't been able to get Spark + Kinesis integration working, I pivoted to plan B: I now push data to S3 where I set up a DStream to monitor an S3 bucket with textFileStream, and that works great. I 3 Spark! Best, Vadim ᐧ On Mon, Apr 6, 2015 at 12:23 PM, Vadim Bichutskiy

Re: Spark + Kinesis

2015-04-06 Thread Tathagata Das
be). ~ Jonathan Kelly From: Vadim Bichutskiy vadim.bichuts...@gmail.com Date: Friday, April 3, 2015 at 12:26 PM To: Jonathan Kelly jonat...@amazon.com Cc: u...@spark.apache.org u...@spark.apache.org Subject: Re: Spark + Kinesis Hi all, Good news! I was able to create a Kinesis