[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-11-08 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Thanks for being so helpful and patient on this one @brkyvz . I will leave this with you now for your final ☑️ if you're happy

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-11-03 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 @brkyvz could you please have a look if it looks good. Would be great if you're happy with the changes and we could merge

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-10-27 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 @brkyvz Squashed multiple commits into one for better readability. Please have a look when you get time. Thanks

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-10-19 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 @brkyvz Please have a look once you have time. Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-10-12 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Thanks for the review @brkyvz . Updated the pull request with version 2.3.0. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #19427: [SparkStreaming] Reset spark.driver.bindAddress when sta...

2017-10-12 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/19427 I don't think there is an automated way. You could create a JIRA ticket and rename this title with the ticket id and component name

[GitHub] spark issue #19427: Reset spark.driver.bindAddress when starting a Checkpoin...

2017-10-11 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/19427 @ssaavedra Could you also update the Title as [SPARK-X][component] Title... please. --- - To unsubscribe, e-mail: reviews

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-10-11 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Thanks for the review @brkyvz . Please have a look at the new PR. I have implemented the review comments. For the API hygiene, I have added both the new and old API, and have marked the old

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-10-11 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Thanks @brkyvz . Appreciate the review comments. I am facing some issues with the scala packaging. I will keep working on it and update the patch soon

[GitHub] spark pull request #19416: [SPARK-22187][SS] Update unsaferow format for sav...

2017-10-03 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/19416#discussion_r142368186 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/FlatMapGroupsWithState_StateManager.scala --- @@ -0,0 +1,143

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-10-03 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Hi @brkyvz can I get some love here please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-09-24 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 @brkyvz Could you please check this for the last suggestions ? --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-09-20 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 @budde could you please do one last review of this one. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #18029: [SPARK-20168] [DStream] Add changes to use kinesi...

2017-09-09 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r137926490 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/InitialPosition.scala --- @@ -0,0 +1,104 @@ +/* + * Licensed

[GitHub] spark pull request #18029: [SPARK-20168] [DStream] Add changes to use kinesi...

2017-09-07 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r137684968 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/InitialPosition.scala --- @@ -0,0 +1,104 @@ +/* + * Licensed

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-09-06 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Thanks for the comments @brkyvz . I will be working on the changes and update the PR very soon. --- - To unsubscribe, e-mail

[GitHub] spark pull request #18029: [SPARK-20168] [DStream] Add changes to use kinesi...

2017-09-06 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r137245127 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/InitialPosition.scala --- @@ -0,0 +1,104 @@ +/* + * Licensed

[GitHub] spark pull request #18029: [SPARK-20168] [DStream] Add changes to use kinesi...

2017-09-06 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r137237166 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/InitialPosition.scala --- @@ -0,0 +1,104 @@ +/* + * Licensed

[GitHub] spark pull request #18029: [SPARK-20168] [DStream] Add changes to use kinesi...

2017-09-06 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r137235013 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/InitialPosition.scala --- @@ -0,0 +1,104 @@ +/* + * Licensed

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-09-03 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Resolved conflict introduced by other code commits. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-08-30 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Could I get some love here from the committers please @brkyvz @HyukjinKwon @srowen . Would love to work on any changes if required. --- If your project is set up for it, you can reply

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-08-23 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Hi @srowen we've iterated this patch to bring it in a good state. Need a committer ✔️ before we can go ahead merging it. --- If your project is set up for it, you can reply to this email

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-08-20 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Will wait for @brkyvz , @HyukjinKwon for final ☑️ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request #18029: [SPARK-20168] [DStream] Add changes to use kinesi...

2017-08-20 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r134138994 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/InitialPosition.scala --- @@ -0,0 +1,104 @@ +/* + * Licensed

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-08-20 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Added review suggestions @budde ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-08-20 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Will update and post another request seen. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-08-19 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 @brkyvz I have made the suggested modifications to the code. Please have a look when you get time. Thanks --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-08-19 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Thanks @budde for the review . Love the suggestions. 👍 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-08-05 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 @brkyvz , @budde, @HyukjinKwon could you please review this sometime. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #18029: [SPARK-20168][DStream] Add changes to use kinesis fetche...

2017-07-26 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Would love to work on the review feedback here @budde @brkyvz . Please have a look when you have time. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #18071: [SPARK-20855][Docs][DStream] Update the Spark kinesis do...

2017-07-24 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18071 Thanks @budde @srowen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #18071: [SPARK-20855][Docs][DStream] Update the Spark kinesis do...

2017-07-24 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18071 Updated the patch @HyukjinKwon . Thanks for the note. 👍 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #18071: [SPARK-20855][Docs][DStream] Update the Spark kinesis do...

2017-07-23 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18071 I noticed that now. Yes I will post an updated patch today. Thanks @HyukjinKwon --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark issue #18071: [SPARK-20855][WIP][DStream] Update the Spark kinesis doc...

2017-07-23 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18071 @HyukjinKwon The PR is ready. Just waiting for some 👍 from the reviewers. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #18029: [SPARK-20168][DStream] Add changes to use kinesis fetche...

2017-07-19 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 @budde @brkyvz - could I get some love here please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-07-13 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Hi, @budde , @brkyvz would love to hear your thoughts on the new patch --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-07-08 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 I really appreciate you taking time to review the changes. I know balancing work and foss contributions is difficult :) I have given a clean new stab at the issue with all the feedbacks. Let me

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-06-19 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 @budde @brkyvz could you suggest if the current patch seems ok, or I should make something similar to the case class/ trait ? --- If your project is set up for it, you can reply to this email

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-06-16 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 @budde What would that mean for existing applications that are sending an `InitialPositionInStream.SOMETHING` to the the builder. We would need to provide multiple overloaded versions

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-06-13 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Thoughts - @budde @brkyvz ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-06-09 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 For review @budde @brkyvz ^ this looks in a good state. @brkyvz - I still didn't quite get the code bit you shared, would love to make changes if you could explain that piece of your mind

[GitHub] spark pull request #18029: [SPARK-20168][WIP][DStream] Add changes to use ki...

2017-06-05 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r120236128 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala --- @@ -193,6 +197,21 @@ object

[GitHub] spark pull request #18029: [SPARK-20168][WIP][DStream] Add changes to use ki...

2017-06-05 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r120236059 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala --- @@ -100,6 +103,7 @@ object

[GitHub] spark pull request #18029: [SPARK-20168][WIP][DStream] Add changes to use ki...

2017-06-05 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r120235619 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala --- @@ -193,6 +197,21 @@ object

[GitHub] spark pull request #18029: [SPARK-20168][WIP][DStream] Add changes to use ki...

2017-06-05 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r120234200 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala --- @@ -100,6 +103,7 @@ object

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-06-05 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 ``` ./bin/spark-shell --jars /Users/ysharma/.m2/repository/org/apache/spark/spark-streaming-kinesis-asl_2.11/2.3.0-SNAPSHOT/spark-streaming-kinesis-asl_2.11-2.3.0-SNAPSHOT.jar --packages

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-06-04 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 @budde @brkyvz - Would appreciate if you could share some tips by which I could include these external kinesis modules on spark-shell to test this directly from shell (both for scala and python

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-06-04 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 @budde Added a new method to pass the timestamp more elegantly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request #18029: [SPARK-20168][WIP][DStream] Add changes to use ki...

2017-06-03 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r119986035 --- Diff: external/kinesis-asl/src/test/java/org/apache/spark/streaming/kinesis/JavaKinesisInputDStreamBuilderSuite.java --- @@ -45,7 +46,7 @@ public

[GitHub] spark pull request #18029: [SPARK-20168][WIP][DStream] Add changes to use ki...

2017-06-03 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r119986280 --- Diff: external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisInputDStreamBuilderSuite.scala --- @@ -111,5 +110,28 @@ class

[GitHub] spark pull request #18029: [SPARK-20168][WIP][DStream] Add changes to use ki...

2017-06-03 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r119984045 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala --- @@ -38,6 +40,7 @@ private[kinesis] class

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-05-31 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Could I get some love here please @budde @brkyvz This change is pretty useful for us and I would like to bake this patch well so others can take advantage of this feature. --- If your

[GitHub] spark issue #18071: [SPARK-20855][WIP][DStream] Update the Spark kinesis doc...

2017-05-24 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18071 Hey @budde , drawing your attention here please. Just wanted to update the docs with the new Builder interface by you. Few questions on top of my mind - - Is it compatible with the python

[GitHub] spark pull request #18071: [SPARK-20855][WIP][DStream] Update the Spark kine...

2017-05-23 Thread yssharma
GitHub user yssharma opened a pull request: https://github.com/apache/spark/pull/18071 [SPARK-20855][WIP][DStream] Update the Spark kinesis docs to use the KinesisInputDStream builder instead of deprecated KinesisUtils ## What changes were proposed in this pull request

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-05-22 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Simplified how timestamp is passed to Kinesis for InitialPositionInStream.AT_TIMESTAMP. cc: @budde , @brkyvz --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-05-20 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Commit https://github.com/apache/spark/commit/424550c8450937f78ce608ff7b18e46f41478a8a should fix the timeouts mentioned in the https://github.com/apache/spark/commit

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-05-18 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 @budde @brkyvz would love to hear your thoughts if this is the best way to add this functionality --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #18028: [DStream][DOC]Add documentation for kinesis retry config...

2017-05-18 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18028 Thanks @brkyvz --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #18029: [SPARK-20168][WIP][DStream] Add changes to use ki...

2017-05-18 Thread yssharma
GitHub user yssharma opened a pull request: https://github.com/apache/spark/pull/18029 [SPARK-20168][WIP][DStream] Add changes to use kinesis fetches from specified timestamp ## What changes were proposed in this pull request? Kinesis client can resume from a specified

[GitHub] spark pull request #18028: [DStream][DOC]Add documentation for kinesis retry...

2017-05-18 Thread yssharma
GitHub user yssharma opened a pull request: https://github.com/apache/spark/pull/18028 [DStream][DOC]Add documentation for kinesis retry configurations ## What changes were proposed in this pull request? The changes were merged as part of - https://github.com/apache/spark

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-05-16 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 Awesome. Thanks @budde @brkyvz for reviews and patch improvements. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-05-14 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @budde @brkyvz - Any feed back on this one please ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-05-05 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 Thanks for all the review comments @budde @brkyvz . Added new review changes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request #17467: [SPARK-20140][DStream] Remove hardcoded kinesis r...

2017-05-05 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17467#discussion_r115110773 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReadConfigurations.scala --- @@ -0,0 +1,71

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-05-04 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 Thanks for the comments @budde @brkyvz . Would be adding the changes soon. I too liked pulling the values of spark conf directly and got it working with the private val

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-05-04 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @brkyvz - Thanks for the review comments. Updated the patch, please review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-26 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @budde would you like to share your thoughts on the new changes when you have time ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-24 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @brkyvz - Added new changes that adds - - A case class `KinesisReadConfigurations` that adds all the kinesis read configs in a single place - A test class that passes the kinesis configs

[GitHub] spark pull request #17467: [SPARK-20140][DStream] Remove hardcoded kinesis r...

2017-04-23 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17467#discussion_r112848855 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala --- @@ -135,7 +139,8 @@ class

[GitHub] spark pull request #17467: [SPARK-20140][DStream] Remove hardcoded kinesis r...

2017-04-23 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17467#discussion_r112848762 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala --- @@ -135,7 +139,8 @@ class

[GitHub] spark pull request #17467: [SPARK-20140][DStream] Remove hardcoded kinesis r...

2017-04-23 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17467#discussion_r112848401 --- Diff: external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDDSuite.scala --- @@ -101,6 +103,36 @@ abstract class

[GitHub] spark pull request #17467: [SPARK-20140][DStream] Remove hardcoded kinesis r...

2017-04-23 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17467#discussion_r112848373 --- Diff: external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDDSuite.scala --- @@ -101,6 +103,36 @@ abstract class

[GitHub] spark pull request #17467: [SPARK-20140][DStream] Remove hardcoded kinesis r...

2017-04-23 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17467#discussion_r112848363 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala --- @@ -135,7 +139,8 @@ class

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-23 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @felixcheung : would love to get more review comments to improve the patch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-23 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @budde - Added changes for the minor review comments and docs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-22 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @budde - Not sure if we can exactly test the configured timeouts. I have debugged the flow in my testcase to verify that the custom values are passed down to the Kinesis backed block rdd

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-22 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @budde : Implemented review changes and checked scala code style checks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request #17467: [SPARK-20140][DStream] Remove hardcoded kinesis r...

2017-04-21 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17467#discussion_r112614624 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala --- @@ -83,7 +83,8 @@ class

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-20 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @budde @brkyvz - Implemented the review changes. Please review. - Using SparkConf for all the user parameters - removed kinesisWait to be val instead of var - fixed documentation

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-19 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @budde - thanks for taking time to review it. Appreciate it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request #17467: [SPARK-20140][DStream] Remove hardcoded kinesis r...

2017-04-19 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17467#discussion_r112347174 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala --- @@ -249,6 +252,17 @@ object

[GitHub] spark pull request #17467: [SPARK-20140][DStream] Remove hardcoded kinesis r...

2017-04-19 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17467#discussion_r112346805 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala --- @@ -147,6 +152,14 @@ class

[GitHub] spark issue #17506: [SPARK-20189][DStream] Fix spark kinesis testcases to re...

2017-04-14 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17506 Thanks @srowen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #17506: [SPARK-20189][DStream] Fix spark kinesis testcase...

2017-04-12 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17506#discussion_r111292109 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala --- @@ -267,7 +267,7 @@ object

[GitHub] spark pull request #17506: [SPARK-20189][DStream] Fix spark kinesis testcase...

2017-04-12 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17506#discussion_r111291842 --- Diff: external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisStreamSuite.scala --- @@ -233,11 +241,15 @@ abstract class

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-11 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 Waiting for review @brkyvz . Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17506: [SPARK-20189][DStream] Fix spark kinesis testcases to re...

2017-04-11 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17506 @srowen do you feel this patch could be merged now ? Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-08 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 Just for info, while trying to use the `sc` in the `KinesisBackedBlockRDD ` : `- Basic reading from Kinesis *** FAILED *** org.apache.spark.SparkException: Task not serializable

[GitHub] spark issue #17506: [SPARK-20189][DStream] Fix spark kinesis testcases to re...

2017-04-08 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17506 Is there anything else that can be done on this patch. The patch fixes all the deprecated api testcases that try to use the aws secret/id credentials instead of the builder. --- If your project

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-08 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @brkyvz - thanks for taking time to review the patch. appreciate it. Implemented all your suggestions. Now passing a new map for the kinesis configs and added mechanism to use the builder

[GitHub] spark issue #17506: [SPARK-20189][DStream] Fix spark kinesis testcases to re...

2017-04-06 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17506 Thanks @srowen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #17506: [SPARK-20189][DStream] Fix spark kinesis testcases to re...

2017-04-05 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17506 @srowen - does the Jenkins re-test trigger automatically? else, could I request a retest on this patch please ? --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #17506: [SPARK-20189][DStream] Fix spark kinesis testcases to re...

2017-04-04 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17506 @srowen : It failed on the earlier patch with KinesisTestUtils.scala changes. This version is clean. Will wait for the next automated build :) Build logs: https://amplab.cs.berkeley.edu

[GitHub] spark issue #17506: [SPARK-20189][DStream] Fix spark kinesis testcases to re...

2017-04-03 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17506 The Scala style check fail because of the double spaced lines probably. But that's how the existing code was so thought of keeping it that way. --- If your project is set up for it, you can reply

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-03 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @srowen - Could I get some love here as well. Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #17506: [SPARK-20189][DStream] Fix spark kinesis testcases to re...

2017-04-03 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17506 @srowen - Thanks for the feedback. Appreciate it. Added a minimal simplified patch which fixes the testcases that fail with the old api. --- If your project is set up for it, you can reply

[GitHub] spark issue #17506: [SPARK-20189][DStream] Fix spark kinesis testcases to re...

2017-04-03 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17506 @srowen - Yes, 2 objectives for the patch: - add new testcases for new api (that also tests aws session tokens) - ability to disable old api test cases For now I have added new test

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-03 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 Can I get some feedback here please @tdas @brkyvz Thanks :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #17506: [SPARK-20189][DStream] Fix spark kinesis testcases to re...

2017-04-03 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17506 @srowen - Have made following improvements to the patch to incorporate your points: - Did not remove old testcases for keeping the checks on old api - Added new testcases that test the new

[GitHub] spark pull request #17506: [SPARK-20189][DStream] Fix spark kinesis testcase...

2017-04-02 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17506#discussion_r109304637 --- Diff: external/kinesis-asl/src/test/java/org/apache/spark/streaming/kinesis/JavaKinesisStreamSuite.java --- @@ -50,48 +56,86 @@ public void

  1   2   >