[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user hmcl commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-186931043 @connieyang @jianbzhou @tgravescs patch is here: https://github.com/apache/storm/pull/1131 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user jianbzhou commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-186820101 Hi **@hmcl **, is there any update on the patch please? thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user tgravescs commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-186254846 @hmcl Sorry to keep hounding on this but how close are you? If you are just doing some final cleanup or have a few minor issues could you post what you have so we can start testing and taking a look. We have customers waiting on this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user jianbzhou commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-185525958 **@hmcl ** thanks a lot i understand the complexity. Appreciate your help and it would be great to see your patch by EOD today your time. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user hmcl commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-185525450 @jianbzhou sorry it took me a bit longer. Trying my best to put it up within the next few hours. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user jianbzhou commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-185258481 Hi **@hmcl **, could you please share the patch please? thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user jianbzhou commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-185257444 Hi @**hmcl **, could you please share the patch please? thanks a lot! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user hmcl commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-184566399 @jianbzhou I am just finishing the final touches. I will uploaded it today. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user jianbzhou commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-184563880 Hi **@hmcl** ,may I have the patch please? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user jianbzhou commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-182423997 Hi @hmcl, that's good news, thanks for that! I will add your patch this weekend or early next Monday to our QA environment- next week we will run majority of the QA test cases to ensure we can release our change by end of Feb. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user connieyang commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-181711832 @hmcl, any update to your patch? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user hmcl commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-181721528 Hi @connieyang @jianbzhou apologies for not replying to you earlier. I was sidetracked with a last minute release requirement that wasn't planned and that got things a bit off schedule. I am available to resume this task again this week. I will upload a patch by the end of the week. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user jianbzhou commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-177696306 Hi @hmcl, this is Wayne from connieyang's team, not sure could you please provide a patch for us to kick off the testing? We are doing the QA testing this week, that would be great if I can add your patch in asap...many thanks!!! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user hmcl commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-174599255 @connieyang I will try go have a patch for review by the end of this week. I will also try to keep you posted on the progress. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user hmcl commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-173498063 @connieyang the goal of my patch is to entirely replace the existing kafka spout; that includes the Trident API. In order to allow for a smooth migration, as well as avoid any backward compatibility issues, I started writing the code in a new package, completely isolated from the existing one. The goal is to do a complete rewrite. It is my opinion that this task should be properly designed to address future requirements, including proper testing and backwards compatibility. However, some patches have recently come in addressing the JIRAs I had assigned (despite my comments), which will inevitably lead to wasted effort by either one of the parts. Nevertheless, I will do my best to make the most of this situation for the benefit of the community. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-173585848 @hsun-cnnxty I agree that STORM-1015 looks independent of the other changes happening to support the 0.9 API. It adds in a new feature for backwards compatibility though. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user connieyang commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-173744934 I too agree that STORM-1015 is beyond the scope of Kafka 0.9 uprev related changes. So, I think it makes sense to keep them separate. @hmcl, any ETA on the rewrite or uprev effort? Thanks much! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-173227353 @hsun-cnnxty I am not sure that we have had much in the way of collaboration in the past, so if others have suggestions on this that would be great. I personally think that first we need to define the goals of each person involved here so we can be sure to know what features are needed to declare this done. We can divide different features up into smaller pieces/JIRA that can be done independently if that is simpler. STORM-822 talks about using the kafka 0.8.3 API, not the 0.9.0 API, and that is it (I assume that they are the same and a version number was changed before release). Hence the need to better define what STORM-822 is intended to cover. So from my perspective the minimum goals that I have for a 0.9.0 API Spout is * It needs to support at least once processing correctly * It needs to support at most once processing correctly (or at least as well as the previous spout did) * it needs to support the security APIs Before we can remove the old spout though it also needs to support the same functionality. Not necessarily a drop in replacement (Although that would be nice too). @hmcl @tgravescs @erikdw @Deepnekroz @harshach what are the requirements that you have? Once we know the requirements we can add them to STORM-822, setup a plan a file subtasks for the different parts of the plan, ideally with dependencies. I personally don't want to run the show in coordinating this. I am up to my eyeballs in 1.x/JStorm, so if someone else here wants to volunteer to help coordinate things that would be great. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user hsun-cnnxty commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-173285410 @revans2 and all, I feel my PR ([STORM-1015]: Allow Kafka offsets to be saved using Kafka's consumer offset management api) is kind of independent of this PR as it is not related to the Kafka consumer API changes. I don't see major changes in consumer offset API in 0.9.0 so my feeling is that once you guys have the 0.9.0 branch ready, I can just merge in my change. I am curious to know what is the plan to support both 0.8.x and 0.9.x in coming releases, or what is the general strategy to support incompatible Kafka versions in the future. Another question that is not related to this topic, but maybe I can get help from you. How does the cross compiling of Scala version 2.10, 2.11 works in the build process? I cannot figure out how to do it locally on my laptop. -thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user connieyang commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-173354953 @hmcl Will your PR include the rev for Trident API to work with Kafka 0.9? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user hmcl commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-172962722 @tgravescs I am actively working on this and I am going to conclude my implementation regardless of the status of the proposed patches. This task was and still is my priority, however, due to an emergency of a team member, I had to interrupt working on this to cover for him. I will resume as soon as possible. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user tgravescs commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-172958019 @hmcl just curious your status on this, are you actively working on it and do you have eta? thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user erikdw commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-173041086 @tgravescs, @hmcl, and @Deepnekroz : @hsun-cnnxty is also working on [a refactor of the kafka-spout, to support the new ZooKeeper-free kafka API] (https://github.com/apache/storm/pull/705). Can y'all please sync up to ensure we have a consistent story after these efforts? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user hsun-cnnxty commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-173093649 I don't have much experience on how the collaborations be done in storm project. Happy to take any advices on what I can help. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user darionyaphet commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-170849861 hi @Deepnekroz using difference version kafka maybe use difference spout ? kafka 0.8.X is `KafkaSpout` and kafka 0.9 is `KafkaJavaApiSpout ` ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user Deepnekroz commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-170855250 Hello @darionyaphet ! Yes, `KafkaJavaApiSpout` is using Kafka 0.9 Consumer API, and `KafkaSpout` is retained for backward compatibility. It also will work with Kafka 0.9, because they left old API, but it is deprecated for now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user darionyaphet commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-170928155 I think the two kind of spout could be merge into one ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user Deepnekroz commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-170499566 Hello again! Here's non-autocommit implementation of Kafka 0.9 spout. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user Deepnekroz commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-169908193 Thanks for comments! @harshach could you please give me ETA of @hmcl PR with this feature? I want to be sure, that new storm-kafka will work nicely with my existing code base. Also, I can write non-autocommit approach by next two days, if its needed. Have a nice day! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user hmcl commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-169910619 @Deepnekroz I am working on this. We wil make sure that all works well with what you have. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user connieyang commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-169437497 That's great! When will this pull request be merged to master? We need it as well. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-169452092 CI seems to be failing with a trident kafka test hanging. The code appears to provide a best effort replay, but that is neither at least once nor at most once, so it is likely to cause issues for some topologies that don't expect this a-typical behavior. I am not an expert on the kafka API, but it looks like auto.commit.enable is inherently incompatible with what storm wants from a well behaved spout. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user harshach commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-169456370 @revans2 @connieyang although this is a good start I don't see it handles lot of cases that current storm-kafka handles. @hmcl is working on a full-fledged integration of new kafka consumer api. You should see his PR soon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822 Implement Kafka 0.9 consumer API
Github user connieyang commented on the pull request: https://github.com/apache/storm/pull/986#issuecomment-169458814 Excellent! I will follow @hmcl for his PR. Thanks for the update! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
