[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user gregoks commented on the issue: https://github.com/apache/nifi/pull/2020 @cammachusa did you happen to write the GetKudu processor that you were talking about in this thread? Thx! Greg ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2020 @cammach @cammachusa please close this PR since it has been merged. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user cammachusa commented on the issue: https://github.com/apache/nifi/pull/2020 Thanks @rickysaltzer , I didn't mean to rush you :-) . It's just because I have to report to my boss, since it's the end of the sprint. It was nice working with you, and sure I will contribute more to the community. Cheer! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user cammachusa commented on the issue: https://github.com/apache/nifi/pull/2020 @rickysaltzer , I though the version should be consistence for all, so. Anyway, I revert it 1.3 for the files above, and rebase to have 1 commit. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user cammachusa commented on the issue: https://github.com/apache/nifi/pull/2020 @rickysaltzer thanks for the guide :-) Working on it now --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user rickysaltzer commented on the issue: https://github.com/apache/nifi/pull/2020 hey @cammachusa sorry for gap in time, been pretty busy. I don't know if this is the best way to do it, but here's a shot :) First reset your `HEAD` inside your branch to the last commit before you started your PR. $ git reset --hard 02c05bc2037ca2e4ee1850a8cb765d85a4f3b8a3 Add my repo as a remote $ git remote add ricky https://github.com/rickysaltzer/nifi.git Fetch my repo and branches $ git fetch ricky Merge my rebase onto your branch. $ git merge ricky/NiFi-3973 Make sure you see the following commit for your latest $ git log --oneline | head -n1 1101ffc [NiFi-3973] Add PutKudu processor to ingest data to --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user cammachusa commented on the issue: https://github.com/apache/nifi/pull/2020 @rickysaltzer , I don't know how to do that, can you give me some detail guidance? (from getting your commit to pushing to this PR) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user rickysaltzer commented on the issue: https://github.com/apache/nifi/pull/2020 You could push my force push my commit to this PR and that'll be fine, too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user cammachusa commented on the issue: https://github.com/apache/nifi/pull/2020 @rickysaltzer , I looked at your branch, it's identical with my branch. So sure, it will work. Just to clarify, what you mean by "pull it down, ... and force push it to this review"? Does that mean, pull your branch, and merge to mine, and push to Pull Request? Or pull your branch, and push to this PR, to clean up all previous commits to have just one? If the second, I haven't done cross push, can you help with what git commands I should run? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user rickysaltzer commented on the issue: https://github.com/apache/nifi/pull/2020 @cammachusa - I updated my branch with a rebase, will you pull it down, make sure things are good for you and then force push it to this review? Thanks!! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user rickysaltzer commented on the issue: https://github.com/apache/nifi/pull/2020 I am going to be going through my final rounds of testing, so by the end of this week, I promise :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user cammachusa commented on the issue: https://github.com/apache/nifi/pull/2020 @rickysaltzer , hahaha. When will you close this PR, I am waiting for your review every day :=) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user rickysaltzer commented on the issue: https://github.com/apache/nifi/pull/2020 False alarm, I got it now :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user rickysaltzer commented on the issue: https://github.com/apache/nifi/pull/2020 @cammachusa I'm having a hard time getting your new commits...as well as even understanding how I got them in the first place. @joewitt am I missing something regarding getting the new commits? I'm running a fetch on the repo located here on github. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user cammachusa commented on the issue: https://github.com/apache/nifi/pull/2020 The last commit, I switched to onScheduled already. Thanks for your review and code @rickysaltzer There are some code style check errors. I'm on it now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user rickysaltzer commented on the issue: https://github.com/apache/nifi/pull/2020 @cammachusa seems I can't update this PR with my own code. My changes are below so you can merge them in and then push them again. https://github.com/rickysaltzer/nifi/commit/c17bbaff08685f74eef037f5e34dd6339b3aac68 I was able to get a sample pipeline working locally which is great. I want to do a little bit more testing before we merge it in. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user rickysaltzer commented on the issue: https://github.com/apache/nifi/pull/2020 I've made some changes that allows the tests to run successfully. I'm going to test out this in an actual NiFi installation and then provide you my changes. Unfortunately this may not happen today due to my schedule. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user rickysaltzer commented on the issue: https://github.com/apache/nifi/pull/2020 @cammachusa thanks for the reminder :) - I've been a bit busy due to business travel. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user cammachusa commented on the issue: https://github.com/apache/nifi/pull/2020 @rickysaltzer , remind :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user rickysaltzer commented on the issue: https://github.com/apache/nifi/pull/2020 Hey @cammachusa - I will take a look today..been traveling. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user cammachusa commented on the issue: https://github.com/apache/nifi/pull/2020 @rickysaltzer any comments? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user cammachusa commented on the issue: https://github.com/apache/nifi/pull/2020 Hi @rickysaltzer , switching to ConfigurationContext still doesn't work, all still null. As I noticed that none of Put/Getxxx processors (e.g. PutHBase, PutParquet, PutCassandra) uses OnEnable, they use OnSchedule instead. And, as I understand OnScheduled only get kicked off once (can be every 5 minutes or 10, depend on user's conf), but onTrigger will be triggered several times depend on how many flowfile you have in one schedule. @joewitt may mean OnSchedule, not OnEnabled, because I never used OnEnable to my PutKudu (recall his comments a couple week ago about thread safe of Kudu client). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user rickysaltzer commented on the issue: https://github.com/apache/nifi/pull/2020 @cammachusa thanks! I think maybe you need to change the arguments given to your method once you use `OnEnabled`. I think it switches to `ConfigurationContext` instead of `ProcessContext` (or the other way) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user cammachusa commented on the issue: https://github.com/apache/nifi/pull/2020 sorry @rickysaltzer , I was rush, so didn't run the test when switch to onEnabled. Interestingly!, the OnEnabled is not being called, so objects defined in it are all null when running test cases. Maybe I should have both OnEnabled (for KuduClient and KuduTable), and OnScheduled for the rest of variables. Any advise? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user rickysaltzer commented on the issue: https://github.com/apache/nifi/pull/2020 @cammach @cammachusa Still seeing this error - what did you do to fix it? testSkipHeadLineTrue(org.apache.nifi.processors.kudu.TestPutKudu) Time elapsed: 0.001 sec <<< ERROR! java.lang.NoClassDefFoundError: Could not initialize class org.apache.nifi.processors.kudu.MockPutKudu at org.apache.nifi.processors.kudu.TestPutKudu.setUp(TestPutKudu.java:65) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user rickysaltzer commented on the issue: https://github.com/apache/nifi/pull/2020 I've got your changes locally to do some testing, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user rickysaltzer commented on the issue: https://github.com/apache/nifi/pull/2020 I will continue reviewing later today or tomorrow, thanks for your patience. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user cammach commented on the issue: https://github.com/apache/nifi/pull/2020 Hi @rickysaltzer , I have figured out the Record Reader reference issue, and fixed. Would you please continue to review the code? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user cammachusa commented on the issue: https://github.com/apache/nifi/pull/2020 @joewitt , would you please help? I couldn't start nifi, it keep saying "ClassNotFoundException: org.apache.nifi.serialization.RecordReaderFactory" . I can build it without any error, but when starting nifi, ... Some of my dependency on RecordReader may not be correct, or missing something? Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user rickysaltzer commented on the issue: https://github.com/apache/nifi/pull/2020 @joewitt might be the best person to answer @cammachusa's question regarding Travis. I seem to recall that it can be finicky, but not 100% the current state of CI stability. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user cammachusa commented on the issue: https://github.com/apache/nifi/pull/2020 Hi @joewitt and @rickysaltzer , I am following the PutParquet to implement the PutKudu, and so have the same dependencies as the PutParquet's, and got it build success. But build on Travis was always fail. And I realized this morning that the reason is Record Reader is not properly referenced and deployed for my PutKudu (I deployed it manually to test the PutKudu first). I checked the log, and here is the message: "ClassNotFoundException: org.apache.nifi.serialization.RecordReaderFactory" I compare every files of PutParquet with my PutKudu, but couldn't find any missing part of it. Would you please advise? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user cammachusa commented on the issue: https://github.com/apache/nifi/pull/2020 Look like, i messed up the history again. I just pushed a commit to the branch, then comments gone :( @rickysaltzer , can you review it again? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user cammachusa commented on the issue: https://github.com/apache/nifi/pull/2020 Hi @joewitt , there is a sandbox that has Kudu instance (and related components) that you can quickly spin up a VM and test the processor. https://kudu.apache.org/docs/quickstart.html In case you need something diff, I can also provision an AWS's VM with KuDu installed and give you the access? Btw, can you find me a second reviewer? Thanks, --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user cammachusa commented on the issue: https://github.com/apache/nifi/pull/2020 @joewitt , my apologize again. I'm still new to the process, so mess it up a bit. I did some manual test with 2 diff Kudu env, and with CSV file as a source. Just in case, you or other reviewers ask about GetKudu processor, I will add one after this PutKudu get approved. Thanks, --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2020 will have to take more of a look. As a general practice to help reviewers it is probably best during the review/feedback phase to just add new commits rather than force commit on top of it. This helps to see/discuss progress rather than starting over. No biggie just bringing it up to help you help your reviewers. At this stage I need an instance of Kudu to test against. Did some looking into it last night and it looked like it was going to be a bit involved than I had time for. So hopefully myself or someone else familiar with kudu can help more soon. You've done some really nice work in here for sure. As for other processors with similar patterns the gotcha can be whether their clients auto reconnect, are thread safe, etc.. Each situation can differ quite a lot. Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user cammachusa commented on the issue: https://github.com/apache/nifi/pull/2020 @joewitt , thanks for the goods points. I looked at PutCassandra, PutHBase, PutKite, to build this PutKudu, and so forget to move those objects to onTrigger when following your suggestion, PutParquet as example. I just fixed it, please review and let me know what else I should adjust? Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2020 this is a continuation of work from #1874 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
