[GitHub] nifi issue #1126: NIFI-1769: Implemented SSE with KMS.
Github user miquillo commented on the issue: https://github.com/apache/nifi/pull/1126 I'm sorry but I don't see any time in the coming weeks to finish this. We're currently not using this feature anymore, so I propose we close this PR and don't add it to the project. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1126: Implemented SSE with KMS. Jira #NIFI-1769
GitHub user miquillo opened a pull request: https://github.com/apache/nifi/pull/1126 Implemented SSE with KMS. Jira #NIFI-1769 Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/miquillo/nifi master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1126.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1126 commit f28b8feb324ed79207e0f03761163c1e8baf64d9 Author: Michiel Moonen <michiel.moonen_exter...@wartsila.com> Date: 2016-10-12T14:13:37Z implemented SSE with KMS --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #362: NIFI-1769: added support for SSE-KMS and signature s...
Github user miquillo closed the pull request at: https://github.com/apache/nifi/pull/362 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #239: Nifi 1540 - AWS Kinesis Put Processor
Github user miquillo commented on the issue: https://github.com/apache/nifi/pull/239 @jvwing I kind of overlooked the validator for 1-500 for the BATCH_SIZE property, didn't realize it actually captures the constraint of the package size. Go with the approach currently implemented. @mans2singh @jvwing great work! Hope to see this is the next release :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #239: Nifi 1540 - AWS Kinesis Put Processor
Github user miquillo commented on the issue: https://github.com/apache/nifi/pull/239 ouch, that's painful... we've just committed very similar code in NIFI-1769 (PR #362, see reference), since we required a kinesis streams processor block. I propose to take this solution further though, as it seems more mature and already has been reviewed a few times. My comments: - At the time of reading, the name of the issue / PR didn't make us realize enough that this processor block was already build. Kinesis is the Umbrella, which currently contains three services: Firehose, Streams and Analytics. This processor block only works for Kinesis Streams, Please rename both code and PR/Jira :) (+ also the tests under a subfolder 'streams') - The API explains some restrictions to PutRecords(): > Each PutRecords request can support up to 500 records. Each record in the request can be as large as 1 MB, up to a limit of 5 MB for the entire request, including partition keys. Each shard can support writes up to 1,000 records per second, up to a maximum data write total of 1 MB per second.[http://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecords.html] Data needs to be chunked into multiple PutRecords calls if the amount of records > 500. We used the following: `List recordChuncks = Lists.partition(records, 500);` - We added a NR_SHARDS parameter for the sake of resharding (scaling up/down) in the future. Although we didn't implement a resharding mechanism, it's perhaps worth considering. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #362: NIFI-1769: added support for SSE-KMS and signature s3v4 aut...
Github user miquillo commented on the issue: https://github.com/apache/nifi/pull/362 @jvwing OUCH! that's painfull to see it's exactly similar to what we've build.. :( I reviewed the code and it seems that it is more mature than our solution and VERY similar :) I'll leave my comments there. I can see if can fix/remove the code we've committed concerning the putkinesis, so this PR will only contain fixes for SSE-KMS. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #362: NIFI-1769: added support for SSE-KMS and signature s3v4 aut...
Github user miquillo commented on the issue: https://github.com/apache/nifi/pull/362 @jvwing So it has been a bit silent here, but we continued last week and now we've merged the current master branch into this project. This is where I realized you've implemented the S3v4 (along with v2) already in master, so I've removed our implementation from this branch. The KMS feature is still in, merged with your implementation of S3V4 [using SetSignerOverride() ]. Furthermore we've included a new processor! It's the Kinesis Streams processor, basically the streaming version of firehose. Hope you like it :) Will check Jira and update accordingly. I was unable to built the current master succesfully b/o failing tests in other packages, but it should be working. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
