[GitHub] nifi pull request #458: NIFI-1829 - Create new DebugFlow processor.

2016-06-01 Thread jskora
Github user jskora commented on a diff in the pull request: https://github.com/apache/nifi/pull/458#discussion_r65487473 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DebugFlow.java --- @@ -0,0 +1,414 @@

[GitHub] nifi pull request #458: NIFI-1829 - Create new DebugFlow processor.

2016-06-01 Thread jskora
Github user jskora commented on a diff in the pull request: https://github.com/apache/nifi/pull/458#discussion_r65487334 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DebugFlow.java --- @@ -0,0 +1,414 @@

[GitHub] nifi pull request #458: NIFI-1829 - Create new DebugFlow processor.

2016-06-01 Thread jskora
Github user jskora commented on a diff in the pull request: https://github.com/apache/nifi/pull/458#discussion_r65486064 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DebugFlow.java --- @@ -0,0 +1,414 @@

[GitHub] nifi pull request #444: NIFI-1118 adding SplitText features of size limit an...

2016-06-01 Thread markobean
Github user markobean commented on a diff in the pull request: https://github.com/apache/nifi/pull/444#discussion_r65471941 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestSplitText.java --- @@ -39,6 +3

Re: cron updating value used by another processor

2016-06-01 Thread Edgardo Vega
Bryan, I had not even thought of using those two processor. I will try tomorrow. Thanks for the tip. Cheers, Edgardo On Wed, Jun 1, 2016 at 8:23 PM, Bryan Bende wrote: > I think PutDistributedMapCache and FetchDistributedMapCache could be used > for this. > > -Bryan > > On Wed, Jun 1, 2016 at

[GitHub] nifi pull request #444: NIFI-1118 adding SplitText features of size limit an...

2016-06-01 Thread mosermw
Github user mosermw commented on a diff in the pull request: https://github.com/apache/nifi/pull/444#discussion_r65469870 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/SplitText.java --- @@ -147,138 +193,

[GitHub] nifi pull request #444: NIFI-1118 adding SplitText features of size limit an...

2016-06-01 Thread markobean
Github user markobean commented on a diff in the pull request: https://github.com/apache/nifi/pull/444#discussion_r65468881 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/SplitText.java --- @@ -147,138 +19

[GitHub] nifi pull request #444: NIFI-1118 adding SplitText features of size limit an...

2016-06-01 Thread markobean
Github user markobean commented on a diff in the pull request: https://github.com/apache/nifi/pull/444#discussion_r65467699 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/SplitText.java --- @@ -147,138 +19

[GitHub] nifi issue #483: NIFI-1899 - Introduce ExtractEmailAttachments processor

2016-06-01 Thread bbende
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/483 Sorry that second line should have been something like: `split = session.putAttribute(split, "split.parent.uuid", parentUUID);` You wouldn't want to overwrite the regular UUID :) --- I

[GitHub] nifi pull request #444: NIFI-1118 adding SplitText features of size limit an...

2016-06-01 Thread markobean
Github user markobean commented on a diff in the pull request: https://github.com/apache/nifi/pull/444#discussion_r65467195 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/SplitText.java --- @@ -147,138 +19

[GitHub] nifi pull request #444: NIFI-1118 adding SplitText features of size limit an...

2016-06-01 Thread markobean
Github user markobean commented on a diff in the pull request: https://github.com/apache/nifi/pull/444#discussion_r65467161 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/SplitText.java --- @@ -94,12 +118,

[GitHub] nifi issue #483: NIFI-1899 - Introduce ExtractEmailAttachments processor

2016-06-01 Thread bbende
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/483 @trixpan if you want to add the UUDI of the original FlowFile you would get it from the attributes: `String parentUuid = originalFlowFile.getAttribute(CoreAttributes.UUID.key());` Then

Re: cron updating value used by another processor

2016-06-01 Thread Bryan Bende
I think PutDistributedMapCache and FetchDistributedMapCache could be used for this. -Bryan On Wed, Jun 1, 2016 at 8:17 PM, Edgardo Vega wrote: > Trying to figure out the best way to do the following. I want to have a > flow that runs every few minutes goes to url, pulls out a piece of json, > a

cron updating value used by another processor

2016-06-01 Thread Edgardo Vega
Trying to figure out the best way to do the following. I want to have a flow that runs every few minutes goes to url, pulls out a piece of json, and puts it somewhere that another processor will use for every flow file the comes in via another stream. The other processor would be something like Rou

[GitHub] nifi issue #483: NIFI-1899 - Introduce ExtractEmailAttachments processor

2016-06-01 Thread trixpan
Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/483 @bbende You are correct, the provenance trail is there, however, generally it seems like the Split* processors tend to add the split source information as well in the shape of > `seg

[GitHub] nifi issue #239: Nifi 1540 - AWS Kinesis Get and Put Processors

2016-06-01 Thread mans2singh
Github user mans2singh commented on the issue: https://github.com/apache/nifi/pull/239 Ok - I will add more logging, let me know if there is any other observation you have. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] nifi pull request #474: NIFI-1919 Add replaceFirst method to expression lang...

2016-06-01 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/474 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

[GitHub] nifi issue #239: Nifi 1540 - AWS Kinesis Get and Put Processors

2016-06-01 Thread jvwing
Github user jvwing commented on the issue: https://github.com/apache/nifi/pull/239 Checkpointing after catching an exception would keep things moving, I can see the benefit there. But what is the advantage of creating a failed FlowFile over just logging the exception? What would a u

[GitHub] nifi issue #239: Nifi 1540 - AWS Kinesis Get and Put Processors

2016-06-01 Thread mans2singh
Github user mans2singh commented on the issue: https://github.com/apache/nifi/pull/239 Hi @jvwing: Regarding exception handling - Kinesis will automatically handle retrying if there is kinesis client error from the last checkpoint created in the GetKinesisStream.

[GitHub] nifi pull request #444: NIFI-1118 adding SplitText features of size limit an...

2016-06-01 Thread mosermw
Github user mosermw commented on a diff in the pull request: https://github.com/apache/nifi/pull/444#discussion_r65440697 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestSplitText.java --- @@ -39,6 +39,

[GitHub] nifi pull request #487: NIFI-1956 added 'keyboard-interactive' option to SFT...

2016-06-01 Thread olegz
GitHub user olegz opened a pull request: https://github.com/apache/nifi/pull/487 NIFI-1956 added 'keyboard-interactive' option to SFTPTransfer You can merge this pull request into a Git repository by running: $ git pull https://github.com/olegz/nifi NIFI-1956 Alternatively yo

[GitHub] nifi issue #239: Nifi 1540 - AWS Kinesis Get and Put Processors

2016-06-01 Thread jvwing
Github user jvwing commented on the issue: https://github.com/apache/nifi/pull/239 Thanks, @mans2singh, I will review the changes. I don't think the AppVeyor build is reliable, there don't appear to be any recent successful builds. For GetKinesisStream exceptions, I think you

[GitHub] nifi issue #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/349 Currently the only tests are integration tests that rely on an AWS instance. I would like to see more true unit tests. For inspiration, check out the Publish/Consume MQTT unit tests[1]. [1]

[GitHub] nifi issue #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/349 A couple places have weird comments like "// @TRACE 252=connect to host {0} port {1} timeout {2}". Is this a convention I'm not aware of or left over from copy/paste? --- If your project is set up

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65425287 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/AbstractAWSIoTProcessor.java --- @@ -0,0 +1,

[GitHub] nifi pull request #444: NIFI-1118 adding SplitText features of size limit an...

2016-06-01 Thread mosermw
Github user mosermw commented on a diff in the pull request: https://github.com/apache/nifi/pull/444#discussion_r65425886 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/SplitText.java --- @@ -147,138 +193,

[GitHub] nifi pull request #444: NIFI-1118 adding SplitText features of size limit an...

2016-06-01 Thread mosermw
Github user mosermw commented on a diff in the pull request: https://github.com/apache/nifi/pull/444#discussion_r65425598 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/SplitText.java --- @@ -147,138 +193,

[GitHub] nifi pull request #481: NiFi 1945 - Corrects minor issue in HashContent "

2016-06-01 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/481 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65424552 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/AbstractAWSIoTProcessor.java --- @@ -0,0 +1,

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread KayLerch
Github user KayLerch commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65423497 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/GetAWSIoT.java --- @@ -0,0 +1,153 @@ +/*

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread KayLerch
Github user KayLerch commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65423267 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/util/MqttWebSocketAsyncClient.java --- @@ -0,0

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65423265 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/GetAWSIoTShadow.java --- @@ -0,0 +1,116 @@

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65422861 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/GetAWSIoTShadow.java --- @@ -0,0 +1,116 @@

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread KayLerch
Github user KayLerch commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65422907 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/PutAWSIoT.java --- @@ -0,0 +1,141 @@ +/*

[GitHub] nifi issue #481: NiFi 1945 - Corrects minor issue in HashContent "

2016-06-01 Thread pvillard31
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/481 Good catch. Will merge. However not sure why the travis build failed... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pr

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread KayLerch
Github user KayLerch commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65422534 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/PutAWSIoT.java --- @@ -0,0 +1,141 @@ +/*

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread KayLerch
Github user KayLerch commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65422291 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/PutAWSIoT.java --- @@ -0,0 +1,141 @@ +/*

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread KayLerch
Github user KayLerch commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65421951 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/AbstractAWSIoTProcessor.java --- @@ -0,0 +1,23

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65421171 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/GetAWSIoT.java --- @@ -0,0 +1,153 @@ +/*

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread KayLerch
Github user KayLerch commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65421271 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/AbstractAWSIoTProcessor.java --- @@ -0,0 +1,23

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread KayLerch
Github user KayLerch commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65421228 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/AbstractAWSIoTProcessor.java --- @@ -0,0 +1,23

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread KayLerch
Github user KayLerch commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65421049 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/AbstractAWSIoTProcessor.java --- @@ -0,0 +1,23

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65420897 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/PutAWSIoT.java --- @@ -0,0 +1,141 @@ +/*

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65420794 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/PutAWSIoT.java --- @@ -0,0 +1,141 @@ +/*

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65420622 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/PutAWSIoTShadow.java --- @@ -0,0 +1,110 @@

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65420217 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/util/IoTMessage.java --- @@ -0,0 +1,44 @@

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65419729 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/GetAWSIoT.java --- @@ -0,0 +1,153 @@ +/*

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65418924 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/GetAWSIoT.java --- @@ -0,0 +1,153 @@ +/*

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65418441 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/AbstractAWSIoTProcessor.java --- @@ -0,0 +1,

[GitHub] nifi pull request #444: NIFI-1118 adding SplitText features of size limit an...

2016-06-01 Thread mosermw
Github user mosermw commented on a diff in the pull request: https://github.com/apache/nifi/pull/444#discussion_r65418261 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/SplitText.java --- @@ -147,138 +193,

[GitHub] nifi issue #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/349 The Get/PutAWSIoT processors read and overwrite mqttClient many times without a read/write lock, this can lead to problems in multi-threaded scenarios. For example, the both processors call "Abstra

[GitHub] nifi issue #474: NIFI-1919 Add replaceFirst method to expression language

2016-06-01 Thread mattyb149
Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/474 I built and ran various tests with the new (and existing) replaceXYZ functions, everything looks good. +1 --- If your project is set up for it, you can reply to this email and have your rep

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65416412 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/AbstractAWSIoTProcessor.java --- @@ -0,0 +1,

[GitHub] nifi pull request #466: NIFI-1915 Fixed '$' escape logic.

2016-06-01 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/466 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

[GitHub] nifi pull request #444: NIFI-1118 adding SplitText features of size limit an...

2016-06-01 Thread mosermw
Github user mosermw commented on a diff in the pull request: https://github.com/apache/nifi/pull/444#discussion_r65414058 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/SplitText.java --- @@ -94,12 +118,16

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65413629 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/AbstractAWSIoTProcessor.java --- @@ -0,0 +1,

[GitHub] nifi issue #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/349 For the new dependencies, NOTICE/LICENSE information needs to be added if applicable. Paho is EPL 1.0 licensed and requires being added to the nar NOTICE like so[1]. Do the other two new deps have l

Re: back pressure

2016-06-01 Thread pradeepbill
ok will get the latest and try, will update you soon. -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/back-pressure-tp10801p10903.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65412507 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/util/MqttWebSocketAsyncClient.java --- @@ -0

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65411771 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/GetAWSIoT.java --- @@ -0,0 +1,153 @@ +/*

Re: back pressure

2016-06-01 Thread Bryan Bende
Just wanted to add, if you are using Maven it would be the spark receiver 0.6.1 which would bring in all the other dependencies: http://central.maven.org/maven2/org/apache/nifi/nifi-spark-receiver/0.6.1/nifi-spark-receiver-0.6.1.pom Otherwise you can grab the 0.6.1 version of each of the jars you

Re: back pressure

2016-06-01 Thread Bryan Bende
Ok can you try getting the 0.6.1 version of those jars and see if the problem still occurrs? I want to make sure it still happens in the latest code before we change anything, because I know we fixed a similar serialization issue before. http://central.maven.org/maven2/org/apache/nifi/nifi-site-t

Re: back pressure

2016-06-01 Thread pradeepbill
I am using the below jars, Please see the respective versions, there is no default storage level really, but in the examples listed below use MEMORY_ONLY storage level, I had to change because the MEMORY_ONLY option does not work always, sometimes when there is a spill we may need DISK as well. ht

[GitHub] nifi issue #474: NIFI-1919 Add replaceFirst method to expression language

2016-06-01 Thread alopresto
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/474 @joewitt No existing methods are changing in this fix. A new ticket is filed for NIFI-1938 to address any breaking changes. --- If your project is set up for it, you can reply to this email and hav

[GitHub] nifi pull request #474: NIFI-1919 Add replaceFirst method to expression lang...

2016-06-01 Thread alopresto
Github user alopresto commented on a diff in the pull request: https://github.com/apache/nifi/pull/474#discussion_r65406019 --- Diff: nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/test/groovy/org/apache/nifi/processors/script/ExecuteScriptGroovyTest.groovy ---

[GitHub] nifi pull request #474: NIFI-1919 Add replaceFirst method to expression lang...

2016-06-01 Thread alopresto
Github user alopresto commented on a diff in the pull request: https://github.com/apache/nifi/pull/474#discussion_r65405940 --- Diff: nifi-commons/nifi-expression-language/src/main/java/org/apache/nifi/attribute/expression/language/evaluation/functions/ReplaceEvaluator.java --- @@

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65405495 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/util/MqttWebSocketAsyncClient.java --- @@ -0

[GitHub] nifi issue #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/349 A bit nit picky and I may be alone here, but I find putting "if" statements and their content on the same line a bit hard to read when trying to read someone else's code. For example line 107 of Get

Re: back pressure

2016-06-01 Thread Bryan Bende
Glad to hear you were able to optimize the NiFi side of things. As for the other error, it looks like we might need to make the ReceiveRunnable class serializable. Can you confirm what version of NiFi you are using? also, I am not that familiar with Spark streaming, what is the default StorageLev

[GitHub] nifi pull request #486: NIFI-1929: Improvements for PutHDFS attribute handli...

2016-06-01 Thread mattyb149
GitHub user mattyb149 opened a pull request: https://github.com/apache/nifi/pull/486 NIFI-1929: Improvements for PutHDFS attribute handling You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattyb149/nifi NIFI-1929 Alternatively y

[GitHub] nifi issue #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/349 Does AWS IoT support communicating over SSL with MQTT or shadows? If so, that option should be added. --- If your project is set up for it, you can reply to this email and have your reply appear on

Re: back pressure

2016-06-01 Thread pradeepbill
here is a follow up on that Spark exception.If I change the StorageLevel option to MEMORY_AND_DISK_SER, JavaReceiverInputDStream packetStream = ssc.receiverStream(new NiFiReceiver(config, StorageLevel.MEMORY_AND_DISK_SER())); I get below exception 16/06/01 12:50:28 ERROR

Re: back pressure

2016-06-01 Thread pradeepbill
HI Bryan, here is my update since yesterday FYI, I was using ListenSyslog with TCP , Also after trying with the recommendations from your website, I could get all of them to work well, except Spark throwing exceptions like "Could not compute split, block input-0-1464774108087 not found", which I

[GitHub] nifi issue #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/349 The more I type out and try to easily distinguish GetAWSIoT and PutAWSIoT the more I think MQTT should be integrated into the name and all four of the processors should be Publish/Consume.

[GitHub] nifi pull request #485: NIFI-1948 Fixed encoding-verison attribute not getti...

2016-06-01 Thread jtstorck
GitHub user jtstorck opened a pull request: https://github.com/apache/nifi/pull/485 NIFI-1948 Fixed encoding-verison attribute not getting created in exp… …orted templates You can merge this pull request into a Git repository by running: $ git pull https://github.com/jtstor

[GitHub] nifi issue #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/349 I actually already committed general MQTT processors [1] and they have some good "best practices" with regards to publishing/consuming messages and MQTT I think would be good to follow here. The fir

[GitHub] nifi pull request #478: NIFI-1754 Rollback log messages should include the f...

2016-06-01 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/478 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread joewitt
Github user joewitt commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65388471 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/PutAWSIoT.java --- @@ -0,0 +1,141 @@ +/*

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65387960 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/PutAWSIoT.java --- @@ -0,0 +1,141 @@ +/*

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65387577 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/PutAWSIoT.java --- @@ -0,0 +1,141 @@ +/*

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65385897 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/PutAWSIoT.java --- @@ -0,0 +1,141 @@ +/*

[GitHub] nifi pull request #484: Nifi-1808-1.0-add-license

2016-06-01 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/484 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

[GitHub] nifi issue #474: NIFI-1919 Add replaceFirst method to expression language

2016-06-01 Thread joewitt
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/474 Please just make sure we don't create problems for old flows. Can change names if we have a smooth transition for existing EL statements. --- If your project is set up for it, you can reply to this

[GitHub] nifi pull request #467: Nifi-1808-0.x-add-license

2016-06-01 Thread JPercivall
Github user JPercivall closed the pull request at: https://github.com/apache/nifi/pull/467 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is en

[GitHub] nifi pull request #474: NIFI-1919 Add replaceFirst method to expression lang...

2016-06-01 Thread mattyb149
Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/474#discussion_r65383800 --- Diff: nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/test/groovy/org/apache/nifi/processors/script/ExecuteScriptGroovyTest.groovy ---

[GitHub] nifi issue #239: Nifi 1540 - AWS Kinesis Get and Put Processors

2016-06-01 Thread mans2singh
Github user mans2singh commented on the issue: https://github.com/apache/nifi/pull/239 Hi @jvwing Just wanted to check if you were able to test the processors and if you have any additional review comments/recommendations for me. Thanks Mans --- If your

[GitHub] nifi issue #483: NIFI-1899 - Introduce ExtractEmailAttachments processor

2016-06-01 Thread bbende
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/483 @joewitt @trixpan I think the provenance linkage should already be happening automatically with the following line: `FlowFile split = session.create(originalFlowFile);` With that call,

[GitHub] nifi issue #424: NIFI-1850 - Initial Commit for JSON-to-JSON Schema Converte...

2016-06-01 Thread mcgilman
Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/424 Functionality after the most recent update looks good. I've made a few minor stylistic changes and updated method visibility here [1]. Will merge to 0.x baseline. Thanks! [1] https://github.

[GitHub] nifi pull request #349: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/349#discussion_r65374220 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/iot/AbstractAWSIoTProcessor.java --- @@ -0,0 +1,

Re: Jetty request filtering of NiFI API requests

2016-06-01 Thread Matt Gilman
Joe, It needs to be baked into the framework. Matt On Wed, Jun 1, 2016 at 9:58 AM, Joe Skora wrote: > Matt, > > That looks like what I'm looking for. > > I haven't looked at how the web.xml pieces get put together, can I package > it separately so it can hot deploy or does it need to be baked

[GitHub] nifi pull request: Nifi-1808-1.0-add-license

2016-06-01 Thread JPercivall
GitHub user JPercivall opened a pull request: https://github.com/apache/nifi/pull/484 Nifi-1808-1.0-add-license You can merge this pull request into a Git repository by running: $ git pull https://github.com/JPercivall/nifi NIFI-1808-1.0-add-license Alternatively you can revi

Re: Jetty request filtering of NiFI API requests

2016-06-01 Thread Joe Skora
Matt, That looks like what I'm looking for. I haven't looked at how the web.xml pieces get put together, can I package it separately so it can hot deploy or does it need to be baked into the framework? Thanks JoeS On Tue, May 31, 2016 at 3:26 PM, Matt Gilman wrote: > Joe, > > There is no exte

[GitHub] nifi pull request: NIFI-1754 Rollback log messages should include the flowfi...

2016-06-01 Thread trkurc
Github user trkurc commented on the pull request: https://github.com/apache/nifi/pull/478 looks good. if you don't mind, I'll amend the commit and throw a static on that constant before pushing --- If your project is set up for it, you can reply to this email and have your reply appe

[GitHub] nifi pull request: NIFI-1118 adding SplitText features of size limit and hea...

2016-06-01 Thread mosermw
Github user mosermw commented on the pull request: https://github.com/apache/nifi/pull/444 Reviewing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] nifi pull request: NIFI-1118 Update SplitText Processor - add support for sp...

2016-06-01 Thread mosermw
Github user mosermw commented on the pull request: https://github.com/apache/nifi/pull/280 @jskora Are we closing this PR? It appears the latest PR for NIFI-1118 is [PR#444](https://github.com/apache/nifi/pull/444) --- If your project is set up for it, you can reply to this email an

[GitHub] nifi pull request: NIFI-1767 AWS IoT processors

2016-06-01 Thread JPercivall
Github user JPercivall commented on the pull request: https://github.com/apache/nifi/pull/349 Yup I will start an in-depth review --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

Re: Help with ReplaceTextWithMapping processor: multi-column mappings

2016-06-01 Thread idioma
Hi Andy, apologies for the delay in replying. I have come back to the task and tried to write a simple Groovy script in order to make the substitutions coming from the mapping files. In order to have something working, I have modified the mapping files, so that each of the incoming Json values init

[GitHub] nifi pull request: NIFI-1899 - Introduce ExtractEmailAttachments processor

2016-06-01 Thread joewitt
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/483 I think the behavior you have now is perfect (it should auto decode from base64) as it honors the user's intent. As far as tracing to the parent you just need to connect the provenance

[GitHub] nifi pull request: NIFI-1919 Add replaceFirst method to expression language

2016-06-01 Thread mattyb149
Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/474#discussion_r65358298 --- Diff: nifi-commons/nifi-expression-language/src/main/java/org/apache/nifi/attribute/expression/language/evaluation/functions/ReplaceEvaluator.java --- @@

[GitHub] nifi pull request: NIFI-1919 Add replaceFirst method to expression language

2016-06-01 Thread mattyb149
Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/474#discussion_r65358238 --- Diff: nifi-docs/src/main/asciidoc/expression-language-guide.adoc --- @@ -884,7 +916,7 @@ Expressions will provide the following results:

  1   2   >