date:20160204

[GitHub] nifi pull request: Fix NiFi-1461

2016-02-04 Thread PuspenduBanerjee

Github user PuspenduBanerjee commented on the pull request:

https://github.com/apache/nifi/pull/204#issuecomment-180234634
  
I understand people are busy with 0.5 rel , still does anyone have time & 
interest  to start reviewing, this major [ as per issue tracker] issue?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: planning for apache nifi 0.5.0

2016-02-04 Thread Aldrin Piri

All,

Just as an update on some of the remaining items.  I have been doing some
review work to help close out some of these final issues.  Finished review
and merged 1257 and 1259 which brings a lot of great security and
encryption functions that will help a long way for providing a nice
foundation for additional support across the entirety of the project.

Along with several of the members of the community, I have been reviewing
and providing feedback 259 along with several other members of the
community and its incorporated tickets of 1223, 1379.  We are doing some
final testing particularly surrounding its interactions and utilization of
Kerberos.  Once that wraps up, we will be able to merge that in, closing
the last of the items scheduled for this release.

Thanks!
Aldrin

On Thu, Feb 4, 2016 at 9:53 AM, Joe Witt  wrote:

> Tony
>
> Agreed.  I will work through these as well and try to ensure the
> wording accurately reflect what happened.  The release notes are a
> really important piece of communication we need to get right.
>
> Thanks
> Joe
>
> On Thu, Feb 4, 2016 at 6:04 AM, Tony Kurc  wrote:
> > I am prepping for the release, going through the jiras that have been
> > closed, I think several have a description that does not match. Good
> > example is:
> >
> > https://issues.apache.org/jira/browse/NIFI-1325
> >
> > I believe the final scope of the patch surpassed the original
> description.
> > Should this be adjusted for people perusing the JIRA report we link to in
> > our release notes? I'd say yes. I made a couple small edits to some
> tickets
> > already for increased readability. One change I wanted to make was change
> > NIFI-259 to "New Feature" rather than subtask, but I was concerned about
> > this disrupting folks.
> >
> > I think a quick consistency and spell check would be good - I've had
> > several people mention out of band that prior release notes had a couple
> > errors. Apparently people love the release notes page on the JIRA
> > NIFI-1107 is done, just needs a merge, which I'll do tonight.
> >
> > On Wed, Feb 3, 2016 at 4:32 PM, Joe Witt  wrote:
> >
> >> Latest update for all
> >>
> >> You can see the current status here [1].
> >>
> >> Looks like several tickets but really it is three efforts:
> >> 1) Improve encryption options in EncryptContent NIFI-1257,1259
> >> 2) Provide state management as a framework function NIFI-259,1223,1339
> >> 3) Support multi-part S3 uploads NIFI-1107
> >>
> >> Based on comments on each they are just in very late stage/detailed
> >> review.  So we're close.
> >>
> >> Thanks
> >> Joe
> >>
> >> [1]
> >>
> https://issues.apache.org/jira/browse/NIFI-1339?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%200.5.0%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
> >>
> >> On Thu, Jan 21, 2016 at 12:55 AM, Joe Witt  wrote:
> >> > Team
> >> >
> >> > Did a full round of 050 JIRA cleanup this evening.  Great progress is
> >> > being made.
> >> >
> >> > For the proposed NiFi 050 release schedule above we have about 10 days
> >> > of code time left.  There is a pretty large backlog of patch awaiting
> >> > items currently showing as aligned to 050.  See here [1].
> >> >
> >> > Also there are 25 (as of this writing) open tickets.  See here [2].
> >> >
> >> > For the proposed goals of the release as listed previous in this
> >> > thread it appears the following are likely to occur in 050:
> >> >   Improved encryption handling: NIFI-1324,1314,1259,
> >> >   Improved Hive/Kite interaction: NIFI-1193
> >> >   Interactive Queue Management: NIFI-108
> >> >   Support for numerous scripting/languages: NIFI-210
> >> >   Support for state management: NIFI-259
> >> >
> >> > The following are at risk or moving out:
> >> >   NiFi kerberos support: NIFI-1274
> >> > No progress on it yet.  Good 060 target.
> >> >   UX/UI improvements: NIFI-951
> >> > These changes appear to be better fit for 1.0.0.
> >> >
> >> > [1]
> >>
> https://issues.apache.org/jira/browse/NIFI-1416?jql=project%20%3D%20NIFI%20AND%20status%20%3D%20%22Patch%20Available%22%20AND%20fixVersion%20%3D%200.5.0
> >> >
> >> > [2]
> >>
> https://issues.apache.org/jira/browse/NIFI-1404?jql=project%20%3D%20NIFI%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%200.5.0
> >> >
> >> > Thanks
> >> > Joe
> >> >
> >> > On Tue, Dec 29, 2015 at 10:08 PM, Joe Witt 
> wrote:
> >> >> Thanks Tony - let's plan on that then.
> >> >>
> >> >> On Tue, Dec 29, 2015 at 9:58 PM, Oleg Zhurakousky
> >> >>  wrote:
> >> >>> Be careful what you wish for ;)
> >> >>>
> >> >>> Sent from my iPhone
> >> >>>
> >>  On Dec 29, 2015, at 17:44, Tony Kurc  wrote:
> >> 
> >>  Joe,
> >>  I might like to give the release a try.
> >> 
> >>  Tony
> >> 
> >> > On Mon, Dec 28, 2015 at 4:30 PM, Joe Witt 
> >> wrote:
> >> >
> >> > Team,
> >> >
> >> > In the spirit of shooting for roughly 6 week release cycles a

Re: ConvertCSVtoAvro | support for "||" delimiter

2016-02-04 Thread Thad Guidry

Non Printable characters are also great to use for this usecase (The ASCII
Control Characters which were designed for exactly this in the early days
of computing!)  (Just copy and paste from an editor or Notepad... on
Windows you can get Char 2 by holding down ALT and then using Numeric
Keypad to type 002 )

CHAR 2 (traditionally the Start Of Text or STX) is a great delimiter.
 \u0002

http://www.fileformat.info/info/unicode/char/0002/index.htm

Thad
+ThadGuidry

[GitHub] nifi pull request: NIFI-1257 and 1259

2016-02-04 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/201


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-924:Nifi-Camel Integration

2016-02-04 Thread PuspenduBanerjee

Github user PuspenduBanerjee commented on the pull request:

https://github.com/apache/nifi/pull/197#issuecomment-180056781
  
Will reopen for after 0.5 release.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: ConvertCSVtoAvro | support for "||" delimiter

2016-02-04 Thread Tony Kurc

I believe the processor supports \u notation for a delimiter as part of
a PR for 0.4.X.

On Thu, Feb 4, 2016 at 4:11 PM, Ryan Blue  wrote:

> I didn't know there was a unit separator character, thanks for the
> suggestion. I think I have a lot of ☃ to replace.
>
> If you can paste the unit separator character in, then it should work. The
> underlying code supports escape sequences, like \t, but the validation
> doesn't take those into account yet. That would be a good starter
> contribution for someone out there...
>
> rb
>
>
> On 02/04/2016 12:39 PM, Alan Jackoway wrote:
>
>> Though I love the concept of ☃ as your separator, my belief is that the
>> correct way to do this to replace your custom delimiter with the ones that
>> are defined in ASCII (and therefore extremely unlikely to appear in your
>> data): https://en.wikipedia.org/wiki/Delimiter#ASCII_delimited_text
>>
>> That said, I have not actually tried this with NiFi, so I don't know how
>> easy it is to specify ASCII character 31 as your separator in the UI.
>>
>> On Thu, Feb 4, 2016 at 2:34 PM, Ryan Blue  wrote:
>>
>> The underlying CSV library only supports a single-character delimiter, so
>>> it would be a bit of work to allow multi-char delimiters. Another
>>> solution
>>> is to use | as your delimiter and simply account for that in your file
>>> header. Everything is mapped by name, so you'd just have a bunch of
>>> columns
>>> named "" and it should work fine otherwise.
>>>
>>> That may not work if your delimiter is || because you might have | in
>>> your
>>> data, though. If that's the case, then I'd go with the suggestion from
>>> Joe
>>> to replace "||" with a single-character delimiter that you won't see in
>>> the
>>> data, like ☃.
>>>
>>> rb
>>>
>>>
>>> On 02/04/2016 06:50 AM, Joe Witt wrote:
>>>
>>> Not a direct answer but:
 With NIFI-210 arriving in the upcoming NiFi 0.5.0 release you will
 have a great option in scripting (Lua, Python, Ruby, Groovy,
 Javascript) that will let you rapidly get past these hurdles without
 having to build your own custom processor until you are sure what you
 need.

 Thanks
 Joe

 On Thu, Feb 4, 2016 at 6:48 AM, Tony Kurc  wrote:

 With that processor alone it doesn't appear so. The validator for that
> property requires it to be one character.
> On Feb 3, 2016 1:01 AM, "shweta"  wrote:
>
> Hi All,
>
>>
>> It seems "ConvertCSVtoAvro" only support single character as delimiter
>> in
>> Nifi. Is there a way to specify "||"
>> delimiter.
>>
>> Thanks,
>> Shweta
>>
>>
>>
>> --
>> View this message in context:
>>
>>
>> http://apache-nifi-developer-list.39713.n7.nabble.com/ConvertCSVtoAvro-support-for-delimiter-tp7116.html
>> Sent from the Apache NiFi Developer List mailing list archive at
>> Nabble.com.
>>
>>
>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Cloudera, Inc.
>>>
>>>
>>
>
> --
> Ryan Blue
> Software Engineer
> Cloudera, Inc.
>

[GitHub] nifi pull request: NIFI-924:Nifi-Camel Integration

2016-02-04 Thread PuspenduBanerjee

Github user PuspenduBanerjee closed the pull request at:

https://github.com/apache/nifi/pull/197


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: ConvertCSVtoAvro | support for "||" delimiter

2016-02-04 Thread Ryan Blue

I didn't know there was a unit separator character, thanks for the
suggestion. I think I have a lot of ☃ to replace.

If you can paste the unit separator character in, then it should work.
The underlying code supports escape sequences, like \t, but the
validation doesn't take those into account yet. That would be a good
starter contribution for someone out there...

On 02/04/2016 12:39 PM, Alan Jackoway wrote:

Though I love the concept of ☃ as your separator, my belief is that the
correct way to do this to replace your custom delimiter with the ones that
are defined in ASCII (and therefore extremely unlikely to appear in your
data): https://en.wikipedia.org/wiki/Delimiter#ASCII_delimited_text

That said, I have not actually tried this with NiFi, so I don't know how
easy it is to specify ASCII character 31 as your separator in the UI.

On Thu, Feb 4, 2016 at 2:34 PM, Ryan Blue wrote:

The underlying CSV library only supports a single-character delimiter, so
it would be a bit of work to allow multi-char delimiters. Another solution
is to use | as your delimiter and simply account for that in your file
header. Everything is mapped by name, so you'd just have a bunch of columns
named "" and it should work fine otherwise.

That may not work if your delimiter is || because you might have | in your
data, though. If that's the case, then I'd go with the suggestion from Joe
to replace "||" with a single-character delimiter that you won't see in the
data, like ☃.

On 02/04/2016 06:50 AM, Joe Witt wrote:

Not a direct answer but:
With NIFI-210 arriving in the upcoming NiFi 0.5.0 release you will
have a great option in scripting (Lua, Python, Ruby, Groovy,
Javascript) that will let you rapidly get past these hurdles without
having to build your own custom processor until you are sure what you
need.

Thanks
Joe

On Thu, Feb 4, 2016 at 6:48 AM, Tony Kurc wrote:

With that processor alone it doesn't appear so. The validator for that
property requires it to be one character.
On Feb 3, 2016 1:01 AM, "shweta" wrote:

Hi All,

It seems "ConvertCSVtoAvro" only support single character as delimiter
in
Nifi. Is there a way to specify "||"
delimiter.

Thanks,
Shweta

--
View this message in context:

http://apache-nifi-developer-list.39713.n7.nabble.com/ConvertCSVtoAvro-support-for-delimiter-tp7116.html
Sent from the Apache NiFi Developer List mailing list archive at
Nabble.com.

--
Ryan Blue
Software Engineer
Cloudera, Inc.

Re: ConvertCSVtoAvro | support for "||" delimiter

2016-02-04 Thread Alan Jackoway

Though I love the concept of ☃ as your separator, my belief is that the
correct way to do this to replace your custom delimiter with the ones that
are defined in ASCII (and therefore extremely unlikely to appear in your
data): https://en.wikipedia.org/wiki/Delimiter#ASCII_delimited_text

That said, I have not actually tried this with NiFi, so I don't know how
easy it is to specify ASCII character 31 as your separator in the UI.

On Thu, Feb 4, 2016 at 2:34 PM, Ryan Blue  wrote:

> The underlying CSV library only supports a single-character delimiter, so
> it would be a bit of work to allow multi-char delimiters. Another solution
> is to use | as your delimiter and simply account for that in your file
> header. Everything is mapped by name, so you'd just have a bunch of columns
> named "" and it should work fine otherwise.
>
> That may not work if your delimiter is || because you might have | in your
> data, though. If that's the case, then I'd go with the suggestion from Joe
> to replace "||" with a single-character delimiter that you won't see in the
> data, like ☃.
>
> rb
>
>
> On 02/04/2016 06:50 AM, Joe Witt wrote:
>
>> Not a direct answer but:
>>With NIFI-210 arriving in the upcoming NiFi 0.5.0 release you will
>> have a great option in scripting (Lua, Python, Ruby, Groovy,
>> Javascript) that will let you rapidly get past these hurdles without
>> having to build your own custom processor until you are sure what you
>> need.
>>
>> Thanks
>> Joe
>>
>> On Thu, Feb 4, 2016 at 6:48 AM, Tony Kurc  wrote:
>>
>>> With that processor alone it doesn't appear so. The validator for that
>>> property requires it to be one character.
>>> On Feb 3, 2016 1:01 AM, "shweta"  wrote:
>>>
>>> Hi All,

 It seems "ConvertCSVtoAvro" only support single character as delimiter
 in
 Nifi. Is there a way to specify "||"
 delimiter.

 Thanks,
 Shweta



 --
 View this message in context:

 http://apache-nifi-developer-list.39713.n7.nabble.com/ConvertCSVtoAvro-support-for-delimiter-tp7116.html
 Sent from the Apache NiFi Developer List mailing list archive at
 Nabble.com.


>
> --
> Ryan Blue
> Software Engineer
> Cloudera, Inc.
>

Re: ConvertCSVtoAvro | support for "||" delimiter

2016-02-04 Thread Ryan Blue

The underlying CSV library only supports a single-character delimiter,
so it would be a bit of work to allow multi-char delimiters. Another
solution is to use | as your delimiter and simply account for that in
your file header. Everything is mapped by name, so you'd just have a
bunch of columns named "" and it should work fine otherwise.

That may not work if your delimiter is || because you might have | in
your data, though. If that's the case, then I'd go with the suggestion
from Joe to replace "||" with a single-character delimiter that you
won't see in the data, like ☃.

On 02/04/2016 06:50 AM, Joe Witt wrote:

Thanks
Joe

On Thu, Feb 4, 2016 at 6:48 AM, Tony Kurc wrote:

With that processor alone it doesn't appear so. The validator for that
property requires it to be one character.
On Feb 3, 2016 1:01 AM, "shweta" wrote:

Hi All,

It seems "ConvertCSVtoAvro" only support single character as delimiter in
Nifi. Is there a way to specify "||"
delimiter.

Thanks,
Shweta

--
View this message in context:
http://apache-nifi-developer-list.39713.n7.nabble.com/ConvertCSVtoAvro-support-for-delimiter-tp7116.html
Sent from the Apache NiFi Developer List mailing list archive at
Nabble.com.

--
Ryan Blue
Software Engineer
Cloudera, Inc.

[GitHub] nifi pull request: NIFI-1257 and 1259

2016-02-04 Thread alopresto

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/201#discussion_r51930882
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/logback-test.xml
 ---
@@ -57,11 +60,12 @@
 
 
 
-
+
 
 
-
+
+
--- End diff --

I never actually got this to work. Reverting. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1257 and 1259

2016-02-04 Thread alopresto

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/201#discussion_r51930434
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/groovy/org/apache/nifi/processors/standard/util/crypto/CipherUtilityGroovyTest.groovy
 ---
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.util.crypto
+
+import org.apache.nifi.security.util.EncryptionMethod
+import org.bouncycastle.jce.provider.BouncyCastleProvider
+import org.junit.After
+import org.junit.Before
+import org.junit.BeforeClass
+import org.junit.Test
+import org.junit.runner.RunWith
+import org.junit.runners.JUnit4
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import java.security.Security
+
+@RunWith(JUnit4.class)
+class CipherUtilityGroovyTest extends GroovyTestCase {
+private static final Logger logger = 
LoggerFactory.getLogger(CipherUtilityGroovyTest.class)
+
+// TripleDES must precede DES for automatic grouping precedence
+private static final List CIPHERS = ["AES", "TRIPLEDES", 
"DES", "RC2", "RC4", "RC5", "TWOFISH"]
+private static final List SYMMETRIC_ALGORITHMS = 
EncryptionMethod.values().findAll { it.algorithm.startsWith("PBE") || 
it.algorithm.startsWith("AES") }*.algorithm
+private static final Map> 
ALGORITHMS_MAPPED_BY_CIPHER = SYMMETRIC_ALGORITHMS.groupBy { String algorithm 
-> CIPHERS.find { algorithm.contains(it) } }
+
+// Manually mapped as of 01/19/16 0.5.0
+private static final Map> 
ALGORITHMS_MAPPED_BY_KEY_LENGTH = [
+(40) : ["PBEWITHSHAAND40BITRC2-CBC",
+"PBEWITHSHAAND40BITRC4"],
+(64) : ["PBEWITHMD5ANDDES",
+"PBEWITHSHA1ANDDES"],
+(112): ["PBEWITHSHAAND2-KEYTRIPLEDES-CBC",
+"PBEWITHSHAAND3-KEYTRIPLEDES-CBC"],
+(128): ["PBEWITHMD5AND128BITAES-CBC-OPENSSL",
+"PBEWITHMD5ANDRC2",
+"PBEWITHSHA1ANDRC2",
+"PBEWITHSHA256AND128BITAES-CBC-BC",
+"PBEWITHSHAAND128BITAES-CBC-BC",
+"PBEWITHSHAAND128BITRC2-CBC",
+"PBEWITHSHAAND128BITRC4",
+"PBEWITHSHAANDTWOFISH-CBC",
+"AES/CBC/PKCS7Padding",
+"AES/CTR/NoPadding",
+"AES/GCM/NoPadding"],
+(192): ["PBEWITHMD5AND192BITAES-CBC-OPENSSL",
+"PBEWITHSHA256AND192BITAES-CBC-BC",
+"PBEWITHSHAAND192BITAES-CBC-BC",
+"AES/CBC/PKCS7Padding",
+"AES/CTR/NoPadding",
+"AES/GCM/NoPadding"],
+(256): ["PBEWITHMD5AND256BITAES-CBC-OPENSSL",
+"PBEWITHSHA256AND256BITAES-CBC-BC",
+"PBEWITHSHAAND256BITAES-CBC-BC",
+"AES/CBC/PKCS7Padding",
+"AES/CTR/NoPadding",
+"AES/GCM/NoPadding"]
+]
+
+@BeforeClass
+static void setUpOnce() {
+Security.addProvider(new BouncyCastleProvider());
+
+// Fix because TRIPLEDES -> DESede
+def tripleDESAlgorithms = 
ALGORITHMS_MAPPED_BY_CIPHER.remove("TRIPLEDES")
+ALGORITHMS_MAPPED_BY_CIPHER.put("DESede", tripleDESAlgorithms)
+
+logger.info("Mapped algorithms: ${ALGORITHMS_MAPPED_BY_CIPHER}")
+}
+
+@Before
+void setUp() throws Exception {
+
+}
+
+@After
+void tearDown() throws Exception {
+
+}
+
+@Test
+void testShouldParseCipherFromAlgorithm() {
+// Arrange
+final def EXPECTED_ALGORITHMS = ALGORITHMS_MAPPED_BY_CIPHER
+
+// Act
+SYMMETRIC_ALGORITHMS.each { String algorithm ->
+String cipher = 
CipherUtili

[GitHub] nifi pull request: NIFI-1257 and 1259

2016-02-04 Thread apiri

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi/pull/201#discussion_r51930432
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/logback-test.xml
 ---
@@ -57,11 +60,12 @@
 
 
 
-
+
 
 
-
+
+
--- End diff --

Did you intend to revert this back to the override log variable and default 
to INFO?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1257 and 1259

2016-02-04 Thread mattyb149

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/201#discussion_r51925709
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/groovy/org/apache/nifi/processors/standard/util/crypto/CipherUtilityGroovyTest.groovy
 ---
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.util.crypto
+
+import org.apache.nifi.security.util.EncryptionMethod
+import org.bouncycastle.jce.provider.BouncyCastleProvider
+import org.junit.After
+import org.junit.Before
+import org.junit.BeforeClass
+import org.junit.Test
+import org.junit.runner.RunWith
+import org.junit.runners.JUnit4
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import java.security.Security
+
+@RunWith(JUnit4.class)
+class CipherUtilityGroovyTest extends GroovyTestCase {
+private static final Logger logger = 
LoggerFactory.getLogger(CipherUtilityGroovyTest.class)
+
+// TripleDES must precede DES for automatic grouping precedence
+private static final List CIPHERS = ["AES", "TRIPLEDES", 
"DES", "RC2", "RC4", "RC5", "TWOFISH"]
+private static final List SYMMETRIC_ALGORITHMS = 
EncryptionMethod.values().findAll { it.algorithm.startsWith("PBE") || 
it.algorithm.startsWith("AES") }*.algorithm
+private static final Map> 
ALGORITHMS_MAPPED_BY_CIPHER = SYMMETRIC_ALGORITHMS.groupBy { String algorithm 
-> CIPHERS.find { algorithm.contains(it) } }
+
+// Manually mapped as of 01/19/16 0.5.0
+private static final Map> 
ALGORITHMS_MAPPED_BY_KEY_LENGTH = [
+(40) : ["PBEWITHSHAAND40BITRC2-CBC",
+"PBEWITHSHAAND40BITRC4"],
+(64) : ["PBEWITHMD5ANDDES",
+"PBEWITHSHA1ANDDES"],
+(112): ["PBEWITHSHAAND2-KEYTRIPLEDES-CBC",
+"PBEWITHSHAAND3-KEYTRIPLEDES-CBC"],
+(128): ["PBEWITHMD5AND128BITAES-CBC-OPENSSL",
+"PBEWITHMD5ANDRC2",
+"PBEWITHSHA1ANDRC2",
+"PBEWITHSHA256AND128BITAES-CBC-BC",
+"PBEWITHSHAAND128BITAES-CBC-BC",
+"PBEWITHSHAAND128BITRC2-CBC",
+"PBEWITHSHAAND128BITRC4",
+"PBEWITHSHAANDTWOFISH-CBC",
+"AES/CBC/PKCS7Padding",
+"AES/CTR/NoPadding",
+"AES/GCM/NoPadding"],
+(192): ["PBEWITHMD5AND192BITAES-CBC-OPENSSL",
+"PBEWITHSHA256AND192BITAES-CBC-BC",
+"PBEWITHSHAAND192BITAES-CBC-BC",
+"AES/CBC/PKCS7Padding",
+"AES/CTR/NoPadding",
+"AES/GCM/NoPadding"],
+(256): ["PBEWITHMD5AND256BITAES-CBC-OPENSSL",
+"PBEWITHSHA256AND256BITAES-CBC-BC",
+"PBEWITHSHAAND256BITAES-CBC-BC",
+"AES/CBC/PKCS7Padding",
+"AES/CTR/NoPadding",
+"AES/GCM/NoPadding"]
+]
+
+@BeforeClass
+static void setUpOnce() {
+Security.addProvider(new BouncyCastleProvider());
+
+// Fix because TRIPLEDES -> DESede
+def tripleDESAlgorithms = 
ALGORITHMS_MAPPED_BY_CIPHER.remove("TRIPLEDES")
+ALGORITHMS_MAPPED_BY_CIPHER.put("DESede", tripleDESAlgorithms)
+
+logger.info("Mapped algorithms: ${ALGORITHMS_MAPPED_BY_CIPHER}")
+}
+
+@Before
+void setUp() throws Exception {
+
+}
+
+@After
+void tearDown() throws Exception {
+
+}
+
+@Test
+void testShouldParseCipherFromAlgorithm() {
+// Arrange
+final def EXPECTED_ALGORITHMS = ALGORITHMS_MAPPED_BY_CIPHER
+
+// Act
+SYMMETRIC_ALGORITHMS.each { String algorithm ->
+String cipher = 
CipherUtili

[GitHub] nifi pull request: NIFI-1257 and 1259

2016-02-04 Thread alopresto

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/201#discussion_r51925068
  
--- Diff: 
nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/EncryptionMethod.java
 ---
@@ -26,37 +26,43 @@
  */
 public enum EncryptionMethod {
 
-MD5_128AES("PBEWITHMD5AND128BITAES-CBC-OPENSSL", "BC", false),
-MD5_192AES("PBEWITHMD5AND192BITAES-CBC-OPENSSL", "BC", false),
-MD5_256AES("PBEWITHMD5AND256BITAES-CBC-OPENSSL", "BC", false),
-MD5_DES("PBEWITHMD5ANDDES", "BC", false),
-MD5_RC2("PBEWITHMD5ANDRC2", "BC", false),
-SHA1_RC2("PBEWITHSHA1ANDRC2", "BC", false),
-SHA1_DES("PBEWITHSHA1ANDDES", "BC", false),
-SHA_128AES("PBEWITHSHAAND128BITAES-CBC-BC", "BC", true),
-SHA_192AES("PBEWITHSHAAND192BITAES-CBC-BC", "BC", true),
-SHA_256AES("PBEWITHSHAAND256BITAES-CBC-BC", "BC", true),
-SHA_40RC2("PBEWITHSHAAND40BITRC2-CBC", "BC", true),
-SHA_128RC2("PBEWITHSHAAND128BITRC2-CBC", "BC", true),
-SHA_40RC4("PBEWITHSHAAND40BITRC4", "BC", true),
-SHA_128RC4("PBEWITHSHAAND128BITRC4", "BC", true),
-SHA256_128AES("PBEWITHSHA256AND128BITAES-CBC-BC", "BC", true),
-SHA256_192AES("PBEWITHSHA256AND192BITAES-CBC-BC", "BC", true),
-SHA256_256AES("PBEWITHSHA256AND256BITAES-CBC-BC", "BC", true),
-SHA_2KEYTRIPLEDES("PBEWITHSHAAND2-KEYTRIPLEDES-CBC", "BC", true),
-SHA_3KEYTRIPLEDES("PBEWITHSHAAND3-KEYTRIPLEDES-CBC", "BC", true),
-SHA_TWOFISH("PBEWITHSHAANDTWOFISH-CBC", "BC", true),
-PGP("PGP", "BC", false),
-PGP_ASCII_ARMOR("PGP-ASCII-ARMOR", "BC", false);
+MD5_128AES("PBEWITHMD5AND128BITAES-CBC-OPENSSL", "BC", false, false),
+MD5_192AES("PBEWITHMD5AND192BITAES-CBC-OPENSSL", "BC", true, false),
+MD5_256AES("PBEWITHMD5AND256BITAES-CBC-OPENSSL", "BC", true, false),
+MD5_DES("PBEWITHMD5ANDDES", "BC", false, false),
+MD5_RC2("PBEWITHMD5ANDRC2", "BC", false, false),
+SHA1_RC2("PBEWITHSHA1ANDRC2", "BC", false, false),
+SHA1_DES("PBEWITHSHA1ANDDES", "BC", false, false),
+SHA_128AES("PBEWITHSHAAND128BITAES-CBC-BC", "BC", false, false),
+SHA_192AES("PBEWITHSHAAND192BITAES-CBC-BC", "BC", true, false),
+SHA_256AES("PBEWITHSHAAND256BITAES-CBC-BC", "BC", true, false),
+SHA_40RC2("PBEWITHSHAAND40BITRC2-CBC", "BC", false, false),
+SHA_128RC2("PBEWITHSHAAND128BITRC2-CBC", "BC", false, false),
+SHA_40RC4("PBEWITHSHAAND40BITRC4", "BC", false, false),
+SHA_128RC4("PBEWITHSHAAND128BITRC4", "BC", false, false),
+SHA256_128AES("PBEWITHSHA256AND128BITAES-CBC-BC", "BC", false, false),
+SHA256_192AES("PBEWITHSHA256AND192BITAES-CBC-BC", "BC", true, false),
+SHA256_256AES("PBEWITHSHA256AND256BITAES-CBC-BC", "BC", true, false),
+SHA_2KEYTRIPLEDES("PBEWITHSHAAND2-KEYTRIPLEDES-CBC", "BC", false, 
false),
+SHA_3KEYTRIPLEDES("PBEWITHSHAAND3-KEYTRIPLEDES-CBC", "BC", false, 
false),
+SHA_TWOFISH("PBEWITHSHAANDTWOFISH-CBC", "BC", false, false),
+PGP("PGP", "BC", false, false),
+PGP_ASCII_ARMOR("PGP-ASCII-ARMOR", "BC", false, false),
+// New encryption methods which used keyed encryption
+AES_CBC("AES/CBC/PKCS7Padding", "BC", false, true),
+AES_CTR("AES/CTR/NoPadding", "BC", false, true),
+AES_GCM("AES/GCM/NoPadding", "BC", false, true);
 
 private final String algorithm;
 private final String provider;
 private final boolean unlimitedStrength;
+private final boolean compatibleWithStrongKDFs;
 
-EncryptionMethod(String algorithm, String provider, boolean 
unlimitedStrength) {
+EncryptionMethod(String algorithm, String provider, boolean 
unlimitedStrength, boolean compatibleWithStrongKDFs) {
--- End diff --

Because this is the enum constructor, it's never used anywhere else. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Smart Load Balancing behavior replicating data to different cluster nodes.

2016-02-04 Thread Paresh Shah

The problem is not quite what we thought.

What we see is that we generate a bunch of flowFiles on the sender
processor and then commit. But all these get delivered to the same remote
node. What we are expecting is they should be split depending on the no of
nodes and the loads on each node.

Paresh

On 2/4/16, 10:53 AM, "Paresh Shah"  wrote:

>We have the pipeline as follows.
>
>Sender Side
>  P1..->Pn -> RPG ( connected to ForwardInputPort )
>
>Receiver Side
>ForwardInputPort -> ProcessGroup
>
>Inside the processGroup
>   InputPort-> P1 Š-> Pn
>
>We see that we sent 10 flow files to the remote cluster. There we see the
>same set of flow files being sent to different nodes of the cluster. We
>are confirming this by looking at the DataProvnance and the fileNames in
>there. They are the same ones for two nodes of the cluster.
>
>This behaviour seems to incorrect.
>
>Paresh
>
>
>The information contained in this transmission may contain privileged and
>confidential information. It is intended only for the use of the
>person(s) named above. If you are not the intended recipient, you are
>hereby notified that any review, dissemination, distribution or
>duplication of this communication is strictly prohibited. If you are not
>the intended recipient, please contact the sender by reply email and
>destroy all copies of the original message.
>

 The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

[GitHub] nifi pull request: NIFI-1257 and 1259

2016-02-04 Thread mattyb149

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/201#discussion_r51919714
  
--- Diff: 
nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/EncryptionMethod.java
 ---
@@ -26,37 +26,43 @@
  */
 public enum EncryptionMethod {
 
-MD5_128AES("PBEWITHMD5AND128BITAES-CBC-OPENSSL", "BC", false),
-MD5_192AES("PBEWITHMD5AND192BITAES-CBC-OPENSSL", "BC", false),
-MD5_256AES("PBEWITHMD5AND256BITAES-CBC-OPENSSL", "BC", false),
-MD5_DES("PBEWITHMD5ANDDES", "BC", false),
-MD5_RC2("PBEWITHMD5ANDRC2", "BC", false),
-SHA1_RC2("PBEWITHSHA1ANDRC2", "BC", false),
-SHA1_DES("PBEWITHSHA1ANDDES", "BC", false),
-SHA_128AES("PBEWITHSHAAND128BITAES-CBC-BC", "BC", true),
-SHA_192AES("PBEWITHSHAAND192BITAES-CBC-BC", "BC", true),
-SHA_256AES("PBEWITHSHAAND256BITAES-CBC-BC", "BC", true),
-SHA_40RC2("PBEWITHSHAAND40BITRC2-CBC", "BC", true),
-SHA_128RC2("PBEWITHSHAAND128BITRC2-CBC", "BC", true),
-SHA_40RC4("PBEWITHSHAAND40BITRC4", "BC", true),
-SHA_128RC4("PBEWITHSHAAND128BITRC4", "BC", true),
-SHA256_128AES("PBEWITHSHA256AND128BITAES-CBC-BC", "BC", true),
-SHA256_192AES("PBEWITHSHA256AND192BITAES-CBC-BC", "BC", true),
-SHA256_256AES("PBEWITHSHA256AND256BITAES-CBC-BC", "BC", true),
-SHA_2KEYTRIPLEDES("PBEWITHSHAAND2-KEYTRIPLEDES-CBC", "BC", true),
-SHA_3KEYTRIPLEDES("PBEWITHSHAAND3-KEYTRIPLEDES-CBC", "BC", true),
-SHA_TWOFISH("PBEWITHSHAANDTWOFISH-CBC", "BC", true),
-PGP("PGP", "BC", false),
-PGP_ASCII_ARMOR("PGP-ASCII-ARMOR", "BC", false);
+MD5_128AES("PBEWITHMD5AND128BITAES-CBC-OPENSSL", "BC", false, false),
+MD5_192AES("PBEWITHMD5AND192BITAES-CBC-OPENSSL", "BC", true, false),
+MD5_256AES("PBEWITHMD5AND256BITAES-CBC-OPENSSL", "BC", true, false),
+MD5_DES("PBEWITHMD5ANDDES", "BC", false, false),
+MD5_RC2("PBEWITHMD5ANDRC2", "BC", false, false),
+SHA1_RC2("PBEWITHSHA1ANDRC2", "BC", false, false),
+SHA1_DES("PBEWITHSHA1ANDDES", "BC", false, false),
+SHA_128AES("PBEWITHSHAAND128BITAES-CBC-BC", "BC", false, false),
+SHA_192AES("PBEWITHSHAAND192BITAES-CBC-BC", "BC", true, false),
+SHA_256AES("PBEWITHSHAAND256BITAES-CBC-BC", "BC", true, false),
+SHA_40RC2("PBEWITHSHAAND40BITRC2-CBC", "BC", false, false),
+SHA_128RC2("PBEWITHSHAAND128BITRC2-CBC", "BC", false, false),
+SHA_40RC4("PBEWITHSHAAND40BITRC4", "BC", false, false),
+SHA_128RC4("PBEWITHSHAAND128BITRC4", "BC", false, false),
+SHA256_128AES("PBEWITHSHA256AND128BITAES-CBC-BC", "BC", false, false),
+SHA256_192AES("PBEWITHSHA256AND192BITAES-CBC-BC", "BC", true, false),
+SHA256_256AES("PBEWITHSHA256AND256BITAES-CBC-BC", "BC", true, false),
+SHA_2KEYTRIPLEDES("PBEWITHSHAAND2-KEYTRIPLEDES-CBC", "BC", false, 
false),
+SHA_3KEYTRIPLEDES("PBEWITHSHAAND3-KEYTRIPLEDES-CBC", "BC", false, 
false),
+SHA_TWOFISH("PBEWITHSHAANDTWOFISH-CBC", "BC", false, false),
+PGP("PGP", "BC", false, false),
+PGP_ASCII_ARMOR("PGP-ASCII-ARMOR", "BC", false, false),
+// New encryption methods which used keyed encryption
+AES_CBC("AES/CBC/PKCS7Padding", "BC", false, true),
+AES_CTR("AES/CTR/NoPadding", "BC", false, true),
+AES_GCM("AES/GCM/NoPadding", "BC", false, true);
 
 private final String algorithm;
 private final String provider;
 private final boolean unlimitedStrength;
+private final boolean compatibleWithStrongKDFs;
 
-EncryptionMethod(String algorithm, String provider, boolean 
unlimitedStrength) {
+EncryptionMethod(String algorithm, String provider, boolean 
unlimitedStrength, boolean compatibleWithStrongKDFs) {
--- End diff --

Can this be a straight override or do we need to keep the original method 
and deprecate it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Smart Load Balancing behavior replicating data to different cluster nodes.

2016-02-04 Thread Paresh Shah

We have the pipeline as follows.

Sender Side
  P1..->Pn -> RPG ( connected to ForwardInputPort )

Receiver Side
ForwardInputPort -> ProcessGroup

Inside the processGroup
   InputPort-> P1 …-> Pn

We see that we sent 10 flow files to the remote cluster. There we see the same 
set of flow files being sent to different nodes of the cluster. We are 
confirming this by looking at the DataProvnance and the fileNames in there. 
They are the same ones for two nodes of the cluster.

This behaviour seems to incorrect.

Paresh


The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

[GitHub] nifi pull request: NIFI-1257 and 1259

2016-02-04 Thread alopresto

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/201#discussion_r51914845
  
--- Diff: 
nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/KeyDerivationFunction.java
 ---
@@ -25,8 +25,11 @@
 public enum KeyDerivationFunction {
 
 NIFI_LEGACY("NiFi legacy KDF", "MD5 @ 1000 iterations"),
-OPENSSL_EVP_BYTES_TO_KEY("OpenSSL EVP_BytesToKey", "Single iteration 
MD5 compatible with PKCS#5 v1.5");
-// TODO: Implement bcrypt, scrypt, and PBKDF2
+OPENSSL_EVP_BYTES_TO_KEY("OpenSSL EVP_BytesToKey", "Single iteration 
MD5 compatible with PKCS#5 v1.5"),
+BCRYPT("Bcrypt", "Bcrypt with configurable work factor: see 
https://cwiki.apache.org/confluence/display/NIFI/Key+Derivation+Function+Explanations";),
--- End diff --

Working on that now. Converting various markup languages. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1257 and 1259

2016-02-04 Thread apiri

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi/pull/201#discussion_r51914669
  
--- Diff: 
nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/KeyDerivationFunction.java
 ---
@@ -25,8 +25,11 @@
 public enum KeyDerivationFunction {
 
 NIFI_LEGACY("NiFi legacy KDF", "MD5 @ 1000 iterations"),
-OPENSSL_EVP_BYTES_TO_KEY("OpenSSL EVP_BytesToKey", "Single iteration 
MD5 compatible with PKCS#5 v1.5");
-// TODO: Implement bcrypt, scrypt, and PBKDF2
+OPENSSL_EVP_BYTES_TO_KEY("OpenSSL EVP_BytesToKey", "Single iteration 
MD5 compatible with PKCS#5 v1.5"),
+BCRYPT("Bcrypt", "Bcrypt with configurable work factor: see 
https://cwiki.apache.org/confluence/display/NIFI/Key+Derivation+Function+Explanations";),
--- End diff --

The resources listed on the wiki are super helpful. I think it would make 
sense for these to additionally get rolled into the administrator's guide so we 
can have a one stop shop for all of it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Lost Trace of FowFiles only with attributes

2016-02-04 Thread Paresh Shah

In the cluster mode we have seeing the following two issues.

  1.  In one scenario we see that the flowFile( only containing attributes ) is 
sent over to the remote cluster and after the first processor following the 
InputPort in the pipeline the data provnance shows no trace of that flow file. 
Where can we locate this flow file in the repository?

2. In another scenario we see that the data provenance on the receiving cluster 
is not showing any updates since 4/5 hours even though the sending cluster 
indicates  that it has successfully sent data to it. So we are not sure where 
the data is dropped.

We have checked that the clocks on all the cluster nodes are in sync.

Paresh



The information contained in this transmission may contain privileged and 
confidential information. It is intended only for the use of the person(s) 
named above. If you are not the intended recipient, you are hereby notified 
that any review, dissemination, distribution or duplication of this 
communication is strictly prohibited. If you are not the intended recipient, 
please contact the sender by reply email and destroy all copies of the original 
message.

[GitHub] nifi pull request: NIFI-1257 and 1259

2016-02-04 Thread alopresto

Github user alopresto commented on the pull request:

https://github.com/apache/nifi/pull/201#issuecomment-179947088
  
Worked with Aldrin and determined that the `brew` method of 
installing/uninstalling JCE USJ files is not correct, as "uninstalling" removes 
the unlimited jurisdiction policy files but does not replace them with the 
originals. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1257 and 1259

2016-02-04 Thread apiri

Github user apiri commented on the pull request:

https://github.com/apache/nifi/pull/201#issuecomment-179914695
  
@alopresto I'm running into issues with executing the built assembly 
without the JCE USJ files installed:

> 2016-02-04 10:52:14,023 ERROR [main] org.apache.nifi.NiFi Failure to 
launch NiFi due to java.util.ServiceConfigurationError: 
org.apache.nifi.controller.ControllerService: Provider 
org.apache.nifi.ssl.StandardSSLContextService could not be instantiated
java.util.ServiceConfigurationError: 
org.apache.nifi.controller.ControllerService: Provider 
org.apache.nifi.ssl.StandardSSLContextService could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232) ~[na:1.8.0_60]
at java.util.ServiceLoader.access$100(ServiceLoader.java:185) 
~[na:1.8.0_60]
at 
java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) 
~[na:1.8.0_60]
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) 
~[na:1.8.0_60]
at java.util.ServiceLoader$1.next(ServiceLoader.java:480) ~[na:1.8.0_60]
at 
org.apache.nifi.nar.ExtensionManager.loadExtensions(ExtensionManager.java:107) 
~[nifi-nar-utils-0.4.2-SNAPSHOT.jar:0.4.2-SNAPSHOT]
at 
org.apache.nifi.nar.ExtensionManager.discoverExtensions(ExtensionManager.java:88)
 ~[nifi-nar-utils-0.4.2-SNAPSHOT.jar:0.4.2-SNAPSHOT]
at org.apache.nifi.NiFi.(NiFi.java:120) 
~[nifi-runtime-0.4.2-SNAPSHOT.jar:0.4.2-SNAPSHOT]
at org.apache.nifi.NiFi.main(NiFi.java:227) 
~[nifi-runtime-0.4.2-SNAPSHOT.jar:0.4.2-SNAPSHOT]
Caused by: java.lang.ExceptionInInitializerError: null
at javax.crypto.JceSecurityManager.(JceSecurityManager.java:65) 
~[na:1.8.0_60]
at javax.crypto.Cipher.getConfiguredPermission(Cipher.java:2587) 
~[na:1.8.0_60]
at javax.crypto.Cipher.getMaxAllowedKeyLength(Cipher.java:2611) 
~[na:1.8.0_60]
at 
sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:548) 
~[na:1.8.0_60]
at 
sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:527) 
~[na:1.8.0_60]
at sun.security.ssl.CipherSuite.isAvailable(CipherSuite.java:194) 
~[na:1.8.0_60]
at 
sun.security.ssl.SSLContextImpl.getApplicableCipherSuiteList(SSLContextImpl.java:346)
 ~[na:1.8.0_60]
at 
sun.security.ssl.SSLContextImpl.getDefaultCipherSuiteList(SSLContextImpl.java:297)
 ~[na:1.8.0_60]
at sun.security.ssl.SSLEngineImpl.init(SSLEngineImpl.java:402) 
~[na:1.8.0_60]
at sun.security.ssl.SSLEngineImpl.(SSLEngineImpl.java:349) 
~[na:1.8.0_60]
at 
sun.security.ssl.SSLContextImpl.engineCreateSSLEngine(SSLContextImpl.java:201) 
~[na:1.8.0_60]
at javax.net.ssl.SSLContext.createSSLEngine(SSLContext.java:329) 
~[na:1.8.0_60]
at 
org.apache.nifi.ssl.StandardSSLContextService.buildAlgorithmAllowableValues(StandardSSLContextService.java:424)
 ~[nifi-ssl-context-service-0.4.2-SNAPSHOT.jar:0.4.2-SNAPSHOT]
at 
org.apache.nifi.ssl.StandardSSLContextService.(StandardSSLContextService.java:103)
 ~[nifi-ssl-context-service-0.4.2-SNAPSHOT.jar:0.4.2-SNAPSHOT]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
Method) ~[na:1.8.0_60]
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 ~[na:1.8.0_60]
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 ~[na:1.8.0_60]
at java.lang.reflect.Constructor.newInstance(Constructor.java:422) 
~[na:1.8.0_60]
at java.lang.Class.newInstance(Class.java:442) ~[na:1.8.0_60]
at 
java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) 
~[na:1.8.0_60]
... 6 common frames omitted
Caused by: java.lang.SecurityException: Can not initialize cryptographic 
mechanism
at javax.crypto.JceSecurity.(JceSecurity.java:89) ~[na:1.8.0_60]
... 26 common frames omitted
Caused by: java.lang.SecurityException: Cannot locate policy or framework 
files!
at 
javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:256) 
~[na:1.8.0_60]
at javax.crypto.JceSecurity.access$000(JceSecurity.java:48) 
~[na:1.8.0_60]
at javax.crypto.JceSecurity$1.run(JceSecurity.java:81) ~[na:1.8.0_60]
at java.security.AccessController.doPrivileged(Native Method) 
~[na:1.8.0_60]
at javax.crypto.JceSecurity.(JceSecurity.java:78) ~[na:1.8.0_60]
... 26 common frames omitted

With them installed, NiFi starts and works as anticipated.  Will dig in a 
bit more to see what changed to cause the issue within
at 
org.apache.nifi.ssl.StandardSSLContextService.buildAlgorithmAllowableValues(StandardSSLContextService.java:424)
 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the fe

[GitHub] nifi pull request: Fix NiFi-1461

2016-02-04 Thread PuspenduBanerjee

GitHub user PuspenduBanerjee opened a pull request:

https://github.com/apache/nifi/pull/204

Fix NiFi-1461

fix for  [NiFi-1461](https://issues.apache.org/jira/browse/NIFI-1461)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/PuspenduBanerjee/nifi nifi-head

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/204.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #204


commit dc120bfdd8578b2f47e6f0f591be78b07762caec
Author: puspendu.baner...@gmail.com 
Date:   2016-02-04T08:40:26Z

Fix NiFi-1461




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: planning for apache nifi 0.5.0

2016-02-04 Thread Joe Witt

Tony

Agreed.  I will work through these as well and try to ensure the
wording accurately reflect what happened.  The release notes are a
really important piece of communication we need to get right.

Thanks
Joe

On Thu, Feb 4, 2016 at 6:04 AM, Tony Kurc  wrote:
> I am prepping for the release, going through the jiras that have been
> closed, I think several have a description that does not match. Good
> example is:
>
> https://issues.apache.org/jira/browse/NIFI-1325
>
> I believe the final scope of the patch surpassed the original description.
> Should this be adjusted for people perusing the JIRA report we link to in
> our release notes? I'd say yes. I made a couple small edits to some tickets
> already for increased readability. One change I wanted to make was change
> NIFI-259 to "New Feature" rather than subtask, but I was concerned about
> this disrupting folks.
>
> I think a quick consistency and spell check would be good - I've had
> several people mention out of band that prior release notes had a couple
> errors. Apparently people love the release notes page on the JIRA
> NIFI-1107 is done, just needs a merge, which I'll do tonight.
>
> On Wed, Feb 3, 2016 at 4:32 PM, Joe Witt  wrote:
>
>> Latest update for all
>>
>> You can see the current status here [1].
>>
>> Looks like several tickets but really it is three efforts:
>> 1) Improve encryption options in EncryptContent NIFI-1257,1259
>> 2) Provide state management as a framework function NIFI-259,1223,1339
>> 3) Support multi-part S3 uploads NIFI-1107
>>
>> Based on comments on each they are just in very late stage/detailed
>> review.  So we're close.
>>
>> Thanks
>> Joe
>>
>> [1]
>> https://issues.apache.org/jira/browse/NIFI-1339?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%200.5.0%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
>>
>> On Thu, Jan 21, 2016 at 12:55 AM, Joe Witt  wrote:
>> > Team
>> >
>> > Did a full round of 050 JIRA cleanup this evening.  Great progress is
>> > being made.
>> >
>> > For the proposed NiFi 050 release schedule above we have about 10 days
>> > of code time left.  There is a pretty large backlog of patch awaiting
>> > items currently showing as aligned to 050.  See here [1].
>> >
>> > Also there are 25 (as of this writing) open tickets.  See here [2].
>> >
>> > For the proposed goals of the release as listed previous in this
>> > thread it appears the following are likely to occur in 050:
>> >   Improved encryption handling: NIFI-1324,1314,1259,
>> >   Improved Hive/Kite interaction: NIFI-1193
>> >   Interactive Queue Management: NIFI-108
>> >   Support for numerous scripting/languages: NIFI-210
>> >   Support for state management: NIFI-259
>> >
>> > The following are at risk or moving out:
>> >   NiFi kerberos support: NIFI-1274
>> > No progress on it yet.  Good 060 target.
>> >   UX/UI improvements: NIFI-951
>> > These changes appear to be better fit for 1.0.0.
>> >
>> > [1]
>> https://issues.apache.org/jira/browse/NIFI-1416?jql=project%20%3D%20NIFI%20AND%20status%20%3D%20%22Patch%20Available%22%20AND%20fixVersion%20%3D%200.5.0
>> >
>> > [2]
>> https://issues.apache.org/jira/browse/NIFI-1404?jql=project%20%3D%20NIFI%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%200.5.0
>> >
>> > Thanks
>> > Joe
>> >
>> > On Tue, Dec 29, 2015 at 10:08 PM, Joe Witt  wrote:
>> >> Thanks Tony - let's plan on that then.
>> >>
>> >> On Tue, Dec 29, 2015 at 9:58 PM, Oleg Zhurakousky
>> >>  wrote:
>> >>> Be careful what you wish for ;)
>> >>>
>> >>> Sent from my iPhone
>> >>>
>>  On Dec 29, 2015, at 17:44, Tony Kurc  wrote:
>> 
>>  Joe,
>>  I might like to give the release a try.
>> 
>>  Tony
>> 
>> > On Mon, Dec 28, 2015 at 4:30 PM, Joe Witt 
>> wrote:
>> >
>> > Team,
>> >
>> > In the spirit of shooting for roughly 6 week release cycles and the
>> > releasing timing of NiFi 0.4.0/0.4.1 I suggest we shoot for Feb 5th,
>> > 2016 as our release vote start date target.  This should mean we're
>> > not altering code past say 1 Feb, 2016 other than for findings that
>> > crop up with the testing/RC.
>> >
>> > The JIRA entries for Apache NiFi 0.5.0 can be quickly found here
>> >
>> >
>> https://issues.apache.org/jira/browse/NIFI-1333?jql=fixVersion%20%3D%200.5.0%20AND%20project%20%3D%20NIFI
>> >
>> > There are some really great features/fixes there already including
>> but
>> > not limited to:
>> >  NiFi kerberos support: NIFI-1274
>> >  Improved encryption handling: NIFI-1324,1314,1259,
>> >  Improved Hive/Kite interaction: NIFI-1193
>> >  UX/UI improvements: NIFI-951
>> >  Interactive Queue Management: NIFI-108
>> >  Support for numerous scripting/languages: NIFI-210
>> >  Support for state management: NIFI-259
>> >
>> > If anyone on the pmc is interested to do the RC work for the release
>> > just

Re: ConvertCSVtoAvro | support for "||" delimiter

2016-02-04 Thread Joe Percivall

A more direct work-around would be to use the ReplaceText processor first in 
order to change instances of "||" to "|" so that it can be used by 
ConvertCSVtoAvro.

Joe- - - - - - 
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com

On Thursday, February 4, 2016 9:50 AM, Joe Witt  wrote:
Not a direct answer but:
  With NIFI-210 arriving in the upcoming NiFi 0.5.0 release you will
have a great option in scripting (Lua, Python, Ruby, Groovy,
Javascript) that will let you rapidly get past these hurdles without
having to build your own custom processor until you are sure what you
need.

Thanks
Joe

On Thu, Feb 4, 2016 at 6:48 AM, Tony Kurc  wrote:
> With that processor alone it doesn't appear so. The validator for that
> property requires it to be one character.
> On Feb 3, 2016 1:01 AM, "shweta"  wrote:
>
>> Hi All,
>>
>> It seems "ConvertCSVtoAvro" only support single character as delimiter in
>> Nifi. Is there a way to specify "||"
>> delimiter.
>>
>> Thanks,
>> Shweta
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-nifi-developer-list.39713.n7.nabble.com/ConvertCSVtoAvro-support-for-delimiter-tp7116.html
>> Sent from the Apache NiFi Developer List mailing list archive at
>> Nabble.com.
>>

Re: ConvertCSVtoAvro | support for "||" delimiter

2016-02-04 Thread Joe Witt

Not a direct answer but:
  With NIFI-210 arriving in the upcoming NiFi 0.5.0 release you will
have a great option in scripting (Lua, Python, Ruby, Groovy,
Javascript) that will let you rapidly get past these hurdles without
having to build your own custom processor until you are sure what you
need.

Thanks
Joe

On Thu, Feb 4, 2016 at 6:48 AM, Tony Kurc  wrote:
> With that processor alone it doesn't appear so. The validator for that
> property requires it to be one character.
> On Feb 3, 2016 1:01 AM, "shweta"  wrote:
>
>> Hi All,
>>
>> It seems "ConvertCSVtoAvro" only support single character as delimiter in
>> Nifi. Is there a way to specify "||"
>> delimiter.
>>
>> Thanks,
>> Shweta
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-nifi-developer-list.39713.n7.nabble.com/ConvertCSVtoAvro-support-for-delimiter-tp7116.html
>> Sent from the Apache NiFi Developer List mailing list archive at
>> Nabble.com.
>>

Re: ConvertCSVtoAvro | support for "||" delimiter

2016-02-04 Thread Tony Kurc

With that processor alone it doesn't appear so. The validator for that
property requires it to be one character.
On Feb 3, 2016 1:01 AM, "shweta"  wrote:

> Hi All,
>
> It seems "ConvertCSVtoAvro" only support single character as delimiter in
> Nifi. Is there a way to specify "||"
> delimiter.
>
> Thanks,
> Shweta
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/ConvertCSVtoAvro-support-for-delimiter-tp7116.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: planning for apache nifi 0.5.0

2016-02-04 Thread Tony Kurc

I am prepping for the release, going through the jiras that have been
closed, I think several have a description that does not match. Good
example is:

https://issues.apache.org/jira/browse/NIFI-1325

I believe the final scope of the patch surpassed the original description.
Should this be adjusted for people perusing the JIRA report we link to in
our release notes? I'd say yes. I made a couple small edits to some tickets
already for increased readability. One change I wanted to make was change
NIFI-259 to "New Feature" rather than subtask, but I was concerned about
this disrupting folks.

I think a quick consistency and spell check would be good - I've had
several people mention out of band that prior release notes had a couple
errors. Apparently people love the release notes page on the JIRA
NIFI-1107 is done, just needs a merge, which I'll do tonight.

On Wed, Feb 3, 2016 at 4:32 PM, Joe Witt  wrote:

> Latest update for all
>
> You can see the current status here [1].
>
> Looks like several tickets but really it is three efforts:
> 1) Improve encryption options in EncryptContent NIFI-1257,1259
> 2) Provide state management as a framework function NIFI-259,1223,1339
> 3) Support multi-part S3 uploads NIFI-1107
>
> Based on comments on each they are just in very late stage/detailed
> review.  So we're close.
>
> Thanks
> Joe
>
> [1]
> https://issues.apache.org/jira/browse/NIFI-1339?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%200.5.0%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
>
> On Thu, Jan 21, 2016 at 12:55 AM, Joe Witt  wrote:
> > Team
> >
> > Did a full round of 050 JIRA cleanup this evening.  Great progress is
> > being made.
> >
> > For the proposed NiFi 050 release schedule above we have about 10 days
> > of code time left.  There is a pretty large backlog of patch awaiting
> > items currently showing as aligned to 050.  See here [1].
> >
> > Also there are 25 (as of this writing) open tickets.  See here [2].
> >
> > For the proposed goals of the release as listed previous in this
> > thread it appears the following are likely to occur in 050:
> >   Improved encryption handling: NIFI-1324,1314,1259,
> >   Improved Hive/Kite interaction: NIFI-1193
> >   Interactive Queue Management: NIFI-108
> >   Support for numerous scripting/languages: NIFI-210
> >   Support for state management: NIFI-259
> >
> > The following are at risk or moving out:
> >   NiFi kerberos support: NIFI-1274
> > No progress on it yet.  Good 060 target.
> >   UX/UI improvements: NIFI-951
> > These changes appear to be better fit for 1.0.0.
> >
> > [1]
> https://issues.apache.org/jira/browse/NIFI-1416?jql=project%20%3D%20NIFI%20AND%20status%20%3D%20%22Patch%20Available%22%20AND%20fixVersion%20%3D%200.5.0
> >
> > [2]
> https://issues.apache.org/jira/browse/NIFI-1404?jql=project%20%3D%20NIFI%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%200.5.0
> >
> > Thanks
> > Joe
> >
> > On Tue, Dec 29, 2015 at 10:08 PM, Joe Witt  wrote:
> >> Thanks Tony - let's plan on that then.
> >>
> >> On Tue, Dec 29, 2015 at 9:58 PM, Oleg Zhurakousky
> >>  wrote:
> >>> Be careful what you wish for ;)
> >>>
> >>> Sent from my iPhone
> >>>
>  On Dec 29, 2015, at 17:44, Tony Kurc  wrote:
> 
>  Joe,
>  I might like to give the release a try.
> 
>  Tony
> 
> > On Mon, Dec 28, 2015 at 4:30 PM, Joe Witt 
> wrote:
> >
> > Team,
> >
> > In the spirit of shooting for roughly 6 week release cycles and the
> > releasing timing of NiFi 0.4.0/0.4.1 I suggest we shoot for Feb 5th,
> > 2016 as our release vote start date target.  This should mean we're
> > not altering code past say 1 Feb, 2016 other than for findings that
> > crop up with the testing/RC.
> >
> > The JIRA entries for Apache NiFi 0.5.0 can be quickly found here
> >
> >
> https://issues.apache.org/jira/browse/NIFI-1333?jql=fixVersion%20%3D%200.5.0%20AND%20project%20%3D%20NIFI
> >
> > There are some really great features/fixes there already including
> but
> > not limited to:
> >  NiFi kerberos support: NIFI-1274
> >  Improved encryption handling: NIFI-1324,1314,1259,
> >  Improved Hive/Kite interaction: NIFI-1193
> >  UX/UI improvements: NIFI-951
> >  Interactive Queue Management: NIFI-108
> >  Support for numerous scripting/languages: NIFI-210
> >  Support for state management: NIFI-259
> >
> > If anyone on the pmc is interested to do the RC work for the release
> > just holler.  If not I'm happy to do it.
> >
> > Thanks
> > Joe
> >
>

[GitHub] nifi pull request: Fix NiFi-1461

Re: planning for apache nifi 0.5.0

Re: ConvertCSVtoAvro | support for "||" delimiter

[GitHub] nifi pull request: NIFI-1257 and 1259

[GitHub] nifi pull request: NIFI-924:Nifi-Camel Integration

Re: ConvertCSVtoAvro | support for "||" delimiter

[GitHub] nifi pull request: NIFI-924:Nifi-Camel Integration

Re: ConvertCSVtoAvro | support for "||" delimiter

Re: ConvertCSVtoAvro | support for "||" delimiter

Re: ConvertCSVtoAvro | support for "||" delimiter

[GitHub] nifi pull request: NIFI-1257 and 1259

[GitHub] nifi pull request: NIFI-1257 and 1259

[GitHub] nifi pull request: NIFI-1257 and 1259

[GitHub] nifi pull request: NIFI-1257 and 1259

[GitHub] nifi pull request: NIFI-1257 and 1259

Re: Smart Load Balancing behavior replicating data to different cluster nodes.

[GitHub] nifi pull request: NIFI-1257 and 1259

Smart Load Balancing behavior replicating data to different cluster nodes.

[GitHub] nifi pull request: NIFI-1257 and 1259

[GitHub] nifi pull request: NIFI-1257 and 1259

Lost Trace of FowFiles only with attributes

[GitHub] nifi pull request: NIFI-1257 and 1259

[GitHub] nifi pull request: NIFI-1257 and 1259

[GitHub] nifi pull request: Fix NiFi-1461

Re: planning for apache nifi 0.5.0

Re: ConvertCSVtoAvro | support for "||" delimiter

Re: ConvertCSVtoAvro | support for "||" delimiter

Re: ConvertCSVtoAvro | support for "||" delimiter

Re: planning for apache nifi 0.5.0

29 matches

Site Navigation

Mail list logo

Footer information