Re: Passwords in EncryptContent

[Andy LoPresto]

Brett,

(I added the dev list back in because this is probably of interest to someone/ 
should be documented. )

This started out as a brief email and spiraled into a couple days of work. Part 
of that is because my Ruby is rusty, but part is because there are some serious 
underlying issues here (previously quickly noted in NIFI-1465 [1] but not fully 
expounded upon at the time). The tl;dr of this email is that I have written a 
Ruby script which will accomplish what you want and it is located here [2]. 
Read on at your own risk (various parts of this email were written over the 
course of the last two days, so it may be repetitive/incoherent where the story 
changed).

This is a confusing case because it is unusual in execution and there are 
multiple layers here. I think I didn’t explain it well last time. I’ll try to 
step through it, but I also apologize in advance, because I found a lot of 
legacy stuff here that was probably written with no intention of ever being 
exposed/integrated with an external source. Kerckhoff would not approve. I will 
add some of this to NIFI-1465.

The mechanism that NiFi uses to encrypt the sensitive property values (i.e. 
passwords for EncryptContent) is as I described previously, but in further 
investigating to help solve your problem, I realized that the Jasypt [3] 
StandardPBEStringEncryptor used in StandardFlowSynchronizer uses a random salt 
generator internally. You can verify this by making a new flow with two 
EncryptContent processors — even if you set the same password for each, the 
resulting cipher texts in the flow.xml will be unique because despite the same 
master key being used, the random salt will cause them to be different.

Now this is actually good news, because it means the salt must be encoded and 
transmitted with the cipher text. If it was not, NiFi would not be able to 
decrypt these values unless it used a fixed salt, and clearly it does not. So 
as long as your Ruby code generates a salt of the correct length and embeds it 
in the cipher text, it will be compatible with NiFi. The salt is the first 16 
bytes (32 hex characters) and the actual cipher text representing the encrypted 
processor sensitive property (happens to be another password, but this is 
irrelevant) is the second 16 bytes.

Note: because your initial plaintext (the password you are trying to encrypt) 
is only 11 UTF-8 characters, it can be represented by 11 bytes. This means that 
when encrypted using AES-CBC (16 byte block size), it requires only and exactly 
one block (11 bytes of plaintext plus 5 bytes padding). The resulting cipher 
text is 16 bytes. If the processor password was longer than 16 characters, it 
would be encrypted in two blocks and encoded as 32 bytes, or 64 hex characters 
alone (remember to add the initial 32 chars for the salt for a total of 96 
chars).

What we are looking for as the output of the Ruby operation is, as you noted, a 
64 character hex string, of the format:

output = hex_encode(salt || encrypt(processor_password, master_key, iv))

where the master_key and iv are derived by

(master_key, iv) = md5(master_passphrase || master_salt) || 
md5(md5(master_passphrase || master_salt) || master_passphrase || master_salt) 
|| md5(md5(md5(…)…)…)

This is an unusual method and is described thusly on the OpenSSL EVP_BytesToKey 
documentation [4]:

> If the total key and IV length is less than the digest length and MD5 is used 
> then the derivation algorithm is compatible with PKCS#5 v1.5 otherwise a non 
> standard extension is used to derive the extra data.

...

> KEY DERIVATION ALGORITHM
> 
> The key and IV is derived by concatenating D_1, D_2, etc until enough data is 
> available for the key and IV. D_i is defined as:
> 
> D_i = HASH^count(D_(i-1) || data || salt)
> where || denotes concatenation, D_0 is empty, HASH is the digest algorithm in 
> use, HASH^1(data) is simply HASH(data), HASH^2(data) is HASH(HASH(data)) and 
> so on.
> 
> The initial bytes are used for the key and the subsequent bytes for the IV.
> 

The reason there are multiple MD5 operations above is because we have specified 
the encryption will use AES-256-CBC, which requires a 256 bit (32 byte) key — 
32 bytes are represented by 64 hex characters. A single iteration of MD5 only 
yields 16 bytes (32 hex chars), so we must concatenate it with another 
invocation. However, as it is deterministic, running it on the same input would 
return the same output, and the key would just repeat the same 16 bytes. To 
counter this, the second ( up to n many) invocation “salts” the input with the 
result of the previous step. If we substitute some variables for the full 
expression above, we can see this more clearly:

let x = “master_passphrase || master_salt”

master_key = md5(master_passphrase || master_salt) || md5(md5(master_passphrase 
|| master_salt) || master_passphrase || master_salt)

master_key = md5(x) || md5(md5(x) || x)

let y = “md5(x)”

master_key = y || md5(y || x)

Re: PutS3Object error

[Joe Skora]

Were you able to resolve this?

I think James is right about the Bucket vs Region relationship, AWS
Signature Version 4 has less tolerance for bucket/region mismatches.  (See this
comment

in the AmazonS3Client source for more detail.)

If that's not it, AWS has a page on signature troubleshooting

.

Regards,
Joe

On Wed, Jul 20, 2016 at 6:15 PM, James Wing  wrote:

> ListS3 processor is a great place to start.  I can think of a couple of
> likely causes:
>
> * Credentials - you might want to double-check how you specified the
> credentials to make sure it is right.  If you specify access key and secret
> key, and the secret key is not correct, I believe you will get this error.
>
> * Region - you should make sure the region you specify in the processors
> matches the S3 bucket's region.  S3 is a bit weird about the rules for
> buckets and regions, but it might help to check.
>
> Thanks,
>
> James
>
> On Wed, Jul 20, 2016 at 12:57 PM, Nabegh  wrote:
>
> > Hi James,
> >
> > I'm using nifi-0.7.0-RC2. I tried ListS3 processor and I received the
> same
> > error.
> >
> > Thanks
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-nifi-developer-list.39713.n7.nabble.com/PutS3Object-error-tp12878p12880.html
> > Sent from the Apache NiFi Developer List mailing list archive at
> > Nabble.com.
> >
>

Re: StandardSSLContextService error

[Jeff]

Hello Nabegh,

Can you supply the logs that include the error you referenced in your
previous email?  Can you also include a directory listing that includes the
file permissions, please?

On Mon, Jul 25, 2016 at 12:31 PM Nabegh  wrote:

> Hi Jeff,
>
> Under the same user who is the owner of the keystore and truststore.
> Yes. All are running under the same user.
>
>
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/StandardSSLContextService-error-tp12887p12893.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: Apache Yetus annotations : clarifying public vs private api

[Joe Skora]

+1

This should help clarify things when learning NiFi and when diving in new
areas of the codebase.

Thanks, I'm looking forward to it.

On Mon, Jul 25, 2016 at 7:45 PM, Matt Burgess  wrote:

> agreed, big +1 here, will make things much more clear re: stability
> and extensibility.
>
> On Mon, Jul 25, 2016 at 7:43 PM, Tony Kurc  wrote:
> > awesome!
> >
> > On Mon, Jul 25, 2016 at 7:32 PM, Joe Witt  wrote:
> >
> >> Team
> >>
> >> For those interested I am going to take a stab at working through the
> >> codebase [1] to apply Apache Yetus [2] annotations to make it more
> >> clear and well communicated what are the public vs private aspects of
> >> the API.
> >>
> >> We've had the 'nifi-api' module for some time now but unless you know
> >> that this was meant to be the public API for supported extension types
> >> you'd not really know that other parts of NiFi were not meant to be
> >> altered and could be changed.  Therefore, we've not been able to
> >> evolve the framework as directly as we'd like at times.  We had to
> >> wait until 1.0 to make changes which otherwise should have been ok to
> >> make.  Also, previously 'nifi-api' had a lot of extraneous bits in it
> >> that have now been refactored out to where they belong such as core
> >> framework itself or in a 'nifi-framework-api' module.  This means
> >> things in 'nifi-api' should be quite stable and public.
> >>
> >> Yetus gives us a chance to clearly articulate which parts of the API
> >> are meant for public use and which are meant for private use only and
> >> therefore are subject to change.  Furthermore, even for those public
> >> elements we can more effectively articulate stability in ways that map
> >> really nicely to our versioning scheme.
> >>
> >> My intent is to be very cautious in labeling anything beyond the
> >> nifi-api as public.  We can open things up as they prove to be stable
> >> or clearly need to be made available as supported points of extension
> >> but we can't really go back the other way.
> >>
> >> We of course also have to realize our 'interface' is more than these
> >> APIs we're talking about.  It is the REST interface, it is
> >> configuration files, it is templates, and so on.
> >>
> >> Anyway, I hope to have the PR up for this very soon but wanted to send
> >> a special note to the dev mailing list in case there are folks
> >> particularly interested in this and who want to share their views.  If
> >> you are please keep an eye out for NIFI-1373.
> >>
> >> [1] https://issues.apache.org/jira/browse/NIFI-1373
> >> [2]
> >>
> https://yetus.apache.org/documentation/0.3.0/audience-annotations-apidocs/
> >>
> >> Thanks
> >> Joe
> >>
>