I just finished testing with Java 1.8 + JCE Unlimited (w/ DH Keys < 1024
disabled) + BC 1.5. still works.
I’m not exactly sure why Java 1.7 didn’t work for this one processor. But
based on what James posted below, he said: “That suggests that your JRE's
java.security provider does not support the clone method used by the AWS SDK.”
just following his recommendation to update the version of java and that
seemed to do the trick.
Thanks,
Dan M
From: Daniel Morris <dgmorri...@gmail.com>
Date: Friday, August 19, 2016 at 9:42 AM
To: <users@nifi.apache.org>
Subject: Re: PutS3Object Error
In my java 1.7 environment, we have the JCE unlimited and BouncyCastle 1.46.
When I added a Java 1.8 environment and ran NiFi there, I have not yet added
JCE or BouncyCastle yet. That’s my plan today, to add each one and do some
testing after adding each of the components.
Thanks,
Dan M
From: Joe Skora <jsk...@gmail.com>
Reply-To: <users@nifi.apache.org>
Date: Friday, August 19, 2016 at 9:34 AM
To: <users@nifi.apache.org>
Subject: Re: PutS3Object Error
That's great, but I'm not sure why that would have fixed it.
Did you change anything when you switched to 1.8?
On Fri, Aug 19, 2016 at 9:16 AM, dgm <dgmorri...@gmail.com> wrote:
So far, upgrading to Java 1.8 resolved the issue I was experiencing.
Thanks,
Dan Morris
443-992-2848
From: James Wing <jvw...@gmail.com>
Reply-To: <users@nifi.apache.org>
Date: Thursday, August 18, 2016 at 12:58 PM
To: <users@nifi.apache.org>
Subject: Re: PutS3Object Error
I think you got it, Joe, it does make sense now. I had wondered about the
Unlimited Strength policy, but for... MD5? I couldn't get over that. Now, I
am a bit surprised more people haven't run into this.
Dan, I'm afraid I don't know anything about configuring the java.security
providers. Telling you to sabotage BouncyCastle doesn't sound like good
advice, but I'm not sure how to configure a happy coexistence between
BouncyCastle and the AWS SDK.
Thanks,
James
On Thu, Aug 18, 2016 at 7:54 AM, Joe Skora <jsk...@gmail.com> wrote:
The JDK does not require MessageDigest implementations to implement Cloneable,
but the AWS code seems to expect digests to be cloneable. Looking at Bouncy
Castle, it's BCMessageDigest does not support Cloneable.
I'm not familiar enough to offer specifics, but if you can change to a
different message digest that could eliminate the problem.
On Thu, Aug 18, 2016 at 9:03 AM, dgm <dgmorri...@gmail.com> wrote:
James, in addition to using the BC security provider, I was already using the
latest JCE Unlimited Strength Jurisdiction Policy Files from Oracle.
Thanks,
Dan Morris
443-992-2848
From: Daniel Morris <dgmorri...@gmail.com>
Date: Thursday, August 18, 2016 at 7:23 AM
To: <users@nifi.apache.org>
Subject: Re: PutS3Object Error
Thanks James. So when you say the JRE security provider, are you referring to
bouncycastle? If so, I am currently using bcprov-jdk16-1.46.jar.
Thanks,
Dan M
From: James Wing <jvw...@gmail.com>
Reply-To: <users@nifi.apache.org>
Date: Wednesday, August 17, 2016 at 11:45 PM
To: <users@nifi.apache.org>
Subject: Re: PutS3Object Error
The IllegalStateException appears to be a response to a
CloneNotSupportException thrown by java.security.MessageDigest. That suggests
that your JRE's java.security provider does not support the clone method used
by the AWS SDK. Is your java.security setup the JRE default? Would it be
possible to try a different, maybe newer, JRE?
And it's always a bonus when the code throwing your exception is foreshadowed
with the ominous "should never occur":
private MessageDigest cloneFrom(MessageDigest from) {
try {
return (MessageDigest)from.clone();
} catch (CloneNotSupportedException e) { // should never occur
throw new IllegalStateException("unexpected", e);
}
}
(see
https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/internal/MD5DigestCalculatingInputStream.java#L54)
Thanks,
James
On Wed, Aug 17, 2016 at 5:40 PM, dgm <dgmorri...@gmail.com> wrote:
Sure thing!
Version: Nifi 0.7.0, RHEL 6.8, with Java JRE 1.7.0_101
Logs/Stacktrace:
2016-08-17 19:01:09,339 INFO [Timer-Driven Process Thread-3]
o.a.n.p.attributes.UpdateAttribute UpdateAttribute[id=7b485633-a6e3-4aee-
a671-d5b6f2599040] Updated attributes for
StandardFlowFileRecord[uuid=bda2e6c2-7a20-4354-a65f-3ed5b9ef4d06,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1471471716616-15,
container=default, section=15], offset=44598, length=3467],offset=0,
name=20160817190051056-low-text_json_good.json,size=3467]; transferring
to 'success'
29 2016-08-17 19:01:09,346 ERROR [Timer-Driven Process Thread-3]
o.a.nifi.processors.aws.s3.PutS3Object PutS3Object[id=a0836995-c787-40b1-
a8bb-c6f39bb9f37a] PutS3Object[id=a0836995-c787-40b1-a8bb-c6f39bb9f37a]
failed to process due to java.lang.IllegalStateException:
unexpected; rolling back session: java.lang.IllegalStateException: unexpected
30 2016-08-17 19:01:09,349 ERROR [Timer-Driven Process Thread-3]
o.a.nifi.processors.aws.s3.PutS3Object
31 java.lang.IllegalStateException: unexpected
32 at
com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream.cloneFrom(MD5DigestCalculatingInputStream.java:54)
~[na:na]
33 at
com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream.mark(MD5DigestCalculatingInputStream.java:66)
~[na:na]
34 at
com.amazonaws.internal.SdkFilterInputStream.mark(SdkFilterInputStream.java:96)
~[na:na]
35 at
com.amazonaws.internal.SdkFilterInputStream.mark(SdkFilterInputStream.java:96)
~[na:na]
36 at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:472)
~[na:na]
37 at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:317) ~[na:na]
38 at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3595)
~[na:na]
39 at
com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1382)
~[na:na]
40 at
org.apache.nifi.processors.aws.s3.PutS3Object$1.process(PutS3Object.java:446)
~[na:na]
41 at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1851)
~[nifi-framework-core-0.7.0. jar:0.7.0]
42 at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1822)
~[nifi-framework-core-0.7.0. jar:0.7.0]
43 at
org.apache.nifi.processors.aws.s3.PutS3Object.onTrigger(PutS3Object.java:400)
~[na:na]
44 at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
~[nifi-api-0.7.0.jar:0.7.0]
45 at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
[nifi-framework-core-0.7.0.jar:0.7.0]
46 at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-0.7.0. jar:0.7.0]
47 at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-0.7.0. jar:0.7.0]
48 at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
[nifi-framework-core-0. 7.0.jar:0.7.0]
49 at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_101]
50 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_101]
51 at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7. 0_101]
52 at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_101]
53 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_101]
54 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_101]
55 at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101]
56 Caused by: java.lang.CloneNotSupportedException: null
57 at java.security.MessageDigest$Delegate.clone(MessageDigest.java:571)
~[na:1.7.0_101]
58 at
com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream.cloneFrom(MD5DigestCalculatingInputStream.java:52)
~[na:na]
59 ... 23 common frames omitted
60 2016-08-17 19:01:09,350 ERROR [Timer-Driven Process Thread-3]
o.a.nifi.processors.aws.s3.PutS3Object PutS3Object[id=a0836995-c787-40b1-
a8bb-c6f39bb9f37a] PutS3Object[id=a0836995-c787-40b1-a8bb-c6f39bb9f37a]
failed to process session due to java.lang.IllegalStateException:
unexpected: java.lang.IllegalStateException: unexpected
61 2016-08-17 19:01:09,354 ERROR [Timer-Driven Process Thread-3]
o.a.nifi.processors.aws.s3.PutS3Object
62 java.lang.IllegalStateException: unexpected
63 at
com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream.cloneFrom(MD5DigestCalculatingInputStream.java:54)
~[na:na]
64 at
com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream.mark(MD5DigestCalculatingInputStream.java:66)
~[na:na]
65 at
com.amazonaws.internal.SdkFilterInputStream.mark(SdkFilterInputStream.java:96)
~[na:na]
66 at
com.amazonaws.internal.SdkFilterInputStream.mark(SdkFilterInputStream.java:96)
~[na:na]
67 at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:472)
~[na:na]
68 at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:317) ~[na:na]
69 at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3595)
~[na:na]
70 at
com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1382)
~[na:na]
71 at
org.apache.nifi.processors.aws.s3.PutS3Object$1.process(PutS3Object.java:446)
~[na:na]
72 at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1851)
~[nifi-framework-core-0.7.0. jar:0.7.0]
73 at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1822)
~[nifi-framework-core-0.7.0. jar:0.7.0]
74 at
org.apache.nifi.processors.aws.s3.PutS3Object.onTrigger(PutS3Object.java:400)
~[na:na]
75 at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
~[nifi-api-0.7.0.jar:0.7.0]
76 at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
~[nifi-framework-core-0.7.0.jar:0.7.0]
77 at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-0.7.0. jar:0.7.0]
78 at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-0.7.0. jar:0.7.0]
79 at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
[nifi-framework-core-0. 7.0.jar:0.7.0]
80 at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_101]
81 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_101]
82 at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7. 0_101]
83 at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_101]
84 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_101]
85 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_101]
86 at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101]
87 Caused by: java.lang.CloneNotSupportedException: null
88 at java.security.MessageDigest$Delegate.clone(MessageDigest.java:571)
~[na:1.7.0_101]
89 at
com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream.cloneFrom(MD5DigestCalculatingInputStream.java:52)
~[na:na]
90 ... 23 common frames omitted
91 2016-08-17 19:01:09,354 WARN [Timer-Driven Process Thread-3]
o.a.nifi.processors.aws.s3.PutS3Object PutS3Object[id=a0836995-c787-40b1-
a8bb-c6f39bb9f37a] Processor Administratively Yielded for 1 sec due to
processing failure
Thanks,
Dan M
From: James Wing <jvw...@gmail.com>
Reply-To: <users@nifi.apache.org>
Date: Wednesday, August 17, 2016 at 8:32 PM
To: <users@nifi.apache.org>
Subject: Re: PutS3Object Error
Dan,
Would you please share the version of NiFi you are using? Also, would you
please look in logs/nifi-app.log for the stack trace of the exception and any
nested exceptions?
Thanks,
James
On Wed, Aug 17, 2016 at 5:05 PM, dgm <dgmorri...@gmail.com> wrote:
I’m just staring to use nifi and having an issue with the PutS3Object
processor. I believe that it's configured consistently with my other S3
processors, using the same accounts/permissions... receiving the following
error in nifi: PutS3Object[id=a3d6e547-9f71-4bcd-8873-108575604ab4]
PutS3Object[id=a3d6e547-9f71-4bcd-8873-108575604ab4] failed to process session
due to java.lang.IllegalStateException: unexpected:
java.lang.IllegalStateException: unexpected
When I configured the processor, I’m completing all the mandatory fields (i.e.
object key, bucket, access key, secret key, storage class, and region). I’ve
made sure these are consistent with my other S3 processors (List and Fetch – of
which I can successful retrieve files).
Any help or guidance would be greatly appreciated!
Thanks,
Dan M
