In my java 1.7 environment, we have the JCE unlimited and BouncyCastle 1.46.
When I added a Java 1.8 environment and ran NiFi there, I have not yet added JCE or BouncyCastle yet. That’s my plan today, to add each one and do some testing after adding each of the components. Thanks, Dan M From: Joe Skora <jsk...@gmail.com> Reply-To: <users@nifi.apache.org> Date: Friday, August 19, 2016 at 9:34 AM To: <users@nifi.apache.org> Subject: Re: PutS3Object Error That's great, but I'm not sure why that would have fixed it. Did you change anything when you switched to 1.8? On Fri, Aug 19, 2016 at 9:16 AM, dgm <dgmorri...@gmail.com> wrote: So far, upgrading to Java 1.8 resolved the issue I was experiencing. Thanks, Dan Morris 443-992-2848 From: James Wing <jvw...@gmail.com> Reply-To: <users@nifi.apache.org> Date: Thursday, August 18, 2016 at 12:58 PM To: <users@nifi.apache.org> Subject: Re: PutS3Object Error I think you got it, Joe, it does make sense now. I had wondered about the Unlimited Strength policy, but for... MD5? I couldn't get over that. Now, I am a bit surprised more people haven't run into this. Dan, I'm afraid I don't know anything about configuring the java.security providers. Telling you to sabotage BouncyCastle doesn't sound like good advice, but I'm not sure how to configure a happy coexistence between BouncyCastle and the AWS SDK. Thanks, James On Thu, Aug 18, 2016 at 7:54 AM, Joe Skora <jsk...@gmail.com> wrote: The JDK does not require MessageDigest implementations to implement Cloneable, but the AWS code seems to expect digests to be cloneable. Looking at Bouncy Castle, it's BCMessageDigest does not support Cloneable. I'm not familiar enough to offer specifics, but if you can change to a different message digest that could eliminate the problem. On Thu, Aug 18, 2016 at 9:03 AM, dgm <dgmorri...@gmail.com> wrote: James, in addition to using the BC security provider, I was already using the latest JCE Unlimited Strength Jurisdiction Policy Files from Oracle. Thanks, Dan Morris 443-992-2848 From: Daniel Morris <dgmorri...@gmail.com> Date: Thursday, August 18, 2016 at 7:23 AM To: <users@nifi.apache.org> Subject: Re: PutS3Object Error Thanks James. So when you say the JRE security provider, are you referring to bouncycastle? If so, I am currently using bcprov-jdk16-1.46.jar. Thanks, Dan M From: James Wing <jvw...@gmail.com> Reply-To: <users@nifi.apache.org> Date: Wednesday, August 17, 2016 at 11:45 PM To: <users@nifi.apache.org> Subject: Re: PutS3Object Error The IllegalStateException appears to be a response to a CloneNotSupportException thrown by java.security.MessageDigest. That suggests that your JRE's java.security provider does not support the clone method used by the AWS SDK. Is your java.security setup the JRE default? Would it be possible to try a different, maybe newer, JRE? And it's always a bonus when the code throwing your exception is foreshadowed with the ominous "should never occur": private MessageDigest cloneFrom(MessageDigest from) { try { return (MessageDigest)from.clone(); } catch (CloneNotSupportedException e) { // should never occur throw new IllegalStateException("unexpected", e); } } (see https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/internal/MD5DigestCalculatingInputStream.java#L54) Thanks, James On Wed, Aug 17, 2016 at 5:40 PM, dgm <dgmorri...@gmail.com> wrote: Sure thing! Version: Nifi 0.7.0, RHEL 6.8, with Java JRE 1.7.0_101 Logs/Stacktrace: 2016-08-17 19:01:09,339 INFO [Timer-Driven Process Thread-3] o.a.n.p.attributes.UpdateAttribute UpdateAttribute[id=7b485633-a6e3-4aee- a671-d5b6f2599040] Updated attributes for StandardFlowFileRecord[uuid=bda2e6c2-7a20-4354-a65f-3ed5b9ef4d06,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1471471716616-15, container=default, section=15], offset=44598, length=3467],offset=0, name=20160817190051056-low-text_json_good.json,size=3467]; transferring to 'success' 29 2016-08-17 19:01:09,346 ERROR [Timer-Driven Process Thread-3] o.a.nifi.processors.aws.s3.PutS3Object PutS3Object[id=a0836995-c787-40b1- a8bb-c6f39bb9f37a] PutS3Object[id=a0836995-c787-40b1-a8bb-c6f39bb9f37a] failed to process due to java.lang.IllegalStateException: unexpected; rolling back session: java.lang.IllegalStateException: unexpected 30 2016-08-17 19:01:09,349 ERROR [Timer-Driven Process Thread-3] o.a.nifi.processors.aws.s3.PutS3Object 31 java.lang.IllegalStateException: unexpected 32 at com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream.cloneFrom(MD5DigestCalculatingInputStream.java:54) ~[na:na] 33 at com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream.mark(MD5DigestCalculatingInputStream.java:66) ~[na:na] 34 at com.amazonaws.internal.SdkFilterInputStream.mark(SdkFilterInputStream.java:96) ~[na:na] 35 at com.amazonaws.internal.SdkFilterInputStream.mark(SdkFilterInputStream.java:96) ~[na:na] 36 at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:472) ~[na:na] 37 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:317) ~[na:na] 38 at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3595) ~[na:na] 39 at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1382) ~[na:na] 40 at org.apache.nifi.processors.aws.s3.PutS3Object$1.process(PutS3Object.java:446) ~[na:na] 41 at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1851) ~[nifi-framework-core-0.7.0. jar:0.7.0] 42 at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1822) ~[nifi-framework-core-0.7.0. jar:0.7.0] 43 at org.apache.nifi.processors.aws.s3.PutS3Object.onTrigger(PutS3Object.java:400) ~[na:na] 44 at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-0.7.0.jar:0.7.0] 45 at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054) [nifi-framework-core-0.7.0.jar:0.7.0] 46 at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-0.7.0. jar:0.7.0] 47 at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-0.7.0. jar:0.7.0] 48 at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127) [nifi-framework-core-0. 7.0.jar:0.7.0] 49 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_101] 50 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_101] 51 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7. 0_101] 52 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_101] 53 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_101] 54 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_101] 55 at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101] 56 Caused by: java.lang.CloneNotSupportedException: null 57 at java.security.MessageDigest$Delegate.clone(MessageDigest.java:571) ~[na:1.7.0_101] 58 at com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream.cloneFrom(MD5DigestCalculatingInputStream.java:52) ~[na:na] 59 ... 23 common frames omitted 60 2016-08-17 19:01:09,350 ERROR [Timer-Driven Process Thread-3] o.a.nifi.processors.aws.s3.PutS3Object PutS3Object[id=a0836995-c787-40b1- a8bb-c6f39bb9f37a] PutS3Object[id=a0836995-c787-40b1-a8bb-c6f39bb9f37a] failed to process session due to java.lang.IllegalStateException: unexpected: java.lang.IllegalStateException: unexpected 61 2016-08-17 19:01:09,354 ERROR [Timer-Driven Process Thread-3] o.a.nifi.processors.aws.s3.PutS3Object 62 java.lang.IllegalStateException: unexpected 63 at com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream.cloneFrom(MD5DigestCalculatingInputStream.java:54) ~[na:na] 64 at com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream.mark(MD5DigestCalculatingInputStream.java:66) ~[na:na] 65 at com.amazonaws.internal.SdkFilterInputStream.mark(SdkFilterInputStream.java:96) ~[na:na] 66 at com.amazonaws.internal.SdkFilterInputStream.mark(SdkFilterInputStream.java:96) ~[na:na] 67 at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:472) ~[na:na] 68 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:317) ~[na:na] 69 at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3595) ~[na:na] 70 at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1382) ~[na:na] 71 at org.apache.nifi.processors.aws.s3.PutS3Object$1.process(PutS3Object.java:446) ~[na:na] 72 at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1851) ~[nifi-framework-core-0.7.0. jar:0.7.0] 73 at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1822) ~[nifi-framework-core-0.7.0. jar:0.7.0] 74 at org.apache.nifi.processors.aws.s3.PutS3Object.onTrigger(PutS3Object.java:400) ~[na:na] 75 at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-0.7.0.jar:0.7.0] 76 at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054) ~[nifi-framework-core-0.7.0.jar:0.7.0] 77 at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-0.7.0. jar:0.7.0] 78 at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-0.7.0. jar:0.7.0] 79 at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127) [nifi-framework-core-0. 7.0.jar:0.7.0] 80 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_101] 81 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_101] 82 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7. 0_101] 83 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_101] 84 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_101] 85 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_101] 86 at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101] 87 Caused by: java.lang.CloneNotSupportedException: null 88 at java.security.MessageDigest$Delegate.clone(MessageDigest.java:571) ~[na:1.7.0_101] 89 at com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream.cloneFrom(MD5DigestCalculatingInputStream.java:52) ~[na:na] 90 ... 23 common frames omitted 91 2016-08-17 19:01:09,354 WARN [Timer-Driven Process Thread-3] o.a.nifi.processors.aws.s3.PutS3Object PutS3Object[id=a0836995-c787-40b1- a8bb-c6f39bb9f37a] Processor Administratively Yielded for 1 sec due to processing failure Thanks, Dan M From: James Wing <jvw...@gmail.com> Reply-To: <users@nifi.apache.org> Date: Wednesday, August 17, 2016 at 8:32 PM To: <users@nifi.apache.org> Subject: Re: PutS3Object Error Dan, Would you please share the version of NiFi you are using? Also, would you please look in logs/nifi-app.log for the stack trace of the exception and any nested exceptions? Thanks, James On Wed, Aug 17, 2016 at 5:05 PM, dgm <dgmorri...@gmail.com> wrote: I’m just staring to use nifi and having an issue with the PutS3Object processor. I believe that it's configured consistently with my other S3 processors, using the same accounts/permissions... receiving the following error in nifi: PutS3Object[id=a3d6e547-9f71-4bcd-8873-108575604ab4] PutS3Object[id=a3d6e547-9f71-4bcd-8873-108575604ab4] failed to process session due to java.lang.IllegalStateException: unexpected: java.lang.IllegalStateException: unexpected When I configured the processor, I’m completing all the mandatory fields (i.e. object key, bucket, access key, secret key, storage class, and region). I’ve made sure these are consistent with my other S3 processors (List and Fetch – of which I can successful retrieve files). Any help or guidance would be greatly appreciated! Thanks, Dan M