Hi James Yes, happy to share the configuration we use:
We have an institute-wide proxy server that requires user credentials for each request (domain, uname, passwd, port 80 and 443 only). We run NiFi on Linux hosts using cntlm as the local proxy. Users provide their domain, uname and passwd to cntlm, and point their applications to localhost:3128 as the proxy, and cntlm sends on the proper credentials to the actual proxy for each request (if that is clear). We point GetHTTP processors etc to cntlm and it works fine, even for https web pages. We have one cert that is imported into browsers, and again, all browsers point to localhost:3128 as the proxy. This seems to work fine, we just export http_proxy=localhost:3128 and https_proxy=localhost:3128 at the bash shell. AWS endpoints are https and unfortunately aws command line tools now only work when we specify --no-verify-ssl option, otherwise we get the following error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590) So I was wondering what further configuration steps I need to take to get S3/SQS working behind our proxy. Many thanks John On Thu, Nov 3, 2016 at 6:42 PM, James Wing <jvw...@gmail.com> wrote: > The short answer is no, PutS3Object does not currently support a direct > equivalent of the AWS CLI's --no-verify-ssl option. There is an option to > provide your own SSLContextService, if you need to establish trust with > your proxy server (maybe, I'm not sure). > > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.ssl. > StandardSSLContextService/index.html > > Can you share a bit more about your use case and proxy setup? I know > there are other NiFi installations using proxy servers against S3, and I do > not believe they have had this problem. > > Last, I believe I foolishly stated in an earlier email that the AWS CLI > was a good comparison tool, but I might have to flip-flop now that we're > bringing proxy settings and SSL verification into the picture. Are you > sure the CLI is using your proxy similarly? > > Thanks, > > James > > On Thu, Nov 3, 2016 at 5:58 AM, John Burns <jzbu...@gmail.com> wrote: > >> Hi, >> >> I have a workflow that compresses an file then invokes PutS3Object to >> store in an S3 bucket. This processor works fine in a non-proxy >> environment, where PutS3Object is parameterised correctly with the proxy >> settings, but in a proxy environment I get the following error shown in the >> stack trace. >> >> Testing from the AWS cli tools, I need to use the --no-verify-ssl >> parameter: >> >> aws s3 ls --no-verify-ssl s3://nifibucket/ >> >> Is there an equivalent "--no-verify-ss"for the PutS3Object processor? >> >> Thanks >> >> John >> >> >> ERROR [Timer-Driven Process Thread-10] o.a.nifi.processors.aws.s3.PutS3Object >> PutS3Object[id=26ea1644-0158-1000-be29-271b59722ea4] Failed to put >> StandardFlowFileRecord[uuid=72488dde-07c8-4236-8116-bd8b34d9 >> 3716,claim=StandardContentClaim >> [resourceClaim=StandardResourceClaim[id=1478122984174-68, >> container=default, section=68], offset=233361, >> length=34033],offset=0,name=bbctext.gz,size=34033] to Amazon S3 due to >> com.amazonaws.AmazonClientException: Unable to execute HTTP request: >> sun.security.validator.ValidatorException: PKIX path building failed: >> sun.security.provider.certpath.SunCertPathBuilderException: unable to >> find valid certification path to requested target: >> com.amazonaws.AmazonClientException: Unable to execute HTTP request: >> sun.security.validator.ValidatorException: PKIX path building failed: >> sun.security.provider.certpath.SunCertPathBuilderException: unable to >> find valid certification path to requested target >> 2016-11-03 12:49:50,876 ERROR [Timer-Driven Process Thread-10] >> o.a.nifi.processors.aws.s3.PutS3Object >> com.amazonaws.AmazonClientException: Unable to execute HTTP request: >> sun.security.validator.ValidatorException: PKIX path building failed: >> sun.security.provider.certpath.SunCertPathBuilderException: unable to >> find valid certification path to requested target >> at >> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:706) >> ~[aws-java-sdk-core-1.11.8.jar:na] >> at >> com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:447) >> ~[aws-java-sdk-core-1.11.8.jar:na] >> at >> com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:409) >> ~[aws-java-sdk-core-1.11.8.jar:na] >> at >> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:358) >> ~[aws-java-sdk-core-1.11.8.jar:na] >> at >> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3787) >> ~[aws-java-sdk-s3-1.11.8.jar:na] >> at >> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1399) >> ~[aws-java-sdk-s3-1.11.8.jar:na] >> at >> org.apache.nifi.processors.aws.s3.PutS3Object$1.process(PutS3Object.java:451) >> ~[nifi-aws-processors-1.0.0.jar:1.0.0] >> at org.apache.nifi.controller.repository.StandardProcessSession >> .read(StandardProcessSession.java:1880) ~[na:na] >> at org.apache.nifi.controller.repository.StandardProcessSession >> .read(StandardProcessSession.java:1851) ~[na:na] >> at >> org.apache.nifi.processors.aws.s3.PutS3Object.onTrigger(PutS3Object.java:401) >> ~[nifi-aws-processors-1.0.0.jar:1.0.0] >> at >> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) >> [nifi-api-1.0.0.jar:1.0.0] >> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(S >> tandardProcessorNode.java:1064) [nifi-framework-core-1.0.0.jar:1.0.0] >> at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask >> .call(ContinuallyRunProcessorTask.java:136) >> [nifi-framework-core-1.0.0.jar:1.0.0] >> at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask >> .call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.0.0.jar >> :1.0.0] >> at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingA >> gent$1.run(TimerDrivenSchedulingAgent.java:132) >> [nifi-framework-core-1.0.0.jar:1.0.0] >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> [na:1.8.0_60] >> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) >> [na:1.8.0_60] >> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu >> tureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_60] >> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu >> tureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_60] >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> [na:1.8.0_60] >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> lExecutor.ja >> > >
