elek opened a new pull request #449: HDDS-2891. Apache NiFi PutFile processor is failing with secure Ozone S3G URL: https://github.com/apache/hadoop-ozone/pull/449 ## What changes were proposed in this pull request? Multipart upload is not working in secure environment as the `OzoneClientProducer` might create the the OzoneClient after executing the `HeaderPreprocessor` filter. `HeaderPreprocessor` modifies the header and the AWS signature check is not possible after executing this filter. It should be done before. Hard to detect the problem, but if you debug both the AWS SDK under NiFi and the signature check of Ozone (OM + S3G) you can see that the canonical message (the base of the signature) contains a wrong content type (`ozone/mpu`) instead of the original one. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-2891 ## How was this patch tested? I started a local NiFi and created a flow with PutS3 processor. It worked ## What has been changed in this patch? Over the time it become a bigger patch, so here is an overview: ``` deleted: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/header/AuthenticationHeaderParser.java renamed: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/AWSAuthParser.java -> hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/SignatureProcessor.java renamed: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/AWSV4AuthParser.java -> hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/AWSV4SignatureProcessor.java ``` The main idea is to remove the duplicated code and use only one single class to process the signature. (Renamed to SignatureProcessor as we already have a header parser to parse the header entry). The other big modification is to do a snapshot in the SignatureProcessor from the HTTP headers. Earlier it was a writable reference which made it very fragile as the HeaderPreprocessor modifies the header. ``` modified: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/OzoneClientProducer.java ``` Just use the one single header parser instead of the other, old one. ``` modified: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/HeaderPreprocessor.java ``` To make it more safe, I save the original header and the SignatureProcessor can restore the original Contant-Type.- ``` new file: hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/TestAWSV4SignatureProcessor.java ``` A new unit test to make sure the Content-Type restore is working ``` modified: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/OzoneS3Util.java ``` That's a tricky bug. We need to use `toLowerCase` as it's already used on the server side. Without that we can get very cryptic error. (For example during the createBucket the bucket is created but during the check at the end of the call the acl check is failing.... ``` modified: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java modified: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/VirtualHostStyleFilter.java modified: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java modified: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/EndpointBase.java modified: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/header/AuthorizationHeaderV4.java modified: hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/TestOzoneClientProducer.java modified: hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/TestVirtualHostStyleFilter.java modified: hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestBucketPut.java modified: hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestRootList.java modified: hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/header/TestAuthorizationHeaderV4.java ``` The remaining modifications just test updated / clarifications. ``` modified: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/RootEndpoint.java ``` This one thing what I couldn't save. The automatic redirect to the S3 browser is removed. (Earlier if you put the s3 bucket URL to the browser it worked. We need a bigger refactor to save it. ~ ~
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
