steveloughran commented on pull request #2675: URL: https://github.com/apache/hadoop/pull/2675#issuecomment-773544133
Latest release adds operation context info to the referer logs so can generate a trace mapping S3 operations to the FS ops and S3A FS instance making the call. No tie-in to opentelemetry tho'. Future work. ``` 183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8 stevel-london [04/Feb/2021:17:10:40 +0000] 109.157.193.10 arn:aws:iam::152813717728:user/stevel-dev 0E574923013D27E3 REST.DELETE.OBJECT fork-0003/test/ITestS3AContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/DELAY_LISTING_ME/outputDir/inputDir/.distcp.tmp.attempt_local1771928054_0008_m_000001_0.1612458639648 "DELETE /fork-0003/test/ITestS3AContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/DELAY_LISTING_ME/outputDir/inputDir/.distcp.tmp.attempt_local1771928054_0008_m_000001_0.1612458639648 HTTP/1.1" 204 - - - 14 - "http://hadoop.apache.org/audit/0ce4df25-02cc-43d0-98a2-6b13f12662dd/span/17.08.39#343/operation/op_rename"; "Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.11.901 Mac_OS_X/10.15.7 OpenJDK_64-Bit_Server_VM/25.252-b09 java/1.8.0_252 vendor/AdoptOpenJDK" - EE5czDIkLDNEPLxZ8XDADyRrZQ5KvstPk1E4Hei1qceDWnG+nAg7CrutshO9eEhmRhBuviommqY= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader stevel-london.s3. eu-west-2.amazonaws.com TLSv1.2 183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8 stevel-london [04/Feb/2021:17:10:40 +0000] 109.157.193.10 arn:aws:iam::152813717728:user/stevel-dev 1CF7A2A5E1AE304E REST.HEAD.OBJECT fork-0003/test/ITestS3AContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/DELAY_LISTING_ME/outputDir/inputDir/subDir2/subDir2/newfile1 "HEAD /fork-0003/test/ITestS3AContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/DELAY_LISTING_ME/outputDir/inputDir/subDir2/subDir2/newfile1 HTTP/1.1" 404 NoSuchKey 417 - 8 - "http://hadoop.apache.org/audit/0ce4df25-02cc-43d0-98a2-6b13f12662dd/span/17.08.39#341/operation/op_exists"; "Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.11.901 Mac_OS_X/10.15.7 OpenJDK_64-Bit_Server_VM/25.252-b09 java/1.8.0_252 vendor/AdoptOpenJDK" - 1FGjqpSb90398PGXq7yUMEWrJnWb5Gk8tGHXCj5adoFJxVleU0ffo93JQBJ/6kEL8+glLrpS0Ik= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader stevel-london.s3.eu-west-2.amazonaws.com TLSv1.2 183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8 stevel-london [04/Feb/2021:17:10:40 +0000] 109.157.193.10 arn:aws:iam::152813717728:user/stevel-dev 243F9FE15CB25A9E REST.HEAD.OBJECT fork-0003/test/ITestS3AContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/DELAY_LISTING_ME/outputDir/inputDir/.distcp.tmp.attempt_local1771928054_0008_m_000001_0.1612458639648 "HEAD /fork-0003/test/ITestS3AContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/DELAY_LISTING_ME/outputDir/inputDir/.distcp.tmp.attempt_local1771928054_0008_m_000001_0.1612458639648 HTTP/1.1" 200 - - 0 5 - "-" "Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.11.901 Mac_OS_X/10.15.7 OpenJDK_64-Bit_Server_VM/25.252-b09 java/1.8.0_252 vendor/AdoptOpenJDK" - h4p6Og/ROqiWR7THNlvl7jK8mWSNbWbne2HfUvi+HO88IsW/seDQ1o8KvTJ6+FxgBKbH2JgCW24= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader stevel-london.s3.eu-west-2.amazonaws.com TLSv1.2 183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8 stevel-london [04/Feb/2021:17:10:40 +0000] 109.157.193.10 arn:aws:iam::152813717728:user/stevel-dev 470666E02E16F8B6 REST.GET.BUCKET - "GET /?list-type=2&delimiter=%2F&max-keys=2&prefix=fork-0003%2Ftest%2FITestS3AContractDistCp%2FtestUpdateDeepDirectoryStructureToRemote%2Fremote%2FDELAY_LISTING_ME%2FoutputDir%2FinputDir%2FsubDir2%2FsubDir2%2Fnewfile1%2F&fetch-owner=false HTTP/1.1" 200 - 412 - 13 12 "http://hadoop.apache.org/audit/0ce4df25-02cc-43d0-98a2-6b13f12662dd/span/17.08.39#343/operation/op_rename"; "Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.11.901 Mac_OS_X/10.15.7 OpenJDK_64-Bit_Server_VM/25.252-b09 java/1.8.0_252 vendor/AdoptOpenJDK" - p6rYqjQGRRT7jzz1tpdyshDJ673ul1A4YYHb/CFLOTv8++McNJYMxev+98vjEU1Kute5ggU3sFM= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader stevel-london.s3.eu-west-2.amazonaws.com TLSv1.2 183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8 stevel-london [04/Feb/2021:17:10:40 +0000] 109.157.193.10 arn:aws:iam::152813717728:user/stevel-dev 5F2E21B0F0071218 REST.COPY.OBJECT fork-0003/test/ITestS3AContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/DELAY_LISTING_ME/outputDir/inputDir/subDir2/subDir2/newfile1 "PUT /fork-0003/test/ITestS3AContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/DELAY_LISTING_ME/outputDir/inputDir/subDir2/subDir2/newfile1 HTTP/1.1" 200 - 234 0 41 19 "http://hadoop.apache.org/audit/0ce4df25-02cc-43d0-98a2-6b13f12662dd/span/17.08.39#343/operation/op_rename"; "Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.11.901 Mac_OS_X/10.15.7 OpenJDK_64-Bit_Server_VM/25.252-b09 java/1.8.0_252 vendor/AdoptOpenJDK com.amazonaws.services.s3.transfer.TransferManager/1.11.901" - bkgJI8wkqzT6Cx+jsbLFB6Mra/qLJiwa0VxvginE5nk3xBT15dBp0Tn8m3HLom2oChaybsbUSFc= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader stevel-london.s3.eu-west-2.amazonaws.com TLSv1.2 183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8 stevel-london [04/Feb/2021:17:10:40 +0000] 109.157.193.10 arn:aws:iam::152813717728:user/stevel-dev 5F2E21B0F0071218 REST.COPY.OBJECT_GET fork-0003/test/ITestS3AContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/DELAY_LISTING_ME/outputDir/inputDir/.distcp.tmp.attempt_local1771928054_0008_m_000001_0.1612458639648 - 200 - - 0 - - - - - bkgJI8wkqzT6Cx+jsbLFB6Mra/qLJiwa0VxvginE5nk3xBT15dBp0Tn8m3HLom2oChaybsbUSFc= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader stevel-london.s3.eu-west-2.amazonaws.com TLSv1.2 183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8 stevel-london [04/Feb/2021:17:10:40 +0000] 109.157.193.10 arn:aws:iam::152813717728:user/stevel-dev 82F908B5DE7BCF3D REST.HEAD.OBJECT fork-0003/test/ITestS3AContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/DELAY_LISTING_ME/outputDir/inputDir/.distcp.tmp.attempt_local1771928054_0008_m_000001_0.1612458639648 "HEAD /fork-0003/test/ITestS3AContractDistCp/testUpdateDeepDirectoryStructureToRemote/remote/DELAY_LISTING_ME/outputDir/inputDir/.distcp.tmp.attempt_local1771928054_0008_m_000001_0.1612458639648 HTTP/1.1" 200 - - 0 6 - "http://hadoop.apache.org/audit/0ce4df25-02cc-43d0-98a2-6b13f12662dd/span/17.08.39#343/operation/op_rename"; "Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.11.901 Mac_OS_X/10.15.7 OpenJDK_64-Bit_Server_VM/25.252-b09 java/1.8.0_252 vendor/AdoptOpenJDK" - 4czEsj1JWzHooqkzqfCW2W/U0ofSs3KUpDUWD9HeYE2POs7AtUP9vG/wDWOCG/vMRZmogEOxMZs= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader stevel-london.s3.eu-we st-2.amazonaws.com TLSv1.2 183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8 stevel-london [04/Feb/2021:17:10:40 +0000] 109.157.193.10 arn:aws:iam::152813717728:user/stevel-dev 9774E0C9B42FB919 BATCH.DELETE.OBJECT fork-0003/ - 204 - - - - - - - - pbNOdYmZ24OIqstCyMnVcnT8PQlxJlYSnFSIiZrGLFjy6/UHFcySp78w0gaB0TgxRO2CJTakx+A= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader stevel-london.s3.eu-west-2.amazonaws.com TLSv1.2 183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8 stevel-london [04/Feb/2021:17:10:40 +0000] 109.157.193.10 arn:aws:iam::152813717728:user/stevel-dev 9774E0C9B42FB919 BATCH.DELETE.OBJECT fork-0003/test/ - 204 - - - - - - - - pbNOdYmZ24OIqstCyMnVcnT8PQlxJlYSnFSIiZrGLFjy6/UHFcySp78w0gaB0TgxRO2CJTakx+A= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader stevel-london.s3.eu-west-2.amazonaws.com TLSv1.2 183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8 stevel-london [04/Feb/2021:17:10:40 +0000] 109.157.193.10 arn:aws:iam::152813717728:user/stevel-dev 9774E0C9B42FB919 BATCH.DELETE.OBJECT fork-0003/test/ITestS3AContractDistCp/ - 204 - - - - - - - - pbNOdYmZ24OIqstCyMnVcnT8PQlxJlYSnFSIiZrGLFjy6/UHFcySp78w0gaB0TgxRO2CJTakx+A= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader stevel-london.s3.eu-west- ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
