[
https://issues.apache.org/jira/browse/HADOOP-19455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17933383#comment-17933383
]
Steve Loughran commented on HADOOP-19455:
-----------------------------------------
It'd be nicest to wire up the logging publisher so it'd automatically to the
pretty logs at trace level -so could be turned on when needed.
However, that'd require some reflection to invoke a constructor which, in the
shaded bundle.jar, needs a reference to the shaded slf4j enum.
Instead, it'll be set to wire up to log at INFO, but only if
DefaultS3ClientFactory is set to log at trace
{code}
log4j.logger.org.apache.hadoop.fs.s3a.DefaultS3ClientFactory=TRACE
{code}
{code}
2025-03-07 17:34:26,086 [s3a-transfer-stevel-london-bounded-pool2-t1] DEBUG
impl.S3AStoreImpl (DurationInfo.java:<init>(80)) - Starting: deleting
test/testXAttrFile
2025-03-07 17:34:26,154 [s3a-transfer-stevel-london-bounded-pool2-t1] INFO
metrics.LoggingMetricPublisher (LoggerAdapter.java:info(165)) - Metrics
published: MetricCollection(name=ApiCall,
metrics=[MetricRecord(metric=MarshallingDuration, value=PT0.000092041S),
MetricRecord(metric=RetryCount, value=0),
MetricRecord(metric=ApiCallSuccessful, value=true),
MetricRecord(metric=OperationName, value=DeleteObject),
MetricRecord(metric=EndpointResolveDuration, value=PT0.000132792S),
MetricRecord(metric=ApiCallDuration, value=PT0.064890875S),
MetricRecord(metric=CredentialsFetchDuration, value=PT0.000017458S),
MetricRecord(metric=ServiceEndpoint,
value=https://stevel-london.s3.eu-west-2.amazonaws.com),
MetricRecord(metric=ServiceId, value=S3)],
children=[MetricCollection(name=ApiCallAttempt,
metrics=[MetricRecord(metric=TimeToFirstByte, value=PT0.06260225S),
MetricRecord(metric=SigningDuration, value=PT0.000293083S),
MetricRecord(metric=ReadThroughput, value=0.0),
MetricRecord(metric=ServiceCallDuration, value=PT0.06260225S),
MetricRecord(metric=AwsExtendedRequestId,
value=zvfoe8Vr8JFB42aDPawtJ3VfqV/xm0zyJ2jt9GG6027aVErpWdQ3UYdLEG8TVIAldxUovsyVJdw=),
MetricRecord(metric=HttpStatusCode, value=204),
MetricRecord(metric=BackoffDelayDuration, value=PT0S),
MetricRecord(metric=TimeToLastByte, value=PT0.064313667S),
MetricRecord(metric=AwsRequestId, value=RKZD44SE5DW91K1G)],
children=[MetricCollection(name=HttpClient,
metrics=[MetricRecord(metric=AvailableConcurrency, value=1),
MetricRecord(metric=LeasedConcurrency, value=0),
MetricRecord(metric=ConcurrencyAcquireDuration, value=PT0S),
MetricRecord(metric=PendingConcurrencyAcquires, value=0),
MetricRecord(metric=MaxConcurrency, value=512),
MetricRecord(metric=HttpClientName, value=Apache)], children=[])])])
{code}
> S3A: support metrics publishers
> -------------------------------
>
> Key: HADOOP-19455
> URL: https://issues.apache.org/jira/browse/HADOOP-19455
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
>
> Add options to enable logging and Cloudwatch metrics
> always turn on logging metrics but log at trace, so never evaluated or
> visible unless requested.
> Cloudwatch would be interesting but complicated, because lots of config. Also
> it should be a singleton: first configuration wins.
> Whoever wants cloudwatch gets to implement that themselves with docs and
> tests (how?)
> https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/metrics/LoggingMetricPublisher.html
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
