[
https://issues.apache.org/jira/browse/HUDI-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Power Liu updated HUDI-4815:
----------------------------
Description:
*HUDI version:* 0.11.1
*Application:* Spark 3.1.3
*Running Environment:* AWS EKS pod (kubernetes version: 1.21)
*HUDI jar:* org.apache.hudi:hudi-spark-3.1-bundle_2.12:0.11.1
*HUDI metric configuration:*
* hoodie.metrics.reporter.type: CLOUDWATCH
* hoodie.metrics.on: true
* hoodie.metrics.cloudwatch.report.period.seconds: 60
* hoodie.metrics.cloudwatch.namespace: (we defined our own namespace with
hyphen inside, like xxxx-xxx-xx)
* hoodie.metrics.cloudwatch.metric.prefix: (we defined our own metric prefix
with hyphen inside, like xxxx-xx-xx)
* hoodie.metrics.cloudwatch.maxDatumsPerRequest: 20
*It seems some of the metric reported to Cloudwatch successfully but some are
not. Is there any configuration need to adjust according to the error below?*
*Error log:*
ERROR [metrics-hudi-cloudWatch-reporter-1-thread-1] CloudWatchReporter:236 -
Error reporting metrics to CloudWatch. The data in this CloudWatch request may
have been discarded, and not made it to CloudWatch.
java.util.concurrent.ExecutionException:
{color:#ff0000}com.amazonaws.services.cloudwatch.model.MissingRequiredParameterException:
The parameter MetricData.member.1.Dimensions.member.1.Value is required.
(Service: AmazonCloudWatch; Status Code: 400; Error Code: MissingParameter;
Request ID: xxx-xxxx){color}
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at
org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:234)
at
org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:212)
at
org.apache.hudi.com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:237)
at
org.apache.hudi.com.codahale.metrics.ScheduledReporter.lambda$start$0(ScheduledReporter.java:177)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by:
com.amazonaws.services.cloudwatch.model.MissingRequiredParameterException: The
parameter MetricData.member.1.Dimensions.member.1.Value is required. (Service:
AmazonCloudWatch; Status Code: 400; Error Code: MissingParameter; Request ID:
3eb96da0-d4bc-4687-81a5-0d65c453559a)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1742)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1371)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1347)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1127)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:784)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:752)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
at
com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.doInvoke(AmazonCloudWatchClient.java:2446)
at
com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:2413)
at
com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:2402)
at
com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.executePutMetricData(AmazonCloudWatchClient.java:2170)
at
com.amazonaws.services.cloudwatch.AmazonCloudWatchAsyncClient$26.call(AmazonCloudWatchAsyncClient.java:1170)
at
com.amazonaws.services.cloudwatch.AmazonCloudWatchAsyncClient$26.call(AmazonCloudWatchAsyncClient.java:1164)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
was:
*HUDI version:* 0.11.1
*Application:* Spark 3.1.3
*Running Environment:* AWS EKS pod (kubernetes version: 1.21)
*HUDI jar:* org.apache.hudi:hudi-spark-3.1-bundle_2.12:0.11.1
*HUDI metric configuration:*
* hoodie.metrics.reporter.type: CLOUDWATCH
* hoodie.metrics.on: true
* hoodie.metrics.cloudwatch.report.period.seconds: 60
* hoodie.metrics.cloudwatch.namespace: (we defined our own namespace with
hyphen inside, like xxxx-xxx-xx)
* hoodie.metrics.cloudwatch.metric.prefix: (we defined our own metric prefix
with hyphen inside, like xxxx-xx-xx)
* hoodie.metrics.cloudwatch.maxDatumsPerRequest: 20
*It seems some of the metric reported to Cloudwatch successfully but some are
not. Is there any configuration need to adjust according to the error below?*
*Error log:*
ERROR [metrics-hudi-cloudWatch-reporter-1-thread-1] CloudWatchReporter:236 -
Error reporting metrics to CloudWatch. The data in this CloudWatch request may
have been discarded, and not made it to CloudWatch.
java.util.concurrent.ExecutionException:
{color:#FF0000}com.amazonaws.services.cloudwatch.model.MissingRequiredParameterException:
The parameter MetricData.member.1.Dimensions.member.1.Value is required.
(Service: AmazonCloudWatch; Status Code: 400; Error Code: MissingParameter;
Request ID: 3eb96da0-d4bc-4687-81a5-0d65c453559a){color}
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at
org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:234)
at
org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:212)
at
org.apache.hudi.com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:237)
at
org.apache.hudi.com.codahale.metrics.ScheduledReporter.lambda$start$0(ScheduledReporter.java:177)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by:
com.amazonaws.services.cloudwatch.model.MissingRequiredParameterException: The
parameter MetricData.member.1.Dimensions.member.1.Value is required. (Service:
AmazonCloudWatch; Status Code: 400; Error Code: MissingParameter; Request ID:
3eb96da0-d4bc-4687-81a5-0d65c453559a)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1742)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1371)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1347)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1127)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:784)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:752)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
at
com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.doInvoke(AmazonCloudWatchClient.java:2446)
at
com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:2413)
at
com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:2402)
at
com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.executePutMetricData(AmazonCloudWatchClient.java:2170)
at
com.amazonaws.services.cloudwatch.AmazonCloudWatchAsyncClient$26.call(AmazonCloudWatchAsyncClient.java:1170)
at
com.amazonaws.services.cloudwatch.AmazonCloudWatchAsyncClient$26.call(AmazonCloudWatchAsyncClient.java:1164)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
> Hudi report metrics to AWS cloudwatch error
> -------------------------------------------
>
> Key: HUDI-4815
> URL: https://issues.apache.org/jira/browse/HUDI-4815
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Power Liu
> Priority: Major
>
> *HUDI version:* 0.11.1
> *Application:* Spark 3.1.3
> *Running Environment:* AWS EKS pod (kubernetes version: 1.21)
> *HUDI jar:* org.apache.hudi:hudi-spark-3.1-bundle_2.12:0.11.1
> *HUDI metric configuration:*
> * hoodie.metrics.reporter.type: CLOUDWATCH
> * hoodie.metrics.on: true
> * hoodie.metrics.cloudwatch.report.period.seconds: 60
> * hoodie.metrics.cloudwatch.namespace: (we defined our own namespace with
> hyphen inside, like xxxx-xxx-xx)
> * hoodie.metrics.cloudwatch.metric.prefix: (we defined our own metric prefix
> with hyphen inside, like xxxx-xx-xx)
> * hoodie.metrics.cloudwatch.maxDatumsPerRequest: 20
> *It seems some of the metric reported to Cloudwatch successfully but some are
> not. Is there any configuration need to adjust according to the error below?*
> *Error log:*
> ERROR [metrics-hudi-cloudWatch-reporter-1-thread-1] CloudWatchReporter:236 -
> Error reporting metrics to CloudWatch. The data in this CloudWatch request
> may have been discarded, and not made it to CloudWatch.
> java.util.concurrent.ExecutionException:
> {color:#ff0000}com.amazonaws.services.cloudwatch.model.MissingRequiredParameterException:
> The parameter MetricData.member.1.Dimensions.member.1.Value is required.
> (Service: AmazonCloudWatch; Status Code: 400; Error Code: MissingParameter;
> Request ID: xxx-xxxx){color}
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:206)
> at
> org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:234)
> at
> org.apache.hudi.aws.cloudwatch.CloudWatchReporter.report(CloudWatchReporter.java:212)
> at
> org.apache.hudi.com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:237)
> at
> org.apache.hudi.com.codahale.metrics.ScheduledReporter.lambda$start$0(ScheduledReporter.java:177)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by:
> com.amazonaws.services.cloudwatch.model.MissingRequiredParameterException:
> The parameter MetricData.member.1.Dimensions.member.1.Value is required.
> (Service: AmazonCloudWatch; Status Code: 400; Error Code: MissingParameter;
> Request ID: 3eb96da0-d4bc-4687-81a5-0d65c453559a)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1742)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1371)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1347)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1127)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:784)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:752)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
> at
> com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.doInvoke(AmazonCloudWatchClient.java:2446)
> at
> com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:2413)
> at
> com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:2402)
> at
> com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.executePutMetricData(AmazonCloudWatchClient.java:2170)
> at
> com.amazonaws.services.cloudwatch.AmazonCloudWatchAsyncClient$26.call(AmazonCloudWatchAsyncClient.java:1170)
> at
> com.amazonaws.services.cloudwatch.AmazonCloudWatchAsyncClient$26.call(AmazonCloudWatchAsyncClient.java:1164)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 common frames omitted
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
