[
https://issues.apache.org/jira/browse/BEAM-9269?focusedWorklogId=385087&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-385087
]
ASF GitHub Bot logged work on BEAM-9269:
----------------------------------------
Author: ASF GitHub Bot
Created on: 11/Feb/20 11:18
Start Date: 11/Feb/20 11:18
Worklog Time Spent: 10m
Work Description: nielm commented on pull request #10752: [BEAM-9269] Add
commit deadline for Spanner writes.
URL: https://github.com/apache/beam/pull/10752#discussion_r377574096
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
##########
@@ -1248,20 +1307,44 @@ public void processElement(ProcessContext c) {
}
}
- private static class WriteToSpannerFn extends DoFn<Iterable<MutationGroup>,
Void> {
+ @VisibleForTesting
+ static class WriteToSpannerFn extends DoFn<Iterable<MutationGroup>, Void> {
private transient SpannerAccessor spannerAccessor;
private final SpannerConfig spannerConfig;
private final FailureMode failureMode;
- private final Counter mutationGroupBatchesCounter =
- Metrics.counter(WriteGrouped.class, "mutation_group_batches");
- private final Counter mutationGroupWriteSuccessCounter =
+
+ @VisibleForTesting static Sleeper sleeper = Sleeper.DEFAULT;
+
+ private final Counter mutationGroupBatchesReceived =
+ Metrics.counter(WriteGrouped.class, "mutation_group_batches_received");
+ private final Counter mutationGroupBatchesWriteSuccess =
+ Metrics.counter(WriteGrouped.class,
"mutation_group_batches_write_success");
+ private final Counter mutationGroupBatchesWriteFail =
+ Metrics.counter(WriteGrouped.class,
"mutation_group_batches_write_fail");
+
+ private final Counter mutationGroupsReceived =
+ Metrics.counter(WriteGrouped.class, "mutation_groups_received");
+ private final Counter mutationGroupsWriteSuccess =
Metrics.counter(WriteGrouped.class, "mutation_groups_write_success");
- private final Counter mutationGroupWriteFailCounter =
+ private final Counter mutationGroupsWriteFail =
Metrics.counter(WriteGrouped.class, "mutation_groups_write_fail");
+ private final Counter spannerWriteSuccess =
+ Metrics.counter(WriteGrouped.class, "spanner_write_success");
+ private final Counter spannerWriteFail =
+ Metrics.counter(WriteGrouped.class, "spanner_write_fail");
+ private final Counter spannerWriteTotalLatency =
Review comment:
Its not possible - the only things we can export are counters and gauges.
User will have to manually calculate average latency.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 385087)
Time Spent: 2h (was: 1h 50m)
> Set shorter Commit Deadline and handle with backoff/retry
> ---------------------------------------------------------
>
> Key: BEAM-9269
> URL: https://issues.apache.org/jira/browse/BEAM-9269
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
> Affects Versions: 2.16.0, 2.17.0, 2.18.0, 2.19.0
> Reporter: Niel Markwick
> Assignee: Niel Markwick
> Priority: Major
> Labels: google-cloud-spanner
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Default commit deadline in Spanner is 1hr, which can lead to a variety of
> issues including database overload and session expiry.
> Shorter deadline should be set with backoff/retry when deadline expires, so
> that the Spanner database does not become overloaded.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
