[
https://issues.apache.org/jira/browse/BEAM-9269?focusedWorklogId=384878&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-384878
]
ASF GitHub Bot logged work on BEAM-9269:
----------------------------------------
Author: ASF GitHub Bot
Created on: 11/Feb/20 00:36
Start Date: 11/Feb/20 00:36
Worklog Time Spent: 10m
Work Description: allenpradeep commented on pull request #10752:
[BEAM-9269] Add commit deadline for Spanner writes.
URL: https://github.com/apache/beam/pull/10752#discussion_r377397609
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
##########
@@ -1248,20 +1307,44 @@ public void processElement(ProcessContext c) {
}
}
- private static class WriteToSpannerFn extends DoFn<Iterable<MutationGroup>,
Void> {
+ @VisibleForTesting
+ static class WriteToSpannerFn extends DoFn<Iterable<MutationGroup>, Void> {
private transient SpannerAccessor spannerAccessor;
private final SpannerConfig spannerConfig;
private final FailureMode failureMode;
- private final Counter mutationGroupBatchesCounter =
- Metrics.counter(WriteGrouped.class, "mutation_group_batches");
- private final Counter mutationGroupWriteSuccessCounter =
+
+ @VisibleForTesting static Sleeper sleeper = Sleeper.DEFAULT;
+
+ private final Counter mutationGroupBatchesReceived =
+ Metrics.counter(WriteGrouped.class, "mutation_group_batches_received");
+ private final Counter mutationGroupBatchesWriteSuccess =
+ Metrics.counter(WriteGrouped.class,
"mutation_group_batches_write_success");
+ private final Counter mutationGroupBatchesWriteFail =
+ Metrics.counter(WriteGrouped.class,
"mutation_group_batches_write_fail");
+
+ private final Counter mutationGroupsReceived =
+ Metrics.counter(WriteGrouped.class, "mutation_groups_received");
+ private final Counter mutationGroupsWriteSuccess =
Metrics.counter(WriteGrouped.class, "mutation_groups_write_success");
- private final Counter mutationGroupWriteFailCounter =
+ private final Counter mutationGroupsWriteFail =
Metrics.counter(WriteGrouped.class, "mutation_groups_write_fail");
+ private final Counter spannerWriteSuccess =
+ Metrics.counter(WriteGrouped.class, "spanner_write_success");
+ private final Counter spannerWriteFail =
+ Metrics.counter(WriteGrouped.class, "spanner_write_fail");
+ private final Counter spannerWriteTotalLatency =
Review comment:
Would an average writeLatency(if its possible to calculate) be more
actionable to the user rather than a total write latency? In the sense, if the
write latency is high then the user can reduce the workers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 384878)
> Set shorter Commit Deadline and handle with backoff/retry
> ---------------------------------------------------------
>
> Key: BEAM-9269
> URL: https://issues.apache.org/jira/browse/BEAM-9269
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
> Affects Versions: 2.16.0, 2.17.0, 2.18.0, 2.19.0
> Reporter: Niel Markwick
> Assignee: Niel Markwick
> Priority: Major
> Labels: google-cloud-spanner
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Default commit deadline in Spanner is 1hr, which can lead to a variety of
> issues including database overload and session expiry.
> Shorter deadline should be set with backoff/retry when deadline expires, so
> that the Spanner database does not become overloaded.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
