[
https://issues.apache.org/jira/browse/BEAM-5906?focusedWorklogId=163408&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-163408
]
ASF GitHub Bot logged work on BEAM-5906:
----------------------------------------
Author: ASF GitHub Bot
Created on: 07/Nov/18 10:26
Start Date: 07/Nov/18 10:26
Worklog Time Spent: 10m
Work Description: echauchot commented on a change in pull request #6886:
[BEAM-5906] Use dedicated BigQuery client for publishing Nexmark results
URL: https://github.com/apache/beam/pull/6886#discussion_r231434528
##########
File path:
sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkUtils.java
##########
@@ -145,53 +145,51 @@
QUERY_RUNNER_AND_MODE
}
+ /** Return a query name with query language (if applicable). */
+ static String fullQueryName(String queryLanguage, String query) {
+ return queryLanguage != null ? query + "_" + queryLanguage : query;
+ }
+
/** Return a BigQuery table spec. */
static String tableSpec(NexmarkOptions options, String queryName, long now,
String version) {
+ return String.format(
+ "%s:%s.%s",
+ options.getProject(),
+ options.getBigQueryDataset(),
+ tableName(options, queryName, now, version));
+ }
+
+ /** Return a BigQuery table name. */
+ static String tableName(NexmarkOptions options, String queryName, long now,
String version) {
String baseTableName = options.getBigQueryTable();
if (Strings.isNullOrEmpty(baseTableName)) {
throw new RuntimeException("Missing --bigQueryTable");
}
+
switch (options.getResourceNameMode()) {
case VERBATIM:
- return String.format(
- "%s:%s.%s_%s",
- options.getProject(), options.getBigQueryDataset(), baseTableName,
version);
+ return String.format("%s_%s", baseTableName, version);
case QUERY:
- return String.format(
- "%s:%s.%s_%s_%s",
- options.getProject(), options.getBigQueryDataset(), baseTableName,
queryName, version);
+ return String.format("%s_%s_%s", baseTableName, queryName, version);
case QUERY_AND_SALT:
- return String.format(
- "%s:%s.%s_%s_%s_%d",
- options.getProject(),
- options.getBigQueryDataset(),
- baseTableName,
- queryName,
- version,
- now);
+ return String.format("%s_%s_%s_%d", baseTableName, queryName, version,
now);
case QUERY_RUNNER_AND_MODE:
- return (version != null)
- ? String.format(
- "%s:%s.%s_%s_%s_%s_%s",
- options.getProject(),
- options.getBigQueryDataset(),
- baseTableName,
- queryName,
- options.getRunner().getSimpleName(),
- options.isStreaming() ? "streaming" : "batch",
- version)
- : String.format(
- "%s:%s.%s_%s_%s_%s",
- options.getProject(),
- options.getBigQueryDataset(),
- baseTableName,
- queryName,
- options.getRunner().getSimpleName(),
- options.isStreaming() ? "streaming" : "batch");
+ String runnerName = options.getRunner().getSimpleName();
+ boolean isStreaming = options.isStreaming();
+
+ String tableName =
+ String.format(
+ "%s_%s_%s_%s", baseTableName, queryName, runnerName,
processingMode(isStreaming));
+
+ return version != null ? String.format("%s_%s", tableName, version) :
tableName;
}
throw new RuntimeException("Unrecognized enum " +
options.getResourceNameMode());
}
+ private static String processingMode(boolean isStreaming) {
Review comment:
inline this method in a variable because it is called only once
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 163408)
Time Spent: 1h 20m (was: 1h 10m)
> Remove pipeline for publishing nexmark results to bigQuery and publish using
> BigQuery API only
> ----------------------------------------------------------------------------------------------
>
> Key: BEAM-5906
> URL: https://issues.apache.org/jira/browse/BEAM-5906
> Project: Beam
> Issue Type: Improvement
> Components: examples-nexmark
> Reporter: Lukasz Gajowy
> Assignee: Lukasz Gajowy
> Priority: Minor
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> There's no need to start a separate pipeline for uploading metrics results
> from Nexmark suites to BigQuery. We can use an API designed for that and
> place it in test-utils. Thanks to that:
> - it won't start a separate pipeline every time it publishes results
> - other suites will be able to use that code
> - We will not face problems like special long to int conversion due to
> problems in BigQueryIO (eg. BEAM-4734) because we will use a thin API instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
