[
https://issues.apache.org/jira/browse/BEAM-6714?focusedWorklogId=202289&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-202289
]
ASF GitHub Bot logged work on BEAM-6714:
----------------------------------------
Author: ASF GitHub Bot
Created on: 21/Feb/19 23:34
Start Date: 21/Feb/19 23:34
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #7907: [BEAM-6714]
Move runner-agnostic code out of FlinkJobServerDriver
URL: https://github.com/apache/beam/pull/7907#discussion_r259160959
##########
File path:
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobServerDriver.java
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.jobsubmission;
+
+import java.io.IOException;
+import java.nio.file.Paths;
+import java.util.function.Function;
+import org.apache.beam.model.pipeline.v1.Endpoints;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.ServerFactory;
+import
org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactStagingService;
+import
org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.kohsuke.args4j.Option;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Shared code for starting and serving an {@link InMemoryJobService}. */
+public abstract class JobServerDriver implements Runnable {
+
+ protected abstract JobInvoker createJobInvoker();
+
+ private static final Logger LOG =
LoggerFactory.getLogger(JobServerDriver.class);
+
+ @VisibleForTesting public ServerConfiguration configuration;
+
+ private final ServerFactory jobServerFactory;
+ private final ServerFactory artifactServerFactory;
+ private volatile GrpcFnServer<InMemoryJobService> jobServer;
+ private volatile GrpcFnServer<BeamFileSystemArtifactStagingService>
artifactStagingServer;
+
+ /** Configuration for the jobServer. */
+ public static class ServerConfiguration {
+ @Option(name = "--job-host", usage = "The job server host name")
+ private String host = "localhost";
+
+ @Option(
+ name = "--job-port",
+ usage = "The job service port. 0 to use a dynamic port. (Default:
8099)")
+ private int port = 8099;
+
+ @Option(
+ name = "--artifact-port",
+ usage = "The artifact service port. 0 to use a dynamic port. (Default:
8098)")
+ private int artifactPort = 8098;
+
+ @Option(name = "--artifacts-dir", usage = "The location to store staged
artifact files")
+ private String artifactStagingPath =
+ Paths.get(System.getProperty("java.io.tmpdir"),
"beam-artifact-staging").toString();
+
+ @Option(
+ name = "--clean-artifacts-per-job",
+ usage = "When true, remove each job's staged artifacts when it
completes")
+ private boolean cleanArtifactsPerJob = false;
+
+ @Option(
+ name = "--sdk-worker-parallelism",
+ usage = "Default parallelism for SDK worker processes (see portable
pipeline options)")
+ private Long sdkWorkerParallelism = 1L;
+
+ @Option(
+ name = "--artifact-service-enabled",
+ usage = "When false, the artifact staging service will not be started
and set to null")
+ private boolean artifactServiceEnabled = true;
Review comment:
Lets add test for `artifactServiceEnabled = false`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 202289)
Time Spent: 4h 20m (was: 4h 10m)
> Move runner-agnostic code out of FlinkJobServerDriver
> -----------------------------------------------------
>
> Key: BEAM-6714
> URL: https://issues.apache.org/jira/browse/BEAM-6714
> Project: Beam
> Issue Type: Task
> Components: runner-flink, runner-samza, runner-spark
> Reporter: Kyle Weaver
> Assignee: Kyle Weaver
> Priority: Major
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> [FlinkJobServerDriver|https://github.com/apache/beam/blob/master/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java]
> contains quite a bit of code that is not actually specific to the Flink
> runner. This runner-agnostic code should be shared so that other runners (ie
> Spark) developing portability can leverage it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
