This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 78dfcba Publishing website 2019/12/20 12:02:19 at commit 0384d16
78dfcba is described below
commit 78dfcbac53dbe75109b1a53caa4213efe29a3503
Author: jenkins <bui...@apache.org>
AuthorDate: Fri Dec 20 12:02:19 2019 +0000
Publishing website 2019/12/20 12:02:19 at commit 0384d16
---
.../documentation/runners/flink/index.html | 310 ++++++++++++---------
1 file changed, 174 insertions(+), 136 deletions(-)
diff --git a/website/generated-content/documentation/runners/flink/index.html
b/website/generated-content/documentation/runners/flink/index.html
index f27a1df..eab90fe 100644
--- a/website/generated-content/documentation/runners/flink/index.html
+++ b/website/generated-content/documentation/runners/flink/index.html
@@ -578,108 +578,99 @@ Many sources like <code
class="highlighter-rouge">PubSubIO</code> rely on their
<!-- Java Options -->
<div class="language-java">
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<!--
+This is an auto-generated file.
+Use generatePipelineOptionsTableJava and generatePipelineOptionsTablePython
respectively
+which should be called before running the tests.
+-->
<table class="table table-bordered">
<tr>
- <th>Field</th>
- <th>Description</th>
- <th>Default Value</th>
-</tr>
-<tr>
- <td><code>runner</code></td>
- <td>The pipeline runner to use. This option allows you to determine the
pipeline runner at runtime.</td>
- <td>Set to <code>FlinkRunner</code> to run using Flink.</td>
-</tr>
-<tr>
- <td><code>streaming</code></td>
- <td>Whether streaming mode is enabled or disabled; <code>true</code> if
enabled. Set to <code>true</code> if running pipelines with unbounded
<code>PCollection</code>s.</td>
- <td><code>false</code></td>
-</tr>
-<tr>
- <td><code>flinkMaster</code></td>
- <td>The url of the Flink JobManager on which to execute pipelines. This can
either be the address of a cluster JobManager, in the form
<code>"host:port"</code> or one of the special Strings <code>"[local]"</code>
or <code>"[auto]"</code>. <code>"[local]"</code> will start a local Flink
Cluster in the JVM while <code>"[auto]"</code> will let the system decide where
to execute the pipeline based on the environment.</td>
- <td><code>[auto]</code></td>
+ <td><code>allowNonRestoredState</code></td>
+ <td>Flag indicating whether non restored state is allowed if the savepoint
contains state for an operator that is no longer part of the pipeline.</td>
+ <td>Default: <code>false</code></td>
</tr>
<tr>
- <td><code>filesToStage</code></td>
- <td>Jar Files to send to all workers and put on the classpath. Here you have
to put the fat jar that contains your program along with all dependencies.</td>
- <td>empty</td>
+ <td><code>autoBalanceWriteFilesShardingEnabled</code></td>
+ <td>Flag indicating whether auto-balance sharding for WriteFiles transform
should be enabled. This might prove useful in streaming use-case, where
pipeline needs to write quite many events into files, typically divided into N
shards. Default behavior on Flink would be, that some workers will receive more
shards to take care of than others. This cause workers to go out of balance in
terms of processing backlog and memory usage. Enabling this feature will make
shards to be spread evenly [...]
+ <td>Default: <code>false</code></td>
</tr>
<tr>
- <td><code>parallelism</code></td>
- <td>The degree of parallelism to be used when distributing operations onto
workers.</td>
- <td>For local execution: <code>Number of available CPU cores</code>
- For remote execution: <code>Default parallelism configuerd at
remote cluster</code>
- Otherwise: <code>1</code>
- </td>
+ <td><code>autoWatermarkInterval</code></td>
+ <td>The interval in milliseconds for automatic watermark emission.</td>
+ <td></td>
</tr>
<tr>
- <td><code>maxParallelism</code></td>
- <td>The pipeline wide maximum degree of parallelism to be used. The maximum
parallelism specifies the upper limit for dynamic scaling and the number of key
groups used for partitioned state.</td>
- <td><code>-1L</code>, meaning same as the parallelism</td>
+ <td><code>checkpointTimeoutMillis</code></td>
+ <td>The maximum time in milliseconds that a checkpoint may take before being
discarded.</td>
+ <td>Default: <code>-1</code></td>
</tr>
<tr>
<td><code>checkpointingInterval</code></td>
- <td>The interval between consecutive checkpoints (i.e. snapshots of the
current pipeline state used for fault tolerance).</td>
- <td><code>-1L</code>, i.e. disabled</td>
+ <td>The interval in milliseconds at which to trigger checkpoints of the
running pipeline. Default: No checkpointing.</td>
+ <td>Default: <code>-1</code></td>
</tr>
<tr>
- <td><code>checkpointMode</code></td>
+ <td><code>checkpointingMode</code></td>
<td>The checkpointing mode that defines consistency guarantee.</td>
- <td><code>EXACTLY_ONCE</code></td>
-</tr>
-<tr>
- <td><code>checkpointTimeoutMillis</code></td>
- <td>The maximum time in milliseconds that a checkpoint may take before being
discarded</td>
- <td><code>-1</code>, the cluster default</td>
+ <td>Default: <code>EXACTLY_ONCE</code></td>
</tr>
<tr>
- <td><code>minPauseBetweenCheckpoints</code></td>
- <td>The minimal pause in milliseconds before the next checkpoint is
triggered.</td>
- <td><code>-1</code>, the cluster default</td>
+ <td><code>disableMetricAccumulator</code></td>
+ <td>By default, uses Flink accumulators to store the metrics which allows to
query metrics from the PipelineResult. If set to true, metrics will still be
reported but can't be queried via PipelineResult. This saves network and
memory.</td>
+ <td>Default: <code>false</code></td>
</tr>
<tr>
- <td><code>failOnCheckpointingErrors</code></td>
- <td>
- Sets the expected behaviour for tasks in case that they encounter an error
in their
- checkpointing procedure. If this is set to true, the task will
fail on checkpointing error.
- If this is set to false, the task will only decline a the
checkpoint and continue running.
- </td>
- <td><code>-1</code>, the cluster default</td>
+ <td><code>disableMetrics</code></td>
+ <td>Disable Beam metrics in Flink Runner</td>
+ <td>Default: <code>false</code></td>
</tr>
<tr>
- <td><code>numberOfExecutionRetries</code></td>
- <td>Sets the number of times that failed tasks are re-executed. A value of
<code>0</code> effectively disables fault tolerance. A value of <code>-1</code>
indicates that the system default value (as defined in the configuration)
should be used.</td>
- <td><code>-1</code></td>
+ <td><code>executionModeForBatch</code></td>
+ <td>Flink mode for data exchange of batch pipelines. Reference {@link
org.apache.flink.api.common.ExecutionMode}. Set this to BATCH_FORCED if
pipelines get blocked, see
https://issues.apache.org/jira/browse/FLINK-10672</td>
+ <td>Default: <code>PIPELINED</code></td>
</tr>
<tr>
<td><code>executionRetryDelay</code></td>
- <td>Sets the delay between executions. A value of <code>-1</code> indicates
that the default value should be used.</td>
- <td><code>-1</code></td>
+ <td>Sets the delay in milliseconds between executions. A value of {@code -1}
indicates that the default value should be used.</td>
+ <td>Default: <code>-1</code></td>
</tr>
<tr>
- <td><code>objectReuse</code></td>
- <td>Sets the behavior of reusing objects.</td>
- <td><code>false</code>, no Object reuse</td>
+ <td><code>externalizedCheckpointsEnabled</code></td>
+ <td>Enables or disables externalized checkpoints. Works in conjunction with
CheckpointingInterval</td>
+ <td>Default: <code>false</code></td>
</tr>
<tr>
- <td><code>stateBackend</code></td>
- <td>Sets the state backend to use in streaming mode. The default is to read
this setting from the Flink config.</td>
- <td><code>empty</code>, i.e. read from Flink config</td>
+ <td><code>failOnCheckpointingErrors</code></td>
+ <td>Sets the expected behaviour for tasks in case that they encounter an
error in their checkpointing procedure. If this is set to true, the task will
fail on checkpointing error. If this is set to false, the task will only
decline a the checkpoint and continue running. </td>
+ <td>Default: <code>true</code></td>
</tr>
<tr>
- <td><code>enableMetrics</code></td>
- <td>Enable/disable Beam metrics in Flink Runner</td>
- <td>Default: <code>true</code></td>
+ <td><code>filesToStage</code></td>
+ <td>Jar-Files to send to all workers and put on the classpath. The default
value is all files from the classpath.</td>
+ <td></td>
</tr>
<tr>
- <td><code>externalizedCheckpointsEnabled</code></td>
- <td>Enables or disables externalized checkpoints. Works in conjunction with
CheckpointingInterval</td>
- <td>Default: <code>false</code></td>
+ <td><code>flinkMaster</code></td>
+ <td>Address of the Flink Master where the Pipeline should be executed. Can
either be of the form "host:port" or one of the special values [local],
[collection] or [auto].</td>
+ <td>Default: <code>[auto]</code></td>
</tr>
<tr>
- <td><code>retainExternalizedCheckpointsOnCancellation</code></td>
- <td>Sets the behavior of externalized checkpoints on cancellation.</td>
- <td>Default: <code>false</code></td>
+ <td><code>latencyTrackingInterval</code></td>
+ <td>Interval in milliseconds for sending latency tracking marks from the
sources to the sinks. Interval value <= 0 disables the feature.</td>
+ <td>Default: <code>0</code></td>
</tr>
<tr>
<td><code>maxBundleSize</code></td>
@@ -692,58 +683,92 @@ Many sources like <code
class="highlighter-rouge">PubSubIO</code> rely on their
<td>Default: <code>1000</code></td>
</tr>
<tr>
- <td><code>shutdownSourcesOnFinalWatermark</code></td>
- <td>If set, shutdown sources when their watermark reaches +Inf.</td>
- <td>Default: <code>false</code></td>
+ <td><code>maxParallelism</code></td>
+ <td>The pipeline wide maximum degree of parallelism to be used. The maximum
parallelism specifies the upper limit for dynamic scaling and the number of key
groups used for partitioned state.</td>
+ <td>Default: <code>-1</code></td>
</tr>
<tr>
- <td><code>latencyTrackingInterval</code></td>
- <td>Interval in milliseconds for sending latency tracking marks from the
sources to the sinks. Interval value <= 0 disables the feature.</td>
- <td>Default: <code>0</code></td>
+ <td><code>minPauseBetweenCheckpoints</code></td>
+ <td>The minimal pause in milliseconds before the next checkpoint is
triggered.</td>
+ <td>Default: <code>-1</code></td>
</tr>
<tr>
- <td><code>autoWatermarkInterval</code></td>
- <td>The interval in milliseconds for automatic watermark emission.</td>
+ <td><code>numberOfExecutionRetries</code></td>
+ <td>Sets the number of times that failed tasks are re-executed. A value of
zero effectively disables fault tolerance. A value of -1 indicates that the
system default value (as defined in the configuration) should be used.</td>
+ <td>Default: <code>-1</code></td>
</tr>
<tr>
- <td><code>executionModeForBatch</code></td>
- <td>Flink mode for data exchange of batch pipelines. Reference {@link
org.apache.flink.api.common.ExecutionMode}. Set this to BATCH_FORCED if
pipelines get blocked, see
https://issues.apache.org/jira/browse/FLINK-10672</td>
- <td>Default: <code>PIPELINED</code></td>
+ <td><code>objectReuse</code></td>
+ <td>Sets the behavior of reusing objects.</td>
+ <td>Default: <code>false</code></td>
+</tr>
+<tr>
+ <td><code>parallelism</code></td>
+ <td>The degree of parallelism to be used when distributing operations onto
workers. If the parallelism is not set, the configured Flink default is used,
or 1 if none can be found.</td>
+ <td>Default: <code>-1</code></td>
+</tr>
+<tr>
+ <td><code>retainExternalizedCheckpointsOnCancellation</code></td>
+ <td>Sets the behavior of externalized checkpoints on cancellation.</td>
+ <td>Default: <code>false</code></td>
</tr>
<tr>
<td><code>savepointPath</code></td>
<td>Savepoint restore path. If specified, restores the streaming pipeline
from the provided path.</td>
- <td>Default: None</td>
+ <td></td>
</tr>
<tr>
- <td><code>allowNonRestoredState</code></td>
- <td>Flag indicating whether non restored state is allowed if the savepoint
contains state for an operator that is no longer part of the pipeline.</td>
+ <td><code>shutdownSourcesOnFinalWatermark</code></td>
+ <td>If set, shutdown sources when their watermark reaches +Inf.</td>
<td>Default: <code>false</code></td>
</tr>
+<tr>
+ <td><code>stateBackendFactory</code></td>
+ <td>Sets the state backend factory to use in streaming mode. Defaults to the
flink cluster's state.backend configuration.</td>
+ <td></td>
+</tr>
</table>
-</div>
+</div>
<!-- Python Options -->
<div class="language-py">
-<table class="table table-bordered">
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<!--
+This is an auto-generated file.
+Use generatePipelineOptionsTableJava and generatePipelineOptionsTablePython
respectively
+which should be called before running the tests.
+-->
+<table class="table table-bordered">
<tr>
- <td><code>files_to_stage</code></td>
- <td>Jar-Files to send to all workers and put on the classpath. The default
value is all files from the classpath.</td>
+ <td><code>allow_non_restored_state</code></td>
+ <td>Flag indicating whether non restored state is allowed if the savepoint
contains state for an operator that is no longer part of the pipeline.</td>
+ <td>Default: <code>false</code></td>
</tr>
<tr>
- <td><code>flink_master</code></td>
- <td>Address of the Flink Master where the Pipeline should be executed. Can
either be of the form "host:port" or one of the special values [local],
[collection] or [auto].</td>
- <td>Default: <code>[auto]</code></td>
+ <td><code>auto_balance_write_files_sharding_enabled</code></td>
+ <td>Flag indicating whether auto-balance sharding for WriteFiles transform
should be enabled. This might prove useful in streaming use-case, where
pipeline needs to write quite many events into files, typically divided into N
shards. Default behavior on Flink would be, that some workers will receive more
shards to take care of than others. This cause workers to go out of balance in
terms of processing backlog and memory usage. Enabling this feature will make
shards to be spread evenly [...]
+ <td>Default: <code>false</code></td>
</tr>
<tr>
- <td><code>parallelism</code></td>
- <td>The degree of parallelism to be used when distributing operations onto
workers. If the parallelism is not set, the configured Flink default is used,
or 1 if none can be found.</td>
- <td>Default: <code>-1</code></td>
+ <td><code>auto_watermark_interval</code></td>
+ <td>The interval in milliseconds for automatic watermark emission.</td>
+ <td></td>
</tr>
<tr>
- <td><code>max_parallelism</code></td>
- <td>The pipeline wide maximum degree of parallelism to be used. The maximum
parallelism specifies the upper limit for dynamic scaling and the number of key
groups used for partitioned state.</td>
+ <td><code>checkpoint_timeout_millis</code></td>
+ <td>The maximum time in milliseconds that a checkpoint may take before being
discarded.</td>
<td>Default: <code>-1</code></td>
</tr>
<tr>
@@ -757,24 +782,19 @@ Many sources like <code
class="highlighter-rouge">PubSubIO</code> rely on their
<td>Default: <code>EXACTLY_ONCE</code></td>
</tr>
<tr>
- <td><code>checkpoint_timeout_millis</code></td>
- <td>The maximum time in milliseconds that a checkpoint may take before being
discarded.</td>
- <td>Default: <code>-1</code></td>
-</tr>
-<tr>
- <td><code>min_pause_between_checkpoints</code></td>
- <td>The minimal pause in milliseconds before the next checkpoint is
triggered.</td>
- <td>Default: <code>-1</code></td>
+ <td><code>disable_metric_accumulator</code></td>
+ <td>By default, uses Flink accumulators to store the metrics which allows to
query metrics from the PipelineResult. If set to true, metrics will still be
reported but can't be queried via PipelineResult. This saves network and
memory.</td>
+ <td>Default: <code>false</code></td>
</tr>
<tr>
- <td><code>fail_on_checkpointing_errors</code></td>
- <td>Sets the expected behaviour for tasks in case that they encounter an
error in their checkpointing procedure. If this is set to true, the task will
fail on checkpointing error. If this is set to false, the task will only
decline a the checkpoint and continue running. </td>
- <td>Default: <code>true</code></td>
+ <td><code>disable_metrics</code></td>
+ <td>Disable Beam metrics in Flink Runner</td>
+ <td>Default: <code>false</code></td>
</tr>
<tr>
- <td><code>number_of_execution_retries</code></td>
- <td>Sets the number of times that failed tasks are re-executed. A value of
zero effectively disables fault tolerance. A value of -1 indicates that the
system default value (as defined in the configuration) should be used.</td>
- <td>Default: <code>-1</code></td>
+ <td><code>execution_mode_for_batch</code></td>
+ <td>Flink mode for data exchange of batch pipelines. Reference {@link
org.apache.flink.api.common.ExecutionMode}. Set this to BATCH_FORCED if
pipelines get blocked, see
https://issues.apache.org/jira/browse/FLINK-10672</td>
+ <td>Default: <code>PIPELINED</code></td>
</tr>
<tr>
<td><code>execution_retry_delay</code></td>
@@ -782,28 +802,29 @@ Many sources like <code
class="highlighter-rouge">PubSubIO</code> rely on their
<td>Default: <code>-1</code></td>
</tr>
<tr>
- <td><code>object_reuse</code></td>
- <td>Sets the behavior of reusing objects.</td>
+ <td><code>externalized_checkpoints_enabled</code></td>
+ <td>Enables or disables externalized checkpoints. Works in conjunction with
CheckpointingInterval</td>
<td>Default: <code>false</code></td>
</tr>
<tr>
- <td><code>state_backend</code></td>
- <td>Sets the state backend to use in streaming mode. Otherwise the default
is read from the Flink config.</td>
+ <td><code>fail_on_checkpointing_errors</code></td>
+ <td>Sets the expected behaviour for tasks in case that they encounter an
error in their checkpointing procedure. If this is set to true, the task will
fail on checkpointing error. If this is set to false, the task will only
decline a the checkpoint and continue running. </td>
+ <td>Default: <code>true</code></td>
</tr>
<tr>
- <td><code>enable_metrics</code></td>
- <td>Enable/disable Beam metrics in Flink Runner</td>
- <td>Default: <code>true</code></td>
+ <td><code>files_to_stage</code></td>
+ <td>Jar-Files to send to all workers and put on the classpath. The default
value is all files from the classpath.</td>
+ <td></td>
</tr>
<tr>
- <td><code>externalized_checkpoints_enabled</code></td>
- <td>Enables or disables externalized checkpoints. Works in conjunction with
CheckpointingInterval</td>
- <td>Default: <code>false</code></td>
+ <td><code>flink_master</code></td>
+ <td>Address of the Flink Master where the Pipeline should be executed. Can
either be of the form "host:port" or one of the special values [local],
[collection] or [auto].</td>
+ <td>Default: <code>[auto]</code></td>
</tr>
<tr>
- <td><code>retain_externalized_checkpoints_on_cancellation</code></td>
- <td>Sets the behavior of externalized checkpoints on cancellation.</td>
- <td>Default: <code>false</code></td>
+ <td><code>latency_tracking_interval</code></td>
+ <td>Interval in milliseconds for sending latency tracking marks from the
sources to the sinks. Interval value <= 0 disables the feature.</td>
+ <td>Default: <code>0</code></td>
</tr>
<tr>
<td><code>max_bundle_size</code></td>
@@ -816,35 +837,52 @@ Many sources like <code
class="highlighter-rouge">PubSubIO</code> rely on their
<td>Default: <code>1000</code></td>
</tr>
<tr>
- <td><code>shutdown_sources_on_final_watermark</code></td>
- <td>If set, shutdown sources when their watermark reaches +Inf.</td>
- <td>Default: <code>false</code></td>
+ <td><code>max_parallelism</code></td>
+ <td>The pipeline wide maximum degree of parallelism to be used. The maximum
parallelism specifies the upper limit for dynamic scaling and the number of key
groups used for partitioned state.</td>
+ <td>Default: <code>-1</code></td>
</tr>
<tr>
- <td><code>latency_tracking_interval</code></td>
- <td>Interval in milliseconds for sending latency tracking marks from the
sources to the sinks. Interval value <= 0 disables the feature.</td>
- <td>Default: <code>0</code></td>
+ <td><code>min_pause_between_checkpoints</code></td>
+ <td>The minimal pause in milliseconds before the next checkpoint is
triggered.</td>
+ <td>Default: <code>-1</code></td>
</tr>
<tr>
- <td><code>auto_watermark_interval</code></td>
- <td>The interval in milliseconds for automatic watermark emission.</td>
+ <td><code>number_of_execution_retries</code></td>
+ <td>Sets the number of times that failed tasks are re-executed. A value of
zero effectively disables fault tolerance. A value of -1 indicates that the
system default value (as defined in the configuration) should be used.</td>
+ <td>Default: <code>-1</code></td>
</tr>
<tr>
- <td><code>execution_mode_for_batch</code></td>
- <td>Flink mode for data exchange of batch pipelines. Reference {@link
org.apache.flink.api.common.ExecutionMode}. Set this to BATCH_FORCED if
pipelines get blocked, see
https://issues.apache.org/jira/browse/FLINK-10672</td>
- <td>Default: <code>PIPELINED</code></td>
+ <td><code>object_reuse</code></td>
+ <td>Sets the behavior of reusing objects.</td>
+ <td>Default: <code>false</code></td>
+</tr>
+<tr>
+ <td><code>parallelism</code></td>
+ <td>The degree of parallelism to be used when distributing operations onto
workers. If the parallelism is not set, the configured Flink default is used,
or 1 if none can be found.</td>
+ <td>Default: <code>-1</code></td>
+</tr>
+<tr>
+ <td><code>retain_externalized_checkpoints_on_cancellation</code></td>
+ <td>Sets the behavior of externalized checkpoints on cancellation.</td>
+ <td>Default: <code>false</code></td>
</tr>
<tr>
<td><code>savepoint_path</code></td>
<td>Savepoint restore path. If specified, restores the streaming pipeline
from the provided path.</td>
+ <td></td>
</tr>
<tr>
- <td><code>allow_non_restored_state</code></td>
- <td>Flag indicating whether non restored state is allowed if the savepoint
contains state for an operator that is no longer part of the pipeline.</td>
+ <td><code>shutdown_sources_on_final_watermark</code></td>
+ <td>If set, shutdown sources when their watermark reaches +Inf.</td>
<td>Default: <code>false</code></td>
</tr>
-
+<tr>
+ <td><code>state_backend_factory</code></td>
+ <td>Sets the state backend factory to use in streaming mode. Defaults to the
flink cluster's state.backend configuration.</td>
+ <td></td>
+</tr>
</table>
+
</div>
<h2 id="capability">Capability</h2>
