This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push: new 78dfcba Publishing website 2019/12/20 12:02:19 at commit 0384d16 78dfcba is described below commit 78dfcbac53dbe75109b1a53caa4213efe29a3503 Author: jenkins <bui...@apache.org> AuthorDate: Fri Dec 20 12:02:19 2019 +0000 Publishing website 2019/12/20 12:02:19 at commit 0384d16 --- .../documentation/runners/flink/index.html | 310 ++++++++++++--------- 1 file changed, 174 insertions(+), 136 deletions(-) diff --git a/website/generated-content/documentation/runners/flink/index.html b/website/generated-content/documentation/runners/flink/index.html index f27a1df..eab90fe 100644 --- a/website/generated-content/documentation/runners/flink/index.html +++ b/website/generated-content/documentation/runners/flink/index.html @@ -578,108 +578,99 @@ Many sources like <code class="highlighter-rouge">PubSubIO</code> rely on their <!-- Java Options --> <div class="language-java"> +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +<!-- +This is an auto-generated file. +Use generatePipelineOptionsTableJava and generatePipelineOptionsTablePython respectively +which should be called before running the tests. +--> <table class="table table-bordered"> <tr> - <th>Field</th> - <th>Description</th> - <th>Default Value</th> -</tr> -<tr> - <td><code>runner</code></td> - <td>The pipeline runner to use. This option allows you to determine the pipeline runner at runtime.</td> - <td>Set to <code>FlinkRunner</code> to run using Flink.</td> -</tr> -<tr> - <td><code>streaming</code></td> - <td>Whether streaming mode is enabled or disabled; <code>true</code> if enabled. Set to <code>true</code> if running pipelines with unbounded <code>PCollection</code>s.</td> - <td><code>false</code></td> -</tr> -<tr> - <td><code>flinkMaster</code></td> - <td>The url of the Flink JobManager on which to execute pipelines. This can either be the address of a cluster JobManager, in the form <code>"host:port"</code> or one of the special Strings <code>"[local]"</code> or <code>"[auto]"</code>. <code>"[local]"</code> will start a local Flink Cluster in the JVM while <code>"[auto]"</code> will let the system decide where to execute the pipeline based on the environment.</td> - <td><code>[auto]</code></td> + <td><code>allowNonRestoredState</code></td> + <td>Flag indicating whether non restored state is allowed if the savepoint contains state for an operator that is no longer part of the pipeline.</td> + <td>Default: <code>false</code></td> </tr> <tr> - <td><code>filesToStage</code></td> - <td>Jar Files to send to all workers and put on the classpath. Here you have to put the fat jar that contains your program along with all dependencies.</td> - <td>empty</td> + <td><code>autoBalanceWriteFilesShardingEnabled</code></td> + <td>Flag indicating whether auto-balance sharding for WriteFiles transform should be enabled. This might prove useful in streaming use-case, where pipeline needs to write quite many events into files, typically divided into N shards. Default behavior on Flink would be, that some workers will receive more shards to take care of than others. This cause workers to go out of balance in terms of processing backlog and memory usage. Enabling this feature will make shards to be spread evenly [...] + <td>Default: <code>false</code></td> </tr> <tr> - <td><code>parallelism</code></td> - <td>The degree of parallelism to be used when distributing operations onto workers.</td> - <td>For local execution: <code>Number of available CPU cores</code> - For remote execution: <code>Default parallelism configuerd at remote cluster</code> - Otherwise: <code>1</code> - </td> + <td><code>autoWatermarkInterval</code></td> + <td>The interval in milliseconds for automatic watermark emission.</td> + <td></td> </tr> <tr> - <td><code>maxParallelism</code></td> - <td>The pipeline wide maximum degree of parallelism to be used. The maximum parallelism specifies the upper limit for dynamic scaling and the number of key groups used for partitioned state.</td> - <td><code>-1L</code>, meaning same as the parallelism</td> + <td><code>checkpointTimeoutMillis</code></td> + <td>The maximum time in milliseconds that a checkpoint may take before being discarded.</td> + <td>Default: <code>-1</code></td> </tr> <tr> <td><code>checkpointingInterval</code></td> - <td>The interval between consecutive checkpoints (i.e. snapshots of the current pipeline state used for fault tolerance).</td> - <td><code>-1L</code>, i.e. disabled</td> + <td>The interval in milliseconds at which to trigger checkpoints of the running pipeline. Default: No checkpointing.</td> + <td>Default: <code>-1</code></td> </tr> <tr> - <td><code>checkpointMode</code></td> + <td><code>checkpointingMode</code></td> <td>The checkpointing mode that defines consistency guarantee.</td> - <td><code>EXACTLY_ONCE</code></td> -</tr> -<tr> - <td><code>checkpointTimeoutMillis</code></td> - <td>The maximum time in milliseconds that a checkpoint may take before being discarded</td> - <td><code>-1</code>, the cluster default</td> + <td>Default: <code>EXACTLY_ONCE</code></td> </tr> <tr> - <td><code>minPauseBetweenCheckpoints</code></td> - <td>The minimal pause in milliseconds before the next checkpoint is triggered.</td> - <td><code>-1</code>, the cluster default</td> + <td><code>disableMetricAccumulator</code></td> + <td>By default, uses Flink accumulators to store the metrics which allows to query metrics from the PipelineResult. If set to true, metrics will still be reported but can't be queried via PipelineResult. This saves network and memory.</td> + <td>Default: <code>false</code></td> </tr> <tr> - <td><code>failOnCheckpointingErrors</code></td> - <td> - Sets the expected behaviour for tasks in case that they encounter an error in their - checkpointing procedure. If this is set to true, the task will fail on checkpointing error. - If this is set to false, the task will only decline a the checkpoint and continue running. - </td> - <td><code>-1</code>, the cluster default</td> + <td><code>disableMetrics</code></td> + <td>Disable Beam metrics in Flink Runner</td> + <td>Default: <code>false</code></td> </tr> <tr> - <td><code>numberOfExecutionRetries</code></td> - <td>Sets the number of times that failed tasks are re-executed. A value of <code>0</code> effectively disables fault tolerance. A value of <code>-1</code> indicates that the system default value (as defined in the configuration) should be used.</td> - <td><code>-1</code></td> + <td><code>executionModeForBatch</code></td> + <td>Flink mode for data exchange of batch pipelines. Reference {@link org.apache.flink.api.common.ExecutionMode}. Set this to BATCH_FORCED if pipelines get blocked, see https://issues.apache.org/jira/browse/FLINK-10672</td> + <td>Default: <code>PIPELINED</code></td> </tr> <tr> <td><code>executionRetryDelay</code></td> - <td>Sets the delay between executions. A value of <code>-1</code> indicates that the default value should be used.</td> - <td><code>-1</code></td> + <td>Sets the delay in milliseconds between executions. A value of {@code -1} indicates that the default value should be used.</td> + <td>Default: <code>-1</code></td> </tr> <tr> - <td><code>objectReuse</code></td> - <td>Sets the behavior of reusing objects.</td> - <td><code>false</code>, no Object reuse</td> + <td><code>externalizedCheckpointsEnabled</code></td> + <td>Enables or disables externalized checkpoints. Works in conjunction with CheckpointingInterval</td> + <td>Default: <code>false</code></td> </tr> <tr> - <td><code>stateBackend</code></td> - <td>Sets the state backend to use in streaming mode. The default is to read this setting from the Flink config.</td> - <td><code>empty</code>, i.e. read from Flink config</td> + <td><code>failOnCheckpointingErrors</code></td> + <td>Sets the expected behaviour for tasks in case that they encounter an error in their checkpointing procedure. If this is set to true, the task will fail on checkpointing error. If this is set to false, the task will only decline a the checkpoint and continue running. </td> + <td>Default: <code>true</code></td> </tr> <tr> - <td><code>enableMetrics</code></td> - <td>Enable/disable Beam metrics in Flink Runner</td> - <td>Default: <code>true</code></td> + <td><code>filesToStage</code></td> + <td>Jar-Files to send to all workers and put on the classpath. The default value is all files from the classpath.</td> + <td></td> </tr> <tr> - <td><code>externalizedCheckpointsEnabled</code></td> - <td>Enables or disables externalized checkpoints. Works in conjunction with CheckpointingInterval</td> - <td>Default: <code>false</code></td> + <td><code>flinkMaster</code></td> + <td>Address of the Flink Master where the Pipeline should be executed. Can either be of the form "host:port" or one of the special values [local], [collection] or [auto].</td> + <td>Default: <code>[auto]</code></td> </tr> <tr> - <td><code>retainExternalizedCheckpointsOnCancellation</code></td> - <td>Sets the behavior of externalized checkpoints on cancellation.</td> - <td>Default: <code>false</code></td> + <td><code>latencyTrackingInterval</code></td> + <td>Interval in milliseconds for sending latency tracking marks from the sources to the sinks. Interval value <= 0 disables the feature.</td> + <td>Default: <code>0</code></td> </tr> <tr> <td><code>maxBundleSize</code></td> @@ -692,58 +683,92 @@ Many sources like <code class="highlighter-rouge">PubSubIO</code> rely on their <td>Default: <code>1000</code></td> </tr> <tr> - <td><code>shutdownSourcesOnFinalWatermark</code></td> - <td>If set, shutdown sources when their watermark reaches +Inf.</td> - <td>Default: <code>false</code></td> + <td><code>maxParallelism</code></td> + <td>The pipeline wide maximum degree of parallelism to be used. The maximum parallelism specifies the upper limit for dynamic scaling and the number of key groups used for partitioned state.</td> + <td>Default: <code>-1</code></td> </tr> <tr> - <td><code>latencyTrackingInterval</code></td> - <td>Interval in milliseconds for sending latency tracking marks from the sources to the sinks. Interval value <= 0 disables the feature.</td> - <td>Default: <code>0</code></td> + <td><code>minPauseBetweenCheckpoints</code></td> + <td>The minimal pause in milliseconds before the next checkpoint is triggered.</td> + <td>Default: <code>-1</code></td> </tr> <tr> - <td><code>autoWatermarkInterval</code></td> - <td>The interval in milliseconds for automatic watermark emission.</td> + <td><code>numberOfExecutionRetries</code></td> + <td>Sets the number of times that failed tasks are re-executed. A value of zero effectively disables fault tolerance. A value of -1 indicates that the system default value (as defined in the configuration) should be used.</td> + <td>Default: <code>-1</code></td> </tr> <tr> - <td><code>executionModeForBatch</code></td> - <td>Flink mode for data exchange of batch pipelines. Reference {@link org.apache.flink.api.common.ExecutionMode}. Set this to BATCH_FORCED if pipelines get blocked, see https://issues.apache.org/jira/browse/FLINK-10672</td> - <td>Default: <code>PIPELINED</code></td> + <td><code>objectReuse</code></td> + <td>Sets the behavior of reusing objects.</td> + <td>Default: <code>false</code></td> +</tr> +<tr> + <td><code>parallelism</code></td> + <td>The degree of parallelism to be used when distributing operations onto workers. If the parallelism is not set, the configured Flink default is used, or 1 if none can be found.</td> + <td>Default: <code>-1</code></td> +</tr> +<tr> + <td><code>retainExternalizedCheckpointsOnCancellation</code></td> + <td>Sets the behavior of externalized checkpoints on cancellation.</td> + <td>Default: <code>false</code></td> </tr> <tr> <td><code>savepointPath</code></td> <td>Savepoint restore path. If specified, restores the streaming pipeline from the provided path.</td> - <td>Default: None</td> + <td></td> </tr> <tr> - <td><code>allowNonRestoredState</code></td> - <td>Flag indicating whether non restored state is allowed if the savepoint contains state for an operator that is no longer part of the pipeline.</td> + <td><code>shutdownSourcesOnFinalWatermark</code></td> + <td>If set, shutdown sources when their watermark reaches +Inf.</td> <td>Default: <code>false</code></td> </tr> +<tr> + <td><code>stateBackendFactory</code></td> + <td>Sets the state backend factory to use in streaming mode. Defaults to the flink cluster's state.backend configuration.</td> + <td></td> +</tr> </table> -</div> +</div> <!-- Python Options --> <div class="language-py"> -<table class="table table-bordered"> +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +<!-- +This is an auto-generated file. +Use generatePipelineOptionsTableJava and generatePipelineOptionsTablePython respectively +which should be called before running the tests. +--> +<table class="table table-bordered"> <tr> - <td><code>files_to_stage</code></td> - <td>Jar-Files to send to all workers and put on the classpath. The default value is all files from the classpath.</td> + <td><code>allow_non_restored_state</code></td> + <td>Flag indicating whether non restored state is allowed if the savepoint contains state for an operator that is no longer part of the pipeline.</td> + <td>Default: <code>false</code></td> </tr> <tr> - <td><code>flink_master</code></td> - <td>Address of the Flink Master where the Pipeline should be executed. Can either be of the form "host:port" or one of the special values [local], [collection] or [auto].</td> - <td>Default: <code>[auto]</code></td> + <td><code>auto_balance_write_files_sharding_enabled</code></td> + <td>Flag indicating whether auto-balance sharding for WriteFiles transform should be enabled. This might prove useful in streaming use-case, where pipeline needs to write quite many events into files, typically divided into N shards. Default behavior on Flink would be, that some workers will receive more shards to take care of than others. This cause workers to go out of balance in terms of processing backlog and memory usage. Enabling this feature will make shards to be spread evenly [...] + <td>Default: <code>false</code></td> </tr> <tr> - <td><code>parallelism</code></td> - <td>The degree of parallelism to be used when distributing operations onto workers. If the parallelism is not set, the configured Flink default is used, or 1 if none can be found.</td> - <td>Default: <code>-1</code></td> + <td><code>auto_watermark_interval</code></td> + <td>The interval in milliseconds for automatic watermark emission.</td> + <td></td> </tr> <tr> - <td><code>max_parallelism</code></td> - <td>The pipeline wide maximum degree of parallelism to be used. The maximum parallelism specifies the upper limit for dynamic scaling and the number of key groups used for partitioned state.</td> + <td><code>checkpoint_timeout_millis</code></td> + <td>The maximum time in milliseconds that a checkpoint may take before being discarded.</td> <td>Default: <code>-1</code></td> </tr> <tr> @@ -757,24 +782,19 @@ Many sources like <code class="highlighter-rouge">PubSubIO</code> rely on their <td>Default: <code>EXACTLY_ONCE</code></td> </tr> <tr> - <td><code>checkpoint_timeout_millis</code></td> - <td>The maximum time in milliseconds that a checkpoint may take before being discarded.</td> - <td>Default: <code>-1</code></td> -</tr> -<tr> - <td><code>min_pause_between_checkpoints</code></td> - <td>The minimal pause in milliseconds before the next checkpoint is triggered.</td> - <td>Default: <code>-1</code></td> + <td><code>disable_metric_accumulator</code></td> + <td>By default, uses Flink accumulators to store the metrics which allows to query metrics from the PipelineResult. If set to true, metrics will still be reported but can't be queried via PipelineResult. This saves network and memory.</td> + <td>Default: <code>false</code></td> </tr> <tr> - <td><code>fail_on_checkpointing_errors</code></td> - <td>Sets the expected behaviour for tasks in case that they encounter an error in their checkpointing procedure. If this is set to true, the task will fail on checkpointing error. If this is set to false, the task will only decline a the checkpoint and continue running. </td> - <td>Default: <code>true</code></td> + <td><code>disable_metrics</code></td> + <td>Disable Beam metrics in Flink Runner</td> + <td>Default: <code>false</code></td> </tr> <tr> - <td><code>number_of_execution_retries</code></td> - <td>Sets the number of times that failed tasks are re-executed. A value of zero effectively disables fault tolerance. A value of -1 indicates that the system default value (as defined in the configuration) should be used.</td> - <td>Default: <code>-1</code></td> + <td><code>execution_mode_for_batch</code></td> + <td>Flink mode for data exchange of batch pipelines. Reference {@link org.apache.flink.api.common.ExecutionMode}. Set this to BATCH_FORCED if pipelines get blocked, see https://issues.apache.org/jira/browse/FLINK-10672</td> + <td>Default: <code>PIPELINED</code></td> </tr> <tr> <td><code>execution_retry_delay</code></td> @@ -782,28 +802,29 @@ Many sources like <code class="highlighter-rouge">PubSubIO</code> rely on their <td>Default: <code>-1</code></td> </tr> <tr> - <td><code>object_reuse</code></td> - <td>Sets the behavior of reusing objects.</td> + <td><code>externalized_checkpoints_enabled</code></td> + <td>Enables or disables externalized checkpoints. Works in conjunction with CheckpointingInterval</td> <td>Default: <code>false</code></td> </tr> <tr> - <td><code>state_backend</code></td> - <td>Sets the state backend to use in streaming mode. Otherwise the default is read from the Flink config.</td> + <td><code>fail_on_checkpointing_errors</code></td> + <td>Sets the expected behaviour for tasks in case that they encounter an error in their checkpointing procedure. If this is set to true, the task will fail on checkpointing error. If this is set to false, the task will only decline a the checkpoint and continue running. </td> + <td>Default: <code>true</code></td> </tr> <tr> - <td><code>enable_metrics</code></td> - <td>Enable/disable Beam metrics in Flink Runner</td> - <td>Default: <code>true</code></td> + <td><code>files_to_stage</code></td> + <td>Jar-Files to send to all workers and put on the classpath. The default value is all files from the classpath.</td> + <td></td> </tr> <tr> - <td><code>externalized_checkpoints_enabled</code></td> - <td>Enables or disables externalized checkpoints. Works in conjunction with CheckpointingInterval</td> - <td>Default: <code>false</code></td> + <td><code>flink_master</code></td> + <td>Address of the Flink Master where the Pipeline should be executed. Can either be of the form "host:port" or one of the special values [local], [collection] or [auto].</td> + <td>Default: <code>[auto]</code></td> </tr> <tr> - <td><code>retain_externalized_checkpoints_on_cancellation</code></td> - <td>Sets the behavior of externalized checkpoints on cancellation.</td> - <td>Default: <code>false</code></td> + <td><code>latency_tracking_interval</code></td> + <td>Interval in milliseconds for sending latency tracking marks from the sources to the sinks. Interval value <= 0 disables the feature.</td> + <td>Default: <code>0</code></td> </tr> <tr> <td><code>max_bundle_size</code></td> @@ -816,35 +837,52 @@ Many sources like <code class="highlighter-rouge">PubSubIO</code> rely on their <td>Default: <code>1000</code></td> </tr> <tr> - <td><code>shutdown_sources_on_final_watermark</code></td> - <td>If set, shutdown sources when their watermark reaches +Inf.</td> - <td>Default: <code>false</code></td> + <td><code>max_parallelism</code></td> + <td>The pipeline wide maximum degree of parallelism to be used. The maximum parallelism specifies the upper limit for dynamic scaling and the number of key groups used for partitioned state.</td> + <td>Default: <code>-1</code></td> </tr> <tr> - <td><code>latency_tracking_interval</code></td> - <td>Interval in milliseconds for sending latency tracking marks from the sources to the sinks. Interval value <= 0 disables the feature.</td> - <td>Default: <code>0</code></td> + <td><code>min_pause_between_checkpoints</code></td> + <td>The minimal pause in milliseconds before the next checkpoint is triggered.</td> + <td>Default: <code>-1</code></td> </tr> <tr> - <td><code>auto_watermark_interval</code></td> - <td>The interval in milliseconds for automatic watermark emission.</td> + <td><code>number_of_execution_retries</code></td> + <td>Sets the number of times that failed tasks are re-executed. A value of zero effectively disables fault tolerance. A value of -1 indicates that the system default value (as defined in the configuration) should be used.</td> + <td>Default: <code>-1</code></td> </tr> <tr> - <td><code>execution_mode_for_batch</code></td> - <td>Flink mode for data exchange of batch pipelines. Reference {@link org.apache.flink.api.common.ExecutionMode}. Set this to BATCH_FORCED if pipelines get blocked, see https://issues.apache.org/jira/browse/FLINK-10672</td> - <td>Default: <code>PIPELINED</code></td> + <td><code>object_reuse</code></td> + <td>Sets the behavior of reusing objects.</td> + <td>Default: <code>false</code></td> +</tr> +<tr> + <td><code>parallelism</code></td> + <td>The degree of parallelism to be used when distributing operations onto workers. If the parallelism is not set, the configured Flink default is used, or 1 if none can be found.</td> + <td>Default: <code>-1</code></td> +</tr> +<tr> + <td><code>retain_externalized_checkpoints_on_cancellation</code></td> + <td>Sets the behavior of externalized checkpoints on cancellation.</td> + <td>Default: <code>false</code></td> </tr> <tr> <td><code>savepoint_path</code></td> <td>Savepoint restore path. If specified, restores the streaming pipeline from the provided path.</td> + <td></td> </tr> <tr> - <td><code>allow_non_restored_state</code></td> - <td>Flag indicating whether non restored state is allowed if the savepoint contains state for an operator that is no longer part of the pipeline.</td> + <td><code>shutdown_sources_on_final_watermark</code></td> + <td>If set, shutdown sources when their watermark reaches +Inf.</td> <td>Default: <code>false</code></td> </tr> - +<tr> + <td><code>state_backend_factory</code></td> + <td>Sets the state backend factory to use in streaming mode. Defaults to the flink cluster's state.backend configuration.</td> + <td></td> +</tr> </table> + </div> <h2 id="capability">Capability</h2>