[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16367310#comment-16367310 ] ASF GitHub Bot commented on FLINK-8475: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5507 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16367286#comment-16367286 ] ASF GitHub Bot commented on FLINK-8475: --- GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/5507 [FLINK-8475][config][docs] Integrate HeartbeatManager options ## What is the purpose of the change This PR integrates the HeartbeatManager `ConfigOptions` into the configuration docs generator. Based on #5506. ## Brief change log * Add missing descriptions to config options (derived from existing description/javadocs) * integrate HeartbeatManager configuration table into `config.md` You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 8475_hm Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5507.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5507 commit 219dbe6985f894df81eb7f67633d1ebe4869a747 Author: zentol Date: 2018-02-16T12:56:29Z [FLINK-8671][docs] Word-wrap default values commit f52b552bf88282909db42768b0430441bac323b2 Author: zentol Date: 2018-01-23T13:48:24Z [FLINK-8475][config][docs] Integrate HeartbeatManager options > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363894#comment-16363894 ] ASF GitHub Bot commented on FLINK-8475: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5459 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363898#comment-16363898 ] ASF GitHub Bot commented on FLINK-8475: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5471 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363895#comment-16363895 ] ASF GitHub Bot commented on FLINK-8475: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5467 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363897#comment-16363897 ] ASF GitHub Bot commented on FLINK-8475: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5470 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363896#comment-16363896 ] ASF GitHub Bot commented on FLINK-8475: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5468 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363901#comment-16363901 ] ASF GitHub Bot commented on FLINK-8475: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5469 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363818#comment-16363818 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5469 merging. > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363814#comment-16363814 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/5469 Ok, sounds fine to me then. 👍 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363777#comment-16363777 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5471 merging. > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363772#comment-16363772 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5470 merging. > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363775#comment-16363775 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5467 merging > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363776#comment-16363776 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5468 merging > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363774#comment-16363774 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5459 merging > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362498#comment-16362498 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5459#discussion_r167901429 --- Diff: flink-core/src/main/java/org/apache/flink/configuration/CoreOptions.java --- @@ -127,7 +130,30 @@ */ public static final ConfigOption DEFAULT_FILESYSTEM_SCHEME = ConfigOptions .key("fs.default-scheme") - .noDefaultValue(); + .noDefaultValue() + .withDescription("The default filesystem scheme, used for paths that do not declare a scheme explicitly."); --- End diff -- Ok, then I'd say merge it like this. > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362497#comment-16362497 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5468#discussion_r167901255 --- Diff: docs/_includes/generated/environment_configuration.html --- @@ -0,0 +1,41 @@ + + + +Key +Default +Description + + + + +env.java.opts +(none) + --- End diff -- KK, go ahead, then. 👍 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362495#comment-16362495 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/5467 Ok, then please go ahead. > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362492#comment-16362492 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/5470 Good to go 👌 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362490#comment-16362490 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/5471 Feel free to merge once you addressed my other comment. I'm eager to merge this no matter what because all of this is better than what we had before... > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362488#comment-16362488 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5471#discussion_r167900592 --- Diff: docs/_includes/generated/task_manager_configuration.html --- @@ -0,0 +1,146 @@ + + + +Key +Default +Description + + + + +task.cancellation.interval --- End diff -- `cancellation.timeout`, I think I was referring to > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362332#comment-16362332 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5471 @aljoscha I've ported the missing options and modified plenty of descriptions. > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362314#comment-16362314 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5471#discussion_r167863017 --- Diff: docs/ops/config.md --- @@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp ### TaskManager -The following parameters configure Flink's TaskManagers. - - -- `taskmanager.hostname`: The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file. - -- `taskmanager.rpc.port`: The task manager's IPC port (DEFAULT: **0**, which lets the OS choose a free port). Flink also accepts a list of ports ("50100,50101"), ranges ("50100-50200") or a combination of both. It is recommended to set a range of ports to avoid collisions when multiple TaskManagers are running on the same machine. - -- `taskmanager.data.port`: The task manager's port used for data exchange operations (DEFAULT: **0**, which lets the OS choose a free port). - -- `taskmanager.data.ssl.enabled`: Enable SSL support for the taskmanager data transport. This is applicable only when the global ssl flag security.ssl.enabled is set to true (DEFAULT: **true**) - -- `taskmanager.heap.mb`: JVM heap size (in megabytes) for the TaskManagers, which are the parallel workers of the system. In contrast to Hadoop, Flink runs operators (e.g., join, aggregate) and user-defined functions (e.g., Map, Reduce, CoGroup) inside the TaskManager (including sorting/hashing/caching), so this value should be as large as possible (DEFAULT: **512**). On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value. --- End diff -- Actually yes, I believe this doesn't belong in there. The YARN note is useful though... > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362304#comment-16362304 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5471#discussion_r167861535 --- Diff: docs/ops/config.md --- @@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp ### TaskManager -The following parameters configure Flink's TaskManagers. - - -- `taskmanager.hostname`: The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file. - -- `taskmanager.rpc.port`: The task manager's IPC port (DEFAULT: **0**, which lets the OS choose a free port). Flink also accepts a list of ports ("50100,50101"), ranges ("50100-50200") or a combination of both. It is recommended to set a range of ports to avoid collisions when multiple TaskManagers are running on the same machine. - -- `taskmanager.data.port`: The task manager's port used for data exchange operations (DEFAULT: **0**, which lets the OS choose a free port). - -- `taskmanager.data.ssl.enabled`: Enable SSL support for the taskmanager data transport. This is applicable only when the global ssl flag security.ssl.enabled is set to true (DEFAULT: **true**) --- End diff -- will revert to existing docs > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362303#comment-16362303 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5471#discussion_r167860765 --- Diff: docs/_includes/generated/task_manager_configuration.html --- @@ -0,0 +1,146 @@ + + + +Key +Default +Description + + + + +task.cancellation.interval --- End diff -- there's a description for cancellation.interval, which other one are you referring to? > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362272#comment-16362272 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5471 man these long descriptions are gonna wreck the table... > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362270#comment-16362270 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5471#discussion_r167856613 --- Diff: docs/ops/config.md --- @@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp ### TaskManager -The following parameters configure Flink's TaskManagers. - - -- `taskmanager.hostname`: The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file. - -- `taskmanager.rpc.port`: The task manager's IPC port (DEFAULT: **0**, which lets the OS choose a free port). Flink also accepts a list of ports ("50100,50101"), ranges ("50100-50200") or a combination of both. It is recommended to set a range of ports to avoid collisions when multiple TaskManagers are running on the same machine. - -- `taskmanager.data.port`: The task manager's port used for data exchange operations (DEFAULT: **0**, which lets the OS choose a free port). - -- `taskmanager.data.ssl.enabled`: Enable SSL support for the taskmanager data transport. This is applicable only when the global ssl flag security.ssl.enabled is set to true (DEFAULT: **true**) - -- `taskmanager.heap.mb`: JVM heap size (in megabytes) for the TaskManagers, which are the parallel workers of the system. In contrast to Hadoop, Flink runs operators (e.g., join, aggregate) and user-defined functions (e.g., Map, Reduce, CoGroup) inside the TaskManager (including sorting/hashing/caching), so this value should be as large as possible (DEFAULT: **512**). On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value. - -- `taskmanager.numberOfTaskSlots`: The number of parallel operator or user function instances that a single TaskManager can run (DEFAULT: **1**). If this value is larger than 1, a single TaskManager takes multiple instances of a function or operator. That way, the TaskManager can utilize multiple CPU cores, but at the same time, the available memory is divided between the different operator or function instances. This value is typically proportional to the number of physical CPU cores that the TaskManager's machine has (e.g., equal to the number of cores, or half the number of cores). - -- `taskmanager.tmp.dirs`: The directory for temporary files, or a list of directories separated by the system's directory delimiter (for example ':' (colon) on Linux/Unix). If multiple directories are specified, then the temporary files will be distributed across the directories in a round robin fashion. The I/O manager component will spawn one reading and one writing thread per directory. A directory may be listed multiple times to have the I/O manager use multiple threads for it (for example if it is physically stored on a very fast disc or RAID) (DEFAULT: **The system's tmp dir**). - -- `taskmanager.network.memory.fraction`: Fraction of JVM memory to use for network buffers. This determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value or the min/max values below. Also note, that `taskmanager.network.memory.min` and `taskmanager.network.memory.max` may override this fraction. (DEFAULT: **0.1**) - -- `taskmanager.network.memory.min`: Minimum memory size for network buffers in bytes (DEFAULT: **64 MB**). Previously, this was determined from `taskmanager.network.numberOfBuffers` and `taskmanager.memory.segment-size`. - -- `taskmanager.network.memory.max`: Maximum memory size for network buffers in bytes (DEFAULT: **1 GB**). Previously, this was determined from `taskmanager.network.numberOfBuffers` and `taskmanager.memory.segment-size`. - -- `taskmanager.network.numberOfBuffers` (deprecated, replaced by the three parameters above): The number of buffers available to the network stack. This number determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value (DEFAULT: **2048**). If set, it
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362271#comment-16362271 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5471#discussion_r167856770 --- Diff: docs/ops/config.md --- @@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp ### TaskManager -The following parameters configure Flink's TaskManagers. - - -- `taskmanager.hostname`: The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file. - -- `taskmanager.rpc.port`: The task manager's IPC port (DEFAULT: **0**, which lets the OS choose a free port). Flink also accepts a list of ports ("50100,50101"), ranges ("50100-50200") or a combination of both. It is recommended to set a range of ports to avoid collisions when multiple TaskManagers are running on the same machine. - -- `taskmanager.data.port`: The task manager's port used for data exchange operations (DEFAULT: **0**, which lets the OS choose a free port). - -- `taskmanager.data.ssl.enabled`: Enable SSL support for the taskmanager data transport. This is applicable only when the global ssl flag security.ssl.enabled is set to true (DEFAULT: **true**) - -- `taskmanager.heap.mb`: JVM heap size (in megabytes) for the TaskManagers, which are the parallel workers of the system. In contrast to Hadoop, Flink runs operators (e.g., join, aggregate) and user-defined functions (e.g., Map, Reduce, CoGroup) inside the TaskManager (including sorting/hashing/caching), so this value should be as large as possible (DEFAULT: **512**). On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value. - -- `taskmanager.numberOfTaskSlots`: The number of parallel operator or user function instances that a single TaskManager can run (DEFAULT: **1**). If this value is larger than 1, a single TaskManager takes multiple instances of a function or operator. That way, the TaskManager can utilize multiple CPU cores, but at the same time, the available memory is divided between the different operator or function instances. This value is typically proportional to the number of physical CPU cores that the TaskManager's machine has (e.g., equal to the number of cores, or half the number of cores). - -- `taskmanager.tmp.dirs`: The directory for temporary files, or a list of directories separated by the system's directory delimiter (for example ':' (colon) on Linux/Unix). If multiple directories are specified, then the temporary files will be distributed across the directories in a round robin fashion. The I/O manager component will spawn one reading and one writing thread per directory. A directory may be listed multiple times to have the I/O manager use multiple threads for it (for example if it is physically stored on a very fast disc or RAID) (DEFAULT: **The system's tmp dir**). - -- `taskmanager.network.memory.fraction`: Fraction of JVM memory to use for network buffers. This determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value or the min/max values below. Also note, that `taskmanager.network.memory.min` and `taskmanager.network.memory.max` may override this fraction. (DEFAULT: **0.1**) - -- `taskmanager.network.memory.min`: Minimum memory size for network buffers in bytes (DEFAULT: **64 MB**). Previously, this was determined from `taskmanager.network.numberOfBuffers` and `taskmanager.memory.segment-size`. - -- `taskmanager.network.memory.max`: Maximum memory size for network buffers in bytes (DEFAULT: **1 GB**). Previously, this was determined from `taskmanager.network.numberOfBuffers` and `taskmanager.memory.segment-size`. - -- `taskmanager.network.numberOfBuffers` (deprecated, replaced by the three parameters above): The number of buffers available to the network stack. This number determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value (DEFAULT: **2048**). If set, it
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362267#comment-16362267 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5471#discussion_r167856481 --- Diff: docs/ops/config.md --- @@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp ### TaskManager -The following parameters configure Flink's TaskManagers. - - -- `taskmanager.hostname`: The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file. - -- `taskmanager.rpc.port`: The task manager's IPC port (DEFAULT: **0**, which lets the OS choose a free port). Flink also accepts a list of ports ("50100,50101"), ranges ("50100-50200") or a combination of both. It is recommended to set a range of ports to avoid collisions when multiple TaskManagers are running on the same machine. - -- `taskmanager.data.port`: The task manager's port used for data exchange operations (DEFAULT: **0**, which lets the OS choose a free port). - -- `taskmanager.data.ssl.enabled`: Enable SSL support for the taskmanager data transport. This is applicable only when the global ssl flag security.ssl.enabled is set to true (DEFAULT: **true**) - -- `taskmanager.heap.mb`: JVM heap size (in megabytes) for the TaskManagers, which are the parallel workers of the system. In contrast to Hadoop, Flink runs operators (e.g., join, aggregate) and user-defined functions (e.g., Map, Reduce, CoGroup) inside the TaskManager (including sorting/hashing/caching), so this value should be as large as possible (DEFAULT: **512**). On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value. - -- `taskmanager.numberOfTaskSlots`: The number of parallel operator or user function instances that a single TaskManager can run (DEFAULT: **1**). If this value is larger than 1, a single TaskManager takes multiple instances of a function or operator. That way, the TaskManager can utilize multiple CPU cores, but at the same time, the available memory is divided between the different operator or function instances. This value is typically proportional to the number of physical CPU cores that the TaskManager's machine has (e.g., equal to the number of cores, or half the number of cores). - -- `taskmanager.tmp.dirs`: The directory for temporary files, or a list of directories separated by the system's directory delimiter (for example ':' (colon) on Linux/Unix). If multiple directories are specified, then the temporary files will be distributed across the directories in a round robin fashion. The I/O manager component will spawn one reading and one writing thread per directory. A directory may be listed multiple times to have the I/O manager use multiple threads for it (for example if it is physically stored on a very fast disc or RAID) (DEFAULT: **The system's tmp dir**). - -- `taskmanager.network.memory.fraction`: Fraction of JVM memory to use for network buffers. This determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value or the min/max values below. Also note, that `taskmanager.network.memory.min` and `taskmanager.network.memory.max` may override this fraction. (DEFAULT: **0.1**) - -- `taskmanager.network.memory.min`: Minimum memory size for network buffers in bytes (DEFAULT: **64 MB**). Previously, this was determined from `taskmanager.network.numberOfBuffers` and `taskmanager.memory.segment-size`. - -- `taskmanager.network.memory.max`: Maximum memory size for network buffers in bytes (DEFAULT: **1 GB**). Previously, this was determined from `taskmanager.network.numberOfBuffers` and `taskmanager.memory.segment-size`. - -- `taskmanager.network.numberOfBuffers` (deprecated, replaced by the three parameters above): The number of buffers available to the network stack. This number determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value (DEFAULT: **2048**). If set, it
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362266#comment-16362266 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5471#discussion_r167856324 --- Diff: docs/ops/config.md --- @@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp ### TaskManager -The following parameters configure Flink's TaskManagers. - - -- `taskmanager.hostname`: The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file. - -- `taskmanager.rpc.port`: The task manager's IPC port (DEFAULT: **0**, which lets the OS choose a free port). Flink also accepts a list of ports ("50100,50101"), ranges ("50100-50200") or a combination of both. It is recommended to set a range of ports to avoid collisions when multiple TaskManagers are running on the same machine. - -- `taskmanager.data.port`: The task manager's port used for data exchange operations (DEFAULT: **0**, which lets the OS choose a free port). - -- `taskmanager.data.ssl.enabled`: Enable SSL support for the taskmanager data transport. This is applicable only when the global ssl flag security.ssl.enabled is set to true (DEFAULT: **true**) - -- `taskmanager.heap.mb`: JVM heap size (in megabytes) for the TaskManagers, which are the parallel workers of the system. In contrast to Hadoop, Flink runs operators (e.g., join, aggregate) and user-defined functions (e.g., Map, Reduce, CoGroup) inside the TaskManager (including sorting/hashing/caching), so this value should be as large as possible (DEFAULT: **512**). On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value. - -- `taskmanager.numberOfTaskSlots`: The number of parallel operator or user function instances that a single TaskManager can run (DEFAULT: **1**). If this value is larger than 1, a single TaskManager takes multiple instances of a function or operator. That way, the TaskManager can utilize multiple CPU cores, but at the same time, the available memory is divided between the different operator or function instances. This value is typically proportional to the number of physical CPU cores that the TaskManager's machine has (e.g., equal to the number of cores, or half the number of cores). - -- `taskmanager.tmp.dirs`: The directory for temporary files, or a list of directories separated by the system's directory delimiter (for example ':' (colon) on Linux/Unix). If multiple directories are specified, then the temporary files will be distributed across the directories in a round robin fashion. The I/O manager component will spawn one reading and one writing thread per directory. A directory may be listed multiple times to have the I/O manager use multiple threads for it (for example if it is physically stored on a very fast disc or RAID) (DEFAULT: **The system's tmp dir**). - -- `taskmanager.network.memory.fraction`: Fraction of JVM memory to use for network buffers. This determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value or the min/max values below. Also note, that `taskmanager.network.memory.min` and `taskmanager.network.memory.max` may override this fraction. (DEFAULT: **0.1**) - -- `taskmanager.network.memory.min`: Minimum memory size for network buffers in bytes (DEFAULT: **64 MB**). Previously, this was determined from `taskmanager.network.numberOfBuffers` and `taskmanager.memory.segment-size`. - -- `taskmanager.network.memory.max`: Maximum memory size for network buffers in bytes (DEFAULT: **1 GB**). Previously, this was determined from `taskmanager.network.numberOfBuffers` and `taskmanager.memory.segment-size`. - -- `taskmanager.network.numberOfBuffers` (deprecated, replaced by the three parameters above): The number of buffers available to the network stack. This number determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value (DEFAULT: **2048**). If set, it
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362265#comment-16362265 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5471#discussion_r167856138 --- Diff: docs/ops/config.md --- @@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp ### TaskManager -The following parameters configure Flink's TaskManagers. - - -- `taskmanager.hostname`: The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file. - -- `taskmanager.rpc.port`: The task manager's IPC port (DEFAULT: **0**, which lets the OS choose a free port). Flink also accepts a list of ports ("50100,50101"), ranges ("50100-50200") or a combination of both. It is recommended to set a range of ports to avoid collisions when multiple TaskManagers are running on the same machine. - -- `taskmanager.data.port`: The task manager's port used for data exchange operations (DEFAULT: **0**, which lets the OS choose a free port). - -- `taskmanager.data.ssl.enabled`: Enable SSL support for the taskmanager data transport. This is applicable only when the global ssl flag security.ssl.enabled is set to true (DEFAULT: **true**) - -- `taskmanager.heap.mb`: JVM heap size (in megabytes) for the TaskManagers, which are the parallel workers of the system. In contrast to Hadoop, Flink runs operators (e.g., join, aggregate) and user-defined functions (e.g., Map, Reduce, CoGroup) inside the TaskManager (including sorting/hashing/caching), so this value should be as large as possible (DEFAULT: **512**). On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value. - -- `taskmanager.numberOfTaskSlots`: The number of parallel operator or user function instances that a single TaskManager can run (DEFAULT: **1**). If this value is larger than 1, a single TaskManager takes multiple instances of a function or operator. That way, the TaskManager can utilize multiple CPU cores, but at the same time, the available memory is divided between the different operator or function instances. This value is typically proportional to the number of physical CPU cores that the TaskManager's machine has (e.g., equal to the number of cores, or half the number of cores). - -- `taskmanager.tmp.dirs`: The directory for temporary files, or a list of directories separated by the system's directory delimiter (for example ':' (colon) on Linux/Unix). If multiple directories are specified, then the temporary files will be distributed across the directories in a round robin fashion. The I/O manager component will spawn one reading and one writing thread per directory. A directory may be listed multiple times to have the I/O manager use multiple threads for it (for example if it is physically stored on a very fast disc or RAID) (DEFAULT: **The system's tmp dir**). - -- `taskmanager.network.memory.fraction`: Fraction of JVM memory to use for network buffers. This determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value or the min/max values below. Also note, that `taskmanager.network.memory.min` and `taskmanager.network.memory.max` may override this fraction. (DEFAULT: **0.1**) - -- `taskmanager.network.memory.min`: Minimum memory size for network buffers in bytes (DEFAULT: **64 MB**). Previously, this was determined from `taskmanager.network.numberOfBuffers` and `taskmanager.memory.segment-size`. - -- `taskmanager.network.memory.max`: Maximum memory size for network buffers in bytes (DEFAULT: **1 GB**). Previously, this was determined from `taskmanager.network.numberOfBuffers` and `taskmanager.memory.segment-size`. - -- `taskmanager.network.numberOfBuffers` (deprecated, replaced by the three parameters above): The number of buffers available to the network stack. This number determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value (DEFAULT: **2048**). If set, it
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362261#comment-16362261 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5471#discussion_r167855709 --- Diff: docs/ops/config.md --- @@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp ### TaskManager -The following parameters configure Flink's TaskManagers. - - -- `taskmanager.hostname`: The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file. - -- `taskmanager.rpc.port`: The task manager's IPC port (DEFAULT: **0**, which lets the OS choose a free port). Flink also accepts a list of ports ("50100,50101"), ranges ("50100-50200") or a combination of both. It is recommended to set a range of ports to avoid collisions when multiple TaskManagers are running on the same machine. - -- `taskmanager.data.port`: The task manager's port used for data exchange operations (DEFAULT: **0**, which lets the OS choose a free port). - -- `taskmanager.data.ssl.enabled`: Enable SSL support for the taskmanager data transport. This is applicable only when the global ssl flag security.ssl.enabled is set to true (DEFAULT: **true**) - -- `taskmanager.heap.mb`: JVM heap size (in megabytes) for the TaskManagers, which are the parallel workers of the system. In contrast to Hadoop, Flink runs operators (e.g., join, aggregate) and user-defined functions (e.g., Map, Reduce, CoGroup) inside the TaskManager (including sorting/hashing/caching), so this value should be as large as possible (DEFAULT: **512**). On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value. - -- `taskmanager.numberOfTaskSlots`: The number of parallel operator or user function instances that a single TaskManager can run (DEFAULT: **1**). If this value is larger than 1, a single TaskManager takes multiple instances of a function or operator. That way, the TaskManager can utilize multiple CPU cores, but at the same time, the available memory is divided between the different operator or function instances. This value is typically proportional to the number of physical CPU cores that the TaskManager's machine has (e.g., equal to the number of cores, or half the number of cores). - -- `taskmanager.tmp.dirs`: The directory for temporary files, or a list of directories separated by the system's directory delimiter (for example ':' (colon) on Linux/Unix). If multiple directories are specified, then the temporary files will be distributed across the directories in a round robin fashion. The I/O manager component will spawn one reading and one writing thread per directory. A directory may be listed multiple times to have the I/O manager use multiple threads for it (for example if it is physically stored on a very fast disc or RAID) (DEFAULT: **The system's tmp dir**). --- End diff -- superceeded by `CoreOptions#TMP_DIRS` > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362252#comment-16362252 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5471#discussion_r167852941 --- Diff: docs/ops/config.md --- @@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp ### TaskManager -The following parameters configure Flink's TaskManagers. - - -- `taskmanager.hostname`: The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file. - -- `taskmanager.rpc.port`: The task manager's IPC port (DEFAULT: **0**, which lets the OS choose a free port). Flink also accepts a list of ports ("50100,50101"), ranges ("50100-50200") or a combination of both. It is recommended to set a range of ports to avoid collisions when multiple TaskManagers are running on the same machine. - -- `taskmanager.data.port`: The task manager's port used for data exchange operations (DEFAULT: **0**, which lets the OS choose a free port). - -- `taskmanager.data.ssl.enabled`: Enable SSL support for the taskmanager data transport. This is applicable only when the global ssl flag security.ssl.enabled is set to true (DEFAULT: **true**) - -- `taskmanager.heap.mb`: JVM heap size (in megabytes) for the TaskManagers, which are the parallel workers of the system. In contrast to Hadoop, Flink runs operators (e.g., join, aggregate) and user-defined functions (e.g., Map, Reduce, CoGroup) inside the TaskManager (including sorting/hashing/caching), so this value should be as large as possible (DEFAULT: **512**). On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value. - -- `taskmanager.numberOfTaskSlots`: The number of parallel operator or user function instances that a single TaskManager can run (DEFAULT: **1**). If this value is larger than 1, a single TaskManager takes multiple instances of a function or operator. That way, the TaskManager can utilize multiple CPU cores, but at the same time, the available memory is divided between the different operator or function instances. This value is typically proportional to the number of physical CPU cores that the TaskManager's machine has (e.g., equal to the number of cores, or half the number of cores). - -- `taskmanager.tmp.dirs`: The directory for temporary files, or a list of directories separated by the system's directory delimiter (for example ':' (colon) on Linux/Unix). If multiple directories are specified, then the temporary files will be distributed across the directories in a round robin fashion. The I/O manager component will spawn one reading and one writing thread per directory. A directory may be listed multiple times to have the I/O manager use multiple threads for it (for example if it is physically stored on a very fast disc or RAID) (DEFAULT: **The system's tmp dir**). - -- `taskmanager.network.memory.fraction`: Fraction of JVM memory to use for network buffers. This determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value or the min/max values below. Also note, that `taskmanager.network.memory.min` and `taskmanager.network.memory.max` may override this fraction. (DEFAULT: **0.1**) --- End diff -- Why not include the longer description? > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362254#comment-16362254 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5471#discussion_r167853223 --- Diff: docs/ops/config.md --- @@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp ### TaskManager -The following parameters configure Flink's TaskManagers. - - -- `taskmanager.hostname`: The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file. - -- `taskmanager.rpc.port`: The task manager's IPC port (DEFAULT: **0**, which lets the OS choose a free port). Flink also accepts a list of ports ("50100,50101"), ranges ("50100-50200") or a combination of both. It is recommended to set a range of ports to avoid collisions when multiple TaskManagers are running on the same machine. - -- `taskmanager.data.port`: The task manager's port used for data exchange operations (DEFAULT: **0**, which lets the OS choose a free port). - -- `taskmanager.data.ssl.enabled`: Enable SSL support for the taskmanager data transport. This is applicable only when the global ssl flag security.ssl.enabled is set to true (DEFAULT: **true**) - -- `taskmanager.heap.mb`: JVM heap size (in megabytes) for the TaskManagers, which are the parallel workers of the system. In contrast to Hadoop, Flink runs operators (e.g., join, aggregate) and user-defined functions (e.g., Map, Reduce, CoGroup) inside the TaskManager (including sorting/hashing/caching), so this value should be as large as possible (DEFAULT: **512**). On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value. - -- `taskmanager.numberOfTaskSlots`: The number of parallel operator or user function instances that a single TaskManager can run (DEFAULT: **1**). If this value is larger than 1, a single TaskManager takes multiple instances of a function or operator. That way, the TaskManager can utilize multiple CPU cores, but at the same time, the available memory is divided between the different operator or function instances. This value is typically proportional to the number of physical CPU cores that the TaskManager's machine has (e.g., equal to the number of cores, or half the number of cores). - -- `taskmanager.tmp.dirs`: The directory for temporary files, or a list of directories separated by the system's directory delimiter (for example ':' (colon) on Linux/Unix). If multiple directories are specified, then the temporary files will be distributed across the directories in a round robin fashion. The I/O manager component will spawn one reading and one writing thread per directory. A directory may be listed multiple times to have the I/O manager use multiple threads for it (for example if it is physically stored on a very fast disc or RAID) (DEFAULT: **The system's tmp dir**). - -- `taskmanager.network.memory.fraction`: Fraction of JVM memory to use for network buffers. This determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value or the min/max values below. Also note, that `taskmanager.network.memory.min` and `taskmanager.network.memory.max` may override this fraction. (DEFAULT: **0.1**) - -- `taskmanager.network.memory.min`: Minimum memory size for network buffers in bytes (DEFAULT: **64 MB**). Previously, this was determined from `taskmanager.network.numberOfBuffers` and `taskmanager.memory.segment-size`. - -- `taskmanager.network.memory.max`: Maximum memory size for network buffers in bytes (DEFAULT: **1 GB**). Previously, this was determined from `taskmanager.network.numberOfBuffers` and `taskmanager.memory.segment-size`. - -- `taskmanager.network.numberOfBuffers` (deprecated, replaced by the three parameters above): The number of buffers available to the network stack. This number determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value (DEFAULT: **2048**). If set, i
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362250#comment-16362250 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5471#discussion_r167836095 --- Diff: docs/ops/config.md --- @@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp ### TaskManager -The following parameters configure Flink's TaskManagers. - - -- `taskmanager.hostname`: The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file. - -- `taskmanager.rpc.port`: The task manager's IPC port (DEFAULT: **0**, which lets the OS choose a free port). Flink also accepts a list of ports ("50100,50101"), ranges ("50100-50200") or a combination of both. It is recommended to set a range of ports to avoid collisions when multiple TaskManagers are running on the same machine. - -- `taskmanager.data.port`: The task manager's port used for data exchange operations (DEFAULT: **0**, which lets the OS choose a free port). - -- `taskmanager.data.ssl.enabled`: Enable SSL support for the taskmanager data transport. This is applicable only when the global ssl flag security.ssl.enabled is set to true (DEFAULT: **true**) --- End diff -- Why don't we keep `This is applicable only when the global ssl flag security.ssl.enabled is set to true`? > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362260#comment-16362260 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5471#discussion_r167854183 --- Diff: docs/_includes/generated/task_manager_configuration.html --- @@ -0,0 +1,146 @@ + + + +Key +Default +Description + + + + +task.cancellation.interval --- End diff -- The description for those two should probably be carried over from the old doc > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362251#comment-16362251 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5471#discussion_r167835147 --- Diff: docs/ops/config.md --- @@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp ### TaskManager -The following parameters configure Flink's TaskManagers. - - -- `taskmanager.hostname`: The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file. --- End diff -- Why don't we have this longer description for the new `taskmanager.host` setting? > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362258#comment-16362258 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5471#discussion_r167853972 --- Diff: docs/ops/config.md --- @@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp ### TaskManager -The following parameters configure Flink's TaskManagers. - - -- `taskmanager.hostname`: The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file. - -- `taskmanager.rpc.port`: The task manager's IPC port (DEFAULT: **0**, which lets the OS choose a free port). Flink also accepts a list of ports ("50100,50101"), ranges ("50100-50200") or a combination of both. It is recommended to set a range of ports to avoid collisions when multiple TaskManagers are running on the same machine. - -- `taskmanager.data.port`: The task manager's port used for data exchange operations (DEFAULT: **0**, which lets the OS choose a free port). - -- `taskmanager.data.ssl.enabled`: Enable SSL support for the taskmanager data transport. This is applicable only when the global ssl flag security.ssl.enabled is set to true (DEFAULT: **true**) - -- `taskmanager.heap.mb`: JVM heap size (in megabytes) for the TaskManagers, which are the parallel workers of the system. In contrast to Hadoop, Flink runs operators (e.g., join, aggregate) and user-defined functions (e.g., Map, Reduce, CoGroup) inside the TaskManager (including sorting/hashing/caching), so this value should be as large as possible (DEFAULT: **512**). On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value. - -- `taskmanager.numberOfTaskSlots`: The number of parallel operator or user function instances that a single TaskManager can run (DEFAULT: **1**). If this value is larger than 1, a single TaskManager takes multiple instances of a function or operator. That way, the TaskManager can utilize multiple CPU cores, but at the same time, the available memory is divided between the different operator or function instances. This value is typically proportional to the number of physical CPU cores that the TaskManager's machine has (e.g., equal to the number of cores, or half the number of cores). - -- `taskmanager.tmp.dirs`: The directory for temporary files, or a list of directories separated by the system's directory delimiter (for example ':' (colon) on Linux/Unix). If multiple directories are specified, then the temporary files will be distributed across the directories in a round robin fashion. The I/O manager component will spawn one reading and one writing thread per directory. A directory may be listed multiple times to have the I/O manager use multiple threads for it (for example if it is physically stored on a very fast disc or RAID) (DEFAULT: **The system's tmp dir**). - -- `taskmanager.network.memory.fraction`: Fraction of JVM memory to use for network buffers. This determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value or the min/max values below. Also note, that `taskmanager.network.memory.min` and `taskmanager.network.memory.max` may override this fraction. (DEFAULT: **0.1**) - -- `taskmanager.network.memory.min`: Minimum memory size for network buffers in bytes (DEFAULT: **64 MB**). Previously, this was determined from `taskmanager.network.numberOfBuffers` and `taskmanager.memory.segment-size`. - -- `taskmanager.network.memory.max`: Maximum memory size for network buffers in bytes (DEFAULT: **1 GB**). Previously, this was determined from `taskmanager.network.numberOfBuffers` and `taskmanager.memory.segment-size`. - -- `taskmanager.network.numberOfBuffers` (deprecated, replaced by the three parameters above): The number of buffers available to the network stack. This number determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value (DEFAULT: **2048**). If set, i
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362253#comment-16362253 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5471#discussion_r167836313 --- Diff: docs/ops/config.md --- @@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp ### TaskManager -The following parameters configure Flink's TaskManagers. - - -- `taskmanager.hostname`: The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file. - -- `taskmanager.rpc.port`: The task manager's IPC port (DEFAULT: **0**, which lets the OS choose a free port). Flink also accepts a list of ports ("50100,50101"), ranges ("50100-50200") or a combination of both. It is recommended to set a range of ports to avoid collisions when multiple TaskManagers are running on the same machine. - -- `taskmanager.data.port`: The task manager's port used for data exchange operations (DEFAULT: **0**, which lets the OS choose a free port). - -- `taskmanager.data.ssl.enabled`: Enable SSL support for the taskmanager data transport. This is applicable only when the global ssl flag security.ssl.enabled is set to true (DEFAULT: **true**) - -- `taskmanager.heap.mb`: JVM heap size (in megabytes) for the TaskManagers, which are the parallel workers of the system. In contrast to Hadoop, Flink runs operators (e.g., join, aggregate) and user-defined functions (e.g., Map, Reduce, CoGroup) inside the TaskManager (including sorting/hashing/caching), so this value should be as large as possible (DEFAULT: **512**). On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value. - -- `taskmanager.numberOfTaskSlots`: The number of parallel operator or user function instances that a single TaskManager can run (DEFAULT: **1**). If this value is larger than 1, a single TaskManager takes multiple instances of a function or operator. That way, the TaskManager can utilize multiple CPU cores, but at the same time, the available memory is divided between the different operator or function instances. This value is typically proportional to the number of physical CPU cores that the TaskManager's machine has (e.g., equal to the number of cores, or half the number of cores). --- End diff -- Why not include this one in the new description? > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362257#comment-16362257 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5471#discussion_r167853874 --- Diff: docs/ops/config.md --- @@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp ### TaskManager -The following parameters configure Flink's TaskManagers. - - -- `taskmanager.hostname`: The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file. - -- `taskmanager.rpc.port`: The task manager's IPC port (DEFAULT: **0**, which lets the OS choose a free port). Flink also accepts a list of ports ("50100,50101"), ranges ("50100-50200") or a combination of both. It is recommended to set a range of ports to avoid collisions when multiple TaskManagers are running on the same machine. - -- `taskmanager.data.port`: The task manager's port used for data exchange operations (DEFAULT: **0**, which lets the OS choose a free port). - -- `taskmanager.data.ssl.enabled`: Enable SSL support for the taskmanager data transport. This is applicable only when the global ssl flag security.ssl.enabled is set to true (DEFAULT: **true**) - -- `taskmanager.heap.mb`: JVM heap size (in megabytes) for the TaskManagers, which are the parallel workers of the system. In contrast to Hadoop, Flink runs operators (e.g., join, aggregate) and user-defined functions (e.g., Map, Reduce, CoGroup) inside the TaskManager (including sorting/hashing/caching), so this value should be as large as possible (DEFAULT: **512**). On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value. - -- `taskmanager.numberOfTaskSlots`: The number of parallel operator or user function instances that a single TaskManager can run (DEFAULT: **1**). If this value is larger than 1, a single TaskManager takes multiple instances of a function or operator. That way, the TaskManager can utilize multiple CPU cores, but at the same time, the available memory is divided between the different operator or function instances. This value is typically proportional to the number of physical CPU cores that the TaskManager's machine has (e.g., equal to the number of cores, or half the number of cores). - -- `taskmanager.tmp.dirs`: The directory for temporary files, or a list of directories separated by the system's directory delimiter (for example ':' (colon) on Linux/Unix). If multiple directories are specified, then the temporary files will be distributed across the directories in a round robin fashion. The I/O manager component will spawn one reading and one writing thread per directory. A directory may be listed multiple times to have the I/O manager use multiple threads for it (for example if it is physically stored on a very fast disc or RAID) (DEFAULT: **The system's tmp dir**). - -- `taskmanager.network.memory.fraction`: Fraction of JVM memory to use for network buffers. This determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value or the min/max values below. Also note, that `taskmanager.network.memory.min` and `taskmanager.network.memory.max` may override this fraction. (DEFAULT: **0.1**) - -- `taskmanager.network.memory.min`: Minimum memory size for network buffers in bytes (DEFAULT: **64 MB**). Previously, this was determined from `taskmanager.network.numberOfBuffers` and `taskmanager.memory.segment-size`. - -- `taskmanager.network.memory.max`: Maximum memory size for network buffers in bytes (DEFAULT: **1 GB**). Previously, this was determined from `taskmanager.network.numberOfBuffers` and `taskmanager.memory.segment-size`. - -- `taskmanager.network.numberOfBuffers` (deprecated, replaced by the three parameters above): The number of buffers available to the network stack. This number determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value (DEFAULT: **2048**). If set, i
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362249#comment-16362249 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5471#discussion_r167836229 --- Diff: docs/ops/config.md --- @@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp ### TaskManager -The following parameters configure Flink's TaskManagers. - - -- `taskmanager.hostname`: The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file. - -- `taskmanager.rpc.port`: The task manager's IPC port (DEFAULT: **0**, which lets the OS choose a free port). Flink also accepts a list of ports ("50100,50101"), ranges ("50100-50200") or a combination of both. It is recommended to set a range of ports to avoid collisions when multiple TaskManagers are running on the same machine. - -- `taskmanager.data.port`: The task manager's port used for data exchange operations (DEFAULT: **0**, which lets the OS choose a free port). - -- `taskmanager.data.ssl.enabled`: Enable SSL support for the taskmanager data transport. This is applicable only when the global ssl flag security.ssl.enabled is set to true (DEFAULT: **true**) - -- `taskmanager.heap.mb`: JVM heap size (in megabytes) for the TaskManagers, which are the parallel workers of the system. In contrast to Hadoop, Flink runs operators (e.g., join, aggregate) and user-defined functions (e.g., Map, Reduce, CoGroup) inside the TaskManager (including sorting/hashing/caching), so this value should be as large as possible (DEFAULT: **512**). On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value. --- End diff -- You didn't want to include that whole description of how Flink works? 😉 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362259#comment-16362259 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5471#discussion_r167854639 --- Diff: docs/ops/config.md --- @@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp ### TaskManager -The following parameters configure Flink's TaskManagers. - - -- `taskmanager.hostname`: The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file. - -- `taskmanager.rpc.port`: The task manager's IPC port (DEFAULT: **0**, which lets the OS choose a free port). Flink also accepts a list of ports ("50100,50101"), ranges ("50100-50200") or a combination of both. It is recommended to set a range of ports to avoid collisions when multiple TaskManagers are running on the same machine. - -- `taskmanager.data.port`: The task manager's port used for data exchange operations (DEFAULT: **0**, which lets the OS choose a free port). - -- `taskmanager.data.ssl.enabled`: Enable SSL support for the taskmanager data transport. This is applicable only when the global ssl flag security.ssl.enabled is set to true (DEFAULT: **true**) - -- `taskmanager.heap.mb`: JVM heap size (in megabytes) for the TaskManagers, which are the parallel workers of the system. In contrast to Hadoop, Flink runs operators (e.g., join, aggregate) and user-defined functions (e.g., Map, Reduce, CoGroup) inside the TaskManager (including sorting/hashing/caching), so this value should be as large as possible (DEFAULT: **512**). On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value. - -- `taskmanager.numberOfTaskSlots`: The number of parallel operator or user function instances that a single TaskManager can run (DEFAULT: **1**). If this value is larger than 1, a single TaskManager takes multiple instances of a function or operator. That way, the TaskManager can utilize multiple CPU cores, but at the same time, the available memory is divided between the different operator or function instances. This value is typically proportional to the number of physical CPU cores that the TaskManager's machine has (e.g., equal to the number of cores, or half the number of cores). - -- `taskmanager.tmp.dirs`: The directory for temporary files, or a list of directories separated by the system's directory delimiter (for example ':' (colon) on Linux/Unix). If multiple directories are specified, then the temporary files will be distributed across the directories in a round robin fashion. The I/O manager component will spawn one reading and one writing thread per directory. A directory may be listed multiple times to have the I/O manager use multiple threads for it (for example if it is physically stored on a very fast disc or RAID) (DEFAULT: **The system's tmp dir**). - -- `taskmanager.network.memory.fraction`: Fraction of JVM memory to use for network buffers. This determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value or the min/max values below. Also note, that `taskmanager.network.memory.min` and `taskmanager.network.memory.max` may override this fraction. (DEFAULT: **0.1**) - -- `taskmanager.network.memory.min`: Minimum memory size for network buffers in bytes (DEFAULT: **64 MB**). Previously, this was determined from `taskmanager.network.numberOfBuffers` and `taskmanager.memory.segment-size`. - -- `taskmanager.network.memory.max`: Maximum memory size for network buffers in bytes (DEFAULT: **1 GB**). Previously, this was determined from `taskmanager.network.numberOfBuffers` and `taskmanager.memory.segment-size`. - -- `taskmanager.network.numberOfBuffers` (deprecated, replaced by the three parameters above): The number of buffers available to the network stack. This number determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value (DEFAULT: **2048**). If set, i
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362256#comment-16362256 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5471#discussion_r167853447 --- Diff: docs/ops/config.md --- @@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp ### TaskManager -The following parameters configure Flink's TaskManagers. - - -- `taskmanager.hostname`: The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file. - -- `taskmanager.rpc.port`: The task manager's IPC port (DEFAULT: **0**, which lets the OS choose a free port). Flink also accepts a list of ports ("50100,50101"), ranges ("50100-50200") or a combination of both. It is recommended to set a range of ports to avoid collisions when multiple TaskManagers are running on the same machine. - -- `taskmanager.data.port`: The task manager's port used for data exchange operations (DEFAULT: **0**, which lets the OS choose a free port). - -- `taskmanager.data.ssl.enabled`: Enable SSL support for the taskmanager data transport. This is applicable only when the global ssl flag security.ssl.enabled is set to true (DEFAULT: **true**) - -- `taskmanager.heap.mb`: JVM heap size (in megabytes) for the TaskManagers, which are the parallel workers of the system. In contrast to Hadoop, Flink runs operators (e.g., join, aggregate) and user-defined functions (e.g., Map, Reduce, CoGroup) inside the TaskManager (including sorting/hashing/caching), so this value should be as large as possible (DEFAULT: **512**). On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value. - -- `taskmanager.numberOfTaskSlots`: The number of parallel operator or user function instances that a single TaskManager can run (DEFAULT: **1**). If this value is larger than 1, a single TaskManager takes multiple instances of a function or operator. That way, the TaskManager can utilize multiple CPU cores, but at the same time, the available memory is divided between the different operator or function instances. This value is typically proportional to the number of physical CPU cores that the TaskManager's machine has (e.g., equal to the number of cores, or half the number of cores). - -- `taskmanager.tmp.dirs`: The directory for temporary files, or a list of directories separated by the system's directory delimiter (for example ':' (colon) on Linux/Unix). If multiple directories are specified, then the temporary files will be distributed across the directories in a round robin fashion. The I/O manager component will spawn one reading and one writing thread per directory. A directory may be listed multiple times to have the I/O manager use multiple threads for it (for example if it is physically stored on a very fast disc or RAID) (DEFAULT: **The system's tmp dir**). - -- `taskmanager.network.memory.fraction`: Fraction of JVM memory to use for network buffers. This determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value or the min/max values below. Also note, that `taskmanager.network.memory.min` and `taskmanager.network.memory.max` may override this fraction. (DEFAULT: **0.1**) - -- `taskmanager.network.memory.min`: Minimum memory size for network buffers in bytes (DEFAULT: **64 MB**). Previously, this was determined from `taskmanager.network.numberOfBuffers` and `taskmanager.memory.segment-size`. - -- `taskmanager.network.memory.max`: Maximum memory size for network buffers in bytes (DEFAULT: **1 GB**). Previously, this was determined from `taskmanager.network.numberOfBuffers` and `taskmanager.memory.segment-size`. - -- `taskmanager.network.numberOfBuffers` (deprecated, replaced by the three parameters above): The number of buffers available to the network stack. This number determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value (DEFAULT: **2048**). If set, i
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362248#comment-16362248 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5471#discussion_r167836369 --- Diff: docs/ops/config.md --- @@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp ### TaskManager -The following parameters configure Flink's TaskManagers. - - -- `taskmanager.hostname`: The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file. - -- `taskmanager.rpc.port`: The task manager's IPC port (DEFAULT: **0**, which lets the OS choose a free port). Flink also accepts a list of ports ("50100,50101"), ranges ("50100-50200") or a combination of both. It is recommended to set a range of ports to avoid collisions when multiple TaskManagers are running on the same machine. - -- `taskmanager.data.port`: The task manager's port used for data exchange operations (DEFAULT: **0**, which lets the OS choose a free port). - -- `taskmanager.data.ssl.enabled`: Enable SSL support for the taskmanager data transport. This is applicable only when the global ssl flag security.ssl.enabled is set to true (DEFAULT: **true**) - -- `taskmanager.heap.mb`: JVM heap size (in megabytes) for the TaskManagers, which are the parallel workers of the system. In contrast to Hadoop, Flink runs operators (e.g., join, aggregate) and user-defined functions (e.g., Map, Reduce, CoGroup) inside the TaskManager (including sorting/hashing/caching), so this value should be as large as possible (DEFAULT: **512**). On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value. - -- `taskmanager.numberOfTaskSlots`: The number of parallel operator or user function instances that a single TaskManager can run (DEFAULT: **1**). If this value is larger than 1, a single TaskManager takes multiple instances of a function or operator. That way, the TaskManager can utilize multiple CPU cores, but at the same time, the available memory is divided between the different operator or function instances. This value is typically proportional to the number of physical CPU cores that the TaskManager's machine has (e.g., equal to the number of cores, or half the number of cores). - -- `taskmanager.tmp.dirs`: The directory for temporary files, or a list of directories separated by the system's directory delimiter (for example ':' (colon) on Linux/Unix). If multiple directories are specified, then the temporary files will be distributed across the directories in a round robin fashion. The I/O manager component will spawn one reading and one writing thread per directory. A directory may be listed multiple times to have the I/O manager use multiple threads for it (for example if it is physically stored on a very fast disc or RAID) (DEFAULT: **The system's tmp dir**). --- End diff -- Does this not work anymore? > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362255#comment-16362255 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5471#discussion_r167853029 --- Diff: docs/ops/config.md --- @@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp ### TaskManager -The following parameters configure Flink's TaskManagers. - - -- `taskmanager.hostname`: The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file. - -- `taskmanager.rpc.port`: The task manager's IPC port (DEFAULT: **0**, which lets the OS choose a free port). Flink also accepts a list of ports ("50100,50101"), ranges ("50100-50200") or a combination of both. It is recommended to set a range of ports to avoid collisions when multiple TaskManagers are running on the same machine. - -- `taskmanager.data.port`: The task manager's port used for data exchange operations (DEFAULT: **0**, which lets the OS choose a free port). - -- `taskmanager.data.ssl.enabled`: Enable SSL support for the taskmanager data transport. This is applicable only when the global ssl flag security.ssl.enabled is set to true (DEFAULT: **true**) - -- `taskmanager.heap.mb`: JVM heap size (in megabytes) for the TaskManagers, which are the parallel workers of the system. In contrast to Hadoop, Flink runs operators (e.g., join, aggregate) and user-defined functions (e.g., Map, Reduce, CoGroup) inside the TaskManager (including sorting/hashing/caching), so this value should be as large as possible (DEFAULT: **512**). On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value. - -- `taskmanager.numberOfTaskSlots`: The number of parallel operator or user function instances that a single TaskManager can run (DEFAULT: **1**). If this value is larger than 1, a single TaskManager takes multiple instances of a function or operator. That way, the TaskManager can utilize multiple CPU cores, but at the same time, the available memory is divided between the different operator or function instances. This value is typically proportional to the number of physical CPU cores that the TaskManager's machine has (e.g., equal to the number of cores, or half the number of cores). - -- `taskmanager.tmp.dirs`: The directory for temporary files, or a list of directories separated by the system's directory delimiter (for example ':' (colon) on Linux/Unix). If multiple directories are specified, then the temporary files will be distributed across the directories in a round robin fashion. The I/O manager component will spawn one reading and one writing thread per directory. A directory may be listed multiple times to have the I/O manager use multiple threads for it (for example if it is physically stored on a very fast disc or RAID) (DEFAULT: **The system's tmp dir**). - -- `taskmanager.network.memory.fraction`: Fraction of JVM memory to use for network buffers. This determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value or the min/max values below. Also note, that `taskmanager.network.memory.min` and `taskmanager.network.memory.max` may override this fraction. (DEFAULT: **0.1**) --- End diff -- This question is probably true for a lot of these changes. > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362185#comment-16362185 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5470#discussion_r167839893 --- Diff: docs/_includes/generated/resource_manager_configuration.html --- @@ -0,0 +1,36 @@ + + + +Key +Default +Description + + + + +containerized.heap-cutoff-min +600 +Minimum amount of heap memory to remove in containers, as a safety margin. + + +containerized.heap-cutoff-ratio +0.25 +Percentage of heap space to remove from containers (YARN / Mesos), to compensate for other JVM memory usage. + + +local.number-resourcemanager +1 + + + +resourcemanager.job.timeout +"5 minutes" +Timeout for jobs which don't have a job manager as leader assigned. + + +resourcemanager.rpc.port --- End diff -- Must've missed it by accident, added it back. > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362187#comment-16362187 ] ASF GitHub Bot commented on FLINK-8475: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5463 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362186#comment-16362186 ] ASF GitHub Bot commented on FLINK-8475: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5462 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362178#comment-16362178 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5467#discussion_r167838921 --- Diff: docs/_includes/generated/high_availability_configuration.html --- @@ -0,0 +1,36 @@ + + + +Key +Default +Description + + + + +high-availability +"NONE" +Defines high-availability mode used for the cluster execution. To enable high-availability, set this mode to "ZOOKEEPER". --- End diff -- high-availability: how Zookeeper-based HA works is/should be explained in the [HA docs](https://ci.apache.org/projects/flink/flink-docs-master/ops/jobmanager_high_availability.html#yarn-cluster-high-availability) cluster-id: derived from javadocs, as the existing docs were entirely about ZK setups job.delay: derived from javadocs, as i assume it to be more up-to-date than the ZK specific description jobmanager.port: undocumented before storageDir: derived from javadocs, as i assume it to be more up-to-date than the ZK specific description > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362172#comment-16362172 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5463 merging. > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362171#comment-16362171 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5462 merging. > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362166#comment-16362166 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5469 I only did that for the sake of this PR to avoid merge conflicts; this won't be necessary in the future. > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362158#comment-16362158 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5468#discussion_r167833518 --- Diff: docs/_includes/generated/environment_configuration.html --- @@ -0,0 +1,41 @@ + + + +Key +Default +Description + + + + +env.java.opts +(none) + --- End diff -- There are plenty of missing descriptions, but I wanted to do that as a follow-up to get the basic setup in place first. > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362157#comment-16362157 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5470#discussion_r167833506 --- Diff: docs/_includes/generated/resource_manager_configuration.html --- @@ -0,0 +1,36 @@ + + + +Key +Default +Description + + + + +containerized.heap-cutoff-min +600 +Minimum amount of heap memory to remove in containers, as a safety margin. + + +containerized.heap-cutoff-ratio +0.25 +Percentage of heap space to remove from containers (YARN / Mesos), to compensate for other JVM memory usage. + + +local.number-resourcemanager +1 + + + +resourcemanager.job.timeout +"5 minutes" +Timeout for jobs which don't have a job manager as leader assigned. + + +resourcemanager.rpc.port --- End diff -- Why does this not have the description from the original doc? > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362152#comment-16362152 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/5469 Couldn't these be called "Classloader options" and have their own group? That way you don't have to manually mess with the generated table. > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362145#comment-16362145 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5468#discussion_r167831861 --- Diff: docs/_includes/generated/environment_configuration.html --- @@ -0,0 +1,41 @@ + + + +Key +Default +Description + + + + +env.java.opts +(none) + --- End diff -- We could add a short descriptions about what these `opts` things actually do. > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362141#comment-16362141 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5467#discussion_r167831138 --- Diff: docs/_includes/generated/high_availability_configuration.html --- @@ -0,0 +1,36 @@ + + + +Key +Default +Description + + + + +high-availability +"NONE" +Defines high-availability mode used for the cluster execution. To enable high-availability, set this mode to "ZOOKEEPER". --- End diff -- Why do these not have all the description that the original doc has? > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362142#comment-16362142 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5459#discussion_r167831341 --- Diff: flink-core/src/main/java/org/apache/flink/configuration/CoreOptions.java --- @@ -127,7 +130,30 @@ */ public static final ConfigOption DEFAULT_FILESYSTEM_SCHEME = ConfigOptions .key("fs.default-scheme") - .noDefaultValue(); + .noDefaultValue() + .withDescription("The default filesystem scheme, used for paths that do not declare a scheme explicitly."); --- End diff -- Yes, in a practical sense "file:///" is the default due to how `FileSystem` works. We can't define it as the official default at the moment because of OS compatibility, see LocalFileSystem: ``` LOCAL_URI = OperatingSystem.isWindows() ? URI.create("file:/") : URI.create("file:///"); ``` Not sure whether that is really necessary though, but the scheme parsing by `Paths` is a [bit wonky on Windows](https://issues.apache.org/jira/browse/FLINK-6889). > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362137#comment-16362137 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/5463 Ok, please go ahead, then. 👍 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362129#comment-16362129 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5459#discussion_r167826760 --- Diff: flink-core/src/main/java/org/apache/flink/configuration/CoreOptions.java --- @@ -127,7 +130,30 @@ */ public static final ConfigOption DEFAULT_FILESYSTEM_SCHEME = ConfigOptions .key("fs.default-scheme") - .noDefaultValue(); + .noDefaultValue() + .withDescription("The default filesystem scheme, used for paths that do not declare a scheme explicitly."); --- End diff -- Ah, because we have no default in here but isn't the `FileSystem` code using that as the default fallback? > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362119#comment-16362119 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/5462 Ok, looks good now. 👍 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362002#comment-16362002 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5462#discussion_r167793610 --- Diff: docs/ops/config.md --- @@ -502,27 +502,7 @@ Previously this key was named `recovery.mode` and the default value was `standal ZooKeeper-based HA Mode -- `high-availability.zookeeper.quorum`: Defines the ZooKeeper quorum URL which is used to connect to the ZooKeeper cluster when the 'zookeeper' HA mode is selected. Previously this key was named `recovery.zookeeper.quorum`. - -- `high-availability.zookeeper.path.root`: (Default `/flink`) Defines the root dir under which the ZooKeeper HA mode will create namespace directories. Previously this ket was named `recovery.zookeeper.path.root`. - -- `high-availability.zookeeper.path.latch`: (Default `/leaderlatch`) Defines the znode of the leader latch which is used to elect the leader. Previously this key was named `recovery.zookeeper.path.latch`. - -- `high-availability.zookeeper.path.leader`: (Default `/leader`) Defines the znode of the leader which contains the URL to the leader and the current leader session ID. Previously this key was named `recovery.zookeeper.path.leader`. - -- `high-availability.storageDir`: Defines the directory in the state backend where the JobManager metadata will be stored (ZooKeeper only keeps pointers to it). Required for HA. Previously this key was named `recovery.zookeeper.storageDir` and `high-availability.zookeeper.storageDir`. - -- `high-availability.zookeeper.client.session-timeout`: (Default `6`) Defines the session timeout for the ZooKeeper session in ms. Previously this key was named `recovery.zookeeper.client.session-timeout` - -- `high-availability.zookeeper.client.connection-timeout`: (Default `15000`) Defines the connection timeout for ZooKeeper in ms. Previously this key was named `recovery.zookeeper.client.connection-timeout`. - -- `high-availability.zookeeper.client.retry-wait`: (Default `5000`) Defines the pause between consecutive retries in ms. Previously this key was named `recovery.zookeeper.client.retry-wait`. - -- `high-availability.zookeeper.client.max-retry-attempts`: (Default `3`) Defines the number of connection retries before the client gives up. Previously this key was named `recovery.zookeeper.client.max-retry-attempts`. - -- `high-availability.job.delay`: (Default `akka.ask.timeout`) Defines the delay before persisted jobs are recovered in case of a master recovery situation. Previously this key was named `recovery.job.delay`. --- End diff -- re-added in #5467 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362000#comment-16362000 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5462#discussion_r167793417 --- Diff: docs/ops/config.md --- @@ -502,27 +502,7 @@ Previously this key was named `recovery.mode` and the default value was `standal ZooKeeper-based HA Mode -- `high-availability.zookeeper.quorum`: Defines the ZooKeeper quorum URL which is used to connect to the ZooKeeper cluster when the 'zookeeper' HA mode is selected. Previously this key was named `recovery.zookeeper.quorum`. - -- `high-availability.zookeeper.path.root`: (Default `/flink`) Defines the root dir under which the ZooKeeper HA mode will create namespace directories. Previously this ket was named `recovery.zookeeper.path.root`. - -- `high-availability.zookeeper.path.latch`: (Default `/leaderlatch`) Defines the znode of the leader latch which is used to elect the leader. Previously this key was named `recovery.zookeeper.path.latch`. - -- `high-availability.zookeeper.path.leader`: (Default `/leader`) Defines the znode of the leader which contains the URL to the leader and the current leader session ID. Previously this key was named `recovery.zookeeper.path.leader`. - -- `high-availability.storageDir`: Defines the directory in the state backend where the JobManager metadata will be stored (ZooKeeper only keeps pointers to it). Required for HA. Previously this key was named `recovery.zookeeper.storageDir` and `high-availability.zookeeper.storageDir`. --- End diff -- scratch the previous comment (wasn't too helpful was it :D ); this option will now be listed in the general HA section. > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361980#comment-16361980 ] ASF GitHub Bot commented on FLINK-8475: --- GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/5471 [FLINK-8475][config][docs] Integrate TM options ## What is the purpose of the change This PR integrates the TaskManager `ConfigOptions` into the configuration docs generator. ## Brief change log * Port missing taskmanager config constants to config options * Add missing descriptions to config options (derived from existing description/javadocs) * integrate TaskManager configuration table into `config.md` You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 8475_tm Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5471.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5471 commit edb996e1c21962e6d087c941e8d898879021b9fc Author: zentol Date: 2018-01-22T17:15:34Z [FLINK-8475][config][docs] Integrate TM options > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361976#comment-16361976 ] ASF GitHub Bot commented on FLINK-8475: --- GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/5470 [FLINK-8475][config][docs] Integrate RM options ## What is the purpose of the change This PR integrates the RM `ConfigOptions` into the configuration docs generator. ## Brief change log * Add missing descriptions to config options (derived from existing description/javadocs) * integrate RM configuration table into `config.md` You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 8475_rm Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5470.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5470 commit 9f6baefbcbd2cbc5e01cf07f52145443ac4b121f Author: zentol Date: 2018-01-30T13:49:13Z [FLINK-8475][config][docs] Integrate RM options > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361174#comment-16361174 ] Chesnay Schepler commented on FLINK-8475: - partially addressed in: d477786329bc0c3bc9ed75ca1586ce9be6321f98 023ab749cc2d82fcf729d3e3f8052c60280f0af3 99c07d967a7a27ce06ded601cbc4c36d1b6702e2 7910d0544b0dc74a0720055a4b38f10e63dc569c b5d7d7c6bf2b5c4ae157358aecb335a7c06dd41b 00cae7089ff698f23644169ccdb30f100aeb1545 a8f64fb43ab4d3820b10b546a660d2767577a4f9 10ad8c34d6c88d44e39428f0eea49e54f76fd7fc a0e961dff70627e02e559c526ba8fbb0943c56b4 3f2a065d838249b4aa24135dfcbd2d42fbd86651 92026ff392e527a57657d67a9a98adaf96df9643 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361166#comment-16361166 ] ASF GitHub Bot commented on FLINK-8475: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5386 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361169#comment-16361169 ] ASF GitHub Bot commented on FLINK-8475: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5390 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361168#comment-16361168 ] ASF GitHub Bot commented on FLINK-8475: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5389 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361173#comment-16361173 ] ASF GitHub Bot commented on FLINK-8475: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5385 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361171#comment-16361171 ] ASF GitHub Bot commented on FLINK-8475: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5391 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361170#comment-16361170 ] ASF GitHub Bot commented on FLINK-8475: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5392 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361172#comment-16361172 ] ASF GitHub Bot commented on FLINK-8475: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5464 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361167#comment-16361167 ] ASF GitHub Bot commented on FLINK-8475: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5387 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361165#comment-16361165 ] ASF GitHub Bot commented on FLINK-8475: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5460 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361164#comment-16361164 ] ASF GitHub Bot commented on FLINK-8475: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5461 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361126#comment-16361126 ] ASF GitHub Bot commented on FLINK-8475: --- GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/5469 [FLINK-8475][config][docs] Integrate Core options ## What is the purpose of the change This PR integrates the core `ConfigOptions` into the configuration docs generator. Do note that i cheated a bit and manually cut out the fs/env options from the table that are moved to separate tables in #5468 and #5459. ## Brief change log * Add missing descriptions to config options (derived from existing description/javadocs) * integrate core configuration table into `config.md` * added `CoreOptions#TMP_DIRS` to the set of options for which the generated overrides the default You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 8475_core Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5469.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5469 commit 037c8bc93193c7aefb7a56412bba74fdbf39e92d Author: zentol Date: 2018-01-30T13:58:32Z [FLINK-8475][config][docs] Integrate Core options > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361112#comment-16361112 ] ASF GitHub Bot commented on FLINK-8475: --- GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/5468 [FLINK-8475][config][docs] Integrate Environment options ## What is the purpose of the change This PR integrates the environment `ConfigOptions` into the configuration docs generator. Note that the newly added config options are only evaluated in the shell scripts, which is rather unfortunate and may cause the docs to be outdated should we not find a way to guard this with tests. ## Brief change log * Add missing descriptions to config options (derived from existing description/javadocs) * integrate environment configuration table into `config.md` You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 8475_env Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5468.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5468 commit 0393908861b3d311fbf0761261260ebefffbd6eb Author: zentol Date: 2018-01-30T13:45:10Z [FLINK-8475][config][docs] Integrate Environment options > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361109#comment-16361109 ] ASF GitHub Bot commented on FLINK-8475: --- GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/5467 [FLINK-8475][config][docs] Integrate HA options ## What is the purpose of the change This PR integrates the HA `ConfigOptions` into the configuration docs generator. ## Brief change log * Add missing descriptions to config options (derived from existing description/javadocs) * integrate HA configuration table into `config.md` You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 8475_ha Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5467.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5467 commit ac26a32d8fa947706e8bc6f279a1fc5603c5cc41 Author: zentol Date: 2018-01-30T13:38:10Z [FLINK-8475][config][docs] Integrate HA options > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361060#comment-16361060 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5385 merging, > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361046#comment-16361046 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5463#discussion_r167615886 --- Diff: docs/ops/config.md --- @@ -408,38 +408,7 @@ of the JobManager, because the same ActorSystem is used. Its not possible to use ### YARN -- `containerized.heap-cutoff-ratio`: (Default 0.25) Percentage of heap space to remove from containers started by YARN. When a user requests a certain amount of memory for each TaskManager container (for example 4 GB), we can not pass this amount as the maximum heap space for the JVM (`-Xmx` argument) because the JVM is also allocating memory outside the heap. YARN is very strict with killing containers which are using more memory than requested. Therefore, we remove this fraction of the memory from the requested heap as a safety margin and add it to the memory used off-heap. - -- `containerized.heap-cutoff-min`: (Default 600 MB) Minimum amount of memory to cut off the requested heap size. - -- `yarn.maximum-failed-containers` (Default: number of requested containers). Maximum number of containers the system is going to reallocate in case of a failure. - -- `yarn.application-attempts` (Default: 1). Number of ApplicationMaster restarts. Note that that the entire Flink cluster will restart and the YARN Client will loose the connection. Also, the JobManager address will change and you'll need to set the JM host:port manually. It is recommended to leave this option at 1. - -- `yarn.heartbeat-delay` (Default: 5 seconds). Time between heartbeats with the ResourceManager. - -- `yarn.properties-file.location` (Default: temp directory). When a Flink job is submitted to YARN, the JobManager's host and the number of available processing slots is written into a properties file, so that the Flink client is able to pick those details up. This configuration parameter allows changing the default location of that file (for example for environments sharing a Flink installation between users) - -- `yarn.containers.vcores` The number of virtual cores (vcores) per YARN container. By default, the number of `vcores` is set to the number of slots per TaskManager, if set, or to 1, otherwise. - -- `containerized.master.env.`*ENV_VAR1=value* Configuration values prefixed with `containerized.master.env.` will be passed as environment variables to the ApplicationMaster/JobManager process. For example for passing `LD_LIBRARY_PATH` as an env variable to the ApplicationMaster, set: - -`containerized.master.env.LD_LIBRARY_PATH: "/usr/lib/native"` - -- `containerized.taskmanager.env.` Similar to the configuration prefix about, this prefix allows setting custom environment variables for the TaskManager processes. - -- `yarn.container-start-command-template`: Flink uses the following template when starting on YARN: --- End diff -- and porting that one is tricky since it is used by flink-runtime but the yarn options are in flink-yarn... > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361037#comment-16361037 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5463#discussion_r167613681 --- Diff: docs/ops/config.md --- @@ -408,38 +408,7 @@ of the JobManager, because the same ActorSystem is used. Its not possible to use ### YARN -- `containerized.heap-cutoff-ratio`: (Default 0.25) Percentage of heap space to remove from containers started by YARN. When a user requests a certain amount of memory for each TaskManager container (for example 4 GB), we can not pass this amount as the maximum heap space for the JVM (`-Xmx` argument) because the JVM is also allocating memory outside the heap. YARN is very strict with killing containers which are using more memory than requested. Therefore, we remove this fraction of the memory from the requested heap as a safety margin and add it to the memory used off-heap. - -- `containerized.heap-cutoff-min`: (Default 600 MB) Minimum amount of memory to cut off the requested heap size. - -- `yarn.maximum-failed-containers` (Default: number of requested containers). Maximum number of containers the system is going to reallocate in case of a failure. - -- `yarn.application-attempts` (Default: 1). Number of ApplicationMaster restarts. Note that that the entire Flink cluster will restart and the YARN Client will loose the connection. Also, the JobManager address will change and you'll need to set the JM host:port manually. It is recommended to leave this option at 1. - -- `yarn.heartbeat-delay` (Default: 5 seconds). Time between heartbeats with the ResourceManager. - -- `yarn.properties-file.location` (Default: temp directory). When a Flink job is submitted to YARN, the JobManager's host and the number of available processing slots is written into a properties file, so that the Flink client is able to pick those details up. This configuration parameter allows changing the default location of that file (for example for environments sharing a Flink installation between users) - -- `yarn.containers.vcores` The number of virtual cores (vcores) per YARN container. By default, the number of `vcores` is set to the number of slots per TaskManager, if set, or to 1, otherwise. - -- `containerized.master.env.`*ENV_VAR1=value* Configuration values prefixed with `containerized.master.env.` will be passed as environment variables to the ApplicationMaster/JobManager process. For example for passing `LD_LIBRARY_PATH` as an env variable to the ApplicationMaster, set: - -`containerized.master.env.LD_LIBRARY_PATH: "/usr/lib/native"` - -- `containerized.taskmanager.env.` Similar to the configuration prefix about, this prefix allows setting custom environment variables for the TaskManager processes. - -- `yarn.container-start-command-template`: Flink uses the following template when starting on YARN: -`%java% %jvmmem% %jvmopts% %logging% %class% %args% %redirects%`. This configuration parameter allows users -to pass custom settings (such as JVM paths, arguments etc.). Note that in most cases, it is sufficient to -use the `env.java.opts` setting, which is the `%jvmopts%` variable in the String. - -- `yarn.application-master.port` (Default: 0, which lets the OS choose an ephemeral port) With this configuration option, users can specify a port, a range of ports or a list of ports for the Application Master (and JobManager) RPC port. By default we recommend using the default value (0) to let the operating system choose an appropriate port. In particular when multiple AMs are running on the same physical host, fixed port assignments prevent the AM from starting. - - For example when running Flink on YARN on an environment with a restrictive firewall, this option allows specifying a range of allowed ports. - -- `yarn.tags` A comma-separated list of tags to apply to the Flink YARN application. - -- `yarn.per-job-cluster.include-user-jar` (Default: ORDER) Control whether and how the user-jar is included in the system class path for per-job clusters. Setting this parameter to `DISABLED` causes the jar to be included in the user class path instead. Setting this parameter to one of `FIRST`, `LAST` or `ORDER` causes the jar to be included in the system class path, with the jar either being placed at the beginning of the class path (`FIRST`), at the end (`LAST`), or based on the lexicographic order (`ORDER`). --- End diff -- you can, I'll update the description. Another case where enum ConfigOptions would be useful. > Move remaining sections to generated tables >
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361035#comment-16361035 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5463#discussion_r167613260 --- Diff: docs/ops/config.md --- @@ -408,38 +408,7 @@ of the JobManager, because the same ActorSystem is used. Its not possible to use ### YARN -- `containerized.heap-cutoff-ratio`: (Default 0.25) Percentage of heap space to remove from containers started by YARN. When a user requests a certain amount of memory for each TaskManager container (for example 4 GB), we can not pass this amount as the maximum heap space for the JVM (`-Xmx` argument) because the JVM is also allocating memory outside the heap. YARN is very strict with killing containers which are using more memory than requested. Therefore, we remove this fraction of the memory from the requested heap as a safety margin and add it to the memory used off-heap. - -- `containerized.heap-cutoff-min`: (Default 600 MB) Minimum amount of memory to cut off the requested heap size. - -- `yarn.maximum-failed-containers` (Default: number of requested containers). Maximum number of containers the system is going to reallocate in case of a failure. - -- `yarn.application-attempts` (Default: 1). Number of ApplicationMaster restarts. Note that that the entire Flink cluster will restart and the YARN Client will loose the connection. Also, the JobManager address will change and you'll need to set the JM host:port manually. It is recommended to leave this option at 1. - -- `yarn.heartbeat-delay` (Default: 5 seconds). Time between heartbeats with the ResourceManager. - -- `yarn.properties-file.location` (Default: temp directory). When a Flink job is submitted to YARN, the JobManager's host and the number of available processing slots is written into a properties file, so that the Flink client is able to pick those details up. This configuration parameter allows changing the default location of that file (for example for environments sharing a Flink installation between users) - -- `yarn.containers.vcores` The number of virtual cores (vcores) per YARN container. By default, the number of `vcores` is set to the number of slots per TaskManager, if set, or to 1, otherwise. - -- `containerized.master.env.`*ENV_VAR1=value* Configuration values prefixed with `containerized.master.env.` will be passed as environment variables to the ApplicationMaster/JobManager process. For example for passing `LD_LIBRARY_PATH` as an env variable to the ApplicationMaster, set: - -`containerized.master.env.LD_LIBRARY_PATH: "/usr/lib/native"` - -- `containerized.taskmanager.env.` Similar to the configuration prefix about, this prefix allows setting custom environment variables for the TaskManager processes. - -- `yarn.container-start-command-template`: Flink uses the following template when starting on YARN: --- End diff -- looks like you still can. `ConfigConstangs#YARN_CONTAINER_START_COMMAND_TEMPLATE` is read by the yarn cluster descriptor and BootStrapTools. Looks like we missed porting that option. > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361033#comment-16361033 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5463#discussion_r167612255 --- Diff: docs/ops/config.md --- @@ -408,38 +408,7 @@ of the JobManager, because the same ActorSystem is used. Its not possible to use ### YARN -- `containerized.heap-cutoff-ratio`: (Default 0.25) Percentage of heap space to remove from containers started by YARN. When a user requests a certain amount of memory for each TaskManager container (for example 4 GB), we can not pass this amount as the maximum heap space for the JVM (`-Xmx` argument) because the JVM is also allocating memory outside the heap. YARN is very strict with killing containers which are using more memory than requested. Therefore, we remove this fraction of the memory from the requested heap as a safety margin and add it to the memory used off-heap. - -- `containerized.heap-cutoff-min`: (Default 600 MB) Minimum amount of memory to cut off the requested heap size. - -- `yarn.maximum-failed-containers` (Default: number of requested containers). Maximum number of containers the system is going to reallocate in case of a failure. - -- `yarn.application-attempts` (Default: 1). Number of ApplicationMaster restarts. Note that that the entire Flink cluster will restart and the YARN Client will loose the connection. Also, the JobManager address will change and you'll need to set the JM host:port manually. It is recommended to leave this option at 1. - -- `yarn.heartbeat-delay` (Default: 5 seconds). Time between heartbeats with the ResourceManager. - -- `yarn.properties-file.location` (Default: temp directory). When a Flink job is submitted to YARN, the JobManager's host and the number of available processing slots is written into a properties file, so that the Flink client is able to pick those details up. This configuration parameter allows changing the default location of that file (for example for environments sharing a Flink installation between users) - -- `yarn.containers.vcores` The number of virtual cores (vcores) per YARN container. By default, the number of `vcores` is set to the number of slots per TaskManager, if set, or to 1, otherwise. - -- `containerized.master.env.`*ENV_VAR1=value* Configuration values prefixed with `containerized.master.env.` will be passed as environment variables to the ApplicationMaster/JobManager process. For example for passing `LD_LIBRARY_PATH` as an env variable to the ApplicationMaster, set: --- End diff -- yes > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361032#comment-16361032 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5463#discussion_r167612224 --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnConfigOptions.java --- @@ -79,14 +87,18 @@ */ public static final ConfigOption APPLICATION_ATTEMPTS = key("yarn.application-attempts") - .noDefaultValue(); + .noDefaultValue() --- End diff -- I haven't seen any recommendation in the [Yarn docs](https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/yarn_setup.html). In the [HA docs](https://ci.apache.org/projects/flink/flink-docs-master/ops/jobmanager_high_availability.html#yarn-cluster-high-availability) we set it to 10 but never explicitly call this a recommended value. > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361028#comment-16361028 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5463#discussion_r167611073 --- Diff: docs/ops/config.md --- @@ -408,38 +408,7 @@ of the JobManager, because the same ActorSystem is used. Its not possible to use ### YARN -- `containerized.heap-cutoff-ratio`: (Default 0.25) Percentage of heap space to remove from containers started by YARN. When a user requests a certain amount of memory for each TaskManager container (for example 4 GB), we can not pass this amount as the maximum heap space for the JVM (`-Xmx` argument) because the JVM is also allocating memory outside the heap. YARN is very strict with killing containers which are using more memory than requested. Therefore, we remove this fraction of the memory from the requested heap as a safety margin and add it to the memory used off-heap. --- End diff -- correct. > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361025#comment-16361025 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5464 merging. > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361022#comment-16361022 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5462#discussion_r167610029 --- Diff: docs/ops/config.md --- @@ -502,27 +502,7 @@ Previously this key was named `recovery.mode` and the default value was `standal ZooKeeper-based HA Mode -- `high-availability.zookeeper.quorum`: Defines the ZooKeeper quorum URL which is used to connect to the ZooKeeper cluster when the 'zookeeper' HA mode is selected. Previously this key was named `recovery.zookeeper.quorum`. - -- `high-availability.zookeeper.path.root`: (Default `/flink`) Defines the root dir under which the ZooKeeper HA mode will create namespace directories. Previously this ket was named `recovery.zookeeper.path.root`. - -- `high-availability.zookeeper.path.latch`: (Default `/leaderlatch`) Defines the znode of the leader latch which is used to elect the leader. Previously this key was named `recovery.zookeeper.path.latch`. - -- `high-availability.zookeeper.path.leader`: (Default `/leader`) Defines the znode of the leader which contains the URL to the leader and the current leader session ID. Previously this key was named `recovery.zookeeper.path.leader`. - -- `high-availability.storageDir`: Defines the directory in the state backend where the JobManager metadata will be stored (ZooKeeper only keeps pointers to it). Required for HA. Previously this key was named `recovery.zookeeper.storageDir` and `high-availability.zookeeper.storageDir`. - -- `high-availability.zookeeper.client.session-timeout`: (Default `6`) Defines the session timeout for the ZooKeeper session in ms. Previously this key was named `recovery.zookeeper.client.session-timeout` - -- `high-availability.zookeeper.client.connection-timeout`: (Default `15000`) Defines the connection timeout for ZooKeeper in ms. Previously this key was named `recovery.zookeeper.client.connection-timeout`. - -- `high-availability.zookeeper.client.retry-wait`: (Default `5000`) Defines the pause between consecutive retries in ms. Previously this key was named `recovery.zookeeper.client.retry-wait`. - -- `high-availability.zookeeper.client.max-retry-attempts`: (Default `3`) Defines the number of connection retries before the client gives up. Previously this key was named `recovery.zookeeper.client.max-retry-attempts`. - -- `high-availability.job.delay`: (Default `akka.ask.timeout`) Defines the delay before persisted jobs are recovered in case of a master recovery situation. Previously this key was named `recovery.job.delay`. --- End diff -- ah, this is a general HA option and not specifically tied to ZK. It should be moved into the HA section instead of outright removing it. > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361009#comment-16361009 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5462#discussion_r167609071 --- Diff: docs/ops/config.md --- @@ -502,27 +502,7 @@ Previously this key was named `recovery.mode` and the default value was `standal ZooKeeper-based HA Mode -- `high-availability.zookeeper.quorum`: Defines the ZooKeeper quorum URL which is used to connect to the ZooKeeper cluster when the 'zookeeper' HA mode is selected. Previously this key was named `recovery.zookeeper.quorum`. - -- `high-availability.zookeeper.path.root`: (Default `/flink`) Defines the root dir under which the ZooKeeper HA mode will create namespace directories. Previously this ket was named `recovery.zookeeper.path.root`. - -- `high-availability.zookeeper.path.latch`: (Default `/leaderlatch`) Defines the znode of the leader latch which is used to elect the leader. Previously this key was named `recovery.zookeeper.path.latch`. - -- `high-availability.zookeeper.path.leader`: (Default `/leader`) Defines the znode of the leader which contains the URL to the leader and the current leader session ID. Previously this key was named `recovery.zookeeper.path.leader`. - -- `high-availability.storageDir`: Defines the directory in the state backend where the JobManager metadata will be stored (ZooKeeper only keeps pointers to it). Required for HA. Previously this key was named `recovery.zookeeper.storageDir` and `high-availability.zookeeper.storageDir`. - -- `high-availability.zookeeper.client.session-timeout`: (Default `6`) Defines the session timeout for the ZooKeeper session in ms. Previously this key was named `recovery.zookeeper.client.session-timeout` - -- `high-availability.zookeeper.client.connection-timeout`: (Default `15000`) Defines the connection timeout for ZooKeeper in ms. Previously this key was named `recovery.zookeeper.client.connection-timeout`. - -- `high-availability.zookeeper.client.retry-wait`: (Default `5000`) Defines the pause between consecutive retries in ms. Previously this key was named `recovery.zookeeper.client.retry-wait`. - -- `high-availability.zookeeper.client.max-retry-attempts`: (Default `3`) Defines the number of connection retries before the client gives up. Previously this key was named `recovery.zookeeper.client.max-retry-attempts`. - -- `high-availability.job.delay`: (Default `akka.ask.timeout`) Defines the delay before persisted jobs are recovered in case of a master recovery situation. Previously this key was named `recovery.job.delay`. --- End diff -- ehhhcould be that the tables are outdated (ironic isn't it), let me check. > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361006#comment-16361006 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5463#discussion_r167607077 --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnConfigOptions.java --- @@ -79,14 +87,18 @@ */ public static final ConfigOption APPLICATION_ATTEMPTS = key("yarn.application-attempts") - .noDefaultValue(); + .noDefaultValue() --- End diff -- We don't really recommend to leave this at `1` anymore, do we? > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361008#comment-16361008 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5463#discussion_r167607866 --- Diff: docs/ops/config.md --- @@ -408,38 +408,7 @@ of the JobManager, because the same ActorSystem is used. Its not possible to use ### YARN -- `containerized.heap-cutoff-ratio`: (Default 0.25) Percentage of heap space to remove from containers started by YARN. When a user requests a certain amount of memory for each TaskManager container (for example 4 GB), we can not pass this amount as the maximum heap space for the JVM (`-Xmx` argument) because the JVM is also allocating memory outside the heap. YARN is very strict with killing containers which are using more memory than requested. Therefore, we remove this fraction of the memory from the requested heap as a safety margin and add it to the memory used off-heap. - -- `containerized.heap-cutoff-min`: (Default 600 MB) Minimum amount of memory to cut off the requested heap size. - -- `yarn.maximum-failed-containers` (Default: number of requested containers). Maximum number of containers the system is going to reallocate in case of a failure. - -- `yarn.application-attempts` (Default: 1). Number of ApplicationMaster restarts. Note that that the entire Flink cluster will restart and the YARN Client will loose the connection. Also, the JobManager address will change and you'll need to set the JM host:port manually. It is recommended to leave this option at 1. - -- `yarn.heartbeat-delay` (Default: 5 seconds). Time between heartbeats with the ResourceManager. - -- `yarn.properties-file.location` (Default: temp directory). When a Flink job is submitted to YARN, the JobManager's host and the number of available processing slots is written into a properties file, so that the Flink client is able to pick those details up. This configuration parameter allows changing the default location of that file (for example for environments sharing a Flink installation between users) - -- `yarn.containers.vcores` The number of virtual cores (vcores) per YARN container. By default, the number of `vcores` is set to the number of slots per TaskManager, if set, or to 1, otherwise. - -- `containerized.master.env.`*ENV_VAR1=value* Configuration values prefixed with `containerized.master.env.` will be passed as environment variables to the ApplicationMaster/JobManager process. For example for passing `LD_LIBRARY_PATH` as an env variable to the ApplicationMaster, set: - -`containerized.master.env.LD_LIBRARY_PATH: "/usr/lib/native"` - -- `containerized.taskmanager.env.` Similar to the configuration prefix about, this prefix allows setting custom environment variables for the TaskManager processes. - -- `yarn.container-start-command-template`: Flink uses the following template when starting on YARN: --- End diff -- I see this in the doc and it's referenced twice in comments in tests but you can't actually set this anymore, can you? > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361007#comment-16361007 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5463#discussion_r167607428 --- Diff: docs/ops/config.md --- @@ -408,38 +408,7 @@ of the JobManager, because the same ActorSystem is used. Its not possible to use ### YARN -- `containerized.heap-cutoff-ratio`: (Default 0.25) Percentage of heap space to remove from containers started by YARN. When a user requests a certain amount of memory for each TaskManager container (for example 4 GB), we can not pass this amount as the maximum heap space for the JVM (`-Xmx` argument) because the JVM is also allocating memory outside the heap. YARN is very strict with killing containers which are using more memory than requested. Therefore, we remove this fraction of the memory from the requested heap as a safety margin and add it to the memory used off-heap. - -- `containerized.heap-cutoff-min`: (Default 600 MB) Minimum amount of memory to cut off the requested heap size. - -- `yarn.maximum-failed-containers` (Default: number of requested containers). Maximum number of containers the system is going to reallocate in case of a failure. - -- `yarn.application-attempts` (Default: 1). Number of ApplicationMaster restarts. Note that that the entire Flink cluster will restart and the YARN Client will loose the connection. Also, the JobManager address will change and you'll need to set the JM host:port manually. It is recommended to leave this option at 1. - -- `yarn.heartbeat-delay` (Default: 5 seconds). Time between heartbeats with the ResourceManager. - -- `yarn.properties-file.location` (Default: temp directory). When a Flink job is submitted to YARN, the JobManager's host and the number of available processing slots is written into a properties file, so that the Flink client is able to pick those details up. This configuration parameter allows changing the default location of that file (for example for environments sharing a Flink installation between users) - -- `yarn.containers.vcores` The number of virtual cores (vcores) per YARN container. By default, the number of `vcores` is set to the number of slots per TaskManager, if set, or to 1, otherwise. - -- `containerized.master.env.`*ENV_VAR1=value* Configuration values prefixed with `containerized.master.env.` will be passed as environment variables to the ApplicationMaster/JobManager process. For example for passing `LD_LIBRARY_PATH` as an env variable to the ApplicationMaster, set: --- End diff -- These next three will also be in the general RM section? > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361004#comment-16361004 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5463#discussion_r167606495 --- Diff: docs/ops/config.md --- @@ -408,38 +408,7 @@ of the JobManager, because the same ActorSystem is used. Its not possible to use ### YARN -- `containerized.heap-cutoff-ratio`: (Default 0.25) Percentage of heap space to remove from containers started by YARN. When a user requests a certain amount of memory for each TaskManager container (for example 4 GB), we can not pass this amount as the maximum heap space for the JVM (`-Xmx` argument) because the JVM is also allocating memory outside the heap. YARN is very strict with killing containers which are using more memory than requested. Therefore, we remove this fraction of the memory from the requested heap as a safety margin and add it to the memory used off-heap. --- End diff -- This will be in the general resource manager options from now on, I'm guessing? (this option and the next) > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361005#comment-16361005 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5463#discussion_r167608886 --- Diff: docs/ops/config.md --- @@ -408,38 +408,7 @@ of the JobManager, because the same ActorSystem is used. Its not possible to use ### YARN -- `containerized.heap-cutoff-ratio`: (Default 0.25) Percentage of heap space to remove from containers started by YARN. When a user requests a certain amount of memory for each TaskManager container (for example 4 GB), we can not pass this amount as the maximum heap space for the JVM (`-Xmx` argument) because the JVM is also allocating memory outside the heap. YARN is very strict with killing containers which are using more memory than requested. Therefore, we remove this fraction of the memory from the requested heap as a safety margin and add it to the memory used off-heap. - -- `containerized.heap-cutoff-min`: (Default 600 MB) Minimum amount of memory to cut off the requested heap size. - -- `yarn.maximum-failed-containers` (Default: number of requested containers). Maximum number of containers the system is going to reallocate in case of a failure. - -- `yarn.application-attempts` (Default: 1). Number of ApplicationMaster restarts. Note that that the entire Flink cluster will restart and the YARN Client will loose the connection. Also, the JobManager address will change and you'll need to set the JM host:port manually. It is recommended to leave this option at 1. - -- `yarn.heartbeat-delay` (Default: 5 seconds). Time between heartbeats with the ResourceManager. - -- `yarn.properties-file.location` (Default: temp directory). When a Flink job is submitted to YARN, the JobManager's host and the number of available processing slots is written into a properties file, so that the Flink client is able to pick those details up. This configuration parameter allows changing the default location of that file (for example for environments sharing a Flink installation between users) - -- `yarn.containers.vcores` The number of virtual cores (vcores) per YARN container. By default, the number of `vcores` is set to the number of slots per TaskManager, if set, or to 1, otherwise. - -- `containerized.master.env.`*ENV_VAR1=value* Configuration values prefixed with `containerized.master.env.` will be passed as environment variables to the ApplicationMaster/JobManager process. For example for passing `LD_LIBRARY_PATH` as an env variable to the ApplicationMaster, set: - -`containerized.master.env.LD_LIBRARY_PATH: "/usr/lib/native"` - -- `containerized.taskmanager.env.` Similar to the configuration prefix about, this prefix allows setting custom environment variables for the TaskManager processes. - -- `yarn.container-start-command-template`: Flink uses the following template when starting on YARN: -`%java% %jvmmem% %jvmopts% %logging% %class% %args% %redirects%`. This configuration parameter allows users -to pass custom settings (such as JVM paths, arguments etc.). Note that in most cases, it is sufficient to -use the `env.java.opts` setting, which is the `%jvmopts%` variable in the String. - -- `yarn.application-master.port` (Default: 0, which lets the OS choose an ephemeral port) With this configuration option, users can specify a port, a range of ports or a list of ports for the Application Master (and JobManager) RPC port. By default we recommend using the default value (0) to let the operating system choose an appropriate port. In particular when multiple AMs are running on the same physical host, fixed port assignments prevent the AM from starting. - - For example when running Flink on YARN on an environment with a restrictive firewall, this option allows specifying a range of allowed ports. - -- `yarn.tags` A comma-separated list of tags to apply to the Flink YARN application. - -- `yarn.per-job-cluster.include-user-jar` (Default: ORDER) Control whether and how the user-jar is included in the system class path for per-job clusters. Setting this parameter to `DISABLED` causes the jar to be included in the user class path instead. Setting this parameter to one of `FIRST`, `LAST` or `ORDER` causes the jar to be included in the system class path, with the jar either being placed at the beginning of the class path (`FIRST`), at the end (`LAST`), or based on the lexicographic order (`ORDER`). --- End diff -- You can't set `DISABLED` anymore? > Move remaining sections to generated tables > --- > > Key: FLINK-8475 >
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361001#comment-16361001 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5462#discussion_r167608682 --- Diff: docs/ops/config.md --- @@ -502,27 +502,7 @@ Previously this key was named `recovery.mode` and the default value was `standal ZooKeeper-based HA Mode -- `high-availability.zookeeper.quorum`: Defines the ZooKeeper quorum URL which is used to connect to the ZooKeeper cluster when the 'zookeeper' HA mode is selected. Previously this key was named `recovery.zookeeper.quorum`. - -- `high-availability.zookeeper.path.root`: (Default `/flink`) Defines the root dir under which the ZooKeeper HA mode will create namespace directories. Previously this ket was named `recovery.zookeeper.path.root`. - -- `high-availability.zookeeper.path.latch`: (Default `/leaderlatch`) Defines the znode of the leader latch which is used to elect the leader. Previously this key was named `recovery.zookeeper.path.latch`. - -- `high-availability.zookeeper.path.leader`: (Default `/leader`) Defines the znode of the leader which contains the URL to the leader and the current leader session ID. Previously this key was named `recovery.zookeeper.path.leader`. - -- `high-availability.storageDir`: Defines the directory in the state backend where the JobManager metadata will be stored (ZooKeeper only keeps pointers to it). Required for HA. Previously this key was named `recovery.zookeeper.storageDir` and `high-availability.zookeeper.storageDir`. --- End diff -- subsumed by `"high-availability.storageDir"` > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16360995#comment-16360995 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5459#discussion_r167608020 --- Diff: flink-core/src/main/java/org/apache/flink/configuration/CoreOptions.java --- @@ -127,7 +130,30 @@ */ public static final ConfigOption DEFAULT_FILESYSTEM_SCHEME = ConfigOptions .key("fs.default-scheme") - .noDefaultValue(); + .noDefaultValue() + .withDescription("The default filesystem scheme, used for paths that do not declare a scheme explicitly."); --- End diff -- ah I remember now, looking at the description: [Part1] The default filesystem scheme to be used [Part2] , with the necessary authority to contact, e.g. the host:port of the NameNode in the case of HDFS (if needed). [Part3] By default, this is set to file:/// which points to the local filesystem. This means that the local filesystem is going to be used to search for user-specified files without an explicit scheme definition. [Part4] This scheme is used ONLY if no other scheme is specified (explicitly) in the user-provided URI. Part 1 and 4 are contained in the description, part 3 was left out since file:/// isn't the documented default. Only part 2 is really missing. > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16360992#comment-16360992 ] ASF GitHub Bot commented on FLINK-8475: --- Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5459#discussion_r167606303 --- Diff: flink-core/src/main/java/org/apache/flink/configuration/CoreOptions.java --- @@ -127,7 +130,30 @@ */ public static final ConfigOption DEFAULT_FILESYSTEM_SCHEME = ConfigOptions .key("fs.default-scheme") - .noDefaultValue(); + .noDefaultValue() + .withDescription("The default filesystem scheme, used for paths that do not declare a scheme explicitly."); --- End diff -- IMO the missing text isn't really needed, but I'll add it back to keep this PR as a straight port of the existing docs and merge it afterwards. > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16360976#comment-16360976 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5462#discussion_r167603434 --- Diff: docs/ops/config.md --- @@ -502,27 +502,7 @@ Previously this key was named `recovery.mode` and the default value was `standal ZooKeeper-based HA Mode -- `high-availability.zookeeper.quorum`: Defines the ZooKeeper quorum URL which is used to connect to the ZooKeeper cluster when the 'zookeeper' HA mode is selected. Previously this key was named `recovery.zookeeper.quorum`. - -- `high-availability.zookeeper.path.root`: (Default `/flink`) Defines the root dir under which the ZooKeeper HA mode will create namespace directories. Previously this ket was named `recovery.zookeeper.path.root`. - -- `high-availability.zookeeper.path.latch`: (Default `/leaderlatch`) Defines the znode of the leader latch which is used to elect the leader. Previously this key was named `recovery.zookeeper.path.latch`. - -- `high-availability.zookeeper.path.leader`: (Default `/leader`) Defines the znode of the leader which contains the URL to the leader and the current leader session ID. Previously this key was named `recovery.zookeeper.path.leader`. - -- `high-availability.storageDir`: Defines the directory in the state backend where the JobManager metadata will be stored (ZooKeeper only keeps pointers to it). Required for HA. Previously this key was named `recovery.zookeeper.storageDir` and `high-availability.zookeeper.storageDir`. --- End diff -- Did you find out why this is gone? > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16360975#comment-16360975 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5462#discussion_r167603849 --- Diff: docs/ops/config.md --- @@ -502,27 +502,7 @@ Previously this key was named `recovery.mode` and the default value was `standal ZooKeeper-based HA Mode -- `high-availability.zookeeper.quorum`: Defines the ZooKeeper quorum URL which is used to connect to the ZooKeeper cluster when the 'zookeeper' HA mode is selected. Previously this key was named `recovery.zookeeper.quorum`. - -- `high-availability.zookeeper.path.root`: (Default `/flink`) Defines the root dir under which the ZooKeeper HA mode will create namespace directories. Previously this ket was named `recovery.zookeeper.path.root`. - -- `high-availability.zookeeper.path.latch`: (Default `/leaderlatch`) Defines the znode of the leader latch which is used to elect the leader. Previously this key was named `recovery.zookeeper.path.latch`. - -- `high-availability.zookeeper.path.leader`: (Default `/leader`) Defines the znode of the leader which contains the URL to the leader and the current leader session ID. Previously this key was named `recovery.zookeeper.path.leader`. - -- `high-availability.storageDir`: Defines the directory in the state backend where the JobManager metadata will be stored (ZooKeeper only keeps pointers to it). Required for HA. Previously this key was named `recovery.zookeeper.storageDir` and `high-availability.zookeeper.storageDir`. - -- `high-availability.zookeeper.client.session-timeout`: (Default `6`) Defines the session timeout for the ZooKeeper session in ms. Previously this key was named `recovery.zookeeper.client.session-timeout` - -- `high-availability.zookeeper.client.connection-timeout`: (Default `15000`) Defines the connection timeout for ZooKeeper in ms. Previously this key was named `recovery.zookeeper.client.connection-timeout`. - -- `high-availability.zookeeper.client.retry-wait`: (Default `5000`) Defines the pause between consecutive retries in ms. Previously this key was named `recovery.zookeeper.client.retry-wait`. - -- `high-availability.zookeeper.client.max-retry-attempts`: (Default `3`) Defines the number of connection retries before the client gives up. Previously this key was named `recovery.zookeeper.client.max-retry-attempts`. - -- `high-availability.job.delay`: (Default `akka.ask.timeout`) Defines the delay before persisted jobs are recovered in case of a master recovery situation. Previously this key was named `recovery.job.delay`. --- End diff -- And also where this is gone? > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8475) Move remaining sections to generated tables
[ https://issues.apache.org/jira/browse/FLINK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16360974#comment-16360974 ] ASF GitHub Bot commented on FLINK-8475: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5462#discussion_r167603044 --- Diff: docs/ops/config.md --- @@ -502,27 +502,7 @@ Previously this key was named `recovery.mode` and the default value was `standal ZooKeeper-based HA Mode -- `high-availability.zookeeper.quorum`: Defines the ZooKeeper quorum URL which is used to connect to the ZooKeeper cluster when the 'zookeeper' HA mode is selected. Previously this key was named `recovery.zookeeper.quorum`. - -- `high-availability.zookeeper.path.root`: (Default `/flink`) Defines the root dir under which the ZooKeeper HA mode will create namespace directories. Previously this ket was named `recovery.zookeeper.path.root`. --- End diff -- `ket` ... 😅 > Move remaining sections to generated tables > --- > > Key: FLINK-8475 > URL: https://issues.apache.org/jira/browse/FLINK-8475 > Project: Flink > Issue Type: Sub-task > Components: Configuration >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)