[jira] [Created] (FLINK-10640) Enable Slot Resource Profile for Resource Management
Xintong Song created FLINK-10640: Summary: Enable Slot Resource Profile for Resource Management Key: FLINK-10640 URL: https://issues.apache.org/jira/browse/FLINK-10640 Project: Flink Issue Type: New Feature Components: ResourceManager Reporter: Xintong Song Motivation & Backgrounds * The existing concept of task slots roughly represents how many pipeline of tasks a TaskManager can hold. However, it does not consider the differences in resource needs and usage of individual tasks. Enabling resource profiles of slots may allow Flink to better allocate execution resources according to tasks fine-grained resource needs. * The community version Flink already contains APIs and some implementation for slot resource profile. However, such logic is not truly used. (ResourceProfile of slot requests is by default set to UNKNOWN with negative values, thus matches any given slot.) Preliminary Design * Slot Management A slot represents a certain amount of resources for a single pipeline of tasks to run in on a TaskManager. Initially, a TaskManager does not have any slots but a total amount of resources. When allocating, the ResourceManager finds proper TMs to generate new slots for the tasks to run according to the slot requests. Once generated, the slot's size (resource profile) does not change until it's freed. ResourceManager can apply different, portable strategies to allocate slots from TaskManagers. * TM Management The size and number of TaskManagers and when to start them can also be flexible. TMs can be started and released dynamically, and may have different sizes. We may have many different, portable strategies. E.g., an elastic session that can run multiple jobs like the session mode while dynamically adjusting the size of session (number of TMs) according to the realtime working load. * About Slot Sharing Slot sharing is a good heuristic to easily calculate how many slots needed to get the job running and get better utilization when there is no resource profile in slots. However, with resource profiles enabling finer-grained resource management, each individual task has its specific resource need and it does not make much sense to have multiple tasks sharing the resource of the same slot. Instead, we may introduce locality preferences/constraints to support the semantics of putting tasks in same/different TMs in a more general way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-13184) Support launching task executors with multi-thread on YARN.
Xintong Song created FLINK-13184: Summary: Support launching task executors with multi-thread on YARN. Key: FLINK-13184 URL: https://issues.apache.org/jira/browse/FLINK-13184 Project: Flink Issue Type: Bug Components: Deployment / YARN Affects Versions: 1.8.1, 1.9.0 Reporter: Xintong Song Assignee: Xintong Song Currently, YarnResourceManager starts all task executors in main thread. This could cause RM thread becomes unresponsive when launching a large number of TEs (e.g. > 1000), leading to TE registration/heartbeat timeouts. In Blink, we have a thread pool that RM starts TEs through the YARN NMClient in separated threads. I think we should add this feature to the Flink master branch as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-13554) ResourceManager should have a timeout on starting new TaskExecutors.
Xintong Song created FLINK-13554: Summary: ResourceManager should have a timeout on starting new TaskExecutors. Key: FLINK-13554 URL: https://issues.apache.org/jira/browse/FLINK-13554 Project: Flink Issue Type: Improvement Components: Runtime / Coordination Affects Versions: 1.9.0 Reporter: Xintong Song Recently, we encountered a case that one TaskExecutor get stuck during launching on Yarn (without fail), causing that job cannot recover from continuous failovers. The reason the TaskExecutor gets stuck is due to our environment problem. The TaskExecutor gets stuck somewhere after the ResourceManager starts the TaskExecutor and waiting for the TaskExecutor to be brought up and register. Later when the slot request timeouts, the job fails over and requests slots from ResourceManager again, the ResourceManager still see a TaskExecutor (the stuck one) is being started and will not request new container from Yarn. Therefore, the job can not recover from failure. I think to avoid such unrecoverable status, the ResourceManager need to have a timeout on starting new TaskExecutor. If the starting of TaskExecutor takes too long, it should just fail the TaskExecutor and starts a new one. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13555) Failures of slot requests requiring unfulfillable managed memory should not be ignored.
Xintong Song created FLINK-13555: Summary: Failures of slot requests requiring unfulfillable managed memory should not be ignored. Key: FLINK-13555 URL: https://issues.apache.org/jira/browse/FLINK-13555 Project: Flink Issue Type: Improvement Components: Runtime / Coordination Affects Versions: 1.9.0 Reporter: Xintong Song Fix For: 1.9.0 Attachments: flink-unk-standalonesession-0-u-home.log, flink-unk-taskexecutor-0-u-home.log Currently, SlotPool ignores failures of requesting slots from ResourceManager for all batch slot requests. The idea behind this is to allow batch slot requests pending at SlotPool and waiting for other tasks to finish and release slots. A slot request will be failed only if it is not fulfilled in its timeout. However, there could be two kinds of request slots from RM failures. # RM does not have available slots. All slots are in use at the moment. But they might become available later when the currently running tasks finish. # The slot request requires too many resources that can not be fulfilled by any slot (available or not) in the cluster. The request is also not likely to be fulfilled later. For the 2nd kinds of failures, it doesn't make sense to wait for the timeout. We should fail the job immediately, with proper error messages describing the problem and suggesting the user to tune job or cluster configurations. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13579) Failed launching standalone cluster due to improper configured irrelevant config options for active mode.
Xintong Song created FLINK-13579: Summary: Failed launching standalone cluster due to improper configured irrelevant config options for active mode. Key: FLINK-13579 URL: https://issues.apache.org/jira/browse/FLINK-13579 Project: Flink Issue Type: Bug Components: Runtime / Coordination Affects Versions: 1.9.0 Reporter: Xintong Song Fix For: 1.9.0 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (FLINK-13980) FLIP-49 Unified Memory Configuration for TaskExecutors
Xintong Song created FLINK-13980: Summary: FLIP-49 Unified Memory Configuration for TaskExecutors Key: FLINK-13980 URL: https://issues.apache.org/jira/browse/FLINK-13980 Project: Flink Issue Type: Improvement Components: Runtime / Configuration, Runtime / Coordination Affects Versions: 1.9.0 Reporter: Xintong Song Fix For: 1.10.0 This is the umbrella issue of 'FLIP-49: Unified Memory Configuration for TaskExecutors'. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (FLINK-13981) Introduce a switch for enabling the new task executor memory configurations
Xintong Song created FLINK-13981: Summary: Introduce a switch for enabling the new task executor memory configurations Key: FLINK-13981 URL: https://issues.apache.org/jira/browse/FLINK-13981 Project: Flink Issue Type: Sub-task Reporter: Xintong Song -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (FLINK-13985) Use native memory for managed memory.
Xintong Song created FLINK-13985: Summary: Use native memory for managed memory. Key: FLINK-13985 URL: https://issues.apache.org/jira/browse/FLINK-13985 Project: Flink Issue Type: Sub-task Reporter: Xintong Song -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (FLINK-13982) Implement memory calculation logics
Xintong Song created FLINK-13982: Summary: Implement memory calculation logics Key: FLINK-13982 URL: https://issues.apache.org/jira/browse/FLINK-13982 Project: Flink Issue Type: Sub-task Reporter: Xintong Song -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (FLINK-13984) Separate on-heap and off-heap managed memory pools
Xintong Song created FLINK-13984: Summary: Separate on-heap and off-heap managed memory pools Key: FLINK-13984 URL: https://issues.apache.org/jira/browse/FLINK-13984 Project: Flink Issue Type: Sub-task Reporter: Xintong Song -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (FLINK-13986) Clean-up of legacy mode
Xintong Song created FLINK-13986: Summary: Clean-up of legacy mode Key: FLINK-13986 URL: https://issues.apache.org/jira/browse/FLINK-13986 Project: Flink Issue Type: Sub-task Reporter: Xintong Song -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (FLINK-13983) Launch task executor with new memory calculation logics
Xintong Song created FLINK-13983: Summary: Launch task executor with new memory calculation logics Key: FLINK-13983 URL: https://issues.apache.org/jira/browse/FLINK-13983 Project: Flink Issue Type: Sub-task Reporter: Xintong Song -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (FLINK-14058) FLIP-53 Fine Grained Operator Resource Management
Xintong Song created FLINK-14058: Summary: FLIP-53 Fine Grained Operator Resource Management Key: FLINK-14058 URL: https://issues.apache.org/jira/browse/FLINK-14058 Project: Flink Issue Type: New Feature Components: Runtime / Coordination Affects Versions: 1.9.0 Reporter: Xintong Song Fix For: 1.10.0 This is the umbrella issue of 'FLIP-53: Fine Grained Operator Resource Management'. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (FLINK-14059) Introduce option allSourcesInSamePipelinedRegion in ExecutionConfig
Xintong Song created FLINK-14059:
Summary: Introduce option allSourcesInSamePipelinedRegion in
ExecutionConfig
Key: FLINK-14059
URL: https://issues.apache.org/jira/browse/FLINK-14059
Project: Flink
Issue Type: Sub-task
Reporter: Xintong Song
* Introduce option {{allSourcesInSamePipelinedRegion}} in {{ExecutionConfig}}
* Set it to {{true}} by default
* Set it to {{false}} for SQL/Table API bounded batch jobs by the Blink planner
This step should not introduce any behavior changes.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (FLINK-14061) Introduce managed memory fractions to StreamConfig
Xintong Song created FLINK-14061:
Summary: Introduce managed memory fractions to StreamConfig
Key: FLINK-14061
URL: https://issues.apache.org/jira/browse/FLINK-14061
Project: Flink
Issue Type: Sub-task
Reporter: Xintong Song
Introduce {{fracManagedMemOnHeap}} and {{fracManagedMemOffHeap}} in
{{StreamConfig}}, so they can be set by {{StreamingJobGraphGenerator}} and used
by operators in runtime.
This step should not introduce any behavior changes.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (FLINK-14063) Operators use fractions to decide how many managed memory to allocate
Xintong Song created FLINK-14063:
Summary: Operators use fractions to decide how many managed memory
to allocate
Key: FLINK-14063
URL: https://issues.apache.org/jira/browse/FLINK-14063
Project: Flink
Issue Type: Sub-task
Reporter: Xintong Song
* Operators allocate memory segments with the amount returned by
{{MemoryManager#computeNumberOfPages}}.
* Operators reserve memory with the amount returned by
{{MemoryManager#computeMemorySize}}.
This step activates the new fraction based managed memory.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (FLINK-14060) Set slot sharing groups according to pipelined regions
Xintong Song created FLINK-14060:
Summary: Set slot sharing groups according to pipelined regions
Key: FLINK-14060
URL: https://issues.apache.org/jira/browse/FLINK-14060
Project: Flink
Issue Type: Sub-task
Reporter: Xintong Song
{{StreamingJobGraphGenerator}} set slot sharing group for operators at
compiling time.
* Identify pipelined regions, with respect to
{{allSourcesInSamePipelinedRegion}}
* Set slot sharing groups according to pipelined regions
** By default, each pipelined region should go into a separate slot sharing
group
** If the user sets operators in multiple pipelined regions into same slot
sharing group, it should be respected
This step should not introduce any behavior changes, given that later scheduled
pipelined regions can reuse slots from previous scheduled pipelined regions.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (FLINK-14062) Set managed memory fractions according to slot sharing groups
Xintong Song created FLINK-14062:
Summary: Set managed memory fractions according to slot sharing
groups
Key: FLINK-14062
URL: https://issues.apache.org/jira/browse/FLINK-14062
Project: Flink
Issue Type: Sub-task
Reporter: Xintong Song
* For operators with specified {{ResourceSpecs}}, calculate fractions according
to operators {{ResourceSpecs}}
* For operators with unknown {{ResourceSpecs}}, calculate fractions according
to number of operators using managed memory
This step should not introduce any behavior changes.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (FLINK-14106) Make SlotManager pluggable
Xintong Song created FLINK-14106: Summary: Make SlotManager pluggable Key: FLINK-14106 URL: https://issues.apache.org/jira/browse/FLINK-14106 Project: Flink Issue Type: Improvement Components: Runtime / Coordination Affects Versions: 1.9.0 Reporter: Xintong Song Fix For: 1.10.0 As we are enabling fine grained resource management in 1.10, we can have various resource scheduling strategies. Such strategies generally should make the following three decisions. * When to launch new / release existing TMs? (How many TMs) * What and how many resources should TMs be started with? * How to allocate between slot requests and TM resources? We may want to make above decisions differently in different scenarios (active/reactive mode, perjob/session mode, etc.). Therefore, we propose to make the scheduling strategies pluggable. We propose to make the following changes: * Make SlotManager an interface, and implements it differently for different strategies strategies. * Modify ResourceManager-SlotManager interfaces to cover all the three decisions mentioned above in SlotManager. In particular, SlotManager needs to allocate TM resources instead of slot resources from ResourceActions. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (FLINK-14187) FLIP-56 Dynamic Slot Allocation
Xintong Song created FLINK-14187: Summary: FLIP-56 Dynamic Slot Allocation Key: FLINK-14187 URL: https://issues.apache.org/jira/browse/FLINK-14187 Project: Flink Issue Type: New Feature Components: Runtime / Coordination Affects Versions: 1.9.0 Reporter: Xintong Song Fix For: 1.10.0 This is the umbrella issue for 'FLIP-56: Dynamic Slot Allocation'. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14189) Extend TaskExecutor to support dynamic slot allocation
Xintong Song created FLINK-14189: Summary: Extend TaskExecutor to support dynamic slot allocation Key: FLINK-14189 URL: https://issues.apache.org/jira/browse/FLINK-14189 Project: Flink Issue Type: Sub-task Reporter: Xintong Song * TaskSlotTable ** Bookkeep task manager available resources ** Add and implement interface for dynamic allocating slot (with resource profile instead of slot index) ** Create slot report with dynamic allocated slots and remaining available resources * TaskExecutor ** Support request slot with resource profile rather than slot id. The slot report still contain status of legacy free slots. When ResourceManager requests slots with slot id, convert it to default slot resource profiles for bookkeeping. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14188) TaskExecutor derive and register with default slot resource profile
Xintong Song created FLINK-14188: Summary: TaskExecutor derive and register with default slot resource profile Key: FLINK-14188 URL: https://issues.apache.org/jira/browse/FLINK-14188 Project: Flink Issue Type: Sub-task Reporter: Xintong Song * Introduce config option for defaultSlotFraction * Derive default slot resource profile from the new config option, or the legacy config option "taskmanager.numberOfTaskSlots". * Register task executor with the default slot resource profile. This step should not introduce any behavior changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14191) Extend SlotManager to support dynamic slot allocation on pending task executors
Xintong Song created FLINK-14191: Summary: Extend SlotManager to support dynamic slot allocation on pending task executors Key: FLINK-14191 URL: https://issues.apache.org/jira/browse/FLINK-14191 Project: Flink Issue Type: Sub-task Reporter: Xintong Song * Introduce PendingTaskManagerResources * Create PendingTaskManagerSlot on allocation, from PendingTaskManagerResource * Map registered task executors to matching PendingTaskManagerResources, and allocate slots for corresponding PendingTaskManagerSlots Convert registered task executor free slots into equivalent available resources according to default slot resource profiles. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14190) Extend SlotManager to support dynamic slot allocation on registered TaskExecutors.
Xintong Song created FLINK-14190: Summary: Extend SlotManager to support dynamic slot allocation on registered TaskExecutors. Key: FLINK-14190 URL: https://issues.apache.org/jira/browse/FLINK-14190 Project: Flink Issue Type: Sub-task Reporter: Xintong Song * Bookkeep task manager available resources * Match between slot requests and task executor resources ** Find task executors with matching available resources for slot requests ** Find matching pending slot requests for task executors with new available resources * Create TaskManagerSlot on allocation and remove on free. * Request slot from TaskExecutor with resource profiles. Use RM calculated default resource profiles for all slot requests. Convert free slots in SlotReports into equivalent available resources according to default slot resource profiles. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14192) Enable the dynamic slot allocation feature.
Xintong Song created FLINK-14192: Summary: Enable the dynamic slot allocation feature. Key: FLINK-14192 URL: https://issues.apache.org/jira/browse/FLINK-14192 Project: Flink Issue Type: Sub-task Reporter: Xintong Song * ResourceManager uses TaskExecutor registered default slot resource profiles, instead of that calculated on RM side. * ResourceManager uses actual requested resource profiles for slot requests, instead assuming default profile for all requests. * TaskExecutor bookkeep with requested resource profiles instead, instead of assuming default profile for all requests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14193) Update RestAPI / Web UI
Xintong Song created FLINK-14193: Summary: Update RestAPI / Web UI Key: FLINK-14193 URL: https://issues.apache.org/jira/browse/FLINK-14193 Project: Flink Issue Type: Sub-task Reporter: Xintong Song * Update RestAPI / WebUI to properly display information of available resources and allocated slots of task executors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14194) Clean-up of legacy mode.
Xintong Song created FLINK-14194: Summary: Clean-up of legacy mode. Key: FLINK-14194 URL: https://issues.apache.org/jira/browse/FLINK-14194 Project: Flink Issue Type: Sub-task Reporter: Xintong Song -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14405) Align ResourceProfile/ResourceSpec fields with the new TaskExecutor memory setups.
Xintong Song created FLINK-14405: Summary: Align ResourceProfile/ResourceSpec fields with the new TaskExecutor memory setups. Key: FLINK-14405 URL: https://issues.apache.org/jira/browse/FLINK-14405 Project: Flink Issue Type: Sub-task Reporter: Xintong Song -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15023) Remove on-heap managed memory
Xintong Song created FLINK-15023: Summary: Remove on-heap managed memory Key: FLINK-15023 URL: https://issues.apache.org/jira/browse/FLINK-15023 Project: Flink Issue Type: Sub-task Components: Runtime / Configuration Reporter: Xintong Song As mentioned in [this discussion thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Make-Managed-Memory-always-off-heap-Adjustment-to-FLIP-49-td35365.html], we want to make managed memory always off-heap. This task include changes: * Remove `MEMORY_OFF_HEAP`, `MANAGED_MEMORY_OFFHEAP_FRACTION` and `MANAGED_MEMORY_OFFHEAP_SIZE` from `TaskManagerOptions`. * Remove `onHeapManagedMemory` from `ResourceProfile`, `ResourceSpec` and `TaskExecutorResourceSpec`. * Remove on heap managed memory from `MemoryManager` * Remove on heap managed memory fraction from `StreamConfig`. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15053) Configurations with values contains space may cause TM failures on Yarn
Xintong Song created FLINK-15053: Summary: Configurations with values contains space may cause TM failures on Yarn Key: FLINK-15053 URL: https://issues.apache.org/jira/browse/FLINK-15053 Project: Flink Issue Type: Bug Components: Deployment / YARN, Runtime / Coordination Affects Versions: 1.10.0 Reporter: Xintong Song Fix For: 1.10.0 Currently on Yarn setups, we are passing task executor specific configurations through dynamic properties in the starting command (see FLINK-13184). If the value of configuration contains space, the dynamic properties may not be correctly parsed, which could cause task executor failures. On occurrence can be found in FLINK-15047. It would be good to allow spaces when passing dynamic properties. E.g., surrounding the values with double quotation marks, or escaping special characters. cc [~fly_in_gis] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15055) YarnDistributedCacheITCase does not generate logs.
Xintong Song created FLINK-15055: Summary: YarnDistributedCacheITCase does not generate logs. Key: FLINK-15055 URL: https://issues.apache.org/jira/browse/FLINK-15055 Project: Flink Issue Type: Improvement Components: Deployment / YARN, Tests Affects Versions: 1.10.0 Reporter: Xintong Song According to FLINK-14630, in order to properly generate logs, test cases that use starts cluster with YarnClusterDescriptor on MiniYARNCluster should call the util function FlinkYarnSessionCli#setLogConfigFileInConfig to set the path of log4j property file into the configuration. The YarnDistributedCacheITCase failed to do this. We probably should also revisit other Yarn IT cases see if there's any other case that also failed to do this. A discussable alternative is to make it a default behavior for YarnClusterDescriptor to always set the default log4j property file, if it can find one, when the property file is not explicitly configured. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15143) Create document for FLIP-49 TM memory model and configuration guide
Xintong Song created FLINK-15143: Summary: Create document for FLIP-49 TM memory model and configuration guide Key: FLINK-15143 URL: https://issues.apache.org/jira/browse/FLINK-15143 Project: Flink Issue Type: Task Components: Documentation Affects Versions: 1.10.0 Reporter: Xintong Song Fix For: 1.10.0 In release 1.10, with FLIP-49, we introduced significant changes to the TaskExecutor memory model and it's related configuration options / logics. It is very important that we clearly state the changes and potential effects, and guide our users to tune their clusters with the new configuration for both new setups and migrations of previous setups. And we should do that for both documentation and the release note. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15145) Tune default values for FLIP-49 TM memory configurations with real production jobs.
Xintong Song created FLINK-15145: Summary: Tune default values for FLIP-49 TM memory configurations with real production jobs. Key: FLINK-15145 URL: https://issues.apache.org/jira/browse/FLINK-15145 Project: Flink Issue Type: Task Components: Runtime / Configuration Reporter: Xintong Song Fix For: 1.10.0 In release 1.10, with FLIP-49, we introduced significant changes to the TaskExecutor memory model and it's related configuration options / logics. Since the model and configuration logics are changed, it is reasonable that we also change the default configuration values. Currently, the default values are set with the gut feelings and experiences from e2e tests. It would be good that we try and tune the configurations with some real production jobs, of various scales if possible, before exposing the configurations in the release. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15367) Handle backwards compatibility of "taskmanager.heap.size" differently for standalone / active setups
Xintong Song created FLINK-15367: Summary: Handle backwards compatibility of "taskmanager.heap.size" differently for standalone / active setups Key: FLINK-15367 URL: https://issues.apache.org/jira/browse/FLINK-15367 Project: Flink Issue Type: Improvement Components: Runtime / Configuration Reporter: Xintong Song Fix For: 1.10.0 Previously, "taskmanager.heap.size" were used differently for calculating TM memory sizes on standalone / active setups. To fully align with the previous behaviors, we need to map this deprecated key to "taskmanager.memory.flink.size" for standalone setups and "taskmanager.memory.process.size" for active setups. Detailed discussion can be found in this [ML thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Some-feedback-after-trying-out-the-new-FLIP-49-memory-configurations-td36129.html]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15369) MiniCluster use fixed network / managed memory sizes by defualt
Xintong Song created FLINK-15369: Summary: MiniCluster use fixed network / managed memory sizes by defualt Key: FLINK-15369 URL: https://issues.apache.org/jira/browse/FLINK-15369 Project: Flink Issue Type: Improvement Components: Runtime / Configuration Reporter: Xintong Song Fix For: 1.10.0 Currently, Mini Cluster may allocate off-heap memory (managed & network) according to the JVM free heap size and configured off-heap fractions. This could lead to unnecessary large off-heap memory usage and unpredictable / hard-to-understand behaviors. We believe a fix value for managed / network memory would be enough for a such a setup that runs Flink as a library. Detailed discussion can be found in this [ML thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Some-feedback-after-trying-out-the-new-FLIP-49-memory-configurations-td36129.html]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15371) Change FLIP-49 memory configurations to use the new memory type config options
Xintong Song created FLINK-15371: Summary: Change FLIP-49 memory configurations to use the new memory type config options Key: FLINK-15371 URL: https://issues.apache.org/jira/browse/FLINK-15371 Project: Flink Issue Type: Improvement Components: Runtime / Configuration Reporter: Xintong Song Fix For: 1.10.0 FLIP-49 memory configurations can leverage the new strong typed ConfigOption, to make validation automatic and save from breaking the options later. Detailed discussion can be found in this [ML thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Some-feedback-after-trying-out-the-new-FLIP-49-memory-configurations-td36129.html]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15372) Use shorter config keys for FLIP-49 total memory config options
Xintong Song created FLINK-15372: Summary: Use shorter config keys for FLIP-49 total memory config options Key: FLINK-15372 URL: https://issues.apache.org/jira/browse/FLINK-15372 Project: Flink Issue Type: Improvement Components: Runtime / Configuration Reporter: Xintong Song Fix For: 1.10.0 We propose to use shorter keys for total flink / process memory config options, to make it less clumsy without loss of expressiveness. To be specific, we propose to: * Change the config option key "taskmanager.memory.total-flink.size" to "taskmanager.memory.flink.size" * Change the config option key "taskmanager.memory.total-process.size" to "taskmanager.memory.process.size" Detailed discussion can be found in this [ML thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Some-feedback-after-trying-out-the-new-FLIP-49-memory-configurations-td36129.html]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15373) Update descriptions for framework / task off-heap memory config options
Xintong Song created FLINK-15373: Summary: Update descriptions for framework / task off-heap memory config options Key: FLINK-15373 URL: https://issues.apache.org/jira/browse/FLINK-15373 Project: Flink Issue Type: Improvement Components: Runtime / Configuration Reporter: Xintong Song Fix For: 1.10.0 Update descriptions for "taskmanager.memory.framework.off-heap.size" and "taskmanager.memory.task.off-heap.size" to explicitly state that: * Both direct and native memory are accounted * Will be fully counted into MaxDirectMemorySize Detailed discussion can be found in this [ML thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Some-feedback-after-trying-out-the-new-FLIP-49-memory-configurations-td36129.html]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15374) Update descriptions for jvm overhead config options
Xintong Song created FLINK-15374: Summary: Update descriptions for jvm overhead config options Key: FLINK-15374 URL: https://issues.apache.org/jira/browse/FLINK-15374 Project: Flink Issue Type: Improvement Components: Runtime / Configuration Reporter: Xintong Song Fix For: 1.10.0 Update descriptions for "taskmanager.memory.jvm-overhead.[min|max|fraction]" to remove "I/O direct memory" and explicitly state that it's not counted into MaxDirectMemorySize. Detailed discussion can be found in this [ML thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Some-feedback-after-trying-out-the-new-FLIP-49-memory-configurations-td36129.html]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15375) Improve MemorySize to print / parse with better readability.
Xintong Song created FLINK-15375:
Summary: Improve MemorySize to print / parse with better
readability.
Key: FLINK-15375
URL: https://issues.apache.org/jira/browse/FLINK-15375
Project: Flink
Issue Type: Improvement
Components: Runtime / Configuration
Reporter: Xintong Song
Fix For: 1.10.0
* Print MemorySize with proper unit rather than tremendous number of bytes.
* Parse memory size in numbers instead of {{parse(xxx + "m")}}
Detailed discussion can be found in this [ML
thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Some-feedback-after-trying-out-the-new-FLIP-49-memory-configurations-td36129.html].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15382) Flink failed generating python config docs
Xintong Song created FLINK-15382:
Summary: Flink failed generating python config docs
Key: FLINK-15382
URL: https://issues.apache.org/jira/browse/FLINK-15382
Project: Flink
Issue Type: Bug
Components: API / Python, Runtime / Configuration
Reporter: Xintong Song
When generating config option docs with the command suggested by
{{flink-docs/README.md}}, the generated
{{docs/_includes/generated/python_configuration.html}} does not contain any
config options despite that there are 4 options in {{PythonOptions}}.
I encountered this problem at the commit
{{545534e43ed37f518fe59b6ddd8ed56ae82a234b}} on master branch.
Command used to generate doc:
{code:bash}mvn package -Dgenerate-config-docs -pl flink-docs -am -nsu
-DskipTests{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15403) 'State Migration end-to-end test from 1.6' is unstable on travis.
Xintong Song created FLINK-15403:
Summary: 'State Migration end-to-end test from 1.6' is unstable on
travis.
Key: FLINK-15403
URL: https://issues.apache.org/jira/browse/FLINK-15403
Project: Flink
Issue Type: Bug
Components: Tests
Affects Versions: 1.10.0
Reporter: Xintong Song
Fix For: 1.10.0
https://api.travis-ci.org/v3/job/629576631/log.txt
The test case fails because the log contains the following error message.
{code}
2019-12-26 09:19:35,537 ERROR
org.apache.flink.streaming.runtime.tasks.StreamTask - Received
CancelTaskException while we are not canceled. This is a bug and should be
reported
org.apache.flink.runtime.execution.CancelTaskException: Consumed partition
PipelinedSubpartitionView(index: 0) of ResultPartition
3886657fb8cc980139fac67e32d6e380@8cfcbe851fe3bb3fa00e9afc370bd963 has been
released.
at
org.apache.flink.runtime.io.network.partition.consumer.LocalInputChannel.getNextBuffer(LocalInputChannel.java:190)
at
org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.waitAndGetNextData(SingleInputGate.java:509)
at
org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:487)
at
org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.pollNext(SingleInputGate.java:475)
at
org.apache.flink.runtime.taskmanager.InputGateWithMetrics.pollNext(InputGateWithMetrics.java:75)
at
org.apache.flink.streaming.runtime.io.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:125)
at
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:133)
at
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:69)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:311)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:187)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:488)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:470)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:702)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:527)
at java.lang.Thread.run(Thread.java:748)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15480) BlobsCleanupITCase is unstable on travis
Xintong Song created FLINK-15480:
Summary: BlobsCleanupITCase is unstable on travis
Key: FLINK-15480
URL: https://issues.apache.org/jira/browse/FLINK-15480
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.10.0
Reporter: Xintong Song
Fix For: 1.10.0
https://api.travis-ci.com/v3/job/272321636/log.txt
{code}
03:52:11.256 [ERROR]
testBlobServerCleanupFinishedJob(org.apache.flink.runtime.jobmanager.BlobsCleanupITCase)
Time elapsed: 298.556 s <<< FAILURE!
java.lang.AssertionError:
Expected: is
but: was
at
org.apache.flink.runtime.jobmanager.BlobsCleanupITCase.testBlobServerCleanup(BlobsCleanupITCase.java:220)
at
org.apache.flink.runtime.jobmanager.BlobsCleanupITCase.testBlobServerCleanupFinishedJob(BlobsCleanupITCase.java:133)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15530) Replace process memory with flink memory for TMs in default flink-conf.yaml
Xintong Song created FLINK-15530: Summary: Replace process memory with flink memory for TMs in default flink-conf.yaml Key: FLINK-15530 URL: https://issues.apache.org/jira/browse/FLINK-15530 Project: Flink Issue Type: Improvement Components: Runtime / Configuration Reporter: Xintong Song Fix For: 1.10.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-15541) FlinkKinesisConsumerTest.testSourceSynchronization is unstable on travis.
Xintong Song created FLINK-15541:
Summary: FlinkKinesisConsumerTest.testSourceSynchronization is
unstable on travis.
Key: FLINK-15541
URL: https://issues.apache.org/jira/browse/FLINK-15541
Project: Flink
Issue Type: Bug
Components: Connectors / Kinesis
Affects Versions: 1.10.0
Reporter: Xintong Song
Fix For: 1.10.0
[https://api.travis-ci.org/v3/job/634712405/log.txt]
{code:java}
13:16:19.144 [ERROR] Tests run: 11, Failures: 1, Errors: 0, Skipped: 0, Time
elapsed: 4.338 s <<< FAILURE! - in
org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumerTest
13:16:19.144 [ERROR]
testSourceSynchronization(org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumerTest)
Time elapsed: 1.001 s <<< FAILURE!
java.lang.AssertionError: expected null, but was:
at
org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumerTest.testSourceSynchronization(FlinkKinesisConsumerTest.java:1018)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15564) YarnClusterDescriptorTest failed to validate the original intended behavior
Xintong Song created FLINK-15564:
Summary: YarnClusterDescriptorTest failed to validate the original
intended behavior
Key: FLINK-15564
URL: https://issues.apache.org/jira/browse/FLINK-15564
Project: Flink
Issue Type: Bug
Components: Deployment / YARN
Reporter: Xintong Song
As discovered in [PR#10834|https://github.com/apache/flink/pull/10834], the
following test cases of {{YarnClusterDescriptorTest}} have failed to validate
the original intended behavior and are temporally skipped by PR#10834.
- {{testFailIfTaskSlotsHigherThanMaxVcores}}
- {{testConfigOverwrite}}
The original purpose of these two test cases was to verify the validation logic
against yarn max allocation vcores (in
{{5836f7eddb4849b95d4860cf20045bc61d061918}}).
These two cases should have failed when we change the validation logic to get
yarn max allocation vcores from yarnClient instead of configuration (in
{{e959e6d0cd42f0c5b21c0f03ce547f2025ac58d5}}), because there are no yarn
cluster (neither {{MiniYARNCluster}}) started in these cases, thus
{{yarnClient#getNodeReports}} will never return.
The cases have not failed because another {{IllegalConfigurationException}} was
thrown in {{validateClusterSpecification}}, because of memory validation
failure. The memory validation failure was by design, and in order to verify
the original purpose these two test cases should have been updated with
reasonable memory sizes, which is unfortunately overlooked.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15598) Memory accuracy loss in YarnClusterDescriptor may lead to deployment failure.
Xintong Song created FLINK-15598: Summary: Memory accuracy loss in YarnClusterDescriptor may lead to deployment failure. Key: FLINK-15598 URL: https://issues.apache.org/jira/browse/FLINK-15598 Project: Flink Issue Type: Bug Components: Deployment / YARN Reporter: Xintong Song Fix For: 1.10.0 Currently, YarnClusterDescriptor parses/derives TM process memory size from configuration, store it in ClusterSpecification and validate ClusterSpecification, then overwrite the memory size back to configuration. This logic is unnecessary. The memory validation is already covered by creating TaskExecutorResourceSpec from configuration in TaskExecutorResourceUtils. Moreover, the memory size is stored in MB in ClusterSpecification. The accuracy loss may lead to memory validation failure, which prevent the cluster from being deployed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16111) Kubernetes deployment does not respect "taskmanager.cpu.cores".
Xintong Song created FLINK-16111: Summary: Kubernetes deployment does not respect "taskmanager.cpu.cores". Key: FLINK-16111 URL: https://issues.apache.org/jira/browse/FLINK-16111 Project: Flink Issue Type: Bug Components: Deployment / Kubernetes Affects Versions: 1.10.0 Reporter: Xintong Song Fix For: 1.10.1, 1.11.0 The Kubernetes deployment uses `kubernetes.taskmanager.cpu` for configuring TM cpu cores, and will fallback to number-of-slots if not specified. FLINK-14188 introduced a common option `taskmanager.cpu.cores` (ATM not exposed to users and for internal usage only). A common logic is to decide the TM cpu cores following the fallback order of "common option -> K8s/Yarn/Mesos specific option -> numberOfSlot". The above fallback rules are not respected by the Kubernetes deployment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16299) Release containers recovered from previous attempt in which TaskExecutor is not started.
Xintong Song created FLINK-16299:
Summary: Release containers recovered from previous attempt in
which TaskExecutor is not started.
Key: FLINK-16299
URL: https://issues.apache.org/jira/browse/FLINK-16299
Project: Flink
Issue Type: Improvement
Components: Deployment / YARN
Reporter: Xintong Song
As discussed in FLINK-16215, on Yarn deployment, {{YarnResourceManager}} starts
a new {{TaskExecutor}} in two steps:
# Request a new container from Yarn
# Starts a {{TaskExecutor}} process in the allocated container
If JM failover happens between the two steps, in the new attempt
{{YarnResourceManager}} will not start {{TaskExecutor}} processes in recovered
containers. That means such containers are neither used nor released.
A potential fix to this problem, is to query form the container status by
calling {{NMClientAsync#getContainerStatusAsync}}, and release the containers
whose state is {{NEW}}, keeps only those whose state is {{RUNNING}} and waiting
for them to register.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16437) Make SlotManager allocate resource from ResourceManager at the worker granularity.
Xintong Song created FLINK-16437: Summary: Make SlotManager allocate resource from ResourceManager at the worker granularity. Key: FLINK-16437 URL: https://issues.apache.org/jira/browse/FLINK-16437 Project: Flink Issue Type: Sub-task Components: Runtime / Coordination Reporter: Xintong Song Fix For: 1.11.0 This is the first step of FLINK-14106, including all the major changes inside SlotManager and changes to the RM/SM interfaces, except changes for metrics and status. At the end of this step, SlotManager should allocate resource from ResourceManager with a WorkerResourceSpec, instead of slot ResourceProfile. At this step, the WorkerResourceSpec will not be used, and the active RMs will always use `ActiveResourceManager#taskExecutorProcessSpec` for requesting TMs. We will change that in subsequent steps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16438) Make YarnResourceManager starts workers using WorkerResourceSpec requested by SlotManager
Xintong Song created FLINK-16438: Summary: Make YarnResourceManager starts workers using WorkerResourceSpec requested by SlotManager Key: FLINK-16438 URL: https://issues.apache.org/jira/browse/FLINK-16438 Project: Flink Issue Type: Sub-task Reporter: Xintong Song This means YarnResourceManager no longer: - be aware of the default task executor resources - assumes all workers are identical -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16439) Make KubernetesResourceManager starts workers using WorkerResourceSpec requested by SlotManager
Xintong Song created FLINK-16439: Summary: Make KubernetesResourceManager starts workers using WorkerResourceSpec requested by SlotManager Key: FLINK-16439 URL: https://issues.apache.org/jira/browse/FLINK-16439 Project: Flink Issue Type: Sub-task Components: Deployment / Kubernetes Reporter: Xintong Song Fix For: 1.11.0 This means KubernetesResourceManager no longer: - be aware of the default task executor resources - assumes all workers are identical -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16440) Extend SlotManager metrics and status for dynamic slot allocation.
Xintong Song created FLINK-16440: Summary: Extend SlotManager metrics and status for dynamic slot allocation. Key: FLINK-16440 URL: https://issues.apache.org/jira/browse/FLINK-16440 Project: Flink Issue Type: Sub-task Components: Runtime / Metrics Reporter: Xintong Song Fix For: 1.11.0 * Create a slotManagerMetricGroup in resourceManagerMetricGroup, pass it into SM and register slot related metrics there. * This allows registering different metrics for different SM implementation. * For backwards compatibility, the slotManagerMetricGroup should have the same path as the resourceManagerMetricGroup. * Extend ResourceOverview and TaskManagerInfo to contain TM total / free / allocated resources. * Need to add methods to SM for getting TM resource status. * For SlotManagerImpl, * The existing methods for getting number of registered / free slots need no changes. * TM resource status can be computed from TaskExecutorProcessSpec, slot profiles and number of free slots. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16442) Make MesosResourceManager starts workers using WorkerResourceSpec requested by SlotManager
Xintong Song created FLINK-16442: Summary: Make MesosResourceManager starts workers using WorkerResourceSpec requested by SlotManager Key: FLINK-16442 URL: https://issues.apache.org/jira/browse/FLINK-16442 Project: Flink Issue Type: Sub-task Components: Deployment / Mesos Reporter: Xintong Song This means MesosResourceManager no longer: - be aware of the default task executor resources - assumes all workers are identical TBH, I'm not sure how many use cases do we have that needs to bring a different slot allocation strategy to the Mesos deployment. I think we can discuss whether we want to do this step or not. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16563) CommandLineParser should fail with explicit error message when parsing recognized arguments.
Xintong Song created FLINK-16563:
Summary: CommandLineParser should fail with explicit error message
when parsing recognized arguments.
Key: FLINK-16563
URL: https://issues.apache.org/jira/browse/FLINK-16563
Project: Flink
Issue Type: Improvement
Components: Runtime / Configuration
Affects Versions: 1.10.0
Reporter: Xintong Song
Fix For: 1.11.0
Currently, {{CommandLineParser}} will stop parsing silently if it meets an
unrecognized option, leaving the remaining tokens to "args" rather than
"options".
This sometimes lead to problems due to absence of subsequence options, and the
error messages do not point to the true root cause.
[Example|[http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-1-10-container-memory-configuration-with-Mesos-td33594.html]]
reported in the user ML.
I've checked and it seems that the "args" generated by {{CommandLineParser }}is
not really used anywhere. Therefore, I propose to make the parser fail fast
with explicit error message at unrecognized tokens.
The proposed changes are basically as follows:
* In {{CommandLineParser#parse}}, call {{DefaultParser#parse}} with the
argument {{stopAtNonOption}} set to {{false}}.
* Remove args from {{ClusterConfiguration}} and its sub-classes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-27514) Website links to up-to-date committer and PMC member lists
Xintong Song created FLINK-27514: Summary: Website links to up-to-date committer and PMC member lists Key: FLINK-27514 URL: https://issues.apache.org/jira/browse/FLINK-27514 Project: Flink Issue Type: Improvement Components: Project Website Reporter: Xintong Song Assignee: Xintong Song According to the [ML discussion|https://lists.apache.org/thread/679ds6lfqs8f4q8lnt7tnlofl58str4y], we are going to add a link to the up-to-date [committer and PMC member list|https://projects.apache.org/committee.html?flink] from the community page of our project website, as well as a notice about the current list could be outdated. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (FLINK-27719) Create Apache Flink slack workspace
Xintong Song created FLINK-27719: Summary: Create Apache Flink slack workspace Key: FLINK-27719 URL: https://issues.apache.org/jira/browse/FLINK-27719 Project: Flink Issue Type: Improvement Reporter: Xintong Song Assignee: Xintong Song This is an umbrella for tasks related to setting up the Apache Flink slack channel. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (FLINK-27720) Slack: Update the project website regarding communication channels
Xintong Song created FLINK-27720: Summary: Slack: Update the project website regarding communication channels Key: FLINK-27720 URL: https://issues.apache.org/jira/browse/FLINK-27720 Project: Flink Issue Type: Sub-task Reporter: Xintong Song -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (FLINK-27721) Slack: set up archive
Xintong Song created FLINK-27721: Summary: Slack: set up archive Key: FLINK-27721 URL: https://issues.apache.org/jira/browse/FLINK-27721 Project: Flink Issue Type: Sub-task Reporter: Xintong Song -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (FLINK-27722) Slack: set up auto-updated invitation link
Xintong Song created FLINK-27722: Summary: Slack: set up auto-updated invitation link Key: FLINK-27722 URL: https://issues.apache.org/jira/browse/FLINK-27722 Project: Flink Issue Type: Sub-task Reporter: Xintong Song -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (FLINK-27723) Slack: revisit how this communication channel works
Xintong Song created FLINK-27723: Summary: Slack: revisit how this communication channel works Key: FLINK-27723 URL: https://issues.apache.org/jira/browse/FLINK-27723 Project: Flink Issue Type: Sub-task Reporter: Xintong Song -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (FLINK-27731) Cannot build documentation with Hugo docker image
Xintong Song created FLINK-27731: Summary: Cannot build documentation with Hugo docker image Key: FLINK-27731 URL: https://issues.apache.org/jira/browse/FLINK-27731 Project: Flink Issue Type: Bug Components: Documentation Affects Versions: 1.16.0 Reporter: Xintong Song Flink provides 2 ways for building the documentation: 1) using a Hugo docker image, and 2) using a local Hugo installation. Currently, 1) is broken due to the `setup_docs.sh` script requires a local Hugo installation. This was introduced in FLINK-27394. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (FLINK-27848) ZooKeeperLeaderElectionDriver keeps writing leader information, using up zxid
Xintong Song created FLINK-27848:
Summary: ZooKeeperLeaderElectionDriver keeps writing leader
information, using up zxid
Key: FLINK-27848
URL: https://issues.apache.org/jira/browse/FLINK-27848
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.14.4, 1.15.0
Reporter: Xintong Song
Assignee: Weijie Guo
Fix For: 1.14.5, 1.15.1
After a leadership change, the new leader may keeps writing its information
(which is identical) to ZK, causing the zxid on ZK quickly used up.
The problem is that, in
{{ZooKeeperLeaderElectionDriver#retrieveLeaderInformationFromZooKeeper}},
{{leaderElectionEventHandler.onLeaderInformationChange(LeaderInformation.empty())}}
is called no matter {{childData}} is {{null}} or not. In case of non-null,
this will cause the driver keeps re-writing the leader information to ZK.
The problem is introduced in FLINK-24038, and only affects the legacy
{{ZooKeeperHaServices}}. Thus, only 1.14 / 1.15 are affected.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (FLINK-25356) Add benchmarks for performance in OLAP scenarios
Xintong Song created FLINK-25356: Summary: Add benchmarks for performance in OLAP scenarios Key: FLINK-25356 URL: https://issues.apache.org/jira/browse/FLINK-25356 Project: Flink Issue Type: Sub-task Components: Benchmarks Reporter: Xintong Song As discussed in FLINK-25318, we would need a unified, public visible benchmark setups, for supporting OLAP performance improvements and investigations. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25970) SerializedThrowable should record type of the original throwable.
Xintong Song created FLINK-25970:
Summary: SerializedThrowable should record type of the original
throwable.
Key: FLINK-25970
URL: https://issues.apache.org/jira/browse/FLINK-25970
Project: Flink
Issue Type: Improvement
Components: API / Core
Affects Versions: 1.14.3
Reporter: Xintong Song
Currently, only the message and stack of the original throwable is preserved in
{{{}SerializedThrowable{}}}, while the type of the original throwable is
discarded.
Sometimes, it would be helpful if message of {{SerializedThrowable}} can also
include the full class name of the original throwable.
E.g., in the following stack.
{code:java}
Caused by: org.apache.flink.util.SerializedThrowable
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) ~[?:1.8.0_102]
at java.util.HashMap$EntryIterator.next(HashMap.java:1471) ~[?:1.8.0_102]
at java.util.HashMap$EntryIterator.next(HashMap.java:1469) ~[?:1.8.0_102]
...
{code}
It's not that easy to understand what is wrong from this stack. JDK does not
provide a message for the original exception. We have to look into the JDK
source codes to find out what's going on. Sometimes it's even more annoying
having to look for the JDK source codes of the exactly same version in order to
match the line numbers.
Turns out the original exception was a {{ConcurrentModificationException}}. I
think it would be much more straightforward if we can have a stack like the
following.
{code}
Caused by: org.apache.flink.util.SerializedThrowable:
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) ~[?:1.8.0_102]
at java.util.HashMap$EntryIterator.next(HashMap.java:1471) ~[?:1.8.0_102]
at java.util.HashMap$EntryIterator.next(HashMap.java:1469) ~[?:1.8.0_102]
...
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-18581) Cannot find GC cleaner with java version previous 8u202
Xintong Song created FLINK-18581:
Summary: Cannot find GC cleaner with java version previous 8u202
Key: FLINK-18581
URL: https://issues.apache.org/jira/browse/FLINK-18581
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.11.0
Reporter: Xintong Song
{{JavaGcCleanerWrapper}} is looking for the package-private method
{{Reference.tryHandlePending}} using reflection. However, the method is first
introduced in the version 8u202. Therefore, if an older version JDK is used,
the method cannot be found and Flink will fail.
See also this [ML
thread|http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Error-GC-Cleaner-Provider-Flink-1-11-0-td36565.html].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18620) Unify behaviors of active resource managers
Xintong Song created FLINK-18620: Summary: Unify behaviors of active resource managers Key: FLINK-18620 URL: https://issues.apache.org/jira/browse/FLINK-18620 Project: Flink Issue Type: Improvement Components: Runtime / Coordination Reporter: Xintong Song Assignee: Xintong Song Flink supports various deployment modes: standalone, Kubernetes, Yarn & Mesos. For each deployment mode, a resource manager is implemented for managing the resources. While StandaloneResourceManager is quite different from the others by not being able to dynamically request and release resources, the other three (KubernetesResourceManager, YarnResourceManager and MesosResourceManager) share many logics in common. These common logics are currently duplicately implemented by each of the active resource managers. Such duplication leads to extra maintaining overhead and amplifies stability risks. This ticket proposes a refactor design for the resource managers, with better abstraction deduplicating common logics implementations and minimizing the deployment specific behaviors. This proposal is a pure refactor effort. It does not intend to change any of the current resource management behaviors. A detailed design doc and a simplified proof-of-concept implementation for the Kubernetes deployment are linked to this ticket. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18639) Error messages from BashJavaUtils is eaten
Xintong Song created FLINK-18639: Summary: Error messages from BashJavaUtils is eaten Key: FLINK-18639 URL: https://issues.apache.org/jira/browse/FLINK-18639 Project: Flink Issue Type: Bug Components: Deployment / Scripts Affects Versions: 1.11.0, 1.12.0 Reporter: Xintong Song Assignee: Xintong Song Fix For: 1.12.0, 1.11.1 Shell scripts execute BashJavaUtils for generating memory related JVM parameters and dynamic configurations. When there's a problem in configuration that the memory sizes cannot be properly calculated, the script will not launch the Flink daemon and exit with non-zero code. In such cases, error messages from BashJavaUtils describing the reason of failure are missing, making it hard for users to understand what's wrong and how to fix the problem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18719) Define the interfaces and introduce ActiveResourceManager
Xintong Song created FLINK-18719: Summary: Define the interfaces and introduce ActiveResourceManager Key: FLINK-18719 URL: https://issues.apache.org/jira/browse/FLINK-18719 Project: Flink Issue Type: Sub-task Components: Runtime / Coordination Reporter: Xintong Song * Define the interface ResourceManagerDriver and ResourceEventHandler * Rename the original ActiveResourceManager to LegacyActiveResourceManager. * Introduce the new ActiveResourceManager. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18720) Migrate KubernetesResourceManager to the new KubernetesResourceManagerDriver
Xintong Song created FLINK-18720: Summary: Migrate KubernetesResourceManager to the new KubernetesResourceManagerDriver Key: FLINK-18720 URL: https://issues.apache.org/jira/browse/FLINK-18720 Project: Flink Issue Type: Sub-task Components: Deployment / Kubernetes, Runtime / Coordination Reporter: Xintong Song Fix For: 1.12.0 * Introduce KubernetesResourceManagerDriver * Switch to ActiveResourceManager and KubernetesResourceManagerDriver for Kubernetes deployment * Remove KubernetesResourceManager -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18722) Migrate MesosResourceManager to the new MesosResourceManagerDriver
Xintong Song created FLINK-18722: Summary: Migrate MesosResourceManager to the new MesosResourceManagerDriver Key: FLINK-18722 URL: https://issues.apache.org/jira/browse/FLINK-18722 Project: Flink Issue Type: Sub-task Components: Deployment / Mesos, Runtime / Coordination Reporter: Xintong Song -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18721) Migrate YarnResourceManager to the new YarnResourceManagerDriver
Xintong Song created FLINK-18721: Summary: Migrate YarnResourceManager to the new YarnResourceManagerDriver Key: FLINK-18721 URL: https://issues.apache.org/jira/browse/FLINK-18721 Project: Flink Issue Type: Sub-task Components: Deployment / YARN, Runtime / Coordination Reporter: Xintong Song Fix For: 1.12.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18738) Revisit resource management model for python processes.
Xintong Song created FLINK-18738: Summary: Revisit resource management model for python processes. Key: FLINK-18738 URL: https://issues.apache.org/jira/browse/FLINK-18738 Project: Flink Issue Type: Task Components: API / Python, Runtime / Coordination Reporter: Xintong Song Assignee: Xintong Song Fix For: 1.12.0 This ticket is for tracking the effort towards a proper long-term resource management model for python processes. In FLINK-17923, we run into problems due to python processes are not well integrate with the task manager resource management mechanism. A temporal workaround has been merged for release-1.11. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18760) Redundant task managers should be released when there's no job running in session cluster
Xintong Song created FLINK-18760: Summary: Redundant task managers should be released when there's no job running in session cluster Key: FLINK-18760 URL: https://issues.apache.org/jira/browse/FLINK-18760 Project: Flink Issue Type: Improvement Components: Runtime / Coordination Affects Versions: 1.12.0 Reporter: Xintong Song Assignee: Liu Fix For: 1.12.0 In FLINK-18625, we introduced redundant task managers, as backup resources for speeding up job recovery in cases of task manager lost. For a session cluster, when there's no job running, it would be better to not keep such redundant resources. Currently, Flink session cluster will not request redundant task managers until a job is submitted, but it also will not release redundant task managers after all jobs terminated. This ticket proposes to check and release task managers if there are redundant task managers only. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19057) Avoid usage of GlobalConfiguration in ResourceManager
Xintong Song created FLINK-19057:
Summary: Avoid usage of GlobalConfiguration in ResourceManager
Key: FLINK-19057
URL: https://issues.apache.org/jira/browse/FLINK-19057
Project: Flink
Issue Type: Task
Components: Runtime / Coordination
Reporter: Xintong Song
This is a follow up of this PR
[discussion|https://github.com/apache/flink/pull/13186/#discussion_r476459874].
On Kubernetes/Yarn deployments, resource manager try to compare the effective
configurations with the original configuration file shipped from client, and
only set the differences to dynamic properties for task managers.
During which, {{GlobalConfiguration.loadConfiguration()}} is used for getting
the original configuration file. The strongly relies on that Kubernetes/Yarn
entry points do not support custom configuration directories, which is true at
the moment but brittle in future.
It would be better to rethink the usage of GlobalConfiguration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-19068) Filter verbose pod events for KubernetesResourceManagerDriver
Xintong Song created FLINK-19068:
Summary: Filter verbose pod events for
KubernetesResourceManagerDriver
Key: FLINK-19068
URL: https://issues.apache.org/jira/browse/FLINK-19068
Project: Flink
Issue Type: Improvement
Components: Deployment / Kubernetes
Reporter: Xintong Song
A status of a Kubernetes pod consists of many detailed fields. Currently, Flink
receives pod {{MODIFIED}} events from the {{KubernetesPodsWatcher}} on every
single change to these fields, many of which Flink does not care.
The verbose events will not affect the functionality of Flink, but will pollute
the logs with repeated messages, because Flink only looks into the fields it
interested in and those fields are identical.
E.g., when a task manager is stopped due to idle timeout, Flink receives 3
events:
* MODIFIED: container terminated
* MODIFIED: {{deletionGracePeriodSeconds}} changes from 30 to 0, which is a
Kubernetes internal status change after containers are gracefully terminated
* DELETED: Flink removes metadata of the terminated pod
Among the 3 messages, Flink is only interested in the 1st MODIFIED message, but
will try to process all of them because the container status is terminated.
I propose to Filter the verbose events in
{{KubernetesResourceManagerDriver.PodCallbackHandlerImpl}}, to only process the
status changes interested by Flink. This probably requires recording the status
of all living pods, to compare with the incoming events for detecting status
changes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-19117) FLINK-18620
Xintong Song created FLINK-19117: Summary: FLINK-18620 Key: FLINK-19117 URL: https://issues.apache.org/jira/browse/FLINK-19117 Project: Flink Issue Type: Improvement Reporter: Xintong Song -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19151) Flink does not normalize container resource with correct configurations when Yarn FairScheduler is used
Xintong Song created FLINK-19151: Summary: Flink does not normalize container resource with correct configurations when Yarn FairScheduler is used Key: FLINK-19151 URL: https://issues.apache.org/jira/browse/FLINK-19151 Project: Flink Issue Type: Bug Components: Deployment / YARN Affects Versions: 1.11.2 Reporter: Xintong Song h3. Problem It's a Yarn protocol that the requested container resource will be normalized for allocation. That means, the allocated container may have different resource (larger than or equal to) compared to requested. Currently, Flink matches the allocated containers to the original requests by reading the Yarn configurations and calculate how the requested resources should be normalized. What has been overlooked is that, Yarn FairScheduler (and its subclass SLSFairScheduler) has overridden the normalization behavior. To be specific, * By default, Yarn normalize container resources to integer multiple of "yarn.scheduler.minimum-allocation-[mb|vcores]" * FairScheduler normalize container resources to integer multiple of "yarn.resource-types.[memory-mb|vcores].increment-allocation" (or the deprecated keys "yarn.scheduler.increment-allocation-[mb|vcores]"), while making sure the resource is no less than "yarn.scheduler.minimum-allocation-[mb|vcores]" h3. Proposal for short term solution To fix this problem, a quick and easy way is to also read Yarn configuration and learn which scheduler is used, and perform normalization calculations accordingly. This should be good enough to cover behaviors of all the schedulers that Yarn currently provides. The limitation is that, Flink will not be able to deal with custom Yarn schedulers which overrides the normalization behaviors. h3. Proposal for long term solution For long term, it would be good to use Yarn ContainerRequest#allocationRequestId to match the allocated containers with the original requests, so that Flink no longer needs to understand how Yarn normalize container resources. Yarn ContainerRequest#allocationRequestId is introduced in Hadoop 2.9, while ATM Flink claims to be compatible with Hadoop 2.4+. Therefore, this solution would not work at the moment. Another idea is to support various Hadoop versions with different container matching logics. We can abstract the container matching logics into a dedicating component, and provide different implementations for it. This will allow Flink to take advantages of the new versions (e.g., work well with custom schedulers), while stay compatible with the old versions with without those advantages. Given that we need the resource based matching anyway for the old Hadoop versions, and the cost for maintaining two sets of matching logics, I tend to think this approach as a back-up option to be worked on when we indeed see a need for it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19177) FLIP-141: Intra-Slot Managed Memory Sharing
Xintong Song created FLINK-19177: Summary: FLIP-141: Intra-Slot Managed Memory Sharing Key: FLINK-19177 URL: https://issues.apache.org/jira/browse/FLINK-19177 Project: Flink Issue Type: Improvement Components: Runtime / Coordination Reporter: Xintong Song Assignee: Xintong Song Fix For: 1.12.0 This is the umbrella ticket of [FLIP-141: Intra-Slot Managed Memory Sharing|https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing]. [FLIP-53|https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management] introduced a fraction based approach for sharing managed memory within a slot. This approach needs to be extended as python operators, which also use managed memory, are introduced. This FLIP proposes a design for extending intra-slot managed memory sharing for python operators and other potential future managed memory use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19178) Introduce the memory weights configuration option
Xintong Song created FLINK-19178: Summary: Introduce the memory weights configuration option Key: FLINK-19178 URL: https://issues.apache.org/jira/browse/FLINK-19178 Project: Flink Issue Type: Sub-task Reporter: Xintong Song -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19179) Implement the new fraction calculation logic
Xintong Song created FLINK-19179: Summary: Implement the new fraction calculation logic Key: FLINK-19179 URL: https://issues.apache.org/jira/browse/FLINK-19179 Project: Flink Issue Type: Sub-task Reporter: Xintong Song This also means migrating the batch operator use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19180) Make RocksDB respect the calculated fraction
Xintong Song created FLINK-19180: Summary: Make RocksDB respect the calculated fraction Key: FLINK-19180 URL: https://issues.apache.org/jira/browse/FLINK-19180 Project: Flink Issue Type: Sub-task Reporter: Xintong Song -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19182) Update document for intra-slot managed memory sharing
Xintong Song created FLINK-19182: Summary: Update document for intra-slot managed memory sharing Key: FLINK-19182 URL: https://issues.apache.org/jira/browse/FLINK-19182 Project: Flink Issue Type: Sub-task Reporter: Xintong Song -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19181) Make python processes respect the calculated fraction
Xintong Song created FLINK-19181: Summary: Make python processes respect the calculated fraction Key: FLINK-19181 URL: https://issues.apache.org/jira/browse/FLINK-19181 Project: Flink Issue Type: Sub-task Reporter: Xintong Song -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19267) Calculate managed memory fractions for fine grained resource specs
Xintong Song created FLINK-19267: Summary: Calculate managed memory fractions for fine grained resource specs Key: FLINK-19267 URL: https://issues.apache.org/jira/browse/FLINK-19267 Project: Flink Issue Type: Task Components: Runtime / Coordination Reporter: Xintong Song This is a follow-up issue of FLIP-141, to support managed memory calculation for fine grained resource specs w.r.t. various use cases, per the discussion [here|https://github.com/apache/flink/pull/13397#discussion_r489250595]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19324) Map requested/allocated containers with priority on YARN
Xintong Song created FLINK-19324:
Summary: Map requested/allocated containers with priority on YARN
Key: FLINK-19324
URL: https://issues.apache.org/jira/browse/FLINK-19324
Project: Flink
Issue Type: Bug
Components: Deployment / YARN
Reporter: Xintong Song
In the design doc of FLINK-14106, there was a
[discussion|https://docs.google.com/document/d/1f8imSus3QwKEUPAldzR8CSMjZ-2a9O17-rn4oeKGtqw/edit?disco=GPX_tmg]
on how we map allocated containers with the requested ones on YARN. We
rejected the design option that uses container priorities for mapping
containers of different resources, because we do not want to priorities
different container requests (which is the original purpose for this field). As
a result, we have to interpret how the requested container request would be
normalized by Yarn, and map the allocated/requested containers accordingly,
which is complicated and fragile. See also FLINK-19151.
Recently in our POC for fine grained resource management, we surprisingly
discovered that Yarn actually doesn't work with container requests same
priority and different resources. I do not find this described as an official
protocol in any Yarn's documents. The issue has been raised in early Yarn
versions (YARN-314) and has not been fixed util Hadoop 2.9 when
{{allocationRequestId}} is introduced. In Hadoop 2.8, Yarn scheduler is still
internally using priority as the key of a container request (see
[AppSchedulingInfo#updateResourceRequests
|https://github.com/apache/hadoop/blob/eb818cdc64336ade273a960ba3b9b5a5d0c4d4ec/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java#L341]),
thus requests same priority and different resources would overwrite each other.
The new discovery suggests that, if we want to support containers with
different resources on Hadoop 2.8 and earlier versions, we have to give them
different priorities anyway. Thus, I would suggest to get rid of the container
normalization simulation and go back to the previously rejected priority based
design option.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-19568) Offload creating TM launch contexts to the IO executor
Xintong Song created FLINK-19568:
Summary: Offload creating TM launch contexts to the IO executor
Key: FLINK-19568
URL: https://issues.apache.org/jira/browse/FLINK-19568
Project: Flink
Issue Type: Improvement
Components: Deployment / YARN
Reporter: Xintong Song
Fix For: 1.12.0
Currently, for launching each TM container on Yarn, Flink creates a container
launch context in RM's PRC main thread. This includes accessing file status
from remote file systems, which may blocks the RM's main thread, especially
when remote file system is slow. See [this
thread|http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/TM-heartbeat-timeout-due-to-ResourceManager-being-busy-td38626.html].
The creating of TM context does not access nor change any RM's internal states.
Therefore, I propose to offload the work to the IO executor. To be specific, I
think the entire {{YarnResourceManagerDriver#createTaskExecutorLaunchContext}}
can be invoked on the IO executor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16733) Refactor YarnClusterDescriptor
Xintong Song created FLINK-16733:
Summary: Refactor YarnClusterDescriptor
Key: FLINK-16733
URL: https://issues.apache.org/jira/browse/FLINK-16733
Project: Flink
Issue Type: Improvement
Components: Deployment / YARN
Reporter: Xintong Song
Currently, YarnClusterDescriptor is not in a good shape. It has 1600+ lines of
codes, of which the method {{startAppMaster}} alone has 400+ codes, leading to
poor maintainability.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16734) Remove ClusterSpecification
Xintong Song created FLINK-16734:
Summary: Remove ClusterSpecification
Key: FLINK-16734
URL: https://issues.apache.org/jira/browse/FLINK-16734
Project: Flink
Issue Type: Improvement
Components: Command Line Client
Reporter: Xintong Song
Currently, {{ClusterSpecification}} has the following three fields.
* masterMemoryMB
* taskManagerMemoryMB
* slotsPerTaskManager
Among the three fields, {{taskManagerMemoryMB}} is only used in
{{YarnClusterDescriptor#validateClusterResources}}. It can be replaced by
"taskmanager.memory.process.size" in the configuration. Moreover, there are
consistency risks for keeping the process memory in two places with different
precision (MB vs. MemorySize).
{{masterMemoryMB}} should be the same as {{taskManagerMemoryMB}} after
finishing [FLIP-116 Unified Memory Configuration for Job
Managers|https://cwiki.apache.org/confluence/display/FLINK/FLIP+116%3A+Unified+Memory+Configuration+for+Job+Managers].
That leaves only {{slotsPerTaskManager}}, which can easily get from the
configuration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16798) Logs from BashJavaUtils are not properly preserved and passed into TM logs.
Xintong Song created FLINK-16798:
Summary: Logs from BashJavaUtils are not properly preserved and
passed into TM logs.
Key: FLINK-16798
URL: https://issues.apache.org/jira/browse/FLINK-16798
Project: Flink
Issue Type: Bug
Components: Deployment / Scripts
Affects Versions: 1.11.0
Reporter: Xintong Song
Fix For: 1.11.0
With FLINK-15519, in the TM start-up scripts, we have captured logs from
{{BashJavaUtils}} and passed into the TM JVM process via environment variable.
These logs will be merged with other TM logs, writing to same places respecting
user's log configurations.
This effort was broken in FLINK-15727, where the outputs from {{BashJavaUtils}}
are thrown away, except for the result JVM parameters and dynamic
configurations
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16953) TableEnvHiveConnectorTest is unstable on travis.
Xintong Song created FLINK-16953: Summary: TableEnvHiveConnectorTest is unstable on travis. Key: FLINK-16953 URL: https://issues.apache.org/jira/browse/FLINK-16953 Project: Flink Issue Type: Bug Components: Connectors / Hive Affects Versions: 1.11.0 Reporter: Xintong Song [https://api.travis-ci.org/v3/job/670405441/log.txt] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17061) Unset process/flink memory size from configuration once dynamic worker resource is activated.
Xintong Song created FLINK-17061:
Summary: Unset process/flink memory size from configuration once
dynamic worker resource is activated.
Key: FLINK-17061
URL: https://issues.apache.org/jira/browse/FLINK-17061
Project: Flink
Issue Type: Task
Components: Runtime / Configuration, Runtime / Coordination
Affects Versions: 1.11.0
Reporter: Xintong Song
With FLINK-14106, memory of a TaskExecutor is decided in two steps on active
resource managers.
- {{SlotManager}} decides {{WorkerResourceSpec}}, including memory used by
Flink tasks: task heap, task off-heap, network and managed memory.
- {{ResourceManager}} derives {{TaskExecutorProcessSpec}} from
{{WorkerResourceSpec}} and the configuration, deciding sizes of memory used by
Flink framework and JVM: framework heap, framework off-heap, jvm metaspace and
jvm overhead.
This works fine for now, because both {{WorkerResourceSpec}} and
{{TaskExecutorProcessSpec}} are derived from the same configurations. However,
it might cause problem if later we have new {{SlotManager}} implementations
that decides {{WorkerResourceSpec}} dynamically. In such cases, the
process/flink sizes in configuration should be ignored, or it may easily lead
to configuration conflicts.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17257) AbstractYarnClusterTest does not compile with Hadoop 2.10
Xintong Song created FLINK-17257:
Summary: AbstractYarnClusterTest does not compile with Hadoop 2.10
Key: FLINK-17257
URL: https://issues.apache.org/jira/browse/FLINK-17257
Project: Flink
Issue Type: Bug
Components: Deployment / YARN, Tests
Affects Versions: 1.9.3, 1.10.1, 1.11.0
Reporter: Xintong Song
Fix For: 1.11.0, 1.10.2, 1.9.4
In {{AbstractYarnClusterTest}}, we create {{ApplicationReport}} with the static
method {{ApplicationReport.newInstance}}, which is annotated as private and
unstable. This method is no longer compatible in Hadoop 2.10.
As a workaround, we can create {{ApplicationReport}} with its default
constructor and set only the fields that we need.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17390) Container resource cannot be mapped on Hadoop 2.10+
Xintong Song created FLINK-17390:
Summary: Container resource cannot be mapped on Hadoop 2.10+
Key: FLINK-17390
URL: https://issues.apache.org/jira/browse/FLINK-17390
Project: Flink
Issue Type: Bug
Components: Deployment / YARN
Affects Versions: 1.11.0
Reporter: Xintong Song
Fix For: 1.11.0
In FLINK-16438, we introduced {{WorkerSpecContainerResourceAdapter}} for
mapping Yarn container {{Resource}} with Flink {{WorkerResourceSpec}}. Inside
this class, we use {{Resource}} for hash map keys and set elements, assuming
that {{Resource}} instances that describes the same set of resources have the
same hash code.
This assumption is not always true. {{Resource}} is an abstract class and may
have different implementations. In Hadoop 2.10+, {{LightWeightResource}}, a new
implementation of {{Resource}}, is introduced for {{Resource}} generated by
{{Resource.newInstance}} on the AM side, which overrides the {{hashCode}}
method. That means, a {{Resource}} generated on AM may have a different hash
code compared to an equal {{Resource}} returned from Yarn.
To solve this problem, we may introduce an {{InternalResource}} as an inner
class of {{WorkerSpecContainerResourceAdapter}}, with {{hashCode}} method
depends only on the fields needed by Flink (ATM memroy and vcores).
{{WorkerSpecContainerResourceAdapter}} should only use {{InternalResource}} for
internal state management, and do conversions for {{Resource}} passed into and
returned from it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17422) Create user document for the external resource framework and the GPU plugin..
Xintong Song created FLINK-17422: Summary: Create user document for the external resource framework and the GPU plugin.. Key: FLINK-17422 URL: https://issues.apache.org/jira/browse/FLINK-17422 Project: Flink Issue Type: Sub-task Components: Documentation Affects Versions: 1.11.0 Reporter: Xintong Song Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17551) Documentation of stable releases are actually built on top of snapshot code bases.
Xintong Song created FLINK-17551: Summary: Documentation of stable releases are actually built on top of snapshot code bases. Key: FLINK-17551 URL: https://issues.apache.org/jira/browse/FLINK-17551 Project: Flink Issue Type: Bug Components: Project Website Affects Versions: 1.10.0 Reporter: Xintong Song When browsing Flink's documentation on the project website, we can choose from both the latest snapshot version and the stable release versions. However, it seems the documentation of stable release version is actually built on top of the snapshot version of the release branch. E.g., currently the latest stable release is 1.10.0, but the documentation described as "Flink 1.10 (Latest stable release)" is actually built with 1.10-SNAPSHOT. As a consequence, users might be confused when they use release 1.10.0 and some latest documentation changes meant for 1.10.1. [This comment|https://github.com/apache/flink/pull/11300#issuecomment-624776199] shows one of such confusions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17677) FLINK_LOG_PREFIX recommended in docs is not always available
Xintong Song created FLINK-17677:
Summary: FLINK_LOG_PREFIX recommended in docs is not always
available
Key: FLINK-17677
URL: https://issues.apache.org/jira/browse/FLINK-17677
Project: Flink
Issue Type: Bug
Components: Documentation
Affects Versions: 1.10.1, 1.9.3, 1.11.0
Reporter: Xintong Song
Fix For: 1.11.0, 1.10.2, 1.9.4
The [Application Profiling &
Debugging|https://ci.apache.org/projects/flink/flink-docs-master/monitoring/application_profiling.html]
documentation recommend to use the script variable {{FLINK_LOG_PREFIX}} for
defining log file paths. However, this variable is only available in standalone
mode. This is a bit misleading for users of other deployments (see this
[thread|http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Memory-analyze-on-AWS-EMR-td35036.html]).
I propose to replace {{FLINK_LOG_PREFIX}} with a general representation
{{}}, and add a separate section to discuss how to set the log
path (e.g., use {{FLINK_LOG_PREFIX}} with standalone deployments and
{{}} with Yarn deployments).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18175) Add human readable summary for configured and derived memory sizes.
Xintong Song created FLINK-18175: Summary: Add human readable summary for configured and derived memory sizes. Key: FLINK-18175 URL: https://issues.apache.org/jira/browse/FLINK-18175 Project: Flink Issue Type: Improvement Components: Runtime / Coordination Affects Versions: 1.10.1, 1.11.0 Reporter: Xintong Song FLIP-49 & FLIP-116 introduces sophisticated memory configurations for TaskManager and Master processes. Before the JVM processes are started, Flink derives the accurate sizes for all necessary components, based on both explicit user configurations and implicit defaults. To make the configuration results (especially those implicitly derived) clear to users, it would be helpful to print them in the beginning of the process logs. Currently, we only have printed JVM parameters (TM & Master) dynamic memory configurations (TM only). They are incomplete (jvm overhead for both processes and off-heap memory for the master process are not presented) and unfriendly (in bytes). Therefore, I propose to add a human readable summary at the beginning of process logs. See also this [PR discussion|https://github.com/apache/flink/pull/11445#discussion_r435721169]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18226) ResourceManager requests unnecessary new workers if previous workers are allocated but not registered.
Xintong Song created FLINK-18226:
Summary: ResourceManager requests unnecessary new workers if
previous workers are allocated but not registered.
Key: FLINK-18226
URL: https://issues.apache.org/jira/browse/FLINK-18226
Project: Flink
Issue Type: Bug
Components: Deployment / Kubernetes, Deployment / YARN, Runtime /
Coordination
Affects Versions: 1.11.0
Reporter: Xintong Song
Assignee: Xintong Song
Fix For: 1.11.0
h2. Problem
Currently on Kubernetes & Yarn deployment, the ResourceManager compares
*pending workers requested from Kubernetes/Yarn* against *pending workers
required by SlotManager*, for deciding whether new workers should be requested
in case of a worker failure.
* {{KubernetesResourceManager#requestKubernetesPodIfRequired}}
* {{YarnResourceManager#requestYarnContainerIfRequired}}
*Pending workers requested from Kubernetes/Yarn* is decreased when the worker
is allocated, *before the worker is actually started and registered*.
* Decreased in {{ActiveResourceManager#notifyNewWorkerAllocated}}, which is
called in
* {{KubernetesResourceManager#onAdded}}
* {{YarnResourceManager#onContainersOfResourceAllocated}}
On the other hand, *pending workers required by SlotManager* is derived from
the number of pending slots inside SlotManager, which is decreased *when the
new workers/slots are registered*.
* {{SlotManagerImpl#registerSlot}}
Therefore, if a worker {{w1}} is failed after another worker {{w2}} is
allocated but before {{w2}} is registered, the ResourceManager will request an
unnecessary new worker for {{w2}}.
h2. Impact
Normally, the extra worker should be released soon after allocated. But in
cases where the Kubernetes/Yarn cluster does not have enough resources, it
might create more and more pending pods/containers.
It's even more severe for Kubernetes, because
{{KubernetesResourceManager#onAdded}} only suggest that the pod spec has been
successfully added to the cluster, but the pod may not actually been allocated
due to lack of resources. Imagine there are {{N}} pending pods, a failure of a
running pod means requesting another {{N}} new pods.
In a session cluster, such pending pods could take long to be cleared even
after all jobs in the session cluster have terminated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18229) Pending worker requests should be properly cleared
Xintong Song created FLINK-18229:
Summary: Pending worker requests should be properly cleared
Key: FLINK-18229
URL: https://issues.apache.org/jira/browse/FLINK-18229
Project: Flink
Issue Type: Improvement
Components: Deployment / Kubernetes, Deployment / YARN, Runtime /
Coordination
Affects Versions: 1.10.1, 1.9.3, 1.11.0
Reporter: Xintong Song
Currently, if Kubernetes/Yarn does not have enough resources to fulfill Flink's
resource requirement, there will be pending pod/container requests on
Kubernetes/Yarn. These pending resource requirements are never cleared until
either fulfilled or the Flink cluster is shutdown.
However, sometimes Flink no longer needs the pending resources. E.g., the slot
request is then fulfilled by another slots that become available, or the job
failed due to slot request timeout (in a session cluster). In such cases, Flink
does not remove the resource request until the resource is allocated, then it
discovers that it no longer needs the allocated resource and release them. This
would affect the underlying Kubernetes/Yarn cluster, especially when the
cluster is under heavy workload.
It would be good for Flink to cancel pod/container requests as earlier as
possible if it can discover that some of the pending workers are no longer
needed.
There are several approaches potentially achieve this.
# We can always check whether there's a pending worker that can be canceled
when a \{{PendingTaskManagerSlot}} is unassigned.
# We can have a separate timeout for requesting new worker. If the resource
cannot be allocated within the given time since requested, we should cancel
that resource request and claim a resource allocation failure.
# We can share the same timeout for starting new worker (proposed in
FLINK-13554). This is similar to 2), but it requires the worker to be
registered, rather than allocated, before timeout.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
