GitHub user shuai-xu opened a pull request:
https://github.com/apache/flink/pull/4991
[FLINK-7928] [runtime] extend the resources in ResourceProfile for
precisely calculating the resource of task manager
Notes: this pull request contains the #4911 since it depends on it.
## What is the purpose of the change
This pull request makes task extendable with ResourceSpec( #4911), and add
a two field for calculating the memory needed for an operator to communicating
with its upstream and downstream.
## Brief change log
- *Add a extendedResource field for extendable resources in ResourceSpec*
- *Add memoryForInputInMB nad memoryForOutputInMB for the memory needed
for an operator to communicating with its upstream and downstream*
- *Add a fromResourceSpec method for transforming ResourceSpec to
ResourceProfile*
## Verifying this change
This change added tests and can be verified as follows:
- *Added test in ResourceProfileTest*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shuai-xu/flink jira-7928
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4991.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4991
----
commit 3e1d61a33f18b351424d4684cbaebc22674f582c
Author: shuai.xus <[email protected]>
Date: 2017-10-25T06:56:35Z
[FLINK-7878] [api] make resource type extendible in ResourceSpec
Summary:
Now, flink only support user define CPU and MEM,
but some user need to specify the GPU, FPGA and so on resources.
So it need to make the resouce type extendible in the ResourceSpec.
Add a extend field for new resources.
Test Plan: UnitTest
Reviewers: haitao.w
Differential Revision: https://aone.alibaba-inc.com/code/D327427
commit d769fe5d0184cd6ac264fd42552d290ae6978fbb
Author: shuai.xus <[email protected]>
Date: 2017-11-08T09:10:01Z
make Resource abstract and add GPUResource FPGAResource
commit f897d1fa1742c8186c93bb60abfd8719f156c7da
Author: shuai.xus <[email protected]>
Date: 2017-11-08T09:20:22Z
enhance test
commit b8e882b9f39f5588338297ce227e200c6527b84b
Author: shuai.xus <[email protected]>
Date: 2017-11-10T02:00:08Z
make create protected
commit 41cf6e4c7e68ef84d9d84e909b417fc6ddc794a6
Author: shuai.xus <[email protected]>
Date: 2017-11-10T03:02:21Z
make constructor public
commit 931e279e5a85f38e6cd9e53169fd37b8ce2d87ad
Author: shuai.xus <[email protected]>
Date: 2017-10-26T09:38:04Z
[FLINK-7928] [runtime] extend the resources in ResourceProfile for
precisely calculating the resource of task manager
Summary:
ResourceProfile denotes the resource requirements of a task. It should
contains:
1. The resource for the operators: the resources in ResourceSpec (please
refer to jira-7878)
2. The resource for the task to communicate with its upstreams.
3. The resource for the task to communicate with its downstreams.
Now the ResourceProfile only contains the first part. Adding the last two
parts.
Test Plan: UnitTests
Reviewers: haitao.w
Differential Revision: https://aone.alibaba-inc.com/code/D330364
commit 6665d570882efa49e35251092385efc8fb6adeb8
Author: shuai.xus <[email protected]>
Date: 2017-10-27T07:43:25Z
modify compare
commit 739564db031febd5bb029f08df3ced1ef539c7e6
Author: shuai.xus <[email protected]>
Date: 2017-10-30T04:01:42Z
add more denotes
commit c39c3597c1094bb258556d8d6dc12e5305903ea8
Author: shuai.xus <[email protected]>
Date: 2017-11-10T02:55:26Z
rebase with 7878
----
---
