Hi Rex,
We're running this in a local environment so that may be contributing to
> what we're seeing.
>
Just to double check on this. By `local environment`, you mean running
flink without setting up a standalone cluster or submitting it to a
K8s/Yarn cluster? (Typically executing from an IDE, running `flink run -t
local`, or running your own application that calls
`ExecutionEnvironment#execute`).
If yes, then this is kind of expected.
A couple of things that might help you understand this.
- Running on a local environment means setting up the Flink cluster
within the current process (the IDE process if executed from an IDE, the
flink client process if using `flink run -t local`, or your own application
process). That also means most of the resource configurations cannot take
effect, because the resources of the JVM are already determined. Please
refer to the memory configuration documents for options that still take
effect in local execution. [1][2]
- David is correct that `taskmanager.cpu.cores` is only intended for
internal usages. I assume you learnt about this configuration by reading
the source codes? If true, please also be aware that the JavaDoc
of `TaskManagerOption#CPU_CORES` says "DO NOT USE THIS CONFIG OPTION", and
it is also annotated with `ExcludeFromDocumentation` so that users do not
learn this option from the documents.
- Flink does not really control how many cpu cores it uses. However,
when running on an external resource management system (K8s, Yarn, Mesos),
it requires a certain amount of cpu resources for its containers/pods, and
allows the external system to control its cpu usage. You can use the
following configuration options to control how many cpu cores are requested
in such cases.
- kubernetes.jobmanager.cpu
- kubernetes.taskmanager.cpu
- yarn.appmaster.vcores
- yarn.containers.vcores
- mesos.resourcemanager.tasks.cpus
Thank you~
Xintong Song
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/memory/mem_setup_tm.html#local-execution
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/memory/mem_setup_jobmanager.html#local-execution
On Mon, Dec 7, 2020 at 10:32 AM Yangze Guo <[email protected]> wrote:
> Hi, Rex,
>
> Can you share more logs for it. Did you see something like "The
> configuration option taskmanager.cpu.cores required for local
> execution is not set, setting it to" in your logs?
>
> Best,
> Yangze Guo
>
> Best,
> Yangze Guo
>
>
> On Sat, Dec 5, 2020 at 6:53 PM David Anderson <[email protected]>
> wrote:
> >
> > taskmanager.cpu.cores is intended for internal use only -- you aren't
> meant to set this option. What happens if you leave it alone?
> >
> > Regards,
> > David
> >
> >
> > On Sat, Dec 5, 2020 at 8:04 AM Rex Fenley <[email protected]> wrote:
> >>
> >> We're running this in a local environment so that may be contributing
> to what we're seeing.
> >>
> >> On Fri, Dec 4, 2020 at 10:41 PM Rex Fenley <[email protected]> wrote:
> >>>
> >>> Hello,
> >>>
> >>> I'm tuning flink for parallelism right now and when I look at the
> JobManager I see
> >>> taskmanager.cpu.cores1.7976931348623157E308
> >>> Which looks like the maximum double number.
> >>>
> >>> We have 8 cpu cores, so we figured we'd bump to 16 for hyper
> threading. We have 37 operators so we rounded up and set 40 task slots.
> >>>
> >>> Here is our configuration
> >>>
> >>> "vmArgs": "-Xmx16g -Xms16g -XX:MaxDirectMemorySize=1207959552
> -XX:MaxMetaspaceSize=268435456 -Dlog.file=/tmp/flink.log
> -Dtaskmanager.memory.framework.off-heap.size=134217728b
> -Dtaskmanager.memory.network.max=1073741824b
> -Dtaskmanager.memory.network.min=1073741824b
> -Dtaskmanager.memory.framework.heap.size=134217728b
> -Dtaskmanager.memory.managed.size=6335076856b
> -Dtaskmanager.memory.task.heap.size=8160437768b
> -Dtaskmanager.memory.task.off-heap.size=0b
> -Dtaskmanager.numberOfTaskSlots=40 -Dtaskmanager.cpu.cores=16.0"
> >>>
> >>> We then tried with -Dtaskmanager.cpu.cores=7.0 and still ended up with
> that very odd value for cpu cores.
> >>>
> >>> How do we correctly adjust this?
> >>>
> >>> Thanks!
> >>> --
> >>>
> >>> Rex Fenley | Software Engineer - Mobile and Backend
> >>>
> >>>
> >>> Remind.com | BLOG | FOLLOW US | LIKE US
> >>
> >>
> >>
> >> --
> >>
> >> Rex Fenley | Software Engineer - Mobile and Backend
> >>
> >>
> >> Remind.com | BLOG | FOLLOW US | LIKE US
>
