Hi Juha,
> I can also submit the more complex test with the bigger operator and and a
> window operator. There's just gonna be more code to read. Can I attach a
> file here or how should I submit a larger chuck of code?
You can just attach the file with the code.
> 2. I'm not sure what would / should I look for.
> For 'taskmanager.memory.managed.fraction' I tried
> configuration.setDouble("taskmanager.memory.managed.fraction", 0.8);
I think Yu meant increasing the managed memory because it might be not enough
to host both X and window operator.
You can do it by increasing this option: taskmanager.memory.managed.size [1],
[2]
also if you run Flink locally from your IDE, see notes for local execution [3].
When you enable ‘state.backend.rocksdb.memory.managed’, RocksDB does not use
more memory than the configured or default size of managed memory. Therefore,
it starts to spill to disk and performance degrades but the memory usage is
deterministic and you do not risk that your container gets killed with
out-of-memory error.
If you disable ‘state.backend.rocksdb.memory.managed’, RocksDB does some
internal decisions about how much memory to allocate, so it can allocate more
to be more performant and do less frequent spilling to disk. So maybe it gives
more memory to window operator to spill less.
Therefore, it would be nice to compare memory consumption of Flink process with
‘state.backend.rocksdb.memory.managed’ to be true and false.
Anyways I do not know how we could control splitting of the configured managed
memory among operators in a more optimal way.
Best,
Andrey
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_setup.html#managed-memory
<https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_setup.html#managed-memory>
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html#taskmanager-memory-managed-size
<https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html#taskmanager-memory-managed-size>
[3]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_detail.html#local-execution
<https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_detail.html#local-execution>
> On 26 Jun 2020, at 08:45, Juha Mynttinen <juha.myntti...@king.com> wrote:
>
> Andrey,
>
> A small clarification. The tweaked WordCount I posted earlier doesn't
> illustrate the issue I originally explained, i.e. the one where there's a
> bigger operator and a smallest possible windows operator. Instead, the
> modified WordCount illustrates the degraded performance of a very simple
> Flink application when using managed memory and increasing parallelism over
> a certain treshold. The performance is not degraded if not using managed
> memory (and parallelism is increased).
>
> I was hoping this kind of simple program would be easier to debug / profile.
>
> I can also submit the more complex test with the bigger operator and and a
> window operator. There's just gonna be more code to read. Can I attach a
> file here or how should I submit a larger chuck of code?
>
> Regards,
> Juha
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
