Thanks for the suggestion, @Stephan.
DATAPROC makes good sense to me. +1 here
Regarding the Scope, it is meant for calculating fractions from the
weights. The idea is that the algorithm looks into the scopes and
calculates fractions without understanding the individual use cases.
I guess I shoul
Hi!
I read through the FLIP and looks good to me. One suggestion and one
question:
Regarding naming, we could call the ROCKSDB/BATCH_OP category DATAPROC
because this is the memory that goes into holding (and structuring) the
data.
I am a bit confused about the Scope enum (with values Slot and O
Thanks Till, `taskmanager.memory.managed.consumer-weights` sounds good to
me.
Thank you~
Xintong Song
On Thu, Sep 3, 2020 at 8:44 PM Till Rohrmann wrote:
> Thanks for updating the FLIP Xintong. It looks good to me. One minor
> comment is that we could name the configuration parameter
> also
Thanks for updating the FLIP Xintong. It looks good to me. One minor
comment is that we could name the configuration parameter
also taskmanager.memory.managed.consumer-weights which might be a bit more
expressive what this option does.
Cheers,
Till
On Thu, Sep 3, 2020 at 12:44 PM Xintong Song wr
Thanks all for the feedback.
FYI, I've opened a voting thread[1] on this.
Thank you~
Xintong Song
[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44358.html
On Thu, Sep 3, 2020 at 2:54 PM Zhu Zhu wrote:
> Thanks for propo
Thanks for proposing this improvement! @Xintong
The proposal looks good to me. Agreed that we should make it as simple as
possible for users to understand.
Thanks,
Zhu
Dian Fu 于2020年9月3日周四 下午2:11写道：
> Thanks for driving this FLIP, Xintong! +1 to the updated version.
>
> > 在 2020年9月2日，下午6:09，Xin
Thanks for driving this! The newest version LGTM. +1 for this FLIP.
Best,
Yangze Guo
On Thu, Sep 3, 2020 at 2:11 PM Dian Fu wrote:
>
> Thanks for driving this FLIP, Xintong! +1 to the updated version.
>
> > 在 2020年9月2日，下午6:09，Xintong Song 写道：
> >
> > Thanks for the input, Yu.
> >
> > I believe
Thanks for driving this FLIP, Xintong! +1 to the updated version.
> 在 2020年9月2日，下午6:09，Xintong Song 写道：
>
> Thanks for the input, Yu.
>
> I believe the current proposal should work with RocksDB, or any other state
> backend, using memory at either the slot or the scope. With the proposed
> appr
Thanks for the input, Yu.
I believe the current proposal should work with RocksDB, or any other state
backend, using memory at either the slot or the scope. With the proposed
approach, all we need is an indicator (e.g., a configuration option)
telling us which scope should we calculate the fractio
Thanks for compiling the FLIP Xintong, and +1 for the updated doc.
Just one supplement for the RocksDB state backend part:
It's true that currently we're using managed memory at the slot scope.
However, IMHO, we may support setting weights for different stateful
operators (for advanced usage) in
Thanks all for the feedback and discussion.
I have updated the FLIP, with the following changes.
- Choose the main proposal over the alternative approach
- Combine weights of RocksDB and batch operators
- Expose weights through configuration options, rather than via
ExecutionConfig.
Thanks for the inputs, Aljoscha & Till.
# Weight Configuration
I think exposing the knobs incrementally is a good idea. However, I'm not
sure about non-configurable as the first step.
Currently, users can tune memory for rocksdb
('taskmanager.memory.managed.size') and python
('python.fn-execu
Thanks for creating this FLIP Xintong.
I agree with the previous comments that the memory configuration should be
as easy as possible. Every new knob has the potential to confuse users
and/or allows him to shoot himself in the foot. Consequently, I am +1 for
the first proposal in the FLIP since it
Hi,
playing devils advocate here: should we even make the memory weights
configurable? We could go with weights that should make sense for most
cases in the first version and only introduce configurable weights when
(if) users need them.
Regarding where/how things are configured, I think tha
Thanks for the feedbacks, @Stephan
- There is a push to make as much as possible configurable via the main
> configuration, and not only in code. Specifically values for operations and
> tuning.
> I think it would be more important to have such memory weights in the
> config, compared to in
Thanks for driving this proposal. A few thoughts on the current design:
- There is a push to make as much as possible configurable via the main
configuration, and not only in code. Specifically values for operations and
tuning.
I think it would be more important to have such memory weights i
>
> A quick question, does network memory treated as managed memory now? Or in
> the future?
>
No, network memory is independent from managed memory ATM. And I'm not
aware of any plan to combine these two.
Any insights there?
Thank you~
Xintong Song
On Fri, Aug 28, 2020 at 4:35 PM Kurt Young
A quick question, does network memory treated as managed memory now? Or in
the future?
Best,
Kurt
On Wed, Aug 26, 2020 at 5:32 PM Xintong Song wrote:
> Hi devs,
>
> I'd like to bring the discussion over FLIP-141[1], which proposes how
> managed memory should be shared by various use cases with
Hi devs,
I'd like to bring the discussion over FLIP-141[1], which proposes how
managed memory should be shared by various use cases within a slot. This is
an extension to FLIP-53[2], where we assumed that RocksDB state backend and
batch operators are the only use cases of managed memory for stream
19 matches
Mail list logo
