That is a very good observation!
In an ideal world, I would say we disallow #remove() because we cannot
efficiently implement it for RocksDB and we should keep the behaviour
consistent between the backends. Now that we already have the
functionality for the heap-based backends I think we cannot remove it
because some users might have come to rely on it.
The next best thing would be throwing an exception for RocksDB to at
least not silently ignore ineffective #remove() calls.
Best,
Aljoscha
On 23.07.20 20:40, Ken Krugler wrote:
Hi devs,
If you use the FsStateBackend (or MemoryStateBackend), and you have ListState,
then you can get an iterator and remove() an entry, and it all works as
expected.
If you use the RocksDBStateBackend, the remove() call doesn’t throw an
exception, but the ListState isn’t updated.
Seems like either you should get an exception w/the remove() call, or the
operation should work as expected.
I see https://issues.apache.org/jira/browse/FLINK-5651
<https://issues.apache.org/jira/browse/FLINK-5651>, though that seems only to
be talking about FsStateBackend/MemoryStateBackend.
And I don’t understand the comment on that issue: "Actually, it can be fine to
use Iterator#remove() as long as the user does not reply on these changes in the
backing store”.
Thanks,
— Ken
PS - I understand there are many reasons to not remove arbitrary elements from
a ListState when using RocksDB (serde cost for entire list), so I’d be in favor
of the remove() call throwing an exception, at least with RocksDB.
--------------------------
Ken Krugler
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr
