Re: RocksDBMapState example?
For KeyedState, apart from https://ci.apache.org/projects/flink/flink-docs-
release-1.4/dev/stream/state/state.html#keyed-state-and-operator-state ,
you can refer to docs/dev/migration.md :
public void initializeState(FunctionInitializationContext context)
throws Exception {
counter = context.getKeyedStateStore().getState(
new ValueStateDescriptor<>("counter", Integer.class, 0));
FYI
On Tue, Apr 10, 2018 at 7:24 AM, NEKRASSOV, ALEXEI wrote:
> Yes, I've read the documentation on working with state.
> It talks about MapState. When I looked at Javadoc, I learned that
> MapState is an interface, with RocksDBMapState as one of the implementing
> classes.
>
> I'm not sure what you mean by KeyedState; I don't see a class with that
> name.
>
> I'm not clear how ValueState can be used to store key-value mapping. Can
> you please clarify?
>
> Thanks,
> Alex
>
> -Original Message-
> From: Dawid Wysakowicz [mailto:wysakowicz.da...@gmail.com]
> Sent: Tuesday, April 10, 2018 8:54 AM
> To: NEKRASSOV, ALEXEI
> Cc: user@flink.apache.org
> Subject: Re: RocksDBMapState example?
>
> Hi Alexei,
>
> You should not use RocksDBMapState directly. Have you went through the doc
> page regarding working with state[1]?
> I think you want to use KeyedState, assuming the size of your keyspace.
> Probably a way to go would be to key your stream and then even ValueState
> (which will be scoped to that key) might be sufficient.
> You can configure flink further to use RocksDB as the underlying state
> backend[2]
>
> Regards,
> Dawid
>
> [1] https://ci.apache.org/projects/flink/flink-docs-
> release-1.4/dev/stream/state/state.html#working-with-state
> [2] https://ci.apache.org/projects/flink/flink-docs-
> master/ops/state/state_backends.html#state-backends
>
> > On 9 Apr 2018, at 17:41, NEKRASSOV, ALEXEI wrote:
> >
> > Hi,
> >
> > I’d like to use RocksDB to store a key-value mapping table (with 45
> million keys).
> > Can someone please point me to an example of RocksDBMapState()
> constructor invocation? Or an explanation of constructor arguments?..
> >
> > Thanks,
> > Alex Nekrassov
> > nekras...@att.com
>
>
RE: RocksDBMapState example?
Yes, I've read the documentation on working with state. It talks about MapState. When I looked at Javadoc, I learned that MapState is an interface, with RocksDBMapState as one of the implementing classes. I'm not sure what you mean by KeyedState; I don't see a class with that name. I'm not clear how ValueState can be used to store key-value mapping. Can you please clarify? Thanks, Alex -Original Message- From: Dawid Wysakowicz [mailto:wysakowicz.da...@gmail.com] Sent: Tuesday, April 10, 2018 8:54 AM To: NEKRASSOV, ALEXEI Cc: user@flink.apache.org Subject: Re: RocksDBMapState example? Hi Alexei, You should not use RocksDBMapState directly. Have you went through the doc page regarding working with state[1]? I think you want to use KeyedState, assuming the size of your keyspace. Probably a way to go would be to key your stream and then even ValueState (which will be scoped to that key) might be sufficient. You can configure flink further to use RocksDB as the underlying state backend[2] Regards, Dawid [1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/state.html#working-with-state [2] https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html#state-backends > On 9 Apr 2018, at 17:41, NEKRASSOV, ALEXEI wrote: > > Hi, > > I’d like to use RocksDB to store a key-value mapping table (with 45 million > keys). > Can someone please point me to an example of RocksDBMapState() constructor > invocation? Or an explanation of constructor arguments?.. > > Thanks, > Alex Nekrassov > nekras...@att.com
Re: RocksDBMapState example?
Hi Alexei, You should not use RocksDBMapState directly. Have you went through the doc page regarding working with state[1]? I think you want to use KeyedState, assuming the size of your keyspace. Probably a way to go would be to key your stream and then even ValueState (which will be scoped to that key) might be sufficient. You can configure flink further to use RocksDB as the underlying state backend[2] Regards, Dawid [1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/state.html#working-with-state [2] https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html#state-backends > On 9 Apr 2018, at 17:41, NEKRASSOV, ALEXEI wrote: > > Hi, > > I’d like to use RocksDB to store a key-value mapping table (with 45 million > keys). > Can someone please point me to an example of RocksDBMapState() constructor > invocation? Or an explanation of constructor arguments?.. > > Thanks, > Alex Nekrassov > nekras...@att.com signature.asc Description: Message signed with OpenPGP
RE: RocksDBMapState example?
I looked at that code, but I’m still not clear. new RocksDBMapState<>(columnFamily, namespaceSerializer, stateDesc, this); columnFamily is determined by 50-line function; is this necessary for a simple use case like mine? What should I use as state descriptor in that function?.. Last argument is set to “this”; does this mean I need to implement AbstractKeyedStateBackend, like RocksDBKeyedStateBackend does? Again, I’m looking for a simple equivalent to new HashMap(); or to JedisPool pool = new JedisPool(new JedisPoolConfig(), redisHost); Jedis jedis = pool.getResource(); Thanks, Alex From: Ted Yu [mailto:yuzhih...@gmail.com] Sent: Monday, April 09, 2018 11:48 AM To: user Subject: Re: RocksDBMapState example? Hi, Have you looked at the ctor call in : flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java around line 1261 ? Cheers
Re: RocksDBMapState example?
Hi, Have you looked at the ctor call in : flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java around line 1261 ? Cheers
