The new backend would be for unit tests (instead of a RocksDB mock). It's kind of the mock for out-of-core behavior that you initially requested.
To use rocksDB in an IT Case with multiple task managers, you would adjust the configuration in the usual minicluster setup, for example [1]. Note that you can do the same with the test harness [2], but I'd recommend the test harness only for testing new operators or complex ProcessFunctions (e.g., using timers) and not just for a map. Test harness is non-public API and we need to adjust it from time to time to reflect refactoring on the operators. [1] https://github.com/apache/flink/blob/4518d18a726b35de9ff802d155fd8100dc711a63/flink-table/flink-sql-client/src/test/java/org/apache/flink/table/client/gateway/local/LocalExecutorITCase.java#L132 [2] https://github.com/apache/flink/blob/5acbfedf754fa4d063931ea30432716374c2f8cf/flink-tests/src/test/java/org/apache/flink/test/state/StatefulOperatorChainedTaskTest.java#L143 On Tue, Sep 15, 2020 at 4:18 AM Alexey Trenikhun <yen...@msn.com> wrote: > Thank you for ideas. > Do you suggest to use new backend with unit test or integration test? > > Thanks, > Alexey > > ------------------------------ > *From:* Arvid Heise <ar...@ververica.com> > *Sent:* Monday, September 14, 2020 4:26:47 AM > *To:* Dawid Wysakowicz <dwysakow...@apache.org> > *Cc:* Alexey Trenikhun <yen...@msn.com>; Flink User Mail List < > user@flink.apache.org> > *Subject:* Re: Unit Test for KeyedProcessFunction with out-of-core state > > Hi Alexey, > > Definition of test levels are always a bit blurry when writing tests for a > data processing framework, but I'm convinced that in your case, you should > rather think in terms of integration tests than unit tests: > * Unit test should really just be about business logic > * If it's about specific implementation details of other components, it > should rather go in an integration test. > > You can still structure your code that only half of the pipeline or even > just one step is executed in an ITCase, but it's much harder to do all the > mocking than simply executing a small Flink program with a local runner. > ITCases are really fast and will not limit the portability of your program > to newer Flink version (which mocking of components usually do). > > Another idea for your specific use case would be to implement a backend > that delegates to HeapMemory but copies all values on retrieval. > > On Mon, Sep 7, 2020 at 5:49 PM Dawid Wysakowicz <dwysakow...@apache.org> > wrote: > > Hi Alexey, > > There is no mock for RocksDB. Moreover I am not sure what would be the use > case for one. If you want to test specifically against RocksDB then you can > use it in the test harness Gordon mentioned. > On 04/09/2020 16:31, Alexey Trenikhun wrote: > > Hi Gordon, > We already use [1]. Unfortunately it doesn’t allow to detect out-of-core > specific bugs like this: > POJO v = myMapState.get(myKey): > v.setStatus(1); > return; > // missing myMapState.put(myKey, v); > > Thanks, > Alexey > > ------------------------------ > *From:* Tzu-Li (Gordon) Tai <tzuli...@apache.org> <tzuli...@apache.org> > *Sent:* Friday, September 4, 2020 12:35:48 AM > *To:* Alexey Trenikhun <yen...@msn.com> <yen...@msn.com> > *Cc:* Flink User Mail List <user@flink.apache.org> <user@flink.apache.org> > *Subject:* Re: Unit Test for KeyedProcessFunction with out-of-core state > > Hi Alexey, > > Is there a specific reason why you want to test against RocksDB? > > Otherwise, in Flink tests we use a > `KeyedOneInputStreamOperatorTestHarness` [1] that allows you to wrap a user > function and eliminate the need to worry about setting up heavy runtime > context / dependencies such as the state backend. > As a unit test, this should be sufficient for you to implement basic test > scenarios for your function, such as expected output given inputs, state > etc. > Does this provide what you are looking for? > > Cheers, > Gordon > > [1] > https://github.com/apache/flink/blob/1d5f44710270d1c615537f0d05ab49e699d3a6e5/flink-streaming-java/src/test/java/org/apache/flink/streaming/util/KeyedOneInputStreamOperatorTestHarness.java > > On Fri, Sep 4, 2020 at 2:28 AM Alexey Trenikhun <yen...@msn.com> wrote: > > Hello, > I want to unit test KeyedProcessFunction which uses with out-of-core state > (like rocksdb). > Does Flink has mock for rocksdb, which can be used in unit tests ? > > Thanks, > Alexey > > > > -- > > Arvid Heise | Senior Java Developer > > <https://www.ververica.com/> > > Follow us @VervericaData > > -- > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > Conference > > Stream Processing | Event Driven | Real Time > > -- > > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany > > -- > Ververica GmbH > Registered at Amtsgericht Charlottenburg: HRB 158244 B > Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji > (Toni) Cheng > -- Arvid Heise | Senior Java Developer <https://www.ververica.com/> Follow us @VervericaData -- Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng
