Thank you for ideas. Do you suggest to use new backend with unit test or integration test?
Thanks, Alexey ________________________________ From: Arvid Heise <ar...@ververica.com> Sent: Monday, September 14, 2020 4:26:47 AM To: Dawid Wysakowicz <dwysakow...@apache.org> Cc: Alexey Trenikhun <yen...@msn.com>; Flink User Mail List <user@flink.apache.org> Subject: Re: Unit Test for KeyedProcessFunction with out-of-core state Hi Alexey, Definition of test levels are always a bit blurry when writing tests for a data processing framework, but I'm convinced that in your case, you should rather think in terms of integration tests than unit tests: * Unit test should really just be about business logic * If it's about specific implementation details of other components, it should rather go in an integration test. You can still structure your code that only half of the pipeline or even just one step is executed in an ITCase, but it's much harder to do all the mocking than simply executing a small Flink program with a local runner. ITCases are really fast and will not limit the portability of your program to newer Flink version (which mocking of components usually do). Another idea for your specific use case would be to implement a backend that delegates to HeapMemory but copies all values on retrieval. On Mon, Sep 7, 2020 at 5:49 PM Dawid Wysakowicz <dwysakow...@apache.org<mailto:dwysakow...@apache.org>> wrote: Hi Alexey, There is no mock for RocksDB. Moreover I am not sure what would be the use case for one. If you want to test specifically against RocksDB then you can use it in the test harness Gordon mentioned. On 04/09/2020 16:31, Alexey Trenikhun wrote: Hi Gordon, We already use [1]. Unfortunately it doesn’t allow to detect out-of-core specific bugs like this: POJO v = myMapState.get(myKey): v.setStatus(1); return; // missing myMapState.put(myKey, v); Thanks, Alexey ________________________________ From: Tzu-Li (Gordon) Tai <tzuli...@apache.org><mailto:tzuli...@apache.org> Sent: Friday, September 4, 2020 12:35:48 AM To: Alexey Trenikhun <yen...@msn.com><mailto:yen...@msn.com> Cc: Flink User Mail List <user@flink.apache.org><mailto:user@flink.apache.org> Subject: Re: Unit Test for KeyedProcessFunction with out-of-core state Hi Alexey, Is there a specific reason why you want to test against RocksDB? Otherwise, in Flink tests we use a `KeyedOneInputStreamOperatorTestHarness` [1] that allows you to wrap a user function and eliminate the need to worry about setting up heavy runtime context / dependencies such as the state backend. As a unit test, this should be sufficient for you to implement basic test scenarios for your function, such as expected output given inputs, state etc. Does this provide what you are looking for? Cheers, Gordon [1] https://github.com/apache/flink/blob/1d5f44710270d1c615537f0d05ab49e699d3a6e5/flink-streaming-java/src/test/java/org/apache/flink/streaming/util/KeyedOneInputStreamOperatorTestHarness.java On Fri, Sep 4, 2020 at 2:28 AM Alexey Trenikhun <yen...@msn.com<mailto:yen...@msn.com>> wrote: Hello, I want to unit test KeyedProcessFunction which uses with out-of-core state (like rocksdb). Does Flink has mock for rocksdb, which can be used in unit tests ? Thanks, Alexey -- Arvid Heise | Senior Java Developer [https://lh5.googleusercontent.com/ODbO0aq1IqKMfuoy_pw2YH8r6dqDRTq37rg3ytg11FCGJx12jJ1ff_SANPBxTHzSJTUQY9JLuoXq4NB7Om7j6Vq1lg6jIOKz8S5g2VKDGwicbj5fbY09PVb6mD5TdRuWEUvEMZTG]<https://www.ververica.com/> Follow us @VervericaData -- Join Flink Forward<https://flink-forward.org/> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng
