rinkako created BAHIR-276:
-----------------------------
Summary: flink redis sink bug fix and add pipeline support with at
least once semantics
Key: BAHIR-276
URL: https://issues.apache.org/jira/browse/BAHIR-276
Project: Bahir
Issue Type: Improvement
Components: Flink Streaming Connectors
Reporter: rinkako
I've found some bugs and improvements which I would like to work on it in
flink-connector-redis.
Bug:
* when redis container call open(), it will get a Jedis resource for `echo` to
validate redis connection, but this resource never close.
Improvement:
* redis container use many try..catch..finally(release jedis) block, which can
be replaced by `try with resource`.
* redis container use Jedis Pool to provide Jedis resource instance, every
element send into sink will get a Jedis instance, do redis command, and then
close Jedis instance. When upstream produces tons of data (such as 10w ops), it
will suffer extremely performance degradation. With Jedis pipeline, the
performance will be improved outstandingly (in our test cases, `hincrBy` can be
improved from 5w+ ops to 60w+ ops), and still guarantees flink `at least once`
semantics.
I've already finished the improvement above and used in our own production
environment. I would like to contribute it to the bahir project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
