Re: How to figure out what's the size of ListState?
Thank you Yun for your reply. Yes, I did the same thing and posted my solution here: https://stackoverflow.com/questions/76308686/how-to-figure-out-whats-the-size-of-liststate?noredirect=1#comment134577594_76308686 Best, Anir On Wed, May 24, 2023 at 5:13 AM Yun Tang wrote: > Hi Amir, > > For the current Flink, you have to iterator the returned Iterable of > ListState#get(). > > Why Flink lacks an API to get the size of listState directly? This is > because Flink leverages RocksDB's merge operator [1] for high-performance > listState#add [2], that is to say, we would not even read the original data > when appending the listState. I think you can record the size in another > valueState when implementing your logic. > > > [1] https://github.com/facebook/rocksdb/wiki/Merge-Operator > [2] > https://github.com/apache/flink/blob/d8c64a808484cab78c8bd7b74a287edf7d1f3b01/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBListState.java#L131 > > Best > Yun Tang > > From: Amir Hossein Sharifzadeh > Sent: Tuesday, May 23, 2023 23:51 > To: dev > Subject: How to figure out what's the size of ListState? > > Dear Flink Dev team, > > It's about a while since I am dealing with an issue that can't figure out > myself. I spent quite a lot of time trying to solve the problem myself, but > I feel stuck. > > I explained the problem statement and the issue here: > > https://stackoverflow.com/questions/76308686/how-to-figure-out-whats-the-size-of-liststate > > I really appreciate any suggestion. > > Best, > Amir >
[ANNOUNCE] Apache flink-connector-cassandra 3.1.0 released
The Apache Flink community is very happy to announce the release of Apache flink-connector-cassandra 3.1.0. This connector supports Flink 1.16 and 1.17. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available for download at: https://flink.apache.org/downloads.html The full release notes are available in Jira: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353030 We would like to thank all contributors of the Apache Flink community who made this release possible! Regards, Danny
[jira] [Created] (FLINK-32210) Error import Pyflink.table.descriptors due to python3.10 version mismatch
Alireza Omidvar created FLINK-32210: --- Summary: Error import Pyflink.table.descriptors due to python3.10 version mismatch Key: FLINK-32210 URL: https://issues.apache.org/jira/browse/FLINK-32210 Project: Flink Issue Type: Bug Components: API / Python, Connectors / Kafka, Connectors / MongoDB Affects Versions: 1.13.0 Reporter: Alireza Omidvar Following to the issue[jira] [Created] https://issues.apache.org/jira/browse/FLINK-32207(FLINK-32206) I decided to install latest 1.13 version where Kafka and Json imports are working which needed to create env python 3.8. I faced a few issues 1: CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'. To initialize your shell, run $ conda init Currently supported shells are: - bash - fish - tcsh - xonsh - zsh - powershell See 'conda init --help' for more information and options. IMPORTANT: You may need to close and restart your shell after running 'conda init'. 2. I tried to initiate but the new error faced No Pyflink module found -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction
Hi Jing, Thanks for your response. As stated in the FLIP, the purpose of this FLIP is meant to support user-defined async table function. As described in flink document [1] Async table functions are special functions for table sources that perform > a lookup. > So end user can not directly define and use async table function now. An user case is reported in [2] So, in conclusion, no new interface is introduced, but we extend the ability to support user-defined async table function. [1]: https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/ [2]: https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b Thanks. Aitozi. Jing Ge 于2023年5月27日周六 06:40写道： > Hi Aitozi, > > Thanks for your proposal. I am not quite sure if I understood your thoughts > correctly. You described a special case implementation of the > AsyncTableFunction with on public API changes. Would you please elaborate > your purpose of writing a FLIP according to the FLIP documentation[1]? > Thanks! > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > > Best regards, > Jing > > On Wed, May 24, 2023 at 1:07 PM Aitozi wrote: > > > May I ask for some feedback :D > > > > Thanks, > > Aitozi > > > > Aitozi 于2023年5月23日周二 19:14写道： > > > > > > Just catch an user case report from Giannis Polyzos for this usage: > > > > > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b > > > > > > Aitozi 于2023年5月23日周二 17:45写道： > > > > > > > > Hi guys, > > > > I want to bring up a discussion about adding support of User > > > > Defined AsyncTableFunction in Flink. > > > > Currently, async table function are special functions for table > source > > > > to perform > > > > async lookup. However, it's worth to support the user defined async > > > > table function. > > > > Because, in this way, the end SQL user can leverage it to perform the > > > > async operation > > > > which is useful to maximum the system throughput especially for IO > > > > bottleneck case. > > > > > > > > You can find some more detail in [1]. > > > > > > > > Looking forward to feedback > > > > > > > > > > > > [1]: > > > https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction > > > > > > > > Thanks, > > > > Aitozi. > > >
