Flink Sql中Kafka-Source怎么获取Key值
Flink Sql中Kafka-Source只看到了关于Value的Deserialize,要怎么获取对应的Key及meta(offset, topic, partition)信息呢?
Re: Flink Sql中Kafka-Source怎么获取Key值
Hi, 目前 Flink SQL 还不支持访问 key 及 meta,不过这功能已经在计划中了,有望在1.11 中支持。 我建了个 issue 来追踪这个需求:https://issues.apache.org/jira/browse/FLINK-15869 Best, Jark On Tue, 4 Feb 2020 at 00:10, wei0727wei wrote: > Flink Sql中Kafka-Source只看到了关于Value的Deserialize,要怎么获取对应的Key及meta(offset, > topic, partition)信息呢?
[DISCUSS] Upload the Flink Python API 1.9.x to PyPI for user convenience.
Hi folks, I am very happy to receive some user inquiries about the use of Flink Python API (PyFlink) recently. One of the more common questions is whether it is possible to install PyFlink without using source code build. The most convenient and natural way for users is to use `pip install apache-flink`. We originally planned to support the use of `pip install apache-flink` in Flink 1.10, but the reason for this decision was that when Flink 1.9 was released at August 22, 2019[1], Flink's PyPI account system was not ready. At present, our PyPI account is available at October 09, 2019 [2](Only PMC can access), So for the convenience of users I propose: - Publish the latest release version (1.9.2) of Flink 1.9 to PyPI. - Update Flink 1.9 documentation to add support for `pip install`. As we all know, Flink 1.9.2 was just completed released at January 31, 2020 [3]. There is still at least 1 to 2 months before the release of 1.9.3, so my proposal is completely considered from the perspective of user convenience. Although the proposed work is not large, we have not set a precedent for independent release of the Flink Python API(PyFlink) in the previous release process. I hereby initiate the current discussion and look forward to your feedback! Best, Jincheng [1] https://lists.apache.org/thread.html/4a4d23c449f26b66bc58c71cc1a5c6079c79b5049c6c6744224c5f46%40%3Cdev.flink.apache.org%3E [2] https://lists.apache.org/thread.html/8273a5e8834b788d8ae552a5e177b69e04e96c0446bb90979444deee%40%3Cprivate.flink.apache.org%3E [3] https://lists.apache.org/thread.html/ra27644a4e111476b6041e8969def4322f47d5e0aae8da3ef30cd2926%40%3Cdev.flink.apache.org%3E
Re: [DISCUSS] Upload the Flink Python API 1.9.x to PyPI for user convenience.
Hi Jincheng, Thanks for bring up this discussion! +1 for this proposal. Building from source takes long time and requires a good network environment. Some users may not have such an environment. Uploading to PyPI will greatly improve the user experience. Best, Wei jincheng sun 于2020年2月4日周二 上午11:49写道: > Hi folks, > > I am very happy to receive some user inquiries about the use of Flink > Python API (PyFlink) recently. One of the more common questions is whether > it is possible to install PyFlink without using source code build. The most > convenient and natural way for users is to use `pip install apache-flink`. > We originally planned to support the use of `pip install apache-flink` in > Flink 1.10, but the reason for this decision was that when Flink 1.9 was > released at August 22, 2019[1], Flink's PyPI account system was not ready. > At present, our PyPI account is available at October 09, 2019 [2](Only PMC > can access), So for the convenience of users I propose: > > - Publish the latest release version (1.9.2) of Flink 1.9 to PyPI. > - Update Flink 1.9 documentation to add support for `pip install`. > > As we all know, Flink 1.9.2 was just completed released at January 31, 2020 > [3]. There is still at least 1 to 2 months before the release of 1.9.3, so > my proposal is completely considered from the perspective of user > convenience. Although the proposed work is not large, we have not set a > precedent for independent release of the Flink Python API(PyFlink) in the > previous release process. I hereby initiate the current discussion and look > forward to your feedback! > > Best, > Jincheng > > [1] > > https://lists.apache.org/thread.html/4a4d23c449f26b66bc58c71cc1a5c6079c79b5049c6c6744224c5f46%40%3Cdev.flink.apache.org%3E > [2] > > https://lists.apache.org/thread.html/8273a5e8834b788d8ae552a5e177b69e04e96c0446bb90979444deee%40%3Cprivate.flink.apache.org%3E > [3] > > https://lists.apache.org/thread.html/ra27644a4e111476b6041e8969def4322f47d5e0aae8da3ef30cd2926%40%3Cdev.flink.apache.org%3E >
Re: [DISCUSS] Upload the Flink Python API 1.9.x to PyPI for user convenience.
Hi Jincheng, Thanks for driving this. +1 for this proposal. Compared to building from source, downloading directly from PyPI will greatly save the development cost of Python users. Best, Xingbo Wei Zhong 于2020年2月4日周二 下午12:43写道: > Hi Jincheng, > > Thanks for bring up this discussion! > > +1 for this proposal. Building from source takes long time and requires a > good network environment. Some users may not have such an environment. > Uploading to PyPI will greatly improve the user experience. > > Best, > Wei > > jincheng sun 于2020年2月4日周二 上午11:49写道: > >> Hi folks, >> >> I am very happy to receive some user inquiries about the use of Flink >> Python API (PyFlink) recently. One of the more common questions is whether >> it is possible to install PyFlink without using source code build. The >> most >> convenient and natural way for users is to use `pip install apache-flink`. >> We originally planned to support the use of `pip install apache-flink` in >> Flink 1.10, but the reason for this decision was that when Flink 1.9 was >> released at August 22, 2019[1], Flink's PyPI account system was not ready. >> At present, our PyPI account is available at October 09, 2019 [2](Only PMC >> can access), So for the convenience of users I propose: >> >> - Publish the latest release version (1.9.2) of Flink 1.9 to PyPI. >> - Update Flink 1.9 documentation to add support for `pip install`. >> >> As we all know, Flink 1.9.2 was just completed released at January 31, >> 2020 >> [3]. There is still at least 1 to 2 months before the release of 1.9.3, so >> my proposal is completely considered from the perspective of user >> convenience. Although the proposed work is not large, we have not set a >> precedent for independent release of the Flink Python API(PyFlink) in the >> previous release process. I hereby initiate the current discussion and >> look >> forward to your feedback! >> >> Best, >> Jincheng >> >> [1] >> >> https://lists.apache.org/thread.html/4a4d23c449f26b66bc58c71cc1a5c6079c79b5049c6c6744224c5f46%40%3Cdev.flink.apache.org%3E >> [2] >> >> https://lists.apache.org/thread.html/8273a5e8834b788d8ae552a5e177b69e04e96c0446bb90979444deee%40%3Cprivate.flink.apache.org%3E >> [3] >> >> https://lists.apache.org/thread.html/ra27644a4e111476b6041e8969def4322f47d5e0aae8da3ef30cd2926%40%3Cdev.flink.apache.org%3E >> >
Re: [DISCUSS] Upload the Flink Python API 1.9.x to PyPI for user convenience.
+1 Xingbo Huang 于2020年2月4日周二 下午1:07写道: > Hi Jincheng, > > Thanks for driving this. > +1 for this proposal. > > Compared to building from source, downloading directly from PyPI will > greatly save the development cost of Python users. > > Best, > Xingbo > > > > Wei Zhong 于2020年2月4日周二 下午12:43写道: > >> Hi Jincheng, >> >> Thanks for bring up this discussion! >> >> +1 for this proposal. Building from source takes long time and requires a >> good network environment. Some users may not have such an environment. >> Uploading to PyPI will greatly improve the user experience. >> >> Best, >> Wei >> >> jincheng sun 于2020年2月4日周二 上午11:49写道: >> >>> Hi folks, >>> >>> I am very happy to receive some user inquiries about the use of Flink >>> Python API (PyFlink) recently. One of the more common questions is >>> whether >>> it is possible to install PyFlink without using source code build. The >>> most >>> convenient and natural way for users is to use `pip install >>> apache-flink`. >>> We originally planned to support the use of `pip install apache-flink` in >>> Flink 1.10, but the reason for this decision was that when Flink 1.9 was >>> released at August 22, 2019[1], Flink's PyPI account system was not >>> ready. >>> At present, our PyPI account is available at October 09, 2019 [2](Only >>> PMC >>> can access), So for the convenience of users I propose: >>> >>> - Publish the latest release version (1.9.2) of Flink 1.9 to PyPI. >>> - Update Flink 1.9 documentation to add support for `pip install`. >>> >>> As we all know, Flink 1.9.2 was just completed released at January 31, >>> 2020 >>> [3]. There is still at least 1 to 2 months before the release of 1.9.3, >>> so >>> my proposal is completely considered from the perspective of user >>> convenience. Although the proposed work is not large, we have not set a >>> precedent for independent release of the Flink Python API(PyFlink) in the >>> previous release process. I hereby initiate the current discussion and >>> look >>> forward to your feedback! >>> >>> Best, >>> Jincheng >>> >>> [1] >>> >>> https://lists.apache.org/thread.html/4a4d23c449f26b66bc58c71cc1a5c6079c79b5049c6c6744224c5f46%40%3Cdev.flink.apache.org%3E >>> [2] >>> >>> https://lists.apache.org/thread.html/8273a5e8834b788d8ae552a5e177b69e04e96c0446bb90979444deee%40%3Cprivate.flink.apache.org%3E >>> [3] >>> >>> https://lists.apache.org/thread.html/ra27644a4e111476b6041e8969def4322f47d5e0aae8da3ef30cd2926%40%3Cdev.flink.apache.org%3E >>> >> -- Best Regards Jeff Zhang
Re: [DISCUSS] Upload the Flink Python API 1.9.x to PyPI for user convenience.
Hi Jincheng, +1 for this proposal. >From the perspective of users, I think it would nice to have PyFlink on PyPI which makes it much easier to install PyFlink. Best, Hequn On Tue, Feb 4, 2020 at 1:09 PM Jeff Zhang wrote: > +1 > > > Xingbo Huang 于2020年2月4日周二 下午1:07写道: > >> Hi Jincheng, >> >> Thanks for driving this. >> +1 for this proposal. >> >> Compared to building from source, downloading directly from PyPI will >> greatly save the development cost of Python users. >> >> Best, >> Xingbo >> >> >> >> Wei Zhong 于2020年2月4日周二 下午12:43写道: >> >>> Hi Jincheng, >>> >>> Thanks for bring up this discussion! >>> >>> +1 for this proposal. Building from source takes long time and requires >>> a good network environment. Some users may not have such an environment. >>> Uploading to PyPI will greatly improve the user experience. >>> >>> Best, >>> Wei >>> >>> jincheng sun 于2020年2月4日周二 上午11:49写道: >>> Hi folks, I am very happy to receive some user inquiries about the use of Flink Python API (PyFlink) recently. One of the more common questions is whether it is possible to install PyFlink without using source code build. The most convenient and natural way for users is to use `pip install apache-flink`. We originally planned to support the use of `pip install apache-flink` in Flink 1.10, but the reason for this decision was that when Flink 1.9 was released at August 22, 2019[1], Flink's PyPI account system was not ready. At present, our PyPI account is available at October 09, 2019 [2](Only PMC can access), So for the convenience of users I propose: - Publish the latest release version (1.9.2) of Flink 1.9 to PyPI. - Update Flink 1.9 documentation to add support for `pip install`. As we all know, Flink 1.9.2 was just completed released at January 31, 2020 [3]. There is still at least 1 to 2 months before the release of 1.9.3, so my proposal is completely considered from the perspective of user convenience. Although the proposed work is not large, we have not set a precedent for independent release of the Flink Python API(PyFlink) in the previous release process. I hereby initiate the current discussion and look forward to your feedback! Best, Jincheng [1] https://lists.apache.org/thread.html/4a4d23c449f26b66bc58c71cc1a5c6079c79b5049c6c6744224c5f46%40%3Cdev.flink.apache.org%3E [2] https://lists.apache.org/thread.html/8273a5e8834b788d8ae552a5e177b69e04e96c0446bb90979444deee%40%3Cprivate.flink.apache.org%3E [3] https://lists.apache.org/thread.html/ra27644a4e111476b6041e8969def4322f47d5e0aae8da3ef30cd2926%40%3Cdev.flink.apache.org%3E >>> > > -- > Best Regards > > Jeff Zhang >
Re:Flink Sql中Kafka-Source怎么获取Key值
source从新 订阅,作为新字段加入 字段列表里面 可行不? 我们改了source,对binlog 进行处理,源头很多字段 都加入进去。下游让SQL 处理 在 2020-02-04 00:09:18,"wei0727wei" 写道: >Flink Sql中Kafka-Source只看到了关于Value的Deserialize,要怎么获取对应的Key及meta(offset, topic, >partition)信息呢?