????
flink-cdc????????
HI: flink-cdc,,??eventTime,??watermark ??mysql??
?????? flink ????
Hi: ??Leonard,??flink cdc??(??), join ??(debezium - kafka), eventTime , ??eventTime lookupeventTime 1, changlog, debezium - kafka , ??flink cdc, flink cdc??,flink cdc??watermark? 2, ??watermark??? ?? ---- ??: "user-zh" http://a.id/; = B.id <http://b.id/; ??(proctime??) , Leonard ?? 2021??5??2716:35??liujian <13597820...@qq.com ?? flink lookup??,lookup,?
flink ????
flink lookup??,lookup,?
jdbc connectors
Hi: jdbc connectors sink??mysql, ?? replace into??,,mysqlid,AUTO_INCREMENT ??2
??????flink??????????
??,,??,flinkjoin??,hbase,?? ---- ??: "user-zh"
flink??????????
Hi: ??,??3??,??,,??,kudu?? : jdbc-connecter,join,??,
?????? flink-shaded-hadoop-2-uber????????????
Thanks,flink-confhistory server,??hdfs??,??web ui??, ---- ??: "user-zh" https://ci.apache.org/projects/flink/flink-docs-master/deployment/advanced/historyserver.html Best, Yang liujian <13597820...@qq.comgt; ??2020??12??21?? 1:35?? gt; ??history-server,,,??, gt; gt; gt; gt; gt; --amp;nbsp;amp;nbsp;-- gt; ??: gt;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp; "user-zh" gt;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp; < gt; danrtsey...@gmail.comamp;gt;; gt; :amp;nbsp;2020??12??21??(??) 10:15 gt; ??:amp;nbsp;"user-zh"https://github.com/apache/flink-docker/blob/dev-master/docker-entrypoint.sh gt <https://github.com/apache/flink-docker/blob/dev-master/docker-entrypoint.shgt; ; gt; Best, gt; Yang gt; gt; gt; liujian <13597820...@qq.comamp;gt; ??2020??12??20?? 12:45?? gt; gt; amp;gt; Thanks,amp;amp;nbsp; gt; amp;gt; amp;amp;nbsp; amp;amp;nbsp; amp;amp;nbsp;??historyServer,flink gt; amp;gt; 1.12.0??Dockerfile ??CMD ["history-server"]amp;amp;nbsp; gt; ??8082,?? gt; amp;gt; gt; amp;gt; gt; amp;gt; gt; amp;gt; gt; amp;gt; --amp;amp;nbsp;amp;amp;nbsp;-- gt; amp;gt; ??: gt; amp;gt;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp; gt; "user-zh" gt; amp;gt;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp;amp;nbsp; gt; < gt; amp;gt; danrtsey...@gmail.comamp;amp;gt;; gt; amp;gt; :amp;amp;nbsp;2020??12??19??(??) 9:35 gt; amp;gt; ??:amp;amp;nbsp;"user-zh"https://github.com/apache/flink-shaded gt <https://github.com/apache/flink-shadedgt;; amp;gt < https://github.com/apache/flink-shadedamp;gtgt;; amp;amp;gt; amp;amp;amp;gt; gt; [2]. gt; amp;gt; amp;amp;gt; amp;amp;amp;gt; gt; amp;gt; amp;amp;gt; amp;amp;amp;gt; gt; amp;gt; amp;amp;gt; gt; amp;gt; gt; https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/yarn.html#preparation gt <https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/yarn.html#preparationgt;; amp;gt gt; < https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/yarn.html#preparationamp;gtgt ;; gt; amp;amp;gt gt; amp;gt; < gt; https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/yarn.html#preparationamp;amp;gtamp;gt gt <https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/yarn.html#preparationamp;amp;gtamp;gtgt;; ;; gt; amp;gt; amp;amp;amp;gt gt; amp;gt; amp;amp;gt; < gt; amp;gt; gt; https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/yarn.html#preparationamp;amp;amp;gtamp;amp;gt gt <https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/yarn.html#preparationamp;amp;amp;gtamp;amp;gtgt;; amp;gt gt; < https://ci.apache.org/projects/flink/flink-docs-mas
?????? flink-shaded-hadoop-2-uber????????????
Thanks, docker??,??Native K8s??,?? Dockerfile?? COPY ./jar/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar /opt/flink/lib/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar ENTRYPOINT ["/docker-entrypoint.sh"] EXPOSE 6123 8081 8082 CMD ["help","history-server"] COPY ./jar/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar /opt/flink/lib/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar ENTRYPOINT ["/docker-entrypoint.sh","history-server"] EXPOSE 6123 8081 8082 CMD ["help"] ??...,?? ---- ??: "user-zh" https://ci.apache.org/projects/flink/flink-docs-master/deployment/advanced/historyserver.html Best, Yang liujian <13597820...@qq.com ??2020??12??21?? 1:35?? ??history-server,,,??, --nbsp;nbsp;-- ??: "user-zh" < danrtsey...@gmail.comgt;; :nbsp;2020??12??21??(??) 10:15 ??:nbsp;"user-zh"https://github.com/apache/flink-docker/blob/dev-master/docker-entrypoint.sh Best, Yang liujian <13597820...@qq.comgt; ??2020??12??20?? 12:45?? gt; Thanks,amp;nbsp; gt; amp;nbsp; amp;nbsp; amp;nbsp;??historyServer,flink gt; 1.12.0??Dockerfile ??CMD ["history-server"]amp;nbsp; ??8082,?? gt; gt; gt; gt; gt; --amp;nbsp;amp;nbsp;-- gt; ??: gt;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp; "user-zh" gt;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp; < gt; danrtsey...@gmail.comamp;gt;; gt; :amp;nbsp;2020??12??19??(??) 9:35 gt; ??:amp;nbsp;"user-zh"https://github.com/apache/flink-shaded gt <https://github.com/apache/flink-shadedgt;; amp;gt; amp;amp;gt; [2]. gt; amp;gt; amp;amp;gt; gt; amp;gt; amp;amp;gt; gt; amp;gt; gt; https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/yarn.html#preparation gt <https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/yarn.html#preparationgt;; amp;gt gt; < https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/yarn.html#preparationamp;gtgt ;; gt; amp;amp;gt gt; amp;gt; < gt; https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/yarn.html#preparationamp;amp;gtamp;gt gt <https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/yarn.html#preparationamp;amp;gtamp;gtgt;; ; gt; amp;gt; ; gt; amp;gt; amp;amp;gt; Best, gt; amp;gt; amp;amp;gt; Yang gt; amp;gt; amp;amp;gt; gt; amp;gt; amp;amp;gt;
?????? flink-shaded-hadoop-2-uber????????????
??history-server,,,??, ---- ??: "user-zh" https://github.com/apache/flink-docker/blob/dev-master/docker-entrypoint.sh Best, Yang liujian <13597820...@qq.com ??2020??12??20?? 12:45?? Thanks,nbsp; nbsp; nbsp; nbsp;??historyServer,flink 1.12.0??Dockerfile ??CMD ["history-server"]nbsp; ??8082,?? --nbsp;nbsp;-- ??: "user-zh" < danrtsey...@gmail.comgt;; :nbsp;2020??12??19??(??) 9:35 ??:nbsp;"user-zh"https://github.com/apache/flink-shaded gt; amp;gt; [2]. gt; amp;gt; gt; amp;gt; gt; https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/yarn.html#preparation gt <https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/yarn.html#preparationgt;; amp;gt gt; < https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/yarn.html#preparationamp;gtgt ; gt; ; gt; amp;gt; Best, gt; amp;gt; Yang gt; amp;gt; gt; amp;gt;
?????? flink-shaded-hadoop-2-uber????????????
Thanks, ??historyServer,flink 1.12.0??Dockerfile ??CMD ["history-server"] ??8082,?? ---- ??: "user-zh" https://github.com/apache/flink-shaded gt; [2]. gt; gt; https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/yarn.html#preparation gt
?????? flink1.12 docker ????????????
git clone??,?? ---- ??: "user-zh" https://github.com/apache/flink-docker.git - cd scala_2.11-java8-debian - sudo docker build -t flink:1.12.0 . - docker push Best, Yang superainbower https://apache.website-solution.net/flink/flink-1.12.0/flink-1.12.0-bin-scala_2.11.tgz -n flink-1.12.0 git clone https://github.com/apache/flink-docker.git 1.12??2.11dock build -t flink??1.12.0 .?? ??2020??12??16?? 10:56??Yang Wang ?? docker hub?? ??PR https://github.com/docker-library/official-images/pull/9249 build?? git clone https://github.com/apache/flink-docker.git git checkout dev-master./add-custom.sh -u https://apache.website-solution.net/flink/flink-1.12.0/flink-1.12.0-bin-scala_2.11.tgz -n < https://apache.website-solution.net/flink/flink-1.12.0/flink-1.12.0-bin-scala_2.11.tgz-n flink-1.12.0cd dev/flink-1.12.0-debiandocker build . -t flink:flink-1.12.0docker push flink:flink-1.12.0 jiangjiguang719
?????? flink-shaded-hadoop-2-uber????????????
HDFS??Ha,hdfs-site.xml,,configMap??hdfs-site.xml??$FLINK_HOME/conf?? ---- ??: "user-zh" https://github.com/apache/flink-shaded [2]. https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/yarn.html#preparation Best, Yang
Native Kubernetes ????????HDFS
Hi: ??Native Kubernetes HDFS,??flink-shaded-hadoop-2-uber-2.8.3-10.0.jarlib hdfs??HA,??hdfs-site.xml,
??????flink 1.11 ????sql??????????????hive
??Hive??: ?? checkpoint ?? wartermark ?? ??,??..,, ---- ??: "user-zh"