Re: Checkpoint Error

Hi All, Any suggestions? Thanks On Mon, Jan 18, 2021 at 7:38 PM Navneeth Krishnan wrote: > Hi All, > > We are running our streaming job on flink 1.7.2 and we are noticing the > below error. Not sure what's causing it, any pointers would help. We have > 10 TM's checkpointing to AWS EFS. > >

??flink ???? Kerberos??????hive ??

?? flink1.12.1 hive2.1.0 CDH6.2.0 Kerberoshive?? Kerberos hive metastore?? ??sql-client.sh embedded?? Flink SQL show tables; dimension_table dimension_table1 test Flink SQL select * from

Re: Dynamic JDBC Sink Support

Hey Rion, I had exactly the same problem and implemented this functionality in my Flink fork with XA sink taken from the development branch. As I see that it's not only my problem, I've created a Jira task for it - FLINK-21643 - and will provide draft PR for it. @David - for traditional

Re: reading file from s3

I had a typo in my previous answer, the env name was missing an 'S' ENABLE_BUILT_IN_PLUGIN --> ENABLE_BUILT_IN_PLUGINS once again, the value is the plugin jar name : flink-s3-fs-hadoop-.jar The complete list can be found

Re: reading file from s3

Hey Avi, Do you use 'Hadoop S3 plugin' to read from S3? If yes, what is its version? If not try to read from S3 as follow (ref) 1. set an environment variable to use hadoop plugin (it's

Re: Flink Job 如何集成到自己的系统，方便管理

有的，通过 FLINK 和 YARN 或 k8s 的接口进行编程，管理元数据，管理用户文件，支持提交作业及之后管理作业状态，这是许多公司应用 Flink 的实现方式。 你说的前端的东西是对接口返回值的友好展示，Flink 本身有一个 Web 前端，可以支持你要的大部分功能，但是多任务可能会有一些缺陷，尤其你不是使用 session 模式的情况下。 向下整合 Flink 能力以及实际部署的集群信息，向上提供人性化的交互页面，按照传统的 Web App 开发思路就可以的。 Best, tison. Jacob <17691150...@163.com> 于2021年3月6日周六

Re: Running Pyflink job on K8s Flink Cluster Deployment?

Hi Kevin, For your information, bellow is an example for running a PyFlink table API WordCount job. 1. Building a Docker image with Python and PyFlink Installed: Dockerfile: FROM flink:1.12.0 # install python3 and pip3 RUN apt-get update -y && \ apt-get install -y python3.7 python3-pip

Re: Running Pyflink job on K8s Flink Cluster Deployment?

Hi Kevin, You are able to run PyFlink applications on kuberetes cluster, both native k8s mode and resource definition mode are supported since release-1.12.0. Currently, Python and PyFlink are not enabled in official flink docker image, that you might need to build a custom image with Python and

flink 1.12.2??????DataStream????Table??????rowtime????????????????

?? ?? StreamExecutionEnvironment bsEnv = StreamExecutionEnvironment.getExecutionEnvironment(); StreamTableEnvironment bsTableEnv = StreamTableEnvironment.create(bsEnv); DataStream

java options to generate heap dump in EMR not working

Hi, I am trying to generate a heap dump to debug a GC overhead OOM. For that I added the below java options in flink-conf.yaml, however after adding this the yarn is not able to launch the containers. The job logs show it goes on requesting for containers from yarn and it gets them, again

Flink Job 如何集成到自己的系统，方便管理

我们现在提交Flink Job 是通过flink客户端run命令提交job，进行实时任务的计算，每次提交都要登录prd机器，上传jar包，过程比较麻烦。 后期规划把实时计算的任务集成到我们已有的一个系统中，把上面描述的过程封装起来，给用户提供一些按钮、菜单等，理想状态下，在这个系统增加一些模块、菜单之类的东西，就能完成对Job的维护，包括提交Job、查看正在运行的Job、停止Job等等 上面所说的这个系统是我们自研的一个数据处理平台，实时计算任务也是其中的一环，因此就想把实时计算的任务的模块也集成到其中去。 不知道这有没有可能实现