Re: 关于kafka connector通过python链接

您好， 图片看不到，可以把图片上传到第三方网站，然后贴个链接，或者贴一下文本的报错信息。 > 在 2020年4月9日，下午4:41，秦寒 写道： > > 您好 >根据你们的说明我做了如下配置，我用的是flink 1.10版本 > 1在pyflink/lib下面添加了kafka-clients-2.2.0.jar > > > 2 在build-target/lib下面添加了flink-sql-connector-kafka_2.11-1.10.0.jar > flink-connector-kafka_2.11-1.10.0.jar 以及

回复： 关于Flink1.10.0 flink-hbase guava依赖冲突问题

Hi， 具体的哪个版本不太确定，我只差了11，12，14，16，18，23这些….12以上都没有的 Best, Xinghalo

Re: Temporal Table Function in SQL

Hi, - Temporal Table It is implemented as dimension table, you can define a dimension table using DDL, for example, JDBC [1]. - Temporal Table Function Currently it cannot created from pure SQL, you can create it through table api[2], and then use it in sql. [1]

flink反压问题求助

在跑Flink任务时，遇到了operator反压问题，任务执行图如下，source(读Kafka), KeyBy(抽取数据字段供keyBy操作使用)，Parser(业务处理逻辑)，Sink(写Kafka)，除了KeyBy->Parser使用hash(KeyBy操作)链接，其他都使用RESCALE链接。(并发度仅供参考，这个是解决问题后的并发度，最初的并发度为 500->1000->3000->500) [image: image.png] 相关metric [image: image.png] [image: image.png] 为了解决反压问题做的处理： 1.

Re: 关于flink state的问题

你好： 1 state 如果不需要了，可以自己删除（如果是 Window 中使用的，可以进行配置在 Window 结束时删除） 2 可以使用 TTL State[1] 3 如果仅考虑 OOM 风险，可以考虑使用 RocksDBStateBackend [1] https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/stream/state/state.html#state-time-to-live-ttl Best, Congxian guanyq 于2020年4月11日周六 上午7:21写道： >

Re: Storing Operator state in RocksDb during runtime - plans

Hi KristoffSC I'm not aware of any concrete plans for such a feature. I also CCed Yu, he may give more information about this. Best, Congxian Fabian Hueske 于2020年4月7日周二 上午4:27写道： > Hi Kristoff, > > I'm not aware of any concrete plans for such a feature. > > Best, > Fabian > > Am So., 5. Apr.

Re: Making job fail on Checkpoint Expired?

Hi Robin Glad to hear that my reply can help. >From my side, I do not think concurrent checkpoints can help, because it may cause more disk pressure problems. Currently, this is an issue[1] wants to support Unalign checkpoint, unaligned checkpoint wants to fix the problem of checkpoint under

Re: FlinkRuntimeException: Unexpected list element deserialization failure

Hi As Yun said, could you please share the whole stack trace(normally, you can copy more lines below the given stack trace), can not tell why this happened from the given stack trace. Best, Congxian Yun Tang 于2020年4月10日周五 上午1:48写道： > Hi > > I think you have missed the "caused by" exception

Re: Setting different idleStateRetentionTime for different queries executed in the same TableEnvironment in Flink 1.10

Just looked into the source code a bit further and realized that for StreamTableEnvironmentImpl, even for sinks it's also doing translation lazily. Any way we can have different transformation to have different queryConfig? From: Jiahui Jiang Sent: Friday, April

Re: 重复声明watermark的问题

谢谢，主要是我再次声明watermark后，再转成table,然后再window操作，就一直报错，现在能确定是时间属性字段的问题。我用的阿里云的blink,他们开发人员说好像不能这么操作。 jun su 于 2020年4月10日周五 23:36写道： > hi, > 1. 以我的理解, 再次下发watermark会覆盖之前的, 所以在经过n个operator后，可以用再次声明watermark的方式来调整. > 2. 如果是DataStream模式下, 两个流join后, 下游收到的watermark是较小的流的watermark, > 如果是stream模式下的sql window

Setting different idleStateRetentionTime for different queries executed in the same TableEnvironment in Flink 1.10

Hello! I'm using Table API to write a pipeline with multiple queries. And I want to set up different idleStateRetentionTime for different queries. In Flink 1.8, it seems to be the case where I can pass in a streamQueryConfig when converting each output table into datastreams. And the translate

关于flink state的问题

您好： 1.随着程序的运行，task内存中的状态会不断增加，迟早会出现内存溢出问题，想知道一般都如何解决这个问题？

Re: [PROPOSAL] Contribute training materials to Apache Flink

Yes, the training materials could become part of the docs, as you suggest. The reason why I'm somewhat resistant to this idea is that the documentation is versioned. It will be easier to maintain the training content and its exercises and examples if we only have to be concerned with having one

Re: upgrade flink from 1.9.1 to 1.10.0 on EMR

Hi Robert, attached the full application log file. Thanks, Anuj Container: container_1585301409143_0044_01_01 on ip-172-25-2-209.ap-south-1.compute.internal_8041

Flink job consuming all available memory on host

We are having an issue with a Flink Job that gradually consumes all available memory on a Docker host machine, crashing the machine. * We are running Flink 1.10.0 * We are running Flink in a Docker container on AWS ECS with EC2 instances * The Flink task manager UI does not show high memory usage

Temporal Table Function in SQL

Hi Is it possible to create a Temporal Table and a Temporal Table Function from the SQL API? Any example of DDL? Thanks

Re: [PROPOSAL] Contribute training materials to Apache Flink

+1 to the idea, David regarding your question, could it go in a quickstart/get started with Flink section? Next to or replacing the current 'Getting started' section? On Thu, 9 Apr 2020, 22:11 Seth Wiesman, wrote: > Hi David, > > +1 to add to the project. > > I agree that flink.apache.org and

Re: 重复声明watermark的问题

hi, 1. 以我的理解, 再次下发watermark会覆盖之前的, 所以在经过n个operator后，可以用再次声明watermark的方式来调整. 2. 如果是DataStream模式下, 两个流join后, 下游收到的watermark是较小的流的watermark, 如果是stream模式下的sql window join, 那么下游收到的 watermark是较小一侧表的watermark - sql设定的时间窗口. lec ssmi 于2020年4月8日周三 下午2:05写道： > 大家好： >

Re: how to hold a stream until another stream is drained?

+Sun Haibo who added that validation in https://issues.apache.org/jira/browse/FLINK-11879 Hi Haibo, Any suggestion how to enable checkpointing for InputSelectable and BounedMultiInput? Thanks, Yubao Liu On 2020/4/10, 10:21 PM, "刘宇宝" wrote: Hi Fabian, Thank you very much, I

Re: 如何合并 binlog stream 和 table stream?

抄了下 https://github.com/apache/flink/blob/release-1.10.0/flink-streaming-java/src/test/java/org/apache/flink/streaming/util/TestSequentialReadingStreamOperator.java ， 可以达到串行的效果了： DataStream binlogStream = env.addSource(new FlinkKafkaConsumer(…)); DataStream snapshotStream =

Re: how to hold a stream until another stream is drained?

Hi Fabian, Thank you very much, I almost get it working with InputSelectable: DataStream binlogStream = env.addSource(new FlinkKafkaConsumer(…)); DataStream snapshotStream = env.createInput(JDBCInputFormat.buildJDBCInputFormat()….); DataStream tableStream =

Re: Is it possible to emulate keyed state with operator state?

Hypothetically, yes, I think this is possible to some extent. You would have to give up all the things that require a KeyedStream, such as timers, and the RocksDB state backend. And performance would suffer. As for the question of determining which key groups (and ultimately, which keys) are

Re: Inserting nullable data into NOT NULL columns

Thanks Timo for the answer! We are specifically working on implementing Kudu Table connectors and catalog and we hit a problem while handling keys. Naturally columns that will be used as keys in Kudu cannot be null so we set their type when the catalog returns it to NOT NULL. Unfortunately this

on YARN question

Hello, everyone! I have a question for you [root@hadoop131 bin]#./yarn-session.sh -n 2 -s 2 -jm 1024 -tm 1024 -nm test -d The following command reports an error ./flink run -m yarn-cluster -c com.scala.DateSetWcApp /opt/module/datas/flink/Flink_test-1.7.2.jar --input

Re: Re: fink sql client not able to read parquet format table

https://issues.apache.org/jira/browse/FLINK-17086 It is my first time to create a flink jira issue. Just point it out and correct it if I write something wrong. Thanks, Lei wangl...@geekplus.com.cn From: Jingsong Li Date: 2020-04-10 11:03 To: wangl...@geekplus.com.cn CC: Jark Wu; lirui;

Re: Possible memory leak in JobManager (Flink 1.10.0)?

What you could also try out is whether the same problem occurs with Flink 1.7.3. We did the executor change in this bug fix release. This could help us validating my suspicion. Cheers, Till On Thu, Apr 9, 2020 at 4:24 PM Till Rohrmann wrote: > For further reference, I've created this issue [1]

Re: Javadocs Broken?

s/out/our/

Javadocs Broken?

Hi guys, Right now when I click "JavaDocs" in out docsite[1] it jumps to a page[2] I think is definitely not out api documentation. Any thoughts? Best, tison. [1] https://ci.apache.org/projects/flink/flink-docs-master/ [2] https://ci.apache.org/projects/flink/flink-docs-master/api/java/

Re: Re: Multiple SQL Optimization

Hi forideal, You are using `StreamTableEnvironment` which doesn't support multi-sink optimization in 1.10 :) You should change `StreamTableEnvironment.create` to `TableEnvironment.create`. Btw, StreamTableEnvironment will also support multi-sink optimization in 1.11. Best, Jark On Fri, 10

Re:Re: Multiple SQL Optimization

Hi Jark Thanks for your reploy. In my code, i use `TableEnvironemnt` with Blink planner. this.tableEnv = StreamTableEnvironment.create( env, EnvironmentSettings.newInstance() .useBlinkPlanner() .inStreamingMode() .build()); // update three sql sqlsWithoutFunc.forEach(sql -> {

dayofweek异常

hi， 在flink中使用hql函数的时候 dayofweek 报错，编译都没有通过。我的使用方式 用-MM-dd也是一样的错误。哪位大神遇到过！ DAYOFWEEK(to_date(from_unixtime (unix_timestamp(cast(dt as string),'MMdd'),'-MM-dd HH:mm:ss'))) ，Cannot apply 'EXTRACT' to arguments of type 'EXTRACT( FROM )'. Supported form(s): 'EXTRACT( FROM )'

Re: Multiple SQL Optimization

Hi forideal, Are you using `StreamTableEnvironment` or SQL CLI? Currently, only `TableEnvironemnt` with Blink planner have the multi-sink optimization (reuse shared upstream operators). Best, Jark On Fri, 10 Apr 2020 at 16:31, forideal wrote: > Hello > >There are 3 SQLs all querying the

Re: Multiple SQL Optimization

Hi forideal, Currently, Blink planner with TableEnvironment supports multiple sinks optimization which will try best to reuse common sub-graph. Best, Godfrey forideal 于2020年4月10日周五 下午4:31写道： > Hello > >There are 3 SQLs all querying the same table, but the generated GAG is > 3 independent

Multiple SQL Optimization

Hello There are 3 SQLs all querying the same table, but the generated GAG is 3 independent topologies.I think, the better result is that there is one Source and 3 Sinks. createtablegood_sink(datavarchar)with( 'connector.type'='console',

Re: on YARN question

Hi, I think you need to remove "-n 2" from your start up command. The "-n" option has been removed and no longer take effect. Flink will stop parsing the command line options once it see an un-recognized token. That means all the options after "-n 2" are ignored, including "-d". As a result,

Re: Flink SQL文件连接器中proctime()创建表失败的问题

这应该是个已知问题，https://issues.apache.org/jira/browse/FLINK-16160 请先使用 DDL 吧。 Best， Jark On Fri, 10 Apr 2020 at 16:21, Night_xing <1029681...@qq.com> wrote: > Flink版本：1.10.0 > > 使用了BlinkPlanner之后，使用java代码创建本地的CSV表时，不支持proctime的配置 > > 创建代码如下： > tableEnv.connect(new FileSystem() >

Flink SQL????????????proctime()????????????????

Flink??1.10.0 ??BlinkPlanner??java??CSVproctime?? ?? tableEnv.connect(new FileSystem() .path("file:///Users/test/csv/demo.csv") ) .withFormat(new Csv()) .withSchema(

Re: 关于Flink1.10.0 flink-hbase guava依赖冲突问题

Hi, 这是一个已知问题，而且确实挺影响开发效率的。 > 其次HBase需要guava12.0版本（更高的版本就移除这个方法了） 你知道 HBase 从哪个版本开始不使用这个方法/版本了吗？ 和 HBase 1.4.x server 兼容吗？ cc @ZhengHu who may have more knowledge on this. Best, Jark On Fri, 10 Apr 2020 at 08:21, 111 wrote: > Hi, > > > 原来IDEA的问题，那这样就不能debug调试了。 >

Flink SQL????????????proctime()????????????????

Flink??1.10.0 ??BlinkPlanner??java??CSVproctime?? ?? tableEnv.connect(new FileSystem() .path("file:///Users/test/csv/demo.csv") ) .withFormat(new Csv()) .withSchema( new Schema()

Re: job doesn't start via cli after migrating Flink from 1.8 to 1.10

I am trying to answer your question inline. > The server has twice more than that, and on flink 1.8 this configuration > works, why when switching to 1.10 it is not enough resources? >From 1.10, the taskmanager resource related configuration has changed and the default value is bigger than