Re: datadog metrics

Hi Fanbin, I have commented on the ticket: https://issues.apache.org/jira/browse/FLINK-16611. - Steve On Mon, Mar 16, 2020 at 6:39 PM Fanbin Bu wrote: > Hi Steve, > > could you please share your work around solution in more detail in the > above ticket? > > Thanks, > Fanbin > > On Mon, Mar

Re: Re: flink sql-client read hive orc table no result

Seems the root cause is "transactional"='true'. After remove this，the table can be queryed from flink sql-client，even i add "clustered by (robot_id) into 3 buckets" again. Thanks, Lei wangl...@geekplus.com.cn From: Kurt Young Date: 2020-03-18 18:04 To: wangl...@geekplus.com.cn CC: lirui;

Re: FlinkCEP - Detect absence of a certain event

Hi Humberto, I guess Fuji is familiar with Flink CEP and he can answer your proposed question. I already cc him. Best, Zhijiang -- From:Humberto Rodriguez Avila Send Time:2020 Mar. 18 (Wed.) 17:31 To:user Subject:FlinkCEP -

Re: Field types of query result and registered TableSink [Result] do not match

好的，了解了，多谢 Jark Jark Wu 于2020年3月19日周四 上午10:39写道： > Hi zhisheng, > > 目前 kafka source & jdbc sink 都是用的 TypeInformation ，所以都只能声明成 38, 18 或者直接写 > DECIMAL ，默认就是 38, 18。 > 这个问题会在升级到 new source/sink interface （FLIP-95）后有效解决。 > > Best, > Jark > > On Thu, 19 Mar 2020 at 10:31, zhisheng wrote: > > > hi,

Re: Upgrade flink fail from 1.9.1 to 1.10.0

Hey Andrey, Thanks for your answer. I know use savepoint to upgrade the flink cluster is the available, but that mean when I upgrading my flink cluster I need to cancel all jobs from JM. And the stream process will be stopped, that will have an impact for the production system which is time

Re: Field types of query result and registered TableSink [Result] do not match

Hi zhisheng, 目前 kafka source & jdbc sink 都是用的 TypeInformation ，所以都只能声明成 38, 18 或者直接写 DECIMAL ，默认就是 38, 18。 这个问题会在升级到 new source/sink interface （FLIP-95）后有效解决。 Best, Jark On Thu, 19 Mar 2020 at 10:31, zhisheng wrote: > hi, Jark > > 我只是使用了 flink-jdbc 这个 connector，发下我本地测试的 DDL 和 SQL 如下： > >

Can't create a savepoint with State Processor API

Hi everyone, I'm looking for a way to modify state inside an operator in Flink. I found State Processor API that allows to modify savepoints, which looks great. But I can't

Re: Field types of query result and registered TableSink [Result] do not match

Hi zhisheng, 我猜测你的 TableSink 的实现用了 legacy type, i.e. TypeInformation， 而不是 DataType。 legacy type 是无法表达精度的，所以只能 mapping 到默认的 38, 18 上。 这是框架做的一个合法性校验。 Best, Jark On Thu, 19 Mar 2020 at 09:33, zhisheng wrote: > hi, Jark > > 我刚使用 1.10.0 测试，报错异常如下： > > Exception in thread "main"

Re: Timestamp Erasure

Hi Dom, If you are converting a DataStream to a Table with a rowtime attribute, then the DataStream should hold event-time timestamp. For example, call `assignTimestampsAndWatermarks` before converting to table. You can find more details in the doc [1]. Best, Jark [1]:

Re: Field types of query result and registered TableSink [Result] do not match

对的，这是生产的一个 1.9 blink planner 作业，今天我测试一下 1.10 blink planner Jark Wu 于2020年3月18日周三 下午11:47写道： > Hi zhisheng, > > 你用的是1.9吗？ 试过 1.10.0 blink planner 么？ > > On Wed, 18 Mar 2020 at 22:21, zhisheng wrote: > > > hi, all > > > > 我定义的一个表的一个字段（yidun_score）是 numeric(5,2) 类型，写入 PostgreSQL 的 DDL >

Re: Flink Schema Validation - Kafka

Hi Robert, Thanks for your reply. This helps, was looking into similar direction. Thanks, Hemant On Wed, 18 Mar 2020 at 8:44 PM, Robert Metzger wrote: > Hi Hemant, > > you could let the Kafka consumer just deserialize your JSON data as into a > DataStream, then you use a custom

Timestamp Erasure

Hey, I just wanted to ask about one thing about timestamps. So, currently If I have a KeyedBroadcastProcess function followed by Temporal Table Join, it works like a charm. But, say I want to delay emitting some of the results due to any reason. So If I *registerProcessingTimeTimer* and any

Re: Flink checkStateMappingCompleteness doesn't include UserDefinedOperatorIDs

Thanks a lot for reporting this potential issue. The ticket looks good. I would suggest to keep the discussion on the ticket. Otherwise, information will potentially be split between this list and the ticket. I'm sure a committer will soon look at it. On Wed, Mar 18, 2020 at 5:19 AM Bashar

Re: time-windowed joins and tumbling windows

Hi Vinod, thanks for answering my questions. The == Optimized Logical Plan == looks as expected. However, the == Physical Execution Plan == seems to be quite complex. Are you sure that watermarks don't get lost in some of those custom operators before entering the SQL part of the pipeline?

Re: KafkaConsumer keeps getting InstanceAlreadyExistsException

Hi Becket/Till, Thanks for the detail explanation. Just to confirm: the issue in FLINK-8093 refers to multiple Kafka consumer within the same TM - thus the fix should be to make consumer client.id unique for different tasks ? and the issue here is an issue internal to the Kafka consumer, where

Re: Field types of query result and registered TableSink [Result] do not match

Hi zhisheng, 你用的是1.9吗？ 试过 1.10.0 blink planner 么？ On Wed, 18 Mar 2020 at 22:21, zhisheng wrote: > hi, all > > 我定义的一个表的一个字段（yidun_score）是 numeric(5,2) 类型，写入 PostgreSQL 的 DDL yidun_score > 字段也是定义的 numeric(5,2) 类型，结果会报异常。 > > org.apache.flink.client.program.ProgramInvocationException: The main

Re: Flink Schema Validation - Kafka

Hi Hemant, you could let the Kafka consumer just deserialize your JSON data as into a DataStream, then you use a custom processFunction to parse the string to JSON. In your custom function, you can handle the error more flexibly (like outputting erroneous records through a side output). I hope

Re: Custom Exception Handling

Hi Anil, You are never throwing any exception outside of your process function?, so your Flink job is not restarting because of a failure (your Flink job would restart because of a failure if you would throw an exception out of your user code). If you can rule-out a job restart (check the logs

Field types of query result and registered TableSink [Result] do not match

hi, all 我定义的一个表的一个字段（yidun_score）是 numeric(5,2) 类型，写入 PostgreSQL 的 DDL yidun_score 字段也是定义的 numeric(5,2) 类型，结果会报异常。 org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Field types of query result and registered TableSink [Result] do not match. Query result

Re: Upgrade flink fail from 1.9.1 to 1.10.0

Hi Reo, I do not think this is always guaranteed by Flink API. The usual supported way is to: - take a savepoint - upgrade the cluster (JM and TM) - maybe rebuild the job against the new Flink version - start the job from the savepoint [1] The externalised checkpoints also do not have to be

Upgrade flink fail from 1.9.1 to 1.10.0

Hi all, I encountered a problem when I upgrade flink from 1.9.1 to 1.10.0. At first, my job is running on flink stably which JM and TM is flink 1.9.1. And then I try to upgrade to 1.10.0. I stop the JM progress and start another JM progress. At this time, the JM is 1.10.0 and the TM is 1.9.1,

Re: Re: sql关键字问题

Hi, 你的这个问题就是 FLINK-16526 修复的问题，会在即将到来的 1.10.1 中发布。对于普通字段也是会加 ` ` 的，可以看这行代码 [1]。 Best, Jark [1]: https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/schema/CatalogSourceTable.scala#L116 On Wed, 18 Mar 2020 at

ProcessWindowFunction 问题求助

大神： 报错信息： Error:(73, 27) java: 对于process(com.linkedsee.aiops.CountErrorFunction), 找不到合适的方法 方法 org.apache.flink.streaming.api.datastream.DataStream.process(org.apache.flink.streaming.api.functions.ProcessFunction)不适用 (无法推断类型变量 R (参数不匹配;

Re: Re: sql关键字问题

Hi lucas, 你说的问题应该在https://issues.apache.org/jira/browse/FLINK-16068里面修复了。 lucas.wu 于2020年3月18日周三 下午5:27写道： > 原始邮件 > 发件人:lucas.wulucas...@xiaoying.com > 收件人:imj...@gmail.com > 发送时间:2020年3月18日(周三) 17:21 > 主题:Re: sql关键字问题 > > > Hi,jark >

Re: [DISCUSS] FLIP-111: Docker image unification

Thanks for creating this FLIP Andrey. I like the general idea pretty much. I tried to group some of the above-mentioned points in order to give my 50 cent. # Configuration How to configure the Flink process seems to be the biggest question. Due to historical reasons we have a plethora of

Re: Setting the operator-id to measure percentile latency over several jobs

I talked to Chesnay about this offline. Shipping the operator names with the latency markers would significantly increase their size (potentially affecting performance) There is no global lookup from operatorId to operatorName. On Fri, Mar 6, 2020 at 5:34 PM Robert Metzger wrote: > @Chesnay

Flink Schema Validation - Kafka

Hi Users, Is there a way I can do a schema validation on read from Kafka in a Flink job. I have a pipeline like this Kafka Topic Raw(json data) -> Kafka Topic Avro(avro data) -> Kafka Topic Transformed(avro data) -> Sink While reading from Raw topic I wanted to validate the schema so that in

Re: Re: flink sql-client read hive orc table no result

also try to remove "transactional"='true'? Best, Kurt On Wed, Mar 18, 2020 at 5:54 PM wangl...@geekplus.com.cn < wangl...@geekplus.com.cn> wrote: > > Tried again. Even i remove the "clustered by (robot_id) into 3 buckets" > statement, no result from flink sql-client > > Thanks, > Lei > >

Re: Re: flink sql-client read hive orc table no result

Tried again. Even i remove the "clustered by (robot_id) into 3 buckets" statement, no result from flink sql-client Thanks, Lei wangl...@geekplus.com.cn From: Kurt Young Date: 2020-03-18 17:41 To: wangl...@geekplus.com.cn; lirui CC: user Subject: Re: flink sql-client read hive orc table no

Re: flink sql-client read hive orc table no result

My guess is we haven't support hive bucket table yet. cc Rui Li for confirmation. Best, Kurt On Wed, Mar 18, 2020 at 5:19 PM wangl...@geekplus.com.cn < wangl...@geekplus.com.cn> wrote: > Hive table store as orc format: > CREATE TABLE `robot_tr`(`robot_id` int, `robot_time` bigint, >

flink maven打包

现在官网提供的maven打包方式，直接把第三方包解开后按照目录方式存放 而不是维持maven depend on的标准的jar包方式（带版本） 现在这种方式不利于软件的项目长期管理，项目长期累月运行后， 随着人员变化以及版本升级，会带来很多版本兼容和识别的工程问题 期望flink在1.10.0的后续版本改进该问题，可能需要更改运行时的classloader 发送自 Windows 10 版邮件应用

FlinkCEP - Detect absence of a certain event

In the documentation of FlinkCEP, I found that I can enforce that a particular event doesn't occur between two other events using notFollowedBy or notNext. However, I was wondering If I could detect the absence of a certain event after a time X. For example, if an event A is not followed by

转发：Re: sql关键字问题

原始邮件 发件人:lucas.wulucas...@xiaoying.com 收件人:imj...@gmail.com 发送时间:2020年3月18日(周三) 17:21 主题:Re: sql关键字问题 Hi,jark 看到了你修复的这个jirahttps://issues.apache.org/jira/browse/FLINK-16526 但是看了你的代码和描述，你只是针对SqlBasicCall这种node的字段名加了`` ,也就是说只会对有computed_column_expression的字段加上，但是对于普通的字段并没有覆盖到，请问我理解的正确吗？ 原始邮件

flink sql-client read hive orc table no result

Hive table store as orc format: CREATE TABLE `robot_tr`(`robot_id` int, `robot_time` bigint, `linear_velocity` double, `track_side_error` int) partitioned by (warehouseid STRING) clustered by (robot_id) into 3 buckets stored as orc tblproperties("transactional"='true'); Under hive

Re: 使用Flink sql insert 数据 to hive 之乱码问题 （已解决）

Hi, 会在1.10.1里面修复。 JIRA: https://issues.apache.org/jira/browse/FLINK-16647 谢谢 @li...@apache.org Best, Jingsong Lee On Wed, Mar 18, 2020 at 4:50 PM 吕先生 wrote: > Hi > > > 经过阿里工程师的协助，现定位到这是一个压缩问题。我的Hive 开启了压缩，以减小磁盘空间的使用。Flink > sql在写入压缩文件后（textfile

Re: How do I get the outPoolUsage value inside my own stream operator?

Hi Felipe, I checked the code path, and the metric of outPoolUsage is under the following layer: TaskMetricGroup -> TaskIOMetricGroup -> "buffers" group -> "outPoolUsage". It seems that you missed the `TaskIOMetricGroup` from below samples. You can get it from TaskMetricGroup. Hope it solve

Re:使用Flink sql insert 数据 to hive 之乱码问题 （已解决）

Hi 经过阿里工程师的协助，现定位到这是一个压缩问题。我的Hive 开启了压缩，以减小磁盘空间的使用。Flink sql在写入压缩文件后（textfile 格式），没有添加后缀名（正常情况下应该有“.snappy”后缀）。所以读的时候，把压缩文件当普通的txt 文件读了，造成乱码。 后面阿里的工程师会跟进这个问题。 解决办法：将hive-site.xml 里面的hive.exec.compress.output 设置成 false，或使用其他存储格式。 另外确认Flink sql 对Parquet、ORC 格式Hive 表的压缩是支持的。

Re: sql关键字问题

好像已经有了，应该是这个jira： https://issues.apache.org/jira/browse/FLINK-16526 Best, Kurt On Wed, Mar 18, 2020 at 4:19 PM Jingsong Li wrote: > Hi lucas, > > 赞专业的分析，看起来是Flink的bug，你可以建个Jira来跟踪。 > CC: @Yuzhao Chen > > Best, > Jingsong Lee > > On Wed, Mar 18, 2020 at 4:15 PM lucas.wu wrote: > > > 初步找到了原因

Re: Help me understand this Exception

Thanks, Zhijiang and Gordon. I will see the logs to find out more. On Wed, Mar 18, 2020 at 1:44 PM Zhijiang wrote: > Agree with Gordon's below explanation! > > Besides that, maybe you can also check the job master's log which might > probably show the specific exception to cause this failure.

Re: sql关键字问题

Hi lucas, 赞专业的分析，看起来是Flink的bug，你可以建个Jira来跟踪。 CC: @Yuzhao Chen Best, Jingsong Lee On Wed, Mar 18, 2020 at 4:15 PM lucas.wu wrote: > 初步找到了原因 > 原来我的建表语句用了computed_column_expression 这种语义。 > 然后flink内部在使用的时候其实是把它转成了select 语句 > ... > if (columnExprs.nonEmpty) { > val fieldExprs = fieldNames >

Re: Help me understand this Exception

Agree with Gordon's below explanation! Besides that, maybe you can also check the job master's log which might probably show the specific exception to cause this failure. I was thinking whether it is necessary to improve ExceptionInChainedOperatorException to also provide the message from the

回复：sql关键字问题

初步找到了原因 原来我的建表语句用了computed_column_expression 这种语义。 然后flink内部在使用的时候其实是把它转成了select 语句 ... if (columnExprs.nonEmpty) { val fieldExprs = fieldNames .map { name = if (columnExprs.contains(name)) { columnExprs(name) } else { name } }.toArray val rexNodes =

sql关键字问题

create table `source_table`( `SeqNo` varchar, `Type` varchar, `Table` varchar, `ServerId` varchar, `Database` varchar, `OldData` varchar, `GTID` varchar, `Offset` varchar, `event_ts` as to_timestamp(from_unixtime(Data.`FuiUpdateTime`),'-MM-ddHH:mm:ss'), WATERMARK FOR event_ts AS

转发：（无主题）

| | apache22 | | apach...@163.com | 签名由网易邮箱大师定制 - 转发邮件信息 - 发件人： 酷酷的浑蛋 发送日期： 2020年3月18日 15:15 发送至： user-zh 现在我发现个问题：flink sql实时 inner join ，结果会发生乱序，请问这是正常的吗 | | apache22 | | apach...@163.com | 签名由网易邮箱大师定制

Re: JobMaster does not register with ResourceManager in high availability setup

It seems that your zookeeper service is not stable. From the the log i find that resourcemanager leader is granted and taskmanager could register to resourcemanager successfully. That means the resourcemanager address has been published to the ZK successfully. Also a

Re: Scala Shell gives error "rest.address must be set"

Sure, will do. Thanks! On Tue, Mar 17, 2020 at 7:05 PM Jeff Zhang wrote: > I agree, this is really confusing for users. Do you mind to create a > ticket for that ? > > Craig Foster 于2020年3月18日周三 上午8:36写道： > >> If I specify these options, it seems to work...but I thought I could >> have this