Flink Stateful Functions 3.4

Hi there, Anyone knows which Flink version that Flink Stateful Functions 3.4 is compatible with? https://nightlies.apache.org/flink/flink-statefun-docs-master/docs/deployment/state-bootstrap/ I know Stateful Functions 3.3 is compatible with Flink 1.16.2, and Stateful Functions 3.2 is good with

RE: aws zeppelin glue table connector

Hi- Don't see a response for this so taking one last stab. Any reply would be greatly appreciated. Thanks, -Dave David L Katz (he/him/his) why?<https://sites.jpmchase.com/sites/companyhome/about%20us/diversity/pages/language,-inclusion-%26-gender-pronous.aspx> | Executive Director |

aws zeppelin glue table connector

outlined above expected to be supported with that JIRA? 2. Do we know when the connector will be released? 3. Do we have a general sense of when AWS would make that version of flink will be available as a flink runtime? Thanks in advance! -Dave David L Katz (he/him/his) why?<ht

RE: [EXTERNAL]Re: aws glue connector

backed by an s3 bucket in one account and a streaming glue table backed by a kinesis data stream in another account and want to join the two as part of a streaming application by just going to the central lake database as a consumer. Thanks again! -Dave David L Katz (he/him/his) why?<ht

aws glue connector

, -Dave David L Katz (he/him/his) why?<https://sites.jpmchase.com/sites/companyhome/about%20us/diversity/pages/language,-inclusion-%26-gender-pronous.aspx> | Executive Director | Macro Technology | 545 Washington Blvd, Floor 4 | Jersey City, NJ 07310 This message is confidential and s

RE: [EXTERNAL]multiple s3 sinks

Hi- Never mind. Error must have been from shutting down the prior version of the app (timestamp was after restart), issue is now resolved. Thanks, -Dave David L Katz (he/him/his) why?<https://sites.jpmchase.com/sites/companyhome/about%20us/diversity/pages/language,-inclusion-%26-gen

multiple s3 sinks

nk_table_det", resultsDet); stmtSet.execute(); I read that switching to a StatementSet is the way to go, but I'm still missing something. Any help would be greatly appreciated! Thanks, -Dave David L Katz (he/him/his) why?<https://sites.jpmchase.com/sites/companyhome/about%20us/diversity/pag

Re: Flink 1.12.8 release

> Martijn Visser > https://twitter.com/MartijnVisser82 > > [1] https://flink.apache.org/downloads.html#update-policy-for-old-releases > > > On Wed, 16 Feb 2022 at 08:54, Joey L wrote: > >> Hi, >> >> Is there a planned release date for 1.12.8 or scheduled release cycle for >> minor versions? >> >> Regards, >> J >> >

Flink 1.12.8 release

Hi, Is there a planned release date for 1.12.8 or scheduled release cycle for minor versions? Regards, J

Re: Kubernetes HA: New jobs stuck in Initializing for a long time after a certain number of existing jobs are running

anager pod) but fails. > SlotRequests are triggered but never actually fulfilled. They are put in > the queue for pending SlotRequests. The timeout kicks in after trying to > reach the ResourceManager for some time. That's > the NoResourcesAvailableException you are experiencing. > >

Re: NoClassDefFoundError for RMIServerImpl when running Flink with Scala 2.12 and Java 11

running some of > the examples with Java 11, without encountering the issue you describe (with > or without HA). Can you give more details on how exactly you built Flink > (ideally the full command), and how you ran the job? > > Best, > Nico > > On Thu, Oct 28, 2021 at 10:42 A

NoClassDefFoundError for RMIServerImpl when running Flink with Scala 2.12 and Java 11

Hi Flink developers, I encountered a weird error like follows. It only happens when I build Flink with Scala 2.12 profile and run with Java 11. No such error for Scala 2.11 + Java 8/11. Just search for relative info, but don't find any useful. Do you have any idea about this? Thanks.

Re: NoClassDefFoundError if Kafka classes are assemblied in application jar

> I don't think this is a general Flink issue or else no one would be able to > read from Kafka at all. > > On Fri, Oct 15, 2021 at 4:16 AM L. C. Hsieh wrote: >> >> Hi, Flink developers, >> >> Does anyone encounter the following error? >> >> java.lan

NoClassDefFoundError if Kafka classes are assemblied in application jar

Hi, Flink developers, Does anyone encounter the following error? java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/ByteArrayDeserializer at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.setDeserializer(FlinkKafkaConsumer.java:322) at

Re: Got NoClassDefFoundError on Flink native K8S cluster

ing(FlinkUserCodeClassLoader.java:63) So it looks like user code cannot use plugin's classes (e.g. Hadoop classes)? I don't see hadoop-common is shaded at the plugin. On 2021/08/22 18:24:24, L. C. Hsieh wrote: > > As I know, flink-shaded-hadoop is not officially supported since Flink 1.11

Re: Got NoClassDefFoundError on Flink native K8S cluster

en2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-10.0/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar. > May help. > > L. C. Hsieh 于2021年8月22日周日 上午1:40写道： > > > > > BTW, I checked dependency tree, the flink-iceberg demo only has one Hadoop > > common dependency. So I'

Re: Got NoClassDefFoundError on Flink native K8S cluster

Iceberg and Flink. On 2021/08/21 08:50:05, L. C. Hsieh wrote: > > Thanks for replying. > > I'm using Flink 1.12.x. And I think Iceberg 0.12 uses Flink 1.12 actually. > > Once I upgraded the Iceberg from 0.11.0 to 0.12.0 for the Java application. I > got n

Re: Got NoClassDefFoundError on Flink native K8S cluster

HadoopConf(FlinkCatalogFactory.java:152) On 2021/08/21 08:11:33, Manong Karl wrote: > Iceberg v0.11 or v0.12 not capable with flink v1.13.x. > > L. C. Hsieh 于2021年8月21日周六 下午3:52写道： > > > Hi, I'm testing using Flink to write Iceberg table. I run Flink native K8S > > c

Got NoClassDefFoundError on Flink native K8S cluster

Hi, I'm testing using Flink to write Iceberg table. I run Flink native K8S cluster locally and submit a simple Java program that writes out Iceberg table (https://github.com/spancer/flink-iceberg-demo). But got an exception: java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration

固定间隔重启策略 - 计数逻辑

Hi 大家好： 我想问一下当重启策略为：restart-strategy: fixed-delay 时，其参数 restart-strategy.fixed-delay.attempts 是全局计数（任务生命周期）？还是每次当任务从HA失败策略中恢复后，会重置重试次数，下次失败重新从0开始？

????flink????state

import org.apache.flink.api.java.functions.KeySelector; import org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows; import org.apache.flink.streaming.api.windowing.time.Time; ... DataStream

??????flinkSQL??ValueStateDescriptor????????StateTtlConfig

streamTableEnv.getConfig().setIdleStateRetention(Duration.ofDays(1)); ---- ??: "stgztsw"http://apache-flink.147419.n8.nabble.com/

flink sql

hi, ?? ??1.12flink sql ??datastream?,

??????????: ????: ??????????

th-idle-sources best, amenhub nbsp; ???L ?? 2021-02-01 17:20 user-zh ?? ?? flink1.12, kafka??3??, flink??3??. ??, , ??,, ?

??????????: ??????????

th-idle-sources best, amenhub ???L ?? 2021-02-01 17:20 user-zh ?? ?? flink1.12, kafka??3??, flink??3??. ??, , ??,, ?

??????????

flink1.12, kafka??3??, flink??3??. ??, , ??,, ?

?????? flink sql????????

streamTableEnv.getConfig().setLocalTimeZone(ZoneId.of("+8")) . ,??, ---- ??:

flink sql????????

streamTableEnv.getConfig().setLocalTimeZone(ZoneId.of("+8"))

flink sql ????????????

flink sql+8,??

?????? ????????????

kafka ??3??, ??, flink??3 ---- ??: "user-zh"

????????????

EnvironmentSettings settings = EnvironmentSettings.newInstance().inStreamingMode().useBlinkPlanner().build(); StreamExecutionEnvironment executionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment(); executionEnvironment.setParallelism(3); Map

FlinkSQL Filter Error With Float Column on flink-1.12.0

Hi: Flink SQL filter data throw an exception, code: def main(args: Array[String]): Unit = { val env = StreamExecutionEnvironment.getExecutionEnvironment val source = env.fromElements( (1.0f, 11.0f, 12.0f), (2.0f, 21.0f, 22.0f), (3.0f, 31.0f, 32.0f), (4.0f, 41.0f,

FlinkSQL Filter Error With Float Column on flink-1.12.0

Hi: Flink SQL filter data throw an exception, code: def main(args: Array[String]): Unit = { val env = StreamExecutionEnvironment.getExecutionEnvironment val source = env.fromElements( (1.0f, 11.0f, 12.0f), (2.0f, 21.0f, 22.0f), (3.0f, 31.0f, 32.0f), (4.0f, 41.0f,

DynamicTableSource中Filter push down

Hi: 各位大佬，请教一个问题。 我再flink-1.12.0上自定义一个DynamicTableSource，并支持SupportsFilterPushDown，SupportsProjectionPushDown等特性。 然后我的ScanRuntimeProvider使用的是InputFormatProvider。 我运行时，下推的filters在创建InputFormat和copy()方法之后，那我还怎么在InputFormat中根据filters去过滤数据源呢？

TableEnvironment中怎么添加参数，使参数能在TaskManagerServices中能获取到？

Hi： 我的程序是Flink Table/SQL开发的，我想要设置参数能在TaskManagerServices中生效，怎么设置？ 我使用如下设置，在TaskManagerServices中并不能获取到我设置的值。 val settings = EnvironmentSettings.newInstance() .inBatchMode() .build() val tEnv = TableEnvironment.create(settings)

java.lang.IllegalStateException: Trying to access closed classloader.

Hi: 我在Idea里面运行我的flink程序，报了如下异常： Exception in thread "Thread-22" java.lang.IllegalStateException: Trying to access closed classloader. Please check if you store classloaders directly or indirectly in static fields. If the stacktrace suggests that the leak occurs in a third party library and cannot

flink1.10.0中的filter下推还能在flink1.12.0中使用吗？

Hi： 我们在flink-1.10.0中自定义了connect，并实现类了FilterableTableSource接口。目前flink-1.12.0发布了，我们想将之前解析Expression放到新版本中来使用。但是发现之前解析的方式在新版本中不能使用了。 是新版本不再支持flink-1.10.0中的Expression解析方式了吗？还是之前的依旧可以用，只是我的打开方式不对，那在1.12.0中怎么解析Expression生成Or、EqualTo、LessThan等下推的Filter？ 祝好！

tEnv.executeSql(query).print() 这样不能成功消费kafka的数据

Hi： 我Flink消费kafka的数据，我创建一张表如下： val kafkaSource = """ |create table kafka_order( |order_id string, |order_price decimal(10,2), |order_time timestamp(3) |) |with( |'connector' = 'kafka', |'topic' = 'iceberg.order',

创建的临时表，在新的catalog中找不到对应的临时表

Hi, 通过以下API创建了一张临时表： val oStream: DataStream[Order] = env.addSource(new BugSource) tEnv.createTemporaryView("t_order", oStream, 'order_id, 'order_price, 'order_time) 然后又创建了新的catalog以及数据库，过程如下： val catalog = """ |create catalog hive_catalog with ( | 'type'='iceberg', |

ScanTableSource 为什么不支持SupportsFilterPushDown的问题

Hi: 由于业务需要，我想在flink1.11.2中自定义一个ScanTableSource， 并想实现SupportsFilterPushDown特性。 但是在摸索的过程中发现，以下代码确似乎禁止使用SupportsFilterPushDown特性。 val unsupportedAbilities = List( classOf[SupportsFilterPushDown], classOf[SupportsLimitPushDown], classOf[SupportsPartitionPushDown],

SQL Cli中找不到DeserializationSchemaFactory

Hi： flink版本1.12.0： 我想在sql-client-defaults.yaml中配置一张表，配置如下： tables: - name: t_users type: source-table connector: property-version: 1 type: kafka version: universal topic: ods.userAnalysis.user_profile startup-mode: latest-offset

Re: Flink SQL的灵异事件----查询语句中增加表中的某个字段时就没法正常查询了。

我使用的是release-1.12.0-rc1 Best Jark Wu 于2020年11月24日周二 上午11:42写道： > 看报错像是一个 bug。 请问使用的是哪个版本呢？ > 可以去 JIRA issue 提个 issue。 > > Best, > Jark > > On Tue, 24 Nov 2020 at 11:27, jy l wrote: > > > Hi： > > FlinkSQL我在使用时发生一件很诡异的事件。具体如下： > > > > 我的DDL： &g

Flink SQL的灵异事件----查询语句中增加表中的某个字段时就没法正常查询了。

Hi： FlinkSQL我在使用时发生一件很诡异的事件。具体如下： 我的DDL： create table if not exists t_order( id int PRIMARY KEY comment '订单id', timestamps bigint comment '订单创建时间', orderInformationId string comment '订单信息ID', userId string comment '用户ID', categoryId int comment '商品类别', productId int comment '商品ID', price

Flink SQL 对延迟数据怎么处理？

Hi： 请教一下，FlinkSQL中，我在创建表时设置了watermark并设置了最大延迟，可是还是有数据依旧会迟到晚到，对于这样的数据我们又不想直接丢弃，那这个依旧迟到的数据我该怎么收集？是否有与StreamAPI一样可以将依旧迟到的数据进行分流的方案？ 祝好！

Re: FlinkSQL CDC 窗口分组聚合求助

好的，我试一下。谢谢 Best Jark Wu 于2020年11月23日周一 下午2:06写道： > 那是不是用非窗口聚合，开5s mini batch，是不是可以达到你的需求？ > > Best, > Jark > > On Mon, 23 Nov 2020 at 13:16, jy l wrote: > > > 使用场景就是我们想用Flink 根据订单系统中的订单表，每5s钟计算一次总的交易金额。 > > 目前我们的系统大致架构是mysql(debezium)>kafka--->flink

Re: FlinkSQL CDC 窗口分组聚合求助

> > On Mon, 23 Nov 2020 at 10:28, jy l wrote: > > > Hi： > > 我使用CDC组件debezium将MySQL中的维表数据实时发送到kafka，然后使用flinkSQL消费kafka的数据，具体流程如下： > > [image: image.png] > > [image: image.png] > > 分组计算的SQL如下： > > [image: image.png] > > 在执行计算时

FlinkSQL CDC 窗口分组聚合求助

Hi： 我使用CDC组件debezium将MySQL中的维表数据实时发送到kafka，然后使用flinkSQL消费kafka的数据，具体流程如下： [image: image.png] [image: image.png] 分组计算的SQL如下： [image: image.png] 在执行计算时，报了如下异常： Exception in thread "main" org.apache.flink.table.api.TableException: GroupWindowAggregate doesn't support consuming update and delete

关于global window

您好，我在看global window的时候有一些疑问，首先是global window是全局唯一的，也就是所以的元素都会流到这一个window实例上，这样的话前面加keyBy还有什么意义呢？反正都要流到一个window里面，我测试的结果好像也确实是这样，我自定义了一个global window，然后设置了process的并行度，但是window确实是只有一个 示例如下： dataUnion.keyBy(0).window(new StreamToBatchWindow()).process(new

Re: Flink Batch 模式下，The rpc invocation size 113602196 exceeds the maximum akka framesize

如果使用了print()等算子，会将上一个task的结果一次全部pull过来，pull时数据超过了akka framesize大小导致。 李加燕 于2020年9月28日周一 下午3:07写道： > Flink batch 模式消费hdfs上的文件，并做了一个word count > 操作，但是task一直运行，查看taskmanager的log，发现如下异常： > java.lang.reflect.UndeclaredThrowableException: null > at

通过Flink web ui上传jar包时无响应

通过Flink web ui上传jar包时链接被重置，不清楚是什么原因，请问各位大佬遇到过么？ [image: pic_2020-08-15_10-39-37.png] [image: pic_2020-08-15_10-40-09.png]

Request for removal from subscription

Unsubscribe

Re: Re: 怎样把 state 定时写到外部存储

使用 window 聚合一下批量写呢？ wangl...@geekplus.com.cn 于2019年11月1日周五 上午10:17写道： > Hi Congxian， > > 以 sink 的方式写出去最终也要落在某个地方才能供查询使用啊。 > 我们的 case 是写到 MySQL 中 > > > > wangl...@geekplus.com.cn > > Sender: Congxian Qiu > Send Time: 2019-11-01 10:10 > Receiver: user-zh > Subject: Re: 怎样把 state 定时写到外部存储 >

Re: [Discuss] What should the "Data Source" be translated into Chinese

我也更倾向于不翻译 Data Source 和 Data Sink Kurt Young 于2019年8月13日周二 下午6:21写道： > "更倾向不去翻译Data Source和Data Sink, 通过用中文对其做解释即可" +1 > > Best, > Kurt > > > On Tue, Aug 13, 2019 at 6:18 PM Simon Su wrote: > > > 更倾向不去翻译Data Source和Data Sink, 通过用中文对其做解释即可 > > > > > > Thanks, > > SImon > > > > > > On

Query - External SSL setup

Hi, I am trying to authenticate Flink using NGINX. In the document it is mentioned to deploy a "side car proxy", here is the link for the section of the document which I have referred to

Flink window operation based on event time is triggered when watermark is less than the end of window ends

Please refer to the stackoverflow . Thanks. -- Thanks. · Lx wlxwol...@gmail.com

Re: Duplicates in self join

ed on its timestamp and the join window interval. > > Best, > Fabian > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/joining.html#interval-join > > Am Mo., 8. Okt. 2018 um 16:44 Uhr schrieb Eric L Goodman < > eric.good...@colorado.

Re: Duplicates in self join

assumption is correct, you can add a ProcessFunction after the join > to do distinct. > > Best, Hequn > > On Mon, Oct 8, 2018 at 10:37 PM Eric L Goodman > wrote: > >> If I change it to a Tumbling window some of the results will be lost >> since the pattern I'm match

Re: Duplicates in self join

ocs-release-1.6/api/java/org/apache/flink/streaming/api/windowing/evictors/Evictor.html >> >> Best Regards, >> Dominik. >> >> pon., 8 paź 2018 o 08:00 Eric L Goodman >> napisał(a): >> >>> What is the best way to avoid or remove dupli

Duplicates in self join

What is the best way to avoid or remove duplicates when joining a stream with itself? I'm performing a streaming temporal triangle computation and the first part is to find triads of two edges of the form vertexA->vertexB and vertexB->vertexC (and there are temporal constraints where the first

multiple input streams

If I have a standalone cluster running flink, what is the best way to ingest multiple streams of the same type of data? For example, if I open a socket text stream, does the socket only get opened on the master node and then the stream is partitioned out to the worker nodes? DataStream text =