Hi all, my flink job consume kafka topic A, and write to kafka topic B.
When i restart my flink job via savepoint, topic B have some duplicate
message. Any one can help me how to solve this problem? Thanks!
My Versions:
Flink 1.12.4
Kafka 2.0.1
Java 1.8
Core code:
我本地试了一下没有复现你的问题，你的hive版本是什么呢？
On Fri, Jul 30, 2021 at 3:00 PM Asahi Lee <978466...@qq.com.invalid> wrote:
> CREATE TABLE `cosldatacenter.ods_emp_maindata_iadc_paramvalue`(
> `paramvalue_id` string COMMENT '',
> `platform_id` string COMMENT '',
> `equipment_id` string COMMENT '',
> `param_id`
Hi Leonard,
I am using flink 1.11.2 and using debezium-json to read CDC data generated
by debezium.
For each table, I convert the Kafka dynamic table to a retract stream and
finally that stream is converted to DataStream. Here's the sample
function
private DataStream getDataStream(String sql) {
你好，我试了一下，如果表的ddl是三个字段，但是insert只指定两个字段的话，会报错：
[ERROR] Could not execute SQL statement. Reason:
org.apache.flink.table.api.ValidationException: Column types of query result
and sink for registered table 'default_catalog.default_database.t' do not match.
Cause: Different number of columns.
你好，upsert是全字段更新，必须指定所有的字段值。我们的需求是想根据主键更新部分字段，其余字段不变。
现有table
CREATE TABLE t (
abigint,
bbigint,
cbigint,
PRIMARY KEY (a) NOT ENFORCED
) WITH (
...
);
我们的场景只想根据主键a更新部分字段b，其余的字段c保持不变，例如
mysql 支持 insert into
1. I use the filesystem as the state backend, and the state should be in memory.
2. The mini-batch function is disabled.
3. Does mini-batch reduce memory usage? I found that the memory usage of the
overwindows grew fast and the JVM FunllGC was frequent. Tenured Gen occupies a
large amount of
如果只想更新部分字段的话可以试下
insert into t(a,b) select a,b from x
--
发件人：Ye Chen
发送时间：2021年7月30日(星期五) 17:57
收件人：user-zh
主 题：场景题：Flink SQL 不支持 INSERT INTO… ON DUPLICATE KEY UPDATE ?
现有table
CREATE TABLE t (
abigint,
b
AFAIK, the ClusterClient should not be exposed through the public API.
Would you like to explain your use case and why you need to get the
web UI programmatically?
Best,
Yangze Guo
On Fri, Jul 30, 2021 at 9:54 PM Hailu, Andreas [Engineering]
wrote:
>
> Hello Yangze, thanks for responding.
>
>
Hi, Ayush
Thanks for the detailed description.
Before analyze the issue, I have two questions that which Flink and Flink CDC
version are you using? Is Flink CDC used in SQL or DataStream ?
That’s helpful if you can post you Flink CDC connector parameters.
Best,
Leonard
> 在
Hi!
As the state grows the processing speed will slow down a bit. Which state
backend are you using? Is mini batch enabled[1]?
[1]
https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/config/#table-exec-mini-batch-enabled
Wanghui (HiCampus) 于2021年7月30日周五 下午3:59写道：
> Hi :
>
>
你在你的sink ddl定义了主键会自动的按主键进行upsert的
参考https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/table/jdbc/#idempotent-writes
--
发件人：Ye Chen
发送时间：2021年7月30日(星期五) 17:57
收件人：user-zh
主 题：场景题：Flink SQL 不支持 INSERT
11 matches
Mail list logo
