[jira] [Updated] (FLINK-20953) canal-json.table.include Supports regular or wildcard characters
[ https://issues.apache.org/jira/browse/FLINK-20953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangfei updated FLINK-20953: Description: |h5. canal-json.database.include|optional|(none)|String|Only read changelog rows which match the specific database (by comparing the "database" meta field in the Canal record).| |h5. canal-json.table.include|optional|(none)|String|Only read changelog rows which match the specific table (by comparing the "table" meta field in the Canal record).| h5. canal-json.database.include and canal-json.table.include Supports regular or wildcard characters was: |h5. canal-json.database.include|optional|(none)|String|仅读取指定数据库的 changelog 记录（通过对比 Canal 记录中的 "database" 元数据字段）| |h5. canal-json.table.include|optional|(none)|String|仅读取指定表的 changelog 记录（通过对比 Canal 记录中的 "table" 元数据字段）。| h5. canal-json.database.include and canal-json.table.include Supports regular or wildcard characters > canal-json.table.include Supports regular or wildcard characters > > > Key: FLINK-20953 > URL: https://issues.apache.org/jira/browse/FLINK-20953 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Reporter: wangfei >Priority: Major > Fix For: 1.13.0 > > > |h5. canal-json.database.include|optional|(none)|String|Only read changelog > rows which match the specific database (by comparing the "database" meta > field in the Canal record).| > |h5. canal-json.table.include|optional|(none)|String|Only read changelog rows > which match the specific table (by comparing the "table" meta field in the > Canal record).| > h5. canal-json.database.include and canal-json.table.include Supports > regular or wildcard characters > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20953) canal-json.table.include Supports regular or wildcard characters
[ https://issues.apache.org/jira/browse/FLINK-20953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17263963#comment-17263963 ] wangfei commented on FLINK-20953: - @[~jark] @[~nicholasjiang] > canal-json.table.include Supports regular or wildcard characters > > > Key: FLINK-20953 > URL: https://issues.apache.org/jira/browse/FLINK-20953 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Reporter: wangfei >Priority: Major > Fix For: 1.13.0 > > > |h5. canal-json.database.include|optional|(none)|String|仅读取指定数据库的 changelog > 记录（通过对比 Canal 记录中的 "database" 元数据字段）| > |h5. canal-json.table.include|optional|(none)|String|仅读取指定表的 changelog > 记录（通过对比 Canal 记录中的 "table" 元数据字段）。| > h5. canal-json.database.include and canal-json.table.include Supports > regular or wildcard characters > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20953) canal-json.table.include Supports regular or wildcard characters
wangfei created FLINK-20953: --- Summary: canal-json.table.include Supports regular or wildcard characters Key: FLINK-20953 URL: https://issues.apache.org/jira/browse/FLINK-20953 Project: Flink Issue Type: Improvement Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) Reporter: wangfei Fix For: 1.13.0 |h5. canal-json.database.include|optional|(none)|String|仅读取指定数据库的 changelog 记录（通过对比 Canal 记录中的 "database" 元数据字段）| |h5. canal-json.table.include|optional|(none)|String|仅读取指定表的 changelog 记录（通过对比 Canal 记录中的 "table" 元数据字段）。| h5. canal-json.database.include and canal-json.table.include Supports regular or wildcard characters -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-20885) Exception when use 'canal-json.table.include' to filter Canal binlog but table contains 'source' column
[
https://issues.apache.org/jira/browse/FLINK-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17263923#comment-17263923
]
wangfei edited comment on FLINK-20885 at 1/13/21, 7:03 AM:
---
Remove \, the JIRA automatically adds \
!image-2021-01-13-15-02-33-266.png!
was (Author: wangfeiair2324):
{"data":[\{"id":"0","username":"zhangsna","password":"passwd","AddTime":"2020-12-25
14:23:07"}],"database":"plink","es":1610520384000,"id":92151,"isDdl":false,"mysqlType":\{"id":"int(11)","username":"varchar(255)","password":"varchar(255)","AddTime":"datetime"},"old":[\{"password":""}],"pkNames":["id"],"sql":"","sqlType":\{"id":4,"username":12,"password":12,"AddTime":93},"table":"test_1","ts":1610520384579,"type":"UPDATE"}
> Exception when use 'canal-json.table.include' to filter Canal binlog but
> table contains 'source' column
> ---
>
> Key: FLINK-20885
> URL: https://issues.apache.org/jira/browse/FLINK-20885
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Ecosystem
>Reporter: Jark Wu
>Assignee: Nicholas Jiang
>Priority: Major
> Attachments: image-2021-01-08-14-52-53-443.png,
> image-2021-01-08-14-53-40-476.png, image-2021-01-13-15-02-33-266.png,
> screenshot-1.png
>
>
> I found a bug in the canal code. 'canal-json.table.include' does not filter
> out the binlog of the specified table correctly, which will cause an error in
> the parsing section. For example, if I want to read the binlog of
> canal-json.table.include = 'a' table, there is a source field of int in table
> a, but at this time if table b also has a source field of string, An error
> will be reported.
> !screenshot-1.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20885) Exception when use 'canal-json.table.include' to filter Canal binlog but table contains 'source' column
[
https://issues.apache.org/jira/browse/FLINK-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17263923#comment-17263923
]
wangfei commented on FLINK-20885:
-
{"data":[\{"id":"0","username":"zhangsna","password":"passwd","AddTime":"2020-12-25
14:23:07"}],"database":"plink","es":1610520384000,"id":92151,"isDdl":false,"mysqlType":\{"id":"int(11)","username":"varchar(255)","password":"varchar(255)","AddTime":"datetime"},"old":[\{"password":""}],"pkNames":["id"],"sql":"","sqlType":\{"id":4,"username":12,"password":12,"AddTime":93},"table":"test_1","ts":1610520384579,"type":"UPDATE"}
> Exception when use 'canal-json.table.include' to filter Canal binlog but
> table contains 'source' column
> ---
>
> Key: FLINK-20885
> URL: https://issues.apache.org/jira/browse/FLINK-20885
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Ecosystem
>Reporter: Jark Wu
>Assignee: Nicholas Jiang
>Priority: Major
> Attachments: image-2021-01-08-14-52-53-443.png,
> image-2021-01-08-14-53-40-476.png, screenshot-1.png
>
>
> I found a bug in the canal code. 'canal-json.table.include' does not filter
> out the binlog of the specified table correctly, which will cause an error in
> the parsing section. For example, if I want to read the binlog of
> canal-json.table.include = 'a' table, there is a source field of int in table
> a, but at this time if table b also has a source field of string, An error
> will be reported.
> !screenshot-1.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20885) Exception when use 'canal-json.table.include' to filter Canal binlog but table contains 'source' column
[ https://issues.apache.org/jira/browse/FLINK-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17263917#comment-17263917 ] wangfei commented on FLINK-20885: - You can use the JSON above。 and canal-json.database.include and canal-json.table.include， It cannot be modified to support regular or wildcard characters？ > Exception when use 'canal-json.table.include' to filter Canal binlog but > table contains 'source' column > --- > > Key: FLINK-20885 > URL: https://issues.apache.org/jira/browse/FLINK-20885 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / Ecosystem >Reporter: Jark Wu >Assignee: Nicholas Jiang >Priority: Major > Attachments: image-2021-01-08-14-52-53-443.png, > image-2021-01-08-14-53-40-476.png, screenshot-1.png > > > I found a bug in the canal code. 'canal-json.table.include' does not filter > out the binlog of the specified table correctly, which will cause an error in > the parsing section. For example, if I want to read the binlog of > canal-json.table.include = 'a' table, there is a source field of int in table > a, but at this time if table b also has a source field of string, An error > will be reported. > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20885) Exception when use 'canal-json.table.include' to filter Canal binlog but table contains 'source' column
[
https://issues.apache.org/jira/browse/FLINK-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17263916#comment-17263916
]
wangfei commented on FLINK-20885:
-
|{"data":[\{"id":"0","username":"zhangsna","password":"passwd","AddTime":"2020-12-25
14:23:07"}],"database":"plink","es":1610520384000,"id":92151,"isDdl":false,"mysqlType":\{"id":"int(11)","username":"varchar(255)","password":"varchar(255)","AddTime":"datetime"},"old":[\{"password":""}],"pkNames":["id"],"sql":"","sqlType":\{"id":4,"username":12,"password":12,"AddTime":93},"table":"test_1","ts":1610520384579,"type":"UPDATE"}|
> Exception when use 'canal-json.table.include' to filter Canal binlog but
> table contains 'source' column
> ---
>
> Key: FLINK-20885
> URL: https://issues.apache.org/jira/browse/FLINK-20885
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Ecosystem
>Reporter: Jark Wu
>Assignee: Nicholas Jiang
>Priority: Major
> Attachments: image-2021-01-08-14-52-53-443.png,
> image-2021-01-08-14-53-40-476.png, screenshot-1.png
>
>
> I found a bug in the canal code. 'canal-json.table.include' does not filter
> out the binlog of the specified table correctly, which will cause an error in
> the parsing section. For example, if I want to read the binlog of
> canal-json.table.include = 'a' table, there is a source field of int in table
> a, but at this time if table b also has a source field of string, An error
> will be reported.
> !screenshot-1.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-20885) Exception when use 'canal-json.table.include' to filter Canal binlog but table contains 'source' column
[
https://issues.apache.org/jira/browse/FLINK-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17263858#comment-17263858
]
wangfei edited comment on FLINK-20885 at 1/13/21, 3:16 AM:
---
table test_2.userName (int); table test_1.userName (varchar);
You create a ddl with canal-json.table.include=test_2 .
Then add the binlog of test_1, as in the binlog format above
{"data":[\{"id":"0","username":"zhangsna","password":"","AddTime":"-00-00
00:00:00"}],"database":"plink","es":1610507775000,"id":90557,"isDdl":false,"mysqlType":\{"id":"int(11)","username":"varchar(255)","password":"varchar(255)","AddTime":"datetime"},"old":[\{"password":"111"}],"pkNames":["id"],"sql":"","sqlType":\{"id":4,"username":12,"password":12,"AddTime":93},"table":"test_1","ts":1610507775184,"type":"UPDATE"}
Then you get an error
was (Author: wangfeiair2324):
table test_2.userName (int); table test_1.userName (varchar);
You create a ddl with canal-json.table.include=test_2 .
Then add the binlog of test_1, as in the binlog format above
{"data":[\{"id":"0","username":"zhangsna","password":"111","AddTime":"-00-00
00:00:00"}],"database":"plink","es":1610507126000,"id":90469,"isDdl":false,"mysqlType":\{"id":"int(11)","username":"varchar(255)","password":"varchar(255)","AddTime":"datetime"},"old":[\{"username":null,"password":null}],"pkNames":["id"],"sql":"","sqlType":\{"id":4,"username":12,"password":12,"AddTime":93},"table":"test_1","ts":1610507127009,"type":"UPDATE"}
Then you get an error
> Exception when use 'canal-json.table.include' to filter Canal binlog but
> table contains 'source' column
> ---
>
> Key: FLINK-20885
> URL: https://issues.apache.org/jira/browse/FLINK-20885
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Ecosystem
>Reporter: Jark Wu
>Assignee: Nicholas Jiang
>Priority: Major
> Attachments: image-2021-01-08-14-52-53-443.png,
> image-2021-01-08-14-53-40-476.png, screenshot-1.png
>
>
> I found a bug in the canal code. 'canal-json.table.include' does not filter
> out the binlog of the specified table correctly, which will cause an error in
> the parsing section. For example, if I want to read the binlog of
> canal-json.table.include = 'a' table, there is a source field of int in table
> a, but at this time if table b also has a source field of string, An error
> will be reported.
> !screenshot-1.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-20885) Exception when use 'canal-json.table.include' to filter Canal binlog but table contains 'source' column
[
https://issues.apache.org/jira/browse/FLINK-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17263858#comment-17263858
]
wangfei edited comment on FLINK-20885 at 1/13/21, 3:14 AM:
---
table test_2.userName (int); table test_1.userName (varchar);
You create a ddl with canal-json.table.include=test_2 .
Then add the binlog of test_1, as in the binlog format above
{"data":[\{"id":"0","username":"zhangsna","password":"111","AddTime":"-00-00
00:00:00"}],"database":"plink","es":1610507126000,"id":90469,"isDdl":false,"mysqlType":\{"id":"int(11)","username":"varchar(255)","password":"varchar(255)","AddTime":"datetime"},"old":[\{"username":null,"password":null}],"pkNames":["id"],"sql":"","sqlType":\{"id":4,"username":12,"password":12,"AddTime":93},"table":"test_1","ts":1610507127009,"type":"UPDATE"}
Then you get an error
was (Author: wangfeiair2324):
table test_2.userName (int); table test_1.userName (varchar);
You create a canal-json.table.include=test_2
Then you get an error
> Exception when use 'canal-json.table.include' to filter Canal binlog but
> table contains 'source' column
> ---
>
> Key: FLINK-20885
> URL: https://issues.apache.org/jira/browse/FLINK-20885
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Ecosystem
>Reporter: Jark Wu
>Assignee: Nicholas Jiang
>Priority: Major
> Attachments: image-2021-01-08-14-52-53-443.png,
> image-2021-01-08-14-53-40-476.png, screenshot-1.png
>
>
> I found a bug in the canal code. 'canal-json.table.include' does not filter
> out the binlog of the specified table correctly, which will cause an error in
> the parsing section. For example, if I want to read the binlog of
> canal-json.table.include = 'a' table, there is a source field of int in table
> a, but at this time if table b also has a source field of string, An error
> will be reported.
> !screenshot-1.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20885) Exception when use 'canal-json.table.include' to filter Canal binlog but table contains 'source' column
[ https://issues.apache.org/jira/browse/FLINK-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17263858#comment-17263858 ] wangfei commented on FLINK-20885: - table test_2.userName (int); table test_1.userName (varchar); You create a canal-json.table.include=test_2 Then you get an error > Exception when use 'canal-json.table.include' to filter Canal binlog but > table contains 'source' column > --- > > Key: FLINK-20885 > URL: https://issues.apache.org/jira/browse/FLINK-20885 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / Ecosystem >Reporter: Jark Wu >Assignee: Nicholas Jiang >Priority: Major > Attachments: image-2021-01-08-14-52-53-443.png, > image-2021-01-08-14-53-40-476.png, screenshot-1.png > > > I found a bug in the canal code. 'canal-json.table.include' does not filter > out the binlog of the specified table correctly, which will cause an error in > the parsing section. For example, if I want to read the binlog of > canal-json.table.include = 'a' table, there is a source field of int in table > a, but at this time if table b also has a source field of string, An error > will be reported. > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20885) Exception when use 'canal-json.table.include' to filter Canal binlog but table contains 'source' column
[
https://issues.apache.org/jira/browse/FLINK-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17263857#comment-17263857
]
wangfei commented on FLINK-20885:
-
[~nicholasjiang]
CREATE TABLE `test_2` (
`id` int(11) NOT NULL,
`username` int(11) DEFAULT NULL COMMENT '用户名',
`password` varchar(255) DEFAULT NULL COMMENT '密码',
`AddTime` datetime NOT NULL COMMENT '创建时间',
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
{"data":[\{"id":"0","username":"22","password":"11","AddTime":"-00-00
00:00:00"}],"database":"plink","es":1610507187000,"id":90481,"isDdl":false,"mysqlType":\{"id":"int(11)","username":"int(11)","password":"varchar(255)","AddTime":"datetime"},"old":[\{"password":null}],"pkNames":["id"],"sql":"","sqlType":\{"id":4,"username":4,"password":12,"AddTime":93},"table":"test_2","ts":1610507187259,"type":"UPDATE"}
CREATE TABLE `test_1` (
`id` int(11) NOT NULL,
`username` varchar(255) DEFAULT NULL COMMENT '用户名',
`password` varchar(255) DEFAULT NULL COMMENT '密码',
`AddTime` datetime NOT NULL COMMENT '创建时间',
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
{"data":[\{"id":"0","username":"zhangsna","password":"111","AddTime":"-00-00
00:00:00"}],"database":"plink","es":1610507126000,"id":90469,"isDdl":false,"mysqlType":\{"id":"int(11)","username":"varchar(255)","password":"varchar(255)","AddTime":"datetime"},"old":[\{"username":null,"password":null}],"pkNames":["id"],"sql":"","sqlType":\{"id":4,"username":12,"password":12,"AddTime":93},"table":"test_1","ts":1610507127009,"type":"UPDATE"}
> Exception when use 'canal-json.table.include' to filter Canal binlog but
> table contains 'source' column
> ---
>
> Key: FLINK-20885
> URL: https://issues.apache.org/jira/browse/FLINK-20885
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Ecosystem
>Reporter: Jark Wu
>Assignee: Nicholas Jiang
>Priority: Major
> Attachments: image-2021-01-08-14-52-53-443.png,
> image-2021-01-08-14-53-40-476.png, screenshot-1.png
>
>
> I found a bug in the canal code. 'canal-json.table.include' does not filter
> out the binlog of the specified table correctly, which will cause an error in
> the parsing section. For example, if I want to read the binlog of
> canal-json.table.include = 'a' table, there is a source field of int in table
> a, but at this time if table b also has a source field of string, An error
> will be reported.
> !screenshot-1.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20885) Exception when use 'canal-json.table.include' to filter Canal binlog but table contains 'source' column
[ https://issues.apache.org/jira/browse/FLINK-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261054#comment-17261054 ] wangfei commented on FLINK-20885: - This is just a temporary solution for my production environment, looking forward to the community's update, Thanks. > Exception when use 'canal-json.table.include' to filter Canal binlog but > table contains 'source' column > --- > > Key: FLINK-20885 > URL: https://issues.apache.org/jira/browse/FLINK-20885 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / Ecosystem >Reporter: Jark Wu >Priority: Major > Attachments: image-2021-01-08-14-52-53-443.png, > image-2021-01-08-14-53-40-476.png, screenshot-1.png > > > I found a bug in the canal code. 'canal-json.table.include' does not filter > out the binlog of the specified table correctly, which will cause an error in > the parsing section. For example, if I want to read the binlog of > canal-json.table.include = 'a' table, there is a source field of int in table > a, but at this time if table b also has a source field of string, An error > will be reported. > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20885) Exception when use 'canal-json.table.include' to filter Canal binlog but table contains 'source' column
[ https://issues.apache.org/jira/browse/FLINK-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangfei updated FLINK-20885: Attachment: image-2021-01-08-14-53-40-476.png > Exception when use 'canal-json.table.include' to filter Canal binlog but > table contains 'source' column > --- > > Key: FLINK-20885 > URL: https://issues.apache.org/jira/browse/FLINK-20885 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / Ecosystem >Reporter: Jark Wu >Priority: Major > Attachments: image-2021-01-08-14-52-53-443.png, > image-2021-01-08-14-53-40-476.png, screenshot-1.png > > > I found a bug in the canal code. 'canal-json.table.include' does not filter > out the binlog of the specified table correctly, which will cause an error in > the parsing section. For example, if I want to read the binlog of > canal-json.table.include = 'a' table, there is a source field of int in table > a, but at this time if table b also has a source field of string, An error > will be reported. > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20885) Exception when use 'canal-json.table.include' to filter Canal binlog but table contains 'source' column
[ https://issues.apache.org/jira/browse/FLINK-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261052#comment-17261052 ] wangfei commented on FLINK-20885: - !image-2021-01-08-14-53-40-476.png! > Exception when use 'canal-json.table.include' to filter Canal binlog but > table contains 'source' column > --- > > Key: FLINK-20885 > URL: https://issues.apache.org/jira/browse/FLINK-20885 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / Ecosystem >Reporter: Jark Wu >Priority: Major > Attachments: image-2021-01-08-14-52-53-443.png, > image-2021-01-08-14-53-40-476.png, screenshot-1.png > > > I found a bug in the canal code. 'canal-json.table.include' does not filter > out the binlog of the specified table correctly, which will cause an error in > the parsing section. For example, if I want to read the binlog of > canal-json.table.include = 'a' table, there is a source field of int in table > a, but at this time if table b also has a source field of string, An error > will be reported. > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20885) Exception when use 'canal-json.table.include' to filter Canal binlog but table contains 'source' column
[ https://issues.apache.org/jira/browse/FLINK-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261051#comment-17261051 ] wangfei commented on FLINK-20885: - !image-2021-01-08-14-52-53-443.png! > Exception when use 'canal-json.table.include' to filter Canal binlog but > table contains 'source' column > --- > > Key: FLINK-20885 > URL: https://issues.apache.org/jira/browse/FLINK-20885 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / Ecosystem >Reporter: Jark Wu >Priority: Major > Attachments: image-2021-01-08-14-52-53-443.png, screenshot-1.png > > > I found a bug in the canal code. 'canal-json.table.include' does not filter > out the binlog of the specified table correctly, which will cause an error in > the parsing section. For example, if I want to read the binlog of > canal-json.table.include = 'a' table, there is a source field of int in table > a, but at this time if table b also has a source field of string, An error > will be reported. > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20885) Exception when use 'canal-json.table.include' to filter Canal binlog but table contains 'source' column
[ https://issues.apache.org/jira/browse/FLINK-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangfei updated FLINK-20885: Attachment: image-2021-01-08-14-52-53-443.png > Exception when use 'canal-json.table.include' to filter Canal binlog but > table contains 'source' column > --- > > Key: FLINK-20885 > URL: https://issues.apache.org/jira/browse/FLINK-20885 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / Ecosystem >Reporter: Jark Wu >Priority: Major > Attachments: image-2021-01-08-14-52-53-443.png, screenshot-1.png > > > I found a bug in the canal code. 'canal-json.table.include' does not filter > out the binlog of the specified table correctly, which will cause an error in > the parsing section. For example, if I want to read the binlog of > canal-json.table.include = 'a' table, there is a source field of int in table > a, but at this time if table b also has a source field of string, An error > will be reported. > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20885) Exception when use 'canal-json.table.include' to filter Canal binlog but table contains 'source' column
[ https://issues.apache.org/jira/browse/FLINK-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261050#comment-17261050 ] wangfei commented on FLINK-20885: - I have modified the code locally to fix this bug, and if you do, it is recommended that canal-json.database.include and canal-json.table.include support regular matching, and contain relationships rather than equal relationships. Filter isDdl=true for all binlogs that have an alter or create error. 我在本地修改了代码 来解决这个bug，如果你们来解决的话，建议canal-json.database.include 和 canal-json.table.include 应该支持正则匹配，应该是包含关系 而不是等于关系。并且如果有alter 或者create 等binlog也会报错，建议过滤掉isDdl=true的binlog。 > Exception when use 'canal-json.table.include' to filter Canal binlog but > table contains 'source' column > --- > > Key: FLINK-20885 > URL: https://issues.apache.org/jira/browse/FLINK-20885 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / Ecosystem >Reporter: Jark Wu >Priority: Major > Attachments: image-2021-01-08-14-52-53-443.png, screenshot-1.png > > > I found a bug in the canal code. 'canal-json.table.include' does not filter > out the binlog of the specified table correctly, which will cause an error in > the parsing section. For example, if I want to read the binlog of > canal-json.table.include = 'a' table, there is a source field of int in table > a, but at this time if table b also has a source field of string, An error > will be reported. > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (FLINK-20885) Exception when use 'canal-json.table.include' to filter Canal binlog but table contains 'source' column
[ https://issues.apache.org/jira/browse/FLINK-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangfei updated FLINK-20885: Comment: was deleted (was: [~jark] I've modified and tested it, but don't know how to participate in the community) > Exception when use 'canal-json.table.include' to filter Canal binlog but > table contains 'source' column > --- > > Key: FLINK-20885 > URL: https://issues.apache.org/jira/browse/FLINK-20885 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / Ecosystem >Reporter: Jark Wu >Priority: Major > Attachments: screenshot-1.png > > > I found a bug in the canal code. 'canal-json.table.include' does not filter > out the binlog of the specified table correctly, which will cause an error in > the parsing section. For example, if I want to read the binlog of > canal-json.table.include = 'a' table, there is a source field of int in table > a, but at this time if table b also has a source field of string, An error > will be reported. > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20885) Exception when use 'canal-json.table.include' to filter Canal binlog but table contains 'source' column
[ https://issues.apache.org/jira/browse/FLINK-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261031#comment-17261031 ] wangfei commented on FLINK-20885: - [~jark] I've modified and tested it, but don't know how to participate in the community > Exception when use 'canal-json.table.include' to filter Canal binlog but > table contains 'source' column > --- > > Key: FLINK-20885 > URL: https://issues.apache.org/jira/browse/FLINK-20885 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / Ecosystem >Reporter: Jark Wu >Priority: Major > Attachments: screenshot-1.png > > > I found a bug in the canal code. 'canal-json.table.include' does not filter > out the binlog of the specified table correctly, which will cause an error in > the parsing section. For example, if I want to read the binlog of > canal-json.table.include = 'a' table, there is a source field of int in table > a, but at this time if table b also has a source field of string, An error > will be reported. > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20885) Exception when use 'canal-json.table.include' to filter Canal binlog but table contains 'source' column
[ https://issues.apache.org/jira/browse/FLINK-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260988#comment-17260988 ] wangfei commented on FLINK-20885: - [~jark] I'm using Flink Master based compilation, probably 1.13. > Exception when use 'canal-json.table.include' to filter Canal binlog but > table contains 'source' column > --- > > Key: FLINK-20885 > URL: https://issues.apache.org/jira/browse/FLINK-20885 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / Ecosystem >Reporter: Jark Wu >Priority: Major > Attachments: screenshot-1.png > > > I found a bug in the canal code. 'canal-json.table.include' does not filter > out the binlog of the specified table correctly, which will cause an error in > the parsing section. For example, if I want to read the binlog of > canal-json.table.include = 'a' table, there is a source field of int in table > a, but at this time if table b also has a source field of string, An error > will be reported. > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20885) Exception when use 'canal-json.table.include' to filter Canal binlog but table contains 'source' column
[ https://issues.apache.org/jira/browse/FLINK-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260946#comment-17260946 ] wangfei commented on FLINK-20885: - [~jark] When is this expected to be fixed > Exception when use 'canal-json.table.include' to filter Canal binlog but > table contains 'source' column > --- > > Key: FLINK-20885 > URL: https://issues.apache.org/jira/browse/FLINK-20885 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / Ecosystem >Reporter: Jark Wu >Priority: Major > Attachments: screenshot-1.png > > > I found a bug in the canal code. 'canal-json.table.include' does not filter > out the binlog of the specified table correctly, which will cause an error in > the parsing section. For example, if I want to read the binlog of > canal-json.table.include = 'a' table, there is a source field of int in table > a, but at this time if table b also has a source field of string, An error > will be reported. > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20385) Allow to read metadata for Canal-json format
[ https://issues.apache.org/jira/browse/FLINK-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260402#comment-17260402 ] wangfei commented on FLINK-20385: - [~jark] 3q > Allow to read metadata for Canal-json format > > > Key: FLINK-20385 > URL: https://issues.apache.org/jira/browse/FLINK-20385 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Leonard Xu >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Attachments: image-2021-01-07-16-56-37-330.png, > image-2021-01-07-16-57-20-838.png > > > In FLIP-107, we support read meta from CDC format Debezium, Canal-json is also > another widely used CDC format , we need to support read metadata too. > > The requirement comes from user-zh mail list, the user want to read meta > information(database table name) from Canal-json. > [1] [http://apache-flink.147419.n8.nabble.com/canal-json-tt8939.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20385) Allow to read metadata for Canal-json format
[ https://issues.apache.org/jira/browse/FLINK-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangfei updated FLINK-20385: Attachment: image-2021-01-07-16-57-20-838.png > Allow to read metadata for Canal-json format > > > Key: FLINK-20385 > URL: https://issues.apache.org/jira/browse/FLINK-20385 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Leonard Xu >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Attachments: image-2021-01-07-16-56-37-330.png, > image-2021-01-07-16-57-20-838.png > > > In FLIP-107, we support read meta from CDC format Debezium, Canal-json is also > another widely used CDC format , we need to support read metadata too. > > The requirement comes from user-zh mail list, the user want to read meta > information(database table name) from Canal-json. > [1] [http://apache-flink.147419.n8.nabble.com/canal-json-tt8939.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20385) Allow to read metadata for Canal-json format
[ https://issues.apache.org/jira/browse/FLINK-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260328#comment-17260328 ] wangfei commented on FLINK-20385: - 在源码 这步会解析报错 !image-2021-01-07-16-57-20-838.png! > Allow to read metadata for Canal-json format > > > Key: FLINK-20385 > URL: https://issues.apache.org/jira/browse/FLINK-20385 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Leonard Xu >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Attachments: image-2021-01-07-16-56-37-330.png, > image-2021-01-07-16-57-20-838.png > > > In FLIP-107, we support read meta from CDC format Debezium, Canal-json is also > another widely used CDC format , we need to support read metadata too. > > The requirement comes from user-zh mail list, the user want to read meta > information(database table name) from Canal-json. > [1] [http://apache-flink.147419.n8.nabble.com/canal-json-tt8939.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20385) Allow to read metadata for Canal-json format
[ https://issues.apache.org/jira/browse/FLINK-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260326#comment-17260326 ] wangfei commented on FLINK-20385: - !image-2021-01-07-16-56-37-330.png! > Allow to read metadata for Canal-json format > > > Key: FLINK-20385 > URL: https://issues.apache.org/jira/browse/FLINK-20385 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Leonard Xu >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Attachments: image-2021-01-07-16-56-37-330.png > > > In FLIP-107, we support read meta from CDC format Debezium, Canal-json is also > another widely used CDC format , we need to support read metadata too. > > The requirement comes from user-zh mail list, the user want to read meta > information(database table name) from Canal-json. > [1] [http://apache-flink.147419.n8.nabble.com/canal-json-tt8939.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20385) Allow to read metadata for Canal-json format
[ https://issues.apache.org/jira/browse/FLINK-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260324#comment-17260324 ] wangfei commented on FLINK-20385: - 我发现了一个bug 在canal代码，在使用时候 配置’canal-json.table.include’ 并不能正确的过滤掉指定表的binlog ，在解析那步会报错，比如 我想读取 canal-json.table.include = 'a' 表的binlog，a表有一个source字段为int类型， 但是这个时候如果b表也有source字段，但是source字段是string类型，这个时候b的source存了一个非int的数据，这个时候会报错。 > Allow to read metadata for Canal-json format > > > Key: FLINK-20385 > URL: https://issues.apache.org/jira/browse/FLINK-20385 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Leonard Xu >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Attachments: image-2021-01-07-16-56-37-330.png > > > In FLIP-107, we support read meta from CDC format Debezium, Canal-json is also > another widely used CDC format , we need to support read metadata too. > > The requirement comes from user-zh mail list, the user want to read meta > information(database table name) from Canal-json. > [1] [http://apache-flink.147419.n8.nabble.com/canal-json-tt8939.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20385) Allow to read metadata for Canal-json format
[ https://issues.apache.org/jira/browse/FLINK-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangfei updated FLINK-20385: Attachment: image-2021-01-07-16-56-37-330.png > Allow to read metadata for Canal-json format > > > Key: FLINK-20385 > URL: https://issues.apache.org/jira/browse/FLINK-20385 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Leonard Xu >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Attachments: image-2021-01-07-16-56-37-330.png > > > In FLIP-107, we support read meta from CDC format Debezium, Canal-json is also > another widely used CDC format , we need to support read metadata too. > > The requirement comes from user-zh mail list, the user want to read meta > information(database table name) from Canal-json. > [1] [http://apache-flink.147419.n8.nabble.com/canal-json-tt8939.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20385) Allow to read metadata for Canal-json format
[ https://issues.apache.org/jira/browse/FLINK-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260323#comment-17260323 ] wangfei commented on FLINK-20385: - I found a bug in the canal code. 'canal-json.table.include' does not filter out the binlog of the specified table correctly, which will cause an error in the parsing section. For example, if I want to read the binlog of canal-json.table.include = 'a' table, there is a source field of int in table a, but at this time if table b also has a source field of string, An error will be reported. > Allow to read metadata for Canal-json format > > > Key: FLINK-20385 > URL: https://issues.apache.org/jira/browse/FLINK-20385 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Leonard Xu >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > In FLIP-107, we support read meta from CDC format Debezium, Canal-json is also > another widely used CDC format , we need to support read metadata too. > > The requirement comes from user-zh mail list, the user want to read meta > information(database table name) from Canal-json. > [1] [http://apache-flink.147419.n8.nabble.com/canal-json-tt8939.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20385) Allow to read metadata for Canal-json format
[ https://issues.apache.org/jira/browse/FLINK-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260322#comment-17260322 ] wangfei commented on FLINK-20385: - i try it > Allow to read metadata for Canal-json format > > > Key: FLINK-20385 > URL: https://issues.apache.org/jira/browse/FLINK-20385 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Leonard Xu >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > In FLIP-107, we support read meta from CDC format Debezium, Canal-json is also > another widely used CDC format , we need to support read metadata too. > > The requirement comes from user-zh mail list, the user want to read meta > information(database table name) from Canal-json. > [1] [http://apache-flink.147419.n8.nabble.com/canal-json-tt8939.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20385) Allow to read metadata for Canal-json format
[ https://issues.apache.org/jira/browse/FLINK-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17253298#comment-17253298 ] wangfei commented on FLINK-20385: - [~nicholasjiang] [~jark]Thank you. When did this get merged into the master > Allow to read metadata for Canal-json format > > > Key: FLINK-20385 > URL: https://issues.apache.org/jira/browse/FLINK-20385 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Leonard Xu >Assignee: Nicholas Jiang >Priority: Major > > In FLIP-107, we support read meta from CDC format Debezium, Canal-json is also > another widely used CDC format , we need to support read metadata too. > > The requirement comes from user-zh mail list, the user want to read meta > information(database table name) from Canal-json. > [1] [http://apache-flink.147419.n8.nabble.com/canal-json-tt8939.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20385) Allow to read metadata for Canal-json format
[ https://issues.apache.org/jira/browse/FLINK-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249429#comment-17249429 ] wangfei commented on FLINK-20385: - When will this version come online？ > Allow to read metadata for Canal-json format > > > Key: FLINK-20385 > URL: https://issues.apache.org/jira/browse/FLINK-20385 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Leonard Xu >Assignee: Nicholas Jiang >Priority: Major > > In FLIP-107, we support read meta from CDC format Debezium, Canal-json is also > another widely used CDC format , we need to support read metadata too. > > The requirement comes from user-zh mail list, the user want to read meta > information(database table name) from Canal-json. > [1] [http://apache-flink.147419.n8.nabble.com/canal-json-tt8939.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
