[jira] [Commented] (FLINK-20385) Allow to read metadata for Canal-json format
[ https://issues.apache.org/jira/browse/FLINK-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260402#comment-17260402 ] wangfei commented on FLINK-20385: - [~jark] 3q > Allow to read metadata for Canal-json format > > > Key: FLINK-20385 > URL: https://issues.apache.org/jira/browse/FLINK-20385 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Leonard Xu >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Attachments: image-2021-01-07-16-56-37-330.png, > image-2021-01-07-16-57-20-838.png > > > In FLIP-107, we support read meta from CDC format Debezium, Canal-json is also > another widely used CDC format , we need to support read metadata too. > > The requirement comes from user-zh mail list, the user want to read meta > information(database table name) from Canal-json. > [1] [http://apache-flink.147419.n8.nabble.com/canal-json-tt8939.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20385) Allow to read metadata for Canal-json format
[ https://issues.apache.org/jira/browse/FLINK-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260376#comment-17260376 ] Jark Wu commented on FLINK-20385: - [~wangfeiair2324], I helped to create an issue for this problem: FLINK-20885. > Allow to read metadata for Canal-json format > > > Key: FLINK-20385 > URL: https://issues.apache.org/jira/browse/FLINK-20385 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Leonard Xu >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Attachments: image-2021-01-07-16-56-37-330.png, > image-2021-01-07-16-57-20-838.png > > > In FLIP-107, we support read meta from CDC format Debezium, Canal-json is also > another widely used CDC format , we need to support read metadata too. > > The requirement comes from user-zh mail list, the user want to read meta > information(database table name) from Canal-json. > [1] [http://apache-flink.147419.n8.nabble.com/canal-json-tt8939.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20385) Allow to read metadata for Canal-json format
[ https://issues.apache.org/jira/browse/FLINK-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260328#comment-17260328 ] wangfei commented on FLINK-20385: - 在源码 这步会解析报错 !image-2021-01-07-16-57-20-838.png! > Allow to read metadata for Canal-json format > > > Key: FLINK-20385 > URL: https://issues.apache.org/jira/browse/FLINK-20385 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Leonard Xu >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Attachments: image-2021-01-07-16-56-37-330.png, > image-2021-01-07-16-57-20-838.png > > > In FLIP-107, we support read meta from CDC format Debezium, Canal-json is also > another widely used CDC format , we need to support read metadata too. > > The requirement comes from user-zh mail list, the user want to read meta > information(database table name) from Canal-json. > [1] [http://apache-flink.147419.n8.nabble.com/canal-json-tt8939.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20385) Allow to read metadata for Canal-json format
[ https://issues.apache.org/jira/browse/FLINK-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260326#comment-17260326 ] wangfei commented on FLINK-20385: - !image-2021-01-07-16-56-37-330.png! > Allow to read metadata for Canal-json format > > > Key: FLINK-20385 > URL: https://issues.apache.org/jira/browse/FLINK-20385 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Leonard Xu >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Attachments: image-2021-01-07-16-56-37-330.png > > > In FLIP-107, we support read meta from CDC format Debezium, Canal-json is also > another widely used CDC format , we need to support read metadata too. > > The requirement comes from user-zh mail list, the user want to read meta > information(database table name) from Canal-json. > [1] [http://apache-flink.147419.n8.nabble.com/canal-json-tt8939.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20385) Allow to read metadata for Canal-json format
[ https://issues.apache.org/jira/browse/FLINK-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260324#comment-17260324 ] wangfei commented on FLINK-20385: - 我发现了一个bug 在canal代码，在使用时候 配置’canal-json.table.include’ 并不能正确的过滤掉指定表的binlog ，在解析那步会报错，比如 我想读取 canal-json.table.include = 'a' 表的binlog，a表有一个source字段为int类型， 但是这个时候如果b表也有source字段，但是source字段是string类型，这个时候b的source存了一个非int的数据，这个时候会报错。 > Allow to read metadata for Canal-json format > > > Key: FLINK-20385 > URL: https://issues.apache.org/jira/browse/FLINK-20385 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Leonard Xu >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Attachments: image-2021-01-07-16-56-37-330.png > > > In FLIP-107, we support read meta from CDC format Debezium, Canal-json is also > another widely used CDC format , we need to support read metadata too. > > The requirement comes from user-zh mail list, the user want to read meta > information(database table name) from Canal-json. > [1] [http://apache-flink.147419.n8.nabble.com/canal-json-tt8939.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20385) Allow to read metadata for Canal-json format
[ https://issues.apache.org/jira/browse/FLINK-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260323#comment-17260323 ] wangfei commented on FLINK-20385: - I found a bug in the canal code. 'canal-json.table.include' does not filter out the binlog of the specified table correctly, which will cause an error in the parsing section. For example, if I want to read the binlog of canal-json.table.include = 'a' table, there is a source field of int in table a, but at this time if table b also has a source field of string, An error will be reported. > Allow to read metadata for Canal-json format > > > Key: FLINK-20385 > URL: https://issues.apache.org/jira/browse/FLINK-20385 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Leonard Xu >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > In FLIP-107, we support read meta from CDC format Debezium, Canal-json is also > another widely used CDC format , we need to support read metadata too. > > The requirement comes from user-zh mail list, the user want to read meta > information(database table name) from Canal-json. > [1] [http://apache-flink.147419.n8.nabble.com/canal-json-tt8939.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20385) Allow to read metadata for Canal-json format
[ https://issues.apache.org/jira/browse/FLINK-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260322#comment-17260322 ] wangfei commented on FLINK-20385: - i try it > Allow to read metadata for Canal-json format > > > Key: FLINK-20385 > URL: https://issues.apache.org/jira/browse/FLINK-20385 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Leonard Xu >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > In FLIP-107, we support read meta from CDC format Debezium, Canal-json is also > another widely used CDC format , we need to support read metadata too. > > The requirement comes from user-zh mail list, the user want to read meta > information(database table name) from Canal-json. > [1] [http://apache-flink.147419.n8.nabble.com/canal-json-tt8939.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20385) Allow to read metadata for Canal-json format
[ https://issues.apache.org/jira/browse/FLINK-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256800#comment-17256800 ] Nicholas Jiang commented on FLINK-20385: [~wangfeiair2324], this has already been merge into master branch. Please check for it. > Allow to read metadata for Canal-json format > > > Key: FLINK-20385 > URL: https://issues.apache.org/jira/browse/FLINK-20385 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Leonard Xu >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > In FLIP-107, we support read meta from CDC format Debezium, Canal-json is also > another widely used CDC format , we need to support read metadata too. > > The requirement comes from user-zh mail list, the user want to read meta > information(database table name) from Canal-json. > [1] [http://apache-flink.147419.n8.nabble.com/canal-json-tt8939.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20385) Allow to read metadata for Canal-json format
[ https://issues.apache.org/jira/browse/FLINK-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17253298#comment-17253298 ] wangfei commented on FLINK-20385: - [~nicholasjiang] [~jark]Thank you. When did this get merged into the master > Allow to read metadata for Canal-json format > > > Key: FLINK-20385 > URL: https://issues.apache.org/jira/browse/FLINK-20385 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Leonard Xu >Assignee: Nicholas Jiang >Priority: Major > > In FLIP-107, we support read meta from CDC format Debezium, Canal-json is also > another widely used CDC format , we need to support read metadata too. > > The requirement comes from user-zh mail list, the user want to read meta > information(database table name) from Canal-json. > [1] [http://apache-flink.147419.n8.nabble.com/canal-json-tt8939.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20385) Allow to read metadata for Canal-json format
[ https://issues.apache.org/jira/browse/FLINK-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17250158#comment-17250158 ] Nicholas Jiang commented on FLINK-20385: [~wangfeiair2324], I would push the pull request this week. About the version, this could ask [~jark]. > Allow to read metadata for Canal-json format > > > Key: FLINK-20385 > URL: https://issues.apache.org/jira/browse/FLINK-20385 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Leonard Xu >Assignee: Nicholas Jiang >Priority: Major > > In FLIP-107, we support read meta from CDC format Debezium, Canal-json is also > another widely used CDC format , we need to support read metadata too. > > The requirement comes from user-zh mail list, the user want to read meta > information(database table name) from Canal-json. > [1] [http://apache-flink.147419.n8.nabble.com/canal-json-tt8939.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20385) Allow to read metadata for Canal-json format
[ https://issues.apache.org/jira/browse/FLINK-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249429#comment-17249429 ] wangfei commented on FLINK-20385: - When will this version come online？ > Allow to read metadata for Canal-json format > > > Key: FLINK-20385 > URL: https://issues.apache.org/jira/browse/FLINK-20385 > Project: Flink > Issue Type: Sub-task > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Leonard Xu >Assignee: Nicholas Jiang >Priority: Major > > In FLIP-107, we support read meta from CDC format Debezium, Canal-json is also > another widely used CDC format , we need to support read metadata too. > > The requirement comes from user-zh mail list, the user want to read meta > information(database table name) from Canal-json. > [1] [http://apache-flink.147419.n8.nabble.com/canal-json-tt8939.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20385) Allow to read metadata for Canal-json format
[ https://issues.apache.org/jira/browse/FLINK-20385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242228#comment-17242228 ] Nicholas Jiang commented on FLINK-20385: [~Leonard Xu], I could work for this issue which supports to allow read metadata for canal-json format. And in my opinion, should maxwell-json format also support to allow read metadata from the format? What's more, could you please assign this ticket to me? > Allow to read metadata for Canal-json format > > > Key: FLINK-20385 > URL: https://issues.apache.org/jira/browse/FLINK-20385 > Project: Flink > Issue Type: New Feature > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / API >Reporter: Leonard Xu >Priority: Major > > In FLIP-107, we support read meta from CDC format Debezium, Canal-json is also > another widely used CDC format , we need to support read metadata too. > > The requirement comes from user-zh mail list, the user want to read meta > information(database table name) from Canal-json. > [1] [http://apache-flink.147419.n8.nabble.com/canal-json-tt8939.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
