Re: [PR] [FLINK-34908][pipeline-connector][doris] Fix Mysql pipeline to doris will lost precision for timestamp [flink-cdc]
gong commented on PR #3207: URL: https://github.com/apache/flink-cdc/pull/3207#issuecomment-2159656572 > @gong would you like to open a PR for release-3.1 branch? ok, I will open the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34908][pipeline-connector][doris] Fix Mysql pipeline to doris will lost precision for timestamp [flink-cdc]
leonardBang commented on PR #3207: URL: https://github.com/apache/flink-cdc/pull/3207#issuecomment-2152195587 @gong would you like to open a PR for release-3.1 branch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34908][pipeline-connector][doris] Fix Mysql pipeline to doris will lost precision for timestamp [flink-cdc]
leonardBang merged PR #3207: URL: https://github.com/apache/flink-cdc/pull/3207 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34908][pipeline-connector][doris] Fix Mysql pipeline to doris will lost precision for timestamp [flink-cdc]
gong commented on PR #3207: URL: https://github.com/apache/flink-cdc/pull/3207#issuecomment-2121831876 @PatrickRen Hello, I rebase master. CI fail is not relation with the PR. Please help to rerun CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34908][pipeline-connector][doris] Fix Mysql pipeline to doris will lost precision for timestamp [flink-cdc]
gong commented on PR #3207: URL: https://github.com/apache/flink-cdc/pull/3207#issuecomment-2061042815 @PatrickRen Hello, Please help to rerun CI. CI fail is not relation with the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34908][pipeline-connector][doris] Fix Mysql pipeline to doris will lost precision for timestamp [flink-cdc]
gong commented on code in PR #3207: URL: https://github.com/apache/flink-cdc/pull/3207#discussion_r1568504940 ## flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-doris/src/test/java/org/apache/flink/cdc/connectors/doris/sink/DorisRowConverterTest.java: ## @@ -86,7 +113,9 @@ public void testExternalConvert() { row.add(converter.serialize(i, recordData)); } Assert.assertEquals( -"[true, 1.2, 1.2345, 1, 32, 64, 128, 2021-01-01 08:00:00, 2021-01-01, a, doris]", +"[true, 1.2, 1.2345, 1, 32, 64, 128, 2021-01-01 08:00:00.00, 2021-01-01, a, doris, 2021-01-01 " ++ "08:01:11.00, 2021-01-01 08:01:11.123000, 2021-01-01 08:01:11.123456, 2021-01-01 " ++ "16:01:11.00, 2021-01-01 16:01:11.123000, 2021-01-01 16:01:11.123456]", Review Comment: I set pipelineZoneId GMT+08:00 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34908][pipeline-connector][doris] Fix Mysql pipeline to doris will lost precision for timestamp [flink-cdc]
yuxiqian commented on code in PR #3207: URL: https://github.com/apache/flink-cdc/pull/3207#discussion_r1568485784 ## flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-doris/src/test/java/org/apache/flink/cdc/connectors/doris/sink/DorisRowConverterTest.java: ## @@ -86,7 +113,9 @@ public void testExternalConvert() { row.add(converter.serialize(i, recordData)); } Assert.assertEquals( -"[true, 1.2, 1.2345, 1, 32, 64, 128, 2021-01-01 08:00:00, 2021-01-01, a, doris]", +"[true, 1.2, 1.2345, 1, 32, 64, 128, 2021-01-01 08:00:00.00, 2021-01-01, a, doris, 2021-01-01 " ++ "08:01:11.00, 2021-01-01 08:01:11.123000, 2021-01-01 08:01:11.123456, 2021-01-01 " ++ "16:01:11.00, 2021-01-01 16:01:11.123000, 2021-01-01 16:01:11.123456]", Review Comment: This assertion will fail when running on any machine which doesn't use UTC+8 timezone. Maybe dynamically generating expected data based on local timezone? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34908][pipeline-connector][doris] Fix Mysql pipeline to doris will lost precision for timestamp [flink-cdc]
gong commented on code in PR #3207: URL: https://github.com/apache/flink-cdc/pull/3207#discussion_r1568450309 ## flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-doris/src/main/java/org/apache/flink/cdc/connectors/doris/sink/DorisRowConverter.java: ## @@ -103,7 +103,8 @@ static SerializationConverter createExternalConverter(DataType type, ZoneId pipe return (index, val) -> val.getTimestamp(index, DataTypeChecks.getPrecision(type)) .toLocalDateTime() - .format(DorisEventSerializer.DATE_TIME_FORMATTER); +.toString() +.replace('T', ' '); Review Comment: @yuxiqian ok -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34908][pipeline-connector][doris] Fix Mysql pipeline to doris will lost precision for timestamp [flink-cdc]
yuxiqian commented on code in PR #3207: URL: https://github.com/apache/flink-cdc/pull/3207#discussion_r1568134905 ## flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-doris/src/main/java/org/apache/flink/cdc/connectors/doris/sink/DorisRowConverter.java: ## @@ -103,7 +103,8 @@ static SerializationConverter createExternalConverter(DataType type, ZoneId pipe return (index, val) -> val.getTimestamp(index, DataTypeChecks.getPrecision(type)) .toLocalDateTime() - .format(DorisEventSerializer.DATE_TIME_FORMATTER); +.toString() +.replace('T', ' '); Review Comment: What about fixing `DorisEventSerializer.DATE_TIME_FORMATTER` instead of doing string-level operations? It should make code clearer and more efficient. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34908][pipeline-connector][doris] Fix Mysql pipeline to doris will lost precision for timestamp [flink-cdc]
gong commented on PR #3207: URL: https://github.com/apache/flink-cdc/pull/3207#issuecomment-2041406982 > I wonder if changing `DorisEventSerializer.DATE_TIME_FORMATTER` to `-MM-dd HH:mm:ss.SS` fixes this problem, too? @yuxiqian I think that it can fix this problem, too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34908][pipeline-connector][doris] Fix Mysql pipeline to doris will lost precision for timestamp [flink-cdc]
yuxiqian commented on PR #3207: URL: https://github.com/apache/flink-cdc/pull/3207#issuecomment-2041372727 I wonder if changing `DorisEventSerializer.DATE_TIME_FORMATTER` to `-MM-dd HH:mm:ss.SS` would fix this problem, too? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34908][pipeline-connector][doris] Fix Mysql pipeline to doris will lost precision for timestamp [flink-cdc]
gong commented on PR #3207: URL: https://github.com/apache/flink-cdc/pull/3207#issuecomment-2041325325 @leonardBang PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org