[
https://issues.apache.org/jira/browse/HUDI-8354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davis Zhang updated HUDI-8354:
------------------------------
Parent: (was: HUDI-8708)
Issue Type: Bug (was: Sub-task)
> Support completion time as checkpoint in S3Event/GcsEvent incremental source
> in Hudi Streamer
> ---------------------------------------------------------------------------------------------
>
> Key: HUDI-8354
> URL: https://issues.apache.org/jira/browse/HUDI-8354
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Shawn Chang
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.0.1
>
> Original Estimate: 20h
> Remaining Estimate: 20h
>
> Currently, S3Event/GcsEvent source uses `QueryInfo` to get time range for
> incremental queries.
> `QueryContext` uses completion time to get time range for incremental queries
> and is more ideal. But QueryContext misses some logic that are required by
> S3/GCSEventSource. These logic can be found under
> IncrSourceHelper#generateQueryInfo:
> https://github.com/apache/hudi/blob/622c8a37cafea903b59c66fd4d40c80380df8e94/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java#L158
> The goal is to:
> 1. migrate these logic to QueryContext
> 2. Have S3/GCSEvent source uses QueryContext instead
> 3. Remove QueryInfo
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
