kaori-seasons commented on PR #1872: URL: https://github.com/apache/iggy/pull/1872#issuecomment-3071619573
@spetz Hi, dear spetz, I introduced a state manager to track the most recently fetched record ID in a persistent way, and added corresponding integration tests. Welcome your review. The purpose of the function change is as follows: ### Core Function - Optional state management: Enable/disable state management function through configuration - Multiple storage backends: Support file, Elasticsearch, Redis and other storage methods - Rich state information: Track processing progress, error statistics, performance indicators, etc. - Automatic state saving: Configurable automatic save interval - State recovery: Continue processing from the last position after restart #### Technical Implementation - SDK extension: Added StatefulSource and StatefulSink traits in iggy_connector_sdk - State storage abstraction: Defined StateStorage trait to support multiple storage implementations - ES connector enhancement: Extended ES connector to support state management - Configuration drive: Control state management behavior through configuration files #### State information - Processing status: Last polling time, total number of documents, number of polls, etc. - Cursor information: Last document ID, scroll ID, offset, etc. - Error tracking: Error count, last error information - Performance statistics: Processing time, number of bytes, success rate, etc. #### How to use - Configuration enablement: Add state management configuration in the configuration file - Automatic integration: The connector automatically loads and saves state - Manual control: Provide API for manual state management - Tool support: Provide tool functions such as state statistics and cleanup -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
