Thanks for all the replies. I will provide more detailed design documents in the future.
-- Best Regards ------------ Liugddx [email protected] hailin0 <[email protected]> 于2025年1月13日周一 17:07写道： > The results are great, but more design details are needed to show > feasibility > > hadoop.wiki <[email protected]> 于2025年1月13日周一 15:20写道： > > > It looks very 👍， At present, Seattunnel web independently manages this > > information, and how to use it on Seattunnel web also needs to be > > considered. > > > > > 2025年1月13日 下午3:11，Jia Fan <[email protected]> 写道： > > > > > > The final result looks good, any detailed design? I wonder how to > > > implement it? We currently have inconsistent ways and locations for > > > different connectors to obtain credentials, and I don't know how to > > > accomplish this functionality in a way that is minimally invasive to > > > existing code. > > > > > > Guangdong Liu <[email protected]> 于2025年1月13日周一 11:58写道： > > >> > > >> Hey SeaTunnel Community, > > >> > > >> Currently, the catalog information for connectors is directly > > configured in > > >> the configuration files. This approach has the following issues: > > >> > > >> 1. **Information Leakage**: Because these configurations are stored in > > >> plain text, they can easily lead to access permission leaks, posing > > >> security risks. > > >> 2. **Difficult to Modify**: When database access information changes, > > all > > >> related task configurations need to be updated individually, which is > > >> time-consuming and error-prone. > > >> > > >> To address the above issues, we propose introducing a unified catalog > > >> management layer to centralize and manage these configurations, > > enhancing > > >> both security and maintainability. > > >> > > >> **Previous Configuration:** > > >> > > >> ```yaml > > >> env { > > >> job.mode = "BATCH" > > >> } > > >> > > >> source { > > >> Jdbc { > > >> url = > > >> > > > "jdbc:mysql://localhost:3306/test?serverTimezone=GMT%2b8&useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true" > > >> driver = "com.mysql.cj.jdbc.Driver" > > >> connection_check_timeout_sec = 100 > > >> user = "root" > > >> password = "123456" > > >> query = "select * from type_bin limit 16" > > >> } > > >> } > > >> > > >> transform { > > >> } > > >> > > >> sink { > > >> jdbc { > > >> url = > > >> > > > "jdbc:mysql://localhost:3306/test?useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true" > > >> driver = "com.mysql.cj.jdbc.Driver" > > >> user = "root" > > >> password = "123456" > > >> query = "insert into test_table(name,age) values(?,?)" > > >> } > > >> } > > >> ``` > > >> > > >> **Configuration Based on a Unified Catalog Layer (e.g., Apache > > Gravitino):** > > >> > > >> ```yaml > > >> env { > > >> job.mode = "BATCH" > > >> catalog.kind = 'gravitino' > > >> gravitino.uri=http://gravitino-server-host:8090 > > >> gravitino.metalake=test > > >> } > > >> > > >> source { > > >> Jdbc { > > >> catalog="mysql_test" > > >> connection_check_timeout_sec = 100 > > >> query = "select * from type_bin limit 16" > > >> } > > >> } > > >> > > >> transform { > > >> } > > >> > > >> sink { > > >> jdbc { > > >> catalog="mysql_test" > > >> query = "insert into test_table(name,age) values(?,?)" > > >> } > > >> } > > >> ``` > > >> > > >> By introducing a catalog management layer (e.g., Apache Gravitino), we > > can > > >> achieve the following improvements: > > >> > > >> - **Enhanced Security**: Sensitive configuration information is > managed > > by > > >> the unified catalog layer and is no longer exposed in configuration > > files. > > >> - **Ease of Maintenance**: When database access information changes, > > >> updates can be made in the catalog layer without modifying individual > > task > > >> configurations. > > >> > > >> Please evaluate whether this solution fits our current requirements. > If > > you > > >> have any questions, feel free to reach out. > > >> > > >> > > >> -- > > >> > > >> Best Regards > > >> > > >> ------------ > > >> > > >> Liugddx > > >> [email protected] > > > > > > > >
