The results are great, but more design details are needed to show feasibility
hadoop.wiki <[email protected]> 于2025年1月13日周一 15:20写道： > It looks very 👍， At present, Seattunnel web independently manages this > information, and how to use it on Seattunnel web also needs to be > considered. > > > 2025年1月13日 下午3:11，Jia Fan <[email protected]> 写道： > > > > The final result looks good, any detailed design? I wonder how to > > implement it? We currently have inconsistent ways and locations for > > different connectors to obtain credentials, and I don't know how to > > accomplish this functionality in a way that is minimally invasive to > > existing code. > > > > Guangdong Liu <[email protected]> 于2025年1月13日周一 11:58写道： > >> > >> Hey SeaTunnel Community, > >> > >> Currently, the catalog information for connectors is directly > configured in > >> the configuration files. This approach has the following issues: > >> > >> 1. **Information Leakage**: Because these configurations are stored in > >> plain text, they can easily lead to access permission leaks, posing > >> security risks. > >> 2. **Difficult to Modify**: When database access information changes, > all > >> related task configurations need to be updated individually, which is > >> time-consuming and error-prone. > >> > >> To address the above issues, we propose introducing a unified catalog > >> management layer to centralize and manage these configurations, > enhancing > >> both security and maintainability. > >> > >> **Previous Configuration:** > >> > >> ```yaml > >> env { > >> job.mode = "BATCH" > >> } > >> > >> source { > >> Jdbc { > >> url = > >> > "jdbc:mysql://localhost:3306/test?serverTimezone=GMT%2b8&useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true" > >> driver = "com.mysql.cj.jdbc.Driver" > >> connection_check_timeout_sec = 100 > >> user = "root" > >> password = "123456" > >> query = "select * from type_bin limit 16" > >> } > >> } > >> > >> transform { > >> } > >> > >> sink { > >> jdbc { > >> url = > >> > "jdbc:mysql://localhost:3306/test?useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true" > >> driver = "com.mysql.cj.jdbc.Driver" > >> user = "root" > >> password = "123456" > >> query = "insert into test_table(name,age) values(?,?)" > >> } > >> } > >> ``` > >> > >> **Configuration Based on a Unified Catalog Layer (e.g., Apache > Gravitino):** > >> > >> ```yaml > >> env { > >> job.mode = "BATCH" > >> catalog.kind = 'gravitino' > >> gravitino.uri=http://gravitino-server-host:8090 > >> gravitino.metalake=test > >> } > >> > >> source { > >> Jdbc { > >> catalog="mysql_test" > >> connection_check_timeout_sec = 100 > >> query = "select * from type_bin limit 16" > >> } > >> } > >> > >> transform { > >> } > >> > >> sink { > >> jdbc { > >> catalog="mysql_test" > >> query = "insert into test_table(name,age) values(?,?)" > >> } > >> } > >> ``` > >> > >> By introducing a catalog management layer (e.g., Apache Gravitino), we > can > >> achieve the following improvements: > >> > >> - **Enhanced Security**: Sensitive configuration information is managed > by > >> the unified catalog layer and is no longer exposed in configuration > files. > >> - **Ease of Maintenance**: When database access information changes, > >> updates can be made in the catalog layer without modifying individual > task > >> configurations. > >> > >> Please evaluate whether this solution fits our current requirements. If > you > >> have any questions, feel free to reach out. > >> > >> > >> -- > >> > >> Best Regards > >> > >> ------------ > >> > >> Liugddx > >> [email protected] > > > >
