The final result looks good, any detailed design? I wonder how to implement it? We currently have inconsistent ways and locations for different connectors to obtain credentials, and I don't know how to accomplish this functionality in a way that is minimally invasive to existing code.
Guangdong Liu <[email protected]> 于2025年1月13日周一 11:58写道： > > Hey SeaTunnel Community, > > Currently, the catalog information for connectors is directly configured in > the configuration files. This approach has the following issues: > > 1. **Information Leakage**: Because these configurations are stored in > plain text, they can easily lead to access permission leaks, posing > security risks. > 2. **Difficult to Modify**: When database access information changes, all > related task configurations need to be updated individually, which is > time-consuming and error-prone. > > To address the above issues, we propose introducing a unified catalog > management layer to centralize and manage these configurations, enhancing > both security and maintainability. > > **Previous Configuration:** > > ```yaml > env { > job.mode = "BATCH" > } > > source { > Jdbc { > url = > "jdbc:mysql://localhost:3306/test?serverTimezone=GMT%2b8&useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true" > driver = "com.mysql.cj.jdbc.Driver" > connection_check_timeout_sec = 100 > user = "root" > password = "123456" > query = "select * from type_bin limit 16" > } > } > > transform { > } > > sink { > jdbc { > url = > "jdbc:mysql://localhost:3306/test?useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true" > driver = "com.mysql.cj.jdbc.Driver" > user = "root" > password = "123456" > query = "insert into test_table(name,age) values(?,?)" > } > } > ``` > > **Configuration Based on a Unified Catalog Layer (e.g., Apache Gravitino):** > > ```yaml > env { > job.mode = "BATCH" > catalog.kind = 'gravitino' > gravitino.uri=http://gravitino-server-host:8090 > gravitino.metalake=test > } > > source { > Jdbc { > catalog="mysql_test" > connection_check_timeout_sec = 100 > query = "select * from type_bin limit 16" > } > } > > transform { > } > > sink { > jdbc { > catalog="mysql_test" > query = "insert into test_table(name,age) values(?,?)" > } > } > ``` > > By introducing a catalog management layer (e.g., Apache Gravitino), we can > achieve the following improvements: > > - **Enhanced Security**: Sensitive configuration information is managed by > the unified catalog layer and is no longer exposed in configuration files. > - **Ease of Maintenance**: When database access information changes, > updates can be made in the catalog layer without modifying individual task > configurations. > > Please evaluate whether this solution fits our current requirements. If you > have any questions, feel free to reach out. > > > -- > > Best Regards > > ------------ > > Liugddx > [email protected]
