Hey SeaTunnel Community,
Currently, the catalog information for connectors is directly configured in
the configuration files. This approach has the following issues:
1. **Information Leakage**: Because these configurations are stored in
plain text, they can easily lead to access permission leaks, posing
security risks.
2. **Difficult to Modify**: When database access information changes, all
related task configurations need to be updated individually, which is
time-consuming and error-prone.
To address the above issues, we propose introducing a unified catalog
management layer to centralize and manage these configurations, enhancing
both security and maintainability.
**Previous Configuration:**
```yaml
env {
job.mode = "BATCH"
}
source {
Jdbc {
url =
"jdbc:mysql://localhost:3306/test?serverTimezone=GMT%2b8&useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true"
driver = "com.mysql.cj.jdbc.Driver"
connection_check_timeout_sec = 100
user = "root"
password = "123456"
query = "select * from type_bin limit 16"
}
}
transform {
}
sink {
jdbc {
url =
"jdbc:mysql://localhost:3306/test?useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true"
driver = "com.mysql.cj.jdbc.Driver"
user = "root"
password = "123456"
query = "insert into test_table(name,age) values(?,?)"
}
}
```
**Configuration Based on a Unified Catalog Layer (e.g., Apache Gravitino):**
```yaml
env {
job.mode = "BATCH"
catalog.kind = 'gravitino'
gravitino.uri=http://gravitino-server-host:8090
gravitino.metalake=test
}
source {
Jdbc {
catalog="mysql_test"
connection_check_timeout_sec = 100
query = "select * from type_bin limit 16"
}
}
transform {
}
sink {
jdbc {
catalog="mysql_test"
query = "insert into test_table(name,age) values(?,?)"
}
}
```
By introducing a catalog management layer (e.g., Apache Gravitino), we can
achieve the following improvements:
- **Enhanced Security**: Sensitive configuration information is managed by
the unified catalog layer and is no longer exposed in configuration files.
- **Ease of Maintenance**: When database access information changes,
updates can be made in the catalog layer without modifying individual task
configurations.
Please evaluate whether this solution fits our current requirements. If you
have any questions, feel free to reach out.
--
Best Regards
------------
Liugddx
[email protected]
