This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 52d25f41a4 [feature](multi-catalog) Rename multi-catalog config
'specified_database_list' to 'include_database_list', and introduce new
multi-catalog config 'exclude_database_list' (#18834)
52d25f41a4 is described below
commit 52d25f41a44cc5470b6658e33cb1db5b67858ff5
Author: DuRipeng <[email protected]>
AuthorDate: Thu May 4 09:30:02 2023 +0800
[feature](multi-catalog) Rename multi-catalog config
'specified_database_list' to 'include_database_list', and introduce new
multi-catalog config 'exclude_database_list' (#18834)
In my scene, We need to specify databases that are excluded to synchronize
to doris,
like some databases store temporary table.
Since #17803 introduce `specified_database_list` to specify 'include
databases',
this pr introduce new config `exclude_database_list` to specify 'exclude
databases',
and rename `specified_database_list` to `include_database_list` for naming
symmetry.
BTW, when `include_database_list` and `exclude_database_list` specify
overlapping databases, `exclude_database_list` would take effect with higher
privilege over `include_database_list`.
---
docs/en/docs/lakehouse/multi-catalog/hive.md | 5 ---
docs/en/docs/lakehouse/multi-catalog/iceberg.md | 5 ---
docs/en/docs/lakehouse/multi-catalog/jdbc.md | 15 +++++--
.../docs/lakehouse/multi-catalog/multi-catalog.md | 12 +++++
docs/zh-CN/docs/lakehouse/multi-catalog/hive.md | 5 ---
docs/zh-CN/docs/lakehouse/multi-catalog/iceberg.md | 5 ---
docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md | 37 +++++++++-------
.../docs/lakehouse/multi-catalog/multi-catalog.md | 12 +++++
.../org/apache/doris/catalog/JdbcResource.java | 13 +++---
.../java/org/apache/doris/catalog/Resource.java | 3 +-
.../apache/doris/datasource/ExternalCatalog.java | 12 ++++-
.../doris/datasource/HMSExternalCatalog.java | 9 +++-
.../doris/datasource/JdbcExternalCatalog.java | 9 ++--
.../datasource/iceberg/IcebergExternalCatalog.java | 9 +++-
.../org/apache/doris/external/jdbc/JdbcClient.java | 27 ++++++++----
.../jdbc_catalog_p0/test_mysql_jdbc_catalog.out | 13 +++++-
.../data/jdbc_catalog_p0/test_pg_jdbc_catalog.out | 13 +++++-
.../jdbc_catalog_p0/test_mysql_jdbc_catalog.groovy | 51 +++++++++++++++++++---
.../jdbc_catalog_p0/test_pg_jdbc_catalog.groovy | 45 +++++++++++++++++--
19 files changed, 222 insertions(+), 78 deletions(-)
diff --git a/docs/en/docs/lakehouse/multi-catalog/hive.md
b/docs/en/docs/lakehouse/multi-catalog/hive.md
index 11ba2ecc1f..f82df22ab7 100644
--- a/docs/en/docs/lakehouse/multi-catalog/hive.md
+++ b/docs/en/docs/lakehouse/multi-catalog/hive.md
@@ -67,11 +67,6 @@ CREATE CATALOG hive PROPERTIES (
In addition to `type` and `hive.metastore.uris` , which are required, you
can specify other parameters regarding the connection.
-> `specified_database_list`:
->
-> only synchronize the specified databases, split with ','. Default values is
'' will synchronize all databases. db name is case sensitive.
->
-
For example, to specify HDFS HA:
```sql
diff --git a/docs/en/docs/lakehouse/multi-catalog/iceberg.md
b/docs/en/docs/lakehouse/multi-catalog/iceberg.md
index 91af94462e..f4452a95a5 100644
--- a/docs/en/docs/lakehouse/multi-catalog/iceberg.md
+++ b/docs/en/docs/lakehouse/multi-catalog/iceberg.md
@@ -61,11 +61,6 @@ CREATE CATALOG iceberg PROPERTIES (
);
```
-> `specified_database_list`:
->
-> only synchronize the specified databases, split with ','. Default values is
'' will synchronize all databases. db name is case sensitive.
->
-
### Iceberg Native Catalog
<version since="dev">
diff --git a/docs/en/docs/lakehouse/multi-catalog/jdbc.md
b/docs/en/docs/lakehouse/multi-catalog/jdbc.md
index 83bc625fb2..3c2606194f 100644
--- a/docs/en/docs/lakehouse/multi-catalog/jdbc.md
+++ b/docs/en/docs/lakehouse/multi-catalog/jdbc.md
@@ -202,6 +202,7 @@ When Trino is mapped, Doris's Database corresponds to a
Schema in Trino that spe
9. OceanBase
+<<<<<<< HEAD
<version since="dev"></version>
```sql
@@ -237,8 +238,9 @@ CREATE CATALOG jdbc_oceanbase_oracle PROPERTIES (
| `driver_class ` | Yes | | JDBC Driver
Class |
| `only_specified_database` | No | "false" | Whether only
the database specified to be synchronized. |
| `lower_case_table_names` | No | "false" | Whether to
synchronize jdbc external data source table names in lower case. |
-| `specified_database_list` | No | "" | When
only_specified_database=true，only synchronize the specified databases. split
with ','. db name is case sensitive.|
| `oceanbase_mode` | No | "" | When the
connected external data source is OceanBase, the mode must be specified as
mysql or oracle |
+| `include_database_list` | No | "" | When
only_specified_database=true，only synchronize the specified databases. split
with ','. db name is case sensitive. |
+| `exclude_database_list` | No | "" | When
only_specified_database=true，do not synchronize the specified databases. split
with ','. db name is case sensitive. |
> `driver_url` can be specified in three ways:
>
> 1. File name. For example, `mysql-connector-java-5.1.47.jar`. Please place
> the Jar file package in `jdbc_drivers/` under the FE/BE deployment
> directory in advance so the system can locate the file. You can change the
> location of the file by modifying `jdbc_drivers_dir` in fe.conf and
> be.conf.
@@ -248,9 +250,16 @@ CREATE CATALOG jdbc_oceanbase_oracle PROPERTIES (
> 3. HTTP address. For example,
> `https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/jdbc_driver/mysql-connector-java-5.1.47.jar`.
> The system will download the Driver file from the HTTP address. This only
> supports HTTP services with no authentication requirements.
> `only_specified_database`:
+> When the JDBC is connected, you can specify which database/schema to
connect. For example, you can specify the DataBase in mysql `jdbc_url`; you can
specify the CurrentSchema in PG `jdbc_url`.
+>
+> `include_database_list`:
+> When `only_specified_database=true`, only synchronize the specified
databases. split with ',', default value is '', means no filter takes effect,
synchronizes all databases. db name is case sensitive.
+>
+> `exclude_database_list`:
+> When `only_specified_database=true`, specify databases that do not need to
synchronize. split with ',', default value is '', means no filter takes effect,
synchronizes all databases. db name is case sensitive.
+>
+> When `include_database_list` and `exclude_database_list` specify overlapping
databases, `exclude_database_list` would take effect with higher privilege over
`include_database_list`.
>
-> When the JDBC is connected, you can specify which database/schema to
connect. For example, you can specify the DataBase in mysql `jdbc_url`; you can
specify the CurrentSchema in PG `jdbc_url`. When `only_specified_database=true`
and `specified_database_list` is empty, only the database in jdbc_url specified
to be synchronized. When `only_specified_database=true` and
`specified_database_list` with some database names，and these names will
specified to be synchronized。
->
> If you connect the Oracle database when using this property, please use the
> version of the jar package above 8 or more (such as ojdbc8.jar).
diff --git a/docs/en/docs/lakehouse/multi-catalog/multi-catalog.md
b/docs/en/docs/lakehouse/multi-catalog/multi-catalog.md
index e0a6244bc0..92de24b727 100644
--- a/docs/en/docs/lakehouse/multi-catalog/multi-catalog.md
+++ b/docs/en/docs/lakehouse/multi-catalog/multi-catalog.md
@@ -301,6 +301,18 @@ Access from Doris to databases and tables in an External
Catalog is not under th
Along with the new Multi-Catalog feature, we also added privilege management
at the Catalog level (See [Privilege
Management](https://doris.apache.org/docs/dev/admin-manual/privilege-ldap/user-privilege/)
for details).
+## Database synchronizing management
+
+Setting `include_database_list` and `exclude_database_list` in Catalog
properties to specify databases to synchronize.
+
+`include_database_list`: Only synchronize the specified databases. split with
',', default value is '', means no filter takes effect, synchronizes all
databases. db name is case sensitive.
+
+`exclude_database_list`: Specify databases that do not need to synchronize.
split with ',', default value is '', means no filter takes effect, synchronizes
all databases. db name is case sensitive.
+
+> When `include_database_list` and `exclude_database_list` specify overlapping
databases, `exclude_database_list` would take effect with higher privilege over
`include_database_list`.
+>
+> To connect JDBC, these two properties should work with
`only_specified_database`, see [JDBC](./jdbc.md) for more detail.
+
## Metadata Update
### Manual Update
diff --git a/docs/zh-CN/docs/lakehouse/multi-catalog/hive.md
b/docs/zh-CN/docs/lakehouse/multi-catalog/hive.md
index 456d16cd34..98d32b8631 100644
--- a/docs/zh-CN/docs/lakehouse/multi-catalog/hive.md
+++ b/docs/zh-CN/docs/lakehouse/multi-catalog/hive.md
@@ -64,11 +64,6 @@ CREATE CATALOG hive PROPERTIES (
```
除了 `type` 和 `hive.metastore.uris` 两个必须参数外，还可以通过更多参数来传递连接所需要的信息。
-
-> `specified_database_list`:
->
-> 支持只同步指定的同步多个database，以','分隔。默认为''，同步所有database。db名称是大小写敏感的。
->
如提供 HDFS HA 信息，示例如下：
diff --git a/docs/zh-CN/docs/lakehouse/multi-catalog/iceberg.md
b/docs/zh-CN/docs/lakehouse/multi-catalog/iceberg.md
index 87f1ff429d..0cf10e23fe 100644
--- a/docs/zh-CN/docs/lakehouse/multi-catalog/iceberg.md
+++ b/docs/zh-CN/docs/lakehouse/multi-catalog/iceberg.md
@@ -59,11 +59,6 @@ CREATE CATALOG iceberg PROPERTIES (
);
```
-> `specified_database_list`:
->
-> 支持只同步指定的同步多个database，以','分隔。默认为''，同步所有database。db名称是大小写敏感的。
->
-
### 基于Iceberg API创建Catalog
<version since="dev">
diff --git a/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
b/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
index b1671e6b62..78292960ff 100644
--- a/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
+++ b/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
@@ -229,18 +229,18 @@ CREATE CATALOG jdbc_oceanbase_oracle PROPERTIES (
### 参数说明
-| 参数 | 是否必须 | 默认值 | 说明
|
-|---------------------------|----------|-----------|-------------------------------------------------------------------
|
-| `user` | 是 | | 对应数据库的用户名
|
-| `password` | 是 | | 对应数据库的密码
|
-| `jdbc_url` | 是 | | JDBC 连接串
|
-| `driver_url` | 是 | | JDBC Driver Jar 包名称*
|
-| `driver_class` | 是 | | JDBC Driver Class 名称
|
-| `only_specified_database` | 否 | "false" | 指定是否只同步指定的 database
|
-| `lower_case_table_names` | 否 | "false" | 是否以小写的形式同步jdbc外部数据源的表名
|
-| `specified_database_list` | 否 | "" |
当only_specified_database=true时，指定同步多个database，以','分隔。db名称是大小写敏感的。 |
-| `oceanbase_mode` | 否 | "" |
当连接的外部数据源为OceanBase时，必须为其指定模式为mysql或oracle |
-
+| 参数 | 是否必须 | 默认值 | 说明
|
+|---------------------------|------|---------|-------------------------------------------------------------------
|
+| `user` | 是 | | 对应数据库的用户名
|
+| `password` | 是 | | 对应数据库的密码
|
+| `jdbc_url` | 是 | | JDBC 连接串
|
+| `driver_url` | 是 | | JDBC Driver Jar 包名称*
|
+| `driver_class` | 是 | | JDBC Driver Class 名称
|
+| `only_specified_database` | 否 | "false" | 指定是否只同步指定的 database
|
+| `lower_case_table_names` | 否 | "false" | 是否以小写的形式同步jdbc外部数据源的表名
|
+| `oceanbase_mode` | 否 | "" |
当连接的外部数据源为OceanBase时，必须为其指定模式为mysql或oracle |
+| `include_database_list` | 否 | "" |
当only_specified_database=true时，指定同步多个database，以','分隔。db名称是大小写敏感的。 |
+| `exclude_database_list` | 否 | "" |
当only_specified_database=true时，指定不需要同步的多个database，以','分割。db名称是大小写敏感的。|
> `driver_url` 可以通过以下三种方式指定：
>
@@ -251,9 +251,16 @@ CREATE CATALOG jdbc_oceanbase_oracle PROPERTIES (
> 3. Http
> 地址。如：`https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/jdbc_driver/mysql-connector-java-5.1.47.jar`。系统会从这个
> http 地址下载 Driver 文件。仅支持无认证的 http 服务。
> `only_specified_database`:
->
-> 在jdbc连接时可以指定链接到哪个database/schema, 如：mysql中jdbc_url中可以指定database,
pg的jdbc_url中可以指定currentSchema。`only_specified_database=true`
且`specified_database_list`为空时，可以只同步指定的
database。当`only_specified_database=true`且`specified_database_list`指定了database列表时，则会同步指定的多个database。
->
+> 在jdbc连接时可以指定链接到哪个database/schema, 如：mysql中jdbc_url中可以指定database,
pg的jdbc_url中可以指定currentSchema。
+>
+> `include_database_list`:
+> 当`only_specified_database=true`时，指定需要同步的
database，以','分割。默认为''，即不做任何过滤，同步所有database。db名称是大小写敏感的
+>
+> `exclude_database_list`:
+>
当`only_specified_database=true`时，指定不需要同步的多个database，以','分割。默认为''，即不做任何过滤，同步所有database。db名称是大小写敏感的。
+>
+> 当 `include_database_list` 和 `exclude_database_list`
有重合的database配置时，`exclude_database_list`会优先生效。
+>
> 如果使用该参数时连接oracle数据库，要求使用ojdbc8.jar以上版本jar包。
## 数据查询
diff --git a/docs/zh-CN/docs/lakehouse/multi-catalog/multi-catalog.md
b/docs/zh-CN/docs/lakehouse/multi-catalog/multi-catalog.md
index 2025ffaefa..39e78f9d78 100644
--- a/docs/zh-CN/docs/lakehouse/multi-catalog/multi-catalog.md
+++ b/docs/zh-CN/docs/lakehouse/multi-catalog/multi-catalog.md
@@ -303,6 +303,18 @@ select k1, k4 from table; // Query OK.
Doris 的权限管理功能提供了对 Catalog 层级的扩展，具体可参阅
[权限管理](../../admin-manual/privilege-ldap/user-privilege.md) 文档。
+## 指定需要同步的数据库
+
+通过在 Catalog 配置中设置 `include_database_list` 和 `exclude_database_list`
可以指定需要同步的数据库。
+
+`include_database_list`:
支持只同步指定的多个database，以','分隔。默认为''，同步所有database。db名称是大小写敏感的。
+
+`exclude_database_list`:
支持指定不需要同步的多个database，以','分割。默认为''，即不做任何过滤，同步所有database。db名称是大小写敏感的。
+
+> 当 `include_database_list` 和 `exclude_database_list`
有重合的database配置时，`exclude_database_list`会优先生效。
+>
+> 连接 JDBC 时，上述 2 个配置需要和配置 `only_specified_database` 搭配使用，详见 [JDBC](./jdbc.md)
+
## 元数据更新
### 手动刷新
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java
index 536744b724..3220b69b58 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java
@@ -101,14 +101,16 @@ public class JdbcResource extends Resource {
TYPE,
ONLY_SPECIFIED_DATABASE,
LOWER_CASE_TABLE_NAMES,
- SPECIFIED_DATABASE_LIST,
- OCEANBASE_MODE
+ OCEANBASE_MODE,
+ INCLUDE_DATABASE_LIST,
+ EXCLUDE_DATABASE_LIST
).build();
private static final ImmutableList<String> OPTIONAL_PROPERTIES = new
ImmutableList.Builder<String>().add(
ONLY_SPECIFIED_DATABASE,
LOWER_CASE_TABLE_NAMES,
- SPECIFIED_DATABASE_LIST,
- OCEANBASE_MODE
+ OCEANBASE_MODE,
+ INCLUDE_DATABASE_LIST,
+ EXCLUDE_DATABASE_LIST
).build();
// The default value of optional properties
@@ -118,8 +120,9 @@ public class JdbcResource extends Resource {
static {
OPTIONAL_PROPERTIES_DEFAULT_VALUE.put(ONLY_SPECIFIED_DATABASE,
"false");
OPTIONAL_PROPERTIES_DEFAULT_VALUE.put(LOWER_CASE_TABLE_NAMES, "false");
- OPTIONAL_PROPERTIES_DEFAULT_VALUE.put(SPECIFIED_DATABASE_LIST, "");
OPTIONAL_PROPERTIES_DEFAULT_VALUE.put(OCEANBASE_MODE, "");
+ OPTIONAL_PROPERTIES_DEFAULT_VALUE.put(INCLUDE_DATABASE_LIST, "");
+ OPTIONAL_PROPERTIES_DEFAULT_VALUE.put(EXCLUDE_DATABASE_LIST, "");
}
// timeout for both connection and read. 10 seconds is long enough.
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Resource.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/Resource.java
index 6c39553926..781c5fb3f6 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Resource.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Resource.java
@@ -46,7 +46,8 @@ import java.util.stream.Collectors;
public abstract class Resource implements Writable, GsonPostProcessable {
private static final Logger LOG =
LogManager.getLogger(OdbcCatalogResource.class);
public static final String REFERENCE_SPLIT = "@";
- public static final String SPECIFIED_DATABASE_LIST =
"specified_database_list";
+ public static final String INCLUDE_DATABASE_LIST = "include_database_list";
+ public static final String EXCLUDE_DATABASE_LIST = "exclude_database_list";
public enum ResourceType {
UNKNOWN,
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
index bcf62b5e49..1c5c4b4944 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
@@ -449,8 +449,16 @@ public abstract class ExternalCatalog implements
CatalogIf<ExternalDatabase>, Wr
throw new NotImplementedException("createDatabase not implemented");
}
- public Map getSpecifiedDatabaseMap() {
- String specifiedDatabaseList =
catalogProperty.getOrDefault(Resource.SPECIFIED_DATABASE_LIST, "");
+ public Map getIncludeDatabaseMap() {
+ return getSpecifiedDatabaseMap(Resource.INCLUDE_DATABASE_LIST);
+ }
+
+ public Map getExcludeDatabaseMap() {
+ return getSpecifiedDatabaseMap(Resource.EXCLUDE_DATABASE_LIST);
+ }
+
+ public Map getSpecifiedDatabaseMap(String catalogPropertyKey) {
+ String specifiedDatabaseList =
catalogProperty.getOrDefault(catalogPropertyKey, "");
Map<String, Boolean> specifiedDatabaseMap = Maps.newHashMap();
specifiedDatabaseList = specifiedDatabaseList.trim();
if (specifiedDatabaseList.isEmpty()) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/HMSExternalCatalog.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/HMSExternalCatalog.java
index 40054f4af2..6a359f3398 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/HMSExternalCatalog.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/HMSExternalCatalog.java
@@ -130,10 +130,15 @@ public class HMSExternalCatalog extends ExternalCatalog {
initCatalogLog.setCatalogId(id);
initCatalogLog.setType(InitCatalogLog.Type.HMS);
List<String> allDatabases = client.getAllDatabases();
- Map<String, Boolean> specifiedDatabaseMap = getSpecifiedDatabaseMap();
+ Map<String, Boolean> includeDatabaseMap = getIncludeDatabaseMap();
+ Map<String, Boolean> excludeDatabaseMap = getExcludeDatabaseMap();
// Update the db name to id map.
for (String dbName : allDatabases) {
- if (!specifiedDatabaseMap.isEmpty() &&
specifiedDatabaseMap.get(dbName) == null) {
+ // Exclude database map take effect with higher priority over
include database map
+ if (!excludeDatabaseMap.isEmpty() &&
excludeDatabaseMap.containsKey(dbName)) {
+ continue;
+ }
+ if (!includeDatabaseMap.isEmpty() &&
includeDatabaseMap.containsKey(dbName)) {
continue;
}
long dbId;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/JdbcExternalCatalog.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/JdbcExternalCatalog.java
index 256737197d..3ce9a1ad94 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/JdbcExternalCatalog.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/JdbcExternalCatalog.java
@@ -129,18 +129,15 @@ public class JdbcExternalCatalog extends ExternalCatalog {
return
catalogProperty.getOrDefault(JdbcResource.LOWER_CASE_TABLE_NAMES, "false");
}
- public String getSpecifiedDatabaseList() {
- return
catalogProperty.getOrDefault(JdbcResource.SPECIFIED_DATABASE_LIST, "");
- }
-
public String getOceanBaseMode() {
return catalogProperty.getOrDefault(JdbcResource.OCEANBASE_MODE, "");
}
@Override
protected void initLocalObjectsImpl() {
- jdbcClient = new JdbcClient(getJdbcUser(), getJdbcPasswd(),
getJdbcUrl(), getDriverUrl(), getDriverClass(),
- getOnlySpecifiedDatabase(), getLowerCaseTableNames(),
getSpecifiedDatabaseMap(), getOceanBaseMode());
+ jdbcClient = new JdbcClient(getJdbcUser(), getJdbcPasswd(),
getJdbcUrl(), getDriverUrl(),
+ getDriverClass(), getOnlySpecifiedDatabase(),
getLowerCaseTableNames(),
+ getOceanBaseMode(), getIncludeDatabaseMap(),
getExcludeDatabaseMap());
}
@Override
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalCatalog.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalCatalog.java
index d7846a9012..22aa3bf5c6 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalCatalog.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalCatalog.java
@@ -67,9 +67,14 @@ public abstract class IcebergExternalCatalog extends
ExternalCatalog {
initCatalogLog.setCatalogId(id);
initCatalogLog.setType(InitCatalogLog.Type.ICEBERG);
List<String> allDatabaseNames = listDatabaseNames();
- Map<String, Boolean> specifiedDatabaseMap = getSpecifiedDatabaseMap();
+ Map<String, Boolean> includeDatabaseMap = getIncludeDatabaseMap();
+ Map<String, Boolean> excludeDatabaseMap = getExcludeDatabaseMap();
for (String dbName : allDatabaseNames) {
- if (!specifiedDatabaseMap.isEmpty() &&
specifiedDatabaseMap.get(dbName) == null) {
+ // Exclude database map take effect with higher priority over
include database map
+ if (!excludeDatabaseMap.isEmpty() &&
excludeDatabaseMap.containsKey(dbName)) {
+ continue;
+ }
+ if (!includeDatabaseMap.isEmpty() &&
includeDatabaseMap.containsKey(dbName)) {
continue;
}
long dbId;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/external/jdbc/JdbcClient.java
b/fe/fe-core/src/main/java/org/apache/doris/external/jdbc/JdbcClient.java
index bcab8f09ef..8c02d48367 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/external/jdbc/JdbcClient.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/external/jdbc/JdbcClient.java
@@ -61,7 +61,8 @@ public class JdbcClient {
private boolean isLowerCaseTableNames = false;
- private Map<String, Boolean> specifiedDatabaseMap = Maps.newHashMap();
+ private Map<String, Boolean> includeDatabaseMap = Maps.newHashMap();
+ private Map<String, Boolean> excludeDatabaseMap = Maps.newHashMap();
// only used when isLowerCaseTableNames = true.
private Map<String, String> lowerTableToRealTable = Maps.newHashMap();
@@ -69,13 +70,16 @@ public class JdbcClient {
private String oceanbaseMode = "";
public JdbcClient(String user, String password, String jdbcUrl, String
driverUrl, String driverClass,
- String onlySpecifiedDatabase, String isLowerCaseTableNames, Map
specifiedDatabaseMap,
- String oceanbaseMode) {
+ String onlySpecifiedDatabase, String isLowerCaseTableNames, String
oceanbaseMode, Map includeDatabaseMap,
+ Map excludeDatabaseMap) {
this.jdbcUser = user;
this.isOnlySpecifiedDatabase =
Boolean.valueOf(onlySpecifiedDatabase).booleanValue();
this.isLowerCaseTableNames =
Boolean.valueOf(isLowerCaseTableNames).booleanValue();
- if (specifiedDatabaseMap != null) {
- this.specifiedDatabaseMap = specifiedDatabaseMap;
+ if (includeDatabaseMap != null) {
+ this.includeDatabaseMap = includeDatabaseMap;
+ }
+ if (excludeDatabaseMap != null) {
+ this.excludeDatabaseMap = excludeDatabaseMap;
}
this.oceanbaseMode = oceanbaseMode;
try {
@@ -180,7 +184,7 @@ public class JdbcClient {
Connection conn = getConnection();
Statement stmt = null;
ResultSet rs = null;
- if (isOnlySpecifiedDatabase && specifiedDatabaseMap.isEmpty()) {
+ if (isOnlySpecifiedDatabase && includeDatabaseMap.isEmpty() &&
excludeDatabaseMap.isEmpty()) {
return getSpecifiedDatabase(conn);
}
List<String> databaseNames = Lists.newArrayList();
@@ -216,11 +220,16 @@ public class JdbcClient {
while (rs.next()) {
tempDatabaseNames.add(rs.getString(1));
}
- if (isOnlySpecifiedDatabase && !specifiedDatabaseMap.isEmpty()) {
+ if (isOnlySpecifiedDatabase) {
for (String db : tempDatabaseNames) {
- if (specifiedDatabaseMap.get(db) != null) {
- databaseNames.add(db);
+ // Exclude database map take effect with higher priority
over include database map
+ if (!excludeDatabaseMap.isEmpty() &&
excludeDatabaseMap.containsKey(db)) {
+ continue;
+ }
+ if (!includeDatabaseMap.isEmpty() &&
includeDatabaseMap.containsKey(db)) {
+ continue;
}
+ databaseNames.add(db);
}
} else {
databaseNames = tempDatabaseNames;
diff --git a/regression-test/data/jdbc_catalog_p0/test_mysql_jdbc_catalog.out
b/regression-test/data/jdbc_catalog_p0/test_mysql_jdbc_catalog.out
index b372579976..66aefa5996 100644
--- a/regression-test/data/jdbc_catalog_p0/test_mysql_jdbc_catalog.out
+++ b/regression-test/data/jdbc_catalog_p0/test_mysql_jdbc_catalog.out
@@ -181,12 +181,21 @@ doris3 20
-- !test_insert4 --
1 abcHa1.12345 1.123450xkalowadawd 2022-10-01 3.14159 1
2 0 100000 1.2345678 24.000 07:09:51 2022
2022-11-27T07:09:51 2022-11-27T07:09:51
--- !specified_database --
+-- !specified_database_1 --
doris_test
--- !specified_database --
+-- !specified_database_2 --
doris_test
+-- !specified_database_3 --
+information_schema
+init_db
+mysql
+performance_schema
+sys
+
+-- !specified_database_4 --
+
-- !ex_tb1 --
{"k1":"v1", "k2":"v2"}
diff --git a/regression-test/data/jdbc_catalog_p0/test_pg_jdbc_catalog.out
b/regression-test/data/jdbc_catalog_p0/test_pg_jdbc_catalog.out
index 6986a6b289..0fc0be189d 100644
--- a/regression-test/data/jdbc_catalog_p0/test_pg_jdbc_catalog.out
+++ b/regression-test/data/jdbc_catalog_p0/test_pg_jdbc_catalog.out
@@ -2152,12 +2152,21 @@ doris2 19
doris3 20
doris3 20
--- !specified_database --
+-- !specified_database_1 --
doris_test
--- !specified_database --
+-- !specified_database_2 --
doris_test
+-- !specified_database_3 --
+catalog_pg_test
+information_schema
+pg_catalog
+pg_toast
+public
+
+-- !specified_database_4 --
+
-- !test_old --
123 abc
123 abc
diff --git
a/regression-test/suites/jdbc_catalog_p0/test_mysql_jdbc_catalog.groovy
b/regression-test/suites/jdbc_catalog_p0/test_mysql_jdbc_catalog.groovy
index b0bf171dee..b0e9409ebe 100644
--- a/regression-test/suites/jdbc_catalog_p0/test_mysql_jdbc_catalog.groovy
+++ b/regression-test/suites/jdbc_catalog_p0/test_mysql_jdbc_catalog.groovy
@@ -138,27 +138,68 @@ suite("test_mysql_jdbc_catalog", "p0") {
sql """CREATE CATALOG ${catalog_name} WITH RESOURCE ${resource_name}"""
sql """switch ${catalog_name}"""
- qt_specified_database """ show databases; """
+ qt_specified_database_1 """ show databases; """
sql """ drop catalog if exists ${catalog_name} """
sql """ drop resource if exists ${resource_name} """
- // test only_specified_database and specified_database_list argument
+ // test only_specified_database and include_database_list argument
sql """create resource if not exists ${resource_name} properties(
"type"="jdbc",
"user"="root",
"password"="123456",
- "jdbc_url" =
"jdbc:mysql://127.0.0.1:${mysql_port}/doris_test?useSSL=false",
+ "jdbc_url" = "jdbc:mysql://127.0.0.1:${mysql_port}?useSSL=false",
"driver_url" =
"https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/jdbc_driver/mysql-connector-java-8.0.25.jar";,
"driver_class" = "com.mysql.cj.jdbc.Driver",
"only_specified_database" = "true",
- "specified_database_list" = "doris_test"
+ "include_database_list" = "doris_test"
);"""
sql """CREATE CATALOG ${catalog_name} WITH RESOURCE ${resource_name}"""
sql """switch ${catalog_name}"""
- qt_specified_database """ show databases; """
+ qt_specified_database_2 """ show databases; """
+
+ sql """ drop catalog if exists ${catalog_name} """
+ sql """ drop resource if exists ${resource_name} """
+
+ // test only_specified_database and exclude_database_list argument
+ sql """create resource if not exists ${resource_name} properties(
+ "type"="jdbc",
+ "user"="root",
+ "password"="123456",
+ "jdbc_url" = "jdbc:mysql://127.0.0.1:${mysql_port}?useSSL=false",
+ "driver_url" =
"https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/jdbc_driver/mysql-connector-java-8.0.25.jar";,
+ "driver_class" = "com.mysql.cj.jdbc.Driver",
+ "only_specified_database" = "true",
+ "exclude_database_list" = "doris_test"
+ );"""
+
+ sql """CREATE CATALOG ${catalog_name} WITH RESOURCE ${resource_name}"""
+ sql """switch ${catalog_name}"""
+
+ qt_specified_database_3 """ show databases; """
+
+ sql """ drop catalog if exists ${catalog_name} """
+ sql """ drop resource if exists ${resource_name} """
+
+ // test include_database_list and exclude_database_list have
overlapping items case
+ sql """create resource if not exists ${resource_name} properties(
+ "type"="jdbc",
+ "user"="root",
+ "password"="123456",
+ "jdbc_url" = "jdbc:mysql://127.0.0.1:${mysql_port}?useSSL=false",
+ "driver_url" =
"https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/jdbc_driver/mysql-connector-java-8.0.25.jar";,
+ "driver_class" = "com.mysql.cj.jdbc.Driver",
+ "only_specified_database" = "true",
+ "include_database_list" = "doris_test",
+ "exclude_database_list" = "doris_test"
+ );"""
+
+ sql """CREATE CATALOG ${catalog_name} WITH RESOURCE ${resource_name}"""
+ sql """switch ${catalog_name}"""
+
+ qt_specified_database_4 """ show databases; """
sql """ drop catalog if exists ${catalog_name} """
sql """ drop resource if exists ${resource_name} """
diff --git a/regression-test/suites/jdbc_catalog_p0/test_pg_jdbc_catalog.groovy
b/regression-test/suites/jdbc_catalog_p0/test_pg_jdbc_catalog.groovy
index 8416fa2fef..1676d3d807 100644
--- a/regression-test/suites/jdbc_catalog_p0/test_pg_jdbc_catalog.groovy
+++ b/regression-test/suites/jdbc_catalog_p0/test_pg_jdbc_catalog.groovy
@@ -102,12 +102,12 @@ suite("test_pg_jdbc_catalog", "p0") {
);"""
sql """CREATE CATALOG ${catalog_name} WITH RESOURCE ${resource_name}
"""
sql """switch ${catalog_name} """
- qt_specified_database """ show databases; """
+ qt_specified_database_1 """ show databases; """
sql """drop catalog if exists ${catalog_name} """
sql """drop resource if exists ${resource_name}"""
- // test only_specified_database and specified_database_list argument
+ // test only_specified_database and include_database_list argument
sql """create resource if not exists ${resource_name} properties(
"type"="jdbc",
"user"="postgres",
@@ -116,11 +116,48 @@ suite("test_pg_jdbc_catalog", "p0") {
"driver_url" =
"https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/jdbc_driver/postgresql-42.5.0.jar";,
"driver_class" = "org.postgresql.Driver",
"only_specified_database" = "true",
- "specified_database_list" = "doris_test"
+ "include_database_list" = "doris_test"
);"""
sql """CREATE CATALOG ${catalog_name} WITH RESOURCE ${resource_name}
"""
sql """switch ${catalog_name} """
- qt_specified_database """ show databases; """
+ qt_specified_database_2 """ show databases; """
+
+ sql """drop catalog if exists ${catalog_name} """
+ sql """drop resource if exists ${resource_name}"""
+
+ // test only_specified_database and exclude_database_list argument
+ sql """create resource if not exists ${resource_name} properties(
+ "type"="jdbc",
+ "user"="postgres",
+ "password"="123456",
+ "jdbc_url" =
"jdbc:postgresql://127.0.0.1:${pg_port}/postgres?currentSchema=doris_test&useSSL=false",
+ "driver_url" =
"https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/jdbc_driver/postgresql-42.5.0.jar";,
+ "driver_class" = "org.postgresql.Driver",
+ "only_specified_database" = "true",
+ "exclude_database_list" = "doris_test"
+ );"""
+ sql """CREATE CATALOG ${catalog_name} WITH RESOURCE ${resource_name}
"""
+ sql """switch ${catalog_name} """
+ qt_specified_database_3 """ show databases; """
+
+ sql """drop catalog if exists ${catalog_name} """
+ sql """drop resource if exists ${resource_name}"""
+
+ // test include_database_list and exclude_database_list have
overlapping items case
+ sql """create resource if not exists ${resource_name} properties(
+ "type"="jdbc",
+ "user"="postgres",
+ "password"="123456",
+ "jdbc_url" =
"jdbc:postgresql://127.0.0.1:${pg_port}/postgres?currentSchema=doris_test&useSSL=false",
+ "driver_url" =
"https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/jdbc_driver/postgresql-42.5.0.jar";,
+ "driver_class" = "org.postgresql.Driver",
+ "only_specified_database" = "true",
+ "include_database_list" = "doris_test",
+ "exclude_database_list" = "doris_test"
+ );"""
+ sql """CREATE CATALOG ${catalog_name} WITH RESOURCE ${resource_name}
"""
+ sql """switch ${catalog_name} """
+ qt_specified_database_4 """ show databases; """
sql """drop catalog if exists ${catalog_name} """
sql """drop resource if exists ${resource_name}"""
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
