This is an automated email from the ASF dual-hosted git repository.
casion pushed a commit to branch dev-1.3.1
in repository https://gitbox.apache.org/repos/asf/linkis.git
The following commit(s) were added to refs/heads/dev-1.3.1 by this push:
new a65227696 [feat:4107] Mysql connection security check extract utils
(#4108)
a65227696 is described below
commit a652276961e39a6e0d6f79698e2b082a72f160d1
Author: aiceflower <[email protected]>
AuthorDate: Thu Jan 12 11:27:19 2023 +0800
[feat:4107] Mysql connection security check extract utils (#4108)
* mysql connection security check extract utils
---
CONTRIBUTING.md | 2 +-
CONTRIBUTING_CN.md | 2 +-
docs/info-1.3.1.md | 10 +-
.../common/exception/LinkisSecurityException.java | 30 ++++
.../apache/linkis/common/utils/SecurityUtils.java | 192 +++++++++++++++++++++
.../linkis/common/utils/SecurityUtilsTest.java | 169 ++++++++++++++++++
linkis-dist/package/conf/linkis.properties | 2 +
.../engineplugin/jdbc/utils/JdbcParamUtils.java | 76 +-------
.../jdbc/utils/JdbcParamUtilsTest.java | 83 +++++----
.../query/service/mysql/SqlConnection.java | 62 +------
10 files changed, 462 insertions(+), 166 deletions(-)
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 4514f4b45..2eb3e45e2 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -145,7 +145,7 @@ git push origin dev-fix dev-fix
- If you still don’t know how to initiate a PR to an open source project,
please refer to [About pull
requests](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests)
Whether it is a bug fix or a new feature development, please submit a PR to
the dev-* branch
-- PR and submission name follow the principle of `<type>(<scope>): <subject>`,
for details, please refer to [Commit message and Change log writing
guide](https://linkis.apache.org/community/development-specification/commit-message)
+- PR and submission name follow the principle of `<type>(<scope>): <subject>`,
for details, please refer to [Commit message and Change log writing
guide](https://linkis.apache.org/docs/1.3.1/development/development-specification/commit-message)
- If the PR contains new features, the document update should be included in
this PR
- If this PR is not ready to merge, please add [WIP] prefix to the head of the
name (WIP = work-in-progress)
- All submissions to dev-* branches need to go through at least one review
before they can be merged
diff --git a/CONTRIBUTING_CN.md b/CONTRIBUTING_CN.md
index a16ebf102..e6b67f492 100644
--- a/CONTRIBUTING_CN.md
+++ b/CONTRIBUTING_CN.md
@@ -139,7 +139,7 @@ git push origin dev-fix dev-fix
- 如果您还不知道怎样向开源项目发起 PR，请参考[About pull
requests](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests)
- 无论是 Bug 修复，还是新功能开发，请将 PR 提交到 dev-* 分支
-- PR 和提交名称遵循 `<type>(<scope>): <subject>` 原则，详情可以参考[Commit message 和 Change
log
编写指南](https://linkis.apache.org/zh-CN/community/development-specification/commit-message)
+- PR 和提交名称遵循 `<type>(<scope>): <subject>` 原则，详情可以参考[Commit message 和 Change
log
编写指南](https://linkis.apache.org/zh-CN/docs/1.3.1/development/development-specification/commit-message)
- 如果 PR 中包含新功能，理应将文档更新包含在本次 PR 中
- 如果本次 PR 尚未准备好合并，请在名称头部加上 [WIP] 前缀（WIP = work-in-progress）
- 所有提交到 dev-* 分支的提交至少需要经过一次 Review 才可以被合并
diff --git a/docs/info-1.3.1.md b/docs/info-1.3.1.md
index 835d83651..73c3630a8 100644
--- a/docs/info-1.3.1.md
+++ b/docs/info-1.3.1.md
@@ -1,6 +1,8 @@
## 参数变化
-| 模块名(服务名) | 类型 | 参数名
| 默认值 | 描述
|
-|---------------------------------------------------|-----|----------------------------------------------------------------------|-------|
------------------------------------------------------- |
-| ps-linkismanager | 修改 |
pipeline.output.isoverwtite <br/>-><br/> pipeline.output.isoverwrite | true
|取值范围：true或false|
-| linkis-engineconn-plugins <br/> linkis-datasource | 新增 |
linkis.mysql.strong.security.enable | false |取值范围：true或false|
+| 模块名(服务名) | 类型 | 参数名
| 默认值 | 描述 |
+|---------------------------------------------------|-----|----------------------------------------------------------------------|-------|-----------------|
+| ps-linkismanager | 修改 |
pipeline.output.isoverwtite <br/>-><br/> pipeline.output.isoverwrite | true |
取值范围：true或false |
+| linkis-engineconn-plugins <br/> linkis-datasource | 新增 |
linkis.mysql.strong.security.enable | false | 取值范围：true或false |
+| linkis-common | 新增 |
linkis.mysql.force.params |
allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false
| mysql连接强制携带参数 |
+| linkis-common | 新增 | linkis.mysql.sensitive.params |
allowLoadLocalInfile,autoDeserialize,allowLocalInfile,allowUrlInLocalInfile,# |
mysql连接安全校验参数 |
diff --git
a/linkis-commons/linkis-common/src/main/java/org/apache/linkis/common/exception/LinkisSecurityException.java
b/linkis-commons/linkis-common/src/main/java/org/apache/linkis/common/exception/LinkisSecurityException.java
new file mode 100644
index 000000000..e38ab6acd
--- /dev/null
+++
b/linkis-commons/linkis-common/src/main/java/org/apache/linkis/common/exception/LinkisSecurityException.java
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.linkis.common.exception;
+
+public class LinkisSecurityException extends LinkisRuntimeException {
+
+ @Override
+ public ExceptionLevel getLevel() {
+ return null;
+ }
+
+ public LinkisSecurityException(int errCode, String desc) {
+ super(errCode, desc);
+ }
+}
diff --git
a/linkis-commons/linkis-common/src/main/java/org/apache/linkis/common/utils/SecurityUtils.java
b/linkis-commons/linkis-common/src/main/java/org/apache/linkis/common/utils/SecurityUtils.java
new file mode 100644
index 000000000..5333b2432
--- /dev/null
+++
b/linkis-commons/linkis-common/src/main/java/org/apache/linkis/common/utils/SecurityUtils.java
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.linkis.common.utils;
+
+import org.apache.linkis.common.conf.CommonVars;
+import org.apache.linkis.common.conf.CommonVars$;
+import org.apache.linkis.common.exception.LinkisSecurityException;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public abstract class SecurityUtils {
+
+ private static final Logger logger =
LoggerFactory.getLogger(SecurityUtils.class);
+
+ private static final String COMMA = ",";
+
+ private static final String EQUAL_SIGN = "=";
+
+ private static final String AND_SYMBOL = "&";
+
+ private static final String QUESTION_MARK = "?";
+
+ /** allowLoadLocalInfile,allowLoadLocalInfiled,# */
+ public static final CommonVars<String> MYSQL_SENSITIVE_PARAMS =
+ CommonVars$.MODULE$.apply(
+ "linkis.mysql.sensitive.params",
+
"allowLoadLocalInfile,autoDeserialize,allowLocalInfile,allowUrlInLocalInfile,#");
+
+ /**
+ *
"allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false"
+ */
+ public static final CommonVars<String> MYSQL_FORCE_PARAMS =
+ CommonVars$.MODULE$.apply(
+ "linkis.mysql.force.params",
+
"allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false");
+
+ public static final CommonVars<String> MYSQL_STRONG_SECURITY_ENABLE =
+ CommonVars$.MODULE$.apply("linkis.mysql.strong.security.enable",
"false");
+
+ /**
+ * mysql url append force params
+ *
+ * @param url
+ * @return
+ */
+ public static String appendMysqlForceParams(String url) {
+ if (StringUtils.isBlank(url)) {
+ return "";
+ }
+
+ String extraParamString = MYSQL_FORCE_PARAMS.getValue();
+
+ if (url.endsWith(QUESTION_MARK)) {
+ url = url + extraParamString;
+ } else if (url.lastIndexOf(QUESTION_MARK) < 0) {
+ url = url + QUESTION_MARK + extraParamString;
+ } else {
+ url = url + AND_SYMBOL + extraParamString;
+ }
+ return url;
+ }
+
+ public static void appendMysqlForceParams(Map<String, Object> extraParams) {
+
extraParams.putAll(parseMysqlUrlParamsToMap(MYSQL_FORCE_PARAMS.getValue()));
+ }
+
+ public static String checkJdbcSecurity(String url) {
+ logger.info("checkJdbcSecurity origin url: {}", url);
+ if (StringUtils.isBlank(url)) {
+ throw new LinkisSecurityException(35000, "Invalid mysql connection cul,
url is empty");
+ }
+ if (url.endsWith(QUESTION_MARK) || !url.contains(QUESTION_MARK)) {
+ logger.info("checkJdbcSecurity target url: {}", url);
+ return url;
+ }
+ String[] items = url.split("\\?");
+ if (items.length != 2) {
+ logger.warn("Invalid url: {}", url);
+ throw new LinkisSecurityException(35000, "Invalid mysql connection cul:
" + url);
+ }
+ Map<String, Object> params = parseMysqlUrlParamsToMap(items[1]);
+ Map<String, Object> securityMap = checkJdbcSecurity(params);
+ String paramUrl = parseParamsMapToMysqlParamUrl(securityMap);
+ url = items[0] + QUESTION_MARK + paramUrl;
+ logger.info("checkJdbcSecurity target url: {}", url);
+ return url;
+ }
+
+ /**
+ * check jdbc params
+ *
+ * @param paramsMap
+ */
+ public static Map<String, Object> checkJdbcSecurity(Map<String, Object>
paramsMap) {
+ if (paramsMap == null) {
+ return new HashMap<>();
+ }
+
+ // mysql url strong security
+ if (Boolean.valueOf(MYSQL_STRONG_SECURITY_ENABLE.getValue())) {
+ paramsMap.clear();
+ return paramsMap;
+ }
+
+ Iterator<Map.Entry<String, Object>> iterator =
paramsMap.entrySet().iterator();
+ while (iterator.hasNext()) {
+ Map.Entry<String, Object> entry = iterator.next();
+ String key = entry.getKey();
+ Object value = entry.getValue();
+ if (StringUtils.isBlank(key) || value == null ||
StringUtils.isBlank(value.toString())) {
+ logger.warn("Invalid parameter key or value is blank.");
+ iterator.remove();
+ continue;
+ }
+ if (isNotSecurity(key, value.toString())) {
+ logger.warn("Sensitive param : key={} and value={}", key, value);
+ throw new LinkisSecurityException(
+ 35000,
+ "Invalid mysql connection parameters: " +
parseParamsMapToMysqlParamUrl(paramsMap));
+ }
+ }
+ return paramsMap;
+ }
+
+ public static String parseParamsMapToMysqlParamUrl(Map<String, Object>
forceParams) {
+ if (forceParams == null) {
+ return "";
+ }
+ return forceParams.entrySet().stream()
+ .map(e -> String.join(EQUAL_SIGN, e.getKey(),
String.valueOf(e.getValue())))
+ .collect(Collectors.joining(AND_SYMBOL));
+ }
+
+ private static Map<String, Object> parseMysqlUrlParamsToMap(String
paramsUrl) {
+ String[] params = paramsUrl.split(AND_SYMBOL);
+ Map<String, Object> map = new LinkedHashMap<>(params.length);
+ for (String param : params) {
+ String[] item = param.split(EQUAL_SIGN);
+ if (item.length != 2) {
+ logger.warn("mysql force param {} error.", param);
+ continue;
+ }
+ map.put(item[0], item[1]);
+ }
+ return map;
+ }
+
+ private static boolean isNotSecurity(String key, String value) {
+ boolean res = true;
+ String sensitiveParamsStr = MYSQL_SENSITIVE_PARAMS.getValue();
+ if (StringUtils.isBlank(sensitiveParamsStr)) {
+ return false;
+ }
+ String[] forceParams = sensitiveParamsStr.split(COMMA);
+ for (String forceParam : forceParams) {
+ if (isNotSecurity(key, value, forceParam)) {
+ res = false;
+ break;
+ }
+ }
+ return !res;
+ }
+
+ private static boolean isNotSecurity(String key, String value, String param)
{
+ return key.toLowerCase().contains(param.toLowerCase())
+ || value.toLowerCase().contains(param.toLowerCase());
+ }
+}
diff --git
a/linkis-commons/linkis-common/src/test/java/org/apache/linkis/common/utils/SecurityUtilsTest.java
b/linkis-commons/linkis-common/src/test/java/org/apache/linkis/common/utils/SecurityUtilsTest.java
new file mode 100644
index 000000000..9d4893e46
--- /dev/null
+++
b/linkis-commons/linkis-common/src/test/java/org/apache/linkis/common/utils/SecurityUtilsTest.java
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.linkis.common.utils;
+
+import org.apache.linkis.common.exception.LinkisSecurityException;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+/** SecurityUtils Tester */
+public class SecurityUtilsTest {
+
+ @Test
+ public void testAppendMysqlForceParamsUrl() throws Exception {
+ //
allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false
+ String url = "jdbc:mysql://127.0.0.1:10000/db_name";
+ String newUrl = SecurityUtils.appendMysqlForceParams(url);
+ Assertions.assertEquals(
+ url
+ +
"?allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false",
+ newUrl);
+
+ url = "jdbc:mysql://127.0.0.1:10000/db_name?";
+ newUrl = SecurityUtils.appendMysqlForceParams(url);
+ Assertions.assertEquals(
+ url
+ +
"allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false",
+ newUrl);
+
+ url = "jdbc:mysql://127.0.0.1:10000/db_name?p1=v1";
+ newUrl = SecurityUtils.appendMysqlForceParams(url);
+ Assertions.assertEquals(
+ url
+ + "&"
+ +
"allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false",
+ newUrl);
+ }
+
+ @Test
+ public void testAppendMysqlForceParamsExtraParams() throws Exception {
+ Map<String, Object> extraParams = new HashMap<>();
+ extraParams.put("testKey", "testValue");
+ SecurityUtils.appendMysqlForceParams(extraParams);
+ Assertions.assertEquals("false", extraParams.get("allowLoadLocalInfile"));
+ Assertions.assertEquals("false", extraParams.get("autoDeserialize"));
+ Assertions.assertEquals("false", extraParams.get("allowLocalInfile"));
+ Assertions.assertEquals("false", extraParams.get("allowUrlInLocalInfile"));
+ Assertions.assertEquals("testValue", extraParams.get("testKey"));
+ Assertions.assertEquals(null, extraParams.get("otherKey"));
+ }
+
+ @Test
+ public void testCheckJdbcSecurityUrl() throws Exception {
+ String url = "jdbc:mysql://127.0.0.1:10000/db_name";
+ String newUrl = SecurityUtils.checkJdbcSecurity(url);
+ Assertions.assertEquals(url, newUrl);
+
+ url = "jdbc:mysql://127.0.0.1:10000/db_name?";
+ newUrl = SecurityUtils.checkJdbcSecurity(url);
+ Assertions.assertEquals(url, newUrl);
+
+ url = "jdbc:mysql://127.0.0.1:10000/db_name?p1=v1";
+ newUrl = SecurityUtils.checkJdbcSecurity(url);
+ Assertions.assertEquals(url, newUrl);
+
+ // key is not security
+ url = "jdbc:mysql://127.0.0.1:10000/db_name?p1=v1&allowLocalInfile=true";
+ AtomicReference<String> atomUrl = new AtomicReference<>(url);
+ Assertions.assertThrows(
+ LinkisSecurityException.class,
+ () -> {
+ SecurityUtils.checkJdbcSecurity(atomUrl.get());
+ });
+
+ // value is not security
+ url = "jdbc:mysql://127.0.0.1:10000/db_name?p1=allowLocalInfile";
+ atomUrl.set(url);
+ Assertions.assertThrows(
+ LinkisSecurityException.class,
+ () -> {
+ SecurityUtils.checkJdbcSecurity(atomUrl.get());
+ });
+
+ // contains #
+ url = "jdbc:mysql://127.0.0.1:10000/db_name?p1=v1&#p2=v2";
+ atomUrl.set(url);
+ Assertions.assertThrows(
+ LinkisSecurityException.class,
+ () -> {
+ SecurityUtils.checkJdbcSecurity(atomUrl.get());
+ });
+ }
+
+ @Test
+ public void testCheckJdbcSecurityParamsMap() throws Exception {
+ Map<String, Object> paramsMap = new HashMap<>();
+ paramsMap.put("p1", "v1");
+ Map<String, Object> newMap = SecurityUtils.checkJdbcSecurity(paramsMap);
+ Assertions.assertEquals("v1", newMap.get("p1"));
+
+ // key not security
+ paramsMap.put("allowLocalInfile", "false");
+ Assertions.assertThrows(
+ LinkisSecurityException.class,
+ () -> {
+ SecurityUtils.checkJdbcSecurity(paramsMap);
+ });
+
+ // value not security
+ paramsMap.clear();
+ paramsMap.put("p1", "allowLocalInfile");
+ Assertions.assertThrows(
+ LinkisSecurityException.class,
+ () -> {
+ SecurityUtils.checkJdbcSecurity(paramsMap);
+ });
+
+ // contains #
+ paramsMap.clear();
+ paramsMap.put("p1#", "v1");
+ Assertions.assertThrows(
+ LinkisSecurityException.class,
+ () -> {
+ SecurityUtils.checkJdbcSecurity(paramsMap);
+ });
+
+ paramsMap.clear();
+ paramsMap.put("p1", "v1#");
+ Assertions.assertThrows(
+ LinkisSecurityException.class,
+ () -> {
+ SecurityUtils.checkJdbcSecurity(paramsMap);
+ });
+ }
+
+ @Test
+ public void testMapToString() throws Exception {
+ Map<String, Object> paramsMap = new HashMap<>();
+ paramsMap.put("p1", "v1");
+ String str = SecurityUtils.parseParamsMapToMysqlParamUrl(paramsMap);
+ Assertions.assertEquals("p1=v1", str);
+
+ paramsMap.clear();
+ str = SecurityUtils.parseParamsMapToMysqlParamUrl(paramsMap);
+ Assertions.assertEquals("", str);
+
+ str = SecurityUtils.parseParamsMapToMysqlParamUrl(null);
+ Assertions.assertEquals("", str);
+ }
+}
diff --git a/linkis-dist/package/conf/linkis.properties
b/linkis-dist/package/conf/linkis.properties
index d4a602ecc..2f6ef7fc1 100644
--- a/linkis-dist/package/conf/linkis.properties
+++ b/linkis-dist/package/conf/linkis.properties
@@ -28,6 +28,8 @@ wds.linkis.server.mybatis.datasource.password=
# mysql
wds.linkis.mysql.is.encrypt=false
linkis.mysql.strong.security.enable=false
+linkis.mysql.force.params=allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false
+linkis.mysql.sensitive.params=allowLoadLocalInfile,autoDeserialize,allowLocalInfile,allowUrlInLocalInfile,#
#hadoop/hive/spark config
#hadoop.config.dir=/appcom/config/hadoop-config
diff --git
a/linkis-engineconn-plugins/jdbc/src/main/java/org/apache/linkis/manager/engineplugin/jdbc/utils/JdbcParamUtils.java
b/linkis-engineconn-plugins/jdbc/src/main/java/org/apache/linkis/manager/engineplugin/jdbc/utils/JdbcParamUtils.java
index 32518d1a2..40e6a2ae9 100644
---
a/linkis-engineconn-plugins/jdbc/src/main/java/org/apache/linkis/manager/engineplugin/jdbc/utils/JdbcParamUtils.java
+++
b/linkis-engineconn-plugins/jdbc/src/main/java/org/apache/linkis/manager/engineplugin/jdbc/utils/JdbcParamUtils.java
@@ -19,15 +19,14 @@ package org.apache.linkis.manager.engineplugin.jdbc.utils;
import org.apache.linkis.common.conf.CommonVars;
import org.apache.linkis.common.conf.CommonVars$;
+import org.apache.linkis.common.utils.SecurityUtils;
import org.apache.linkis.manager.engineplugin.jdbc.JDBCPropertiesParser;
import
org.apache.linkis.manager.engineplugin.jdbc.constant.JDBCEngineConnConstant;
import
org.apache.linkis.manager.engineplugin.jdbc.exception.JDBCParamsIllegalException;
import org.apache.commons.lang3.StringUtils;
-import java.util.HashMap;
import java.util.Map;
-import java.util.stream.Collectors;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -71,6 +70,9 @@ public class JdbcParamUtils {
}
public static String filterJdbcUrl(String url) {
+
+ LOG.info("the filter source url is: {}", url);
+
if (StringUtils.isBlank(url)) {
return url;
}
@@ -79,78 +81,16 @@ public class JdbcParamUtils {
return url;
}
- // no params
- if (!url.contains(String.valueOf(QUESTION_MARK))) {
- return url + QUESTION_MARK + APPEND_PARAMS;
- }
+ // security check
+ url = SecurityUtils.checkJdbcSecurity(url);
- // enable strong security
- if (Boolean.valueOf(MYSQL_STRONG_SECURITY_ENABLE.getValue())) {
- LOG.info("mysql engine use strong security configuration. Remove all
connection parameters.");
- return url + QUESTION_MARK + APPEND_PARAMS;
- }
+ // append force params
+ url = SecurityUtils.appendMysqlForceParams(url);
- // deal with params
- String[] items = url.split("\\?");
- // params error: multiple question marks
- if (items.length != 2) {
- LOG.warn("JDBC params error, the url is : " + url);
- return items[0];
- }
-
- String[] params = items[1].split("&");
- Map<String, String> paramsMap = new HashMap<>(params.length);
- for (String param : params) {
- String[] keyAndValues = param.split("=");
- // params error: key and value error
- if (keyAndValues.length != 2) {
- continue;
- }
- String key = keyAndValues[0];
- String value = keyAndValues[1];
- // key and value is blank
- if (StringUtils.isBlank(key) || StringUtils.isBlank(value)) {
- continue;
- }
- if (isSecurity(key, value)) {
- paramsMap.put(key, value);
- } else {
- LOG.warn("Sensitive param : key={} and value={}", key, value);
- }
- }
- String extraParamString =
- paramsMap.entrySet().stream()
- .map(e -> String.join("=", e.getKey(),
String.valueOf(e.getValue())))
- .collect(Collectors.joining("&"));
-
- if (StringUtils.isBlank(extraParamString)) {
- url = items[0];
- } else {
- url = items[0] + String.valueOf(QUESTION_MARK) + extraParamString;
- }
- if (url.endsWith(String.valueOf(QUESTION_MARK))) {
- url = url + APPEND_PARAMS;
- } else if (url.lastIndexOf(QUESTION_MARK) < 0) {
- url = url + QUESTION_MARK + APPEND_PARAMS;
- } else {
- url = url + AND_SYMBOL + APPEND_PARAMS;
- }
LOG.info("The filtered jdbc url is: {}", url);
return url;
}
- private static boolean isSecurity(String key, String value) {
- return !(isNotSecurity(key) || isNotSecurity(value));
- }
-
- private static boolean isNotSecurity(String key) {
- return key.toLowerCase().contains("allowLoadLocalInfile".toLowerCase())
- || key.toLowerCase().contains("autoDeserialize".toLowerCase())
- || key.toLowerCase().contains("allowLocalInfile".toLowerCase())
- || key.toLowerCase().contains("allowUrlInLocalInfile".toLowerCase())
- || key.toLowerCase().contains("#".toLowerCase());
- }
-
public static String getJdbcUsername(Map<String, String> properties)
throws JDBCParamsIllegalException {
String username =
diff --git
a/linkis-engineconn-plugins/jdbc/src/test/java/org/apache/linkis/manager/engineplugin/jdbc/utils/JdbcParamUtilsTest.java
b/linkis-engineconn-plugins/jdbc/src/test/java/org/apache/linkis/manager/engineplugin/jdbc/utils/JdbcParamUtilsTest.java
index fbaa6c3f2..26d7ed424 100644
---
a/linkis-engineconn-plugins/jdbc/src/test/java/org/apache/linkis/manager/engineplugin/jdbc/utils/JdbcParamUtilsTest.java
+++
b/linkis-engineconn-plugins/jdbc/src/test/java/org/apache/linkis/manager/engineplugin/jdbc/utils/JdbcParamUtilsTest.java
@@ -17,11 +17,13 @@
package org.apache.linkis.manager.engineplugin.jdbc.utils;
+import org.apache.linkis.common.exception.LinkisSecurityException;
import
org.apache.linkis.manager.engineplugin.jdbc.constant.JDBCEngineConnConstant;
import
org.apache.linkis.manager.engineplugin.jdbc.exception.JDBCParamsIllegalException;
import java.util.HashMap;
import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.DisplayName;
@@ -31,49 +33,56 @@ public class JdbcParamUtilsTest {
@Test
@DisplayName("testFilterJdbcUrl")
public void testFilterJdbcUrl() {
- String url =
-
"jdbc:mysql://localhost:3306/db_name?allowLoadLocalInfile=true&autoDeserialize=true&#sk=13&p1=v1";
- url = JdbcParamUtils.filterJdbcUrl(url);
- Assertions.assertEquals(
-
"jdbc:mysql://localhost:3306/db_name?p1=v1&allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false",
- url);
+ String securityParam =
+
"allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false";
+ String url = "jdbc:mysql://127.0.0.1:10000/db_name";
+ String newUrl = JdbcParamUtils.filterJdbcUrl(url);
+ Assertions.assertEquals(url + "?" + securityParam, newUrl);
- url =
-
"jdbc:mysql://localhost:3306/db_name?\\#allowLoadLocalInfile=true&autoDeserialize=true&#sk=13&p1=v1";
- url = JdbcParamUtils.filterJdbcUrl(url);
- Assertions.assertEquals(
-
"jdbc:mysql://localhost:3306/db_name?p1=v1&allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false",
- url);
+ // not mysql url
+ url = "h2:mysql";
+ newUrl = JdbcParamUtils.filterJdbcUrl(url);
+ Assertions.assertEquals(url, newUrl);
- url = "jdbc:mysql://localhost:3306/db_name";
- url = JdbcParamUtils.filterJdbcUrl(url);
- Assertions.assertEquals(
-
"jdbc:mysql://localhost:3306/db_name?allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false",
- url);
+ // start with JDBC
+ url = "JDBC:mysql://127.0.0.1:10000/db_name?";
+ newUrl = JdbcParamUtils.filterJdbcUrl(url);
+ Assertions.assertEquals(url + securityParam, newUrl);
- url = "jdbc:mysql://localhost:3306/db_name?p1=v1";
- url = JdbcParamUtils.filterJdbcUrl(url);
- Assertions.assertEquals(
-
"jdbc:mysql://localhost:3306/db_name?p1=v1&allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false",
- url);
+ url = "jdbc:mysql://127.0.0.1:10000/db_name?";
+ newUrl = JdbcParamUtils.filterJdbcUrl(url);
+ Assertions.assertEquals(url + securityParam, newUrl);
- url = "jdbc:mysql://localhost:3306/db_name?autoDeserialize=true";
- url = JdbcParamUtils.filterJdbcUrl(url);
- Assertions.assertEquals(
-
"jdbc:mysql://localhost:3306/db_name?allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false",
- url);
+ url = "jdbc:mysql://127.0.0.1:10000/db_name?p1=v1";
+ newUrl = JdbcParamUtils.filterJdbcUrl(url);
+ Assertions.assertEquals(url + "&" + securityParam, newUrl);
- url = "jdbc:mysql://localhost:3306/db_name?p1=v1&autoDeserialize=true";
- url = JdbcParamUtils.filterJdbcUrl(url);
- Assertions.assertEquals(
-
"jdbc:mysql://localhost:3306/db_name?p1=v1&allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false",
- url);
+ // key is not security
+ url = "jdbc:mysql://127.0.0.1:10000/db_name?p1=v1&allowLocalInfile=true";
+ AtomicReference<String> atomUrl = new AtomicReference<>(url);
+ Assertions.assertThrows(
+ LinkisSecurityException.class,
+ () -> {
+ JdbcParamUtils.filterJdbcUrl(atomUrl.get());
+ });
- url =
"jdbc:mysql://localhost:3306/db_name?p1=v1&autoDeserialize=true&p2=v2";
- url = JdbcParamUtils.filterJdbcUrl(url);
- Assertions.assertEquals(
-
"jdbc:mysql://localhost:3306/db_name?p1=v1&p2=v2&allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false",
- url);
+ // value is not security
+ url = "jdbc:mysql://127.0.0.1:10000/db_name?p1=allowLocalInfile";
+ atomUrl.set(url);
+ Assertions.assertThrows(
+ LinkisSecurityException.class,
+ () -> {
+ JdbcParamUtils.filterJdbcUrl(atomUrl.get());
+ });
+
+ // contains #
+ url = "jdbc:mysql://127.0.0.1:10000/db_name?p1=v1&#p2=v2";
+ atomUrl.set(url);
+ Assertions.assertThrows(
+ LinkisSecurityException.class,
+ () -> {
+ JdbcParamUtils.filterJdbcUrl(atomUrl.get());
+ });
}
@Test
diff --git
a/linkis-public-enhancements/linkis-datasource/linkis-metadata-query/service/jdbc/src/main/java/org/apache/linkis/metadata/query/service/mysql/SqlConnection.java
b/linkis-public-enhancements/linkis-datasource/linkis-metadata-query/service/jdbc/src/main/java/org/apache/linkis/metadata/query/service/mysql/SqlConnection.java
index dc2819a2a..0b5099cce 100644
---
a/linkis-public-enhancements/linkis-datasource/linkis-metadata-query/service/jdbc/src/main/java/org/apache/linkis/metadata/query/service/mysql/SqlConnection.java
+++
b/linkis-public-enhancements/linkis-datasource/linkis-metadata-query/service/jdbc/src/main/java/org/apache/linkis/metadata/query/service/mysql/SqlConnection.java
@@ -18,18 +18,15 @@
package org.apache.linkis.metadata.query.service.mysql;
import org.apache.linkis.common.conf.CommonVars;
+import org.apache.linkis.common.utils.SecurityUtils;
import org.apache.linkis.metadata.query.common.domain.MetaColumnInfo;
-import org.apache.commons.lang.StringUtils;
-
import java.io.Closeable;
import java.io.IOException;
import java.sql.*;
import java.util.*;
import java.util.stream.Collectors;
-import scala.annotation.meta.param;
-
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -83,60 +80,15 @@ public class SqlConnection implements Closeable {
return;
}
- // enable strong security
- if (MYSQL_STRONG_SECURITY_ENABLE.getValue()) {
- LOG.info(
- "mysql metadata use strong security configuration. Remove all
connection parameters.");
- extraParams.clear();
- }
-
- // Delete suspected vulnerability parameters
- Iterator<Map.Entry<String, Object>> iterator =
extraParams.entrySet().iterator();
- while (iterator.hasNext()) {
- Map.Entry<String, Object> entry = iterator.next();
- String key = entry.getKey();
- if (StringUtils.isBlank(key)
- || entry.getValue() == null
- || StringUtils.isBlank(entry.getValue().toString())) {
- iterator.remove();
- continue;
- }
- String value = entry.getValue().toString();
- if (keyAndValueIsNotSecurity(key, value, "allowLoadLocalInfile")
- || keyAndValueIsNotSecurity(key, value, "autoDeserialize")
- || keyAndValueIsNotSecurity(key, value, "allowLocalInfile")
- || keyAndValueIsNotSecurity(key, value, "allowUrlInLocalInfile")
- || keyAndValueIsNotSecurity(key, value, "#")) {
- iterator.remove();
- LOG.warn("mysql metadata sensitive param : key={} and value={}", key,
value);
- }
- }
+ // security check
+ SecurityUtils.checkJdbcSecurity(extraParams);
- // Set all vulnerability parameters to false
- extraParams.put("allowLoadLocalInfile", "false");
- extraParams.put("autoDeserialize", "false");
- extraParams.put("allowLocalInfile", "false");
- extraParams.put("allowUrlInLocalInfile", "false");
+ // append force params
+ SecurityUtils.appendMysqlForceParams(extraParams);
// print extraParams
- StringBuilder sb = new StringBuilder("mysql metadata url extraParams: [ ");
- for (Map.Entry<String, Object> paramEntry : extraParams.entrySet()) {
-
sb.append(paramEntry.getKey()).append("=").append(paramEntry.getValue()).append("
,");
- }
- sb.deleteCharAt(sb.length() - 1);
- sb.append("]");
- LOG.info(sb.toString());
- }
-
- private boolean keyAndValueIsNotSecurity(String key, String value, String
param) {
- return !(isSecurity(key, param) && isSecurity(value, param));
- }
-
- private boolean isSecurity(String noSecurityKey, String param) {
- if (StringUtils.isBlank(param) || StringUtils.isBlank(noSecurityKey)) {
- return true;
- }
- return !noSecurityKey.toLowerCase().contains(param.toLowerCase());
+ String logStr = SecurityUtils.parseParamsMapToMysqlParamUrl(extraParams);
+ LOG.info("mysql metadata url extraParams: {}", logStr);
}
public List<String> getAllDatabases() throws SQLException {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
