This is an automated email from the ASF dual-hosted git repository.
zjffdu pushed a commit to branch branch-0.8
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/branch-0.8 by this push:
new 7d4b4b4 [ZEPPELIN-1070]: Inject Credentials in any Interpreter-Code -
0.8x
7d4b4b4 is described below
commit 7d4b4b4f698692e77e9b651b495658365fd0f39a
Author: Pascal Pellmont <git...@ppo2.ch>
AuthorDate: Mon Aug 12 14:45:55 2019 -0400
[ZEPPELIN-1070]: Inject Credentials in any Interpreter-Code - 0.8x
### What is this PR for?
This PR is a re-submission of the original ZEPPELIN-1070 PR. The original
PR seems to be abandoned and I am currently creating custom builds of Zeppelin
with the ZEPPELIN-1070 PR included so I am interested in getting the PR merged.
I am submitting two PRs one for 0.8X branch and one that includes fixes for
merge conflicts to the master branch.
Original PR Description:
> This PR enables a generic syntax for inserting credentials. A username
can be inserted by $[user.entry] where "entry" is the name of the credential. A
password can be inserted by $[password.entry].
> To avoid output of the password all occurences of the password-String in
the Interpreter-output will be replaced by "###". This should not be a really
secure feature (since the runner of the notebook knows the password anyway),
but it should avoid accidential exposure of the used passwords by any sort of
interpreter
### What type of PR is it?
Feature
### Todos
* [ ] - Documentation
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1070
### How should this be tested?
Unit tests are included in PR
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? **No**
* Is there breaking changes for older versions? **Only in very unlikely
circumstances. IE: code that matched {user.VALID_CREDENTIAL_ENTITY} or
{password.VALID_CREDENTIAL_ENTITY}.**
* Does this needs documentation? **Yes**
Author: Pascal Pellmont <git...@ppo2.ch>
Author: jpmcmu <jpm...@gmail.com>
Closes #3415 from jpmcmu/ZEPPELIN-1070-0.8 and squashes the following
commits:
66e69441e [jpmcmu] Code review changes
7e56bf443 [jpmcmu] Code review changes
de714c31c [Pascal Pellmont] [ZEPPELIN-1070] if credential entry is not
found then leave the pattern as is
21d9556db [Pascal Pellmont] [ZEPPELIN-1070] Replaced $[...] pattern with
{...} pattern
e7060f56d [Pascal Pellmont] [ZEPPELIN-1070] Inject Credentials in any
Interpreter-Code
---
.../zeppelin/img/screenshots/credential_entry.png | Bin 0 -> 3067 bytes
.../screenshots/credential_injection_setting.PNG | Bin 0 -> 2183 bytes
docs/usage/interpreter/overview.md | 16 +++
.../org/apache/zeppelin/interpreter/Constants.java | 2 +
.../zeppelin/notebook/CredentialInjector.java | 110 +++++++++++++++++++++
.../org/apache/zeppelin/notebook/Paragraph.java | 15 ++-
.../zeppelin/notebook/CredentialInjectorTest.java | 86 ++++++++++++++++
.../apache/zeppelin/notebook/ParagraphTest.java | 63 ++++++++++--
8 files changed, 285 insertions(+), 7 deletions(-)
diff --git a/docs/assets/themes/zeppelin/img/screenshots/credential_entry.png
b/docs/assets/themes/zeppelin/img/screenshots/credential_entry.png
new file mode 100644
index 0000000..745e91d
Binary files /dev/null and
b/docs/assets/themes/zeppelin/img/screenshots/credential_entry.png differ
diff --git
a/docs/assets/themes/zeppelin/img/screenshots/credential_injection_setting.PNG
b/docs/assets/themes/zeppelin/img/screenshots/credential_injection_setting.PNG
new file mode 100644
index 0000000..ca98ca5
Binary files /dev/null and
b/docs/assets/themes/zeppelin/img/screenshots/credential_injection_setting.PNG
differ
diff --git a/docs/usage/interpreter/overview.md
b/docs/usage/interpreter/overview.md
index 5b567c7..4098202 100644
--- a/docs/usage/interpreter/overview.md
+++ b/docs/usage/interpreter/overview.md
@@ -152,3 +152,19 @@ In such cases, interpreter process recovery is necessary.
Starting from 0.8.0, u
`org.apache.zeppelin.interpreter.recovery.FileSystemRecoveryStorage` or other
implementations if available in future, by default it is
`org.apache.zeppelin.interpreter.recovery.NullRecoveryStorage`
which means recovery is not enabled. Enable recover means shutting down
Zeppelin would not terminating interpreter process,
and when Zeppelin is restarted, it would try to reconnect to the existing
running interpreter processes. If you want to kill all the interpreter
processes after terminating Zeppelin even when recovery is enabled, you can run
`bin/stop-interpreter.sh`
+
+## Credential Injection
+
+Credentials from the credential manager can be injected into Notebooks.
Credential injection works by replacing the following patterns in Notebooks
with matching credentials for the Credential Manager:
`{user.CREDENTIAL_ENTITY}` and `{password.CREDENTIAL_ENTITY}`. However,
credential injection must be enabled per Interpreter, by adding a boolean
`injectCredentials` setting in the Interpreters configuration. Injected
passwords are removed from Notebook output to prevent accidentally leaki [...]
+
+**Credential Injection Setting**
+<img
src="{{BASE_PATH}}/assets/themes/zeppelin/img/screenshots/credential_injection_setting.png"
width="500px">
+
+**Credential Entry Example**
+<img
src="{{BASE_PATH}}/assets/themes/zeppelin/img/screenshots/credential_entry.png"
width="500px">
+
+**Credential Injection Example**
+```
+val password = "{password.SOME_CREDENTIAL_ENTITY}"
+val username = "{user.SOME_CREDENTIAL_ENTITY}"
+```
diff --git
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/Constants.java
b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/Constants.java
index 87748ff..fe2f674 100644
---
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/Constants.java
+++
b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/Constants.java
@@ -30,6 +30,8 @@ public class Constants {
public static final String ZEPPELIN_INTERPRETER_PORT =
"zeppelin.interpreter.port";
public static final String ZEPPELIN_INTERPRETER_HOST =
"zeppelin.interpreter.host";
+
+ public static final String INJECT_CREDENTIALS = "injectCredentials";
public static final String EXISTING_PROCESS = "existing_process";
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/CredentialInjector.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/CredentialInjector.java
new file mode 100644
index 0000000..bc683a7
--- /dev/null
+++
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/CredentialInjector.java
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.zeppelin.notebook;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.zeppelin.interpreter.InterpreterResult;
+import org.apache.zeppelin.interpreter.InterpreterResultMessage;
+import org.apache.zeppelin.user.UserCredentials;
+import org.apache.zeppelin.user.UsernamePassword;
+
+/**
+ * Class for replacing {user.>credentialkey<} and
+ * {password.>credentialkey<} tags with the matching credentials from
+ * zeppelin
+ */
+class CredentialInjector {
+
+ private Set<String> passwords = new HashSet<>();
+ private final UserCredentials creds;
+ private static final Pattern userpattern =
Pattern.compile("\\{user\\.([^\\}]+)\\}");
+ private static final Pattern passwordpattern =
Pattern.compile("\\{password\\.([^\\}]+)\\}");
+
+
+ public CredentialInjector(UserCredentials creds) {
+ this.creds = creds;
+ }
+
+ public String replaceCredentials(String code) {
+ if (code == null) {
+ return null;
+ }
+ String replaced = code;
+ Matcher matcher = userpattern.matcher(replaced);
+ while (matcher.find()) {
+ String key = matcher.group(1);
+ UsernamePassword usernamePassword = creds.getUsernamePassword(key);
+ if (usernamePassword != null) {
+ String value = usernamePassword.getUsername();
+ replaced = matcher.replaceFirst(value);
+ matcher = userpattern.matcher(replaced);
+ }
+ }
+ matcher = passwordpattern.matcher(replaced);
+ while (matcher.find()) {
+ String key = matcher.group(1);
+ UsernamePassword usernamePassword = creds.getUsernamePassword(key);
+ if (usernamePassword != null) {
+ passwords.add(usernamePassword.getPassword());
+ String value = usernamePassword.getPassword();
+ replaced = matcher.replaceFirst(value);
+ matcher = passwordpattern.matcher(replaced);
+ }
+ }
+ return replaced;
+ }
+
+ public InterpreterResult hidePasswords(InterpreterResult ret) {
+ if (ret == null) {
+ return null;
+ }
+ return new InterpreterResult(ret.code(), replacePasswords(ret.message()));
+ }
+
+ private List<InterpreterResultMessage>
replacePasswords(List<InterpreterResultMessage> original) {
+ List<InterpreterResultMessage> replaced = new ArrayList<>();
+ for (InterpreterResultMessage msg : original) {
+ switch(msg.getType()) {
+ case HTML:
+ case TEXT:
+ case TABLE: {
+ String replacedMessages = replacePasswords(msg.getData());
+ replaced.add(new InterpreterResultMessage(msg.getType(),
replacedMessages));
+ break;
+ }
+ default:
+ replaced.add(msg);
+ }
+ }
+ return replaced;
+ }
+
+ private String replacePasswords(String str) {
+ String result = str;
+ for (String password : passwords) {
+ result = result.replace(password, "###");
+ }
+ return result;
+ }
+
+}
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java
index 57756b8..f5a3c22 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java
@@ -37,6 +37,7 @@ import org.apache.zeppelin.display.AngularObjectRegistry;
import org.apache.zeppelin.display.GUI;
import org.apache.zeppelin.display.Input;
import org.apache.zeppelin.helium.HeliumPackage;
+import org.apache.zeppelin.interpreter.Constants;
import org.apache.zeppelin.interpreter.Interpreter;
import org.apache.zeppelin.interpreter.Interpreter.FormType;
import org.apache.zeppelin.interpreter.InterpreterContext;
@@ -434,7 +435,19 @@ public class Paragraph extends Job implements Cloneable,
JsonSerializable {
try {
InterpreterContext context = getInterpreterContext();
InterpreterContext.set(context);
- InterpreterResult ret = interpreter.interpret(script, context);
+ UserCredentials creds =
context.getAuthenticationInfo().getUserCredentials();
+
+ boolean shouldInjectCredentials = Boolean.parseBoolean(
+ interpreter.getProperty(Constants.INJECT_CREDENTIALS, "false"));
+ InterpreterResult ret = null;
+ if (shouldInjectCredentials) {
+ CredentialInjector credinjector = new CredentialInjector(creds);
+ String code = credinjector.replaceCredentials(script);
+ ret = interpreter.interpret(code, context);
+ ret = credinjector.hidePasswords(ret);
+ } else {
+ ret = interpreter.interpret(script, context);
+ }
if (interpreter.getFormType() == FormType.NATIVE) {
note.setNoteParams(context.getNoteGui().getParams());
diff --git
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/CredentialInjectorTest.java
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/CredentialInjectorTest.java
new file mode 100644
index 0000000..9b0c93a
--- /dev/null
+++
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/CredentialInjectorTest.java
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.notebook;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import org.apache.zeppelin.interpreter.InterpreterResult;
+import org.apache.zeppelin.interpreter.InterpreterResult.Code;
+import org.apache.zeppelin.user.UserCredentials;
+import org.apache.zeppelin.user.UsernamePassword;
+import org.junit.Test;
+
+public class CredentialInjectorTest {
+
+ private static final String TEMPLATE =
+ "val jdbcUrl =
\"jdbc:mysql://localhost/emp?user={user.mysql}&password={password.mysql}\"";
+ private static final String CORRECT_REPLACED =
+ "val jdbcUrl = \"jdbc:mysql://localhost/emp?user=username&password=pwd\"";
+
+ private static final String ANSWER =
+ "jdbcUrl: String =
jdbc:mysql://localhost/employees?user=username&password=pwd";
+ private static final String HIDDEN =
+ "jdbcUrl: String =
jdbc:mysql://localhost/employees?user=username&password=###";
+
+ @Test
+ public void replaceCredentials() {
+ UserCredentials userCredentials = mock(UserCredentials.class);
+ UsernamePassword usernamePassword = new UsernamePassword("username",
"pwd");
+
when(userCredentials.getUsernamePassword("mysql")).thenReturn(usernamePassword);
+ CredentialInjector testee = new CredentialInjector(userCredentials);
+ String actual = testee.replaceCredentials(TEMPLATE);
+ assertEquals(CORRECT_REPLACED, actual);
+
+ InterpreterResult ret = new InterpreterResult(Code.SUCCESS, ANSWER);
+ InterpreterResult hiddenResult = testee.hidePasswords(ret);
+ assertEquals(1, hiddenResult.message().size());
+ assertEquals(HIDDEN, hiddenResult.message().get(0).getData());
+ }
+
+ @Test
+ public void replaceCredentialNoTexts() {
+ UserCredentials userCredentials = mock(UserCredentials.class);
+ CredentialInjector testee = new CredentialInjector(userCredentials);
+ String actual = testee.replaceCredentials(null);
+ assertNull(actual);
+ }
+
+ @Test
+ public void replaceCredentialsNotExisting() {
+ UserCredentials userCredentials = mock(UserCredentials.class);
+ CredentialInjector testee = new CredentialInjector(userCredentials);
+ String actual = testee.replaceCredentials(TEMPLATE);
+ assertEquals(TEMPLATE, actual);
+
+ InterpreterResult ret = new InterpreterResult(Code.SUCCESS, ANSWER);
+ InterpreterResult hiddenResult = testee.hidePasswords(ret);
+ assertEquals(1, hiddenResult.message().size());
+ assertEquals(ANSWER, hiddenResult.message().get(0).getData());
+ }
+
+ @Test
+ public void hidePasswordsNoResult() {
+ UserCredentials userCredentials = mock(UserCredentials.class);
+ CredentialInjector testee = new CredentialInjector(userCredentials);
+ assertNull(testee.hidePasswords(null));
+ }
+
+}
diff --git
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/ParagraphTest.java
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/ParagraphTest.java
index e46b739..f5580a4 100644
---
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/ParagraphTest.java
+++
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/ParagraphTest.java
@@ -23,6 +23,7 @@ import static org.junit.Assert.assertNotEquals;
import static org.junit.Assert.assertNotNull;
import static org.mockito.Matchers.any;
import static org.mockito.Matchers.anyString;
+import static org.mockito.Matchers.eq;
import static org.mockito.Mockito.doNothing;
import static org.mockito.Mockito.doReturn;
import static org.mockito.Mockito.mock;
@@ -30,33 +31,39 @@ import static org.mockito.Mockito.spy;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
-import com.google.common.collect.Lists;
-
import java.util.Arrays;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
import org.apache.commons.lang3.tuple.Triple;
import org.apache.zeppelin.display.AngularObject;
import org.apache.zeppelin.display.AngularObjectBuilder;
import org.apache.zeppelin.display.AngularObjectRegistry;
import org.apache.zeppelin.display.Input;
-import org.apache.zeppelin.interpreter.*;
+import org.apache.zeppelin.interpreter.AbstractInterpreterTest;
+import org.apache.zeppelin.interpreter.Constants;
+import org.apache.zeppelin.interpreter.Interpreter;
import org.apache.zeppelin.interpreter.Interpreter.FormType;
import org.apache.zeppelin.interpreter.InterpreterContext;
import org.apache.zeppelin.interpreter.InterpreterOption;
import org.apache.zeppelin.interpreter.InterpreterResult;
import org.apache.zeppelin.interpreter.InterpreterResult.Code;
import org.apache.zeppelin.interpreter.InterpreterResult.Type;
+import org.apache.zeppelin.interpreter.InterpreterResultMessage;
+import org.apache.zeppelin.interpreter.InterpreterSetting;
import org.apache.zeppelin.interpreter.InterpreterSetting.Status;
+import org.apache.zeppelin.interpreter.ManagedInterpreterGroup;
import org.apache.zeppelin.resource.ResourcePool;
import org.apache.zeppelin.user.AuthenticationInfo;
import org.apache.zeppelin.user.Credentials;
+import org.apache.zeppelin.user.UserCredentials;
+import org.apache.zeppelin.user.UsernamePassword;
import org.junit.Test;
-
-import java.util.HashMap;
-import java.util.Map;
import org.mockito.Mockito;
+import com.google.common.collect.Lists;
+
public class ParagraphTest extends AbstractInterpreterTest {
@Test
@@ -299,4 +306,48 @@ public class ParagraphTest extends AbstractInterpreterTest
{
}
}
+ @Test
+ public void credentialReplacement() throws Throwable {
+ Note mockNote = mock(Note.class);
+ Credentials creds = mock(Credentials.class);
+ when(mockNote.getCredentials()).thenReturn(creds);
+ Paragraph spyParagraph = spy(new Paragraph("para_1", mockNote, null,
null));
+ UserCredentials uc = mock(UserCredentials.class);
+ when(creds.getUserCredentials(anyString())).thenReturn(uc);
+ UsernamePassword up = new UsernamePassword("user", "pwd");
+ when(uc.getUsernamePassword("ent")).thenReturn(up );
+
+ Interpreter mockInterpreter = mock(Interpreter.class);
+ spyParagraph.setInterpreter(mockInterpreter);
+ doReturn(mockInterpreter).when(spyParagraph).getBindedInterpreter();
+
+ ManagedInterpreterGroup mockInterpreterGroup =
mock(ManagedInterpreterGroup.class);
+
when(mockInterpreter.getInterpreterGroup()).thenReturn(mockInterpreterGroup);
+ when(mockInterpreterGroup.getId()).thenReturn("mock_id_1");
+
when(mockInterpreterGroup.getAngularObjectRegistry()).thenReturn(mock(AngularObjectRegistry.class));
+
when(mockInterpreterGroup.getResourcePool()).thenReturn(mock(ResourcePool.class));
+ when(mockInterpreter.getFormType()).thenReturn(FormType.NONE);
+
+ ParagraphJobListener mockJobListener = mock(ParagraphJobListener.class);
+ doReturn(mockJobListener).when(spyParagraph).getListener();
+
doNothing().when(mockJobListener).onOutputUpdateAll(Mockito.<Paragraph>any(),
Mockito.anyList());
+
+ InterpreterResult mockInterpreterResult = mock(InterpreterResult.class);
+ when(mockInterpreter.interpret(anyString(),
Mockito.<InterpreterContext>any())).thenReturn(mockInterpreterResult);
+ when(mockInterpreterResult.code()).thenReturn(Code.SUCCESS);
+
+ AuthenticationInfo user1 = new AuthenticationInfo("user1");
+ spyParagraph.setAuthenticationInfo(user1);
+
+ spyParagraph.setText("val x = \"usr={user.ent}&pass={password.ent}\"");
+
+ // Credentials should only be injected when it is enabled for an
interpreter
+ mockInterpreter.setProperty(Constants.INJECT_CREDENTIALS, "false");
+ spyParagraph.jobRun();
+ verify(mockInterpreter).interpret(eq("val x =
\"usr={user.ent}&pass={password.ent}\""), any(InterpreterContext.class));
+
+ mockInterpreter.setProperty(Constants.INJECT_CREDENTIALS, "true");
+ spyParagraph.jobRun();
+ verify(mockInterpreter).interpret(eq("val x = \"usr=user&pass=pwd\""),
any(InterpreterContext.class));
+ }
}
