This is an automated email from the ASF dual-hosted git repository.
zjffdu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/master by this push:
new 780a7e8 [ZEPPELIN-1070]: Inject Credentials in any Interpreter-Code -
Master
780a7e8 is described below
commit 780a7e8b77999419978d982c7dc8a8a5d4f542f1
Author: jpmcmu <jpm...@gmail.com>
AuthorDate: Tue Oct 29 10:34:41 2019 -0400
[ZEPPELIN-1070]: Inject Credentials in any Interpreter-Code - Master
### What is this PR for?
This PR is a re-submission of the original ZEPPELIN-1070 PR. The original
PR seems to be abandoned and I am currently creating custom builds of Zeppelin
with the ZEPPELIN-1070 PR included so I am interested in getting the PR merged.
I am submitting two PRs one for 0.8X branch and one that includes fixes for
merge conflicts to the master branch.
Original PR Description:
> This PR enables a generic syntax for inserting credentials. A username
can be inserted by $[user.entry] where "entry" is the name of the credential. A
password can be inserted by $[password.entry].
> To avoid output of the password all occurences of the password-String in
the Interpreter-output will be replaced by "###". This should not be a really
secure feature (since the runner of the notebook knows the password anyway),
but it should avoid accidential exposure of the used passwords by any sort of
interpreter
### What type of PR is it?
Feature
### Todos
* [ ] - Documentation
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1070
### How should this be tested?
Unit tests included in PR
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? **No**
* Is there breaking changes for older versions? **Only in very unlikely
circumstances. IE: code that matched {user.VALID_CREDENTIAL_ENTITY} or
{password.VALID_CREDENTIAL_ENTITY}.**
* Does this needs documentation? **Yes**
Author: jpmcmu <jpm...@gmail.com>
Closes #3416 from jpmcmu/ZEPPELIN-1070 and squashes the following commits:
be300d33a [jpmcmu] Added fix for test interference
19dca6a9b [jpmcmu] Quoted password before replacement to allow $ and / in
passwords.
0ae0f5882 [jpmcmu] Code review changes
bc837dfbc [jpmcmu] WIP
86e9ffb2f [jpmcmu] Code review changes
712a1d32c [jpmcmu] Merge branch 'ZEPPELIN-1070' of
https://github.com/pellmont/zeppelin into ZEPPELIN-1070
---
.../zeppelin/img/screenshots/credential_entry.png | Bin 0 -> 3067 bytes
.../screenshots/credential_injection_setting.PNG | Bin 0 -> 2183 bytes
docs/usage/interpreter/overview.md | 16 +++
.../org/apache/zeppelin/interpreter/Constants.java | 2 +
.../zeppelin/notebook/CredentialInjector.java | 112 +++++++++++++++++++++
.../org/apache/zeppelin/notebook/Paragraph.java | 19 +++-
.../zeppelin/notebook/CredentialInjectorTest.java | 86 ++++++++++++++++
.../apache/zeppelin/notebook/ParagraphTest.java | 70 ++++++++++++-
8 files changed, 300 insertions(+), 5 deletions(-)
diff --git a/docs/assets/themes/zeppelin/img/screenshots/credential_entry.png
b/docs/assets/themes/zeppelin/img/screenshots/credential_entry.png
new file mode 100644
index 0000000..745e91d
Binary files /dev/null and
b/docs/assets/themes/zeppelin/img/screenshots/credential_entry.png differ
diff --git
a/docs/assets/themes/zeppelin/img/screenshots/credential_injection_setting.PNG
b/docs/assets/themes/zeppelin/img/screenshots/credential_injection_setting.PNG
new file mode 100644
index 0000000..ca98ca5
Binary files /dev/null and
b/docs/assets/themes/zeppelin/img/screenshots/credential_injection_setting.PNG
differ
diff --git a/docs/usage/interpreter/overview.md
b/docs/usage/interpreter/overview.md
index da59076..3fe0f5f 100644
--- a/docs/usage/interpreter/overview.md
+++ b/docs/usage/interpreter/overview.md
@@ -150,3 +150,19 @@ Before 0.8.0, shutting down Zeppelin also meant to
shutdown all the running inte
In such cases, interpreter process recovery is necessary. Starting from 0.8.0,
users can enable interpreter process recovery via the setting
`zeppelin.recovery.storage.class` as
`org.apache.zeppelin.interpreter.recovery.FileSystemRecoveryStorage` or other
implementations if available in the future. By default it is
`org.apache.zeppelin.interpreter.recovery.NullRecoveryStorage`,
which means recovery is not enabled. Enabling recovery means shutting down
Zeppelin would not terminate interpreter processes, and when Zeppelin is
restarted, it would try to reconnect to the existing running interpreter
processes. If you want to kill all the interpreter processes after terminating
Zeppelin even when recovery is enabled, you can run `bin/stop-interpreter.sh`
+
+## Credential Injection
+
+Credentials from the credential manager can be injected into Notebooks.
Credential injection works by replacing the following patterns in Notebooks
with matching credentials for the Credential Manager:
`{user.CREDENTIAL_ENTITY}` and `{password.CREDENTIAL_ENTITY}`. However,
credential injection must be enabled per Interpreter, by adding a boolean
`injectCredentials` setting in the Interpreters configuration. Injected
passwords are removed from Notebook output to prevent accidentally leaki [...]
+
+**Credential Injection Setting**
+<img
src="{{BASE_PATH}}/assets/themes/zeppelin/img/screenshots/credential_injection_setting.png"
width="500px">
+
+**Credential Entry Example**
+<img
src="{{BASE_PATH}}/assets/themes/zeppelin/img/screenshots/credential_entry.png"
width="500px">
+
+**Credential Injection Example**
+```
+val password = "{password.SOME_CREDENTIAL_ENTITY}"
+val username = "{user.SOME_CREDENTIAL_ENTITY}"
+```
diff --git
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/Constants.java
b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/Constants.java
index 87748ff..fe2f674 100644
---
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/Constants.java
+++
b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/Constants.java
@@ -30,6 +30,8 @@ public class Constants {
public static final String ZEPPELIN_INTERPRETER_PORT =
"zeppelin.interpreter.port";
public static final String ZEPPELIN_INTERPRETER_HOST =
"zeppelin.interpreter.host";
+
+ public static final String INJECT_CREDENTIALS = "injectCredentials";
public static final String EXISTING_PROCESS = "existing_process";
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/CredentialInjector.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/CredentialInjector.java
new file mode 100644
index 0000000..7f7226f
--- /dev/null
+++
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/CredentialInjector.java
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.zeppelin.notebook;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.zeppelin.interpreter.InterpreterResult;
+import org.apache.zeppelin.interpreter.InterpreterResultMessage;
+import org.apache.zeppelin.user.UserCredentials;
+import org.apache.zeppelin.user.UsernamePassword;
+
+/**
+ * Class for replacing {user.>credentialkey<} and
+ * {password.>credentialkey<} tags with the matching credentials from
+ * zeppelin
+ */
+class CredentialInjector {
+
+ private Set<String> passwords = new HashSet<>();
+ private final UserCredentials creds;
+ private static final Pattern userpattern =
Pattern.compile("\\{user\\.([^\\}]+)\\}");
+ private static final Pattern passwordpattern =
Pattern.compile("\\{password\\.([^\\}]+)\\}");
+
+
+ public CredentialInjector(UserCredentials creds) {
+ this.creds = creds;
+ }
+
+ public String replaceCredentials(String code) {
+ if (code == null) {
+ return null;
+ }
+ String replaced = code;
+ Matcher matcher = userpattern.matcher(replaced);
+ while (matcher.find()) {
+ String key = matcher.group(1);
+ UsernamePassword usernamePassword = creds.getUsernamePassword(key);
+ if (usernamePassword != null) {
+ String value = usernamePassword.getUsername();
+ String quotedValue = Matcher.quoteReplacement(value);
+ replaced = matcher.replaceFirst(quotedValue);
+ matcher = userpattern.matcher(replaced);
+ }
+ }
+ matcher = passwordpattern.matcher(replaced);
+ while (matcher.find()) {
+ String key = matcher.group(1);
+ UsernamePassword usernamePassword = creds.getUsernamePassword(key);
+ if (usernamePassword != null) {
+ passwords.add(usernamePassword.getPassword());
+ String value = usernamePassword.getPassword();
+ String quotedValue = Matcher.quoteReplacement(value);
+ replaced = matcher.replaceFirst(quotedValue);
+ matcher = passwordpattern.matcher(replaced);
+ }
+ }
+ return replaced;
+ }
+
+ public InterpreterResult hidePasswords(InterpreterResult ret) {
+ if (ret == null) {
+ return null;
+ }
+ return new InterpreterResult(ret.code(), replacePasswords(ret.message()));
+ }
+
+ private List<InterpreterResultMessage>
replacePasswords(List<InterpreterResultMessage> original) {
+ List<InterpreterResultMessage> replaced = new ArrayList<>();
+ for (InterpreterResultMessage msg : original) {
+ switch(msg.getType()) {
+ case HTML:
+ case TEXT:
+ case TABLE: {
+ String replacedMessages = replacePasswords(msg.getData());
+ replaced.add(new InterpreterResultMessage(msg.getType(),
replacedMessages));
+ break;
+ }
+ default:
+ replaced.add(msg);
+ }
+ }
+ return replaced;
+ }
+
+ private String replacePasswords(String str) {
+ String result = str;
+ for (String password : passwords) {
+ result = result.replace(password, "###");
+ }
+ return result;
+ }
+
+}
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java
index 0100afe..ae3db7b 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java
@@ -40,6 +40,7 @@ import org.apache.zeppelin.display.AngularObjectRegistry;
import org.apache.zeppelin.display.GUI;
import org.apache.zeppelin.display.Input;
import org.apache.zeppelin.helium.HeliumPackage;
+import org.apache.zeppelin.interpreter.Constants;
import org.apache.zeppelin.interpreter.Interpreter;
import org.apache.zeppelin.interpreter.Interpreter.FormType;
import org.apache.zeppelin.interpreter.InterpreterContext;
@@ -490,11 +491,27 @@ public class Paragraph extends
JobWithProgressPoller<InterpreterResult> implemen
script = Input.getSimpleQuery(note.getNoteParams(), scriptBody, true);
script = Input.getSimpleQuery(settings.getParams(), script, false);
}
+
LOGGER.debug("RUN : " + script);
try {
InterpreterContext context = getInterpreterContext();
InterpreterContext.set(context);
- InterpreterResult ret = interpreter.interpret(script, context);
+
+ // Inject credentials
+ String injectPropStr =
interpreter.getProperty(Constants.INJECT_CREDENTIALS, "false");
+ injectPropStr =
context.getStringLocalProperty(Constants.INJECT_CREDENTIALS, injectPropStr);
+ boolean shouldInjectCredentials = Boolean.parseBoolean(injectPropStr);
+
+ InterpreterResult ret = null;
+ if (shouldInjectCredentials) {
+ UserCredentials creds =
context.getAuthenticationInfo().getUserCredentials();
+ CredentialInjector credinjector = new CredentialInjector(creds);
+ String code = credinjector.replaceCredentials(script);
+ ret = interpreter.interpret(code, context);
+ ret = credinjector.hidePasswords(ret);
+ } else {
+ ret = interpreter.interpret(script, context);
+ }
if (interpreter.getFormType() == FormType.NATIVE) {
note.setNoteParams(context.getNoteGui().getParams());
diff --git
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/CredentialInjectorTest.java
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/CredentialInjectorTest.java
new file mode 100644
index 0000000..9b0c93a
--- /dev/null
+++
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/CredentialInjectorTest.java
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.notebook;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import org.apache.zeppelin.interpreter.InterpreterResult;
+import org.apache.zeppelin.interpreter.InterpreterResult.Code;
+import org.apache.zeppelin.user.UserCredentials;
+import org.apache.zeppelin.user.UsernamePassword;
+import org.junit.Test;
+
+public class CredentialInjectorTest {
+
+ private static final String TEMPLATE =
+ "val jdbcUrl =
\"jdbc:mysql://localhost/emp?user={user.mysql}&password={password.mysql}\"";
+ private static final String CORRECT_REPLACED =
+ "val jdbcUrl = \"jdbc:mysql://localhost/emp?user=username&password=pwd\"";
+
+ private static final String ANSWER =
+ "jdbcUrl: String =
jdbc:mysql://localhost/employees?user=username&password=pwd";
+ private static final String HIDDEN =
+ "jdbcUrl: String =
jdbc:mysql://localhost/employees?user=username&password=###";
+
+ @Test
+ public void replaceCredentials() {
+ UserCredentials userCredentials = mock(UserCredentials.class);
+ UsernamePassword usernamePassword = new UsernamePassword("username",
"pwd");
+
when(userCredentials.getUsernamePassword("mysql")).thenReturn(usernamePassword);
+ CredentialInjector testee = new CredentialInjector(userCredentials);
+ String actual = testee.replaceCredentials(TEMPLATE);
+ assertEquals(CORRECT_REPLACED, actual);
+
+ InterpreterResult ret = new InterpreterResult(Code.SUCCESS, ANSWER);
+ InterpreterResult hiddenResult = testee.hidePasswords(ret);
+ assertEquals(1, hiddenResult.message().size());
+ assertEquals(HIDDEN, hiddenResult.message().get(0).getData());
+ }
+
+ @Test
+ public void replaceCredentialNoTexts() {
+ UserCredentials userCredentials = mock(UserCredentials.class);
+ CredentialInjector testee = new CredentialInjector(userCredentials);
+ String actual = testee.replaceCredentials(null);
+ assertNull(actual);
+ }
+
+ @Test
+ public void replaceCredentialsNotExisting() {
+ UserCredentials userCredentials = mock(UserCredentials.class);
+ CredentialInjector testee = new CredentialInjector(userCredentials);
+ String actual = testee.replaceCredentials(TEMPLATE);
+ assertEquals(TEMPLATE, actual);
+
+ InterpreterResult ret = new InterpreterResult(Code.SUCCESS, ANSWER);
+ InterpreterResult hiddenResult = testee.hidePasswords(ret);
+ assertEquals(1, hiddenResult.message().size());
+ assertEquals(ANSWER, hiddenResult.message().get(0).getData());
+ }
+
+ @Test
+ public void hidePasswordsNoResult() {
+ UserCredentials userCredentials = mock(UserCredentials.class);
+ CredentialInjector testee = new CredentialInjector(userCredentials);
+ assertNull(testee.hidePasswords(null));
+ }
+
+}
diff --git
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/ParagraphTest.java
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/ParagraphTest.java
index 524dde7..6b2161b 100644
---
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/ParagraphTest.java
+++
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/ParagraphTest.java
@@ -23,6 +23,7 @@ import static org.junit.Assert.assertNotEquals;
import static org.junit.Assert.assertNotNull;
import static org.mockito.Matchers.any;
import static org.mockito.Matchers.anyString;
+import static org.mockito.Matchers.eq;
import static org.mockito.Mockito.doNothing;
import static org.mockito.Mockito.doReturn;
import static org.mockito.Mockito.mock;
@@ -30,36 +31,45 @@ import static org.mockito.Mockito.spy;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
-import com.google.common.collect.Lists;
-
import java.util.Arrays;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
import org.apache.commons.lang3.tuple.Triple;
import org.apache.zeppelin.display.AngularObject;
import org.apache.zeppelin.display.AngularObjectBuilder;
import org.apache.zeppelin.display.AngularObjectRegistry;
import org.apache.zeppelin.display.Input;
-import org.apache.zeppelin.interpreter.*;
+import org.apache.zeppelin.interpreter.AbstractInterpreterTest;
+import org.apache.zeppelin.interpreter.Constants;
+import org.apache.zeppelin.interpreter.Interpreter;
import org.apache.zeppelin.interpreter.Interpreter.FormType;
import org.apache.zeppelin.interpreter.InterpreterContext;
import org.apache.zeppelin.interpreter.InterpreterOption;
import org.apache.zeppelin.interpreter.InterpreterResult;
import org.apache.zeppelin.interpreter.InterpreterResult.Code;
import org.apache.zeppelin.interpreter.InterpreterResult.Type;
+import org.apache.zeppelin.interpreter.InterpreterResultMessage;
+import org.apache.zeppelin.interpreter.InterpreterSetting;
import org.apache.zeppelin.interpreter.InterpreterSetting.Status;
+import org.apache.zeppelin.interpreter.ManagedInterpreterGroup;
import org.apache.zeppelin.resource.ResourcePool;
import org.apache.zeppelin.user.AuthenticationInfo;
import org.apache.zeppelin.user.Credentials;
+import org.apache.zeppelin.user.UserCredentials;
+import org.apache.zeppelin.user.UsernamePassword;
import org.junit.Rule;
import org.junit.Test;
+import org.junit.rules.ExpectedException;
import java.util.HashMap;
import java.util.Map;
-import org.junit.rules.ExpectedException;
import org.mockito.Mockito;
+import com.google.common.collect.Lists;
+
public class ParagraphTest extends AbstractInterpreterTest {
@Test
@@ -342,4 +352,56 @@ public class ParagraphTest extends AbstractInterpreterTest
{
}
}
+ @Test
+ public void credentialReplacement() throws Throwable {
+ Note mockNote = mock(Note.class);
+ Credentials creds = mock(Credentials.class);
+ when(mockNote.getCredentials()).thenReturn(creds);
+ Paragraph spyParagraph = spy(new Paragraph("para_1", mockNote, null));
+ UserCredentials uc = mock(UserCredentials.class);
+ when(creds.getUserCredentials(anyString())).thenReturn(uc);
+ UsernamePassword up = new UsernamePassword("user", "pwd");
+ when(uc.getUsernamePassword("ent")).thenReturn(up );
+
+ Interpreter mockInterpreter = mock(Interpreter.class);
+ spyParagraph.setInterpreter(mockInterpreter);
+ doReturn(mockInterpreter).when(spyParagraph).getBindedInterpreter();
+
+ ManagedInterpreterGroup mockInterpreterGroup =
mock(ManagedInterpreterGroup.class);
+
when(mockInterpreter.getInterpreterGroup()).thenReturn(mockInterpreterGroup);
+ when(mockInterpreterGroup.getId()).thenReturn("mock_id_1");
+
when(mockInterpreterGroup.getAngularObjectRegistry()).thenReturn(mock(AngularObjectRegistry.class));
+
when(mockInterpreterGroup.getResourcePool()).thenReturn(mock(ResourcePool.class));
+ when(mockInterpreter.getFormType()).thenReturn(FormType.NONE);
+
+ ParagraphJobListener mockJobListener = mock(ParagraphJobListener.class);
+ doReturn(mockJobListener).when(spyParagraph).getListener();
+
doNothing().when(mockJobListener).onOutputUpdateAll(Mockito.<Paragraph>any(),
Mockito.anyList());
+
+ InterpreterResult mockInterpreterResult = mock(InterpreterResult.class);
+ when(mockInterpreter.interpret(anyString(),
Mockito.<InterpreterContext>any())).thenReturn(mockInterpreterResult);
+ when(mockInterpreterResult.code()).thenReturn(Code.SUCCESS);
+
+ AuthenticationInfo user1 = new AuthenticationInfo("user1");
+ spyParagraph.setAuthenticationInfo(user1);
+
+ spyParagraph.setText("val x = \"usr={user.ent}&pass={password.ent}\"");
+
+ // Credentials should only be injected when it is enabled for an
interpreter or when specified in a local property
+ when(mockInterpreter.getProperty(Constants.INJECT_CREDENTIALS,
"false")).thenReturn("false");
+ spyParagraph.jobRun();
+ verify(mockInterpreter).interpret(eq("val x =
\"usr={user.ent}&pass={password.ent}\""), any(InterpreterContext.class));
+
+ when(mockInterpreter.getProperty(Constants.INJECT_CREDENTIALS,
"false")).thenReturn("true");
+ mockInterpreter.setProperty(Constants.INJECT_CREDENTIALS, "true");
+ spyParagraph.jobRun();
+ verify(mockInterpreter).interpret(eq("val x = \"usr=user&pass=pwd\""),
any(InterpreterContext.class));
+
+ // Check if local property override works
+ when(mockInterpreter.getProperty(Constants.INJECT_CREDENTIALS,
"false")).thenReturn("true");
+ spyParagraph.getLocalProperties().put(Constants.INJECT_CREDENTIALS,
"true");
+ spyParagraph.jobRun();
+ verify(mockInterpreter).interpret(eq("val x = \"usr=user&pass=pwd\""),
any(InterpreterContext.class));
+
+ }
}
