[GitHub] nifi pull request #2630: NIFI-5041 Adds SPNEGO authentication to LivySession...
GitHub user peter-toth opened a pull request: https://github.com/apache/nifi/pull/2630 NIFI-5041 Adds SPNEGO authentication to LivySessionController Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/peter-toth/nifi NIFI-5041 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2630.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2630 commit 2ca8d0ea6e883b3f646a320522c9dee52bacd78f Author: Peter Toth Date: 2018-04-10T12:27:51Z NIFI-5041: Adds SPNEGO authentication to LivySessionController ---
[GitHub] nifi issue #2630: NIFI-5041 Adds SPNEGO authentication to LivySessionControl...
Github user peter-toth commented on the issue: https://github.com/apache/nifi/pull/2630 This improvement of LivySessionController uses the KerberosCredentialsService to fetch a principal and a keytab and provide SPNEGO authentication towards a Livy server. In the implementation I switched from HttpURLConnections to HttpClient, which can be configured to do an SPNEGO handshake. ---
[GitHub] nifi issue #2630: NIFI-5041 Adds SPNEGO authentication to LivySessionControl...
Github user peter-toth commented on the issue: https://github.com/apache/nifi/pull/2630 Some of the travis builds failed, but I don't see any issue in the logs except that the size of log exceeded 4MB. Could someone help me with this please? ---
[GitHub] nifi pull request #2630: NIFI-5041 Adds SPNEGO authentication to LivySession...
Github user peter-toth commented on a diff in the pull request: https://github.com/apache/nifi/pull/2630#discussion_r181127144 --- Diff: pom.xml --- @@ -94,6 +94,7 @@ 1.7.25 0.7.1 9.4.3.v20170317 +4.5.5 --- End diff -- I'm not sure I got this issue right. I removed this property and replaced it to a direct version specification where it was used. Is that enough? ---
[GitHub] nifi pull request #2630: NIFI-5041 Adds SPNEGO authentication to LivySession...
Github user peter-toth commented on a diff in the pull request: https://github.com/apache/nifi/pull/2630#discussion_r181441723 --- Diff: pom.xml --- @@ -94,6 +94,7 @@ 1.7.25 0.7.1 9.4.3.v20170317 +4.5.5 --- End diff -- I think I see the issue now. I changed the httpclient and hadoop-auth dependency to "provided" in nifi-hadoop-util so they won't be included into all NARs depending on nifi-hadoop-util. ---
[GitHub] nifi pull request #2630: NIFI-5041 Adds SPNEGO authentication to LivySession...
Github user peter-toth commented on a diff in the pull request: https://github.com/apache/nifi/pull/2630#discussion_r181746461 --- Diff: pom.xml --- @@ -94,6 +94,7 @@ 1.7.25 0.7.1 9.4.3.v20170317 +4.5.5 --- End diff -- @mattyb149, could you please review my change and let me know if I did the right thing? Any suggestions are welcome. I changed the httpclient dependency to provided so it won't get into all NARs depending on nifi-hadoop-utils. Unfortunately httpclient seems to change a lot from version to version and the minimum required version that my changes can work with is 4.4.1. All my additions to hadoop-utils are in new classes, so I believe even a hadoop-utils dependent NAR that specifies a lower version of httpclient will be fine as long as it doesn't start to use my classes directly. ---
[GitHub] nifi pull request #2630: NIFI-5041 Adds SPNEGO authentication to LivySession...
Github user peter-toth commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2630#discussion_r184309195
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-hadoop-utils/src/main/java/org/apache/nifi/hadoop/KerberosConfiguration.java
---
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.hadoop;
+
+import org.apache.hadoop.security.authentication.util.KerberosUtil;
+
+import javax.security.auth.login.AppConfigurationEntry;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Modified Kerberos configuration class from {@link
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.KerberosConfiguration}
+ * that requires authentication from a keytab.
+ */
+public class KerberosConfiguration extends
javax.security.auth.login.Configuration {
--- End diff --
I've added the new entries.
---
[GitHub] nifi pull request #2630: NIFI-5041 Adds SPNEGO authentication to LivySession...
Github user peter-toth commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2630#discussion_r186738068
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-hadoop-utils/src/main/java/org/apache/nifi/hadoop/KerberosConfiguration.java
---
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.hadoop;
+
+import org.apache.hadoop.security.authentication.util.KerberosUtil;
+
+import javax.security.auth.login.AppConfigurationEntry;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Modified Kerberos configuration class from {@link
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.KerberosConfiguration}
+ * that requires authentication from a keytab.
+ */
+public class KerberosConfiguration extends
javax.security.auth.login.Configuration {
--- End diff --
@mattyb149, @joewitt can you please help me and review this PR and let me
know if I need to change anything.
---
[GitHub] nifi issue #2630: NIFI-5041 Adds SPNEGO authentication to LivySessionControl...
Github user peter-toth commented on the issue: https://github.com/apache/nifi/pull/2630 Hi @mattyb149, Thanks for the feedback, I amended the error handling a bit: - Processor is yielded now if there is no session available - KerberosKeytabSPNegoScheme throws a ProcessException on a Kerberos error, so the flow file will be penalized automatically - CS session manager thread no longer dies on an exception, it goes to sleep instead and then retries to fill up the sessions ---
[GitHub] nifi pull request #2630: NIFI-5041 Adds SPNEGO authentication to LivySession...
Github user peter-toth commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2630#discussion_r188609838
--- Diff:
nifi-nar-bundles/nifi-spark-bundle/nifi-livy-controller-service/src/main/java/org/apache/nifi/controller/livy/LivySessionController.java
---
@@ -241,12 +241,14 @@ public void onConfigured(final ConfigurationContext
context) {
while (enabled) {
try {
manageSessions();
+} catch (Exception e) {
+getLogger().error("Livy Session Manager Thread run
into an error, but continues to run", e);
--- End diff --
I agree, changed to what you suggested.
---
[GitHub] nifi pull request #2630: NIFI-5041 Adds SPNEGO authentication to LivySession...
Github user peter-toth commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2630#discussion_r189932016
--- Diff:
nifi-nar-bundles/nifi-spark-bundle/nifi-livy-controller-service/src/main/java/org/apache/nifi/controller/livy/LivySessionController.java
---
@@ -551,4 +561,11 @@ private SSLContext
getSslSocketFactory(SSLContextService sslService)
return sslContext;
}
+private void checkSessionManagerError() throws IOException {
+Exception exception = sessionManagerError;
+if (exception != null) {
+throw new IOException(exception);
--- End diff --
good idea @mattyb149, added a new SessionManagerException
---
[GitHub] nifi pull request #2768: NIFI-5278: fixes JSON escaping of code parameter in...
GitHub user peter-toth opened a pull request: https://github.com/apache/nifi/pull/2768 NIFI-5278: fixes JSON escaping of code parameter in Execute Spark Interactive processor Change-Id: I2cb0e6c658d4a0f2aad9c4aab9201a3334ee54df Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/peter-toth/nifi NIFI-5278 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2768.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2768 commit f54792cd58e69bc43323ef9a063cce4c5c956e61 Author: Peter Toth Date: 2018-06-07T10:13:21Z NIFI-5278: fixes JSON escaping of code Change-Id: I2cb0e6c658d4a0f2aad9c4aab9201a3334ee54df ---
[GitHub] nifi issue #2768: NIFI-5278: fixes JSON escaping of code parameter in Execut...
Github user peter-toth commented on the issue: https://github.com/apache/nifi/pull/2768 @joewitt , thanks for the feedback. I've added Apache Commons Text to NOTICE of the nifi-livy-nar and nifi-assembly as you suggested. I checked that it does not bring in any new transitive dependency and also amended the existing test. ---
[GitHub] nifi pull request #2768: NIFI-5278: fixes JSON escaping of code parameter in...
Github user peter-toth commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2768#discussion_r194157588
--- Diff:
nifi-nar-bundles/nifi-spark-bundle/nifi-livy-processors/src/test/java/org/apache/nifi/processors/livy/TestExecuteSparkInteractiveSSL.java
---
@@ -109,13 +109,15 @@ private static TestServer createServer() throws
IOException {
public void testSslSparkSession() throws Exception {
addHandler(new LivyAPIHandler());
-runner.enqueue("print \"hello world\"");
+String code = "print \"hello world\" //'?!<>[]{}()$&*=%;.|_-\\";
--- End diff --
Removed.
---
[GitHub] nifi pull request #2768: NIFI-5278: fixes JSON escaping of code parameter in...
Github user peter-toth commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2768#discussion_r194157501
--- Diff:
nifi-nar-bundles/nifi-spark-bundle/nifi-livy-processors/src/test/java/org/apache/nifi/processors/livy/ExecuteSparkInteractiveTestBase.java
---
@@ -64,33 +66,37 @@ public void handle(String target, Request baseRequest,
HttpServletRequest reques
}
session1Requests++;
}
-
-response.setContentLength(responseBody.length());
-
-try (PrintWriter writer = response.getWriter()) {
-writer.print(responseBody);
-writer.flush();
-}
-
} else if ("POST".equalsIgnoreCase(request.getMethod())) {
-
-String responseBody = "{}";
-response.setContentType("application/json");
-
-if ("/sessions".equalsIgnoreCase(target)) {
-responseBody = "{\"id\": 1, \"kind\": \"spark\",
\"state\": \"idle\"}";
-} else if
("/sessions/1/statements".equalsIgnoreCase(target)) {
-responseBody = "{\"id\": 7}";
+String requestBody = IOUtils.toString(request.getReader());
+try {
+System.out.println("requestBody: " + requestBody);
--- End diff --
Fixed.
---
[GitHub] nifi pull request #2768: NIFI-5278: fixes JSON escaping of code parameter in...
Github user peter-toth commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2768#discussion_r194157547
--- Diff:
nifi-nar-bundles/nifi-spark-bundle/nifi-livy-processors/src/test/java/org/apache/nifi/processors/livy/ExecuteSparkInteractiveTestBase.java
---
@@ -64,33 +66,37 @@ public void handle(String target, Request baseRequest,
HttpServletRequest reques
}
session1Requests++;
}
-
-response.setContentLength(responseBody.length());
-
-try (PrintWriter writer = response.getWriter()) {
-writer.print(responseBody);
-writer.flush();
-}
-
} else if ("POST".equalsIgnoreCase(request.getMethod())) {
-
-String responseBody = "{}";
-response.setContentType("application/json");
-
-if ("/sessions".equalsIgnoreCase(target)) {
-responseBody = "{\"id\": 1, \"kind\": \"spark\",
\"state\": \"idle\"}";
-} else if
("/sessions/1/statements".equalsIgnoreCase(target)) {
-responseBody = "{\"id\": 7}";
+String requestBody = IOUtils.toString(request.getReader());
+try {
+System.out.println("requestBody: " + requestBody);
+
+new ObjectMapper().readTree(requestBody);
--- End diff --
Done.
---
[GitHub] nifi pull request #2768: NIFI-5278: fixes JSON escaping of code parameter in...
Github user peter-toth commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2768#discussion_r194157756
--- Diff:
nifi-nar-bundles/nifi-spark-bundle/nifi-livy-processors/src/test/java/org/apache/nifi/processors/livy/TestExecuteSparkInteractive.java
---
@@ -85,16 +85,17 @@ private static TestServer createServer() throws
IOException {
@Test
public void testSparkSession() throws Exception {
-
addHandler(new LivyAPIHandler());
-runner.enqueue("print \"hello world\"");
+String code = "print \"hello world\" //'?!<>[]{}()$&*=%;.|_-\\";
--- End diff --
Thanks for the review @mgaido91. I made a minor refactor to the tests and
separated the cases.
---
[GitHub] nifi issue #2768: NIFI-5278: fixes JSON escaping of code parameter in Execut...
Github user peter-toth commented on the issue: https://github.com/apache/nifi/pull/2768 @mattyb149 I've rebased this onto latest master. ---
[jira] [Created] (NIFI-5041) Add convenient SPNEGO/Kerberos authentication support to LivySessionController
Peter Toth created NIFI-5041: Summary: Add convenient SPNEGO/Kerberos authentication support to LivySessionController Key: NIFI-5041 URL: https://issues.apache.org/jira/browse/NIFI-5041 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.5.0 Reporter: Peter Toth Livy requires SPNEGO/Kerberos authentication on a secured cluster. Initiating such an authentication from NiFi is a viable by providing a java.security.auth.login.config system property (https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/lab/part6.html), but this is a bit cumbersome and needs kinit running outside of NiFi. An alternative and more sophisticated solution would be to do the SPNEGO negotiation programmatically. * This solution would add some new properties to the LivySessionController to fetch kerberos principal and password/keytab * Add the required HTTP Negotiate header (with an SPNEGO token) to the HttpURLConnection to do the authentication programmatically (https://tools.ietf.org/html/rfc4559) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5278) ExecuteSparkInteractive processor fails on code containing a quote
Peter Toth created NIFI-5278: Summary: ExecuteSparkInteractive processor fails on code containing a quote Key: NIFI-5278 URL: https://issues.apache.org/jira/browse/NIFI-5278 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.6.0 Reporter: Peter Toth ExecuteSparkInteractive uses org.apache.commons.lang.StringEscapeUtils.escapeJavaScript() which results quotes to be escaped as \' this breaks JSON payload. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5278) ExecuteSparkInteractive processor fails on code containing a quote
[ https://issues.apache.org/jira/browse/NIFI-5278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated NIFI-5278: - Description: ExecuteSparkInteractive uses org.apache.commons.lang.StringEscapeUtils.escapeJavaScript() which results quotes to be escaped as \'. This breaks JSON payload of the Livy REST API. (was: ExecuteSparkInteractive uses org.apache.commons.lang.StringEscapeUtils.escapeJavaScript() which results quotes to be escaped as \' this breaks JSON payload.) > ExecuteSparkInteractive processor fails on code containing a quote > -- > > Key: NIFI-5278 > URL: https://issues.apache.org/jira/browse/NIFI-5278 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.6.0 >Reporter: Peter Toth >Priority: Major > > ExecuteSparkInteractive uses > org.apache.commons.lang.StringEscapeUtils.escapeJavaScript() which results > quotes to be escaped as \'. This breaks JSON payload of the Livy REST API. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
