[GitHub] nifi issue #3114: NIFI-5765 Fixing WriteJsonResult to use chosenDataType whe...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/3114 Thanks @ijokarumawak for the quick review! ---
[GitHub] nifi issue #3114: NIFI-5765 Fixing WriteJsonResult to use chosenDataType whe...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/3114 Follow-up commit is b/c I found the same bug in PutElasticsearchHttpRecord. I believe that's the only other occurrence. ---
[GitHub] nifi issue #3114: NIFI-5765 Fixing WriteJsonResult to use chosenDataType whe...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/3114 Template for testing on the ticket: https://issues.apache.org/jira/browse/NIFI-5765 ---
[GitHub] nifi pull request #3114: NIFI-5765 Fixing WriteJsonResult to use chosenDataT...
GitHub user JPercivall opened a pull request: https://github.com/apache/nifi/pull/3114 NIFI-5765 Fixing WriteJsonResult to use chosenDataType when writing a⦠â¦n Array value Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [X] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [X] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JPercivall/nifi NIFI-5765_handling_WriteJsonResult_choice_array Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3114.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3114 commit 6babefda6a533c0699a20d02a0dc84b25acc7ae8 Author: Joe Percivall Date: 2018-10-29T20:04:05Z NIFI-5765 Fixing WriteJsonResult to use chosenDataType when writing an Array value ---
[GitHub] nifi issue #3090: NIFI-3792 Adding a RetryCount processor to faciliate retry...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/3090 Hey @patricker, here's the processor I mentioned. It attempts to avert the livelock scenario by looking at the input queue if it's over a certain size/count divert FFs but there's still the chance for livelock in some scenarios. ---
[GitHub] nifi pull request #3090: NIFI-3792 Adding a RetryCount processor to faciliat...
GitHub user JPercivall opened a pull request: https://github.com/apache/nifi/pull/3090 NIFI-3792 Adding a RetryCount processor to faciliate retry logic as w⦠â¦ell as penalizing a FlowFile. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JPercivall/nifi NIFI-3792_Adding_a_RetryCount_processor Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3090.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3090 commit 2a91eff9948c5b6600a5b64515170d240803d1a3 Author: Joe Percivall Date: 2018-10-17T19:44:04Z NIFI-3792 Adding a RetryCount processor to faciliate retry logic as well as penalizing a FlowFile. ---
[GitHub] nifi issue #2813: NIFI-5084 Added GenerateRecord processor.
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2813 Ah yup, totally different lol. I was thinking GenerateFlowFile but with just a writer + dynamic properties to set fields. Sorry for the confusion! ---
[GitHub] nifi issue #2813: NIFI-5084 Added GenerateRecord processor.
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2813 Is there a reason this processor wasn't just a part of the normal record bundle instead of being thought of as part of a larger reprocessing bundle? If it's just another file in that same nar, it wouldn't be a lot of space. For reference, I have the use-case where I have attributes on a FF and the content is a binary file. After putting the file to disk, I want to create a nested json object to continue processing. I essentially want to create a new record just from EL (same as UpdateRecord but with no reader). I believe GenerateRecord would fill this need? ---
[GitHub] nifi issue #2813: NIFI-5084 Added GenerateRecord processor.
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2813 @MikeThomsen I'm looking for the discussion on the mailing list on why this was decided to not be merged in and I can't find it. I only see the "How would we handle this?" thread[1]. Could you link to it? [1] http://apache-nifi-developer-list.39713.n7.nabble.com/How-would-we-handle-this-td19151.html ---
[GitHub] nifi issue #2929: NIFI-5474 ReplaceText RegexReplace evaluates payload as Ex...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2929 Hey @ottobackwards, thanks for the PR. I should be able to review this around mid next week. ---
[GitHub] nifi-minifi pull request #117: MINIFI-424 Adding the ability to evaluate boo...
Github user JPercivall closed the pull request at: https://github.com/apache/nifi-minifi/pull/117 ---
[GitHub] nifi issue #2616: NIFI-5052 Added DeleteByQuery ElasticSearch processor.
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2616 Hey @MikeThomsen sorry to leave you hanging but I just got super busy with my day job. I may be able to help in a couple weeks can't at the moment. ---
[GitHub] nifi-minifi pull request #117: MINIFI-424 Adding the ability to evaluate boo...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/117#discussion_r183258883
--- Diff:
minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/RunMiNiFi.java
---
@@ -1176,11 +1191,18 @@ public boolean accept(final File dir, final String
filename) {
@SuppressWarnings({"rawtypes", "unchecked"})
public void start() throws IOException, InterruptedException {
+Properties bootstrapProperties = getBootstrapProperties();
+
+final String confDir =
bootstrapProperties.getProperty(CONF_DIR_KEY);
+final File configFile = new
File(bootstrapProperties.getProperty(MINIFI_CONFIG_FILE_KEY));
+final Properties transformationConfigProperties =
getConfigTransformationProperties(bootstrapProperties);
+
+for(String name:
transformationConfigProperties.stringPropertyNames()) {
+defaultLogger.error("config property: name:[" + name + "]
value:[" + transformationConfigProperties.getProperty(name) + "] ");
--- End diff --
Most definitely better as `info`
---
[GitHub] nifi pull request #2637: NIFI-5085 In InvokeHttp, moving the OkHttp Response...
Github user JPercivall closed the pull request at: https://github.com/apache/nifi/pull/2637 ---
[GitHub] nifi issue #2637: NIFI-5085 In InvokeHttp, moving the OkHttp Response object...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2637 Will close, thanks for the review @mattyb149! For linkage: https://github.com/apache/nifi/commit/a0c9bebe241e8960363490cc0378f22d034f8db7 ---
[GitHub] nifi pull request #2637: NIFI-5085 In InvokeHttp, moving the OkHttp Response...
GitHub user JPercivall opened a pull request: https://github.com/apache/nifi/pull/2637 NIFI-5085 In InvokeHttp, moving the OkHttp Response object to a try w⦠â¦ith resources Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [X] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [X] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JPercivall/nifi NIFI-5085_closing_OkHttpResponse_in_InvokeHttp Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2637.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2637 commit a320601fbd20f9ae77044c9babe2728dc89d6426 Author: Joe Percivall <jpercivall@...> Date: 2018-04-17T02:36:26Z NIFI-5085 In InvokeHttp, moving the OkHttp Response object to a try with resources ---
[GitHub] nifi issue #2586: NIFI-5021 Moving nifi-elasticsearch-client-service-api to ...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2586 @bbende @MikeThomsen since there is another RC of 1.6.0, did you want to merge this in? ---
[GitHub] nifi issue #2547: NIFI-4977 Adding expression language support to the Sender...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2547 Awesome, thanks @bbende. I appreciate the review ---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 +1 Overall looks good and works as expected. I've built with contrib check and tested using a vanilla instance. I will squash and merge to master. Thanks @MikeThomsen. ---
[GitHub] nifi issue #2547: NIFI-4977 Adding expression language support to the Sender...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2547 @bbende I've added the changes and rebased to master ---
[GitHub] nifi pull request #2547: NIFI-4977 Adding expression language support to the...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2547#discussion_r177146046
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractSyslogProcessor.java
---
@@ -29,34 +29,28 @@
public static final AllowableValue TCP_VALUE = new
AllowableValue("TCP", "TCP");
public static final AllowableValue UDP_VALUE = new
AllowableValue("UDP", "UDP");
-public static final PropertyDescriptor PROTOCOL = new
PropertyDescriptor
+public static final PropertyDescriptor PROTOCOL_PROP = new
PropertyDescriptor
.Builder().name("Protocol")
.description("The protocol for Syslog communication.")
.required(true)
.allowableValues(TCP_VALUE, UDP_VALUE)
.defaultValue(UDP_VALUE.getValue())
.build();
-public static final PropertyDescriptor PORT = new PropertyDescriptor
+public static final PropertyDescriptor.Builder PORT_PROP_BUILDER = new
PropertyDescriptor
--- End diff --
Yeah, the reason was just that I was shying away from modifying the
ListenSyslog processor as well but that's probably a better way to go. Will fix
---
[GitHub] nifi pull request #2547: NIFI-4977 Adding expression language support to the...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2547#discussion_r177145747
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractSyslogProcessor.java
---
@@ -29,34 +29,28 @@
public static final AllowableValue TCP_VALUE = new
AllowableValue("TCP", "TCP");
public static final AllowableValue UDP_VALUE = new
AllowableValue("UDP", "UDP");
-public static final PropertyDescriptor PROTOCOL = new
PropertyDescriptor
+public static final PropertyDescriptor PROTOCOL_PROP = new
PropertyDescriptor
.Builder().name("Protocol")
.description("The protocol for Syslog communication.")
.required(true)
.allowableValues(TCP_VALUE, UDP_VALUE)
--- End diff --
Yup, you're right will adjust
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Looks good, was able to do a couple simple queries and aggregations. I wasn't able to test HTTPS though as I don't have access to the X-Pack. Were you able to do so @MikeThomsen? The only lingering issue, that I've mentioned multiple times now, is to add a note about the entire body of the response being loaded into memory. ---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Hey @MikeThomsen, just reviewed the changes. They look good and I think we're almost there. One thing that is missing is the comment regarding pulling the results into memory. Also, I tried to build NiFi and test the processor an instance but hit the following error when I attempted to start up NiFi (./bin/nifi.sh start). I tried rebasing it to the latest master but to no avail. > 2018-03-24 02:06:53,171 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.elasticsearch.JsonQueryElasticsearch could not be instantiated java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.elasticsearch.JsonQueryElasticsearch could not be instantiated at java.util.ServiceLoader.fail(ServiceLoader.java:232) at java.util.ServiceLoader.access$100(ServiceLoader.java:185) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at org.apache.nifi.nar.ExtensionManager.loadExtensions(ExtensionManager.java:148) at org.apache.nifi.nar.ExtensionManager.discoverExtensions(ExtensionManager.java:123) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:771) at org.apache.nifi.NiFi.(NiFi.java:157) at org.apache.nifi.NiFi.(NiFi.java:71) at org.apache.nifi.NiFi.main(NiFi.java:292) Caused by: java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper at org.apache.nifi.processors.elasticsearch.JsonQueryElasticsearch.(JsonQueryElasticsearch.java:206) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ... 8 common frames omitted Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.ObjectMapper at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 15 common frames omitted ---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Hey @MikeThomsen, I'm planning on reviewing this tomorrow evening ---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Hey @MikeThomsen, just finished up another round of reviewing. Again, mostly focused on the error handling portion. I hope my review doesn't come off in the wrong way, there is a lot of great work here and I just want to make sure the usability is top notch. Error handling is a huge focus in NiFi because we're writing up a toolbox of processors for potentially non-technical users to work with however they want. Those users need to be able to see what they messed up and what went wrong for efficient and effective rectification. Thanks for the hard work, and the willingness to iterate and learn! ---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175306495
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java
---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
--- End diff --
I don't see this in the latest commit
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175306340
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends
AbstractControllerService implements ElasticSearchClientService {
+private ObjectMapper mapper = new ObjectMapper();
+
+static final private List properties;
+
+private RestClient client;
+
+private String url;
+
+static {
+List _props = new ArrayList();
+_props.add(ElasticSearchClientService.HTTP_HOSTS);
+_props.add(ElasticSearchClientService.USERNAME);
+_props.add(ElasticSearchClientService.PASSWORD);
+_props.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+_props.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+_props.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+_props.add(ElasticSearchClientService.RETRY_TIMEOUT);
+
+properties = Collections.unmodifiableList(_props);
+}
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+return properties;
+}
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) throws
InitializationException {
+try {
+setupClient(context);
+} catch (Exception ex) {
+getLogger().error("Could not initialize ElasticSearch
client.", ex);
+throw new InitializationException(ex);
+}
+}
+
+@OnDisabled
+public void onDisabled() throws IOException {
+this.client.close();
+this.url = null;
+}
+
+private void setupClient(ConfigurationContext context) throws
MalformedURLException {
+final String hosts =
context.getProperty(HTTP_HOSTS).evaluateAttributeExpressions().getValue();
+String[] hostsSplit = hosts.split(",[\\s]*");
+this.url = hostsSplit[0];
+final SSLContextService sslService =
+
context.getProperty(PROP_
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175306133
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends
AbstractControllerService implements ElasticSearchClientService {
+private ObjectMapper mapper = new ObjectMapper();
+
+static final private List properties;
+
+private RestClient client;
+
+private String url;
+
+static {
+List _props = new ArrayList();
+_props.add(ElasticSearchClientService.HTTP_HOSTS);
+_props.add(ElasticSearchClientService.USERNAME);
+_props.add(ElasticSearchClientService.PASSWORD);
+_props.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+_props.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+_props.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+_props.add(ElasticSearchClientService.RETRY_TIMEOUT);
+
+properties = Collections.unmodifiableList(_props);
+}
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+return properties;
+}
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) throws
InitializationException {
+try {
+setupClient(context);
+} catch (Exception ex) {
+getLogger().error("Could not initialize ElasticSearch
client.", ex);
+throw new InitializationException(ex);
+}
+}
+
+@OnDisabled
+public void onDisabled() throws IOException {
+this.client.close();
+this.url = null;
+}
+
+private void setupClient(ConfigurationContext context) throws
MalformedURLException {
+final String hosts =
context.getProperty(HTTP_HOSTS).evaluateAttributeExpressions().getValue();
+String[] hostsSplit = hosts.split(",[\\s]*");
+this.url = hostsSplit[0];
+final SSLContextService sslService =
+
context.getProperty(PROP_
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175305604
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends
AbstractControllerService implements ElasticSearchClientService {
+private ObjectMapper mapper = new ObjectMapper();
+
+static final private List properties;
+
+private RestClient client;
+
+private String url;
+
+static {
+List _props = new ArrayList();
+_props.add(ElasticSearchClientService.HTTP_HOSTS);
+_props.add(ElasticSearchClientService.USERNAME);
+_props.add(ElasticSearchClientService.PASSWORD);
+_props.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+_props.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+_props.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+_props.add(ElasticSearchClientService.RETRY_TIMEOUT);
+
+properties = Collections.unmodifiableList(_props);
+}
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+return properties;
+}
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) throws
InitializationException {
+try {
+setupClient(context);
+} catch (Exception ex) {
+getLogger().error("Could not initialize ElasticSearch
client.", ex);
+throw new InitializationException(ex);
+}
+}
+
+@OnDisabled
+public void onDisabled() throws IOException {
+this.client.close();
+this.url = null;
+}
+
+private void setupClient(ConfigurationContext context) throws
MalformedURLException {
+final String hosts =
context.getProperty(HTTP_HOSTS).evaluateAttributeExpressions().getValue();
+String[] hostsSplit = hosts.split(",[\\s]*");
+this.url = hostsSplit[0];
+final SSLContextService sslService =
+
context.getProperty(PROP_
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175305459
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java
---
@@ -0,0 +1,329 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+publi
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175305389
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java
---
@@ -0,0 +1,329 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+publi
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175304957
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -61,22 +58,23 @@
public class ElasticSearchClientServiceImpl extends
AbstractControllerService implements ElasticSearchClientService {
private ObjectMapper mapper = new ObjectMapper();
-private List properties;
+static final private List properties;
private RestClient client;
private String url;
-@Override
-protected void init(ControllerServiceInitializationContext config) {
-properties = new ArrayList<>();
-properties.add(ElasticSearchClientService.HTTP_HOSTS);
-properties.add(ElasticSearchClientService.USERNAME);
-properties.add(ElasticSearchClientService.PASSWORD);
-
properties.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
-properties.add(ElasticSearchClientService.CONNECT_TIMEOUT);
-properties.add(ElasticSearchClientService.SOCKET_TIMEOUT);
-properties.add(ElasticSearchClientService.RETRY_TIMEOUT);
+static {
+List _props = new ArrayList();
--- End diff --
Bit of a nit pick but can this init be changed so it's not of raw type:
> List\<PropertyDescriptor\> _props = new ArrayList\<\>();
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175304809
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -121,7 +119,7 @@ private void setupClient(ConfigurationContext context)
throws Exception {
RestClientBuilder builder = RestClient.builder(hh)
.setHttpClientConfigCallback(httpClientBuilder -> {
-if (sslService != null) {
+if (sslService != null &&
sslService.isKeyStoreConfigured() && sslService.isTrustStoreConfigured()) {
try {
--- End diff --
Given that this is a callback, we probably want to move as much of the
logic out of here as possible. That way we can better handle any errors and
reduce performance cost if this is called multiple times. Currently, if there
is an error which hits line 145, the error will be caught and logged but it
will continue to create the http client as if nothing went wrong. What we
should probably do is have a final SSLContext which gets created above,
bubbling up any errors so that the CS doesn't enable in a bad state, and set
the SSLContext same way as below in the context.
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Taking a look now. WRT: > Ok, I can do that from now on. Shouldn't they be squashed before a merge into master? Yup, they will be squashed prior to merge to master but the reviewer can do that at the same time they add the comment to the commit to close the PR. ---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175265212
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java
---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
--- End diff --
Related to the note about pagination, should add something indicating that
very large searches should be done as the response is loaded into memory.
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175265070
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintWriter;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends
AbstractControllerService implements ElasticSearchClientService {
+private ObjectMapper mapper = new ObjectMapper();
+
+private List properties;
+
+private RestClient client;
+
+private String url;
+
+@Override
+protected void init(ControllerServiceInitializationContext config) {
+properties = new ArrayList<>();
+properties.add(ElasticSearchClientService.HTTP_HOSTS);
+properties.add(ElasticSearchClientService.USERNAME);
+properties.add(ElasticSearchClientService.PASSWORD);
+
properties.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+properties.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+properties.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+properties.add(ElasticSearchClientService.RETRY_TIMEOUT);
+}
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+return properties;
+}
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) throws
InitializationException {
+try {
+setupClient(context);
+} catch (Exception ex) {
+getLogger().error("Could not initialize ElasticSearch
client.", ex);
+throw new InitializationException(ex);
+}
+}
+
+@OnDisabled
+public void onDisabled() throws IOException {
+this.client.close();
+this.url = null;
+}
+
+private void setupClient(ConfigurationContext context) throws
Exception {
+final String hosts =
context.getProperty(HTTP_HOSTS).evaluateAttributeExpressions().getValue();
+Stri
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175265049
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintWriter;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends
AbstractControllerService implements ElasticSearchClientService {
+private ObjectMapper mapper = new ObjectMapper();
+
+private List properties;
+
+private RestClient client;
+
+private String url;
+
+@Override
+protected void init(ControllerServiceInitializationContext config) {
+properties = new ArrayList<>();
+properties.add(ElasticSearchClientService.HTTP_HOSTS);
+properties.add(ElasticSearchClientService.USERNAME);
+properties.add(ElasticSearchClientService.PASSWORD);
+
properties.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+properties.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+properties.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+properties.add(ElasticSearchClientService.RETRY_TIMEOUT);
+}
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+return properties;
+}
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) throws
InitializationException {
+try {
+setupClient(context);
+} catch (Exception ex) {
+getLogger().error("Could not initialize ElasticSearch
client.", ex);
+throw new InitializationException(ex);
+}
+}
+
+@OnDisabled
+public void onDisabled() throws IOException {
+this.client.close();
+this.url = null;
+}
+
+private void setupClient(ConfigurationContext context) throws
Exception {
+final String hosts =
context.getProperty(HTTP_HOSTS).evaluateAttributeExpressions().getValue();
+Stri
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175264989
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintWriter;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends
AbstractControllerService implements ElasticSearchClientService {
+private ObjectMapper mapper = new ObjectMapper();
+
+private List properties;
+
+private RestClient client;
+
+private String url;
+
+@Override
+protected void init(ControllerServiceInitializationContext config) {
+properties = new ArrayList<>();
+properties.add(ElasticSearchClientService.HTTP_HOSTS);
+properties.add(ElasticSearchClientService.USERNAME);
+properties.add(ElasticSearchClientService.PASSWORD);
+
properties.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+properties.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+properties.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+properties.add(ElasticSearchClientService.RETRY_TIMEOUT);
+}
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+return properties;
+}
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) throws
InitializationException {
+try {
+setupClient(context);
+} catch (Exception ex) {
+getLogger().error("Could not initialize ElasticSearch
client.", ex);
+throw new InitializationException(ex);
+}
+}
+
+@OnDisabled
+public void onDisabled() throws IOException {
+this.client.close();
+this.url = null;
+}
+
+private void setupClient(ConfigurationContext context) throws
Exception {
--- End diff --
Try to avoid "throws Exception". Looks like the only one thrown is
"MalformedURLException".
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175264961
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintWriter;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends
AbstractControllerService implements ElasticSearchClientService {
+private ObjectMapper mapper = new ObjectMapper();
+
+private List properties;
--- End diff --
Can this be declared similar to the properties in the processor and marked
final?
---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175264906
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java
---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+publi
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175264876
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java
---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+publi
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175264845
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java
---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+publi
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/2113#discussion_r175264784 --- Diff: nifi-nar-bundles/nifi-standard-services/pom.xml --- @@ -15,6 +15,18 @@ --> http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> 4.0.0 + --- End diff -- This dependencies section shouldn't be needed here. This is the parent pom for all of the standard services. ---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Thanks @MikeThomsen, looks like that fixed the NPE. Still needs the License and Notice in the client service nar. Also, it's a bit of an anti-pattern to have the logging and error handling in the Controller Service for methods called by the processor. This is because any time there is a problem with the search, the bulletin is logged at the Controller Service. This means that not only will a user not know which processor is the cause of the error (since it could be shared) but also when looking at the map the processor appears to be working without issue. Lastly, since the error isn't bubbled back to the processor, it never yields so if there is something wrong with the query it will continually error and log without pause. Related, there's also some weirdness with error handling when there is an incoming connection. Currently, input is only sent to Failure instead of Original when an error is caught at the top level. This means that if a query is messed up and is caught as a JsonProcessingException in the Controller service, the input will be routed to original and no other outputs. As a user, I would expect that flowfile to be sent to Failure. So in summary, the CS should let the calling processor decide how to handle any exceptions encountered during processing. This allows the processor to more granularlly decide what to do under different situations. For example, was it a JSON parsing exception and it's acting as a source processor? well we better yield the whole processor since it's probably just going to continually fail. Is there an incoming connection but some network error happened when searching? again, later FlowFiles will probably fail too, so we should yield. If you have any questions on when to yield the processor vs penalizing the FlowFile, feel free to ask. Lastly, it's preferred if you don't squash your commits every time. If you don't, that allows the reviewer to more easily see exactly what changed since they last reviewed it. Also allows reviewers to see how the PR evolved over time in response to different comments. ---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175249262
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java
---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+publi
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Apologises, won't be able to finish the review this evening. I reviewed the restapi nar License and Notice files they look good to go. Just need to do a bit of functional testing and verify @mattyb149 comments were addressed and we should be ready to merge (pending the L/N for the service nar). ---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 The nifi-elasticsearch-client-service-nar is missing a license and notice. ---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Thanks @MikeThomsen I should be able to take another look later tonight ---
[GitHub] nifi pull request #2547: NIFI-4977 Adding expression language support to the...
GitHub user JPercivall opened a pull request: https://github.com/apache/nifi/pull/2547 NIFI-4977 Adding expression language support to the Sender properties⦠⦠of PutSyslog Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [X] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [X] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JPercivall/nifi NIFI-4977_Exposing_EL_in_PutSyslog Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2547.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2547 commit e9b731529a243ec439760a758cb0c809f7f8cd93 Author: Joe Percivall <jpercivall@...> Date: 2018-03-15T00:39:04Z NIFI-4977 Adding expression language support to the Sender properties of PutSyslog ---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Github user JPercivall commented on the issue:
https://github.com/apache/nifi/pull/2113
I'm seeing NPE and FlowFile handling exceptions. I didn't do anything
special, just using the example query. I'll attach a template. Errors from the
logs below.
> 2018-03-12 23:58:11,386 ERROR [Timer-Driven Process Thread-10]
o.a.n.p.e.JsonQueryElasticsearch
JsonQueryElasticsearch[id=1d5d83d1-0162-1000-cdf2-92fa9e61ef42] Error
processing flowfile.: java.lang.NullPointerException
java.lang.NullPointerException: null
at
org.apache.nifi.processors.elasticsearch.JsonQueryElasticsearch.onTrigger(JsonQueryElasticsearch.java:248)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1123)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2018-03-12 23:58:11,389 ERROR [Timer-Driven Process Thread-10]
o.a.n.p.e.JsonQueryElasticsearch
JsonQueryElasticsearch[id=1d5d83d1-0162-1000-cdf2-92fa9e61ef42]
JsonQueryElasticsearch[id=1d5d83d1-0162-1000-cdf2-92fa9e61ef42] failed to
process due to org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=77ad5fff-d4b6-4e11-8d8c-b07ac1d0a6cb,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1520913052434-1, container=default,
section=1], offset=30, length=228],offset=226,name=46172028496405,size=2]
transfer relationship not specified; rolling back session: {}
org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=77ad5fff-d4b6-4e11-8d8c-b07ac1d0a6cb,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1520913052434-1, container=default,
section=1], offset=30, length=228],offset=226,name=46172028496405,size=2]
transfer relationship not specified
at
org.apache.nifi.controller.repository.StandardProcessSession.checkpoint(StandardProcessSession.java:251)
at
org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:321)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1123)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Template for FlowFile handling exception. [FlowFile_Handling_Exception.txt](https://github.com/apache/nifi/files/1805420/FlowFile_Handling_Exception.txt) ---
[GitHub] nifi pull request #2113: NIFI-4325 Added new processor that uses the JSON DS...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/2113#discussion_r173996495 --- Diff: nifi-assembly/pom.xml --- @@ -445,6 +445,24 @@ language governing permissions and limitations under the License. --> 1.6.0-SNAPSHOT nar + +org.apache.nifi + nifi-elasticsearch-client-service-api-nar --- End diff -- My build is failing due to this. I don't see this artifact created anywhere. ---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Looks like the master build is failing with some check style issues, hence why your builds failed. I'll work on fixing them. ---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Reviewing ---
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Hey @mattyb149, what's your status for this review? From a cursory look, it appears like it just needs an updated pom from @MikeThomsen and the final approval. With talks of 1.6.0 happening it would be nice to get this in so those using ES 6 aren't limited to the HTTP processor. If help is needed to finalize things just let me know where I can help. ---
[GitHub] nifi-minifi pull request #117: MINIFI-424 Adding the ability to evaluate boo...
GitHub user JPercivall opened a pull request: https://github.com/apache/nifi-minifi/pull/117 MINIFI-424 Adding the ability to evaluate bootstrap properties in the⦠⦠config YAML prior to translation into flow.xml and nifi.properties Thank you for submitting a contribution to Apache NiFi - MiNiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi-minifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] If applicable, have you updated the LICENSE file, including the main LICENSE file under minifi-assembly? - [X] If applicable, have you updated the NOTICE file, including the main NOTICE file found under minifi-assembly? ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JPercivall/nifi-minifi MINIFI-424 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi/pull/117.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #117 commit 959a7cfffe76ab3795a1bfa361c3580f1f194775 Author: Joe Percivall <jpercivall@...> Date: 2018-02-28T02:44:39Z MINIFI-424 Adding the ability to evaluate bootstrap properties in the config YAML prior to translation into flow.xml and nifi.properties ---
[GitHub] nifi issue #2219: NiFi-4436: Add UI controls for starting/stopping/reverting...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2219 Small bug with the visuals on nested flows: - Start with a versioned flow that contains a nested versioned flow. Both are up to date. - Make a change to the nested flow. - Notice that it correctly has the "*" saying it has local uncommitted changes. - Go to the top versioned flow, notice that it says that it is still up to date. This is different than if the nested flow wasn't versioned. In that case, the top level properly shows the "*". Fortunately, if I go to commit the top level versioned flow it correctly tells me I can't because I have a nested versioned flow with uncommitted changes. ---
[GitHub] nifi issue #2219: NiFi-4436: Add UI controls for starting/stopping/reverting...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2219 Not sure if I should comment here or capture in a new ticket since this PR is so big but from the Registry Clients page, it would be nice to have an icon to click to open the Registry in a new tab. ---
[GitHub] nifi issue #2337: NIFI-4475 Changing the get(batchSize) method in StandardPr...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2337 Fantastic thanks @markap14 ---
[GitHub] nifi issue #2337: NIFI-4475 Changing the get(batchSize) method in StandardPr...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2337 Thanks for the review @markap14, I pushed out a new commit with those two changes. ---
[GitHub] nifi pull request #2337: NIFI-4475 Changing the get(batchSize) method in Sta...
GitHub user JPercivall opened a pull request: https://github.com/apache/nifi/pull/2337 NIFI-4475 Changing the get(batchSize) method in StandardProcessSessio⦠â¦n so that it checks all connections before returning nothing Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [X] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [X] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JPercivall/nifi NIFI-4475_Making_session_get_x_round_robin_queues Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2337.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2337 commit 5307d406374c5448d1913a3130459b6724104b10 Author: Joe Percivall <jperciv...@apache.org> Date: 2017-12-13T22:17:05Z NIFI-4475 Changing the get(batchSize) method in StandardProcessSession so that it checks all connections before returning nothing ---
[GitHub] nifi-minifi pull request #103: MINIFI-417 Upgrading underlying NiFi libs to ...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/103#discussion_r156687353
--- Diff:
minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/util/ConfigTransformer.java
---
@@ -77,7 +77,7 @@
public final class ConfigTransformer {
// Underlying version of NIFI will be using
-public static final String NIFI_VERSION = "1.2.0";
+public static final String NIFI_VERSION = "1.4.0";
--- End diff --
There appear to be a couple of different ways[1] but most of them are
pretty hacky and not sure it's worth putting in for this one variable. If the
variable was logic that was relied on for MiNiFi to work I'd probably lean
towards adding it but it's currently only used in one place[2] and that is just
for documentation within a file that in general isn't viewed by users (the
underlying nifi.properties).
[1]
https://stackoverflow.com/questions/11500533/access-maven-properties-defined-in-the-pom
[2]
https://github.com/JPercivall/nifi-minifi/blob/baac89b1ddd7a019635bd150c0cce9a37314a371/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/util/ConfigTransformer.java#L163
---
[GitHub] nifi-minifi pull request #103: MINIFI-417 Upgrading underlying NiFi libs to ...
GitHub user JPercivall opened a pull request: https://github.com/apache/nifi-minifi/pull/103 MINIFI-417 Upgrading underlying NiFi libs to 1.4.0 from 1.2.0 Thank you for submitting a contribution to Apache NiFi - MiNiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi-minifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] If applicable, have you updated the LICENSE file, including the main LICENSE file under minifi-assembly? - [X] If applicable, have you updated the NOTICE file, including the main NOTICE file found under minifi-assembly? ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JPercivall/nifi-minifi MINIFI-417-nifi_1.2.0-deps Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi/pull/103.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #103 commit baac89b1ddd7a019635bd150c0cce9a37314a371 Author: Joe Percivall <jperciv...@apache.org> Date: 2017-12-13T06:23:22Z MINIFI-417 Upgrading underlying NiFi libs to 1.4.0 from 1.2.0 ---
[GitHub] nifi issue #2154: NIFI-2663: Add WebSocket support for MQTT processors
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2154 Hello Sebastian, yup that subclassing strategy makes sense. ---
[GitHub] nifi issue #2154: NIFI-2663: Add WebSocket support for MQTT processors
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2154 @SebastianCarroll the way the test cases are set up you don't need to copy and paste the tests themselves. What you'll do is extend common/Test*MqttCommon in order to get all the junit tests. See [TestPublishMQTT](https://github.com/apache/nifi/blob/7923fd04c35737df8145b213536bdf333ef72713/nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/test/java/org/apache/nifi/processors/mqtt/integration/TestPublishMQTT.java#L38) in the integration folder as an example. While it may be just a 3 letter change on our end, you're changing a core configuration of the underlying implementation. So verifying that is helpful. Also, what is the purpose of the changes you've made to MqttTestClient.java and where did you copy & paste the comments/code from? Lastly, you have some check style issues (can be seen by running "mvn clean install -Pcontrib-check"). Here's what I see: `[WARNING] src/main/java/org/apache/nifi/processors/mqtt/common/AbstractMQTTProcessor.java[104:52] (whitespace) OperatorWrap: '||' should be on a new line. [WARNING] src/main/java/org/apache/nifi/processors/mqtt/common/AbstractMQTTProcessor.java[105:53] (whitespace) OperatorWrap: '||' should be on a new line. [WARNING] src/main/java/org/apache/nifi/processors/mqtt/common/AbstractMQTTProcessor.java[106:52] (whitespace) OperatorWrap: '||' should be on a new line. [WARNING] src/main/java/org/apache/nifi/processors/mqtt/common/AbstractMQTTProcessor.java[296:62] (whitespace) OperatorWrap: '||' should be on a new line. [WARNING] src/test/java/org/apache/nifi/processors/mqtt/common/MqttTestClient.java[20] (imports) AvoidStarImport: Using the '.*' form of import should be avoided - org.eclipse.paho.client.mqttv3.*. [WARNING] src/test/java/org/apache/nifi/processors/mqtt/common/MqttTestClient.java[375:43] (whitespace) FileTabCharacter: Line contains a tab character. [WARNING] src/test/java/org/apache/nifi/processors/mqtt/common/MqttTestClient.java[476:43] (whitespace) FileTabCharacter: Line contains a tab character.` ---
[GitHub] nifi issue #2004: NIFI-2162 Updating OkHttp to 3.8.1 and OkHttp-Digest to 1....
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2004 Thanks for the review @trkurc, sorry it took me so long. ---
[GitHub] nifi issue #2004: NIFI-2162 Updating OkHttp to 3.8.1 and OkHttp-Digest to 1....
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2004 @trkurc finally got around to adding comments. I also rebased to the latest master ---
[GitHub] nifi issue #2004: NIFI-2162 Updating OkHttp to 3.8.1 and OkHttp-Digest to 1....
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2004 Thanks for the review @trkurc and sorry for the delay in my responses. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #2004: NIFI-2162 Updating OkHttp to 3.8.1 and OkHttp-Diges...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2004#discussion_r132318693
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
---
@@ -500,48 +512,88 @@ public void onPropertyModified(final
PropertyDescriptor descriptor, final String
}
@OnScheduled
-public void setUpClient(final ProcessContext context) throws
IOException {
+public void setUpClient(final ProcessContext context) throws
IOException, UnrecoverableKeyException, CertificateException,
NoSuchAlgorithmException, KeyStoreException, KeyManagementException {
okHttpClientAtomicReference.set(null);
-OkHttpClient okHttpClient = new OkHttpClient();
+OkHttpClient.Builder okHttpClientBuilder = new
OkHttpClient().newBuilder();
// Add a proxy if set
final String proxyHost =
context.getProperty(PROP_PROXY_HOST).getValue();
final Integer proxyPort =
context.getProperty(PROP_PROXY_PORT).asInteger();
if (proxyHost != null && proxyPort != null) {
final Proxy proxy = new Proxy(Type.HTTP, new
InetSocketAddress(proxyHost, proxyPort));
-okHttpClient.setProxy(proxy);
+okHttpClientBuilder.proxy(proxy);
}
// Set timeouts
-
okHttpClient.setConnectTimeout((context.getProperty(PROP_CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue()),
TimeUnit.MILLISECONDS);
-
okHttpClient.setReadTimeout(context.getProperty(PROP_READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue(),
TimeUnit.MILLISECONDS);
+
okHttpClientBuilder.connectTimeout((context.getProperty(PROP_CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue()),
TimeUnit.MILLISECONDS);
+
okHttpClientBuilder.readTimeout(context.getProperty(PROP_READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue(),
TimeUnit.MILLISECONDS);
// Set whether to follow redirects
-
okHttpClient.setFollowRedirects(context.getProperty(PROP_FOLLOW_REDIRECTS).asBoolean());
+
okHttpClientBuilder.followRedirects(context.getProperty(PROP_FOLLOW_REDIRECTS).asBoolean());
final SSLContextService sslService =
context.getProperty(PROP_SSL_CONTEXT_SERVICE).asControllerService(SSLContextService.class);
final SSLContext sslContext = sslService == null ? null :
sslService.createSSLContext(ClientAuth.NONE);
// check if the ssl context is set and add the factory if so
if (sslContext != null) {
-
okHttpClient.setSslSocketFactory(sslContext.getSocketFactory());
+setSslSocketFactory(okHttpClientBuilder, sslService,
sslContext);
}
// check the trusted hostname property and override the
HostnameVerifier
String trustedHostname =
trimToEmpty(context.getProperty(PROP_TRUSTED_HOSTNAME).getValue());
if (!trustedHostname.isEmpty()) {
-okHttpClient.setHostnameVerifier(new
OverrideHostnameVerifier(trustedHostname, okHttpClient.getHostnameVerifier()));
+okHttpClientBuilder.hostnameVerifier(new
OverrideHostnameVerifier(trustedHostname, OkHostnameVerifier.INSTANCE));
}
-setAuthenticator(okHttpClient, context);
+setAuthenticator(okHttpClientBuilder, context);
useChunked =
context.getProperty(PROP_USE_CHUNKED_ENCODING).asBoolean();
-okHttpClientAtomicReference.set(okHttpClient);
+okHttpClientAtomicReference.set(okHttpClientBuilder.build());
+}
+
+private void setSslSocketFactory(OkHttpClient.Builder
okHttpClientBuilder, SSLContextService sslService, SSLContext sslContext)
+throws IOException, KeyStoreException, CertificateException,
NoSuchAlgorithmException, UnrecoverableKeyException, KeyManagementException {
+final String keystoreLocation = sslService.getKeyStoreFile();
+final String keystorePass = sslService.getKeyStorePassword();
+final String keystoreType = sslService.getKeyStoreType();
+
+// prepare the keystore
+final KeyStore keyStore = KeyStore.getInstance(keystoreType);
+
+try (FileInputStream keyStoreStream = new
FileInputStream(keystoreLocation)) {
+keyStore.load(keyStoreStream, keystorePass.toCharArray());
+}
+
+final KeyManagerFactory keyManagerFactory =
KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
+keyManagerFactory.init(keyStore, keystorePass.toCharArray());
+
+// load truststore
+final String truststoreLocation = sslService.getTrustS
[GitHub] nifi pull request #2004: NIFI-2162 Updating OkHttp to 3.8.1 and OkHttp-Diges...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2004#discussion_r132318357
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/ProxyAuthenticator.java
---
@@ -16,35 +16,28 @@
*/
package org.apache.nifi.processors.standard.util;
+import okhttp3.Authenticator;
+import okhttp3.Credentials;
+import okhttp3.Request;
+import okhttp3.Response;
+import okhttp3.Route;
+
+import javax.annotation.Nullable;
import java.io.IOException;
-import java.net.Proxy;
-import java.util.HashMap;
-import java.util.Map;
-import com.burgstaller.okhttp.DispatchingAuthenticator;
-import com.squareup.okhttp.Authenticator;
-import com.squareup.okhttp.Credentials;
-import com.squareup.okhttp.Request;
-import com.squareup.okhttp.Response;
+public class ProxyAuthenticator implements Authenticator {
--- End diff --
I was targeting 1.4.0
I understand that it's a public class but I wouldn't consider it part of
the public API. Looking at the items explicitly listed that fall under the
public API, the closest one I see is "Any extension such as Processor,
Controller Service, Reporting Task." I see it falling outside of that though
since it's not a component itself. In what way do you see it falling under the
items listed as our public API?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #2004: NIFI-2162 Updating OkHttp to 3.8.1 and OkHttp-Diges...
GitHub user JPercivall opened a pull request: https://github.com/apache/nifi/pull/2004 NIFI-2162 Updating OkHttp to 3.8.1 and OkHttp-Digest to 1.13 and refa⦠â¦ctoring InvokeHttp to adjust for changes Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [X] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [X] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JPercivall/nifi NIFI-2162 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2004.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2004 commit b67ca1532765e8296442e78e7d6a781bed1167d9 Author: Joe Percivall <jperciv...@apache.org> Date: 2017-07-13T03:16:15Z NIFI-2162 Updating OkHttp to 3.8.1 and OkHttp-Digest to 1.13 and refactoring InvokeHttp to adjust for changes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1984: NIFI-4155: Expand EnforceOrder capability to cluster
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/1984 @ijokarumawak ah I remember seeing that come up but didn't realize it was finished. That's awesome! Definitely agree with your second statement but I'd add that we should warn the users of the danger in the description of the cluster state option here and potentially add it to the processor capability description too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1984: NIFI-4155: Expand EnforceOrder capability to cluster
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/1984 @ijokarumawak while we are still using ZooKeeper as the only cluster state provider and using it for the cluster coordinator, are we sure we want to add this option? This goes hand-in-hand with the work to add state to UpdateAttribute, specifically this comment[1]. @markap14 do you have opinions on this? [1] https://issues.apache.org/jira/browse/NIFI-1582 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1922: NIFI-4075 Updating 'addRemoteProcessGroup' in Stand...
GitHub user JPercivall opened a pull request: https://github.com/apache/nifi/pull/1922 NIFI-4075 Updating 'addRemoteProcessGroup' in StandardFlowSerializer ⦠â¦to properly use ScheduledStateLookup Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [X] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [X] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JPercivall/nifi NIFI-4075 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1922.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1922 commit 0a837687bdd293700ea1ae93a08883fcb028129e Author: Joe Percivall <jperciv...@apache.org> Date: 2017-06-16T16:39:26Z NIFI-4075 Updating 'addRemoteProcessGroup' in StandardFlowSerializer to properly use ScheduledStateLookup --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1907: NIFI-4047: Addressed issue with inconsistent Lexical polic...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/1907 +1 Contrib check build, tested the template and other potential quoting/character related issues and all worked as expected. The only thing of note that I found is that the Sql Parser for the validator is "similar to Oracle" while the lexer we're using is "MYSQL_ANSI". There was no apparent issue but something we may want to address/evaluate outside this ticket. I will merge this in, thanks @markap14 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1907: NIFI-4047: Addressed issue with inconsistent Lexical polic...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/1907 Reviewing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1553: NIFI-1449 - Migrate PutEmail tests from Mock class to List...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/1553 Also the travis failures appear to be unrelated but there are merge conflicts. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1553: NIFI-1449 - Migrate PutEmail tests from Mock class ...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1553#discussion_r114464910
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestPutEmail.java
---
@@ -17,83 +17,64 @@
package org.apache.nifi.processors.standard;
import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertNull;
+
import static org.junit.Assert.assertTrue;
-import java.io.InputStream;
-import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import javax.mail.BodyPart;
-import javax.mail.Message;
-import javax.mail.MessagingException;
-import javax.mail.internet.MimeMessage.RecipientType;
-import javax.mail.internet.MimeMultipart;
-
import org.apache.commons.codec.binary.Base64;
import org.apache.commons.codec.binary.StringUtils;
-import org.apache.commons.io.IOUtils;
+import org.apache.nifi.processors.email.ExtractEmailAttachments;
+import org.apache.nifi.processors.email.ExtractEmailHeaders;
+import org.apache.nifi.processors.email.ListenSMTP;
+import org.apache.nifi.remote.io.socket.NetworkUtils;
import org.apache.nifi.util.LogMessage;
+import org.apache.nifi.util.MockFlowFile;
import org.apache.nifi.util.TestRunner;
import org.apache.nifi.util.TestRunners;
+import org.hamcrest.CoreMatchers;
+import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-public class TestPutEmail {
-/**
- * Extension to PutEmail that stubs out the calls to
- * Transport.sendMessage().
- *
- *
- * All sent messages are records in a list available via the
- * {@link #getMessages()} method.
- * Calling
- * {@link #setException(MessagingException)} will cause the supplied
exception to be
- * thrown when sendMessage is invoked.
- *
- */
-private static final class PutEmailExtension extends PutEmail {
-private MessagingException e;
-private ArrayList messages = new ArrayList<>();
-
-@Override
-protected void send(Message msg) throws MessagingException {
-messages.add(msg);
-if (this.e != null) {
-throw e;
-}
-}
-
-void setException(final MessagingException e) {
-this.e = e;
-}
-
-List getMessages() {
-return messages;
-}
-}
-PutEmailExtension processor;
+public class TestPutEmail {
+
TestRunner runner;
+TestRunner smtpRunner;
+TestRunner extractHeadersRunner;
+TestRunner extractAttachmentsRunner;
+int port;
@Before
public void setup() {
-processor = new PutEmailExtension();
-runner = TestRunners.newTestRunner(processor);
+runner = TestRunners.newTestRunner(PutEmail.class);
+smtpRunner = TestRunners.newTestRunner(ListenSMTP.class);
+extractHeadersRunner =
TestRunners.newTestRunner(ExtractEmailHeaders.class);
+extractAttachmentsRunner =
TestRunners.newTestRunner(ExtractEmailAttachments.class);
+
+port = NetworkUtils.availablePort();
+smtpRunner.setProperty("SMTP_PORT", String.valueOf(port));
+smtpRunner.setProperty("SMTP_MAXIMUM_CONNECTIONS", "3");
+smtpRunner.setProperty("SMTP_TIMEOUT", "10 seconds");
+smtpRunner.run(1,false);
--- End diff --
I believe this line is the only one that needs to be run before each method
and the rest can be run before the class, specifically the network port.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1705: NIFI-3633 Adding a try with resources so the OkHttp...
GitHub user JPercivall opened a pull request: https://github.com/apache/nifi/pull/1705 NIFI-3633 Adding a try with resources so the OkHttp response is prope⦠â¦rly closed Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JPercivall/nifi NIFI-3633_2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1705.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1705 commit 102fbb6facd32525c56901ffd7b2ae1c3002a03b Author: Joe Percivall <jperciv...@apache.org> Date: 2017-04-27T02:23:30Z NIFI-3633 Adding a try with resources so the OkHttp response is properly closed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1576: NIFI-3518 Create a Morphlines processor
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1576#discussion_r110553180
--- Diff:
nifi-nar-bundles/nifi-morphlines-bundle/nifi-morphlines-processors/src/main/java/org/apache/nifi/processors/morphlines/ImplementMorphlines.java
---
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.morphlines;
+
+import com.google.common.base.Preconditions;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.*;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.kitesdk.morphline.api.Command;
+import org.kitesdk.morphline.api.MorphlineContext;
+import org.kitesdk.morphline.api.Record;
+import org.kitesdk.morphline.base.Fields;
+
+import org.kitesdk.morphline.base.Notifications;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.*;
+import java.util.stream.*;
+import java.util.concurrent.atomic.*;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.processor.exception.*;
+
+@Tags({"kitesdk", "morphlines", "ETL", "HDFS", "avro", "Solr", "HBase"})
+@CapabilityDescription("Implements Morphlines
(http://kitesdk.org/docs/1.1.0/morphlines/) framework, which performs in-memory
container of transformation commands in oder to perform tasks such as loading,
parsing, transforming, or otherwise processing a single record.")
+public class MorphlinesProcessor extends AbstractProcessor {
+
+private Command morphline;
+private volatile Record record = new Record();
+private volatile Collector collector = new Collector();
+
+public static final PropertyDescriptor MORPHLINES_ID = new
PropertyDescriptor
+.Builder().name("Morphlines ID")
+.description("Identifier of the morphlines context")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+
+public static final PropertyDescriptor MORPHLINES_FILE = new
PropertyDescriptor
+.Builder().name("Morphlines File")
+.description("File for the morphlines context")
+.required(true)
+.addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+
+public static final PropertyDescriptor MORPHLINES_OUTPUT_FIELD = new
PropertyDescriptor
+.Builder().name("Morphlines output field")
+.description("Field name of output in Morphlines. Default is
'_attachment_body'.")
+.required(false)
+ .expressionLanguageSupported(true)
+.defaultValue("_attachment_body")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final Relationship REL_SUCCESS = new
Relationship.Builder()
+
[GitHub] nifi pull request #1576: NIFI-3518 Create a Morphlines processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/1576#discussion_r110552745 --- Diff: nifi-nar-bundles/nifi-morphlines-bundle/nifi-morphlines-nar/pom.xml --- @@ -0,0 +1,41 @@ + + +http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> +4.0.0 + + +org.apache.com --- End diff -- This is wrong, should be "org.apache.nifi". --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1576: NIFI-3518 Create a Morphlines processor
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1576#discussion_r110553085
--- Diff:
nifi-nar-bundles/nifi-morphlines-bundle/nifi-morphlines-processors/src/main/java/org/apache/nifi/processors/morphlines/ImplementMorphlines.java
---
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.morphlines;
+
+import com.google.common.base.Preconditions;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.*;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.kitesdk.morphline.api.Command;
+import org.kitesdk.morphline.api.MorphlineContext;
+import org.kitesdk.morphline.api.Record;
+import org.kitesdk.morphline.base.Fields;
+
+import org.kitesdk.morphline.base.Notifications;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.*;
+import java.util.stream.*;
+import java.util.concurrent.atomic.*;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.processor.exception.*;
+
+@Tags({"kitesdk", "morphlines", "ETL", "HDFS", "avro", "Solr", "HBase"})
+@CapabilityDescription("Implements Morphlines
(http://kitesdk.org/docs/1.1.0/morphlines/) framework, which performs in-memory
container of transformation commands in oder to perform tasks such as loading,
parsing, transforming, or otherwise processing a single record.")
+public class MorphlinesProcessor extends AbstractProcessor {
--- End diff --
Please change the file name or classname in order to match.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1576: NIFI-3518 Create a Morphlines processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/1576#discussion_r110552988 --- Diff: nifi-nar-bundles/nifi-morphlines-bundle/nifi-morphlines-nar/pom.xml --- @@ -0,0 +1,41 @@ + + +http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> +4.0.0 + + +org.apache.com +nifi-morphlines-bundle +1.2.0-SNAPSHOT + + +nifi-morphlines-nar +1.2.0-SNAPSHOT +nar + +true +true + + + + +org.apache.nifi +nifi-morphlines-processors --- End diff -- You need to add a license/notice file for this nar and add any needed additions to the nifi-assembly license/notice files. Every dependency you bring in (including their transitive deps) need to be accounted for. You can see everything you're bringing in by running "mvn dependency:tree -Dverbose". Our licensing guide is here: https://nifi.apache.org/licensing-guide.html --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1576: NIFI-3518 Create a Morphlines processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/1576#discussion_r110552955 --- Diff: nifi-nar-bundles/nifi-morphlines-bundle/nifi-morphlines-nar/pom.xml --- @@ -0,0 +1,41 @@ + + +http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> +4.0.0 + + +org.apache.com +nifi-morphlines-bundle +1.2.0-SNAPSHOT + + +nifi-morphlines-nar --- End diff -- In order for the nifi-assembly pom to know to include it into the final package, you need to add it as a dependency of nifi-assembly/pom.xml. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1576: NIFI-3518 Create a Morphlines processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/1576#discussion_r110553363 --- Diff: nifi-nar-bundles/nifi-morphlines-bundle/nifi-morphlines-processors/pom.xml --- @@ -0,0 +1,110 @@ + + +http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> +4.0.0 + + +org.apache.nifi +nifi-morphlines-bundle +1.2.0-SNAPSHOT + + +nifi-morphlines-processors +jar + + + +org.apache.nifi +nifi-api + + +org.apache.nifi +nifi-processor-utils + + +org.apache.nifi +nifi-mock +test + + +org.slf4j +slf4j-simple +test + + +junit +junit +4.11 +test + + +org.kitesdk +kite-morphlines-core + + +org.kitesdk +kite-morphlines-avro + + +org.kitesdk +kite-morphlines-json + + +org.kitesdk +kite-morphlines-saxon + + +org.kitesdk +kite-morphlines-hadoop-core + + +org.kitesdk +kite-morphlines-hadoop-parquet-avro + + +org.kitesdk +kite-morphlines-hadoop-sequencefile + + +org.kitesdk +kite-morphlines-hadoop-rcfile + + +org.kitesdk +kite-morphlines-tika-core + + +org.kitesdk +kite-morphlines-tika-decompress + + +org.kitesdk +kite-morphlines-twitter + + +org.kitesdk +kite-morphlines-maxmind + + +org.kitesdk +kite-morphlines-metrics-servlets + + +org.kitesdk +kite-morphlines-useragent --- End diff -- I'm a bit worried that importing all of these dependencies is a too broad in scope. From my understanding not all of them are inherently needed but instead represent a toolbox of things that could potentially be used. I'd much prefer we limit this to a core group of functionality that we can't already do in NiFi instead. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1576: NIFI-3518 Create a Morphlines processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/1576#discussion_r110552730 --- Diff: nifi-nar-bundles/nifi-morphlines-bundle/nifi-morphlines-processors/pom.xml --- @@ -0,0 +1,110 @@ + + +http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> +4.0.0 + + +org.apache.nifi +nifi-morphlines-bundle +1.2.0-SNAPSHOT + + +nifi-morphlines-processors +jar + + + +org.apache.nifi +nifi-api + + +org.apache.nifi +nifi-processor-utils + + +org.apache.nifi +nifi-mock +test + + +org.slf4j +slf4j-simple +test + + +junit +junit +4.11 +test + + +org.kitesdk +kite-morphlines-core --- End diff -- All of these dependencies are missing versions. I can't build it at all. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1576: NIFI-3518 Create a Morphlines processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/1576#discussion_r110553041 --- Diff: nifi-nar-bundles/nifi-morphlines-bundle/nifi-morphlines-processors/src/main/java/org/apache/nifi/processors/morphlines/ImplementMorphlines.java --- @@ -0,0 +1,219 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.morphlines; + +import com.google.common.base.Preconditions; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.PropertyValue; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.*; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.io.OutputStreamCallback; +import org.apache.nifi.processor.io.StreamCallback; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.stream.io.StreamUtils; +import org.kitesdk.morphline.api.Command; +import org.kitesdk.morphline.api.MorphlineContext; +import org.kitesdk.morphline.api.Record; +import org.kitesdk.morphline.base.Fields; + +import org.kitesdk.morphline.base.Notifications; +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableSet; +import org.apache.nifi.annotation.lifecycle.OnScheduled; + +import java.io.File; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.util.*; +import java.util.stream.*; +import java.util.concurrent.atomic.*; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.processor.exception.*; --- End diff -- Please run "mvn clean install -Pcontrib-check" in order to see all your checkstyle issues. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1576: NIFI-3518 Create a Morphlines processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/1576#discussion_r110552680 --- Diff: nifi-nar-bundles/nifi-morphlines-bundle/pom.xml --- @@ -0,0 +1,35 @@ + + +http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> +4.0.0 + + +org.apache.nifi +nifi-nar-bundles +1.1.1 --- End diff -- You didn't address this comment in the previous PR[1]. This version should be 1.2.0-SNAPSHOT to match the others. [1] https://github.com/apache/nifi/pull/1529#discussion_r102965258 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1576: NIFI-3518 Create a Morphlines processor
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1576#discussion_r110552773
--- Diff:
nifi-nar-bundles/nifi-morphlines-bundle/nifi-morphlines-processors/src/main/java/org/apache/nifi/processors/morphlines/ImplementMorphlines.java
---
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.morphlines;
+
+import com.google.common.base.Preconditions;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.*;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.kitesdk.morphline.api.Command;
+import org.kitesdk.morphline.api.MorphlineContext;
+import org.kitesdk.morphline.api.Record;
+import org.kitesdk.morphline.base.Fields;
+
+import org.kitesdk.morphline.base.Notifications;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.*;
+import java.util.stream.*;
+import java.util.concurrent.atomic.*;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.processor.exception.*;
+
+@Tags({"kitesdk", "morphlines", "ETL", "HDFS", "avro", "Solr", "HBase"})
+@CapabilityDescription("Implements Morphlines
(http://kitesdk.org/docs/1.1.0/morphlines/) framework, which performs in-memory
container of transformation commands in oder to perform tasks such as loading,
parsing, transforming, or otherwise processing a single record.")
+public class MorphlinesProcessor extends AbstractProcessor {
+
+private Command morphline;
+private volatile Record record = new Record();
+private volatile Collector collector = new Collector();
+
+public static final PropertyDescriptor MORPHLINES_ID = new
PropertyDescriptor
+.Builder().name("Morphlines ID")
+.description("Identifier of the morphlines context")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+
+public static final PropertyDescriptor MORPHLINES_FILE = new
PropertyDescriptor
+.Builder().name("Morphlines File")
+.description("File for the morphlines context")
+.required(true)
+.addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+
+public static final PropertyDescriptor MORPHLINES_OUTPUT_FIELD = new
PropertyDescriptor
+.Builder().name("Morphlines output field")
+.description("Field name of output in Morphlines. Default is
'_attachment_body'.")
+.required(false)
+ .expressionLanguageSupported(true)
+.defaultValue("_attachment_body")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final Relationship REL_SUCCESS = new
Relationship.Builder()
+
[GitHub] nifi issue #1576: NIFI-3518 Create a Morphlines processor
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/1576 Sorry for the delay @WilliamNouet . I am going to look at it now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1610: NIFI-3633 Adding HttpNotificationService.
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/1610#discussion_r110550161 --- Diff: nifi-bootstrap/pom.xml --- @@ -34,13 +34,26 @@ language governing permissions and limitations under the License. --> nifi-utils +org.apache.nifi +nifi-security-utils + + javax.mail mail org.apache.nifi nifi-expression-language + +com.squareup.okhttp3 --- End diff -- Yup when doing a "mvn dependency:tree -Dverbose" the only related deps are: ``` +- com.squareup.okhttp3:okhttp:jar:3.6.0:compile | \- com.squareup.okio:okio:jar:1.11.0:compile +- com.squareup.okhttp3:mockwebserver:jar:3.6.0:test | +- (com.squareup.okhttp3:okhttp:jar:3.6.0:test - omitted for duplicate) | +- (org.bouncycastle:bcprov-jdk15on:jar:1.55:test - version managed from 1.50; omitted for duplicate) | \- (junit:junit:jar:4.12:test - omitted for duplicate) ``` The only other compile related one is "okio" which is offered under the same license: https://github.com/square/okio --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1610: NIFI-3633 Adding HttpNotificationService.
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/1610#discussion_r110550645 --- Diff: nifi-bootstrap/pom.xml --- @@ -34,13 +34,26 @@ language governing permissions and limitations under the License. --> nifi-utils +org.apache.nifi --- End diff -- It was already added to the RPM[1] even though it wasn't included in the bootstrap pom. Its transitive deps are also already included: ``` +- org.apache.nifi:nifi-security-utils:jar:1.2.0-SNAPSHOT:compile | +- (org.apache.nifi:nifi-api:jar:1.2.0-SNAPSHOT:compile - omitted for duplicate) | +- (org.apache.nifi:nifi-utils:jar:1.2.0-SNAPSHOT:compile - omitted for duplicate) | +- org.apache.commons:commons-lang3:jar:3.4:compile | +- org.bouncycastle:bcprov-jdk15on:jar:1.55:compile | \- org.bouncycastle:bcpkix-jdk15on:jar:1.55:compile | \- (org.bouncycastle:bcprov-jdk15on:jar:1.55:compile - omitted for duplicate) ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1610: NIFI-3633 Adding HttpNotificationService.
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1610#discussion_r110550466
--- Diff:
nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/NotificationServiceManager.java
---
@@ -420,15 +424,15 @@ public static Element getChild(final Element element,
final String tagName) {
}
private static class ConfiguredNotificationService {
-private final NotificationService service;
+private final AbstractNotificationService service;
--- End diff --
Hmm, I thought I had a good reason, and maybe in some iteration there was,
but it appears there isn't anymore. I will revert.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1610: NIFI-3633 Adding HttpNotificationService.
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1610#discussion_r110550470
--- Diff:
nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/NotificationServiceManager.java
---
@@ -420,15 +424,15 @@ public static Element getChild(final Element element,
final String tagName) {
}
private static class ConfiguredNotificationService {
-private final NotificationService service;
+private final AbstractNotificationService service;
private final Map<String, String> properties;
-public ConfiguredNotificationService(final NotificationService
service, final Map<String, String> properties) {
+public ConfiguredNotificationService(final
AbstractNotificationService service, final Map<String, String> properties) {
--- End diff --
See above, reverting.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1610: NIFI-3633 Adding HttpNotificationService.
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/1610#discussion_r110550270 --- Diff: nifi-bootstrap/pom.xml --- @@ -34,13 +34,26 @@ language governing permissions and limitations under the License. --> nifi-utils +org.apache.nifi +nifi-security-utils + + javax.mail mail org.apache.nifi nifi-expression-language + +com.squareup.okhttp3 +okhttp --- End diff -- The only dep that's brought in is "okio" (as shown in the previous comment). I did forget to add it to the rpm plugin build, adding it to the new commit. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1576: NIFI-3518 Create a Morphlines processor
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/1576 Hey @WilliamNouet, sorry for not giving an update. Been too busy with other obligations. I'll try to get back to this when I have more free time. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1610: NIFI-3633 Adding HttpNotificationService.
GitHub user JPercivall opened a pull request: https://github.com/apache/nifi/pull/1610 NIFI-3633 Adding HttpNotificationService. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [X] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [X] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JPercivall/nifi NIFI-3633 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1610.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1610 commit 4e5b0a5bd3b7872f0058033e34971ab6bae97c93 Author: Joe Percivall <jperciv...@apache.org> Date: 2017-03-22T02:50:06Z NIFI-3633 Adding HttpNotificationService. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1598: NIFI-3614 Adding "return" statement to HandleHttpRe...
GitHub user JPercivall opened a pull request: https://github.com/apache/nifi/pull/1598 NIFI-3614 Adding "return" statement to HandleHttpResponse when status⦠â¦Code is not a number Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [X] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [X] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JPercivall/nifi NIFI-3614 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1598.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1598 commit 8c4a5a649000b17adc386299a7712cd04940a84f Author: Joe Percivall <jperciv...@apache.org> Date: 2017-03-16T18:11:46Z NIFI-3614 Adding "return" statement to HandleHttpResponse when statusCode is not a number --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1328: NIFI-1682 Adding RollingWindowOperation processor
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/1328 Hey @trixpan, sorry for the delay but I just rebased it again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
