[GitHub] nifi issue #2671: NiFi-5102 - Adding Processors for MarkLogic DB
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2671 If you take a look here https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#core-properties-br and checkout the 'nifi.nar.library.directory' section you can see how someone could be guided to create a directory for custom nars, add that in there, and start up. Once we have support for extensions in the NiFi Registry this will be beautiful/easy. ---
[GitHub] nifi issue #2671: NiFi-5102 - Adding Processors for MarkLogic DB
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2671 as a good example where we should have though it through more as well we have some cool work done by the InfluxDB folks. They're now wanting to improve it in https://github.com/apache/nifi/pull/2743 But the reality is we just don't have people knowledgeable enough to do reliable code reviews/testing of this. ---
[GitHub] nifi issue #2671: NiFi-5102 - Adding Processors for MarkLogic DB
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2671 @ryanjdew We have addressed Travis-CI related tests issues and the build has been stable now since. It could break again as people add timing dependent tests that behave wildly different on slow environments but we'll see. This PR, assuming the L&N and such are sorted now, is just tricky because we need a committer with time to devote to learning MarkLogic enough to setup an environment and verify function or leverage a provided instance (not a sustainable model). It is a good example where the limits of what the community can reasonably support comes into play. Now, this said this is probably a really cool and useful thing to offer folks and beneficial to both the NiFi userbase and MarkLogic user base. This is why I suggested MarkLogic folks just hang onto this code in some public github repo and have it be ALv2. They can publish their nars into maven central or wherever they do and provide instructions to it. I'd be supportive, and I'd assume the community at large would, of having links to such extensions on the apache website. This feels to me like the best tradeoff right now for all parties. ---
[GitHub] nifi pull request #3165: NIFI-5318 Implement NiFi test harness
Github user joewitt commented on a diff in the pull request: https://github.com/apache/nifi/pull/3165#discussion_r236045703 --- Diff: nifi-testharness/src/test/resources/sample_technology_rss.xml --- @@ -0,0 +1,28 @@ + + --- End diff -- this xml document looks like a real http response. We cannot use it as test material unless we can assert its license and it is ALv2 compatible. Its better to use entirely synthetic test data. ---
[GitHub] nifi pull request #3181: NIFI-5836 changed the brittle timing based tests to...
GitHub user joewitt opened a pull request: https://github.com/apache/nifi/pull/3181 NIFI-5836 changed the brittle timing based tests to integration tests Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/joewitt/incubator-nifi NIFI-5836 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3181.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3181 commit 7201f2c018b3fff31e70175fb7bb300bc46e0e31 Author: joewitt Date: 2018-11-21T17:26:13Z NIFI-5836 changed the brittle timing based tests to integration tests ---
[GitHub] nifi issue #3000: NIFI-5560 Added Follow SYMLINK support for ListFTP & ListS...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3000 +1 merged to master - thanks ---
[GitHub] nifi issue #3000: NIFI-5560 Added Follow SYMLINK support for ListFTP & ListS...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3000 reviewing now ---
[GitHub] nifi issue #3126: NIFI-5753 Add SSL support to HortonworksSchemaRegistry ser...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3126 thanks for the contrib. i've not tested it on a live/secured instance. Have you done so? ---
[GitHub] nifi issue #3180: NIFI-5833 Treat GetTwitter API properties as sensitive
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3180 actually looks like the todo may be legit to hang around in general and isn't specific to this twitter scenario so will leave that alone. given kojis review and comments and mine am +1 will merge to master ---
[GitHub] nifi pull request #3180: NIFI-5833 Treat GetTwitter API properties as sensit...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3180#discussion_r235417790
--- Diff:
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/GetTwitter.java
---
@@ -56,24 +67,10 @@
import org.apache.nifi.processor.io.OutputStreamCallback;
import org.apache.nifi.processor.util.StandardValidators;
-import com.twitter.hbc.ClientBuilder;
-import com.twitter.hbc.core.Client;
-import com.twitter.hbc.core.Constants;
-import com.twitter.hbc.core.endpoint.Location ;
-import com.twitter.hbc.core.endpoint.Location.Coordinate ;
-import com.twitter.hbc.core.endpoint.StatusesFilterEndpoint;
-import com.twitter.hbc.core.endpoint.StatusesFirehoseEndpoint;
-import com.twitter.hbc.core.endpoint.StatusesSampleEndpoint;
-import com.twitter.hbc.core.endpoint.StreamingEndpoint;
-import com.twitter.hbc.core.event.Event;
-import com.twitter.hbc.core.processor.StringDelimitedProcessor;
-import com.twitter.hbc.httpclient.auth.Authentication;
-import com.twitter.hbc.httpclient.auth.OAuth1;
-
@SupportsBatching
@InputRequirement(Requirement.INPUT_FORBIDDEN)
@Tags({"twitter", "tweets", "social media", "status", "json"})
-@CapabilityDescription("Pulls status changes from Twitter's streaming API")
+@CapabilityDescription("Pulls status changes from Twitter's streaming API.
In versions starting with 1.9.0, the Consumer Key and Access Token are marked
as sensitive according to
https://developer.twitter.com/en/docs/basics/authentication/guides/securing-keys-and-tokens";)
--- End diff --
i think it probably makes sense to leave it here because that link in
general is good for users to read and it is more of a higher level concern that
the secrets and tokens should be sensitive. Dont have a strong opinion on this
but given your +1 koji already i'm going to go ahead and merge this thing as-is
after removing the flagged todo.
---
[GitHub] nifi issue #3178: NIFI-4914: Add Apache Pulsar processors
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3178 the output log had "14:11:44.805 [pool-67-thread-1] ERROR org.apache.nifi.processors.pulsar.pubsub.PublishPulsarRecord - PublishPulsarRecord[id=9c298cea-0e2a-4a59-8864-92e95563d7f4] Unable to publish to topic " not sure if that is from the same test or not ---
[GitHub] nifi issue #3178: NIFI-4914: Add Apache Pulsar processors
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3178 build on large machine now has a unit test failure [ERROR] Failures: [ERROR] TestAsyncPublishPulsarRecord.pulsarClientExceptionTest:56 expected:<1> but was:<0> [INFO] [ERROR] Tests run: 70, Failures: 1, Errors: 0, Skipped: 0 ---
[GitHub] nifi pull request #3178: NIFI-4914: Add Apache Pulsar processors
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3178#discussion_r235108402
--- Diff:
nifi-nar-bundles/nifi-pulsar-bundle/nifi-pulsar-processors/src/main/java/org/apache/nifi/processors/pulsar/pubsub/ConsumePulsarRecord.java
---
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.pulsar.pubsub;
+
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.pulsar.AbstractPulsarConsumerProcessor;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.WriteResult;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.PulsarClientException;
+
+@CapabilityDescription("Consumes messages from Apache Pulsar. "
++ "The complementary NiFi processor for sending messages is
PublishPulsarRecord. Please note that, at this time, "
++ "the Processor assumes that all records that are retrieved have
the same schema. If any of the Pulsar messages "
++ "that are pulled but cannot be parsed or written with the
configured Record Reader or Record Writer, the contents "
++ "of the message will be written to a separate FlowFile, and that
FlowFile will be transferred to the 'parse.failure' "
++ "relationship. Otherwise, each FlowFile is sent to the 'success'
relationship and may contain many individual "
++ "messages within the single FlowFile. A 'record.count' attribute
is added to indicate how many messages are contained in the "
++ "FlowFile. No two Pulsar messages will be placed into the same
FlowFile if they have different schemas.")
+@Tags({"Pulsar", "Get", "Record", "csv", "avro", "json", "Ingest",
"Ingress", "Topic", "PubSub", "Consume"})
+@WritesAttributes({
+@WritesAttribute(attribute = "record.count", description = "The number
of records received")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@SeeAlso({PublishPulsar.c
[GitHub] nifi pull request #3178: NIFI-4914: Add Apache Pulsar processors
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3178#discussion_r235107847
--- Diff:
nifi-nar-bundles/nifi-pulsar-bundle/nifi-pulsar-processors/src/main/java/org/apache/nifi/processors/pulsar/pubsub/ConsumePulsar.java
---
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.pulsar.pubsub;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processors.pulsar.AbstractPulsarConsumerProcessor;
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.PulsarClientException;
+
+@SeeAlso({PublishPulsar.class, ConsumePulsarRecord.class,
PublishPulsarRecord.class})
+@Tags({"Pulsar", "Get", "Ingest", "Ingress", "Topic", "PubSub", "Consume"})
+@CapabilityDescription("Consumes messages from Apache Pulsar. The
complementary NiFi processor for sending messages is PublishPulsar.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+public class ConsumePulsar extends AbstractPulsarConsumerProcessor
{
+
+@Override
+public void onTrigger(ProcessContext context, ProcessSession session)
throws ProcessException {
+try {
+Consumer consumer = getConsumer(context,
getConsumerId(context, session.get()));
+
+if (consumer == null) {
+context.yield();
+return;
+}
+
+if (context.getProperty(ASYNC_ENABLED).asBoolean()) {
+consumeAsync(consumer, context, session);
+handleAsync(consumer, context, session);
+} else {
+consume(consumer, context, session);
+}
+} catch (PulsarClientException e) {
+getLogger().error("Unable to consume from Pulsar Topic ", e);
+context.yield();
+throw new ProcessException(e);
+}
+}
+
+private void handleAsync(final Consumer consumer,
ProcessContext context, ProcessSession session) {
+try {
+Future> done = getConsumerService().poll(50,
TimeUnit.MILLISECONDS);
+
+if (done != null) {
+ Message msg = done.get();
+
+ if (msg != null) {
+ FlowFile flowFile = null;
+ final byte[] value = msg.getData();
+ if (value != null && value.length > 0) {
+ flowFile = session.create();
+ flowFile = session.write(flowFile, out -> {
+ out.write(value);
+ });
+
+ session.getProvenanceReporter().receive(flowFile,
getPulsarClientService().getPulsarBrokerRootURL() + "/" + consumer.getTopic());
+ session.transfer(flowFile, REL_SUCCESS);
+ session.commit();
+ }
+ // Acknowledge consuming the message
+ getAckService().submit(new Callable() {
+ @Override
+ public Object call() throws Exception {
+ return co
[GitHub] nifi pull request #3178: NIFI-4914: Add Apache Pulsar processors
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3178#discussion_r235106784
--- Diff:
nifi-nar-bundles/nifi-pulsar-bundle/nifi-pulsar-processors/src/main/java/org/apache/nifi/processors/pulsar/AbstractPulsarProducerProcessor.java
---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.pulsar;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.pulsar.PulsarClientService;
+import org.apache.nifi.pulsar.cache.LRUCache;
+import org.apache.nifi.util.StringUtils;
+import org.apache.pulsar.client.api.CompressionType;
+import org.apache.pulsar.client.api.MessageRoutingMode;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.ProducerBuilder;
+import org.apache.pulsar.client.api.PulsarClientException;
+import
org.apache.pulsar.shade.org.apache.commons.collections.CollectionUtils;
+
+public abstract class AbstractPulsarProducerProcessor extends
AbstractProcessor {
+
+public static final String MSG_COUNT = "msg.count";
+public static final String TOPIC_NAME = "topic.name";
+
+static final AllowableValue COMPRESSION_TYPE_NONE = new
AllowableValue("NONE", "None", "No compression");
+static final AllowableValue COMPRESSION_TYPE_LZ4 = new
AllowableValue("LZ4", "LZ4", "Compress with LZ4 algorithm.");
+static final AllowableValue COMPRESSION_TYPE_ZLIB = new
AllowableValue("ZLIB", "ZLIB", "Compress with ZLib algorithm");
+
+static final AllowableValue MESSAGE_ROUTING_MODE_CUSTOM_PARTITION =
new AllowableValue("CustomPartition", "Custom Partition", "Route messages to a
custom partition");
+static final AllowableValue MESSAGE_ROUTING_MODE_ROUND_ROBIN_PARTITION
= new AllowableValue("RoundRobinPartition", "Round Robin Partition", "Route
messages to all "
+
+ "partitions in a round robin
manner");
+static final AllowableValue MESSAGE_ROUTING_MODE_SINGLE_PARTITION =
new AllowableValue("SinglePartition", "Single Partition", "Route messages to a
single partition");
+
+public static final Relationship REL_SUCCESS = new
Relationship.Builder()
+.name("success")
+.description("FlowFiles for which all content was sent to
Pulsar.")
+.build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder()
+.name("failure")
+.description("A
[GitHub] nifi pull request #3178: NIFI-4914: Add Apache Pulsar processors
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3178#discussion_r235102894
--- Diff:
nifi-nar-bundles/nifi-pulsar-bundle/nifi-pulsar-processors/pom.xml ---
@@ -0,0 +1,84 @@
+
+
+http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+4.0.0
+
+
+org.apache.nifi
+nifi-pulsar-bundle
+1.9.0-SNAPSHOT
+
+
+nifi-pulsar-processors
+jar
+
+
+
+org.apache.nifi
+nifi-api
+
+
+org.apache.nifi
+nifi-record-serialization-service-api
+
+
+org.apache.nifi
+nifi-record
+
+
+org.apache.nifi
+nifi-utils
+1.9.0-SNAPSHOT
+
+
+org.apache.nifi
+nifi-ssl-context-service-api
+
+
+org.apache.nifi
+nifi-pulsar-client-service-api
+1.9.0-SNAPSHOT
+provided
+
+
+org.apache.pulsar
+pulsar-client
+${pulsar.version}
+
+
+org.apache.nifi
+nifi-mock
+test
+1.9.0-SNAPSHOT
+
+
+org.slf4j
+slf4j-simple
+test
+
+
+junit
+junit
+test
+
+
+ org.apache.commons
+ commons-lang3
+ 3.7
--- End diff --
assuming we really need it
---
[GitHub] nifi pull request #3178: NIFI-4914: Add Apache Pulsar processors
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3178#discussion_r235102831
--- Diff:
nifi-nar-bundles/nifi-pulsar-bundle/nifi-pulsar-processors/pom.xml ---
@@ -0,0 +1,84 @@
+
+
+http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+4.0.0
+
+
+org.apache.nifi
+nifi-pulsar-bundle
+1.9.0-SNAPSHOT
+
+
+nifi-pulsar-processors
+jar
+
+
+
+org.apache.nifi
+nifi-api
+
+
+org.apache.nifi
+nifi-record-serialization-service-api
+
+
+org.apache.nifi
+nifi-record
+
+
+org.apache.nifi
+nifi-utils
+1.9.0-SNAPSHOT
+
+
+org.apache.nifi
+nifi-ssl-context-service-api
+
+
+org.apache.nifi
+nifi-pulsar-client-service-api
+1.9.0-SNAPSHOT
+provided
+
+
+org.apache.pulsar
+pulsar-client
+${pulsar.version}
+
+
+org.apache.nifi
+nifi-mock
+test
+1.9.0-SNAPSHOT
+
+
+org.slf4j
+slf4j-simple
+test
+
+
+junit
+junit
+test
+
+
+ org.apache.commons
+ commons-lang3
+ 3.7
--- End diff --
this should move up to current release which I think is 3.8.1
---
[GitHub] nifi pull request #3178: NIFI-4914: Add Apache Pulsar processors
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3178#discussion_r235102049
--- Diff:
nifi-nar-bundles/nifi-pulsar-bundle/nifi-pulsar-client-service/src/main/java/org/apache/nifi/pulsar/StandardPulsarClientService.java
---
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.pulsar;
+
+import java.net.MalformedURLException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnShutdown;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.pulsar.client.api.ClientBuilder;
+import org.apache.pulsar.client.api.PulsarClient;
+import org.apache.pulsar.client.api.PulsarClientException;
+import
org.apache.pulsar.client.api.PulsarClientException.UnsupportedAuthenticationException;
+import org.apache.pulsar.client.impl.auth.AuthenticationTls;
+
+public class StandardPulsarClientService extends AbstractControllerService
implements PulsarClientService {
+
+public static final PropertyDescriptor PULSAR_SERVICE_URL = new
PropertyDescriptor.Builder()
+.name("PULSAR_SERVICE_URL")
+.displayName("Pulsar Service URL")
+.description("URL for the Pulsar cluster, e.g localhost:6650")
+.required(true)
+.addValidator(StandardValidators.HOSTNAME_PORT_LIST_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.build();
+
+public static final PropertyDescriptor
ACCEPT_UNTRUSTED_TLS_CERTIFICATE_FROM_BROKER = new PropertyDescriptor.Builder()
+.name("ACCEPT_UNTRUSTED_TLS_CERTIFICATE_FROM_BROKER")
+.displayName("Allow TLS insecure connection")
+.description("If a valid trusted certificate is provided in
the 'TLS Trust Certs File Path' property of this controller service,"
++ " then, by default, all communication between this
controller service and the Apache Pulsar broker will be secured via"
++ " TLS and only use the trusted TLS certificate from
broker. Setting this property to 'false' will allow this controller"
++ " service to accept an untrusted TLS certificate
from broker as well. This property should only be set to false if you trust"
++ " the broker you are connecting to, but do not have
access to the TLS certificate file.")
+.required(false)
+.allowableValues("true", "false")
+.defaultValue("false")
+.build();
+
+public static final PropertyDescriptor CONCURRENT_LOOKUP_REQUESTS =
new PropertyDescriptor.Builder()
+.name("CONCURRENT_LOOKUP_REQUESTS")
+.displayName("Maximum concurrent lookup-requests")
+.description("Number of concurrent lookup-requests allowed on
each broker-connection.")
+.required(false)
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.default
[GitHub] nifi pull request #3178: NIFI-4914: Add Apache Pulsar processors
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3178#discussion_r235101491
--- Diff:
nifi-nar-bundles/nifi-pulsar-bundle/nifi-pulsar-client-service/src/main/java/org/apache/nifi/pulsar/StandardPulsarClientService.java
---
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.pulsar;
+
+import java.net.MalformedURLException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnShutdown;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.pulsar.client.api.ClientBuilder;
+import org.apache.pulsar.client.api.PulsarClient;
+import org.apache.pulsar.client.api.PulsarClientException;
+import
org.apache.pulsar.client.api.PulsarClientException.UnsupportedAuthenticationException;
+import org.apache.pulsar.client.impl.auth.AuthenticationTls;
+
+public class StandardPulsarClientService extends AbstractControllerService
implements PulsarClientService {
+
+public static final PropertyDescriptor PULSAR_SERVICE_URL = new
PropertyDescriptor.Builder()
+.name("PULSAR_SERVICE_URL")
+.displayName("Pulsar Service URL")
+.description("URL for the Pulsar cluster, e.g localhost:6650")
+.required(true)
+.addValidator(StandardValidators.HOSTNAME_PORT_LIST_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.build();
+
+public static final PropertyDescriptor
ACCEPT_UNTRUSTED_TLS_CERTIFICATE_FROM_BROKER = new PropertyDescriptor.Builder()
--- End diff --
@david-streamlio this is the property i am saying we should eliminate.
It's fine that pulsar supports this but we dont need to from nifi. We spend a
ton of time/energy on security and we need to get more serious on it. I see no
reason to keep this here. If we later find this to be a real problem then we
could talk about ways to improve the situation by helping them generate proper
certs/etc.. Also, I dont think we claim this is ok 'if you trust the broker
you're connecting to' - the point is it could be any broker... This is
inherently not a good thing to use. We should solve the problem. We can do
that later.
---
[GitHub] nifi pull request #3178: NIFI-4914: Add Apache Pulsar processors
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3178#discussion_r235099937
--- Diff:
nifi-nar-bundles/nifi-pulsar-bundle/nifi-pulsar-client-service-api/src/main/java/org/apache/nifi/pulsar/cache/LRUCache.java
---
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.pulsar.cache;
+
+import java.io.Closeable;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.Map;
+
+public class LRUCache {
--- End diff --
@david-streamlio can you confirm this is a uniquely written class for this
purpose and not taken from elsewhere? I ask because I've had the displeasure
of finding copied code before and it makes for some build/release messes and
this is the type of thing that often hits that trigger. This is fresh/clean
room stuff?
---
[GitHub] nifi issue #3178: NIFI-4914: Add Apache Pulsar processors
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3178 cool thanks. can you please take a look at my comment regarding the security related property. ---
[GitHub] nifi issue #3178: NIFI-4914: Add Apache Pulsar processors
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3178 just did a full clean build with 6 threads on an older macbook and no problems. i'll try to reproduce my build issue from yesterday on a much higher thread build/machine and advise if issue i saw remains ---
[GitHub] nifi issue #3178: NIFI-4914: Add Apache Pulsar processors
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3178 @david-streamlio i had comments on the previous PR. Can you please review those/pull them forward to this. In the future you could rebase, squash, and force push the PR if you want to get back to a clean state but the history could all stay on the same PR. It helps both you and reviewer. I'd love to help you get this thing in but it is a bit elusive to get the timing right between contrib, review cycles, pr resets, etc.. ---
[GitHub] nifi pull request #2882: NIFI-4914
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2882#discussion_r234765596
--- Diff:
nifi-nar-bundles/nifi-pulsar-bundle/nifi-pulsar-client-service/src/main/java/org/apache/nifi/pulsar/StandardPulsarClientService.java
---
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.pulsar;
+
+import java.net.MalformedURLException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnShutdown;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.pulsar.client.api.ClientBuilder;
+import org.apache.pulsar.client.api.PulsarClient;
+import org.apache.pulsar.client.api.PulsarClientException;
+import
org.apache.pulsar.client.api.PulsarClientException.UnsupportedAuthenticationException;
+import org.apache.pulsar.client.impl.auth.AuthenticationTls;
+
+public class StandardPulsarClientService extends AbstractControllerService
implements PulsarClientService {
+
+public static final PropertyDescriptor PULSAR_SERVICE_URL = new
PropertyDescriptor
+.Builder().name("PULSAR_SERVICE_URL")
+.displayName("Pulsar Service URL")
+.description("URL for the Pulsar cluster, e.g localhost:6650")
+.required(true)
+.addValidator(StandardValidators.HOSTNAME_PORT_LIST_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.build();
+
+public static final PropertyDescriptor ALLOW_TLS_INSECURE_CONNECTION =
new PropertyDescriptor.Builder()
--- End diff --
This property should be removed as its meaning is too vague. There are a
lot of ways a TLS connection could be considered or made insecure. If you mean
to allow for untrusted certs, and I really dont understand why we would, then
it should be clear to what it is doing.
---
[GitHub] nifi issue #2882: NIFI-4914
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2882 @david-streamlio its not likely we'll put a feature like this into an already released line. It doesn't fit our semver scheme. Also, on my local branch i have the changes i needed from your last commit so this would actually build against master. However, if I do parallel builds the build gets hung up. Single threaded builds are fine. I cannot tell why yet. If I do a full clean build parallel just on master things are fine. So, i believe there is something in the way these tests run that break parallel builds ---
[GitHub] nifi issue #2882: NIFI-4914
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2882 am building against master - is 1.9.0. changing it to 1.9.0-SNAPSHOT and ignoring last commit addresses it. travis fails for a similar reason i'd guess. Probably makes sense to ditch the merge commits, squash, rebase to latest. But in any case is building now i think ---
[GitHub] nifi issue #2882: NIFI-4914
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2882 the build doesn't work...at least for me or travis. also not with the new patch. i've not looked into why yet but we dont use relative paths anywhere that i know of so not sure what the latest change was for/doing ---
[GitHub] nifi issue #2882: NIFI-4914
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2882 travis and local builds dont work due to some parent relative path issues. can you please confirm you're following the pattern that other components follow with regards to pom contents ---
[GitHub] nifi issue #3174: [WIP] NIFI-5820 NiFi built on Java 1.8 can run on Java 9/1...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3174 it built with java 8. switched to 11. nifi starts up and appears to be working great. I did notice this on startup nifi.sh: JAVA_HOME not set; results may vary Java home: NiFi home: /Users/joe/development/nifi.git/nifi-assembly/target/nifi-1.9.0-SNAPSHOT-bin/nifi-1.9.0-SNAPSHOT Bootstrap Config File: /../development/nifi.git/nifi-assembly/target/nifi-1.9.0-SNAPSHOT-bin/nifi-1.9.0-SNAPSHOT/conf/bootstrap.conf WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.nifi.bootstrap.util.OSUtils (file:/../development/nifi.git/nifi-assembly/target/nifi-1.9.0-SNAPSHOT-bin/nifi-1.9.0-SNAPSHOT/lib/bootstrap/nifi-bootstrap-1.9.0-SNAPSHOT.jar) to method java.lang.ProcessImpl.pid() WARNING: Please consider reporting this to the maintainers of org.apache.nifi.bootstrap.util.OSUtils WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release ---
[GitHub] nifi issue #3130: NIFI-5791: Add Apache Daffodil (incubating) bundle
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3130 First of all this looks like a really thought out contribution and huge thanks for taking the time to create what looks like a well done NOTICE file! I'm not sure about the naming/intent that is communicated to the user. And in general we try to name things VerbSubject style. What are some other names to consider? ---
[GitHub] nifi issue #2671: NiFi-5102 - Adding Processors for MarkLogic DB
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2671 team; given https://github.com/marklogic/nifi/releases could we consider closing this PR and keeping the MarkLogic artifact creation/maintenance something MarkLogic takes care of at this time? It is a perfectly fine model. We could even create a nifi web page to point at vendor/other community managed/supported extensions possibly. ---
[GitHub] nifi pull request #3033: NIFI-5629 GetFile vast listing performance
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3033#discussion_r226293537
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetFile.java
---
@@ -308,32 +321,45 @@ public boolean accept(final File file) {
};
}
-private Set performListing(final File directory, final
FileFilter filter, final boolean recurseSubdirectories) {
-Path p = directory.toPath();
-if (!Files.isWritable(p) || !Files.isReadable(p)) {
-throw new IllegalStateException("Directory '" + directory + "'
does not have sufficient permissions (i.e., not writable and readable)");
-}
-final Set queue = new HashSet<>();
-if (!directory.exists()) {
-return queue;
-}
-
-final File[] children = directory.listFiles();
-if (children == null) {
-return queue;
-}
-
-for (final File child : children) {
-if (child.isDirectory()) {
+private Set performListing(final File directory, final
FileFilter filter, final boolean recurseSubdirectories, final int batchSize) {
+try {
+if (directoryStream == null || !this.fileIterator.hasNext()) {
+final Path p = directory.toPath();
+if (!Files.isReadable(p) || !Files.isWritable(p)) {
+throw new IllegalStateException("Directory '" +
directory + "' does not have sufficient permissions (i.e., not writable and
readable)");
+}
+
+if (!directory.exists()) {
+return Collections.emptySet();
+}
+
+Stream listStream;
if (recurseSubdirectories) {
-queue.addAll(performListing(child, filter,
recurseSubdirectories));
+listStream = Files.walk(p);
--- End diff --
@markap14 hadn't we switched to using the streaming api previously and
backed this out? If so what was the problem? Or maybe that was ListFile?
---
[GitHub] nifi pull request #3033: NIFI-5629 GetFile vast listing performance
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3033#discussion_r226292672
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetFile.java
---
@@ -270,6 +280,9 @@ private FileFilter createFileFilter(final
ProcessContext context) {
return new FileFilter() {
@Override
public boolean accept(final File file) {
+if (file.isDirectory()) {
+return false;
--- End diff --
does filtering work differently with the streams? I ask because we want to
be able to support the same depth/recursive searches and this filter seems like
it would stop at dir level.
---
[GitHub] nifi issue #3078: NIFI-4805 Allow Delayed Transfer
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3078 @patricker yeah sorry - i agree. I just meant that once you pull a flowfile from the queue and find it isn't time (and perhaps not even close to time) to send it to success you need to do something with it. That something could be penalize it and route to a 'not ready' process so you're not constantly interrogating the same file and others can go through ---
[GitHub] nifi issue #3078: NIFI-4805 Allow Delayed Transfer
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3078 I do think so to be honest and i say that knowing it would be a good bit more work. You can penalize flowfiles though that are 'not ready' - we might need to allow penalization values on a per flowfile basis (add an api call for it i mean). I think it makes a lot more sense to the user and would be more powerful/configurable. ---
[GitHub] nifi issue #3078: NIFI-4805 Allow Delayed Transfer
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3078 We should avoid calling this processor a 'PenalizeFlowFile' processor. First, we have a notion of penalization in the framework already and it has a very specific meaning and it kicks in when processors reach certain failure conditions that are believed to be specific to a given flowfile or set of flowfiles and that by penalizing them the condition might improve on its own. Second, the use for this processor in a flow might not be for the purposes of penalizing that flowfile but rather might simply be to cause a given wait or hold to occur. I'd recommend calling it DelayFlowFile. <...then I went and looked at the code> It is literally just calling penalize flow file so it makes sense why the name was given. What use case is this for again? If we're really wanting to create a DelayFlowFile processor, which is what it sounds like was the intent, we should do that because penalization means something else. ---
[GitHub] nifi issue #3084: NIFI-5689 ReplaceText does not handle end of line correctl...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3084 we cannot edit/alter NLKBufferedReader as-is. It needs to be completely redone as per https://issues.apache.org/jira/browse/NIFI-5711 ---
[GitHub] nifi issue #3069: NIFI-4806: Bumping to tika 1.19.1
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3069 +1 for sure assuming build work sout ---
[GitHub] nifi issue #3062: NIFI-5686: Updated StandardProcessScheduler so that if it ...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3062 @markap14 at least one relevant to this change test appears brittle and possible another... [ERROR] Tests run: 10, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 10.413 s <<< FAILURE! - in org.apache.nifi.controller.queue.clustered.TestSocketLoadBalancedFlowFileQueue [ERROR] testChangeInClusterTopologyTriggersRebalanceOnlyOnRemovedNodeIfNecessary(org.apache.nifi.controller.queue.clustered.TestSocketLoadBalancedFlowFileQueue) Time elapsed: 10.008 s <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 1 milliseconds at org.apache.nifi.controller.queue.clustered.TestSocketLoadBalancedFlowFileQueue.testChangeInClusterTopologyTriggersRebalanceOnlyOnRemovedNodeIfNecessary(TestSocketLoadBalancedFlowFileQueue.java:363) [ERROR] Tests run: 15, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 37.795 s <<< FAILURE! - in org.apache.nifi.controller.scheduling.TestProcessorLifecycle [ERROR] validateProcessorCanBeStoppedWhenOnTriggerThrowsException(org.apache.nifi.controller.scheduling.TestProcessorLifecycle) Time elapsed: 21.061 s <<< FAILURE! java.lang.AssertionError at org.apache.nifi.controller.scheduling.TestProcessorLifecycle.assertCondition(TestProcessorLifecycle.java:120) at org.apache.nifi.controller.scheduling.TestProcessorLifecycle.validateProcessorCanBeStoppedWhenOnTriggerThrowsException(TestProcessorLifecycle.java:498) ---
[GitHub] nifi pull request #3053: NIFI-5660: JMSPublisher should not set header prope...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3053#discussion_r223748531
--- Diff:
nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/JMSPublisher.java
---
@@ -89,11 +89,14 @@ void setMessageHeaderAndProperties(final Message
message, final Map entry :
flowFileAttributesToSend.entrySet()) {
try {
if (entry.getKey().equals(JmsHeaders.DELIVERY_MODE)) {
-
message.setJMSDeliveryMode(Integer.parseInt(entry.getValue()));
+this.jmsTemplate.setExplicitQosEnabled(true);
--- End diff --
are setting these values explicitly and without any configurability going
to limit usage of these processors for certain JMS brokers?
---
[GitHub] nifi pull request #3052: NIFI-5666 Updated all usages of Spring, beanutils, ...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3052#discussion_r223551634
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/scheduling/TestStandardProcessScheduler.java
---
@@ -169,6 +169,7 @@ public void after() throws Exception {
* run. This unit test is intended to verify that we have this
resolved.
*/
@Test
+@Ignore("This test appears to be buggy")
--- End diff --
@markap14 any chance you can take a look here? I noticed 40+MB of test
output docs on surefire output loaded with exception stacks and it all related
to a private static reporting task in the test classes (see below for where i
changed from private to public). However, after doing that and seeing the logs
appear much better the test for ensuring reporting task stops was now failing.
I cant quite make sense of it.
---
[GitHub] nifi pull request #3052: NIFI-5666 Updated all usages of Spring, beanutils, ...
GitHub user joewitt opened a pull request: https://github.com/apache/nifi/pull/3052 NIFI-5666 Updated all usages of Spring, beanutils, collections to mov⦠â¦e beyond deps with cves Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/joewitt/incubator-nifi NIFI-5666 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3052.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3052 commit 2b894d7e5c7a025b3bb085cd0521ae092c7c6bff Author: joewitt Date: 2018-10-08T17:35:01Z NIFI-5666 Updated all usages of Spring, beanutils, collections to move beyond deps with cves ---
[GitHub] nifi issue #3034: NIFI-5479 - Fixed up dependencies to remove the WARNs caus...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3034 also in checking the jolt transform processor's advanced ui i get this error now 2018-10-04 11:46:42,041 WARN [NiFi Web Server-87] o.e.j.s.h.C._nifi_jolt_transform_json_ui_1_8_0_SNAPSHOT unavailable java.lang.IllegalStateException: InjectionManagerFactory not found. at org.glassfish.jersey.internal.inject.Injections.lambda$lookupInjectionManagerFactory$0(Injections.java:98) at java.util.Optional.orElseThrow(Optional.java:290) at org.glassfish.jersey.internal.inject.Injections.lookupInjectionManagerFactory(Injections.java:98) at org.glassfish.jersey.internal.inject.Injections.createInjectionManager(Injections.java:93) at org.glassfish.jersey.server.ApplicationHandler.(ApplicationHandler.java:282) at org.glassfish.jersey.servlet.WebComponent.(WebComponent.java:335) at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:178) at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:370) at javax.servlet.GenericServlet.init(GenericServlet.java:244) at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:670) at org.eclipse.jetty.servlet.ServletHolder.getServlet(ServletHolder.java:519) at org.eclipse.jetty.servlet.ServletHolder.prepare(ServletHolder.java:803) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:530) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:724) at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:531) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) at java.lang.Thread.run(Thread.java:748) 2018-10-04 11:46:42,042 WARN [NiFi Web Server-87] org.eclipse.jetty.server.HttpChannel /nifi-jolt-transform-json-ui-1.8.0-SNAPSHOT/api/standard/processor/details javax.servlet.ServletException: javax.servlet.ServletException: api@17a1a==org.glassfish.jersey.servlet.ServletContainer,jsp=null,order=-1,inst=false at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:724) at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61)
[GitHub] nifi issue #3034: NIFI-5479 - Fixed up dependencies to remove the WARNs caus...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3034 I still see this one 2018-10-04 10:36:50,626 WARN [main] o.e.jetty.annotations.AnnotationParser Unknown asm implementation version, assuming version 393216 Did you see it? ---
[GitHub] nifi issue #3028: Nifi 4806
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3028 @mcgilman let me know off-list that the guava changes I made were too dangerous without more thorough evaluation. I've removed all guava changes from this PR as those can be taken up separately. ---
[GitHub] nifi issue #3040: NIFI-5648: Enable to launch NiFi and ignore the failed ext...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3040 @kemixkoo I can understand where the desire for this comes from but I think this is a dangerous approach. Fundamentally we're saying that we know at startup time there is a classpath issue. Continuing to startup as if it appears ok despite knowing that and without a more holistic approach leads to a poor user experience. Instead of this change I think we should focus on adding an extension registry api to the Flow Registry project. There we can have nars live and NiFi instances can source them at runtime. Issues with those nars are then well managed/clear and don't affect a live instance. It also means instances dont require restart to source new versions/etc.. ---
[GitHub] nifi issue #3034: NIFI-5479 - Fixed up dependencies to remove the WARNs caus...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3034 reviewing ---
[GitHub] nifi issue #3028: Nifi 4806
Github user joewitt commented on the issue:
https://github.com/apache/nifi/pull/3028
and i'll address the checkstyle finding
[WARNING]
src/test/java/org/apache/nifi/atlas/emulator/AtlasAPIV2ServerEmulator.java:[175,21]
(blocks) LeftCurly: '{' at column 21 should be on the previous line.
---
[GitHub] nifi issue #3028: Nifi 4806
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3028 new commons lang came out. grabbing that. will squash too to make reviewing easier from here ---
[GitHub] nifi pull request #3028: Nifi 4806
GitHub user joewitt opened a pull request: https://github.com/apache/nifi/pull/3028 Nifi 4806 Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/joewitt/incubator-nifi NIFI-4806 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3028.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3028 commit 3ba0181f4376e2dd4826608b6832d74d926e0707 Author: joewitt Date: 2018-09-21T03:24:17Z NIFI-4806 updated tika and a ton of other deps as found by dependency versions plugin commit fdd82bc9c8153d99473566ea0907fa4a0c6bd17c Author: joewitt Date: 2018-09-21T03:42:43Z NIFI-4806 updated tika and a ton of other deps as found by dependency versions plugin commit dce7cc15764c15849c79e09d7811d19759ae9434 Author: joewitt Date: 2018-09-21T22:28:40Z NIFI-4806 ---
[GitHub] nifi pull request #2947: [WIP] NIFI-5516: Implement Load-Balanced Connection...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2947#discussion_r218446031
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/RoundRobinPartitioner.java
---
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.queue.clustered.partition;
+
+import org.apache.nifi.controller.repository.FlowFileRecord;
+
+import java.util.concurrent.atomic.AtomicLong;
+
+public class RoundRobinPartitioner implements FlowFilePartitioner {
+private final AtomicLong counter = new AtomicLong(0L);
+
+@Override
+public QueuePartition getPartition(final FlowFileRecord flowFile,
final QueuePartition[] partitions, final QueuePartition localPartition) {
+final long count = counter.getAndIncrement();
--- End diff --
i am screenshot/saving this thread. I look forward to coming back to it in
less than 7 million years.
---
[GitHub] nifi issue #3007: NIFI-5596 Upgraded splunk-sdk-java version
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/3007 @MikeThomsen if the nar now includes a dependency it did not previously include then the L&N will need to be updated, if necessary, to reflect that. I dont recall the case of jbcrypt but I dont think that is a new dependency in nifi overall. If we have it in a L or N of a binary artifact elsewhere then we'll want it here too. Make sense? ---
[GitHub] nifi issue #2882: NIFI-4914
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2882 not clear when we'll be cutting a 1.8 release so i'm not sure about safety but this is clearly a cool capability and it is just a matter of finding a committer to review it with sufficient bandwidth and expertise. things impacting security are super important and not being record oriented makes it less useful for sure. I haven't looked at the details in a while to see if you added that. I'd go so far as to recommend not offering a non record approach but i wouldnt say that is a rule - just a recommendation ---
[GitHub] nifi pull request #2968: NIFI-5456: AWS clients now work with private link e...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2968#discussion_r213431435
--- Diff:
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
---
@@ -286,7 +286,7 @@ protected void
initializeRegionAndEndpoint(ProcessContext context) {
final String urlstr =
StringUtils.trimToEmpty(context.getProperty(ENDPOINT_OVERRIDE).evaluateAttributeExpressions().getValue());
if (!urlstr.isEmpty()) {
getLogger().info("Overriding endpoint with {}", new
Object[]{urlstr});
-this.client.setEndpoint(urlstr);
+this.client.setEndpoint(urlstr,
this.client.getServiceName(), this.region.getName());
--- End diff --
these values will always be present? We dont have to guard from them being
null/undefined/different than the API being called expects?
---
[GitHub] nifi pull request #2933: NIFI-5479 Upgraded Jetty. Moved where we unpack bun...
Github user joewitt closed the pull request at: https://github.com/apache/nifi/pull/2933 ---
[GitHub] nifi issue #2971: NIFI-5557: handling expired ticket by rollback and penaliz...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2971 @ekovacs thanks for your contribution. Are we sure there is no other course for us than to parse the exception cause for this case? If not, that is unfortunate but ok. The other question then is if this is a particular case we need the HDFS client to tell us have we asked the Hadoop community to improve this or filed a JIRA to that effect? If not we definitely should so that we can check this in the future in a way that is safer/more reliable. Thanks! ---
[GitHub] nifi issue #2956: NIFI-5537 Create Neo4J cypher execution processor
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2956 Just having it called nifi-neo4j-bundle is probably sufficient. The intent with the bundles is to put like things with like dependencies together. It is about creating an isolated classloader but at the same time we dont want 'one per component'. Also, unless the resulting nar is extremely small we probably dont want to add it to the nifi-assembly at this time. It is ok, still available, just not in the default build. Lastly, remember with the L&N files that we must account for *all* dependencies in the nar. I didn't check so it might be fine but a reminder. ---
[GitHub] nifi issue #2944: [WIP] NIFI-5506 - support disabling wantClientAuth for use...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2944 On the PR: - Small things: 1) The message has the wrong JIRA number 2) The default of false is incorrect and should be true to remain consistent with current usage. Bigger concern: The design of NiFi security model as discussed in the previous discussion thread is important to consider here. Have you tested that a secure NiFi cluster, cluster replication, and site-to-site capabilities function in this configuration? Can you elaborate on the testing conducted? It is believed these would all break under this code change and configuration. We need to have a more formal feature related discussion on how to get from where we are to where we need to be to properly support this. Your use case is a perfectly good one to support but to get there we cannot break what we do support today. ---
[GitHub] nifi issue #2846: NIFI-5381 Add GetSFTP and PutSFTP Unit Tests
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2846 +1 if the change to the NOTICE is removed. Not needed in nar notice since those are only included in test scope (not in nar) ---
[GitHub] nifi pull request #2846: NIFI-5381 Add GetSFTP and PutSFTP Unit Tests
Github user joewitt commented on a diff in the pull request: https://github.com/apache/nifi/pull/2846#discussion_r207997339 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-nar/src/main/resources/META-INF/NOTICE --- @@ -226,6 +226,11 @@ The following binary components are provided under the Apache Software License v Copyright 2018 simple-syslog-5424 authors. + (ASLv2) Apache MINA --- End diff -- dont need this - test changes/deps don't impact nar contents ---
[GitHub] nifi pull request #2882: NIFI-4914
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2882#discussion_r207611496
--- Diff:
nifi-nar-bundles/nifi-pulsar-bundle/nifi-pulsar-client-service/src/main/java/org/apache/nifi/pulsar/StandardPulsarClientService.java
---
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.pulsar;
+
+import java.net.MalformedURLException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnShutdown;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.pulsar.client.api.ClientBuilder;
+import org.apache.pulsar.client.api.PulsarClient;
+import org.apache.pulsar.client.api.PulsarClientException;
+import
org.apache.pulsar.client.api.PulsarClientException.UnsupportedAuthenticationException;
+import org.apache.pulsar.client.impl.auth.AuthenticationTls;
+
+public class StandardPulsarClientService extends AbstractControllerService
implements PulsarClientService {
+
+public static final PropertyDescriptor PULSAR_SERVICE_URL = new
PropertyDescriptor
+.Builder().name("PULSAR_SERVICE_URL")
+.displayName("Pulsar Service URL")
+.description("URL for the Pulsar cluster, e.g localhost:6650")
+.required(true)
+.addValidator(StandardValidators.HOSTNAME_PORT_LIST_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.build();
+
+public static final PropertyDescriptor ALLOW_TLS_INSECURE_CONNECTION =
new PropertyDescriptor.Builder()
+.name("Allow TLS insecure connection")
+.displayName("Allow TLS insecure connection")
+.description("Whether the Pulsar client will accept untrusted
TLS certificate from broker or not.")
+.required(false)
+.allowableValues("true", "false")
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.defaultValue("false")
+.build();
+
+public static final PropertyDescriptor CONCURRENT_LOOKUP_REQUESTS =
new PropertyDescriptor.Builder()
+.name("Maximum concurrent lookup-requests")
+.description("Number of concurrent lookup-requests allowed on
each broker-connection.")
+.required(false)
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.defaultValue("5000")
+.build();
+
+public static final PropertyDescriptor CONNECTIONS_PER_BROKER = new
PropertyDescriptor.Builder()
+.name("Maximum connects per Pulsar broker")
+.description("Sets the max number of connection that the
client library will open to a single broker.\n" +
+"By default, the connection pool will use a single
connection for all the producers and consumers. " +
+"Increasing this parameter may improve throughput when
using many producers over a high latency connection")
+.required(false)
+.addValidator(StandardValidato
[GitHub] nifi pull request #2882: NIFI-4914
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2882#discussion_r207611364
--- Diff:
nifi-nar-bundles/nifi-pulsar-bundle/nifi-pulsar-client-service/src/main/java/org/apache/nifi/pulsar/StandardPulsarClientService.java
---
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.pulsar;
+
+import java.net.MalformedURLException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnShutdown;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.pulsar.client.api.ClientBuilder;
+import org.apache.pulsar.client.api.PulsarClient;
+import org.apache.pulsar.client.api.PulsarClientException;
+import
org.apache.pulsar.client.api.PulsarClientException.UnsupportedAuthenticationException;
+import org.apache.pulsar.client.impl.auth.AuthenticationTls;
+
+public class StandardPulsarClientService extends AbstractControllerService
implements PulsarClientService {
+
+public static final PropertyDescriptor PULSAR_SERVICE_URL = new
PropertyDescriptor
+.Builder().name("PULSAR_SERVICE_URL")
+.displayName("Pulsar Service URL")
+.description("URL for the Pulsar cluster, e.g localhost:6650")
+.required(true)
+.addValidator(StandardValidators.HOSTNAME_PORT_LIST_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.build();
+
+public static final PropertyDescriptor ALLOW_TLS_INSECURE_CONNECTION =
new PropertyDescriptor.Builder()
--- End diff --
This probably should not be an option and the security context service
should encapsulate such things.
---
[GitHub] nifi issue #2882: NIFI-4914
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2882 sorry @david-streamlio i def would like to get it in as well. i will try to help soon ---
[GitHub] nifi issue #2933: NIFI-5479 Upgraded Jetty. Moved where we unpack bundled de...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2933 that would not enable us to work around this issue and does not bring the benefits that led to unpacking in the first place ---
[GitHub] nifi issue #2933: NIFI-5479 Upgraded Jetty. Moved where we unpack bundled de...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2933 big thanks to @mcgilman for finding the needed dep change in nifi-web-ui and identifying why we needed a workaround for how we extract working dir nar deps due to recent jetty change ---
[GitHub] nifi issue #2933: NIFI-5479 Upgraded Jetty. Moved where we unpack bundled de...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2933 2018-08-02 16:20:07,347 WARN [NiFi Web Server-21] o.e.jetty.annotations.AnnotationParser javax.inject.Inject scanned from multiple locations: jar:file:///Users/jwitt/Development/joewitt-nifi.git/nifi-assembly/target/nifi-1.8.0-SNAPSHOT-bin/nifi-1.8.0-SNAPSHOT/work/jetty/nifi-update-attribute-ui-1.8.0-SNAPSHOT.war/webapp/WEB-INF/lib/javax.inject-1.jar!/javax/inject/Inject.class, jar:file:///Users/jwitt/Development/joewitt-nifi.git/nifi-assembly/target/nifi-1.8.0-SNAPSHOT-bin/nifi-1.8.0-SNAPSHOT/work/jetty/nifi-update-attribute-ui-1.8.0-SNAPSHOT.war/webapp/WEB-INF/lib/javax.inject-2.5.0-b42.jar!/javax/inject/Inject.class 2018-08-02 16:20:07,348 WARN [NiFi Web Server-21] o.e.jetty.annotations.AnnotationParser javax.inject.Named scanned from multiple locations: jar:file:///Users/jwitt/Development/joewitt-nifi.git/nifi-assembly/target/nifi-1.8.0-SNAPSHOT-bin/nifi-1.8.0-SNAPSHOT/work/jetty/nifi-update-attribute-ui-1.8.0-SNAPSHOT.war/webapp/WEB-INF/lib/javax.inject-1.jar!/javax/inject/Named.class, jar:file:///Users/jwitt/Development/joewitt-nifi.git/nifi-assembly/target/nifi-1.8.0-SNAPSHOT-bin/nifi-1.8.0-SNAPSHOT/work/jetty/nifi-update-attribute-ui-1.8.0-SNAPSHOT.war/webapp/WEB-INF/lib/javax.inject-2.5.0-b42.jar!/javax/inject/Named.class 2018-08-02 16:20:07,348 WARN [NiFi Web Server-21] o.e.jetty.annotations.AnnotationParser javax.inject.Provider scanned from multiple locations: jar:file:///Users/jwitt/Development/joewitt-nifi.git/nifi-assembly/target/nifi-1.8.0-SNAPSHOT-bin/nifi-1.8.0-SNAPSHOT/work/jetty/nifi-update-attribute-ui-1.8.0-SNAPSHOT.war/webapp/WEB-INF/lib/javax.inject-1.jar!/javax/inject/Provider.class, jar:file:///Users/jwitt/Development/joewitt-nifi.git/nifi-assembly/target/nifi-1.8.0-SNAPSHOT-bin/nifi-1.8.0-S 2018-08-02 16:20:16,335 WARN [main] o.e.j.webapp.StandardDescriptorProcessor Duplicate mapping from / to default 2018-08-02 16:20:16,429 WARN [main] o.e.jetty.annotations.AnnotationParser Unknown asm implementation version, assuming version 393216 Todo: - identify source of and cleanup warnings that now show up on application startup - test/what if one just copied the contents of a new lib dir on top of their old lib dir as an upgrade process...will our work dirs get cleaned and restart properly - add docs in the unpack method to explain why we move META-INF/bundled-dependencies to NAR-INF/bundled-dependencies - this is just a technique to work with older/current nar creation approach. - this is done because jetty's code assumes that META-INF is only in a directory path once or else it fails to find some tlds. - but we want to keep META-INF for things like META-INF/MANIFEST.mf and maven bits. - we might want to just move META-INF/bundled-dependencies to bundled-dependencies. The 'NAR-INF' part is not value add since the nar metadata is in META-INF/MANIFEST.mf and not easily moved due to jar/manifest loading code - file a JIRA to change where we write them in the nar plugin to NAR-INF/bundled-dependencies directly - Test secure/non-secure clusters/etc.. ---
[GitHub] nifi pull request #2933: NIFI-5479 Upgraded Jetty. Moved where we unpack bun...
GitHub user joewitt opened a pull request: https://github.com/apache/nifi/pull/2933 NIFI-5479 Upgraded Jetty. Moved where we unpack bundled deps to so we⦠⦠can avoid a new jetty bug with META-INF loading logic. WIP for testing/eval. Not ready for merge Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/joewitt/incubator-nifi NIFI-5479 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2933.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2933 commit 29bc70b8198072ddcc9269d02a03314fcd23f5a1 Author: joewitt Date: 2018-08-02T20:29:23Z NIFI-5479 Upgraded Jetty. Moved where we unpack bundled deps to so we can avoid a new jetty bug with META-INF loading logic. WIP for testing/eval. Not ready for merge ---
[GitHub] nifi issue #2931: NIFI-3531 Moved session.recover in JMSConsumer to exceptio...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2931 ok @mosermw sounds fair to me. ---
[GitHub] nifi issue #2931: NIFI-3531 Moved session.recover in JMSConsumer to exceptio...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2931 do we know what the original author intended by 'abrupt end OR exceptional situations'? This change covers exceptional (JMSException anyway) but what about 'abrupt'? ---
[GitHub] nifi issue #2884: NIFI-3993 Updated the ZooKeeper version to 3.4.10
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2884 @jtstorck @YolandaMDavis and others interested - i recall some comments about holding off on updating zookeeper because of compatibility, etc.. Is that right? @HorizonNet did you test the migrator and cluster functions with these changes including secure/unsecure settings by chance? ---
[GitHub] nifi issue #2702: Added Apache Pulsar processors
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2702 looks like we're back in git pr funkystate with tons of non contrib commits in the PR... ---
[GitHub] nifi issue #2838: Update processor group variable from a processor
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2838 I've forced this PR closed. A new PR following the linked guidance can be submitted if there is a change needed. ---
[GitHub] nifi issue #2841: NIFI-5376 Removed deprecation warnings
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2841 ok cool. Am +1 then on all of it. Builds great on travis, my local linux machine, osx. Thanks ---
[GitHub] nifi pull request #2841: NIFI-5376 Removed deprecation warnings
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2841#discussion_r201208939
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/test/java/org/apache/nifi/cluster/coordination/flow/TestPopularVoteFlowElection.java
---
@@ -46,6 +45,19 @@
public class TestPopularVoteFlowElection {
+/**
+ * Utility method which accepts {@link NiFiProperties} object but
calls {@link StringEncryptor#createEncryptor(String, String, String)} with
extracted properties.
--- End diff --
ok yeah this method seems like a good candidate for being in 'nifi-mock' or
some nifi test utils module. This call is needed in a lot of tests. But we
dont want/need additional deps to some module in mainline code as the framework
can pull from nifi props as this is doing and avoid another dependency just to
share across code and tests..
---
[GitHub] nifi pull request #2841: NIFI-5376 Removed deprecation warnings
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2841#discussion_r201208738
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/flow/PopularVoteFlowElectionFactoryBean.java
---
@@ -44,8 +43,10 @@ public PopularVoteFlowElection getObject() throws
Exception {
}
final Integer maxNodes = properties.getFlowElectionMaxCandidates();
-
-final StringEncryptor encryptor =
StringEncryptor.createEncryptor(properties);
--- End diff --
it seems like this building of the encryptor needs to only be in the
framework/main code one time using the properties as shown here. Then for all
the tests we could have a util method. Perhaps in nifi-mock?
---
[GitHub] nifi pull request #2841: NIFI-5376 Removed deprecation warnings
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2841#discussion_r201206790
--- Diff:
nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/client/SiteToSiteClient.java
---
@@ -720,7 +728,6 @@ public HttpProxy getHttpProxy() {
}
-@SuppressWarnings("deprecation")
--- End diff --
i think you do want this line included
---
[GitHub] nifi pull request #2841: NIFI-5376 Removed deprecation warnings
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2841#discussion_r201205703
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/security/util/crypto/PasswordBasedEncryptor.java
---
@@ -121,8 +122,8 @@ public void process(final InputStream in, final
OutputStream out) throws IOExcep
byte[] salt;
try {
// NiFi legacy code determined the salt length based on
the cipher block size
-if (cipherProvider instanceof NiFiLegacyCipherProvider) {
-salt = ((NiFiLegacyCipherProvider)
cipherProvider).readSalt(encryptionMethod, in);
+if (cipherProvider instanceof
org.apache.nifi.security.util.crypto.NiFiLegacyCipherProvider) {
--- End diff --
chatted with andy and he taught me about deprecation handling. importing
the class would have caused a deprecation warning. doing the long form usage
of the class each time is the way to explicitly reference/intentionally, a
deprecated class.
---
[GitHub] nifi pull request #2841: NIFI-5376 Removed deprecation warnings
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2841#discussion_r201204269
--- Diff:
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/test/java/org/apache/nifi/processors/hadoop/TestListHDFS.java
---
@@ -496,7 +496,7 @@ public boolean delete(final Path f, final boolean
recursive) throws IOException
}
@Override
-public FileStatus[] listStatus(final Path f) throws
FileNotFoundException, IOException {
--- End diff --
api change out of scope for now
---
[GitHub] nifi pull request #2841: NIFI-5376 Removed deprecation warnings
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2841#discussion_r201204232
--- Diff:
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/test/java/org/apache/nifi/processors/hadoop/TestGetHDFSFileInfo.java
---
@@ -656,7 +656,7 @@ public boolean delete(final Path f, final boolean
recursive) throws IOException
}
@Override
-public FileStatus[] listStatus(final Path f) throws
FileNotFoundException, IOException {
--- End diff --
another API change probably out of scope for this pR
---
[GitHub] nifi pull request #2841: NIFI-5376 Removed deprecation warnings
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2841#discussion_r201204095
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-site-to-site/src/main/java/org/apache/nifi/remote/protocol/http/StandardHttpFlowFileServerProtocol.java
---
@@ -68,7 +66,7 @@ public FlowFileCodec getPreNegotiatedCodec() {
}
@Override
-protected HandshakeProperties doHandshake(Peer peer) throws
IOException, HandshakeException {
--- End diff --
another spot where we're changing the API/exception handling that we should
back out for now in my opinion
---
[GitHub] nifi pull request #2841: NIFI-5376 Removed deprecation warnings
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2841#discussion_r201202035
--- Diff:
nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/client/http/HttpClient.java
---
@@ -122,7 +120,7 @@ public PeerDescription getBootstrapPeerDescription()
throws IOException {
}
@Override
-public Transaction createTransaction(final TransferDirection
direction) throws HandshakeException, PortNotRunningException,
ProtocolException, UnknownPortException, IOException {
--- End diff --
same comment as above
---
[GitHub] nifi pull request #2841: NIFI-5376 Removed deprecation warnings
Github user joewitt commented on a diff in the pull request: https://github.com/apache/nifi/pull/2841#discussion_r201201974 --- Diff: nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/client/SiteToSiteClient.java --- @@ -107,12 +102,9 @@ * * @return a Transaction to use for sending or receiving data, or * null if all nodes are penalized. - * @throws org.apache.nifi.remote.exception.HandshakeException he - * @throws org.apache.nifi.remote.exception.PortNotRunningException pnre - * @throws IOException ioe - * @throws org.apache.nifi.remote.exception.UnknownPortException upe + * @throws IOException if unable to determine the identifier of the port */ -Transaction createTransaction(TransferDirection direction) throws HandshakeException, PortNotRunningException, ProtocolException, UnknownPortException, IOException; --- End diff -- i dont believe we can or want to dump these just yet. I see they all subclass IOException so just throwing IOException would do the job so to speak but then again they express a specific type of failure that a caller could elect to handle specifically. We could change this logic later such that the cause is of a specific type should the called care but that isn't how it is done now. We should consider such a change as part of some API improvements over time but for now I'd suggest we keep it out of this pr. ---
[GitHub] nifi issue #2854: NIFI-5355 ResizeImage Fails to read PNG type on some OS's
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2854 ok cool thanks. assuming travis is happy i'd be a +1 ---
[GitHub] nifi issue #2854: NIFI-5355 ResizeImage Fails to read PNG type on some OS's
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2854 @patricker I'm not finding good guidance in apache legal JIRA or the apache list archives yet here. However, lets be clear what we're evaluating. Initially this is code from Stackoverflow which we believe based on SO guidance is 'CC SA 3.0 w/Attribution' which is allowed in Apache projects according to https://www.apache.org/legal/resolved.html#cc-sa though that does't clarify if it is consider Cat A or Cat B ( a being permitted for source OR binary where B is permitted only for binary). So we'd have to clarify. But, upon further review you found in the wayback/archives the original source is considered to be AL v2.0 code based on a default legal disclaimer for that now defunct website also in wayback/archives. I'm not saying I disagree with your conclusion. But, one of our most critical jobs in the community is to produce legal/valid releases of source code. This is at the very least murky. We should file a LEGAL discuss thread or JIRA for resolution on this one so we can be sure we're doing the right thing. It isn't clear to me how we'd handle it and track this going forward or which license we'd claim it under/etc.. Best case is to write this code originally... Next best is to generate a legal thread to get legal ASF approval/decision on this with guidance how to handle it. ---
[GitHub] nifi issue #2854: NIFI-5355 ResizeImage Fails to read PNG type on some OS's
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2854 Thanks @patricker . I understand your point about its likely licensing. However, I'm not sure that it is sufficient for Apache Software Foundation project purposes. I've read threads or comments in the past that call these into question. Some research will be needed with ASF guidelines and we might need to ask legal if this isn't already asked. ---
[GitHub] nifi issue #2856: NIFI-4811 Added two missing entries to the nifi-redis-serv...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2856 We need to ensure both nars produced have their LICENSE/NOTICE files representing all source/binary artifacts that will be included. The best way is to look at the content of the nar itself. I used the maven dependency/compile scope in some cases as well but due to how we use Nars those weren't always reliable. ---
[GitHub] nifi issue #2780: NIFI-5289 - Changed nifi-mock junit Dependency to Compile ...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2780 it should be fine for junit dep to be compile scope for nifi-mock. nifi-mock will always be used under test scope so the transitive compile scoped deps will be ok. ---
[GitHub] nifi issue #2838: Update processor group variable from a processor
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2838 @xlamer1st please close this PR as Andy mentions. For help on how to achieve what it looks like the goal is see Andy's advice above. For help on doing contributions see here https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide Thanks ---
[GitHub] nifi issue #2853: NIFI-5382 ConvertRecord Add Option To Drop 0 Record Files
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2853 makes sense to be consistent with https://github.com/patricker/nifi/blob/f234f0b2e654027dcaa7e72acfd0f3d035377bae/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/QueryRecord.java#L131 ---
[GitHub] nifi pull request #2854: NIFI-5355 ResizeImage Fails to read PNG type on som...
Github user joewitt commented on a diff in the pull request: https://github.com/apache/nifi/pull/2854#discussion_r200800108 --- Diff: nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/image/ResizeImage.java --- @@ -196,4 +187,53 @@ public void process(final InputStream rawIn, final OutputStream out) throws IOEx } } +// https://stackoverflow.com/questions/7951290/re-sizing-an-image-without-losing-quality --- End diff -- it is polite to cite the source. however, stackoverflow posts cannot be copied as their origin/licensing is not established or clear. The code to perform the function you desire needs to be original/authentic or sourced to ALv2 or category-A compliant code https://www.apache.org/legal/resolved.html#category-a We cannot accept this code as-is. ---
[GitHub] nifi pull request #2854: NIFI-5355 ResizeImage Fails to read PNG type on som...
Github user joewitt commented on a diff in the pull request: https://github.com/apache/nifi/pull/2854#discussion_r200800076 --- Diff: nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/image/ResizeImage.java --- @@ -17,8 +17,7 @@ package org.apache.nifi.processors.image; -import java.awt.Graphics2D; -import java.awt.Image; +import java.awt.*; --- End diff -- we need to explicitly reference the items we wish to import ---
[GitHub] nifi issue #2848: NIFI-5383 Replaced hard-coded version with Maven property ...
Github user joewitt commented on the issue:
https://github.com/apache/nifi/pull/2848
I see a couple errant uses in nifi-hbase pom(s) but otherwise it is all ok
usage.
In your evaluation of things looking ok it would have been a typical build
I suspect. But we need consistency for maintainability purposes of course and
further we have it this way because this way works not just during typical
build processes developers do but also it worked well in maven during the
apache release process. We can change it but doing so requires a far more
thorough analysis of not just the typical build process but being able to do a
full release cycle.
It is not necessary, however, to use '${project.version}' since the maven
release plugin takes care of it so nicely. This JIRA and PR should be closed
in my opinion.
---
[GitHub] nifi issue #2843: NIFI-5318 Implement NiFi test harness: initial commit of n...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2843 I dont fully understand what this aims to achieve so please understand that in my comments. However, changes to the nifi-toolkit-tls module dont seem to apply to wanting a test harness style capability for flows. Adding a series of 'test' packages to main/source code seems like an anti-pattern.We really need to reset on this effort and make sure it is clear what we're looking to achieve before this goes further as-is. ---
[GitHub] nifi pull request #2843: NIFI-5318 Implement NiFi test harness: initial comm...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2843#discussion_r200799818
--- Diff:
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/test/util/FileUtils.java
---
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+
+package org.apache.nifi.test.util;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.*;
+import java.nio.file.attribute.BasicFileAttributes;
+import java.util.Arrays;
+
+public final class FileUtils {
+
+
+private static final String MAC_DS_STORE_NAME = ".DS_Store";
--- End diff --
the osx specific nature of this is concerning
---
[GitHub] nifi pull request #2843: NIFI-5318 Implement NiFi test harness: initial comm...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2843#discussion_r200799793
--- Diff:
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/test/SimpleNiFiFlowDefinitionEditor.java
---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+
+package org.apache.nifi.test;
+
+import org.apache.nifi.test.api.FlowFileEditorCallback;
+import org.w3c.dom.Document;
+import org.w3c.dom.Node;
+
+import javax.xml.xpath.XPath;
+import javax.xml.xpath.XPathConstants;
+import javax.xml.xpath.XPathFactory;
+import java.util.LinkedList;
+
+
+/**
+ *
+ * A facility to describe simple, common changes to a NiFi flow before it
is installed to the test
+ * NiFi instance. Intended to be used by
+ * {@link
TestNiFiInstance.Builder#modifyFlowXmlBeforeInstalling(FlowFileEditorCallback)}
+ *
+ *
+ *
+ * The desired edits can be configured via the {@link Builder} object
returned by the {@link #builder()}
+ * method. Once fully configured, the {@link Builder#build()} emits a
{@code FlowFileEditorCallback}
+ * object that can be passed to
+ * {@link
TestNiFiInstance.Builder#modifyFlowXmlBeforeInstalling(FlowFileEditorCallback)}.
+ *
+ *
+ *
+ * CAUTION: THIS IS AN EXPERIMENTAL API: EXPECT CHANGES!
+ * Efforts will be made to retain backwards API compatibility, but
+ * no guarantee is given.
+ *
+ *
+ * @see
TestNiFiInstance.Builder#modifyFlowXmlBeforeInstalling(FlowFileEditorCallback)
+ *
+ * @author Peter G. Horvath
--- End diff --
please remove all author tags from all javadocs
---
[GitHub] nifi pull request #2843: NIFI-5318 Implement NiFi test harness: initial comm...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2843#discussion_r200799695
--- Diff: nifi-docs/src/main/asciidoc/developer-guide.adoc ---
@@ -2296,6 +2296,32 @@ threads that should be used to run the Processor can
be set via the `setThreadCount(int)` method.
+=== Experimental NiFi Flow test harness
+
+NiFi now has an experimental feature for full end-to-end testing of flows.
This allows us
+to take a NiFi flow, install it to a test NiFi instance, run it and make
Java unit test
+like asserts regarding its behaviour.
+
+The class `org.apache.nifi.test.TestNiFiInstance` is a thin wrapper that
allows us
+to manipulate a NiFi installation and deploy a flow with some adjustments
+to its configuration, including changing processor properties and
replacing processor
+classes with mocks.
+
+In order to add the necessary classes to your project,
+you can use the Maven dependency:
+
+[source]
+
+
+ org.apache.nifi
+ nifi-test
+ ${nifi version}
+
+
+
+For further documentation, please consult the JavaDoc of
+`org.apache.nifi.test.TestNiFiInstance`. For samples, please take a look at
+link:https://github.com/apache/nifi/tree/master/nifi-test/src/test/java/org/apache/test/samples[samples
on GitHub^].
--- End diff --
what is this link really meant to be?
---
[GitHub] nifi pull request #2843: NIFI-5318 Implement NiFi test harness: initial comm...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2843#discussion_r200799698
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-documentation/src/main/java/org/apache/nifi/documentation/DocGenerator.java
---
@@ -57,6 +57,9 @@
* @param extensionMapping extension mapping
*/
public static void generate(final NiFiProperties properties, final
ExtensionMapping extensionMapping) {
+
--- End diff --
these extra lines should be removed
---
[GitHub] nifi issue #2848: NIFI-5383 Replaced hard-coded version with Maven property ...
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2848 this is not consistent with the rest of all modules in the project. the versions get changed correctly and automatically during the build release process. please revert this. if we want to change the approach then lets do it for everything and by validating the actual release process works. ---
[GitHub] nifi pull request #2841: NIFI-5376 Removed deprecation warnings
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2841#discussion_r200703862
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-nar-utils/src/main/java/org/apache/nifi/nar/NarUnpacker.java
---
@@ -264,7 +263,7 @@ private static void unpack(final File nar, final File
workingDirectory, final by
String name = jarEntry.getName();
File f = new File(workingDirectory, name);
if (jarEntry.isDirectory()) {
-FileUtils.ensureDirectoryExistAndCanAccess(f);
+FileUtils.ensureDirectoryExistAndCanReadAndWrite(f);
--- End diff --
this looks like a change fixing a bug whereby we checked for simple access
instead of ability to write files. Can we link this to an existing JIRA or
expand the scope of NIFI-5376 so we dont lose track of this.
---
[GitHub] nifi issue #2806: NIFI-5068 Script to automate parts of RC validation
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2806 should have them now ---
