[GitHub] nifi pull request: NIFI-1275: Add processor(s) support for Elastic...
Github user JPercivall commented on the pull request: https://github.com/apache/nifi/pull/180#issuecomment-178023848 If the "Identifier Attribute" is being left as the attribute containing the identifier then the "ATTRIBUTE_KEY_VALIDATOR" should be used instead of "NON_EMPTY_VALIDATOR". --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: [REVIEW ONLY]]NIFI-865 added initial support fo...
GitHub user olegz opened a pull request: https://github.com/apache/nifi/pull/199 [REVIEW ONLY]]NIFI-865 added initial support for AMQP publish/subscribe Still working on docs and tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/olegz/nifi NIFI-865 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/199.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #199 commit 15922ca6d717bc085bc65ad7f79ed2f40d0b1c29 Author: Oleg Zhurakousky Date: 2016-01-31T18:20:14Z NIFI-865 added initial support for AMQP publish/subscribe --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: [REVIEW ONLY]]NIFI-865 added initial support fo...
Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/199#discussion_r51441043 --- Diff: nifi-nar-bundles/nifi-amqp-bundle/nifi-amqp-nar/src/main/resources/META-INF/LICENSE --- @@ -0,0 +1,319 @@ + + Apache License + Version 2.0, January 2004 +http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims l
[GitHub] nifi pull request: [REVIEW ONLY]]NIFI-865 added initial support fo...
Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/199#discussion_r51441125 --- Diff: nifi-nar-bundles/nifi-amqp-bundle/nifi-amqp-processors/pom.xml --- @@ -0,0 +1,74 @@ + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + 4.0.0 + + org.apache.nifi + nifi-amqp-bundle + 0.4.2-SNAPSHOT + + nifi-amqp-processors + jar + + + --- End diff -- If not needed it should be removed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: [REVIEW ONLY]]NIFI-865 added initial support fo...
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/199#discussion_r51441537
--- Diff:
nifi-nar-bundles/nifi-amqp-bundle/nifi-amqp-processors/src/main/java/org/apache/nifi/amqp/processors/AMQPConsumer.java
---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.amqp.processors;
+
+import java.io.IOException;
+
+import org.apache.nifi.processor.exception.ProcessException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.GetResponse;
+
+/**
+ * Generic consumer of messages from AMQP-based messaging system. It is
based on
+ * RabbitMQ client API (https://www.rabbitmq.com/api-guide.html)
+ */
+final class AMQPConsumer extends AMQPWorker {
+
+private final static Logger logger =
LoggerFactory.getLogger(AMQPConsumer.class);
--- End diff --
Should we be injecting a ComponentLog here instead of using SLF4J?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: [REVIEW ONLY]]NIFI-865 added initial support fo...
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/199#discussion_r51441942
--- Diff:
nifi-nar-bundles/nifi-amqp-bundle/nifi-amqp-processors/src/main/java/org/apache/nifi/amqp/processors/AMQPPublisher.java
---
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.amqp.processors;
+
+import java.io.IOException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.rabbitmq.client.AMQP.BasicProperties;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ReturnListener;
+
+/**
+ * Generic publisher of messages to AMQP-based messaging system. It is
based on
+ * RabbitMQ client API (https://www.rabbitmq.com/api-guide.html)
+ */
+final class AMQPPublisher extends AMQPWorker {
+
+private final static Logger logger =
LoggerFactory.getLogger(AMQPPublisher.class);
+
+private final String exchangeName;
+
+private final String routingKey;
+
+/**
+ * Creates an instance of this publisher
+ *
+ * @param connection
+ *instance of AMQP {@link Connection}
+ * @param exchangeName
+ *the name of AMQP exchange to which messages will be
published.
+ *If not provided 'default' exchange will be used.
+ * @param routingKey
+ *(required) the name of the routingKey to be used by
AMQP-based
+ *system to route messages to its final destination
(queue).
+ */
+AMQPPublisher(Connection connection, String exchangeName, String
routingKey) {
+super(connection);
+this.validateStringProperty("routingKey", routingKey);
+this.exchangeName = exchangeName == null ? "" :
exchangeName.trim();
+if (this.exchangeName.length() == 0) {
+logger.info("The 'exchangeName' is not specified. Messages
will be sent to default exchange");
+}
+
+this.routingKey = routingKey;
+this.channel.addReturnListener(new UndeliverableMessageLogger());
+logger.info("Successfully connected AMQPPublisher to " +
connection.toString() + " and '" + this.exchangeName
++ "' exchange with '" + routingKey + "' as a routing
key.");
+}
+
+/**
+ * Publishes message without any AMQP properties (see
+ * {@link BasicProperties}) to a pre-defined AMQP Exchange.
+ *
+ * @param bytes
+ *bytes representing a message.
+ */
+void publish(byte[] bytes) {
+this.publish(bytes, null);
+}
+
+/**
+ * Publishes message with provided AMQP properties (see
+ * {@link BasicProperties}) to a pre-defined AMQP Exchange.
+ *
+ * @param bytes
+ *bytes representing a message.
+ * @param properties
+ *instance of {@link BasicProperties}
+ */
+void publish(byte[] bytes, BasicProperties properties) {
+if (this.channel.isOpen()) {
+try {
+this.channel.basicPublish(this.exchangeName,
this.routingKey, true, properties, bytes);
+} catch (Exception e) {
+throw new IllegalStateException("Failed to publish to '" +
+this.exchangeName + "' with '" + this.routingKey +
"'.", e);
+}
+} else {
+throw new IllegalStateException("This instance of
AMQPPublisher is invalid since "
++ "its publishigChannel is closed");
--- End diff --
typo "publishingChannel". Also should this throw ClosedChannelException
rather than IllegalStateException?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, plea
[GitHub] nifi pull request: [REVIEW ONLY]]NIFI-865 added initial support fo...
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/199#discussion_r51442155
--- Diff:
nifi-nar-bundles/nifi-amqp-bundle/nifi-amqp-processors/src/main/java/org/apache/nifi/amqp/processors/AMQPWorker.java
---
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.amqp.processors;
+
+import java.io.IOException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+
+/**
+ * Base class for implementing publishing and consuming AMQP workers.
+ *
+ * @see AMQPPublisher
+ * @see AMQPConsumer
+ */
+abstract class AMQPWorker implements AutoCloseable {
+
+private final static Logger logger =
LoggerFactory.getLogger(AMQPWorker.class);
+
+protected final Channel channel;
+
+/**
+ * Creates an instance of this worker initializing it with AMQP
+ * {@link Connection} and creating a target {@link Channel} used by
+ * sub-classes to interruct with AMQP-based messaging system.
+ *
+ * @param connection
+ *instance of {@link Connection}
+ */
+public AMQPWorker(Connection connection) {
+this.validateConnection(connection);
+try {
+this.channel = connection.createChannel();
+} catch (IOException e) {
+logger.error("Failed to create Channel for " + connection, e);
+throw new IllegalStateException(e);
+}
+}
+
+/**
+ * Closes {@link Channel} created when instance of this class was
created.
+ */
+@Override
+public void close() throws Exception {
+if (logger.isDebugEnabled()) {
+logger.info("Closing AMQP channel for " +
this.channel.getConnection().toString());
--- End diff --
Can this just be logger.debug()? Or does it have to be this way for SLF4J?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: [REVIEW ONLY]]NIFI-865 added initial support fo...
Github user olegz commented on a diff in the pull request:
https://github.com/apache/nifi/pull/199#discussion_r51442275
--- Diff:
nifi-nar-bundles/nifi-amqp-bundle/nifi-amqp-processors/src/main/java/org/apache/nifi/amqp/processors/AMQPConsumer.java
---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.amqp.processors;
+
+import java.io.IOException;
+
+import org.apache.nifi.processor.exception.ProcessException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.GetResponse;
+
+/**
+ * Generic consumer of messages from AMQP-based messaging system. It is
based on
+ * RabbitMQ client API (https://www.rabbitmq.com/api-guide.html)
+ */
+final class AMQPConsumer extends AMQPWorker {
+
+private final static Logger logger =
LoggerFactory.getLogger(AMQPConsumer.class);
--- End diff --
ComponentLog is not to be used by Processors and or supporting classes as
far as I know. It's used by framework
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: [REVIEW ONLY]]NIFI-865 added initial support fo...
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/199#discussion_r51442712
--- Diff:
nifi-nar-bundles/nifi-amqp-bundle/nifi-amqp-processors/src/main/java/org/apache/nifi/amqp/processors/AMQPConsumer.java
---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.amqp.processors;
+
+import java.io.IOException;
+
+import org.apache.nifi.processor.exception.ProcessException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.GetResponse;
+
+/**
+ * Generic consumer of messages from AMQP-based messaging system. It is
based on
+ * RabbitMQ client API (https://www.rabbitmq.com/api-guide.html)
+ */
+final class AMQPConsumer extends AMQPWorker {
+
+private final static Logger logger =
LoggerFactory.getLogger(AMQPConsumer.class);
--- End diff --
ProcessorLog extends ComponentLog, I was suggesting ComponentLog in case we
use the producer/consumers for a ReportingTask someday
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: [REVIEW ONLY]]NIFI-865 added initial support fo...
Github user olegz commented on a diff in the pull request:
https://github.com/apache/nifi/pull/199#discussion_r51443717
--- Diff:
nifi-nar-bundles/nifi-amqp-bundle/nifi-amqp-processors/src/main/java/org/apache/nifi/amqp/processors/AMQPConsumer.java
---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.amqp.processors;
+
+import java.io.IOException;
+
+import org.apache.nifi.processor.exception.ProcessException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.GetResponse;
+
+/**
+ * Generic consumer of messages from AMQP-based messaging system. It is
based on
+ * RabbitMQ client API (https://www.rabbitmq.com/api-guide.html)
+ */
+final class AMQPConsumer extends AMQPWorker {
+
+private final static Logger logger =
LoggerFactory.getLogger(AMQPConsumer.class);
--- End diff --
"IF" we do that we'll change it, but as you can see right now its very
package private ;)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: [REVIEW ONLY]]NIFI-865 added initial support fo...
Github user olegz commented on a diff in the pull request:
https://github.com/apache/nifi/pull/199#discussion_r51443924
--- Diff:
nifi-nar-bundles/nifi-amqp-bundle/nifi-amqp-processors/src/main/java/org/apache/nifi/amqp/processors/AMQPWorker.java
---
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.amqp.processors;
+
+import java.io.IOException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+
+/**
+ * Base class for implementing publishing and consuming AMQP workers.
+ *
+ * @see AMQPPublisher
+ * @see AMQPConsumer
+ */
+abstract class AMQPWorker implements AutoCloseable {
+
+private final static Logger logger =
LoggerFactory.getLogger(AMQPWorker.class);
+
+protected final Channel channel;
+
+/**
+ * Creates an instance of this worker initializing it with AMQP
+ * {@link Connection} and creating a target {@link Channel} used by
+ * sub-classes to interruct with AMQP-based messaging system.
+ *
+ * @param connection
+ *instance of {@link Connection}
+ */
+public AMQPWorker(Connection connection) {
+this.validateConnection(connection);
+try {
+this.channel = connection.createChannel();
+} catch (IOException e) {
+logger.error("Failed to create Channel for " + connection, e);
+throw new IllegalStateException(e);
+}
+}
+
+/**
+ * Closes {@link Channel} created when instance of this class was
created.
+ */
+@Override
+public void close() throws Exception {
+if (logger.isDebugEnabled()) {
+logger.info("Closing AMQP channel for " +
this.channel.getConnection().toString());
--- End diff --
oopos
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: [REVIEW ONLY]]NIFI-865 added initial support fo...
Github user olegz commented on a diff in the pull request:
https://github.com/apache/nifi/pull/199#discussion_r51444249
--- Diff:
nifi-nar-bundles/nifi-amqp-bundle/nifi-amqp-processors/src/main/java/org/apache/nifi/amqp/processors/AMQPPublisher.java
---
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.amqp.processors;
+
+import java.io.IOException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.rabbitmq.client.AMQP.BasicProperties;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ReturnListener;
+
+/**
+ * Generic publisher of messages to AMQP-based messaging system. It is
based on
+ * RabbitMQ client API (https://www.rabbitmq.com/api-guide.html)
+ */
+final class AMQPPublisher extends AMQPWorker {
+
+private final static Logger logger =
LoggerFactory.getLogger(AMQPPublisher.class);
+
+private final String exchangeName;
+
+private final String routingKey;
+
+/**
+ * Creates an instance of this publisher
+ *
+ * @param connection
+ *instance of AMQP {@link Connection}
+ * @param exchangeName
+ *the name of AMQP exchange to which messages will be
published.
+ *If not provided 'default' exchange will be used.
+ * @param routingKey
+ *(required) the name of the routingKey to be used by
AMQP-based
+ *system to route messages to its final destination
(queue).
+ */
+AMQPPublisher(Connection connection, String exchangeName, String
routingKey) {
+super(connection);
+this.validateStringProperty("routingKey", routingKey);
+this.exchangeName = exchangeName == null ? "" :
exchangeName.trim();
+if (this.exchangeName.length() == 0) {
+logger.info("The 'exchangeName' is not specified. Messages
will be sent to default exchange");
+}
+
+this.routingKey = routingKey;
+this.channel.addReturnListener(new UndeliverableMessageLogger());
+logger.info("Successfully connected AMQPPublisher to " +
connection.toString() + " and '" + this.exchangeName
++ "' exchange with '" + routingKey + "' as a routing
key.");
+}
+
+/**
+ * Publishes message without any AMQP properties (see
+ * {@link BasicProperties}) to a pre-defined AMQP Exchange.
+ *
+ * @param bytes
+ *bytes representing a message.
+ */
+void publish(byte[] bytes) {
+this.publish(bytes, null);
+}
+
+/**
+ * Publishes message with provided AMQP properties (see
+ * {@link BasicProperties}) to a pre-defined AMQP Exchange.
+ *
+ * @param bytes
+ *bytes representing a message.
+ * @param properties
+ *instance of {@link BasicProperties}
+ */
+void publish(byte[] bytes, BasicProperties properties) {
+if (this.channel.isOpen()) {
+try {
+this.channel.basicPublish(this.exchangeName,
this.routingKey, true, properties, bytes);
+} catch (Exception e) {
+throw new IllegalStateException("Failed to publish to '" +
+this.exchangeName + "' with '" + this.routingKey +
"'.", e);
+}
+} else {
+throw new IllegalStateException("This instance of
AMQPPublisher is invalid since "
++ "its publishigChannel is closed");
--- End diff --
ClosedChannelException is checked exception, needed a runtime exception
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct
[GitHub] nifi pull request: [REVIEW ONLY]]NIFI-865 added initial support fo...
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/199#discussion_r5168
--- Diff:
nifi-nar-bundles/nifi-amqp-bundle/nifi-amqp-processors/src/main/java/org/apache/nifi/amqp/processors/AMQPConsumer.java
---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.amqp.processors;
+
+import java.io.IOException;
+
+import org.apache.nifi.processor.exception.ProcessException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.GetResponse;
+
+/**
+ * Generic consumer of messages from AMQP-based messaging system. It is
based on
+ * RabbitMQ client API (https://www.rabbitmq.com/api-guide.html)
+ */
+final class AMQPConsumer extends AMQPWorker {
+
+private final static Logger logger =
LoggerFactory.getLogger(AMQPConsumer.class);
--- End diff --
Fair enough, but where do the logs go? Can you see them in the NiFi logs or
do you have to go somewhere else to see what happened?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: [REVIEW ONLY]]NIFI-865 added initial support fo...
Github user olegz commented on a diff in the pull request:
https://github.com/apache/nifi/pull/199#discussion_r51444624
--- Diff:
nifi-nar-bundles/nifi-amqp-bundle/nifi-amqp-processors/src/main/java/org/apache/nifi/amqp/processors/AMQPConsumer.java
---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.amqp.processors;
+
+import java.io.IOException;
+
+import org.apache.nifi.processor.exception.ProcessException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.GetResponse;
+
+/**
+ * Generic consumer of messages from AMQP-based messaging system. It is
based on
+ * RabbitMQ client API (https://www.rabbitmq.com/api-guide.html)
+ */
+final class AMQPConsumer extends AMQPWorker {
+
+private final static Logger logger =
LoggerFactory.getLogger(AMQPConsumer.class);
--- End diff --
Yes
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: [REVIEW ONLY]]NIFI-865 added initial support fo...
Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/199#discussion_r51444798 --- Diff: nifi-nar-bundles/nifi-amqp-bundle/nifi-amqp-processors/src/test/resources/log4j.properties --- @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +log4j.rootLogger=INFO, CONSOLE + +log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender + +log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout +log4j.appender.CONSOLE.layout.ConversionPattern=%-4r [%t] %-5p %c %x \u2013 %m%n + +log4j.logger.org.apache.flume = DEBUG --- End diff -- I know this review (for now) is just for code, but I wanted to leave a note for files that are "functional" and not purely documentation, for the Flume references to be updated --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1093 added support for handling non-requir...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/195 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: [REVIEW ONLY]]NIFI-865 added initial support fo...
Github user olegz commented on a diff in the pull request: https://github.com/apache/nifi/pull/199#discussion_r51445239 --- Diff: nifi-nar-bundles/nifi-amqp-bundle/nifi-amqp-processors/src/test/resources/log4j.properties --- @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +log4j.rootLogger=INFO, CONSOLE + +log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender + +log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout +log4j.appender.CONSOLE.layout.ConversionPattern=%-4r [%t] %-5p %c %x \u2013 %m%n + +log4j.logger.org.apache.flume = DEBUG --- End diff -- Good point and there are bunch of junk files (test data) that I am removing already, so all good. Keep going ;) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: [REVIEW ONLY]]NIFI-865 added initial support fo...
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/199#discussion_r51459684
--- Diff:
nifi-nar-bundles/nifi-amqp-bundle/nifi-amqp-processors/src/main/java/org/apache/nifi/amqp/processors/ConsumeAMQP.java
---
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.amqp.processors;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import com.rabbitmq.client.AMQP.BasicProperties;
+import com.rabbitmq.client.GetResponse;
+
+/**
+ * Consuming AMQP processor which upon each invocation of
+ * {@link #onTrigger(ProcessContext, ProcessSession)} method will
construct a
+ * {@link FlowFile} containing the body of the consumed AMQP message and
AMQP
+ * properties that came with message which are added to a {@link FlowFile}
as
+ * attributes.
+ */
+@Tags({ "amqp", "rabbit", "get", "message", "receive", "consume" })
+@CapabilityDescription("Creates a AMQP Message from the contents of a
FlowFile and sends the message to an AMQP Server")
+public class ConsumeAMQP extends AbstractAMQPProcessor {
+
+public static final PropertyDescriptor QUEUE = new
PropertyDescriptor.Builder()
+.name("Queue")
+.description("Source queue")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final Relationship REL_SUCCESS = new
Relationship.Builder()
+.name("success")
+.description("All FlowFiles that are received from the AMQP
queue are routed to this relationship")
+.build();
+
+private final static List propertyDescriptors;
+
+private final static Set relationships;
+
+/*
+ * Will ensure that the list of property descriptors is build only
once.
+ * Will also create a Set of relationships
+ */
+static {
+List _propertyDescriptors = new ArrayList<>();
+_propertyDescriptors.add(QUEUE);
+_propertyDescriptors.addAll(descriptors);
+propertyDescriptors =
Collections.unmodifiableList(_propertyDescriptors);
+
+Set _relationships = new HashSet<>();
+_relationships.add(REL_SUCCESS);
+relationships = Collections.unmodifiableSet(_relationships);
+}
+
+/**
+ * Will construct a {@link FlowFile} containing the body of the
consumed
+ * AMQP message (if {@link GetResponse} returned by {@link
AMQPConsumer} is
+ * not null) and AMQP properties that came with message which are
added to a
+ * {@link FlowFile} as attributes, transferring {@link FlowFile} to
+ * 'success' {@link Relationship}.
+ */
+@Override
+protected void rendezvousWithAmqp(ProcessContext context,
ProcessSession processSession) throws ProcessException {
+GetResponse response = this.targetResource.consume();
+if (response != null){
+FlowFile flowFile = processSession.create();
+BasicProperties amqpProperties = response.getProps();
+flowFile =
AMQPUtils.updateFlowFileAttributesWithAmqpProperties(amqpProperties, flowFile,
processSession);
+processSession.transfer(flowFile, REL_SUCCESS);
+}
--- End diff --
perhaps an "else { co
Getting error when a previously disconnected node is being brought up and trying to connect back to the NCM
Here’s the scenario. 1. Deploy Nifi Cluster : - NCM - Pirmary - Node. 2. Verified that everything is OK. 3. I then disconnected the slave Node and deployed some pipelines on the cluster. 4. Now I tried to bring up the slave node. It gives the following error. 2016-02-01 19:41:05,393 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down. java.lang.Exception: Unable to load flow due to: java.io.IOException: org.apache.nifi.cluster.ConnectionException: Failed to connect node to cluster because local flow is different than cluster flow. at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:784) ~[nifi-jetty-0.4.1.jar:0.4.1] at org.apache.nifi.NiFi.(NiFi.java:137) [nifi-runtime-0.4.1.jar:0.4.1] at org.apache.nifi.NiFi.main(NiFi.java:227) [nifi-runtime-0.4.1.jar:0.4.1] Caused by: java.io.IOException: org.apache.nifi.cluster.ConnectionException: Failed to connect node to cluster because local flow is different than cluster flow. at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:453) ~[nifi-framework-core-0.4.1.jar:0.4.1] at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:775) ~[nifi-jetty-0.4.1.jar:0.4.1] ... 2 common frames omitted Caused by: org.apache.nifi.cluster.ConnectionException: Failed to connect node to cluster because local flow is different than cluster flow. at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:732) ~[nifi-framework-core-0.4.1.jar:0.4.1] at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:433) ~[nifi-framework-core-0.4.1.jar:0.4.1] ... 3 common frames omitted Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed configuration is not inheritable by the flow controller because of flow differences: Found difference in Templates: Local Fingerprint: fc0a802d-0915-3b71-8694-52b8d7b2c556melissa_rawalert_daq_dataavailabilityNO_CONNECTIONSNO_FUNNELSNO_ Cluster Fingerprint: 42809382-bb1f-3785-b3c3-7d8b875c3e88melissa_rawalert_reqest_dataavailability_daqNO_CONNECTIONSNO_FUN at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:223) ~[nifi-framework-core-0.4.1.jar:0.4.1] at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1203) ~[nifi-framework-core-0.4.1.jar:0.4.1] at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:72) ~[nifi-framework-core-0.4.1.jar:0.4.1] at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:608) ~[nifi-framework-core-0.4.1.jar:0.4.1] at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:709) ~[nifi-framework-core-0.4.1.jar:0.4.1] ... 4 common frames omitted What I see is that the flow.tar file on the slave is 0 bytes. This does not look right. Since the file on the NCM is about 20K bytes. thanks Paresh The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
Re: Getting error when a previously disconnected node is being brought up and trying to connect back to the NCM
Paresh, In the log it says "Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed configuration is not inheritable by the flow controller because of flow differences: Found difference in Templates:". So on your node blow away both the flow.xml.gz, flow.tar, and the templates directory. In an upcoming release we will relax the template matching requirement. That was a mistake. Thanks Joe On Mon, Feb 1, 2016 at 2:45 PM, Paresh Shah wrote: > Here’s the scenario. > > 1. Deploy Nifi Cluster : > - NCM > - Pirmary > - Node. > > 2. Verified that everything is OK. > > 3. I then disconnected the slave Node and deployed some pipelines on the > cluster. > > 4. Now I tried to bring up the slave node. It gives the following error. > > > 2016-02-01 19:41:05,393 WARN [main] org.apache.nifi.web.server.JettyServer > Failed to start web server... shutting down. > > java.lang.Exception: Unable to load flow due to: java.io.IOException: > org.apache.nifi.cluster.ConnectionException: Failed to connect node to > cluster because local flow is different than cluster flow. > > at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:784) > ~[nifi-jetty-0.4.1.jar:0.4.1] > > at org.apache.nifi.NiFi.(NiFi.java:137) [nifi-runtime-0.4.1.jar:0.4.1] > > at org.apache.nifi.NiFi.main(NiFi.java:227) [nifi-runtime-0.4.1.jar:0.4.1] > > Caused by: java.io.IOException: org.apache.nifi.cluster.ConnectionException: > Failed to connect node to cluster because local flow is different than > cluster flow. > > at > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:453) > ~[nifi-framework-core-0.4.1.jar:0.4.1] > > at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:775) > ~[nifi-jetty-0.4.1.jar:0.4.1] > > ... 2 common frames omitted > > Caused by: org.apache.nifi.cluster.ConnectionException: Failed to connect > node to cluster because local flow is different than cluster flow. > > at > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:732) > ~[nifi-framework-core-0.4.1.jar:0.4.1] > > at > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:433) > ~[nifi-framework-core-0.4.1.jar:0.4.1] > > ... 3 common frames omitted > > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed > configuration is not inheritable by the flow controller because of flow > differences: Found difference in Templates: > > Local Fingerprint: > fc0a802d-0915-3b71-8694-52b8d7b2c556melissa_rawalert_daq_dataavailabilityNO_CONNECTIONSNO_FUNNELSNO_ > > Cluster Fingerprint: > 42809382-bb1f-3785-b3c3-7d8b875c3e88melissa_rawalert_reqest_dataavailability_daqNO_CONNECTIONSNO_FUN > > at > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:223) > ~[nifi-framework-core-0.4.1.jar:0.4.1] > > at > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1203) > ~[nifi-framework-core-0.4.1.jar:0.4.1] > > at > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:72) > ~[nifi-framework-core-0.4.1.jar:0.4.1] > > at > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:608) > ~[nifi-framework-core-0.4.1.jar:0.4.1] > > at > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:709) > ~[nifi-framework-core-0.4.1.jar:0.4.1] > > ... 4 common frames omitted > > > What I see is that the flow.tar file on the slave is 0 bytes. This does not > look right. Since the file on the NCM is about 20K bytes. > > > thanks > > Paresh > > > > The information contained in this transmission may contain privileged and > confidential information. It is intended only for the use of the person(s) > named above. If you are not the intended recipient, you are hereby notified > that any review, dissemination, distribution or duplication of this > communication is strictly prohibited. If you are not the intended recipient, > please contact the sender by reply email and destroy all copies of the > original message. >
Re: Getting error when a previously disconnected node is being brought up and trying to connect back to the NCM
But our scenario is as follows. 1. We will deploy a cluster with NCM and Primary. 2. We deploy out pipelines. 3. Add slave nodes to the cluster. In this scenairo we cannot be wiping way the flow.* files since we will loose the pipelines, right? Paresh On 2/1/16, 11:48 AM, "Joe Witt" wrote: >Paresh, > >In the log it says "Caused by: >org.apache.nifi.controller.UninheritableFlowException: Proposed >configuration is not inheritable by the flow controller because of >flow differences: Found difference in Templates:". So on your node >blow away both the flow.xml.gz, flow.tar, and the templates directory. >In an upcoming release we will relax the template matching >requirement. That was a mistake. > >Thanks >Joe > >On Mon, Feb 1, 2016 at 2:45 PM, Paresh Shah >wrote: >> Here¹s the scenario. >> >> 1. Deploy Nifi Cluster : >> - NCM >> - Pirmary >> - Node. >> >> 2. Verified that everything is OK. >> >> 3. I then disconnected the slave Node and deployed some pipelines on >>the cluster. >> >> 4. Now I tried to bring up the slave node. It gives the following error. >> >> >> 2016-02-01 19:41:05,393 WARN [main] >>org.apache.nifi.web.server.JettyServer Failed to start web server... >>shutting down. >> >> java.lang.Exception: Unable to load flow due to: java.io.IOException: >>org.apache.nifi.cluster.ConnectionException: Failed to connect node to >>cluster because local flow is different than cluster flow. >> >> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:784) >>~[nifi-jetty-0.4.1.jar:0.4.1] >> >> at org.apache.nifi.NiFi.(NiFi.java:137) >>[nifi-runtime-0.4.1.jar:0.4.1] >> >> at org.apache.nifi.NiFi.main(NiFi.java:227) >>[nifi-runtime-0.4.1.jar:0.4.1] >> >> Caused by: java.io.IOException: >>org.apache.nifi.cluster.ConnectionException: Failed to connect node to >>cluster because local flow is different than cluster flow. >> >> at >>org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.j >>ava:453) ~[nifi-framework-core-0.4.1.jar:0.4.1] >> >> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:775) >>~[nifi-jetty-0.4.1.jar:0.4.1] >> >> ... 2 common frames omitted >> >> Caused by: org.apache.nifi.cluster.ConnectionException: Failed to >>connect node to cluster because local flow is different than cluster >>flow. >> >> at >>org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse >>(StandardFlowService.java:732) ~[nifi-framework-core-0.4.1.jar:0.4.1] >> >> at >>org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.j >>ava:433) ~[nifi-framework-core-0.4.1.jar:0.4.1] >> >> ... 3 common frames omitted >> >> Caused by: org.apache.nifi.controller.UninheritableFlowException: >>Proposed configuration is not inheritable by the flow controller because >>of flow differences: Found difference in Templates: >> >> Local Fingerprint: >>fc0a802d-0915-3b71-8694-52b8d7b2c556melissa_rawalert_daq_dataavailability >>NO_CONNECTIONSNO_FUNNELSNO_ >> >> Cluster Fingerprint: >>42809382-bb1f-3785-b3c3-7d8b875c3e88melissa_rawalert_reqest_dataavailabil >>ity_daqNO_CONNECTIONSNO_FUN >> >> at >>org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSync >>hronizer.java:223) ~[nifi-framework-core-0.4.1.jar:0.4.1] >> >> at >>org.apache.nifi.controller.FlowController.synchronize(FlowController.java >>:1203) ~[nifi-framework-core-0.4.1.jar:0.4.1] >> >> at >>org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(Standard >>XMLFlowConfigurationDAO.java:72) ~[nifi-framework-core-0.4.1.jar:0.4.1] >> >> at >>org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlow >>Service.java:608) ~[nifi-framework-core-0.4.1.jar:0.4.1] >> >> at >>org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse >>(StandardFlowService.java:709) ~[nifi-framework-core-0.4.1.jar:0.4.1] >> >> ... 4 common frames omitted >> >> >> What I see is that the flow.tar file on the slave is 0 bytes. This does >>not look right. Since the file on the NCM is about 20K bytes. >> >> >> thanks >> >> Paresh >> >> >> >> The information contained in this transmission may contain privileged >>and confidential information. It is intended only for the use of the >>person(s) named above. If you are not the intended recipient, you are >>hereby notified that any review, dissemination, distribution or >>duplication of this communication is strictly prohibited. If you are not >>the intended recipient, please contact the sender by reply email and >>destroy all copies of the original message. >> The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient
[GitHub] nifi pull request: [REVIEW ONLY]]NIFI-865 added initial support fo...
Github user olegz closed the pull request at: https://github.com/apache/nifi/pull/199 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: [REVIEW ONLY]]NIFI-865 added initial support fo...
Github user olegz commented on the pull request: https://github.com/apache/nifi/pull/199#issuecomment-178159306 Closing it after initial review. Will open a new one once all comments are addressed and cleanup. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-865 added initial support for AMQP publish...
GitHub user olegz opened a pull request: https://github.com/apache/nifi/pull/200 NIFI-865 added initial support for AMQP publish/subscribe added initial documentation and testing. More tesing will be added You can merge this pull request into a Git repository by running: $ git pull https://github.com/olegz/nifi NIFI-865 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/200.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #200 commit 0fb8f008a8469d8e05864a7d95ad11f1a18ac0e1 Author: Oleg Zhurakousky Date: 2016-01-31T18:20:14Z NIFI-865 added initial support for AMQP publish/subscribe added initial documentation and testing. More tesing will be added --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Getting error when a previously disconnected node is being brought up and trying to connect back to the NCM
Sorry - i only mean to recommend clearing out the flow config on the node including the templates on the node. When it rejoins the cluster it will inherit the flow from the NCM. On Mon, Feb 1, 2016 at 2:51 PM, Paresh Shah wrote: > But our scenario is as follows. > > 1. We will deploy a cluster with NCM and Primary. > 2. We deploy out pipelines. > 3. Add slave nodes to the cluster. > > In this scenairo we cannot be wiping way the flow.* files since we will > loose the pipelines, right? > > Paresh > > On 2/1/16, 11:48 AM, "Joe Witt" wrote: > >>Paresh, >> >>In the log it says "Caused by: >>org.apache.nifi.controller.UninheritableFlowException: Proposed >>configuration is not inheritable by the flow controller because of >>flow differences: Found difference in Templates:". So on your node >>blow away both the flow.xml.gz, flow.tar, and the templates directory. >>In an upcoming release we will relax the template matching >>requirement. That was a mistake. >> >>Thanks >>Joe >> >>On Mon, Feb 1, 2016 at 2:45 PM, Paresh Shah >>wrote: >>> Here¹s the scenario. >>> >>> 1. Deploy Nifi Cluster : >>> - NCM >>> - Pirmary >>> - Node. >>> >>> 2. Verified that everything is OK. >>> >>> 3. I then disconnected the slave Node and deployed some pipelines on >>>the cluster. >>> >>> 4. Now I tried to bring up the slave node. It gives the following error. >>> >>> >>> 2016-02-01 19:41:05,393 WARN [main] >>>org.apache.nifi.web.server.JettyServer Failed to start web server... >>>shutting down. >>> >>> java.lang.Exception: Unable to load flow due to: java.io.IOException: >>>org.apache.nifi.cluster.ConnectionException: Failed to connect node to >>>cluster because local flow is different than cluster flow. >>> >>> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:784) >>>~[nifi-jetty-0.4.1.jar:0.4.1] >>> >>> at org.apache.nifi.NiFi.(NiFi.java:137) >>>[nifi-runtime-0.4.1.jar:0.4.1] >>> >>> at org.apache.nifi.NiFi.main(NiFi.java:227) >>>[nifi-runtime-0.4.1.jar:0.4.1] >>> >>> Caused by: java.io.IOException: >>>org.apache.nifi.cluster.ConnectionException: Failed to connect node to >>>cluster because local flow is different than cluster flow. >>> >>> at >>>org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.j >>>ava:453) ~[nifi-framework-core-0.4.1.jar:0.4.1] >>> >>> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:775) >>>~[nifi-jetty-0.4.1.jar:0.4.1] >>> >>> ... 2 common frames omitted >>> >>> Caused by: org.apache.nifi.cluster.ConnectionException: Failed to >>>connect node to cluster because local flow is different than cluster >>>flow. >>> >>> at >>>org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse >>>(StandardFlowService.java:732) ~[nifi-framework-core-0.4.1.jar:0.4.1] >>> >>> at >>>org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.j >>>ava:433) ~[nifi-framework-core-0.4.1.jar:0.4.1] >>> >>> ... 3 common frames omitted >>> >>> Caused by: org.apache.nifi.controller.UninheritableFlowException: >>>Proposed configuration is not inheritable by the flow controller because >>>of flow differences: Found difference in Templates: >>> >>> Local Fingerprint: >>>fc0a802d-0915-3b71-8694-52b8d7b2c556melissa_rawalert_daq_dataavailability >>>NO_CONNECTIONSNO_FUNNELSNO_ >>> >>> Cluster Fingerprint: >>>42809382-bb1f-3785-b3c3-7d8b875c3e88melissa_rawalert_reqest_dataavailabil >>>ity_daqNO_CONNECTIONSNO_FUN >>> >>> at >>>org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSync >>>hronizer.java:223) ~[nifi-framework-core-0.4.1.jar:0.4.1] >>> >>> at >>>org.apache.nifi.controller.FlowController.synchronize(FlowController.java >>>:1203) ~[nifi-framework-core-0.4.1.jar:0.4.1] >>> >>> at >>>org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(Standard >>>XMLFlowConfigurationDAO.java:72) ~[nifi-framework-core-0.4.1.jar:0.4.1] >>> >>> at >>>org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlow >>>Service.java:608) ~[nifi-framework-core-0.4.1.jar:0.4.1] >>> >>> at >>>org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse >>>(StandardFlowService.java:709) ~[nifi-framework-core-0.4.1.jar:0.4.1] >>> >>> ... 4 common frames omitted >>> >>> >>> What I see is that the flow.tar file on the slave is 0 bytes. This does >>>not look right. Since the file on the NCM is about 20K bytes. >>> >>> >>> thanks >>> >>> Paresh >>> >>> >>> >>> The information contained in this transmission may contain privileged >>>and confidential information. It is intended only for the use of the >>>person(s) named above. If you are not the intended recipient, you are >>>hereby notified that any review, dissemination, distribution or >>>duplication of this communication is strictly prohibited. If you are not >>>the intended recipient, please contact the sender by reply email and >>>destroy all copies of the original message. >>> > > ___
Re: Getting error when a previously disconnected node is being brought up and trying to connect back to the NCM
The Nodes will not have a flow.tar. Only the NCM has a flow.tar (*do not delete your flow.tar*). The Nodes will get both their templates and flow.xml.gz from the NCM on startup, so it is safe to remove them on the node. On Mon, Feb 1, 2016 at 2:48 PM, Joe Witt wrote: > Paresh, > > In the log it says "Caused by: > org.apache.nifi.controller.UninheritableFlowException: Proposed > configuration is not inheritable by the flow controller because of > flow differences: Found difference in Templates:". So on your node > blow away both the flow.xml.gz, flow.tar, and the templates directory. > In an upcoming release we will relax the template matching > requirement. That was a mistake. > > Thanks > Joe > > On Mon, Feb 1, 2016 at 2:45 PM, Paresh Shah > wrote: > > Here’s the scenario. > > > > 1. Deploy Nifi Cluster : > > - NCM > > - Pirmary > > - Node. > > > > 2. Verified that everything is OK. > > > > 3. I then disconnected the slave Node and deployed some pipelines on the > cluster. > > > > 4. Now I tried to bring up the slave node. It gives the following error. > > > > > > 2016-02-01 19:41:05,393 WARN [main] > org.apache.nifi.web.server.JettyServer Failed to start web server... > shutting down. > > > > java.lang.Exception: Unable to load flow due to: java.io.IOException: > org.apache.nifi.cluster.ConnectionException: Failed to connect node to > cluster because local flow is different than cluster flow. > > > > at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:784) > ~[nifi-jetty-0.4.1.jar:0.4.1] > > > > at org.apache.nifi.NiFi.(NiFi.java:137) > [nifi-runtime-0.4.1.jar:0.4.1] > > > > at org.apache.nifi.NiFi.main(NiFi.java:227) > [nifi-runtime-0.4.1.jar:0.4.1] > > > > Caused by: java.io.IOException: > org.apache.nifi.cluster.ConnectionException: Failed to connect node to > cluster because local flow is different than cluster flow. > > > > at > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:453) > ~[nifi-framework-core-0.4.1.jar:0.4.1] > > > > at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:775) > ~[nifi-jetty-0.4.1.jar:0.4.1] > > > > ... 2 common frames omitted > > > > Caused by: org.apache.nifi.cluster.ConnectionException: Failed to > connect node to cluster because local flow is different than cluster flow. > > > > at > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:732) > ~[nifi-framework-core-0.4.1.jar:0.4.1] > > > > at > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:433) > ~[nifi-framework-core-0.4.1.jar:0.4.1] > > > > ... 3 common frames omitted > > > > Caused by: org.apache.nifi.controller.UninheritableFlowException: > Proposed configuration is not inheritable by the flow controller because of > flow differences: Found difference in Templates: > > > > Local Fingerprint: > > fc0a802d-0915-3b71-8694-52b8d7b2c556melissa_rawalert_daq_dataavailabilityNO_CONNECTIONSNO_FUNNELSNO_ > > > > Cluster Fingerprint: > 42809382-bb1f-3785-b3c3-7d8b875c3e88melissa_rawalert_reqest_dataavailability_daqNO_CONNECTIONSNO_FUN > > > > at > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:223) > ~[nifi-framework-core-0.4.1.jar:0.4.1] > > > > at > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1203) > ~[nifi-framework-core-0.4.1.jar:0.4.1] > > > > at > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:72) > ~[nifi-framework-core-0.4.1.jar:0.4.1] > > > > at > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:608) > ~[nifi-framework-core-0.4.1.jar:0.4.1] > > > > at > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:709) > ~[nifi-framework-core-0.4.1.jar:0.4.1] > > > > ... 4 common frames omitted > > > > > > What I see is that the flow.tar file on the slave is 0 bytes. This does > not look right. Since the file on the NCM is about 20K bytes. > > > > > > thanks > > > > Paresh > > > > > > > > The information contained in this transmission may contain privileged > and confidential information. It is intended only for the use of the > person(s) named above. If you are not the intended recipient, you are > hereby notified that any review, dissemination, distribution or duplication > of this communication is strictly prohibited. If you are not the intended > recipient, please contact the sender by reply email and destroy all copies > of the original message. > > >
Re: Getting error when a previously disconnected node is being brought up and trying to connect back to the NCM
Thanks for the clarification. On 2/1/16, 11:59 AM, "Joe Witt" wrote: >Sorry - i only mean to recommend clearing out the flow config on the >node including the templates on the node. When it rejoins the cluster >it will inherit the flow from the NCM. > >On Mon, Feb 1, 2016 at 2:51 PM, Paresh Shah >wrote: >> But our scenario is as follows. >> >> 1. We will deploy a cluster with NCM and Primary. >> 2. We deploy out pipelines. >> 3. Add slave nodes to the cluster. >> >> In this scenairo we cannot be wiping way the flow.* files since we will >> loose the pipelines, right? >> >> Paresh >> >> On 2/1/16, 11:48 AM, "Joe Witt" wrote: >> >>>Paresh, >>> >>>In the log it says "Caused by: >>>org.apache.nifi.controller.UninheritableFlowException: Proposed >>>configuration is not inheritable by the flow controller because of >>>flow differences: Found difference in Templates:". So on your node >>>blow away both the flow.xml.gz, flow.tar, and the templates directory. >>>In an upcoming release we will relax the template matching >>>requirement. That was a mistake. >>> >>>Thanks >>>Joe >>> >>>On Mon, Feb 1, 2016 at 2:45 PM, Paresh Shah >>>wrote: Here¹s the scenario. 1. Deploy Nifi Cluster : - NCM - Pirmary - Node. 2. Verified that everything is OK. 3. I then disconnected the slave Node and deployed some pipelines on the cluster. 4. Now I tried to bring up the slave node. It gives the following error. 2016-02-01 19:41:05,393 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down. java.lang.Exception: Unable to load flow due to: java.io.IOException: org.apache.nifi.cluster.ConnectionException: Failed to connect node to cluster because local flow is different than cluster flow. at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:784) ~[nifi-jetty-0.4.1.jar:0.4.1] at org.apache.nifi.NiFi.(NiFi.java:137) [nifi-runtime-0.4.1.jar:0.4.1] at org.apache.nifi.NiFi.main(NiFi.java:227) [nifi-runtime-0.4.1.jar:0.4.1] Caused by: java.io.IOException: org.apache.nifi.cluster.ConnectionException: Failed to connect node to cluster because local flow is different than cluster flow. at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService .j ava:453) ~[nifi-framework-core-0.4.1.jar:0.4.1] at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:775) ~[nifi-jetty-0.4.1.jar:0.4.1] ... 2 common frames omitted Caused by: org.apache.nifi.cluster.ConnectionException: Failed to connect node to cluster because local flow is different than cluster flow. at org.apache.nifi.controller.StandardFlowService.loadFromConnectionRespon se (StandardFlowService.java:732) ~[nifi-framework-core-0.4.1.jar:0.4.1] at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService .j ava:433) ~[nifi-framework-core-0.4.1.jar:0.4.1] ... 3 common frames omitted Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed configuration is not inheritable by the flow controller because of flow differences: Found difference in Templates: Local Fingerprint: fc0a802d-0915-3b71-8694-52b8d7b2c556melissa_rawalert_daq_dataavailabili ty NO_CONNECTIONSNO_FUNNELSNO_ Cluster Fingerprint: 42809382-bb1f-3785-b3c3-7d8b875c3e88melissa_rawalert_reqest_dataavailab il ity_daqNO_CONNECTIONSNO_FUN at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSy nc hronizer.java:223) ~[nifi-framework-core-0.4.1.jar:0.4.1] at org.apache.nifi.controller.FlowController.synchronize(FlowController.ja va :1203) ~[nifi-framework-core-0.4.1.jar:0.4.1] at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(Standa rd XMLFlowConfigurationDAO.java:72) ~[nifi-framework-core-0.4.1.jar:0.4.1] at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFl ow Service.java:608) ~[nifi-framework-core-0.4.1.jar:0.4.1] at org.apache.nifi.controller.StandardFlowService.loadFromConnectionRespon se (StandardFlowService.java:709) ~[nifi-framework-core-0.4.1.jar:0.4.1] ... 4 common frames omitted What I see is that the flow.tar file on the slave is 0 bytes. This does not look right. Since the file on the NCM is about 20K bytes. thanks Paresh The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination,
[GitHub] nifi pull request: NIFI-1417 Exposing several connection settings ...
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/194#discussion_r51478207
--- Diff:
nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/java/org/apache/nifi/processors/solr/SolrProcessor.java
---
@@ -120,6 +202,22 @@ protected final SolrClient getSolrClient() {
.explanation("A collection must specified for Solr
Type of Cloud")
.build());
}
+
+if (!context.getProperty(ZK_CLIENT_TIMEOUT).isSet()) {
--- End diff --
This is 'dead code' as isSet() will always return 'true', since there is a
defaultValue
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1417 Exposing several connection settings ...
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/194#discussion_r51478213
--- Diff:
nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/java/org/apache/nifi/processors/solr/SolrProcessor.java
---
@@ -120,6 +202,22 @@ protected final SolrClient getSolrClient() {
.explanation("A collection must specified for Solr
Type of Cloud")
.build());
}
+
+if (!context.getProperty(ZK_CLIENT_TIMEOUT).isSet()) {
+problems.add(new ValidationResult.Builder()
+.subject(ZK_CLIENT_TIMEOUT.getName())
+.valid(false)
+.explanation("A ZooKeeper Client Timeout must
specified for Solr Type of Cloud")
+.build());
+}
+
+if (!context.getProperty(ZK_CONNECTION_TIMEOUT).isSet()) {
--- End diff --
This is 'dead code' as isSet() will always return 'true', since there is a
defaultValue
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1417 Exposing several connection settings ...
Github user markap14 commented on the pull request: https://github.com/apache/nifi/pull/194#issuecomment-178187346 I don't know how to really reproduce the issue either, but i agree that these are good properties to introduce. All new properties have defaults so flows will not be broken. Unit tests pass and contrib-check passes. While I don't think the added checks are necessary in the customValidate, all else looks good. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1275: Add processor(s) support for Elastic...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/180#discussion_r51482142
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractElasticsearchProcessor.java
---
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.util.StringUtils;
+import org.elasticsearch.client.Client;
+import org.elasticsearch.client.transport.TransportClient;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.common.transport.InetSocketTransportAddress;
+
+import java.net.InetSocketAddress;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicReference;
+
+
+public abstract class AbstractElasticsearchProcessor extends
AbstractProcessor {
+
+protected static final PropertyDescriptor CLUSTER_NAME = new
PropertyDescriptor.Builder()
+.name("Cluster Name")
+.description("Name of the ES cluster (for example,
elasticsearch_brew). Defaults to 'elasticsearch'")
+.required(false)
+.addValidator(Validator.VALID)
+.defaultValue("elasticsearch")
+.build();
+protected static final PropertyDescriptor HOSTS = new
PropertyDescriptor.Builder()
+.name("ElasticSearch Hosts")
+.description("ElasticSearch Hosts, which should be comma
separated and colon for hostname/port "
++ "host1:port,host2:port, For example
testcluster:9200.")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PROP_SSL_CONTEXT_SERVICE = new
PropertyDescriptor.Builder()
+.name("SSL Context Service")
+.description("The SSL Context Service used to provide client
certificate information for TLS/SSL "
++ "connections. This service only applies if the
Shield plugin is available.")
+.required(false)
+.identifiesControllerService(SSLContextService.class)
+.build();
+
+public static final PropertyDescriptor PROP_SHIELD_LOCATION = new
PropertyDescriptor.Builder()
+.name("Shield plugin URL")
+.description("Specifies the location of the JAR for the
Elasticsearch Shield plugin."
++ " If the Elasticsearch cluster has been secured with
the Shield plugin, then "
++ "an SSL Context Service must be defined, and the
Shield plugin JAR must also "
++ "be available to this processor.")
+.required(false)
+.addValidator(StandardValidators.URL_VALIDATOR)
--- End diff --
I believe we should force this to be a file path instead of a URL so there
isn't a network call out to grab the jar every time the client needs to restart.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or fi
[GitHub] nifi pull request: NIFI-1275: Add processor(s) support for Elastic...
Github user JPercivall commented on the pull request: https://github.com/apache/nifi/pull/180#issuecomment-178198337 There needs to be support added for passing username/password (specifically for shield integration) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1275: Add processor(s) support for Elastic...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/180#discussion_r51483875
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractElasticsearchProcessor.java
---
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.util.StringUtils;
+import org.elasticsearch.client.Client;
+import org.elasticsearch.client.transport.TransportClient;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.common.transport.InetSocketTransportAddress;
+
+import java.net.InetSocketAddress;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicReference;
+
+
+public abstract class AbstractElasticsearchProcessor extends
AbstractProcessor {
+
+protected static final PropertyDescriptor CLUSTER_NAME = new
PropertyDescriptor.Builder()
+.name("Cluster Name")
+.description("Name of the ES cluster (for example,
elasticsearch_brew). Defaults to 'elasticsearch'")
+.required(false)
+.addValidator(Validator.VALID)
+.defaultValue("elasticsearch")
+.build();
+protected static final PropertyDescriptor HOSTS = new
PropertyDescriptor.Builder()
+.name("ElasticSearch Hosts")
+.description("ElasticSearch Hosts, which should be comma
separated and colon for hostname/port "
++ "host1:port,host2:port, For example
testcluster:9200.")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PROP_SSL_CONTEXT_SERVICE = new
PropertyDescriptor.Builder()
+.name("SSL Context Service")
+.description("The SSL Context Service used to provide client
certificate information for TLS/SSL "
++ "connections. This service only applies if the
Shield plugin is available.")
+.required(false)
+.identifiesControllerService(SSLContextService.class)
+.build();
+
+public static final PropertyDescriptor PROP_SHIELD_LOCATION = new
PropertyDescriptor.Builder()
+.name("Shield plugin URL")
+.description("Specifies the location of the JAR for the
Elasticsearch Shield plugin."
++ " If the Elasticsearch cluster has been secured with
the Shield plugin, then "
--- End diff --
A bit nit picky but, the space at the beginning of this string should be at
the end of the line before
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1417 Exposing several connection settings ...
Github user bbende commented on the pull request: https://github.com/apache/nifi/pull/194#issuecomment-178209934 @markap14 thanks for reviewing! Good point about the isSet(). I was thinking someone could remove the default value, but didn't realize that isn't possible. I'll clean that up and commit. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Nifi jobs support Mesos ?
Hi Folks, Forgive me as I'm new to Nifi. The tool seems fantastic, the UI is super easy to use. It was easy to extend and build a custom processor which is great. Looking at our processing, we're curious if we could use Nifi to replace some of our batch processes. Setting up some of the basic routines, scheduled by cron is all working. However we would like the ability to scale jobs out, horizontally and distribute processing across a number of Mesos nodes. Has anyone done this ? any notes, tips or documentation on how to set this up ? Thanks.
[GitHub] nifi pull request: NIFI-210: Add ExecuteScript and InvokeScriptedP...
Github user joewitt commented on a diff in the pull request:
https://github.com/apache/nifi/pull/185#discussion_r51488029
--- Diff:
nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/processors/script/InvokeScriptedProcessor.java
---
@@ -0,0 +1,606 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.script;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.ControllerServiceLookup;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Processor;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.file.monitor.LastModifiedMonitor;
+import org.apache.nifi.util.file.monitor.SynchronousFileWatcher;
+
+import javax.script.Invocable;
+import java.io.FileInputStream;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+@Tags({"script", "invoke", "groovy", "python", "jython", "jruby", "ruby",
"javascript", "js", "lua", "luaj"})
+@CapabilityDescription("Experimental - Invokes a script engine for a
Processor defined in the given script. The script must define "
++ "a valid class that implements the Processor interface, and it
must set a variable 'processor' to an instance of "
++ "the class. Processor methods such as onTrigger() will be
delegated to the scripted Processor instance. Also any "
++ "Relationships or PropertyDescriptors defined by the scripted
processor will be added to the configuration dialog. "
++ "Experimental: Impact of sustained usage not yet verified.")
+@DynamicProperty(name = "A script engine property to update", value = "The
value to set it to", supportsExpressionLanguage = true,
+description = "Updates a script engine property specified by the
Dynamic Property's key with the value specified by the Dynamic Property's
value")
+@SeeAlso({ExecuteScript.class})
+public class InvokeScriptedProcessor extends AbstractScriptProcessor {
+
+private final AtomicReference processor = new
AtomicReference<>();
+private final AtomicReference>
validationResults =
+new AtomicReference<>((Collection) new
ArrayList());
+
+private final Lock lock = new ReentrantLock();
+private SynchronousFileWatcher scriptWatcher;
+
+private ScheduledExecutorService reloadService = null;
+
+/**
+ * Creates the resources needed by this processor. An attempt is made
to also initialize the scripted processor,
+ * but unless the properties (such as script engine name and script
file path) have already been specifi
[GitHub] nifi pull request: NIFI-1275: Add processor(s) support for Elastic...
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/180#discussion_r51490929
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractElasticsearchProcessor.java
---
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.util.StringUtils;
+import org.elasticsearch.client.Client;
+import org.elasticsearch.client.transport.TransportClient;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.common.transport.InetSocketTransportAddress;
+
+import java.net.InetSocketAddress;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicReference;
+
+
+public abstract class AbstractElasticsearchProcessor extends
AbstractProcessor {
+
+protected static final PropertyDescriptor CLUSTER_NAME = new
PropertyDescriptor.Builder()
+.name("Cluster Name")
+.description("Name of the ES cluster (for example,
elasticsearch_brew). Defaults to 'elasticsearch'")
+.required(false)
+.addValidator(Validator.VALID)
+.defaultValue("elasticsearch")
+.build();
+protected static final PropertyDescriptor HOSTS = new
PropertyDescriptor.Builder()
+.name("ElasticSearch Hosts")
+.description("ElasticSearch Hosts, which should be comma
separated and colon for hostname/port "
++ "host1:port,host2:port, For example
testcluster:9200.")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PROP_SSL_CONTEXT_SERVICE = new
PropertyDescriptor.Builder()
+.name("SSL Context Service")
+.description("The SSL Context Service used to provide client
certificate information for TLS/SSL "
++ "connections. This service only applies if the
Shield plugin is available.")
+.required(false)
+.identifiesControllerService(SSLContextService.class)
+.build();
+
+public static final PropertyDescriptor PROP_SHIELD_LOCATION = new
PropertyDescriptor.Builder()
+.name("Shield plugin URL")
+.description("Specifies the location of the JAR for the
Elasticsearch Shield plugin."
++ " If the Elasticsearch cluster has been secured with
the Shield plugin, then "
++ "an SSL Context Service must be defined, and the
Shield plugin JAR must also "
++ "be available to this processor.")
+.required(false)
+.addValidator(StandardValidators.URL_VALIDATOR)
+.build();
+
+protected static final PropertyDescriptor PING_TIMEOUT = new
PropertyDescriptor.Builder()
+.name("ElasticSearch Ping Timeout")
+.description("The ping timeout used to determine when a node
is unreachable. " +
+"For example, 5s (5 seconds). If non-local recommended
is 30s")
+.required(true)
+.defaultValue("5s")
+.addValidator(
status of 050 tickets
Team, We still have quite a few tickets for 050 outstanding so freezing for the RC just isn't there yet. Of the tickets below all are in review many of which after much discussion and progress. Expect several of them to close very soon. Once we have them knocked out Tony can do the RC process. This is a very exciting RC with not only good stability fixes, new features, but also a ton of very cool new community contributions. Active tickets under review: NIFI-1257,NIFI-1259 - Improve encryption logic NIFI-210 - add support for groovy, python, lua, ruby, javascript NIFI-259, NIFI-1223, NIFI-1339 - Add state management as a framework feature NIFI-865 - Add support for get/put AMQP NIFI-1107 - Upload multi part objects to S3 NIFI-1275 - Add support for get/put ElasticSearch NIFI-1417 - Add timeout/better handling of Put SOLR NIFI-1457 - Fix bulletin/error handling in cluster/primary node situation NIFI-1379 - Kurc is the RM. Awaiting closeout of the above. Thanks Joe
[GitHub] nifi pull request: NIFI-924:Nifi-Camel Integration
Github user PuspenduBanerjee commented on the pull request:
https://github.com/apache/nifi/pull/197#issuecomment-178252892
@joewitt For Non-Uniform charset attached piece should work.
```java
/**
* @param rawBytes
* @param patternToMatch
* @return
*/
private static boolean matchPattern(byte[] rawBytes , final String
patternToMatch){
out.println("Pattern= "+patternToMatch);
final Pattern pattern = Pattern.compile(patternToMatch);
final int patternByteLength=patternToMatch.getBytes().length;
out.println("PatternByteLength= "+patternByteLength);
final int minSliceByteLength=2;
final int sliceByteLength = patternByteLength > minSliceByteLength
? patternByteLength : minSliceByteLength;
int offset = 0;
while (offset <= rawBytes.length) {
String s=new String(Arrays.copyOfRange(rawBytes,
offset>patternByteLength? offset-(patternByteLength+1):0,
(offset +
sliceByteLength)));
out.println("Retrieved byte[]= "+s);
if (pattern.matcher(s).find()) {
return true;
}
offset += sliceByteLength;
};
return false;
}
```
I tried with \:
```java
matchPattern("Iâ¥NYâ¥Yâ¥NABCDdkk;lskiepdmclod;die;'cnjywkl;sfdsXRSHSYâ¥â¥Y".getBytes(),
"â¥â¥Y");
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1417 Exposing several connection settings ...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/194 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-924:Nifi-Camel Integration
Github user olegz commented on the pull request: https://github.com/apache/nifi/pull/197#issuecomment-178287033 @PuspenduBanerjee We are currently in the process of releasing 0.5 which is imminent. We'll be happy to help you through this (review, polish etc) once 0.5 is out. Hope you understand. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Nifi jobs support Mesos ?
Timothy In short it’s very possible. I personally haven’t done that, but I have written some custom code submitting jobs to YARN which I am assuming would be very similar. Would be happy and actually interested to take it a bit further. Could you elaborate on the use case a bit more? For example; Where is the source of data? How does Mesos know about where the data is? Sorry some of these question may be very dumb (not much experience with Mesos other then general understanding) and what responsibility do you envision NiFi to have over such process? I’ll share my example with NiFi and Tez where a custom NiFi Processor would submit job to Tez and upon completion would create a FlowFile that would trigger another processor which would invoke another Tez job and so on. Tez essentially is a processing wrapper that uses YARN for resource management. So I am assuming that something similar would be with Mesos. Am I on the right track? Cheers Oleg > On Feb 1, 2016, at 4:53 PM, Timothy Findlay > wrote: > > Hi Folks, > > > Forgive me as I'm new to Nifi. The tool seems fantastic, the UI is super easy > to use. It was easy to extend and build a custom processor which is great. > > > Looking at our processing, we're curious if we could use Nifi to replace some > of our batch processes. Setting up some of the basic routines, scheduled by > cron is all working. > > > However we would like the ability to scale jobs out, horizontally and > distribute processing across a number of Mesos nodes. > > > Has anyone done this ? any notes, tips or documentation on how to set this up > ? > > > Thanks.
[GitHub] nifi pull request: NIFI-1257 and 1259
GitHub user alopresto opened a pull request: https://github.com/apache/nifi/pull/201 NIFI-1257 and 1259 NIFI-1257 NIFI-1259 Added the mechanics for additional strong `Key Derivation Functions` (KDF) including `PBKDF2`, `Bcrypt`, and `Scrypt` as well as `keyed ciphers` (raw key; no KDF). Integrated with `EncryptContent` processor and added new property and property descriptors. Dependent properties not available until NIFI-1121 is implemented. Explanation of KDFs available here: [Apache NiFi Wiki](https://cwiki.apache.org/confluence/display/NIFI/Key+Derivation+Function+Explanations) You can merge this pull request into a Git repository by running: $ git pull https://github.com/alopresto/nifi NIFI-1257-cherrypick Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/201.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #201 commit ab566da624914d553188495a906a523ce7cbe209 Author: Andy LoPresto Date: 2015-12-22T18:12:02Z NIFI-1257: Added first skeleton of KeyDeriver functionality but re-evaluating to handle OpenSSL and NiFi legacy PBE consistently with modern KDFs. commit aaef4ec6e5551392fdf441fe2bca5f6a924206ea Author: Andy LoPresto Date: 2015-12-22T19:02:27Z NIFI-1257 Skeleton of key deriver mechanics and moving existing classes into crypto package. Added variable substitution with defaults to logback-test.xml. (+11 squashed commits) Squashed commits: [5aa2ae2] NIFI-1257: Minor formatting changes in logback-test.xml. [af12eb5] NIFI-1257: Added CipherFactory. [3e04f5b] NIFI-1257: Improved Javadoc in NiFi legacy cipher provider and OpenSSL cipher provider. [bfdc1d7] NIFI-1257: Added PBKDF2 cipher provider implementation and Java & Groovy unit test skeletons. [20adeef] NIFI-1257: Added KeyedCipherProvider interface. [302a864] NIFI-1257: Moved OpenPGP encryptor unit tests to proper package. [ce5c231] NIFI-1257: Added test logback file with variable substitution expressions. [f2475eb] NIFI-1257: Added Groovy unit test support to allow faster development cycle. [4004cfd] NIFI-1257: Added OpenSSL PKCS#5 v1.5 EVP_BytesToKey cipher provider and unit test. [332b9dc] NIFI-1257: Moved crypto classes into new package. Began refactor from KeyDeriver to CipherProvider. [5277efc] NIFI-1257: Added variables with default value to logback-test.xml. commit 07c1e1219153ee3ab5a030f68f0647ad41bdd3b9 Author: Andy LoPresto Date: 2016-01-13T20:25:34Z NIFI-1257 Committed state before refactor to inherit from shared CipherProvider interface. (+5 squashed commits) Squashed commits: [634802c] NIFI-1257 Added new keyed encryption methods and added boolean field for compatibility with new KDFs. [b74d23c] NIFI-1257 Added parseKeyLengthFromAlgorithm utility method. Added unit tests. [f3cd3b6] NIFI-1257 Moved cipher algorithm parsing utility method to common util class. [b0e785e] NIFI-1257 Applied contrib-check fixes for line length. [94a6ecf] NIFI-1257 Added Apache License to various tests. commit 106de28eeda3838413226185dfe52e0c83bbf04a Author: Andy LoPresto Date: 2016-01-14T02:00:34Z NIFI-1257 Implemented PBKDF2 cipher provider. Added default constructor with strong choices for PBKDF2 cipher provider. (+12 squashed commits) Squashed commits: [19f94a0] NIFI-1257 Implemented NiFiLegacyCipherProvider and added unit tests. [efcb5f6] NIFI-1257 Added convenience methods in OpenSSLPKCS5 provider that ignore desired key lengths (all OpenSSL algorithms specify key length in algorithm). [0381a06] NIFI-1257 Moved BC provider loading in test to single static invocation (decreases test time by ~40%). [b5a8682] NIFI-1257 Added ASF license to openssl_pbkdf2.rb. [87cfca0] NIFI-1257 Test cleanup. [cb3598b] NIFI-1257 Added key length parameter to PBKDF2 cipher provider. [aa50006] NIFI-1257 Added PRF resolution to PBKDF2 cipher provider. [9dfb6b0] NIFI-1257 Changed PBKDF2 and OpenSSLPKCS5 implementations to accept EncryptionMethod instead of individual algorithm and provider combinations. [d6156bd] NIFI-1257 Added test resource to generate PBKDF2-derived key and encrypt content using OpenSSL via Ruby. [6cee174] NIFI-1257 Removed duplicated test. [5ff87a4] NIFI-1257 Temporarily commented out other implementations while interface changes. [2f1ac0c] NIFI-1257 Added RandomIVPBECipherProvider to allow for non-deterministic IVs. commit 3a835c4c9a8390a41090b7a66e905a3985b39977 Author: Andy LoPresto Date: 2016-01-17T06:43:25Z NIFI-1257 Added Bcrypt implementation. Added unit tests. Added jBcrypt dependency to pom because BC Bcrypt is O
[GitHub] nifi pull request: NIFI-210: Add ExecuteScript and InvokeScriptedP...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/185 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-865 added initial support for AMQP publish...
Github user apiri commented on the pull request: https://github.com/apache/nifi/pull/200#issuecomment-178325971 Had checkstyle violations for unused imports: > [INFO] --- maven-checkstyle-plugin:2.15:check (check-style) @ nifi-amqp-processors --- [WARNING] src/test/java/org/apache/nifi/amqp/processors/TestConnection.java[21:8] (imports) UnusedImports: Unused import - java.net.InetSocketAddress. [WARNING] src/test/java/org/apache/nifi/amqp/processors/TestConnection.java[22:8] (imports) UnusedImports: Unused import - java.net.InterfaceAddress. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1434 Prevent array index exception in PutE...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/187#issuecomment-178328472 Hello @rpmiskin Do you mind closing this PR? Looks like @JPercivall just forgot to amend the comit entry to ensure it said 'This closes #187' but it does show as merged. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: status of 050 tickets
Thanks Joe, I'm finishing up reviewing/testing 1107, I was going to pull the trigger on versioning all the poms, but I think i'll wait until more of these close to help people merging. Tony On Mon, Feb 1, 2016 at 5:22 PM, Joe Witt wrote: > Team, > > We still have quite a few tickets for 050 outstanding so freezing for > the RC just isn't there yet. Of the tickets below all are in review > many of which after much discussion and progress. Expect several of > them to close very soon. Once we have them knocked out Tony can do > the RC process. > > This is a very exciting RC with not only good stability fixes, new > features, but also a ton of very cool new community contributions. > > Active tickets under review: > > NIFI-1257,NIFI-1259 > - Improve encryption logic > > NIFI-210 > - add support for groovy, python, lua, ruby, javascript > > NIFI-259, NIFI-1223, NIFI-1339 > - Add state management as a framework feature > > NIFI-865 > - Add support for get/put AMQP > > NIFI-1107 > - Upload multi part objects to S3 > > NIFI-1275 > - Add support for get/put ElasticSearch > > NIFI-1417 > - Add timeout/better handling of Put SOLR > > NIFI-1457 > - Fix bulletin/error handling in cluster/primary node situation > > NIFI-1379 > - Kurc is the RM. Awaiting closeout of the above. > > Thanks > Joe >
[GitHub] nifi pull request: NIFI-865 added initial support for AMQP publish...
Github user olegz commented on the pull request: https://github.com/apache/nifi/pull/200#issuecomment-178347191 Will fix as well as the merge conflict --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1257 and 1259
Github user alopresto commented on the pull request: https://github.com/apache/nifi/pull/201#issuecomment-178369880 The Travis CI failure is due to my use of `java.util.Base64` which was introduced in Java 8. Working on switching out to `org.apache.commons.codec.binary.Base64` but this does not provide an option to encode without padding unless using URL-safe character set (differing in "+" and "/" -> "-" and "_"). This causes compatibility issues in the Scrypt and Bcrypt salt formats. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1434 Prevent array index exception in PutE...
Github user rpmiskin closed the pull request at: https://github.com/apache/nifi/pull/187 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
