[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/543#discussion_r67722269 --- Diff: nifi-commons/nifi-processor-utilities/src/main/java/org/apache/nifi/processor/util/put/AbstractPutEventProcessor.java --- @@ -119,7 +119,24 @@ .defaultValue("10 seconds") .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR) .build(); - +public static final PropertyDescriptor OUTGOING_MESSAGE_DELIMITER = new PropertyDescriptor.Builder() +.name("Outgoing Message Delimiter") +.description("Specifies the delimiter to use when sending messages out over the same TCP stream. The delimiter is appended to each FlowFile message " ++ "that is transmitted over the stream so that the receiver can determine when one message ends and the next message begins. Users should " ++ "ensure that the FlowFile content does not contain the delimiter character to avoid errors. If it is not possible to define a delimiter " ++ "character that is not present in the FlowFile content then the user must use another processor to change the encoding of the data e.g. Base64 " ++ "encoding.") +.required(false) +.addValidator(StandardValidators.NON_EMPTY_VALIDATOR) +.expressionLanguageSupported(true) +.build(); +public static final PropertyDescriptor CONNECTION_PER_FLOWFILE = new PropertyDescriptor.Builder() --- End diff -- While I'm not sure about use-cases I don't see a problem adding it as a property which defaults to false. I assume @aviewfromspace1 had a specific use-case in mind when adding the property and since it is optional I like keeping it. Also PutTCP is marked as "TriggerWhenEmpty" so it will properly pruneIdleSenders just like PutSplunk. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #542: NIFI-1895 for 0.x
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/542#discussion_r67723205 --- Diff: nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/PutHBaseJSON.java --- @@ -89,6 +89,25 @@ .defaultValue(COMPLEX_FIELD_TEXT.getValue()) .build(); +protected static final String STRING_ENCODING_VALUE = "String"; +protected static final String BYTES_ENCODING_VALUE = "Bytes"; + +protected static final AllowableValue FIELD_ENCODING_STRING = new AllowableValue(STRING_ENCODING_VALUE, STRING_ENCODING_VALUE, +"Stores the value of each field as a UTF-8 String."); --- End diff -- That works for me. That's fair, if questions come in to the mailing list asking to clarify then we can address the description. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #542: NIFI-1895 for 0.x
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/542 Specifically looking at Bryan's commit: I looked through the code and built locally (since travis failed for un-related issues) and it looks good. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/543#discussion_r67776824 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java --- @@ -0,0 +1,290 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.Charset; +import java.util.Arrays; +import java.util.List; +import java.util.concurrent.TimeUnit; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.TriggerWhenEmpty; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.SeeAlso; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.DataUnit; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.util.put.AbstractPutEventProcessor; +import org.apache.nifi.processor.util.put.sender.ChannelSender; +import org.apache.nifi.ssl.SSLContextService; +import org.apache.nifi.stream.io.StreamUtils; +import org.apache.nifi.util.StopWatch; + +import javax.net.ssl.SSLContext; + +/** + * + * The PutTCP processor receives a FlowFile and transmits the FlowFile content over a TCP connection to the configured TCP server. By default, the FlowFiles are transmitted over the same TCP + * connection (or pool of TCP connections if multiple input threads are configured). To assist the TCP server with determining message boundaries, an optional "Outgoing Message Delimiter" string can + * be configured which is appended to the end of each FlowFiles content when it is transmitted over the TCP connection. An optional "Connection Per FlowFile" parameter can be specified to change the + * behaviour so that each FlowFiles content is transmitted over a single TCP connection which is opened when the FlowFile is received and closed after the FlowFile has been sent. This option should + * only be used for low message volume scenarios, otherwise the platform may run out of TCP sockets. + * + * + * + * This processor has the following required properties: + * + * Hostname - The IP address or host name of the destination TCP server. + * Port - The TCP port of the destination TCP server. + * + * + * + * + * This processor has the following optional properties: + * + * Connection Per FlowFile - Specifies that each FlowFiles content will be transmitted on a separate TCP connection. + * Idle Connection Expiration - The time threshold after which a TCP sender is deemed eligible for pruning - the associated TCP connection will be closed after this timeout. + * Max Size of Socket Send Buffer - The maximum size of the socket send buffer that should be used. This is a suggestion to the Operating System to indicate how big the socket buffer should + * be. If this value is set too low, the buffer may fill up before the data can be read, and incoming data will be dropped. + * Outgoing Message Delimiter - A string to append to the end of each FlowFiles content to indicate the end of the message to the TCP server. + * Timeout - The timeout period for determining an error has occurred whilst connecting or sending data. + * + * + * + * + * The following relationships are required: + *
[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/543#discussion_r6337 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestPutTCP.java --- @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard; + +import org.apache.nifi.processors.standard.util.TestPutTCPCommon; + +public class TestPutTCP extends TestPutTCPCommon { + +public TestPutTCP() { +super(); +ssl = false; +} + +@Override +public void configureProperties(String host, int port, String outgoingMessageDelimiter, boolean connectionPerFlowFile, boolean expectValid) { +runner.setProperty(PutTCP.HOSTNAME, host); +runner.setProperty(PutTCP.PORT, Integer.toString(port)); +if (outgoingMessageDelimiter != null) { +runner.setProperty(PutTCP.OUTGOING_MESSAGE_DELIMITER, outgoingMessageDelimiter); +} +if (connectionPerFlowFile) { +runner.setProperty(PutTCP.CONNECTION_PER_FLOWFILE, "true"); +} else { +runner.setProperty(PutTCP.CONNECTION_PER_FLOWFILE, "false"); +} --- End diff -- In general, I like the IF/ELSE block better but actually I think neither are appropriate here and it would better to just do `String.valueOf(connectionPerFlowFile)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/543#discussion_r67782856 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java --- @@ -0,0 +1,290 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.Charset; +import java.util.Arrays; +import java.util.List; +import java.util.concurrent.TimeUnit; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.TriggerWhenEmpty; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.SeeAlso; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.DataUnit; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.util.put.AbstractPutEventProcessor; +import org.apache.nifi.processor.util.put.sender.ChannelSender; +import org.apache.nifi.ssl.SSLContextService; +import org.apache.nifi.stream.io.StreamUtils; +import org.apache.nifi.util.StopWatch; + +import javax.net.ssl.SSLContext; + +/** + * + * The PutTCP processor receives a FlowFile and transmits the FlowFile content over a TCP connection to the configured TCP server. By default, the FlowFiles are transmitted over the same TCP + * connection (or pool of TCP connections if multiple input threads are configured). To assist the TCP server with determining message boundaries, an optional "Outgoing Message Delimiter" string can + * be configured which is appended to the end of each FlowFiles content when it is transmitted over the TCP connection. An optional "Connection Per FlowFile" parameter can be specified to change the + * behaviour so that each FlowFiles content is transmitted over a single TCP connection which is opened when the FlowFile is received and closed after the FlowFile has been sent. This option should + * only be used for low message volume scenarios, otherwise the platform may run out of TCP sockets. + * + * + * + * This processor has the following required properties: + * + * Hostname - The IP address or host name of the destination TCP server. + * Port - The TCP port of the destination TCP server. + * + * + * + * + * This processor has the following optional properties: + * + * Connection Per FlowFile - Specifies that each FlowFiles content will be transmitted on a separate TCP connection. + * Idle Connection Expiration - The time threshold after which a TCP sender is deemed eligible for pruning - the associated TCP connection will be closed after this timeout. + * Max Size of Socket Send Buffer - The maximum size of the socket send buffer that should be used. This is a suggestion to the Operating System to indicate how big the socket buffer should + * be. If this value is set too low, the buffer may fill up before the data can be read, and incoming data will be dropped. + * Outgoing Message Delimiter - A string to append to the end of each FlowFiles content to indicate the end of the message to the TCP server. + * Timeout - The timeout period for determining an error has occurred whilst connecting or sending data. + * + * + * + * + * The following relationships are required: + *
[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/543#discussion_r67782966 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java --- @@ -0,0 +1,290 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.Charset; +import java.util.Arrays; +import java.util.List; +import java.util.concurrent.TimeUnit; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.TriggerWhenEmpty; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.SeeAlso; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.DataUnit; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.util.put.AbstractPutEventProcessor; +import org.apache.nifi.processor.util.put.sender.ChannelSender; +import org.apache.nifi.ssl.SSLContextService; +import org.apache.nifi.stream.io.StreamUtils; +import org.apache.nifi.util.StopWatch; + +import javax.net.ssl.SSLContext; + +/** + * + * The PutTCP processor receives a FlowFile and transmits the FlowFile content over a TCP connection to the configured TCP server. By default, the FlowFiles are transmitted over the same TCP + * connection (or pool of TCP connections if multiple input threads are configured). To assist the TCP server with determining message boundaries, an optional "Outgoing Message Delimiter" string can + * be configured which is appended to the end of each FlowFiles content when it is transmitted over the TCP connection. An optional "Connection Per FlowFile" parameter can be specified to change the + * behaviour so that each FlowFiles content is transmitted over a single TCP connection which is opened when the FlowFile is received and closed after the FlowFile has been sent. This option should + * only be used for low message volume scenarios, otherwise the platform may run out of TCP sockets. + * + * + * + * This processor has the following required properties: + * + * Hostname - The IP address or host name of the destination TCP server. + * Port - The TCP port of the destination TCP server. + * + * + * + * + * This processor has the following optional properties: + * + * Connection Per FlowFile - Specifies that each FlowFiles content will be transmitted on a separate TCP connection. + * Idle Connection Expiration - The time threshold after which a TCP sender is deemed eligible for pruning - the associated TCP connection will be closed after this timeout. + * Max Size of Socket Send Buffer - The maximum size of the socket send buffer that should be used. This is a suggestion to the Operating System to indicate how big the socket buffer should + * be. If this value is set too low, the buffer may fill up before the data can be read, and incoming data will be dropped. + * Outgoing Message Delimiter - A string to append to the end of each FlowFiles content to indicate the end of the message to the TCP server. + * Timeout - The timeout period for determining an error has occurred whilst connecting or sending data. + * + * + * + * + * The following relationships are required: + *
[GitHub] nifi issue #252: NIFI-615 - Create a processor to extract WAV file character...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/252 @jskora were you able to rebase and open a new PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/543#discussion_r67799164 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java --- @@ -0,0 +1,290 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.Charset; +import java.util.Arrays; +import java.util.List; +import java.util.concurrent.TimeUnit; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.TriggerWhenEmpty; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.SeeAlso; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.DataUnit; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.util.put.AbstractPutEventProcessor; +import org.apache.nifi.processor.util.put.sender.ChannelSender; +import org.apache.nifi.ssl.SSLContextService; +import org.apache.nifi.stream.io.StreamUtils; +import org.apache.nifi.util.StopWatch; + +import javax.net.ssl.SSLContext; + +/** + * + * The PutTCP processor receives a FlowFile and transmits the FlowFile content over a TCP connection to the configured TCP server. By default, the FlowFiles are transmitted over the same TCP + * connection (or pool of TCP connections if multiple input threads are configured). To assist the TCP server with determining message boundaries, an optional "Outgoing Message Delimiter" string can + * be configured which is appended to the end of each FlowFiles content when it is transmitted over the TCP connection. An optional "Connection Per FlowFile" parameter can be specified to change the + * behaviour so that each FlowFiles content is transmitted over a single TCP connection which is opened when the FlowFile is received and closed after the FlowFile has been sent. This option should + * only be used for low message volume scenarios, otherwise the platform may run out of TCP sockets. + * + * + * + * This processor has the following required properties: + * + * Hostname - The IP address or host name of the destination TCP server. + * Port - The TCP port of the destination TCP server. + * + * + * + * + * This processor has the following optional properties: + * + * Connection Per FlowFile - Specifies that each FlowFiles content will be transmitted on a separate TCP connection. + * Idle Connection Expiration - The time threshold after which a TCP sender is deemed eligible for pruning - the associated TCP connection will be closed after this timeout. + * Max Size of Socket Send Buffer - The maximum size of the socket send buffer that should be used. This is a suggestion to the Operating System to indicate how big the socket buffer should + * be. If this value is set too low, the buffer may fill up before the data can be read, and incoming data will be dropped. + * Outgoing Message Delimiter - A string to append to the end of each FlowFiles content to indicate the end of the message to the TCP server. + * Timeout - The timeout period for determining an error has occurred whilst connecting or sending data. + * + * + * + * + * The following relationships are required: + *
[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/543#discussion_r67800576 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java --- @@ -0,0 +1,290 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.Charset; +import java.util.Arrays; +import java.util.List; +import java.util.concurrent.TimeUnit; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.TriggerWhenEmpty; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.SeeAlso; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.DataUnit; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.util.put.AbstractPutEventProcessor; +import org.apache.nifi.processor.util.put.sender.ChannelSender; +import org.apache.nifi.ssl.SSLContextService; +import org.apache.nifi.stream.io.StreamUtils; +import org.apache.nifi.util.StopWatch; + +import javax.net.ssl.SSLContext; + +/** + * + * The PutTCP processor receives a FlowFile and transmits the FlowFile content over a TCP connection to the configured TCP server. By default, the FlowFiles are transmitted over the same TCP + * connection (or pool of TCP connections if multiple input threads are configured). To assist the TCP server with determining message boundaries, an optional "Outgoing Message Delimiter" string can + * be configured which is appended to the end of each FlowFiles content when it is transmitted over the TCP connection. An optional "Connection Per FlowFile" parameter can be specified to change the + * behaviour so that each FlowFiles content is transmitted over a single TCP connection which is opened when the FlowFile is received and closed after the FlowFile has been sent. This option should + * only be used for low message volume scenarios, otherwise the platform may run out of TCP sockets. + * + * + * + * This processor has the following required properties: + * + * Hostname - The IP address or host name of the destination TCP server. + * Port - The TCP port of the destination TCP server. + * + * + * + * + * This processor has the following optional properties: + * + * Connection Per FlowFile - Specifies that each FlowFiles content will be transmitted on a separate TCP connection. + * Idle Connection Expiration - The time threshold after which a TCP sender is deemed eligible for pruning - the associated TCP connection will be closed after this timeout. + * Max Size of Socket Send Buffer - The maximum size of the socket send buffer that should be used. This is a suggestion to the Operating System to indicate how big the socket buffer should + * be. If this value is set too low, the buffer may fill up before the data can be read, and incoming data will be dropped. + * Outgoing Message Delimiter - A string to append to the end of each FlowFiles content to indicate the end of the message to the TCP server. + * Timeout - The timeout period for determining an error has occurred whilst connecting or sending data. + * + * + * + * + * The following relationships are required: + *
[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/543#discussion_r67801359 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java --- @@ -0,0 +1,290 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.Charset; +import java.util.Arrays; +import java.util.List; +import java.util.concurrent.TimeUnit; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.TriggerWhenEmpty; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.SeeAlso; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.DataUnit; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.util.put.AbstractPutEventProcessor; +import org.apache.nifi.processor.util.put.sender.ChannelSender; +import org.apache.nifi.ssl.SSLContextService; +import org.apache.nifi.stream.io.StreamUtils; +import org.apache.nifi.util.StopWatch; + +import javax.net.ssl.SSLContext; + +/** + * + * The PutTCP processor receives a FlowFile and transmits the FlowFile content over a TCP connection to the configured TCP server. By default, the FlowFiles are transmitted over the same TCP + * connection (or pool of TCP connections if multiple input threads are configured). To assist the TCP server with determining message boundaries, an optional "Outgoing Message Delimiter" string can + * be configured which is appended to the end of each FlowFiles content when it is transmitted over the TCP connection. An optional "Connection Per FlowFile" parameter can be specified to change the + * behaviour so that each FlowFiles content is transmitted over a single TCP connection which is opened when the FlowFile is received and closed after the FlowFile has been sent. This option should + * only be used for low message volume scenarios, otherwise the platform may run out of TCP sockets. + * + * + * + * This processor has the following required properties: + * + * Hostname - The IP address or host name of the destination TCP server. + * Port - The TCP port of the destination TCP server. + * + * + * + * + * This processor has the following optional properties: + * + * Connection Per FlowFile - Specifies that each FlowFiles content will be transmitted on a separate TCP connection. + * Idle Connection Expiration - The time threshold after which a TCP sender is deemed eligible for pruning - the associated TCP connection will be closed after this timeout. + * Max Size of Socket Send Buffer - The maximum size of the socket send buffer that should be used. This is a suggestion to the Operating System to indicate how big the socket buffer should + * be. If this value is set too low, the buffer may fill up before the data can be read, and incoming data will be dropped. + * Outgoing Message Delimiter - A string to append to the end of each FlowFiles content to indicate the end of the message to the TCP server. + * Timeout - The timeout period for determining an error has occurred whilst connecting or sending data. + * + * + * + * + * The following relationships are required: + *
[GitHub] nifi pull request #543: NIFI-1834 Create PutTCP Processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/543#discussion_r67803466 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutTCP.java --- @@ -0,0 +1,290 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.standard; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.Charset; +import java.util.Arrays; +import java.util.List; +import java.util.concurrent.TimeUnit; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.TriggerWhenEmpty; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.SeeAlso; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.DataUnit; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.util.put.AbstractPutEventProcessor; +import org.apache.nifi.processor.util.put.sender.ChannelSender; +import org.apache.nifi.ssl.SSLContextService; +import org.apache.nifi.stream.io.StreamUtils; +import org.apache.nifi.util.StopWatch; + +import javax.net.ssl.SSLContext; + +/** + * + * The PutTCP processor receives a FlowFile and transmits the FlowFile content over a TCP connection to the configured TCP server. By default, the FlowFiles are transmitted over the same TCP + * connection (or pool of TCP connections if multiple input threads are configured). To assist the TCP server with determining message boundaries, an optional "Outgoing Message Delimiter" string can + * be configured which is appended to the end of each FlowFiles content when it is transmitted over the TCP connection. An optional "Connection Per FlowFile" parameter can be specified to change the + * behaviour so that each FlowFiles content is transmitted over a single TCP connection which is opened when the FlowFile is received and closed after the FlowFile has been sent. This option should + * only be used for low message volume scenarios, otherwise the platform may run out of TCP sockets. + * + * + * + * This processor has the following required properties: + * + * Hostname - The IP address or host name of the destination TCP server. + * Port - The TCP port of the destination TCP server. + * + * + * + * + * This processor has the following optional properties: + * + * Connection Per FlowFile - Specifies that each FlowFiles content will be transmitted on a separate TCP connection. + * Idle Connection Expiration - The time threshold after which a TCP sender is deemed eligible for pruning - the associated TCP connection will be closed after this timeout. + * Max Size of Socket Send Buffer - The maximum size of the socket send buffer that should be used. This is a suggestion to the Operating System to indicate how big the socket buffer should + * be. If this value is set too low, the buffer may fill up before the data can be read, and incoming data will be dropped. + * Outgoing Message Delimiter - A string to append to the end of each FlowFiles content to indicate the end of the message to the TCP server. + * Timeout - The timeout period for determining an error has occurred whilst connecting or sending data. + * + * + * + * + * The following relationships are required: + *
[GitHub] nifi issue #543: NIFI-1834 Create PutTCP Processor
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/543 Thanks @olegz, will merge it in --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #543: NIFI-1834 Create PutTCP Processor
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/543 Also, for the sake of transparency, we had a discussion offline debating the various directions to go to handle the "consistent inconsistency" and had the idea for a way for the user to check a box in the UI for whether to enter literal or interpreted text. I created a ticket for it here[1]. For now, in order to keep consistent with the other processors in this area that do automatically convert, the PutTCP processor will convert a "\n" entered in the UI to a new line character (also tabs and carriage returns). [1] https://issues.apache.org/jira/browse/NIFI-2069 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #555: NIFI-2067 ignored intermittently failing MemoryTest
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/555 In cases where validateWarnWhenPercentThresholdReached fails would validateWarnWhenSizeThresholdReached fail too? Wondering if we should ignore both. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #555: NIFI-2067 ignored intermittently failing MemoryTest
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/555 Nope I am missing it, sorry about that. Once the travis build finishes successfully I am +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #548: NIFI-2045 - Removing mqtt message from queue after a...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/548#discussion_r67942458 --- Diff: nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt/ConsumeMQTT.java --- @@ -265,7 +266,13 @@ public void onTrigger(final ProcessContext context, final ProcessSession session transferQueue(session); } -private void transferQueue(ProcessSession session){ +// Protected visibility for unit testing +protected BlockingQueue getMqttQueue() { +return mqttQueue; +} + +// Protected visibility for unit testing +protected void transferQueue(ProcessSession session){ --- End diff -- Instead of making these protected I would prefer to use reflection to reference them. You can see Oleg and I's discussion on the issue here: https://github.com/apache/nifi/pull/392#discussion_r63536461 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #548: NIFI-2045 - Removing mqtt message from queue after a...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/548#discussion_r67948120 --- Diff: nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt/ConsumeMQTT.java --- @@ -289,8 +296,10 @@ public void process(final OutputStream out) throws IOException { String transitUri = new StringBuilder(broker).append(mqttMessage.getTopic()).toString(); session.getProvenanceReporter().receive(messageFlowfile, transitUri); session.transfer(messageFlowfile, REL_MESSAGE); -mqttQueue.remove(mqttMessage); session.commit(); +if (!mqttQueue.remove(mqttMessage)) { +logger.warn("Mqtt message " + mqttMessage + " had already been removed from queue, possible duplication of flow files"); --- End diff -- Thinking through this error message and how it would be useful to a user, I would like some kind of UUID to output here (like a message id) but there aren't any in MQTT. So I think it would be better to word the warning a different way by referencing the FlowFile that could potentially contain duplicate data. By having a warning message with the UUID of the FlowFile containing potentially duplicate data the user may be able to correlate that using provenance to find the other FlowFile with duplicate data. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #549: NIFI-2046 - Adding reference to @TriggerSerially in Process...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/549 +1 LGTM, will merge into 0.x and master, thanks @brosander --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #548: NIFI-2045 - Removing mqtt message from queue after a...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/548#discussion_r67957067 --- Diff: nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt/ConsumeMQTT.java --- @@ -288,9 +289,13 @@ public void process(final OutputStream out) throws IOException { String transitUri = new StringBuilder(broker).append(mqttMessage.getTopic()).toString(); session.getProvenanceReporter().receive(messageFlowfile, transitUri); +String uuid = messageFlowfile.getAttribute(CoreAttributes.UUID.key()); --- End diff -- A bit of a nit pick, this is only referenced in the logger statement below so it should be created in there, not outside of it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #548: NIFI-2045 - Removing mqtt message from queue after a...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/548#discussion_r67957184 --- Diff: nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt/ConsumeMQTT.java --- @@ -288,9 +289,13 @@ public void process(final OutputStream out) throws IOException { String transitUri = new StringBuilder(broker).append(mqttMessage.getTopic()).toString(); session.getProvenanceReporter().receive(messageFlowfile, transitUri); +String uuid = messageFlowfile.getAttribute(CoreAttributes.UUID.key()); session.transfer(messageFlowfile, REL_MESSAGE); -mqttQueue.remove(mqttMessage); session.commit(); +if (!mqttQueue.remove(mqttMessage)) { --- End diff -- In order to avoid concatenating the Strings in the logger message needlessly, this should check if warn is enabled too: `... && getLogger().isWarnEnabled()` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/556#discussion_r67962838 --- Diff: nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java --- @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.media; + +import java.io.IOException; +import java.io.InputStream; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.atomic.AtomicReference; +import java.util.regex.Pattern; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.Validator; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ProcessorLog; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessorInitializationContext; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.util.ObjectHolder; + +import org.apache.tika.exception.TikaException; +import org.apache.tika.io.TikaInputStream; +import org.apache.tika.metadata.Metadata; +import org.apache.tika.parser.AutoDetectParser; +import org.apache.tika.sax.BodyContentHandler; +import org.xml.sax.SAXException; + +@InputRequirement(Requirement.INPUT_REQUIRED) +@Tags({"media", "file", "format", "metadata", "audio", "video", "image", "document", "pdf"}) +@CapabilityDescription("Extract the content metadata from flowfiles containing audio, video, image, and other file " ++ "types. This processor relies on the Apache Tika project for file format detection and parsing. It " ++ "extracts a long list of metadata types for media files including audio, video, and print media " ++ "formats." ++ "For the more details and the list of supported file types, visit the library's website " ++ "at http://tika.apache.org/.";) +@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata " ++ "will be inserted with the attribute name \".\", or \"\" if " ++ "\"Metadata Key Prefix\" is not provided.")}) +@SupportsBatching +public class ExtractMediaMetadata extends AbstractProcessor { + +static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new PropertyDescriptor.Builder() +.name("Max Number of Attributes") +.description("Specify the max number of attributes to add to the flowfile. There is no guarantee in what order" ++ " the tags will be processed. By default it will process all of them.") +
[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/556#discussion_r67963044 --- Diff: nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java --- @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.media; + +import java.io.IOException; +import java.io.InputStream; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.atomic.AtomicReference; +import java.util.regex.Pattern; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.Validator; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ProcessorLog; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessorInitializationContext; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.util.ObjectHolder; + +import org.apache.tika.exception.TikaException; +import org.apache.tika.io.TikaInputStream; +import org.apache.tika.metadata.Metadata; +import org.apache.tika.parser.AutoDetectParser; +import org.apache.tika.sax.BodyContentHandler; +import org.xml.sax.SAXException; + +@InputRequirement(Requirement.INPUT_REQUIRED) +@Tags({"media", "file", "format", "metadata", "audio", "video", "image", "document", "pdf"}) +@CapabilityDescription("Extract the content metadata from flowfiles containing audio, video, image, and other file " ++ "types. This processor relies on the Apache Tika project for file format detection and parsing. It " ++ "extracts a long list of metadata types for media files including audio, video, and print media " ++ "formats." ++ "For the more details and the list of supported file types, visit the library's website " ++ "at http://tika.apache.org/.";) +@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata " ++ "will be inserted with the attribute name \".\", or \"\" if " ++ "\"Metadata Key Prefix\" is not provided.")}) +@SupportsBatching +public class ExtractMediaMetadata extends AbstractProcessor { + +static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new PropertyDescriptor.Builder() +.name("Max Number of Attributes") +.description("Specify the max number of attributes to add to the flowfile. There is no guarantee in what order" ++ " the tags will be processed. By default it will process all of them.") +
[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/556#discussion_r67963320 --- Diff: nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java --- @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.media; + +import java.io.IOException; +import java.io.InputStream; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.atomic.AtomicReference; +import java.util.regex.Pattern; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.Validator; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ProcessorLog; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessorInitializationContext; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.util.ObjectHolder; + +import org.apache.tika.exception.TikaException; +import org.apache.tika.io.TikaInputStream; +import org.apache.tika.metadata.Metadata; +import org.apache.tika.parser.AutoDetectParser; +import org.apache.tika.sax.BodyContentHandler; +import org.xml.sax.SAXException; + +@InputRequirement(Requirement.INPUT_REQUIRED) +@Tags({"media", "file", "format", "metadata", "audio", "video", "image", "document", "pdf"}) +@CapabilityDescription("Extract the content metadata from flowfiles containing audio, video, image, and other file " ++ "types. This processor relies on the Apache Tika project for file format detection and parsing. It " ++ "extracts a long list of metadata types for media files including audio, video, and print media " ++ "formats." ++ "For the more details and the list of supported file types, visit the library's website " ++ "at http://tika.apache.org/.";) +@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata " ++ "will be inserted with the attribute name \".\", or \"\" if " ++ "\"Metadata Key Prefix\" is not provided.")}) +@SupportsBatching +public class ExtractMediaMetadata extends AbstractProcessor { + +static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new PropertyDescriptor.Builder() +.name("Max Number of Attributes") +.description("Specify the max number of attributes to add to the flowfile. There is no guarantee in what order" ++ " the tags will be processed. By default it will process all of them.") +
[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/556#discussion_r67963557 --- Diff: nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java --- @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.media; + +import java.io.IOException; +import java.io.InputStream; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.atomic.AtomicReference; +import java.util.regex.Pattern; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.Validator; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ProcessorLog; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessorInitializationContext; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.util.ObjectHolder; + +import org.apache.tika.exception.TikaException; +import org.apache.tika.io.TikaInputStream; +import org.apache.tika.metadata.Metadata; +import org.apache.tika.parser.AutoDetectParser; +import org.apache.tika.sax.BodyContentHandler; +import org.xml.sax.SAXException; + +@InputRequirement(Requirement.INPUT_REQUIRED) +@Tags({"media", "file", "format", "metadata", "audio", "video", "image", "document", "pdf"}) +@CapabilityDescription("Extract the content metadata from flowfiles containing audio, video, image, and other file " ++ "types. This processor relies on the Apache Tika project for file format detection and parsing. It " ++ "extracts a long list of metadata types for media files including audio, video, and print media " ++ "formats." ++ "For the more details and the list of supported file types, visit the library's website " ++ "at http://tika.apache.org/.";) +@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata " ++ "will be inserted with the attribute name \".\", or \"\" if " ++ "\"Metadata Key Prefix\" is not provided.")}) +@SupportsBatching +public class ExtractMediaMetadata extends AbstractProcessor { + +static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new PropertyDescriptor.Builder() +.name("Max Number of Attributes") +.description("Specify the max number of attributes to add to the flowfile. There is no guarantee in what order" ++ " the tags will be processed. By default it will process all of them.") +
[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/556#discussion_r67964066 --- Diff: nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java --- @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.media; + +import java.io.IOException; +import java.io.InputStream; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.atomic.AtomicReference; +import java.util.regex.Pattern; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.Validator; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ProcessorLog; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessorInitializationContext; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.util.ObjectHolder; + +import org.apache.tika.exception.TikaException; +import org.apache.tika.io.TikaInputStream; +import org.apache.tika.metadata.Metadata; +import org.apache.tika.parser.AutoDetectParser; +import org.apache.tika.sax.BodyContentHandler; +import org.xml.sax.SAXException; + +@InputRequirement(Requirement.INPUT_REQUIRED) +@Tags({"media", "file", "format", "metadata", "audio", "video", "image", "document", "pdf"}) +@CapabilityDescription("Extract the content metadata from flowfiles containing audio, video, image, and other file " ++ "types. This processor relies on the Apache Tika project for file format detection and parsing. It " ++ "extracts a long list of metadata types for media files including audio, video, and print media " ++ "formats." ++ "For the more details and the list of supported file types, visit the library's website " ++ "at http://tika.apache.org/.";) +@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata " --- End diff -- Here it has "." making it seem like there is automatically a "." added but in the property descriptor it says the "." or "-" is not automatically added. I agree that we shouldn't lock the user into using ".", "-", etc. so this should be changed to reflect that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/556#discussion_r67966203 --- Diff: nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java --- @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.media; + +import java.io.IOException; +import java.io.InputStream; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.atomic.AtomicReference; +import java.util.regex.Pattern; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.Validator; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ProcessorLog; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessorInitializationContext; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.util.ObjectHolder; + +import org.apache.tika.exception.TikaException; +import org.apache.tika.io.TikaInputStream; +import org.apache.tika.metadata.Metadata; +import org.apache.tika.parser.AutoDetectParser; +import org.apache.tika.sax.BodyContentHandler; +import org.xml.sax.SAXException; + +@InputRequirement(Requirement.INPUT_REQUIRED) +@Tags({"media", "file", "format", "metadata", "audio", "video", "image", "document", "pdf"}) +@CapabilityDescription("Extract the content metadata from flowfiles containing audio, video, image, and other file " ++ "types. This processor relies on the Apache Tika project for file format detection and parsing. It " ++ "extracts a long list of metadata types for media files including audio, video, and print media " ++ "formats." ++ "For the more details and the list of supported file types, visit the library's website " ++ "at http://tika.apache.org/.";) +@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata " ++ "will be inserted with the attribute name \".\", or \"\" if " ++ "\"Metadata Key Prefix\" is not provided.")}) +@SupportsBatching +public class ExtractMediaMetadata extends AbstractProcessor { + +static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new PropertyDescriptor.Builder() +.name("Max Number of Attributes") +.description("Specify the max number of attributes to add to the flowfile. There is no guarantee in what order" ++ " the tags will be processed. By default it will process all of them.") +
[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/556#discussion_r67967967 --- Diff: nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java --- @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.media; + +import java.io.IOException; +import java.io.InputStream; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.atomic.AtomicReference; +import java.util.regex.Pattern; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.Validator; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ProcessorLog; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessorInitializationContext; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.util.ObjectHolder; + +import org.apache.tika.exception.TikaException; +import org.apache.tika.io.TikaInputStream; +import org.apache.tika.metadata.Metadata; +import org.apache.tika.parser.AutoDetectParser; +import org.apache.tika.sax.BodyContentHandler; +import org.xml.sax.SAXException; + +@InputRequirement(Requirement.INPUT_REQUIRED) +@Tags({"media", "file", "format", "metadata", "audio", "video", "image", "document", "pdf"}) +@CapabilityDescription("Extract the content metadata from flowfiles containing audio, video, image, and other file " ++ "types. This processor relies on the Apache Tika project for file format detection and parsing. It " ++ "extracts a long list of metadata types for media files including audio, video, and print media " ++ "formats." ++ "For the more details and the list of supported file types, visit the library's website " ++ "at http://tika.apache.org/.";) +@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata " ++ "will be inserted with the attribute name \".\", or \"\" if " ++ "\"Metadata Key Prefix\" is not provided.")}) +@SupportsBatching +public class ExtractMediaMetadata extends AbstractProcessor { + +static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new PropertyDescriptor.Builder() +.name("Max Number of Attributes") +.description("Specify the max number of attributes to add to the flowfile. There is no guarantee in what order" ++ " the tags will be processed. By default it will process all of them.") +
[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/556#discussion_r67968149 --- Diff: nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java --- @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.media; + +import java.io.IOException; +import java.io.InputStream; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.atomic.AtomicReference; +import java.util.regex.Pattern; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.Validator; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ProcessorLog; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessorInitializationContext; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.util.ObjectHolder; + +import org.apache.tika.exception.TikaException; +import org.apache.tika.io.TikaInputStream; +import org.apache.tika.metadata.Metadata; +import org.apache.tika.parser.AutoDetectParser; +import org.apache.tika.sax.BodyContentHandler; +import org.xml.sax.SAXException; + +@InputRequirement(Requirement.INPUT_REQUIRED) +@Tags({"media", "file", "format", "metadata", "audio", "video", "image", "document", "pdf"}) +@CapabilityDescription("Extract the content metadata from flowfiles containing audio, video, image, and other file " ++ "types. This processor relies on the Apache Tika project for file format detection and parsing. It " ++ "extracts a long list of metadata types for media files including audio, video, and print media " ++ "formats." ++ "For the more details and the list of supported file types, visit the library's website " ++ "at http://tika.apache.org/.";) +@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata " ++ "will be inserted with the attribute name \".\", or \"\" if " ++ "\"Metadata Key Prefix\" is not provided.")}) +@SupportsBatching +public class ExtractMediaMetadata extends AbstractProcessor { + +static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new PropertyDescriptor.Builder() +.name("Max Number of Attributes") +.description("Specify the max number of attributes to add to the flowfile. There is no guarantee in what order" ++ " the tags will be processed. By default it will process all of them.") +
[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/556#discussion_r67969095 --- Diff: nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/test/java/org/apache/nifi/processors/image/ExtractImageMetadataTest.java --- @@ -37,7 +37,7 @@ private static String BMP_HEADER = "BMP Header."; private static String JPEG_HEADER = "JPEG."; private static String GIF_HEADER = "GIF Header."; -private static String PNG_HEADER = "PNG."; +private static String PNG_HEADER = "PNG-"; --- End diff -- Did the PNG header change when changing version of metadata extractor? If so, does that constitute breaking backwards compatibility? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/556#discussion_r67969885 --- Diff: nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/test/java/org/apache/nifi/processors/media/TestExtractMediaMetadata.java --- @@ -0,0 +1,450 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.media; + +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.attributes.CoreAttributes; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.util.MockFlowFile; +import org.apache.nifi.util.TestRunner; +import org.apache.nifi.util.TestRunners; +import org.junit.Test; + +import java.io.File; +import java.io.IOException; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Random; +import java.util.Set; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +public class TestExtractMediaMetadata { + +@Test +public void testProperties() { +final TestRunner runner = TestRunners.newTestRunner(new ExtractMediaMetadata()); +ProcessContext context = runner.getProcessContext(); +Map propertyValues = context.getProperties(); +assertEquals(6, propertyValues.size()); +} + +@Test +public void testRelationships() { +final TestRunner runner = TestRunners.newTestRunner(new ExtractMediaMetadata()); +ProcessContext context = runner.getProcessContext(); +Set relationships = context.getAvailableRelationships(); +assertEquals(2, relationships.size()); +assertTrue(relationships.contains(ExtractMediaMetadata.SUCCESS)); +assertTrue(relationships.contains(ExtractMediaMetadata.FAILURE)); +} + +@Test +public void testTextBytes() throws IOException { +final TestRunner runner = TestRunners.newTestRunner(new ExtractMediaMetadata()); +runner.setProperty(ExtractMediaMetadata.MIME_TYPE_FILTER, "text/.*"); +runner.setProperty(ExtractMediaMetadata.METADATA_KEY_FILTER, ""); +runner.setProperty(ExtractMediaMetadata.METADATA_KEY_PREFIX, "txt."); +runner.assertValid(); + +final Map attrs = new HashMap<>(); +attrs.put("filename", "test1.txt"); +runner.enqueue("test1".getBytes(), attrs); +runner.run(); + +runner.assertAllFlowFilesTransferred(ExtractMediaMetadata.SUCCESS, 1); +runner.assertTransferCount(ExtractMediaMetadata.FAILURE, 0); + +final List successFiles = runner.getFlowFilesForRelationship(ExtractMediaMetadata.SUCCESS); +MockFlowFile flowFile0 = successFiles.get(0); +flowFile0.assertAttributeExists("filename"); +flowFile0.assertAttributeEquals("filename", "test1.txt"); +flowFile0.assertAttributeExists("txt.Content-Type"); + assertTrue(flowFile0.getAttribute("txt.Content-Type").startsWith("text/plain")); +flowFile0.assertAttributeExists("txt.X-Parsed-By"); + assertTrue(flowFile0.getAttribute("txt.X-Parsed-By").contains("org.apache.tika.parser.DefaultParser")); + assertTrue(flowFile0.getAttribute("txt.X-Parsed-By").contains("org.apache.tika.parser.txt.TXTParser")); +flowFile0.assertAttributeExists("txt.Content-Encoding"); +flowFile0.assertAttributeEquals("txt.Content-Encoding", "ISO-8859-1"); +flowFile0.assertContentEquals("test1".getBytes("UTF-8")); +} + +@Test +public void te
[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/556#discussion_r67982895 --- Diff: nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java --- @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.media; + +import java.io.IOException; +import java.io.InputStream; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.atomic.AtomicReference; +import java.util.regex.Pattern; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.Validator; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ProcessorLog; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessorInitializationContext; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.util.ObjectHolder; + +import org.apache.tika.exception.TikaException; +import org.apache.tika.io.TikaInputStream; +import org.apache.tika.metadata.Metadata; +import org.apache.tika.parser.AutoDetectParser; +import org.apache.tika.sax.BodyContentHandler; +import org.xml.sax.SAXException; + +@InputRequirement(Requirement.INPUT_REQUIRED) +@Tags({"media", "file", "format", "metadata", "audio", "video", "image", "document", "pdf"}) +@CapabilityDescription("Extract the content metadata from flowfiles containing audio, video, image, and other file " ++ "types. This processor relies on the Apache Tika project for file format detection and parsing. It " ++ "extracts a long list of metadata types for media files including audio, video, and print media " ++ "formats." ++ "For the more details and the list of supported file types, visit the library's website " ++ "at http://tika.apache.org/.";) +@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata " ++ "will be inserted with the attribute name \".\", or \"\" if " ++ "\"Metadata Key Prefix\" is not provided.")}) +@SupportsBatching +public class ExtractMediaMetadata extends AbstractProcessor { + +static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new PropertyDescriptor.Builder() +.name("Max Number of Attributes") +.description("Specify the max number of attributes to add to the flowfile. There is no guarantee in what order" ++ " the tags will be processed. By default it will process all of them.") +
[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/556#discussion_r67982943 --- Diff: nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java --- @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.media; + +import java.io.IOException; +import java.io.InputStream; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.atomic.AtomicReference; +import java.util.regex.Pattern; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.Validator; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ProcessorLog; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessorInitializationContext; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.util.ObjectHolder; + +import org.apache.tika.exception.TikaException; +import org.apache.tika.io.TikaInputStream; +import org.apache.tika.metadata.Metadata; +import org.apache.tika.parser.AutoDetectParser; +import org.apache.tika.sax.BodyContentHandler; +import org.xml.sax.SAXException; + +@InputRequirement(Requirement.INPUT_REQUIRED) +@Tags({"media", "file", "format", "metadata", "audio", "video", "image", "document", "pdf"}) +@CapabilityDescription("Extract the content metadata from flowfiles containing audio, video, image, and other file " ++ "types. This processor relies on the Apache Tika project for file format detection and parsing. It " ++ "extracts a long list of metadata types for media files including audio, video, and print media " ++ "formats." ++ "For the more details and the list of supported file types, visit the library's website " ++ "at http://tika.apache.org/.";) +@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata " ++ "will be inserted with the attribute name \".\", or \"\" if " ++ "\"Metadata Key Prefix\" is not provided.")}) +@SupportsBatching +public class ExtractMediaMetadata extends AbstractProcessor { + +static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new PropertyDescriptor.Builder() +.name("Max Number of Attributes") +.description("Specify the max number of attributes to add to the flowfile. There is no guarantee in what order" ++ " the tags will be processed. By default it will process all of them.") +
[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/556#discussion_r68052320 --- Diff: nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java --- @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.media; + +import java.io.IOException; +import java.io.InputStream; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.atomic.AtomicReference; +import java.util.regex.Pattern; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.Validator; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ProcessorLog; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessorInitializationContext; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.util.ObjectHolder; + +import org.apache.tika.exception.TikaException; +import org.apache.tika.io.TikaInputStream; +import org.apache.tika.metadata.Metadata; +import org.apache.tika.parser.AutoDetectParser; +import org.apache.tika.sax.BodyContentHandler; +import org.xml.sax.SAXException; + +@InputRequirement(Requirement.INPUT_REQUIRED) +@Tags({"media", "file", "format", "metadata", "audio", "video", "image", "document", "pdf"}) +@CapabilityDescription("Extract the content metadata from flowfiles containing audio, video, image, and other file " ++ "types. This processor relies on the Apache Tika project for file format detection and parsing. It " ++ "extracts a long list of metadata types for media files including audio, video, and print media " ++ "formats." ++ "For the more details and the list of supported file types, visit the library's website " ++ "at http://tika.apache.org/.";) +@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata " ++ "will be inserted with the attribute name \".\", or \"\" if " ++ "\"Metadata Key Prefix\" is not provided.")}) +@SupportsBatching +public class ExtractMediaMetadata extends AbstractProcessor { + +static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new PropertyDescriptor.Builder() +.name("Max Number of Attributes") +.description("Specify the max number of attributes to add to the flowfile. There is no guarantee in what order" ++ " the tags will be processed. By default it will process all of them.") +
[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/556#discussion_r68052582 --- Diff: nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java --- @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.media; + +import java.io.IOException; +import java.io.InputStream; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.atomic.AtomicReference; +import java.util.regex.Pattern; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.Validator; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ProcessorLog; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessorInitializationContext; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.util.ObjectHolder; + +import org.apache.tika.exception.TikaException; +import org.apache.tika.io.TikaInputStream; +import org.apache.tika.metadata.Metadata; +import org.apache.tika.parser.AutoDetectParser; +import org.apache.tika.sax.BodyContentHandler; +import org.xml.sax.SAXException; + +@InputRequirement(Requirement.INPUT_REQUIRED) +@Tags({"media", "file", "format", "metadata", "audio", "video", "image", "document", "pdf"}) +@CapabilityDescription("Extract the content metadata from flowfiles containing audio, video, image, and other file " ++ "types. This processor relies on the Apache Tika project for file format detection and parsing. It " ++ "extracts a long list of metadata types for media files including audio, video, and print media " ++ "formats." ++ "For the more details and the list of supported file types, visit the library's website " ++ "at http://tika.apache.org/.";) +@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata " ++ "will be inserted with the attribute name \".\", or \"\" if " ++ "\"Metadata Key Prefix\" is not provided.")}) +@SupportsBatching +public class ExtractMediaMetadata extends AbstractProcessor { + +static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new PropertyDescriptor.Builder() +.name("Max Number of Attributes") +.description("Specify the max number of attributes to add to the flowfile. There is no guarantee in what order" ++ " the tags will be processed. By default it will process all of them.") +
[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/556#discussion_r68056970 --- Diff: nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/test/java/org/apache/nifi/processors/media/TestExtractMediaMetadata.java --- @@ -0,0 +1,450 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.media; + +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.attributes.CoreAttributes; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.util.MockFlowFile; +import org.apache.nifi.util.TestRunner; +import org.apache.nifi.util.TestRunners; +import org.junit.Test; + +import java.io.File; +import java.io.IOException; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Random; +import java.util.Set; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +public class TestExtractMediaMetadata { + +@Test +public void testProperties() { +final TestRunner runner = TestRunners.newTestRunner(new ExtractMediaMetadata()); +ProcessContext context = runner.getProcessContext(); +Map propertyValues = context.getProperties(); +assertEquals(6, propertyValues.size()); +} + +@Test +public void testRelationships() { +final TestRunner runner = TestRunners.newTestRunner(new ExtractMediaMetadata()); +ProcessContext context = runner.getProcessContext(); +Set relationships = context.getAvailableRelationships(); +assertEquals(2, relationships.size()); +assertTrue(relationships.contains(ExtractMediaMetadata.SUCCESS)); +assertTrue(relationships.contains(ExtractMediaMetadata.FAILURE)); +} + +@Test +public void testTextBytes() throws IOException { +final TestRunner runner = TestRunners.newTestRunner(new ExtractMediaMetadata()); +runner.setProperty(ExtractMediaMetadata.MIME_TYPE_FILTER, "text/.*"); +runner.setProperty(ExtractMediaMetadata.METADATA_KEY_FILTER, ""); +runner.setProperty(ExtractMediaMetadata.METADATA_KEY_PREFIX, "txt."); +runner.assertValid(); + +final Map attrs = new HashMap<>(); +attrs.put("filename", "test1.txt"); +runner.enqueue("test1".getBytes(), attrs); +runner.run(); + +runner.assertAllFlowFilesTransferred(ExtractMediaMetadata.SUCCESS, 1); +runner.assertTransferCount(ExtractMediaMetadata.FAILURE, 0); + +final List successFiles = runner.getFlowFilesForRelationship(ExtractMediaMetadata.SUCCESS); +MockFlowFile flowFile0 = successFiles.get(0); +flowFile0.assertAttributeExists("filename"); +flowFile0.assertAttributeEquals("filename", "test1.txt"); +flowFile0.assertAttributeExists("txt.Content-Type"); + assertTrue(flowFile0.getAttribute("txt.Content-Type").startsWith("text/plain")); +flowFile0.assertAttributeExists("txt.X-Parsed-By"); + assertTrue(flowFile0.getAttribute("txt.X-Parsed-By").contains("org.apache.tika.parser.DefaultParser")); + assertTrue(flowFile0.getAttribute("txt.X-Parsed-By").contains("org.apache.tika.parser.txt.TXTParser")); +flowFile0.assertAttributeExists("txt.Content-Encoding"); +flowFile0.assertAttributeEquals("txt.Content-Encoding", "ISO-8859-1"); +flowFile0.assertContentEquals("test1".getBytes("UTF-8")); +} + +@Test +public void te
[GitHub] nifi issue #556: NIFI-615 - Create a processor to extract WAV file character...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/556 @jskora when do you think you'll get these changes done and another commit posted? We only have a couple more tickets left for 0.7.0 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #556: NIFI-615 - Create a processor to extract WAV file ch...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/556#discussion_r68074121 --- Diff: nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java --- @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.media; + +import java.io.IOException; +import java.io.InputStream; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.atomic.AtomicReference; +import java.util.regex.Pattern; + +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.components.Validator; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ProcessorLog; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessorInitializationContext; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.util.ObjectHolder; + +import org.apache.tika.exception.TikaException; +import org.apache.tika.io.TikaInputStream; +import org.apache.tika.metadata.Metadata; +import org.apache.tika.parser.AutoDetectParser; +import org.apache.tika.sax.BodyContentHandler; +import org.xml.sax.SAXException; + +@InputRequirement(Requirement.INPUT_REQUIRED) +@Tags({"media", "file", "format", "metadata", "audio", "video", "image", "document", "pdf"}) +@CapabilityDescription("Extract the content metadata from flowfiles containing audio, video, image, and other file " ++ "types. This processor relies on the Apache Tika project for file format detection and parsing. It " ++ "extracts a long list of metadata types for media files including audio, video, and print media " ++ "formats." ++ "For the more details and the list of supported file types, visit the library's website " ++ "at http://tika.apache.org/.";) +@WritesAttributes({@WritesAttribute(attribute = ".", description = "The extracted content metadata " ++ "will be inserted with the attribute name \".\", or \"\" if " ++ "\"Metadata Key Prefix\" is not provided.")}) +@SupportsBatching +public class ExtractMediaMetadata extends AbstractProcessor { + +static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new PropertyDescriptor.Builder() +.name("Max Number of Attributes") +.description("Specify the max number of attributes to add to the flowfile. There is no guarantee in what order" ++ " the tags will be processed. By default it will process all of them.") +
[GitHub] nifi issue #556: NIFI-615 - Create a processor to extract WAV file character...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/556 I am trying to parse my downloads folder again and I came across a couple exceptions (will post in other comment) but as I try to figure why it happened I realize that since ExtractMediaMetatadata doesn't emit a provenance event for the failure transfer so don't get any details. It would be nice to include in the catch block: ` session.getProvenanceReporter().route(flowFile, FAILURE, "Failed to extract media metadata due to " + e);` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #556: NIFI-615 - Create a processor to extract WAV file character...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/556 @markap14 it seems odd to limit the usefulness of provenance for the reason to "keep data in the flow". Information is being created on "why this flowfile failed" but we don't track it in provenance, it seems wrong. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #556: NIFI-615 - Create a processor to extract WAV file character...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/556 Future change, I'll add a ticket which is blocked by NIFI-104 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #556: NIFI-615 - Create a processor to extract WAV file character...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/556 The only relevant exception I saw processing the download folder was an xml parse error related to a underlying limitation of Tika, no need to address: 2016-06-22 11:32:19,774 ERROR [Timer-Driven Process Thread-10] o.a.n.p.media.ExtractMediaMetadata ExtractMediaMetadata[id=1a1f93bd-5499-4ab4-8c86-7391f7aa1f19] Failed to extract media metadata from StandardFlowFileRecord[uuid=ea365723-636f-4fb7-b871-f14241d686f7,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1466609517187-31, container=default, section=31], offset=293539, length=888],offset=0,name=prelude.jspf,size=888] due to org.apache.nifi.processor.exception.ProcessException: IOException thrown from ExtractMediaMetadata[id=1a1f93bd-5499-4ab4-8c86-7391f7aa1f19]: java.io.IOException: org.apache.tika.exception.TikaException: XML parse error: org.apache.nifi.processor.exception.ProcessException: IOException thrown from ExtractMediaMetadata[id=1a1f93bd-5499-4ab4-8c86-7391f7aa1f19]: java.io.IOException: org.apache.tika.exception.TikaException: XML parse error 2016-06-22 11:32:19,776 ERROR [Timer-Driven Process Thread-10] o.a.n.p.media.ExtractMediaMetadata org.apache.nifi.processor.exception.ProcessException: IOException thrown from ExtractMediaMetadata[id=1a1f93bd-5499-4ab4-8c86-7391f7aa1f19]: java.io.IOException: org.apache.tika.exception.TikaException: XML parse error at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1873) ~[na:na] at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1822) ~[na:na] at org.apache.nifi.processors.media.ExtractMediaMetadata.onTrigger(ExtractMediaMetadata.java:211) ~[nifi-media-processors-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_74] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_74] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_74] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_74] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_74] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_74] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74] Caused by: java.io.IOException: org.apache.tika.exception.TikaException: XML parse error at org.apache.nifi.processors.media.ExtractMediaMetadata$2.process(ExtractMediaMetadata.java:218) ~[nifi-media-processors-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1851) ~[na:na] ... 14 common frames omitted Caused by: org.apache.tika.exception.TikaException: XML parse error at org.apache.tika.parser.xml.XMLParser.parse(XMLParser.java:78) ~[tika-parsers-1.8.jar:1.8] at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:281) ~[tika-core-1.8.jar:1.8] at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:281) ~[tika-core-1.8.jar:1.8] at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:120) ~[tika-core-1.8.jar:1.8] at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:136) ~[tika-core-1.8.jar:1.8] at org.apache.nifi.processors.media.ExtractMediaMetadata.tika_parse(ExtractMediaMetadata.java:245) ~[nifi-media-processors-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.processors.media.ExtractMediaMetadata.access$000(ExtractMediaMetadata.java:74) ~[nifi-media-processors-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at
[GitHub] nifi issue #556: NIFI-615 - Create a processor to extract WAV file character...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/556 According to the latest comment from @bbende, yes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #556: NIFI-615 - Create a processor to extract WAV file character...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/556 @jskora what about changing it so there is a media and image bundles? Reading this comment[1] it's what we decided and the only way not to break backwards compatibility. [1] https://github.com/apache/nifi/pull/556#discussion_r68044015 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #556: NIFI-615 - Create a processor to extract WAV file character...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/556 Ah sorry @jskora for misinterpreting it, changes look good. I am gonna do another round of testing and then should be good to merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #556: NIFI-615 - Create a processor to extract WAV file character...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/556 +1 I reviewed the code, built locally, analyzed ~25gb of randoms files with it and it performed well. Looks good to me @jskora. Can you squash your commits and merge it into 0.x and master? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #548: NIFI-2045 - Removing mqtt message from queue after after se...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/548 +1 Comments addressed, builds and addresses issue. Thanks @brosander will merge it into 0.x and master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #568: NIFI-2092 Fixing unit tests which try to clean up di...
GitHub user JPercivall opened a pull request: https://github.com/apache/nifi/pull/568 NIFI-2092 Fixing unit tests which try to clean up directories before … …stopping the FlowController that has locks opened. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JPercivall/nifi NIFI-2092 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/568.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #568 commit 8ed0ddadcd3d494997f231aafb406fa62d23824e Author: jpercivall Date: 2016-06-23T16:42:16Z NIFI-2092 Fixing unit tests which try to clean up directories before stopping the FlowController that has locks opened. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #569: NIFI-2097 Changing RemoteProcessGroupStatusTest.java...
GitHub user JPercivall opened a pull request: https://github.com/apache/nifi/pull/569 NIFI-2097 Changing RemoteProcessGroupStatusTest.java to use a random open port ... ... instead of a statically defined one You can merge this pull request into a Git repository by running: $ git pull https://github.com/JPercivall/nifi NIFI-2097 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/569.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #569 commit 81f303a943193a17daa03ae1fa7c5da3c984a057 Author: jpercivall Date: 2016-06-23T16:55:58Z NIFI-2097 Changing RemoteProcessGroupStatusTest to run on a random open port instead of a statically defined one commit b664fc648a9250dc745a7d60d47d6889d8d633cf Author: jpercivall Date: 2016-06-23T17:21:56Z NIFI-2097 Changing RemoteProcessGroupStatusTest.java to use a random open port instead of a statically defined one --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #570: NIFI-2098 Adding ignore annotation to testManyFilesO...
GitHub user JPercivall opened a pull request: https://github.com/apache/nifi/pull/570 NIFI-2098 Adding ignore annotation to testManyFilesOpened unit test You can merge this pull request into a Git repository by running: $ git pull https://github.com/JPercivall/nifi NIFI-2098 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/570.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #570 commit f029ac28e8cd8d84d2390f8dbb0b8c01a2e6a37b Author: jpercivall Date: 2016-06-23T17:45:21Z NIFI-2098 Adding ignore annotation to testManyFilesOpened unit test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #568: NIFI-2092 Fixing unit tests which try to clean up directori...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/568 Got it, thanks @olegz --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #568: NIFI-2092 Fixing unit tests which try to clean up di...
Github user JPercivall closed the pull request at: https://github.com/apache/nifi/pull/568 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #569: NIFI-2097 Changing RemoteProcessGroupStatusTest.java to use...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/569 Thanks @olegz, closing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #569: NIFI-2097 Changing RemoteProcessGroupStatusTest.java...
Github user JPercivall closed the pull request at: https://github.com/apache/nifi/pull/569 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #573: NIFI-2089: Ensure streams are closed before attempting to r...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/573 +1 Fixes errors seen on Windows, code looks good and passes contrib-check. Thanks Mark --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #586: NIFI-2118 Addresssing License/Notice issues in nifi-...
GitHub user JPercivall opened a pull request: https://github.com/apache/nifi/pull/586 NIFI-2118 Addresssing License/Notice issues in nifi-assumbly Jython's license and notice information can be found here: https://hg.python.org/jython/file/tip Jolt's license can be found here (no notice): https://github.com/bazaarvoice/jolt/tree/jolt-0.0.20 You can merge this pull request into a Git repository by running: $ git pull https://github.com/JPercivall/nifi NIFI-2118 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/586.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #586 commit 7e9887996b93654b2671c794bd8558da76d10805 Author: jpercivall Date: 2016-06-27T01:03:46Z NIFI-2118 Addresssing License/Notice issues in nifi-assumbly --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request #19: MINIFI-40 - Defaulting to empty string for key...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi-minifi/pull/19#discussion_r68591275 --- Diff: minifi-bootstrap/pom.xml --- @@ -43,6 +43,10 @@ limitations under the License. org.apache.nifi +nifi-properties --- End diff -- Instead of adding another pom dependency for one static helper method I would prefer to just create a new private one that does it: private static boolean nullOrEmpty(String input) { return input == null || input.isEmpty(); } --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request #20: MINIFI-43 Fixing provenance instantiation and ...
GitHub user JPercivall opened a pull request: https://github.com/apache/nifi-minifi/pull/20 MINIFI-43 Fixing provenance instantiation and making some default con… …structors private to prevent same problem in other classes I made the no parameter constructor private when there are required properties with no default value. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JPercivall/nifi-minifi MINIFI-43 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi/pull/20.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20 commit 1c5c2d49e5e0477a0b6dbb58cbd2c3ecf73b4a52 Author: Joseph Percivall Date: 2016-06-27T16:52:58Z MINIFI-43 Fixing provenance instantiation and making some default constructors private to prevent same problem in other classes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #576: NIFI-2068: Add Elasticsearch HTTP processors
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/576 There have already been comments asking for a multi-get function for ElasticSearch[1] could this be added to FetchElasticSearchHTTP now and be added as a new feature to PutElasticSearch later (Create a Jira for it)? [1] https://community.hortonworks.com/questions/41951/how-to-get-all-values-with-expression-language-in.html --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/525#discussion_r68660039 --- Diff: nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLog.java --- @@ -0,0 +1,282 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.windows.event.log; + +import com.sun.jna.platform.win32.Kernel32; +import com.sun.jna.platform.win32.Kernel32Util; +import com.sun.jna.platform.win32.WinNT; +import org.apache.commons.io.Charsets; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.TriggerSerially; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.flowfile.attributes.CoreAttributes; +import org.apache.nifi.processor.AbstractSessionFactoryProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.processors.windows.event.log.jna.ErrorLookup; +import org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback; +import org.apache.nifi.processors.windows.event.log.jna.WEvtApi; + +import java.net.URI; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.LinkedBlockingQueue; + + +@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN) +@Tags({"ingest", "event", "windows"}) +@TriggerSerially +@CapabilityDescription("Registers a Windows Event Log Subscribe Callback to receive FlowFiles from Events on Windows. These can be filtered via channel and XPath.") +@WritesAttributes({ +@WritesAttribute(attribute = "mime.type", description = "Will set a MIME type value of application/xml.") +}) +public class ConsumeWindowsEventLog extends AbstractSessionFactoryProcessor { +public static final String DEFAULT_CHANNEL = "System"; +public static final String DEFAULT_XPATH = "*"; +public static final int DEFAULT_MAX_BUFFER = 1024 * 1024; +public static final int DEFAULT_MAX_QUEUE_SIZE = 1024; + +public static final PropertyDescriptor CHANNEL = new PropertyDescriptor.Builder() +.name("channel") +.displayName("Channel") +.required(true) +.defaultValue(DEFAULT_CHANNEL) +.description("The Windows Event Log Channel to listen to.") +.addValidator(StandardValidators.NON_EMPTY_VALIDATOR) +.build(); + +public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder() +.name("query") +.displayName("XPath Query") +.required(true) +.defaultValue(DEFAULT_XPATH) +.description("XPath Que
[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/525#discussion_r68666372 --- Diff: nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLog.java --- @@ -0,0 +1,282 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.windows.event.log; + +import com.sun.jna.platform.win32.Kernel32; +import com.sun.jna.platform.win32.Kernel32Util; +import com.sun.jna.platform.win32.WinNT; +import org.apache.commons.io.Charsets; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.TriggerSerially; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.flowfile.attributes.CoreAttributes; +import org.apache.nifi.processor.AbstractSessionFactoryProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.processors.windows.event.log.jna.ErrorLookup; +import org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback; +import org.apache.nifi.processors.windows.event.log.jna.WEvtApi; + +import java.net.URI; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.LinkedBlockingQueue; + + +@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN) +@Tags({"ingest", "event", "windows"}) +@TriggerSerially +@CapabilityDescription("Registers a Windows Event Log Subscribe Callback to receive FlowFiles from Events on Windows. These can be filtered via channel and XPath.") +@WritesAttributes({ +@WritesAttribute(attribute = "mime.type", description = "Will set a MIME type value of application/xml.") +}) +public class ConsumeWindowsEventLog extends AbstractSessionFactoryProcessor { +public static final String DEFAULT_CHANNEL = "System"; +public static final String DEFAULT_XPATH = "*"; +public static final int DEFAULT_MAX_BUFFER = 1024 * 1024; +public static final int DEFAULT_MAX_QUEUE_SIZE = 1024; + +public static final PropertyDescriptor CHANNEL = new PropertyDescriptor.Builder() +.name("channel") +.displayName("Channel") +.required(true) +.defaultValue(DEFAULT_CHANNEL) +.description("The Windows Event Log Channel to listen to.") +.addValidator(StandardValidators.NON_EMPTY_VALIDATOR) +.build(); + +public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder() +.name("query") +.displayName("XPath Query") +.required(true) +.defaultValue(DEFAULT_XPATH) +.description("XPath Que
[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/525#discussion_r68666289 --- Diff: nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLog.java --- @@ -0,0 +1,282 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.windows.event.log; + +import com.sun.jna.platform.win32.Kernel32; +import com.sun.jna.platform.win32.Kernel32Util; +import com.sun.jna.platform.win32.WinNT; +import org.apache.commons.io.Charsets; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.TriggerSerially; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.flowfile.attributes.CoreAttributes; +import org.apache.nifi.processor.AbstractSessionFactoryProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.processors.windows.event.log.jna.ErrorLookup; +import org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback; +import org.apache.nifi.processors.windows.event.log.jna.WEvtApi; + +import java.net.URI; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.LinkedBlockingQueue; + + +@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN) +@Tags({"ingest", "event", "windows"}) +@TriggerSerially +@CapabilityDescription("Registers a Windows Event Log Subscribe Callback to receive FlowFiles from Events on Windows. These can be filtered via channel and XPath.") +@WritesAttributes({ +@WritesAttribute(attribute = "mime.type", description = "Will set a MIME type value of application/xml.") +}) +public class ConsumeWindowsEventLog extends AbstractSessionFactoryProcessor { +public static final String DEFAULT_CHANNEL = "System"; +public static final String DEFAULT_XPATH = "*"; +public static final int DEFAULT_MAX_BUFFER = 1024 * 1024; +public static final int DEFAULT_MAX_QUEUE_SIZE = 1024; + +public static final PropertyDescriptor CHANNEL = new PropertyDescriptor.Builder() +.name("channel") +.displayName("Channel") +.required(true) +.defaultValue(DEFAULT_CHANNEL) +.description("The Windows Event Log Channel to listen to.") +.addValidator(StandardValidators.NON_EMPTY_VALIDATOR) +.build(); + +public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder() +.name("query") +.displayName("XPath Query") +.required(true) +.defaultValue(DEFAULT_XPATH) +.description("XPath Que
[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/525#discussion_r68671269 --- Diff: nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLog.java --- @@ -0,0 +1,282 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.windows.event.log; + +import com.sun.jna.platform.win32.Kernel32; +import com.sun.jna.platform.win32.Kernel32Util; +import com.sun.jna.platform.win32.WinNT; +import org.apache.commons.io.Charsets; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.TriggerSerially; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.flowfile.attributes.CoreAttributes; +import org.apache.nifi.processor.AbstractSessionFactoryProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.processors.windows.event.log.jna.ErrorLookup; +import org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback; +import org.apache.nifi.processors.windows.event.log.jna.WEvtApi; + +import java.net.URI; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.LinkedBlockingQueue; + + +@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN) +@Tags({"ingest", "event", "windows"}) +@TriggerSerially +@CapabilityDescription("Registers a Windows Event Log Subscribe Callback to receive FlowFiles from Events on Windows. These can be filtered via channel and XPath.") +@WritesAttributes({ +@WritesAttribute(attribute = "mime.type", description = "Will set a MIME type value of application/xml.") +}) +public class ConsumeWindowsEventLog extends AbstractSessionFactoryProcessor { +public static final String DEFAULT_CHANNEL = "System"; +public static final String DEFAULT_XPATH = "*"; +public static final int DEFAULT_MAX_BUFFER = 1024 * 1024; +public static final int DEFAULT_MAX_QUEUE_SIZE = 1024; + +public static final PropertyDescriptor CHANNEL = new PropertyDescriptor.Builder() +.name("channel") +.displayName("Channel") +.required(true) +.defaultValue(DEFAULT_CHANNEL) +.description("The Windows Event Log Channel to listen to.") +.addValidator(StandardValidators.NON_EMPTY_VALIDATOR) +.build(); + +public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder() +.name("query") +.displayName("XPath Query") +.required(true) +.defaultValue(DEFAULT_XPATH) +.description("XPath Que
[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/525#discussion_r68777286 --- Diff: nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLog.java --- @@ -0,0 +1,282 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.windows.event.log; + +import com.sun.jna.platform.win32.Kernel32; +import com.sun.jna.platform.win32.Kernel32Util; +import com.sun.jna.platform.win32.WinNT; +import org.apache.commons.io.Charsets; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.TriggerSerially; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.flowfile.attributes.CoreAttributes; +import org.apache.nifi.processor.AbstractSessionFactoryProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.processors.windows.event.log.jna.ErrorLookup; +import org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback; +import org.apache.nifi.processors.windows.event.log.jna.WEvtApi; + +import java.net.URI; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.LinkedBlockingQueue; + + +@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN) +@Tags({"ingest", "event", "windows"}) +@TriggerSerially +@CapabilityDescription("Registers a Windows Event Log Subscribe Callback to receive FlowFiles from Events on Windows. These can be filtered via channel and XPath.") +@WritesAttributes({ +@WritesAttribute(attribute = "mime.type", description = "Will set a MIME type value of application/xml.") +}) +public class ConsumeWindowsEventLog extends AbstractSessionFactoryProcessor { +public static final String DEFAULT_CHANNEL = "System"; +public static final String DEFAULT_XPATH = "*"; +public static final int DEFAULT_MAX_BUFFER = 1024 * 1024; +public static final int DEFAULT_MAX_QUEUE_SIZE = 1024; + +public static final PropertyDescriptor CHANNEL = new PropertyDescriptor.Builder() +.name("channel") +.displayName("Channel") +.required(true) +.defaultValue(DEFAULT_CHANNEL) +.description("The Windows Event Log Channel to listen to.") +.addValidator(StandardValidators.NON_EMPTY_VALIDATOR) +.build(); + +public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder() +.name("query") +.displayName("XPath Query") +.required(true) +.defaultValue(DEFAULT_XPATH) +.description("XPath Que
[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/525#discussion_r68780983 --- Diff: nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLog.java --- @@ -0,0 +1,282 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.windows.event.log; + +import com.sun.jna.platform.win32.Kernel32; +import com.sun.jna.platform.win32.Kernel32Util; +import com.sun.jna.platform.win32.WinNT; +import org.apache.commons.io.Charsets; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.TriggerSerially; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.flowfile.attributes.CoreAttributes; +import org.apache.nifi.processor.AbstractSessionFactoryProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.processors.windows.event.log.jna.ErrorLookup; +import org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback; +import org.apache.nifi.processors.windows.event.log.jna.WEvtApi; + +import java.net.URI; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.LinkedBlockingQueue; + + +@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN) +@Tags({"ingest", "event", "windows"}) +@TriggerSerially +@CapabilityDescription("Registers a Windows Event Log Subscribe Callback to receive FlowFiles from Events on Windows. These can be filtered via channel and XPath.") +@WritesAttributes({ +@WritesAttribute(attribute = "mime.type", description = "Will set a MIME type value of application/xml.") +}) +public class ConsumeWindowsEventLog extends AbstractSessionFactoryProcessor { +public static final String DEFAULT_CHANNEL = "System"; +public static final String DEFAULT_XPATH = "*"; +public static final int DEFAULT_MAX_BUFFER = 1024 * 1024; +public static final int DEFAULT_MAX_QUEUE_SIZE = 1024; + +public static final PropertyDescriptor CHANNEL = new PropertyDescriptor.Builder() +.name("channel") +.displayName("Channel") +.required(true) +.defaultValue(DEFAULT_CHANNEL) +.description("The Windows Event Log Channel to listen to.") +.addValidator(StandardValidators.NON_EMPTY_VALIDATOR) +.build(); + +public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder() +.name("query") +.displayName("XPath Query") +.required(true) +.defaultValue(DEFAULT_XPATH) +.description("XPath Que
[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/525#discussion_r68780468 --- Diff: nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLog.java --- @@ -0,0 +1,282 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.windows.event.log; + +import com.sun.jna.platform.win32.Kernel32; +import com.sun.jna.platform.win32.Kernel32Util; +import com.sun.jna.platform.win32.WinNT; +import org.apache.commons.io.Charsets; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.TriggerSerially; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.flowfile.attributes.CoreAttributes; +import org.apache.nifi.processor.AbstractSessionFactoryProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.processors.windows.event.log.jna.ErrorLookup; +import org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback; +import org.apache.nifi.processors.windows.event.log.jna.WEvtApi; + +import java.net.URI; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.LinkedBlockingQueue; + + +@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN) +@Tags({"ingest", "event", "windows"}) +@TriggerSerially +@CapabilityDescription("Registers a Windows Event Log Subscribe Callback to receive FlowFiles from Events on Windows. These can be filtered via channel and XPath.") +@WritesAttributes({ +@WritesAttribute(attribute = "mime.type", description = "Will set a MIME type value of application/xml.") +}) +public class ConsumeWindowsEventLog extends AbstractSessionFactoryProcessor { +public static final String DEFAULT_CHANNEL = "System"; +public static final String DEFAULT_XPATH = "*"; +public static final int DEFAULT_MAX_BUFFER = 1024 * 1024; +public static final int DEFAULT_MAX_QUEUE_SIZE = 1024; + +public static final PropertyDescriptor CHANNEL = new PropertyDescriptor.Builder() +.name("channel") +.displayName("Channel") +.required(true) +.defaultValue(DEFAULT_CHANNEL) +.description("The Windows Event Log Channel to listen to.") +.addValidator(StandardValidators.NON_EMPTY_VALIDATOR) +.build(); + +public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder() +.name("query") +.displayName("XPath Query") +.required(true) +.defaultValue(DEFAULT_XPATH) +.description("XPath Que
[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/525#discussion_r68785240 --- Diff: nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLog.java --- @@ -0,0 +1,282 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.windows.event.log; + +import com.sun.jna.platform.win32.Kernel32; +import com.sun.jna.platform.win32.Kernel32Util; +import com.sun.jna.platform.win32.WinNT; +import org.apache.commons.io.Charsets; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.TriggerSerially; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.flowfile.attributes.CoreAttributes; +import org.apache.nifi.processor.AbstractSessionFactoryProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.processors.windows.event.log.jna.ErrorLookup; +import org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback; +import org.apache.nifi.processors.windows.event.log.jna.WEvtApi; + +import java.net.URI; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.LinkedBlockingQueue; + + +@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN) +@Tags({"ingest", "event", "windows"}) +@TriggerSerially +@CapabilityDescription("Registers a Windows Event Log Subscribe Callback to receive FlowFiles from Events on Windows. These can be filtered via channel and XPath.") +@WritesAttributes({ +@WritesAttribute(attribute = "mime.type", description = "Will set a MIME type value of application/xml.") +}) +public class ConsumeWindowsEventLog extends AbstractSessionFactoryProcessor { +public static final String DEFAULT_CHANNEL = "System"; +public static final String DEFAULT_XPATH = "*"; +public static final int DEFAULT_MAX_BUFFER = 1024 * 1024; +public static final int DEFAULT_MAX_QUEUE_SIZE = 1024; + +public static final PropertyDescriptor CHANNEL = new PropertyDescriptor.Builder() +.name("channel") +.displayName("Channel") +.required(true) +.defaultValue(DEFAULT_CHANNEL) +.description("The Windows Event Log Channel to listen to.") +.addValidator(StandardValidators.NON_EMPTY_VALIDATOR) +.build(); + +public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder() +.name("query") +.displayName("XPath Query") +.required(true) +.defaultValue(DEFAULT_XPATH) +.description("XPath Que
[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/525#discussion_r68799710 --- Diff: nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLog.java --- @@ -0,0 +1,282 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.windows.event.log; + +import com.sun.jna.platform.win32.Kernel32; +import com.sun.jna.platform.win32.Kernel32Util; +import com.sun.jna.platform.win32.WinNT; +import org.apache.commons.io.Charsets; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.TriggerSerially; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.flowfile.attributes.CoreAttributes; +import org.apache.nifi.processor.AbstractSessionFactoryProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.processors.windows.event.log.jna.ErrorLookup; +import org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback; +import org.apache.nifi.processors.windows.event.log.jna.WEvtApi; + +import java.net.URI; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.LinkedBlockingQueue; + + +@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN) +@Tags({"ingest", "event", "windows"}) +@TriggerSerially +@CapabilityDescription("Registers a Windows Event Log Subscribe Callback to receive FlowFiles from Events on Windows. These can be filtered via channel and XPath.") +@WritesAttributes({ +@WritesAttribute(attribute = "mime.type", description = "Will set a MIME type value of application/xml.") +}) +public class ConsumeWindowsEventLog extends AbstractSessionFactoryProcessor { +public static final String DEFAULT_CHANNEL = "System"; +public static final String DEFAULT_XPATH = "*"; +public static final int DEFAULT_MAX_BUFFER = 1024 * 1024; +public static final int DEFAULT_MAX_QUEUE_SIZE = 1024; + +public static final PropertyDescriptor CHANNEL = new PropertyDescriptor.Builder() +.name("channel") +.displayName("Channel") +.required(true) +.defaultValue(DEFAULT_CHANNEL) +.description("The Windows Event Log Channel to listen to.") +.addValidator(StandardValidators.NON_EMPTY_VALIDATOR) +.build(); + +public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder() +.name("query") +.displayName("XPath Query") +.required(true) +.defaultValue(DEFAULT_XPATH) +.description("XPath Que
[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/525#discussion_r68943688 --- Diff: nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/ConsumeWindowsEventLog.java --- @@ -0,0 +1,282 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.windows.event.log; + +import com.sun.jna.platform.win32.Kernel32; +import com.sun.jna.platform.win32.Kernel32Util; +import com.sun.jna.platform.win32.WinNT; +import org.apache.commons.io.Charsets; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.TriggerSerially; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.annotation.lifecycle.OnStopped; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.flowfile.attributes.CoreAttributes; +import org.apache.nifi.processor.AbstractSessionFactoryProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessSessionFactory; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.processors.windows.event.log.jna.ErrorLookup; +import org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback; +import org.apache.nifi.processors.windows.event.log.jna.WEvtApi; + +import java.net.URI; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.LinkedBlockingQueue; + + +@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN) +@Tags({"ingest", "event", "windows"}) +@TriggerSerially +@CapabilityDescription("Registers a Windows Event Log Subscribe Callback to receive FlowFiles from Events on Windows. These can be filtered via channel and XPath.") +@WritesAttributes({ +@WritesAttribute(attribute = "mime.type", description = "Will set a MIME type value of application/xml.") +}) +public class ConsumeWindowsEventLog extends AbstractSessionFactoryProcessor { +public static final String DEFAULT_CHANNEL = "System"; +public static final String DEFAULT_XPATH = "*"; +public static final int DEFAULT_MAX_BUFFER = 1024 * 1024; +public static final int DEFAULT_MAX_QUEUE_SIZE = 1024; + +public static final PropertyDescriptor CHANNEL = new PropertyDescriptor.Builder() +.name("channel") +.displayName("Channel") +.required(true) +.defaultValue(DEFAULT_CHANNEL) +.description("The Windows Event Log Channel to listen to.") +.addValidator(StandardValidators.NON_EMPTY_VALIDATOR) +.build(); + +public static final PropertyDescriptor QUERY = new PropertyDescriptor.Builder() +.name("query") +.displayName("XPath Query") +.required(true) +.defaultValue(DEFAULT_XPATH) +.description("XPath Que
[GitHub] nifi issue #525: NIFI-1976 - Windows Event Log native processor
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/525 +1 Did a contrib check build, reviewed the code and any comments were addressed. To test I ran three ConsumeWindowsEventLog processors following multiple Windows channels and did a query to limit one of them. I verified the results by looking at the built in Event Viewer program. Thanks @brosander for the contribution! I will rebase and merge it into master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi-minifi/pull/21#discussion_r69157552 --- Diff: minifi-commons/minifi-commons-schema/pom.xml --- @@ -0,0 +1,46 @@ + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> +4.0.0 + +org.apache.nifi.minifi +minifi-commons +0.0.1-SNAPSHOT + + +minifi-commons-schema +jar + + + +org.yaml +snakeyaml +1.17 + + +org.apache.nifi +nifi-framework-core +0.6.0 + + --- End diff -- I don't think this is applicable for the new assembly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi-minifi/pull/21#discussion_r69159093 --- Diff: minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/SchemaLoader.java --- @@ -0,0 +1,52 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.minifi.commons.schema; + +import org.apache.nifi.minifi.commons.schema.exception.SchemaLoaderException; +import org.yaml.snakeyaml.Yaml; +import org.yaml.snakeyaml.error.YAMLException; + +import java.io.IOException; +import java.io.InputStream; +import java.util.Map; + +public class SchemaLoader { +public static Map loadYamlAsMap(InputStream sourceStream) throws IOException, SchemaLoaderException { +try { +Yaml yaml = new Yaml(); + +// Parse the YAML file +final Object loadedObject = yaml.load(sourceStream); + +// Verify the parsed object is a Map structure +if (loadedObject instanceof Map) { +return (Map) loadedObject; +} else { +throw new SchemaLoaderException("Provided YAML configuration is not a Map"); +} +} catch (YAMLException e ) { +throw new IOException(e); +} finally { +sourceStream.close(); +} +} + +public static ConfigSchema loadConfigSchema(InputStream sourceStream) throws IOException, SchemaLoaderException { --- End diff -- Since this is just a wrapper around loadYamlAsMap and has the requirement that the input is Yaml, I'd prefer to have "Yaml" in somewhere in the method name. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi-minifi/pull/21#discussion_r69162642 --- Diff: minifi-toolkit/minifi-toolkit-assembly/NOTICE --- @@ -0,0 +1,21 @@ +Apache NiFi +Copyright 2014-2016 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +=== +Apache Software License v2 +=== + +The following binary components are provided under the Apache Software License v2 + + (ASLv2) Apache NiFi +The following NOTICE information applies: + Apache NiFi + Copyright 2014-2016 The Apache Software Foundation + + (ASLv2) Jetty +The following NOTICE information applies: + Jetty Web Container --- End diff -- Is Jetty packaged in the toolkit-assembly? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi-minifi/pull/21#discussion_r69165759 --- Diff: minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/BaseSchema.java --- @@ -15,13 +15,17 @@ * limitations under the License. */ -package org.apache.nifi.minifi.bootstrap.util.schema.common; +package org.apache.nifi.minifi.commons.schema; import java.lang.reflect.Constructor; import java.lang.reflect.InvocationTargetException; +import java.util.Collections; +import java.util.LinkedHashMap; import java.util.LinkedList; import java.util.List; import java.util.Map; +import java.util.Set; +import java.util.stream.Collectors; public class BaseSchema { --- End diff -- I think I originally should have made this class Abstract. One of the main things is the toMap method. In this class all it does it return a LinkedHashMap and that just feels off to me. I see that others are overriding it and just using this to get a LinkedHashMap. Also I think it is counter intuitive to have the ability to initialize a BaseSchema. Overall I believe all the "Value Access/Interpretation helper methods" should be make static and toMap would be abstract. Thoughts? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi-minifi/pull/21#discussion_r69166517 --- Diff: minifi-toolkit/minifi-toolkit-assembly/pom.xml --- @@ -0,0 +1,68 @@ + + +http://maven.apache.org/POM/4.0.0"; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> +4.0.0 + +minifi-toolkit +org.apache.nifi.minifi +0.0.1-SNAPSHOT + +minifi-toolkit-assembly +pom +This is the assembly of Apache NiFi - MiNiFi --- End diff -- Should be changed to reflect it being the toolkit --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi-minifi/pull/21#discussion_r69178841 --- Diff: minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/BaseSchema.java --- @@ -15,13 +15,17 @@ * limitations under the License. */ -package org.apache.nifi.minifi.bootstrap.util.schema.common; +package org.apache.nifi.minifi.commons.schema; import java.lang.reflect.Constructor; import java.lang.reflect.InvocationTargetException; +import java.util.Collections; +import java.util.LinkedHashMap; import java.util.LinkedList; import java.util.List; import java.util.Map; +import java.util.Set; +import java.util.stream.Collectors; public class BaseSchema { --- End diff -- Right the "Validation Issue helper methods" access the validation issues, which is something every schema will have, so it's good to keep them implemented how they are. The "Value Access/Interpretation helper methods" are all methods that do not necessarily rely on the internal representation of each schema but instead are helper methods for accessing/interpreting values. So they can be static. For toMap I agree that it's good to encourage the use of LinkedHashMap. Maybe a protected method getMap for the schemas to use. Then toMap would be abstract. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi-minifi/pull/21#discussion_r69180850 --- Diff: minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/BaseSchema.java --- @@ -15,13 +15,17 @@ * limitations under the License. */ -package org.apache.nifi.minifi.bootstrap.util.schema.common; +package org.apache.nifi.minifi.commons.schema; import java.lang.reflect.Constructor; import java.lang.reflect.InvocationTargetException; +import java.util.Collections; +import java.util.LinkedHashMap; import java.util.LinkedList; import java.util.List; import java.util.Map; +import java.util.Set; +import java.util.stream.Collectors; public class BaseSchema { --- End diff -- Ah very true, ok I'm on board with not making them static. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi-minifi/pull/21#discussion_r69210176 --- Diff: minifi-toolkit/minifi-toolkit-assembly/src/main/resources/config.sh --- @@ -0,0 +1,133 @@ +#!/bin/sh +# +#Licensed to the Apache Software Foundation (ASF) under one or more +#contributor license agreements. See the NOTICE file distributed with +#this work for additional information regarding copyright ownership. +#The ASF licenses this file to You under the Apache License, Version 2.0 +#(the "License"); you may not use this file except in compliance with +#the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +#Unless required by applicable law or agreed to in writing, software +#distributed under the License is distributed on an "AS IS" BASIS, +#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +#See the License for the specific language governing permissions and +#limitations under the License. +# +# chkconfig: 2345 20 80 +# description: Apache NiFi - MiNiFi +# + +# Script structure inspired from Apache Karaf and other Apache projects with similar startup approaches + +SCRIPT_DIR=$(dirname "$0") +SCRIPT_NAME=$(basename "$0") +MINIFI_TOOLKIT_HOME=$(cd "${SCRIPT_DIR}" && cd .. && pwd) +PROGNAME=$(basename "$0") + + +warn() { +echo "${PROGNAME}: $*" +} + +die() { +warn "$*" +exit 1 +} + +detectOS() { +# OS specific support (must be 'true' or 'false'). +cygwin=false; +aix=false; +os400=false; +darwin=false; +case "$(uname)" in +CYGWIN*) +cygwin=true +;; +AIX*) +aix=true +;; +OS400*) +os400=true +;; +Darwin) +darwin=true +;; +esac +# For AIX, set an environment variable +if ${aix}; then + export LDR_CNTRL=MAXDATA=0xB000@DSA + echo ${LDR_CNTRL} +fi +} + +locateJava() { +# Setup the Java Virtual Machine +if $cygwin ; then +[ -n "${JAVA}" ] && JAVA=$(cygpath --unix "${JAVA}") +[ -n "${JAVA_HOME}" ] && JAVA_HOME=$(cygpath --unix "${JAVA_HOME}") +fi + +if [ "x${JAVA}" = "x" ] && [ -r /etc/gentoo-release ] ; then +JAVA_HOME=$(java-config --jre-home) +fi +if [ "x${JAVA}" = "x" ]; then +if [ "x${JAVA_HOME}" != "x" ]; then +if [ ! -d "${JAVA_HOME}" ]; then +die "JAVA_HOME is not valid: ${JAVA_HOME}" +fi +JAVA="${JAVA_HOME}/bin/java" +else +warn "JAVA_HOME not set; results may vary" +JAVA=$(type java) +JAVA=$(expr "${JAVA}" : '.* \(/.*\)$') +if [ "x${JAVA}" = "x" ]; then +die "java command not found" +fi +fi +fi +# if command is env, attempt to add more to the classpath +if [ "$1" = "env" ]; then +[ "x${TOOLS_JAR}" = "x" ] && [ -n "${JAVA_HOME}" ] && TOOLS_JAR=$(find -H "${JAVA_HOME}" -name "tools.jar") +[ "x${TOOLS_JAR}" = "x" ] && [ -n "${JAVA_HOME}" ] && TOOLS_JAR=$(find -H "${JAVA_HOME}" -name "classes.jar") +if [ "x${TOOLS_JAR}" = "x" ]; then + warn "Could not locate tools.jar or classes.jar. Please set manually to avail all command features." +fi +fi --- End diff -- Is this block left over from the minifi.sh? I tried running it and got this: Joseph-Percivall:bin jpercivall$ ./config.sh env Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home MiNiFi Toolkit home: /Users/jpercivall/projects/edge/nifi-minifi/minifi-toolkit/minifi-toolkit-assembly/target/minifi-0.0.1-SNAPSHOT-bin/minifi-toolkit-0.0.1-SNAPSHOT Usage: java org.apache.nifi.minifi.toolkit.configuration.ConfigMain options Valid commands include: transform: Transform template xml into MiNiFi config YAML validate: Validate config YAML --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi-minifi/pull/21#discussion_r69214705 --- Diff: minifi-toolkit/minifi-toolkit-configuration/pom.xml --- @@ -0,0 +1,42 @@ + + +http://maven.apache.org/POM/4.0.0"; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> +4.0.0 + +minifi-toolkit +org.apache.nifi.minifi +0.0.1-SNAPSHOT + +minifi-toolkit-configuration +jar + + + +org.apache.nifi.minifi +minifi-commons-schema +${project.version} + + +org.apache.nifi +nifi-framework-core +0.6.0 --- End diff -- This should be added to the Dependency Management portion of the root pom so that it shares the nifi version set there. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request #21: MINIFI-41 - CLI utility for template.xml -> YA...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi-minifi/pull/21#discussion_r69218372 --- Diff: minifi-commons/minifi-commons-schema/pom.xml --- @@ -0,0 +1,46 @@ + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> +4.0.0 + +org.apache.nifi.minifi +minifi-commons +0.0.1-SNAPSHOT + + +minifi-commons-schema +jar + + + +org.yaml +snakeyaml +1.17 + + +org.apache.nifi +nifi-framework-core +0.6.0 + + --- End diff -- (Accidentally commented on the wrong pom originally) This should probably be removed from this pom too since it will be used in both assemblies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/576#discussion_r69306232 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/pom.xml --- @@ -56,6 +56,11 @@ language governing permissions and limitations under the License. --> ${es.version} +com.squareup.okhttp +okhttp +2.7.1 --- End diff -- What's the motiviation for using this version of OkHttp instead of the latest[1][2]? [1] https://github.com/square/okhttp/tree/parent-3.3.1 [2] https://mvnrepository.com/artifact/com.squareup.okhttp3/okhttp --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/576#discussion_r69307262 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractElasticsearchProcessor.java --- @@ -17,129 +17,25 @@ package org.apache.nifi.processors.elasticsearch; import org.apache.nifi.components.PropertyDescriptor; -import org.apache.nifi.components.ValidationContext; -import org.apache.nifi.components.ValidationResult; -import org.apache.nifi.components.Validator; -import org.apache.nifi.logging.ComponentLog; import org.apache.nifi.processor.AbstractProcessor; import org.apache.nifi.processor.ProcessContext; import org.apache.nifi.processor.exception.ProcessException; import org.apache.nifi.processor.util.StandardValidators; import org.apache.nifi.ssl.SSLContextService; -import org.apache.nifi.util.StringUtils; -import org.elasticsearch.client.Client; -import org.elasticsearch.client.transport.TransportClient; -import org.elasticsearch.common.settings.Settings; -import org.elasticsearch.common.transport.InetSocketTransportAddress; - -import java.io.File; -import java.lang.reflect.Constructor; -import java.lang.reflect.InvocationTargetException; -import java.lang.reflect.Method; -import java.net.InetSocketAddress; -import java.net.MalformedURLException; -import java.net.URL; -import java.net.URLClassLoader; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.Collection; -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Set; -import java.util.concurrent.atomic.AtomicReference; - +/** + * A base class for all Elasticsearch processors + */ public abstract class AbstractElasticsearchProcessor extends AbstractProcessor { -/** - * This validator ensures the Elasticsearch hosts property is a valid list of hostname:port entries - */ -private static final Validator HOSTNAME_PORT_VALIDATOR = new Validator() { -@Override -public ValidationResult validate(final String subject, final String input, final ValidationContext context) { -final List esList = Arrays.asList(input.split(",")); -for (String hostnamePort : esList) { -String[] addresses = hostnamePort.split(":"); -// Protect against invalid input like http://127.0.0.1:9300 (URL scheme should not be there) -if (addresses.length != 2) { -return new ValidationResult.Builder().subject(subject).input(input).explanation( -"Must be in hostname:port form (no scheme such as http://";).valid(false).build(); -} -} -return new ValidationResult.Builder().subject(subject).input(input).explanation( -"Valid cluster definition").valid(true).build(); -} -}; - -protected static final PropertyDescriptor CLUSTER_NAME = new PropertyDescriptor.Builder() -.name("Cluster Name") -.description("Name of the ES cluster (for example, elasticsearch_brew). Defaults to 'elasticsearch'") -.required(true) -.addValidator(StandardValidators.NON_EMPTY_VALIDATOR) -.defaultValue("elasticsearch") -.build(); - -protected static final PropertyDescriptor HOSTS = new PropertyDescriptor.Builder() -.name("ElasticSearch Hosts") -.description("ElasticSearch Hosts, which should be comma separated and colon for hostname/port " -+ "host1:port,host2:port, For example testcluster:9300.") -.required(true) -.expressionLanguageSupported(false) -.addValidator(HOSTNAME_PORT_VALIDATOR) -.build(); - public static final PropertyDescriptor PROP_SSL_CONTEXT_SERVICE = new PropertyDescriptor.Builder() .name("SSL Context Service") .description("The SSL Context Service used to provide client certificate information for TLS/SSL " -+ "connections. This service only applies if the Shield plugin is available.") ++ "connections. This service only applies if the Elasticsearch endpoints have been protected by SSLShield plugin is available.") --- End diff -- This sentence doesn't really make sense to me, specifically the "plugin is available" part -
[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/576#discussion_r69308812 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java --- @@ -0,0 +1,293 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.elasticsearch; + +import com.squareup.okhttp.HttpUrl; +import com.squareup.okhttp.OkHttpClient; +import com.squareup.okhttp.Response; +import com.squareup.okhttp.ResponseBody; +import org.apache.commons.lang3.StringUtils; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.stream.io.ByteArrayInputStream; +import org.codehaus.jackson.JsonNode; + +import java.io.IOException; +import java.net.MalformedURLException; +import java.net.URL; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.TimeUnit; +import java.util.stream.Collectors; +import java.util.stream.Stream; + + +@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED) +@EventDriven +@SupportsBatching +@Tags({"elasticsearch", "fetch", "read", "get", "http"}) +@CapabilityDescription("Retrieves a document from Elasticsearch using the specified connection properties and the " ++ "identifier of the document to retrieve.") +@WritesAttributes({ +@WritesAttribute(attribute = "filename", description = "The filename attributes is set to the document identifier"), +@WritesAttribute(attribute = "es.index", description = "The Elasticsearch index containing the document"), +@WritesAttribute(attribute = "es.type", description = "The Elasticsearch document type") +}) +public class FetchElasticsearchHttp extends AbstractElasticsearchHttpProcessor { + +private static final String FIELD_INCLUDE_QUERY_PARAM = "_source_include"; + +public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success") +.description("All FlowFiles that are read from Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure") +.description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_RETRY = new Relationship.Builder().name("retry") +.description("A FlowFile is routed to this relationship if the document cannot be fetched but attempting the operation again may succeed") +.build(); + +public static final Relationship REL_NOT_FOUND = new Relationship.Builder().name("not found") +
[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/576#discussion_r69309177 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java --- @@ -0,0 +1,293 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.elasticsearch; + +import com.squareup.okhttp.HttpUrl; +import com.squareup.okhttp.OkHttpClient; +import com.squareup.okhttp.Response; +import com.squareup.okhttp.ResponseBody; +import org.apache.commons.lang3.StringUtils; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.stream.io.ByteArrayInputStream; +import org.codehaus.jackson.JsonNode; + +import java.io.IOException; +import java.net.MalformedURLException; +import java.net.URL; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.TimeUnit; +import java.util.stream.Collectors; +import java.util.stream.Stream; + + +@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED) +@EventDriven +@SupportsBatching +@Tags({"elasticsearch", "fetch", "read", "get", "http"}) +@CapabilityDescription("Retrieves a document from Elasticsearch using the specified connection properties and the " ++ "identifier of the document to retrieve.") +@WritesAttributes({ +@WritesAttribute(attribute = "filename", description = "The filename attributes is set to the document identifier"), +@WritesAttribute(attribute = "es.index", description = "The Elasticsearch index containing the document"), +@WritesAttribute(attribute = "es.type", description = "The Elasticsearch document type") +}) +public class FetchElasticsearchHttp extends AbstractElasticsearchHttpProcessor { + +private static final String FIELD_INCLUDE_QUERY_PARAM = "_source_include"; + +public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success") +.description("All FlowFiles that are read from Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure") +.description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_RETRY = new Relationship.Builder().name("retry") +.description("A FlowFile is routed to this relationship if the document cannot be fetched but attempting the operation again may succeed") +.build(); + +public static final Relationship REL_NOT_FOUND = new Relationship.Builder().name("not found") +.descripti
[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/576#discussion_r69311424 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java --- @@ -0,0 +1,293 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.elasticsearch; + +import com.squareup.okhttp.HttpUrl; +import com.squareup.okhttp.OkHttpClient; +import com.squareup.okhttp.Response; +import com.squareup.okhttp.ResponseBody; +import org.apache.commons.lang3.StringUtils; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.stream.io.ByteArrayInputStream; +import org.codehaus.jackson.JsonNode; + +import java.io.IOException; +import java.net.MalformedURLException; +import java.net.URL; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.TimeUnit; +import java.util.stream.Collectors; +import java.util.stream.Stream; + + +@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED) +@EventDriven +@SupportsBatching +@Tags({"elasticsearch", "fetch", "read", "get", "http"}) +@CapabilityDescription("Retrieves a document from Elasticsearch using the specified connection properties and the " ++ "identifier of the document to retrieve.") +@WritesAttributes({ +@WritesAttribute(attribute = "filename", description = "The filename attributes is set to the document identifier"), +@WritesAttribute(attribute = "es.index", description = "The Elasticsearch index containing the document"), +@WritesAttribute(attribute = "es.type", description = "The Elasticsearch document type") +}) +public class FetchElasticsearchHttp extends AbstractElasticsearchHttpProcessor { + +private static final String FIELD_INCLUDE_QUERY_PARAM = "_source_include"; + +public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success") +.description("All FlowFiles that are read from Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure") +.description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_RETRY = new Relationship.Builder().name("retry") +.description("A FlowFile is routed to this relationship if the document cannot be fetched but attempting the operation again may succeed") +.build(); + +public static final Relationship REL_NOT_FOUND = new Relationship.Builder().name("not found") +
[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/576#discussion_r69312240 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java --- @@ -0,0 +1,293 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.elasticsearch; + +import com.squareup.okhttp.HttpUrl; +import com.squareup.okhttp.OkHttpClient; +import com.squareup.okhttp.Response; +import com.squareup.okhttp.ResponseBody; +import org.apache.commons.lang3.StringUtils; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.stream.io.ByteArrayInputStream; +import org.codehaus.jackson.JsonNode; + +import java.io.IOException; +import java.net.MalformedURLException; +import java.net.URL; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.TimeUnit; +import java.util.stream.Collectors; +import java.util.stream.Stream; + + +@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED) +@EventDriven +@SupportsBatching +@Tags({"elasticsearch", "fetch", "read", "get", "http"}) +@CapabilityDescription("Retrieves a document from Elasticsearch using the specified connection properties and the " ++ "identifier of the document to retrieve.") +@WritesAttributes({ +@WritesAttribute(attribute = "filename", description = "The filename attributes is set to the document identifier"), +@WritesAttribute(attribute = "es.index", description = "The Elasticsearch index containing the document"), +@WritesAttribute(attribute = "es.type", description = "The Elasticsearch document type") +}) +public class FetchElasticsearchHttp extends AbstractElasticsearchHttpProcessor { + +private static final String FIELD_INCLUDE_QUERY_PARAM = "_source_include"; + +public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success") +.description("All FlowFiles that are read from Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure") +.description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_RETRY = new Relationship.Builder().name("retry") +.description("A FlowFile is routed to this relationship if the document cannot be fetched but attempting the operation again may succeed") +.build(); + +public static final Relationship REL_NOT_FOUND = new Relationship.Builder().name("not found") +
[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/576#discussion_r69312668 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java --- @@ -0,0 +1,293 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.elasticsearch; + +import com.squareup.okhttp.HttpUrl; +import com.squareup.okhttp.OkHttpClient; +import com.squareup.okhttp.Response; +import com.squareup.okhttp.ResponseBody; +import org.apache.commons.lang3.StringUtils; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.stream.io.ByteArrayInputStream; +import org.codehaus.jackson.JsonNode; + +import java.io.IOException; +import java.net.MalformedURLException; +import java.net.URL; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.TimeUnit; +import java.util.stream.Collectors; +import java.util.stream.Stream; + + +@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED) +@EventDriven +@SupportsBatching +@Tags({"elasticsearch", "fetch", "read", "get", "http"}) +@CapabilityDescription("Retrieves a document from Elasticsearch using the specified connection properties and the " ++ "identifier of the document to retrieve.") +@WritesAttributes({ +@WritesAttribute(attribute = "filename", description = "The filename attributes is set to the document identifier"), +@WritesAttribute(attribute = "es.index", description = "The Elasticsearch index containing the document"), +@WritesAttribute(attribute = "es.type", description = "The Elasticsearch document type") +}) +public class FetchElasticsearchHttp extends AbstractElasticsearchHttpProcessor { + +private static final String FIELD_INCLUDE_QUERY_PARAM = "_source_include"; + +public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success") +.description("All FlowFiles that are read from Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure") +.description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_RETRY = new Relationship.Builder().name("retry") +.description("A FlowFile is routed to this relationship if the document cannot be fetched but attempting the operation again may succeed") +.build(); + +public static final Relationship REL_NOT_FOUND = new Relationship.Builder().name("not found") +
[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/576#discussion_r69313182 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java --- @@ -0,0 +1,293 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.elasticsearch; + +import com.squareup.okhttp.HttpUrl; +import com.squareup.okhttp.OkHttpClient; +import com.squareup.okhttp.Response; +import com.squareup.okhttp.ResponseBody; +import org.apache.commons.lang3.StringUtils; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.stream.io.ByteArrayInputStream; +import org.codehaus.jackson.JsonNode; + +import java.io.IOException; +import java.net.MalformedURLException; +import java.net.URL; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.TimeUnit; +import java.util.stream.Collectors; +import java.util.stream.Stream; + + +@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED) +@EventDriven +@SupportsBatching +@Tags({"elasticsearch", "fetch", "read", "get", "http"}) +@CapabilityDescription("Retrieves a document from Elasticsearch using the specified connection properties and the " ++ "identifier of the document to retrieve.") +@WritesAttributes({ +@WritesAttribute(attribute = "filename", description = "The filename attributes is set to the document identifier"), +@WritesAttribute(attribute = "es.index", description = "The Elasticsearch index containing the document"), +@WritesAttribute(attribute = "es.type", description = "The Elasticsearch document type") +}) +public class FetchElasticsearchHttp extends AbstractElasticsearchHttpProcessor { + +private static final String FIELD_INCLUDE_QUERY_PARAM = "_source_include"; + +public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success") +.description("All FlowFiles that are read from Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure") +.description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_RETRY = new Relationship.Builder().name("retry") +.description("A FlowFile is routed to this relationship if the document cannot be fetched but attempting the operation again may succeed") +.build(); + +public static final Relationship REL_NOT_FOUND = new Relationship.Builder().name("not found") +
[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/576#discussion_r69313807 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java --- @@ -0,0 +1,293 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.elasticsearch; + +import com.squareup.okhttp.HttpUrl; +import com.squareup.okhttp.OkHttpClient; +import com.squareup.okhttp.Response; +import com.squareup.okhttp.ResponseBody; +import org.apache.commons.lang3.StringUtils; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.stream.io.ByteArrayInputStream; +import org.codehaus.jackson.JsonNode; + +import java.io.IOException; +import java.net.MalformedURLException; +import java.net.URL; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.TimeUnit; +import java.util.stream.Collectors; +import java.util.stream.Stream; + + +@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED) +@EventDriven +@SupportsBatching +@Tags({"elasticsearch", "fetch", "read", "get", "http"}) +@CapabilityDescription("Retrieves a document from Elasticsearch using the specified connection properties and the " ++ "identifier of the document to retrieve.") +@WritesAttributes({ +@WritesAttribute(attribute = "filename", description = "The filename attributes is set to the document identifier"), +@WritesAttribute(attribute = "es.index", description = "The Elasticsearch index containing the document"), +@WritesAttribute(attribute = "es.type", description = "The Elasticsearch document type") +}) +public class FetchElasticsearchHttp extends AbstractElasticsearchHttpProcessor { + +private static final String FIELD_INCLUDE_QUERY_PARAM = "_source_include"; + +public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success") +.description("All FlowFiles that are read from Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure") +.description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_RETRY = new Relationship.Builder().name("retry") +.description("A FlowFile is routed to this relationship if the document cannot be fetched but attempting the operation again may succeed") +.build(); + +public static final Relationship REL_NOT_FOUND = new Relationship.Builder().name("not found") +
[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/576#discussion_r69314193 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java --- @@ -0,0 +1,293 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.elasticsearch; + +import com.squareup.okhttp.HttpUrl; +import com.squareup.okhttp.OkHttpClient; +import com.squareup.okhttp.Response; +import com.squareup.okhttp.ResponseBody; +import org.apache.commons.lang3.StringUtils; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.stream.io.ByteArrayInputStream; +import org.codehaus.jackson.JsonNode; + +import java.io.IOException; +import java.net.MalformedURLException; +import java.net.URL; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.TimeUnit; +import java.util.stream.Collectors; +import java.util.stream.Stream; + + +@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED) +@EventDriven +@SupportsBatching +@Tags({"elasticsearch", "fetch", "read", "get", "http"}) +@CapabilityDescription("Retrieves a document from Elasticsearch using the specified connection properties and the " ++ "identifier of the document to retrieve.") +@WritesAttributes({ +@WritesAttribute(attribute = "filename", description = "The filename attributes is set to the document identifier"), +@WritesAttribute(attribute = "es.index", description = "The Elasticsearch index containing the document"), +@WritesAttribute(attribute = "es.type", description = "The Elasticsearch document type") +}) +public class FetchElasticsearchHttp extends AbstractElasticsearchHttpProcessor { + +private static final String FIELD_INCLUDE_QUERY_PARAM = "_source_include"; + +public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success") +.description("All FlowFiles that are read from Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure") +.description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_RETRY = new Relationship.Builder().name("retry") +.description("A FlowFile is routed to this relationship if the document cannot be fetched but attempting the operation again may succeed") +.build(); + +public static final Relationship REL_NOT_FOUND = new Relationship.Builder().name("not found") +
[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/576#discussion_r69314307 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java --- @@ -0,0 +1,293 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.elasticsearch; + +import com.squareup.okhttp.HttpUrl; +import com.squareup.okhttp.OkHttpClient; +import com.squareup.okhttp.Response; +import com.squareup.okhttp.ResponseBody; +import org.apache.commons.lang3.StringUtils; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.stream.io.ByteArrayInputStream; +import org.codehaus.jackson.JsonNode; + +import java.io.IOException; +import java.net.MalformedURLException; +import java.net.URL; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.TimeUnit; +import java.util.stream.Collectors; +import java.util.stream.Stream; + + +@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED) +@EventDriven +@SupportsBatching +@Tags({"elasticsearch", "fetch", "read", "get", "http"}) +@CapabilityDescription("Retrieves a document from Elasticsearch using the specified connection properties and the " ++ "identifier of the document to retrieve.") +@WritesAttributes({ +@WritesAttribute(attribute = "filename", description = "The filename attributes is set to the document identifier"), +@WritesAttribute(attribute = "es.index", description = "The Elasticsearch index containing the document"), +@WritesAttribute(attribute = "es.type", description = "The Elasticsearch document type") +}) +public class FetchElasticsearchHttp extends AbstractElasticsearchHttpProcessor { + +private static final String FIELD_INCLUDE_QUERY_PARAM = "_source_include"; + +public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success") +.description("All FlowFiles that are read from Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure") +.description("All FlowFiles that cannot be read from Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_RETRY = new Relationship.Builder().name("retry") +.description("A FlowFile is routed to this relationship if the document cannot be fetched but attempting the operation again may succeed") +.build(); + +public static final Relationship REL_NOT_FOUND = new Relationship.Builder().name("not found") +
[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/576#discussion_r69316019 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java --- @@ -0,0 +1,330 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.elasticsearch; + +import com.squareup.okhttp.MediaType; +import com.squareup.okhttp.OkHttpClient; +import com.squareup.okhttp.RequestBody; +import com.squareup.okhttp.Response; +import com.squareup.okhttp.ResponseBody; +import org.apache.commons.io.IOUtils; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.expression.AttributeExpression; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.stream.io.ByteArrayInputStream; +import org.apache.nifi.util.StringUtils; +import org.codehaus.jackson.JsonNode; +import org.codehaus.jackson.node.ArrayNode; + +import java.io.IOException; +import java.net.URL; +import java.nio.charset.Charset; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.Set; + +import static org.apache.commons.lang3.StringUtils.trimToEmpty; + + +@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED) +@EventDriven +@SupportsBatching +@Tags({"elasticsearch", "insert", "update", "upsert", "delete", "write", "put", "http"}) +@CapabilityDescription("Writes the contents of a FlowFile to Elasticsearch, using the specified parameters such as " ++ "the index to insert into and the type of the document.") +public class PutElasticsearchHttp extends AbstractElasticsearchHttpProcessor { + +public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success") +.description("All FlowFiles that are written to Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure") +.description("All FlowFiles that cannot be written to Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_RETRY = new Relationship.Builder().name("retry") +.description("A FlowFile is routed to this relationship if the database cannot be updated but attempting the operation again may succeed") +.build(); + +public static final PropertyDescriptor ID_ATTRIBUTE = new PropertyDescriptor.Builder() +.name("Identifier Attribute") +.description("The name of the FlowFile attribute containing the identifier for the document. If the Index Operation is \"index\", " ++ "this property may be left empty or evalua
[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/576#discussion_r69316350 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java --- @@ -0,0 +1,330 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.elasticsearch; + +import com.squareup.okhttp.MediaType; +import com.squareup.okhttp.OkHttpClient; +import com.squareup.okhttp.RequestBody; +import com.squareup.okhttp.Response; +import com.squareup.okhttp.ResponseBody; +import org.apache.commons.io.IOUtils; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.expression.AttributeExpression; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.stream.io.ByteArrayInputStream; +import org.apache.nifi.util.StringUtils; +import org.codehaus.jackson.JsonNode; +import org.codehaus.jackson.node.ArrayNode; + +import java.io.IOException; +import java.net.URL; +import java.nio.charset.Charset; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.Set; + +import static org.apache.commons.lang3.StringUtils.trimToEmpty; + + +@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED) +@EventDriven +@SupportsBatching +@Tags({"elasticsearch", "insert", "update", "upsert", "delete", "write", "put", "http"}) +@CapabilityDescription("Writes the contents of a FlowFile to Elasticsearch, using the specified parameters such as " ++ "the index to insert into and the type of the document.") +public class PutElasticsearchHttp extends AbstractElasticsearchHttpProcessor { + +public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success") +.description("All FlowFiles that are written to Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure") +.description("All FlowFiles that cannot be written to Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_RETRY = new Relationship.Builder().name("retry") +.description("A FlowFile is routed to this relationship if the database cannot be updated but attempting the operation again may succeed") +.build(); + +public static final PropertyDescriptor ID_ATTRIBUTE = new PropertyDescriptor.Builder() +.name("Identifier Attribute") +.description("The name of the FlowFile attribute containing the identifier for the document. If the Index Operation is \"index\", " ++ "this property may be left empty or evalua
[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/576#discussion_r69317035 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java --- @@ -0,0 +1,330 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.elasticsearch; + +import com.squareup.okhttp.MediaType; +import com.squareup.okhttp.OkHttpClient; +import com.squareup.okhttp.RequestBody; +import com.squareup.okhttp.Response; +import com.squareup.okhttp.ResponseBody; +import org.apache.commons.io.IOUtils; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.expression.AttributeExpression; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.stream.io.ByteArrayInputStream; +import org.apache.nifi.util.StringUtils; +import org.codehaus.jackson.JsonNode; +import org.codehaus.jackson.node.ArrayNode; + +import java.io.IOException; +import java.net.URL; +import java.nio.charset.Charset; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.Set; + +import static org.apache.commons.lang3.StringUtils.trimToEmpty; + + +@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED) +@EventDriven +@SupportsBatching +@Tags({"elasticsearch", "insert", "update", "upsert", "delete", "write", "put", "http"}) +@CapabilityDescription("Writes the contents of a FlowFile to Elasticsearch, using the specified parameters such as " ++ "the index to insert into and the type of the document.") +public class PutElasticsearchHttp extends AbstractElasticsearchHttpProcessor { + +public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success") +.description("All FlowFiles that are written to Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure") +.description("All FlowFiles that cannot be written to Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_RETRY = new Relationship.Builder().name("retry") +.description("A FlowFile is routed to this relationship if the database cannot be updated but attempting the operation again may succeed") +.build(); + +public static final PropertyDescriptor ID_ATTRIBUTE = new PropertyDescriptor.Builder() +.name("Identifier Attribute") +.description("The name of the FlowFile attribute containing the identifier for the document. If the Index Operation is \"index\", " ++ "this property may be left empty or evalua
[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/576#discussion_r69317281 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java --- @@ -0,0 +1,330 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.elasticsearch; + +import com.squareup.okhttp.MediaType; +import com.squareup.okhttp.OkHttpClient; +import com.squareup.okhttp.RequestBody; +import com.squareup.okhttp.Response; +import com.squareup.okhttp.ResponseBody; +import org.apache.commons.io.IOUtils; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.expression.AttributeExpression; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.stream.io.ByteArrayInputStream; +import org.apache.nifi.util.StringUtils; +import org.codehaus.jackson.JsonNode; +import org.codehaus.jackson.node.ArrayNode; + +import java.io.IOException; +import java.net.URL; +import java.nio.charset.Charset; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.Set; + +import static org.apache.commons.lang3.StringUtils.trimToEmpty; + + +@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED) +@EventDriven +@SupportsBatching +@Tags({"elasticsearch", "insert", "update", "upsert", "delete", "write", "put", "http"}) +@CapabilityDescription("Writes the contents of a FlowFile to Elasticsearch, using the specified parameters such as " ++ "the index to insert into and the type of the document.") +public class PutElasticsearchHttp extends AbstractElasticsearchHttpProcessor { + +public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success") +.description("All FlowFiles that are written to Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure") +.description("All FlowFiles that cannot be written to Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_RETRY = new Relationship.Builder().name("retry") +.description("A FlowFile is routed to this relationship if the database cannot be updated but attempting the operation again may succeed") +.build(); + +public static final PropertyDescriptor ID_ATTRIBUTE = new PropertyDescriptor.Builder() +.name("Identifier Attribute") +.description("The name of the FlowFile attribute containing the identifier for the document. If the Index Operation is \"index\", " ++ "this property may be left empty or evalua
[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/576#discussion_r69317330 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java --- @@ -0,0 +1,330 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.elasticsearch; + +import com.squareup.okhttp.MediaType; +import com.squareup.okhttp.OkHttpClient; +import com.squareup.okhttp.RequestBody; +import com.squareup.okhttp.Response; +import com.squareup.okhttp.ResponseBody; +import org.apache.commons.io.IOUtils; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.expression.AttributeExpression; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.stream.io.ByteArrayInputStream; +import org.apache.nifi.util.StringUtils; +import org.codehaus.jackson.JsonNode; +import org.codehaus.jackson.node.ArrayNode; + +import java.io.IOException; +import java.net.URL; +import java.nio.charset.Charset; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.Set; + +import static org.apache.commons.lang3.StringUtils.trimToEmpty; + + +@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED) +@EventDriven +@SupportsBatching +@Tags({"elasticsearch", "insert", "update", "upsert", "delete", "write", "put", "http"}) +@CapabilityDescription("Writes the contents of a FlowFile to Elasticsearch, using the specified parameters such as " ++ "the index to insert into and the type of the document.") +public class PutElasticsearchHttp extends AbstractElasticsearchHttpProcessor { + +public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success") +.description("All FlowFiles that are written to Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure") +.description("All FlowFiles that cannot be written to Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_RETRY = new Relationship.Builder().name("retry") +.description("A FlowFile is routed to this relationship if the database cannot be updated but attempting the operation again may succeed") +.build(); + +public static final PropertyDescriptor ID_ATTRIBUTE = new PropertyDescriptor.Builder() +.name("Identifier Attribute") +.description("The name of the FlowFile attribute containing the identifier for the document. If the Index Operation is \"index\", " ++ "this property may be left empty or evalua
[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/576#discussion_r69317662 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java --- @@ -0,0 +1,330 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.elasticsearch; + +import com.squareup.okhttp.MediaType; +import com.squareup.okhttp.OkHttpClient; +import com.squareup.okhttp.RequestBody; +import com.squareup.okhttp.Response; +import com.squareup.okhttp.ResponseBody; +import org.apache.commons.io.IOUtils; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.expression.AttributeExpression; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.stream.io.ByteArrayInputStream; +import org.apache.nifi.util.StringUtils; +import org.codehaus.jackson.JsonNode; +import org.codehaus.jackson.node.ArrayNode; + +import java.io.IOException; +import java.net.URL; +import java.nio.charset.Charset; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.Set; + +import static org.apache.commons.lang3.StringUtils.trimToEmpty; + + +@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED) +@EventDriven +@SupportsBatching +@Tags({"elasticsearch", "insert", "update", "upsert", "delete", "write", "put", "http"}) +@CapabilityDescription("Writes the contents of a FlowFile to Elasticsearch, using the specified parameters such as " ++ "the index to insert into and the type of the document.") +public class PutElasticsearchHttp extends AbstractElasticsearchHttpProcessor { + +public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success") +.description("All FlowFiles that are written to Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure") +.description("All FlowFiles that cannot be written to Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_RETRY = new Relationship.Builder().name("retry") +.description("A FlowFile is routed to this relationship if the database cannot be updated but attempting the operation again may succeed") +.build(); + +public static final PropertyDescriptor ID_ATTRIBUTE = new PropertyDescriptor.Builder() +.name("Identifier Attribute") +.description("The name of the FlowFile attribute containing the identifier for the document. If the Index Operation is \"index\", " ++ "this property may be left empty or evalua
[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/576#discussion_r69318008 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java --- @@ -0,0 +1,330 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.elasticsearch; + +import com.squareup.okhttp.MediaType; +import com.squareup.okhttp.OkHttpClient; +import com.squareup.okhttp.RequestBody; +import com.squareup.okhttp.Response; +import com.squareup.okhttp.ResponseBody; +import org.apache.commons.io.IOUtils; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.expression.AttributeExpression; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.stream.io.ByteArrayInputStream; +import org.apache.nifi.util.StringUtils; +import org.codehaus.jackson.JsonNode; +import org.codehaus.jackson.node.ArrayNode; + +import java.io.IOException; +import java.net.URL; +import java.nio.charset.Charset; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.Set; + +import static org.apache.commons.lang3.StringUtils.trimToEmpty; + + +@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED) +@EventDriven +@SupportsBatching +@Tags({"elasticsearch", "insert", "update", "upsert", "delete", "write", "put", "http"}) +@CapabilityDescription("Writes the contents of a FlowFile to Elasticsearch, using the specified parameters such as " ++ "the index to insert into and the type of the document.") +public class PutElasticsearchHttp extends AbstractElasticsearchHttpProcessor { + +public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success") +.description("All FlowFiles that are written to Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure") +.description("All FlowFiles that cannot be written to Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_RETRY = new Relationship.Builder().name("retry") +.description("A FlowFile is routed to this relationship if the database cannot be updated but attempting the operation again may succeed") +.build(); + +public static final PropertyDescriptor ID_ATTRIBUTE = new PropertyDescriptor.Builder() +.name("Identifier Attribute") +.description("The name of the FlowFile attribute containing the identifier for the document. If the Index Operation is \"index\", " ++ "this property may be left empty or evalua
[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/576#discussion_r69318837 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java --- @@ -0,0 +1,330 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.elasticsearch; + +import com.squareup.okhttp.MediaType; +import com.squareup.okhttp.OkHttpClient; +import com.squareup.okhttp.RequestBody; +import com.squareup.okhttp.Response; +import com.squareup.okhttp.ResponseBody; +import org.apache.commons.io.IOUtils; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.expression.AttributeExpression; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.stream.io.ByteArrayInputStream; +import org.apache.nifi.util.StringUtils; +import org.codehaus.jackson.JsonNode; +import org.codehaus.jackson.node.ArrayNode; + +import java.io.IOException; +import java.net.URL; +import java.nio.charset.Charset; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.Set; + +import static org.apache.commons.lang3.StringUtils.trimToEmpty; + + +@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED) +@EventDriven +@SupportsBatching +@Tags({"elasticsearch", "insert", "update", "upsert", "delete", "write", "put", "http"}) +@CapabilityDescription("Writes the contents of a FlowFile to Elasticsearch, using the specified parameters such as " ++ "the index to insert into and the type of the document.") +public class PutElasticsearchHttp extends AbstractElasticsearchHttpProcessor { + +public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success") +.description("All FlowFiles that are written to Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure") +.description("All FlowFiles that cannot be written to Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_RETRY = new Relationship.Builder().name("retry") +.description("A FlowFile is routed to this relationship if the database cannot be updated but attempting the operation again may succeed") +.build(); + +public static final PropertyDescriptor ID_ATTRIBUTE = new PropertyDescriptor.Builder() +.name("Identifier Attribute") +.description("The name of the FlowFile attribute containing the identifier for the document. If the Index Operation is \"index\", " ++ "this property may be left empty or evalua
[GitHub] nifi pull request #576: NIFI-2068: Add Elasticsearch HTTP processors
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/576#discussion_r69319085 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java --- @@ -0,0 +1,330 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.elasticsearch; + +import com.squareup.okhttp.MediaType; +import com.squareup.okhttp.OkHttpClient; +import com.squareup.okhttp.RequestBody; +import com.squareup.okhttp.Response; +import com.squareup.okhttp.ResponseBody; +import org.apache.commons.io.IOUtils; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.SupportsBatching; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.ValidationContext; +import org.apache.nifi.components.ValidationResult; +import org.apache.nifi.expression.AttributeExpression; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.stream.io.ByteArrayInputStream; +import org.apache.nifi.util.StringUtils; +import org.codehaus.jackson.JsonNode; +import org.codehaus.jackson.node.ArrayNode; + +import java.io.IOException; +import java.net.URL; +import java.nio.charset.Charset; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.Set; + +import static org.apache.commons.lang3.StringUtils.trimToEmpty; + + +@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED) +@EventDriven +@SupportsBatching +@Tags({"elasticsearch", "insert", "update", "upsert", "delete", "write", "put", "http"}) +@CapabilityDescription("Writes the contents of a FlowFile to Elasticsearch, using the specified parameters such as " ++ "the index to insert into and the type of the document.") +public class PutElasticsearchHttp extends AbstractElasticsearchHttpProcessor { + +public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success") +.description("All FlowFiles that are written to Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure") +.description("All FlowFiles that cannot be written to Elasticsearch are routed to this relationship").build(); + +public static final Relationship REL_RETRY = new Relationship.Builder().name("retry") +.description("A FlowFile is routed to this relationship if the database cannot be updated but attempting the operation again may succeed") +.build(); + +public static final PropertyDescriptor ID_ATTRIBUTE = new PropertyDescriptor.Builder() +.name("Identifier Attribute") +.description("The name of the FlowFile attribute containing the identifier for the document. If the Index Operation is \"index\", " ++ "this property may be left empty or evalua