[jira] [Updated] (NIFI-673) Create ListSFTP, FetchSFTP Processors
[ https://issues.apache.org/jira/browse/NIFI-673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-673: Attachment: 0001-NIFI-673-Added-Completion-Strategy-to-FetchSFTP.patch > Create ListSFTP, FetchSFTP Processors > - > > Key: NIFI-673 > URL: https://issues.apache.org/jira/browse/NIFI-673 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.4.0 > > Attachments: > 0001-NIFI-673-Initial-implementation-of-ListSFTP-FetchSFT.patch, > 0002-NIFI-673-Added-Completion-Strategy-to-FetchSFTP.patch > > > This will allow us to pull a listing from a single primary node and then > distribute the work of pulling and processing the data across the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-631) Create ListFile and FetchFile processors
[ https://issues.apache.org/jira/browse/NIFI-631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-631: Attachment: 0001-NIFI-631-Initial-implementation-of-FetchFile-process.patch > Create ListFile and FetchFile processors > > > Key: NIFI-631 > URL: https://issues.apache.org/jira/browse/NIFI-631 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mark Payne >Assignee: Joe Skora > Attachments: > 0001-NIFI-631-Initial-implementation-of-FetchFile-process.patch > > > This pair of Processors will provide several benefits over the existing > GetFile processor: > 1. Currently, GetFile will continually pull the same files if the "Keep > Source File" property is set to true. There is no way to pull the file and > leave it in the directory without continually pulling the same file. We could > implement state here, but it would either be a huge amount of state to > remember everything pulled or it would have to always pull the oldest file > first so that we can maintain just the Last Modified Date of the last file > pulled plus all files with the same Last Modified Date that have already been > pulled. > 2. If pulling from a network attached storage such as NFS, this would allow a > single processor to run ListFiles and then distribute those FlowFiles to the > cluster so that the cluster can share the work of pulling the data. > 3. There are use cases when we may want to pull a specific file (for example, > in conjunction with ProcessHttpRequest/ProcessHttpResponse) rather than just > pull all files in a directory. GetFile does not support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-673) Create ListSFTP, FetchSFTP Processors
[ https://issues.apache.org/jira/browse/NIFI-673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14943949#comment-14943949 ] Mark Payne commented on NIFI-673: - I have attached patches for ListSFTP / FetchSFTP. In order to apply the second patch, the first patch must already be applied. I.e., the second patch builds upon the first. > Create ListSFTP, FetchSFTP Processors > - > > Key: NIFI-673 > URL: https://issues.apache.org/jira/browse/NIFI-673 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.4.0 > > Attachments: > 0001-NIFI-673-Initial-implementation-of-ListSFTP-FetchSFT.patch, > 0002-NIFI-673-Added-Completion-Strategy-to-FetchSFTP.patch > > > This will allow us to pull a listing from a single primary node and then > distribute the work of pulling and processing the data across the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-673) Create ListSFTP, FetchSFTP Processors
[ https://issues.apache.org/jira/browse/NIFI-673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-673: Attachment: 0002-NIFI-673-Added-Completion-Strategy-to-FetchSFTP.patch > Create ListSFTP, FetchSFTP Processors > - > > Key: NIFI-673 > URL: https://issues.apache.org/jira/browse/NIFI-673 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.4.0 > > Attachments: > 0001-NIFI-673-Initial-implementation-of-ListSFTP-FetchSFT.patch, > 0002-NIFI-673-Added-Completion-Strategy-to-FetchSFTP.patch > > > This will allow us to pull a listing from a single primary node and then > distribute the work of pulling and processing the data across the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (NIFI-34) Mock Framework should provide option to detect common bad practices/bugs
[
https://issues.apache.org/jira/browse/NIFI-34?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14943548#comment-14943548
]
Mark Payne edited comment on NIFI-34 at 10/5/15 3:44 PM:
-
Another thing that I would like to see here is detecting when the logger is
used with the wrong number of arguments. For example:
{code}
getLogger().warn("Something went wrong with {}", new Object[] {flowFile,
exception});
{code}
or
{code}
getLogger().warn("Something went wrong with {} : {}", new Object[] {flowFile});
{code}
The mock framework should use a logger that is capable of understanding how
many {} pairs we have and look at the number of arguments and ensure that the
two align.
was (Author: markap14):
Another thing that I would like to see here is detecting when the logger is
used with the wrong number of arguments (e.g., getLogger().warn("Something went
wrong with {}", new Object[] {flowFile, exception}); or
getLogger().warn("Something went wrong with {} : {}", new Object[] {flowFile});
The mock framework should use a logger that is capable of understanding how
many {} pairs we have and look at the number of arguments and ensure that the
two align.
> Mock Framework should provide option to detect common bad practices/bugs
>
>
> Key: NIFI-34
> URL: https://issues.apache.org/jira/browse/NIFI-34
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Reporter: Mark Payne
>Priority: Minor
>
> Mock Framework should detect common errors such as:
> * Processor has member variable that is a PropertyDescriptor, but the
> PropertyDescriptor isn't returned in the list of supported property
> descriptors.
> * Processor has member variable that is a Relationship, but the Relationship
> isn't returned in the Set of Relationships.
> * Processor has multiple properties or relationships as member variables with
> the same name.
> * No META-INF/services file
> * META-INF/services file doesn't contain the Component's Fully Qualified
> Class Name
> * No @CapabilityDescription annotation
> * No @Tags annotation
> Mock Framework should automatically detect these things and fail the unit
> test unless checking is disabled. This requires building an object that
> allows developer to enable/disable each of these checks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-34) Mock Framework should provide option to detect common bad practices/bugs
[
https://issues.apache.org/jira/browse/NIFI-34?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14943548#comment-14943548
]
Mark Payne commented on NIFI-34:
Another thing that I would like to see here is detecting when the logger is
used with the wrong number of arguments (e.g., getLogger().warn("Something went
wrong with {}", new Object[] {flowFile, exception}); or
getLogger().warn("Something went wrong with {} : {}", new Object[] {flowFile});
The mock framework should use a logger that is capable of understanding how
many {} pairs we have and look at the number of arguments and ensure that the
two align.
> Mock Framework should provide option to detect common bad practices/bugs
>
>
> Key: NIFI-34
> URL: https://issues.apache.org/jira/browse/NIFI-34
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Reporter: Mark Payne
>Priority: Minor
>
> Mock Framework should detect common errors such as:
> * Processor has member variable that is a PropertyDescriptor, but the
> PropertyDescriptor isn't returned in the list of supported property
> descriptors.
> * Processor has member variable that is a Relationship, but the Relationship
> isn't returned in the Set of Relationships.
> * Processor has multiple properties or relationships as member variables with
> the same name.
> * No META-INF/services file
> * META-INF/services file doesn't contain the Component's Fully Qualified
> Class Name
> * No @CapabilityDescription annotation
> * No @Tags annotation
> Mock Framework should automatically detect these things and fail the unit
> test unless checking is disabled. This requires building an object that
> allows developer to enable/disable each of these checks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-673) Create ListSFTP, FetchSFTP Processors
[ https://issues.apache.org/jira/browse/NIFI-673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-673: Attachment: 0001-NIFI-673-Initial-implementation-of-ListSFTP-FetchSFT.patch > Create ListSFTP, FetchSFTP Processors > - > > Key: NIFI-673 > URL: https://issues.apache.org/jira/browse/NIFI-673 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.4.0 > > Attachments: > 0001-NIFI-673-Initial-implementation-of-ListSFTP-FetchSFT.patch > > > This will allow us to pull a listing from a single primary node and then > distribute the work of pulling and processing the data across the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-673) Create ListSFTP, FetchSFTP Processors
[ https://issues.apache.org/jira/browse/NIFI-673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-673: Fix Version/s: 0.4.0 > Create ListSFTP, FetchSFTP Processors > - > > Key: NIFI-673 > URL: https://issues.apache.org/jira/browse/NIFI-673 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Mark Payne > Fix For: 0.4.0 > > > This will allow us to pull a listing from a single primary node and then > distribute the work of pulling and processing the data across the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (NIFI-673) Create ListSFTP, FetchSFTP Processors
[ https://issues.apache.org/jira/browse/NIFI-673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne reassigned NIFI-673: --- Assignee: Mark Payne > Create ListSFTP, FetchSFTP Processors > - > > Key: NIFI-673 > URL: https://issues.apache.org/jira/browse/NIFI-673 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.4.0 > > > This will allow us to pull a listing from a single primary node and then > distribute the work of pulling and processing the data across the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-1009) InvokeHTTP should be able to be scheduled without any incoming connection for GET operations
Mark Payne created NIFI-1009: Summary: InvokeHTTP should be able to be scheduled without any incoming connection for GET operations Key: NIFI-1009 URL: https://issues.apache.org/jira/browse/NIFI-1009 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Mark Payne Fix For: 0.4.0 Many users have been confused by InvokeHTTP not running when it is a "Source" processor. It probably doesn't make sense to do a POST or a PUT operation without an incoming FlowFile. However, if the Processor is configured to do a GET or DELETE, it should be able to run as a Source processor. Validation logic should indicate that the processor is not valid if it has no incoming connection and is configured to perform a POST or a PUT. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1008) NiFi should swap out FlowFiles to disk even before the session is committed
[ https://issues.apache.org/jira/browse/NIFI-1008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14936853#comment-14936853 ] Mark Payne commented on NIFI-1008: -- Removed Fix Version of 0.4.0. Looking into the implementation details, this could be extremely complex. This is because the FlowFile Repository must be updated atomically with all FlowFiles that are processed in a single session. In order to do this, we need all of the FlowFiles to be passed as a single Collection. Otherwise, if we restart in the middle of a session commit, some of the updates will have taken place but not all of them. This could cause some really odd behavior. One possible solution is to modify the FlowFile Repository's definition to allow an Iterator to be passed instead of a Collection, and then we can implement an iterator that deserializes the objects as needed. > NiFi should swap out FlowFiles to disk even before the session is committed > --- > > Key: NIFI-1008 > URL: https://issues.apache.org/jira/browse/NIFI-1008 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne > > Currently, NiFi will swap out FlowFiles if there are a large number in a > FlowFile Queue. This is done to avoid running out of JVM heap space. However, > if we have a simple flow like GetFile -> SplitText and GetFile pulls in a > large file, SplitText can quickly cause OutOfMemoryError. This is not because > it buffers the content of the FlowFile in memory but rather because it holds > the millions of FlowFile objects in memory. We can do better. > When we call session.transfer for the FlowFiles, once we hit a magical > threshold (say 10,000), we should swap those FlowFiles to disk and the > session should transfer them to the queue "swapped out" flowfiles, rather > than having to buffer all of these in memory and then swapping them out once > they land in the queue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-516) MergeContent when merging 10,000s or more objects can cause too many open files
[ https://issues.apache.org/jira/browse/NIFI-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-516: Assignee: Joseph Percivall > MergeContent when merging 10,000s or more objects can cause too many open > files > --- > > Key: NIFI-516 > URL: https://issues.apache.org/jira/browse/NIFI-516 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Joseph Witt >Assignee: Joseph Percivall > Fix For: 0.4.0 > > > MergeContent really is just exposing an issue where the process session keeps > file handles open longer than necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-994) Processor to tail files
[ https://issues.apache.org/jira/browse/NIFI-994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14936888#comment-14936888 ] Mark Payne commented on NIFI-994: - Agreed. I'd recommend we allow the filename to tail to contain a * so that as things roll over we can still process the data. We could sort on last modified time to know the ordering of the files, and if we keep an offset into a file plus the timestamp when we pulled that file, that should help us to know which file it came from (the one with the smallest Last Modified timestamp >= our timestamp) and then we know which offset we left off at. If the data rolls off then you're right - there's nothing we can do about that. Would recommend we mention in the @CapabilityDescription that we expect logs to be kept around long enough to recover from outages. > Processor to tail files > --- > > Key: NIFI-994 > URL: https://issues.apache.org/jira/browse/NIFI-994 > Project: Apache NiFi > Issue Type: New Feature >Affects Versions: 0.4.0 >Reporter: Joseph Percivall >Assignee: Joseph Percivall > > It's a very common data ingest situation to want to input text into the > system by "tailing" a file, most commonly log files. Currently we don't have > an easy way to do this. > A simple processor to tail a file would benefit many users. There would need > to be an option to not just tail a file but pick up where the processor left > off if it is interrupted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-1006) Change format for configuring repositories in nifi.properties file
Mark Payne created NIFI-1006: Summary: Change format for configuring repositories in nifi.properties file Key: NIFI-1006 URL: https://issues.apache.org/jira/browse/NIFI-1006 Project: Apache NiFi Issue Type: Improvement Components: Configuration, Core Framework Reporter: Mark Payne Fix For: 1.0.0 Currently, the Content Repository, FlowFile Repository, and Provenance Repository are all configured within the nifi.properties file. This includes the repository implementation to use and all properties for those repositories. This becomes quite confusing to configure, and it makes it very difficult to provide examples of each of these repositories. I would like to see this changed to a format like is used to configure the Authority Providers. This way, in the nifi.properties file, we would configure just two things: the .xml file that includes the repository configurations (with IDs) and the ID of the repository to use. This also makes it far easier to create a repository that may wrap one or more other repositories, by defining them all in the .xml file and then specifying the ID of the 'wrapping' repository as the one to use. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-817) Create Processors to interact with HBase
[ https://issues.apache.org/jira/browse/NIFI-817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14935349#comment-14935349 ] Mark Payne commented on NIFI-817: - [~ndimiduk] [~nmaillard] - the basic premise is to provide an ETL-like functionality to pull data from HBase, as well as provide the ability to stream data into HBase. I very much like the idea of plugging into an HBase Firehose. Is this something that is configured on the HBase instance itself, to allow NiFi access to the stream? I will have to look more deeply into how those semantics work for sure. Would also like to have [~bende] looking into this from the NiFi perspective. I definitely agree as well that we need to look into the filter language. Kite is not a NiFi thing. It is a set of libraries developed by Cloudera (quick intro at http://kitesdk.org/docs/1.0.0/Kite-SDK-Guide.html). Some of the Cloudera guys provided some Kite Processors for NiFi help push "Kite Datasets" (Avro based datasets) to HDFS. > Create Processors to interact with HBase > > > Key: NIFI-817 > URL: https://issues.apache.org/jira/browse/NIFI-817 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.4.0 > > Attachments: > 0001-NIFI-817-Initial-implementation-of-HBase-processors.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-1008) NiFi should swap out FlowFiles to disk even before the session is committed
Mark Payne created NIFI-1008: Summary: NiFi should swap out FlowFiles to disk even before the session is committed Key: NIFI-1008 URL: https://issues.apache.org/jira/browse/NIFI-1008 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Reporter: Mark Payne Currently, NiFi will swap out FlowFiles if there are a large number in a FlowFile Queue. This is done to avoid running out of JVM heap space. However, if we have a simple flow like GetFile -> SplitText and GetFile pulls in a large file, SplitText can quickly cause OutOfMemoryError. This is not because it buffers the content of the FlowFile in memory but rather because it holds the millions of FlowFile objects in memory. We can do better. When we call session.transfer for the FlowFiles, once we hit a magical threshold (say 10,000), we should swap those FlowFiles to disk and the session should transfer them to the queue "swapped out" flowfiles, rather than having to buffer all of these in memory and then swapping them out once they land in the queue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-900) Add support for sending/receiving data with Azure Event Hub
[ https://issues.apache.org/jira/browse/NIFI-900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14933816#comment-14933816 ] Mark Payne commented on NIFI-900: - Initially I went down the path of adding support for AMQP 1.0 to the JMS Processors. This worked okay for Putting, but only if there was a single message per transaction, as Event Hub does not allow for transacted sessions. It had a significant limitation for receiving, because the Destination Name that we would have to use would be /ConsumerGroups//Partitions/ This doesn't work well, as we'd need N processors for N partitions. As a result, I created a new nifi-azure-nar and used the eventhubs-client. Since i had already done the work to support AMQP 1.0, I left that in so that we can communicate with other AMQP-enabled JMS Providers, though. As-is, the GetAzureEventHub processor will resume reading only from "this point on" when you start it. It functions similar to a "tail" on a file and does not distribute state, so if run in a cluster will need to be run on primary node only. It does, however, pull from all partitions with a single Processor. Suggest we use this processor as it is now for the time being and implement a more full-featured capability after the NiFi State Management feature proposal has been tackled to supply the ability to easily share state across all nodes in a cluster as well as assign partitions to individual nodes. > Add support for sending/receiving data with Azure Event Hub > --- > > Key: NIFI-900 > URL: https://issues.apache.org/jira/browse/NIFI-900 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Joseph Witt > Fix For: 0.4.0 > > Attachments: > 0001-NIFI-900-Added-Azure-nar-added-support-for-AMQP-1.0-.patch > > > Azure event hubs appears to support AMQP. If so then perhaps we only need to > build a processor to get/put via AMQP. Needs research and determination of > best path forward. But there is clearly an interest to interact with Azure > as a hosting platform and Azure Event hub for messaging. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-900) Add support for sending/receiving data with Azure Event Hub
[ https://issues.apache.org/jira/browse/NIFI-900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-900: Labels: (was: beginner) > Add support for sending/receiving data with Azure Event Hub > --- > > Key: NIFI-900 > URL: https://issues.apache.org/jira/browse/NIFI-900 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Joseph Witt > Fix For: 0.4.0 > > Attachments: > 0001-NIFI-900-Added-Azure-nar-added-support-for-AMQP-1.0-.patch > > > Azure event hubs appears to support AMQP. If so then perhaps we only need to > build a processor to get/put via AMQP. Needs research and determination of > best path forward. But there is clearly an interest to interact with Azure > as a hosting platform and Azure Event hub for messaging. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1003) Ability to auto-terminate relationship connections for a processor
[
https://issues.apache.org/jira/browse/NIFI-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14933785#comment-14933785
]
Mark Payne commented on NIFI-1003:
--
Rick,
I think the approach that would make more sense here, rather than setting some
sort of state on the context would be to set the property on the Relationship
itself. For instance:
{code}
final Relationship SOME_RELATIONSHIP = new Relationship.Builder()
.name("Some Relationship")
.description("Some Description")
.autoTerminateDefault(true)
.build();
{code}
> Ability to auto-terminate relationship connections for a processor
> --
>
> Key: NIFI-1003
> URL: https://issues.apache.org/jira/browse/NIFI-1003
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Core Framework
>Affects Versions: 0.3.0
> Environment: New processor with many dynamically-generated
> relationships
>Reporter: Rick Braddy
>Priority: Minor
>
> Need a way to auto-terminate relationships without user intervention; that
> is, change the default behavior of a processor to automatically
> self-terminate connections for relationships when they are created.
> Ideally, this would be possible to set at processor init() time, so all
> relationships on the processor will be auto-terminated by default. Since this
> primarily applies to relationships added dynamically, it could also be called
> via the ProcessContext object during onTrigger().
> context.autoTerminateConnections( boolean selfAutoTerminate);
> where calling autoTerminateConnections( true ) would cause future
> relationships to be auto-terminated by default (instead of the default, which
> is to require manually setting auto-termination).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-900) Add support for sending/receiving data with Azure Event Hub
[ https://issues.apache.org/jira/browse/NIFI-900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-900: Attachment: 0001-NIFI-900-Added-Azure-nar-added-support-for-AMQP-1.0-.patch > Add support for sending/receiving data with Azure Event Hub > --- > > Key: NIFI-900 > URL: https://issues.apache.org/jira/browse/NIFI-900 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Joseph Witt > Labels: beginner > Fix For: 0.4.0 > > Attachments: > 0001-NIFI-900-Added-Azure-nar-added-support-for-AMQP-1.0-.patch > > > Azure event hubs appears to support AMQP. If so then perhaps we only need to > build a processor to get/put via AMQP. Needs research and determination of > best path forward. But there is clearly an interest to interact with Azure > as a hosting platform and Azure Event hub for messaging. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-1000) GetJMSQueue adds wrong value for jms.JMSDestination attribute
Mark Payne created NIFI-1000: Summary: GetJMSQueue adds wrong value for jms.JMSDestination attribute Key: NIFI-1000 URL: https://issues.apache.org/jira/browse/NIFI-1000 Project: Apache NiFi Issue Type: Bug Components: Extensions Reporter: Mark Payne Fix For: 0.4.0 I used LogAttribute to view the output of messages pulled via GetJMSQueue. The Destination's name was the wrong value: Key: 'jms.JMSDestination' Value: 'org.apache.qpid.amqp_1_0.jms.impl.QueueImpl@1a5bf541' It appears that the attribute is being set to the toString() of the destination, rather than obtaining the actual name -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-604) ExecuteStreamCommand does not support arguments with semicolons
[
https://issues.apache.org/jira/browse/NIFI-604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908017#comment-14908017
]
Mark Payne commented on NIFI-604:
-
[~rickysaltzer] - looks good! Tested it and all seems to work well. Passes
contrib-check. Pushed to master. Thanks for the knocking this out!
> ExecuteStreamCommand does not support arguments with semicolons
>
>
> Key: NIFI-604
> URL: https://issues.apache.org/jira/browse/NIFI-604
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 0.1.0
>Reporter: Ricky Saltzer
>Assignee: Mark Payne
> Fix For: 0.4.0
>
> Attachments: NIFI-604.1.patch, NIFI-604.2.patch
>
>
> The following code in ExecuteStreamCommand assumes you're not passing
> semicolons within your argument. This is a problem for people who need to
> pass semicolons to the executing program as part of the argument.
> {code}
> 224for (String arg : commandArguments.split(";")) {
> {code}
> To allow for escaped semicolons, I propose we change this to the following
> regex.
> {code}
> 224for (String arg : commandArguments.split("[^\\];")) {
> {code}
> *or*... could we just change the way arguments are passed to make it more
> similar to how ExecuteCommand works? The whole semicolon per argument took
> some getting used to, and doesn't seem very intuitive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-973) Create a Getting Started Guide
[ https://issues.apache.org/jira/browse/NIFI-973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-973: Fix Version/s: 0.4.0 > Create a Getting Started Guide > -- > > Key: NIFI-973 > URL: https://issues.apache.org/jira/browse/NIFI-973 > Project: Apache NiFi > Issue Type: Task > Components: Documentation & Website >Reporter: Mark Payne > Fix For: 0.4.0 > > Attachments: 0001-NIFI-973-Created-a-Getting-Started-Guide.patch > > > Currently, we have an overview document that is very high level and is more > of an architectural document, and we also have a user guide that is very > verbose and extensive and several other guides that focus on specific aspects > of nifi in great detail. > We do not have anything that is more brief introduction to the features of > NiFi that a new user would be interested in reading through. The User Guide > is a bit longer than many users want to read and is more of a reference guide. > I'd like to see a guide targeted toward those who are not extremely familiar > with NiFi but don't want to read a 70+ page user guide. > Recommended topics: > - Downloading and starting nifi > - Building a simple flow > - What Processors are available > - Working with Attributes > - Working with Templates > - Data Provenance > - Monitoring nifi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-996) Provide listing of commonly used Attributes in UI when using Expression Language
Mark Payne created NIFI-996: --- Summary: Provide listing of commonly used Attributes in UI when using Expression Language Key: NIFI-996 URL: https://issues.apache.org/jira/browse/NIFI-996 Project: Apache NiFi Issue Type: Improvement Components: Core Framework, Core UI, Documentation & Website Reporter: Mark Payne When a user configures a Processor and uses the Expression Language, the UI provides wonderful contextual help when Ctrl+Space is pressed. However, this help is based purely on the Expression Language and the associated documentation. It would be great to see the UI provide a list of Attributes that would likely make sense at that point in the flow. There are lot of things to consider here. Attributes to suggest could be derived from the entire flow, or based on Attributes that have been seen on FlowFIles only by that Processor. Or based on the processor that introduced the data into the flow. Eventually, this could even perhaps be driven by some process outside of NiFi, by running analytics against provenance data or what have you. Something like this would be quite a bit out of scope for now, though, and would be a later addition, but it's important to keep in mind for design considerations. Additionally, Authorization needs to be considered when determining which Attributes to send. We also don't want to fill up the Java Heap, so we would need some sort of LRU cache type of mechanism. Also consider on restart of NiFi, or when a new processor is added to the graph, how do we provide context there for meaningful attributes? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-48) Startup failures should be more graceful/user friendly
[ https://issues.apache.org/jira/browse/NIFI-48?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14907278#comment-14907278 ] Mark Payne commented on NIFI-48: We've also had people reporting bad error messages when templates are different on node than NCM. The error reports that the flows are different, rather than indicating that the templates are different. From the users mailing list: Hi, I created a 2 node cluster but I was having development issues so I worked with only one until I got a custom processor working. I attempted to add the second node back in but get the error "Failed to connect node to cluster because local flow is different than cluster flow.” I deleted the flow.xml.gz file from the node I’m attempting to add, then restarted, same issue. I even copied the flow.xml.gz file from node 1 to node 2, same issue. Any ideas? Thanks, -jeff > Startup failures should be more graceful/user friendly > -- > > Key: NIFI-48 > URL: https://issues.apache.org/jira/browse/NIFI-48 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Joseph Witt > Fix For: 0.4.0 > > > Many startup issues result in massive strack traces and things which will > make users have to work much harder than necessary to see the real problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-989) Support size() operation in distributed map cache
[ https://issues.apache.org/jira/browse/NIFI-989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904968#comment-14904968 ] Mark Payne commented on NIFI-989: - [~joemeszaros] - one thing to note here is that the DistributedMapCacheClient is an interface that exists in the services-api nar. This means that there can be many different implementations of the interface. I know others do exist, outside of apache nifi. As a result, we can't really change the API of that service in a non-backward-compatible way. I.e., we can't add new methods to that interface. At least, not until version 1.0.0. One way to address this is to provide a new interface, SizableDistributedMapCacheClient that extends DistributedMapCacheClient, and then add the size() method there. Then any processor or extension point that wants to use the size() method would rely on this SizableDistributedMapCacheClient. Thanks -Mark > Support size() operation in distributed map cache > - > > Key: NIFI-989 > URL: https://issues.apache.org/jira/browse/NIFI-989 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Joe Mészáros >Priority: Minor > Labels: cache, command-line, distributed, improvement > > The distributed map cache server is a great tool for caching along with the > cache client (DistributedMapCacheClientService), but after you configure and > enable it, it is a black box. You are unable to get any information from the > cache e.g how much entries live in the cache. > The purpose of this issue to extend the cache interface and associated > implementations to support the size() operation, which returns the number of > entries in the distributed cache. > It could be a first step in a direction of a more transparent cache, where > the user can understand, what happens with the distributed cache. I mean > after the size() operation, it could be really helpful to implement a stats() > command, which s used to query the server about statistics it > maintains and other internal data (e.g. evictions, hit rates, ...). > Similar to: > - memcached : > https://docs.oracle.com/cd/E17952_01/refman-5.0-en/ha-memcached-stats-general.html > - couchbase: http://blog.couchbase.com/monitoring-couchbase-cluster > - redis: http://haydenjames.io/using-redis-stat-for-redis-statistics-tracking/ > I implemented a really simple command line tool, which can interact with the > cache server from the command line, e.g. get a cache entry and it also able > to get the size of the cache, which could be useful, when you would like to > debug cache related problems, or just get basic interaction with the cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-993) GetHTTP should allow the Expression Language to be used for the URL property
Mark Payne created NIFI-993: --- Summary: GetHTTP should allow the Expression Language to be used for the URL property Key: NIFI-993 URL: https://issues.apache.org/jira/browse/NIFI-993 Project: Apache NiFi Issue Type: Improvement Reporter: Mark Payne Priority: Minor Fix For: 0.4.0 Currently, the GetHTTP Processor does not allow the Expression Language to be used for the URL property. This was done because it is a Source Processor and doesn't have any FlowFiles' attributes to evaluate. However, a very valid use cause would be to reference functions like now(), hostname(), etc. or to reference system properties. We should allow the Expression Language to be used for this purpose. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-973) Create a Getting Started Guide
[ https://issues.apache.org/jira/browse/NIFI-973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903291#comment-14903291 ] Mark Payne commented on NIFI-973: - Joe - it is private. NiFi appears to be not in the list of projects. Maybe an iNFRA ticket?? For now i just added you, sean, and myself, and I will be happy to add anyone else who would like to review this. I'm sure there's a far better way to do this, but I have never used this application before either. > Create a Getting Started Guide > -- > > Key: NIFI-973 > URL: https://issues.apache.org/jira/browse/NIFI-973 > Project: Apache NiFi > Issue Type: Task > Components: Documentation & Website >Reporter: Mark Payne > Attachments: 0001-NIFI-973-Created-a-Getting-Started-Guide.patch > > > Currently, we have an overview document that is very high level and is more > of an architectural document, and we also have a user guide that is very > verbose and extensive and several other guides that focus on specific aspects > of nifi in great detail. > We do not have anything that is more brief introduction to the features of > NiFi that a new user would be interested in reading through. The User Guide > is a bit longer than many users want to read and is more of a reference guide. > I'd like to see a guide targeted toward those who are not extremely familiar > with NiFi but don't want to read a 70+ page user guide. > Recommended topics: > - Downloading and starting nifi > - Building a simple flow > - What Processors are available > - Working with Attributes > - Working with Templates > - Data Provenance > - Monitoring nifi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-959) PutSolrContentStream not handling expression correctly for Collection property
[ https://issues.apache.org/jira/browse/NIFI-959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903297#comment-14903297 ] Mark Payne commented on NIFI-959: - Bryan - looks good. Verified that it builds ok with contrib-check. Unit tests pass. Good catch. +1 > PutSolrContentStream not handling expression correctly for Collection > property > --- > > Key: NIFI-959 > URL: https://issues.apache.org/jira/browse/NIFI-959 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > Fix For: 0.4.0 > > Attachments: NIFI-959.patch > > > The Collection property in the Solr processors supports expression language, > and the PutSolrContentStream processor is supposed to support receiving the > Collection value from the incoming FlowFiles and be able to send to different > collections from the same processor. > The processor is evaluating the wrong value when calling > context.getProperty(..).evaluateAttributeExpressions() for the Collection, > which is causing this not to work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-973) Create a Getting Started Guide
[ https://issues.apache.org/jira/browse/NIFI-973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903269#comment-14903269 ] Mark Payne commented on NIFI-973: - [~busbey] - I have created a Review Request for this - https://reviews.apache.org/r/38643/ Thanks -Mark > Create a Getting Started Guide > -- > > Key: NIFI-973 > URL: https://issues.apache.org/jira/browse/NIFI-973 > Project: Apache NiFi > Issue Type: Task > Components: Documentation & Website >Reporter: Mark Payne > Attachments: 0001-NIFI-973-Created-a-Getting-Started-Guide.patch > > > Currently, we have an overview document that is very high level and is more > of an architectural document, and we also have a user guide that is very > verbose and extensive and several other guides that focus on specific aspects > of nifi in great detail. > We do not have anything that is more brief introduction to the features of > NiFi that a new user would be interested in reading through. The User Guide > is a bit longer than many users want to read and is more of a reference guide. > I'd like to see a guide targeted toward those who are not extremely familiar > with NiFi but don't want to read a 70+ page user guide. > Recommended topics: > - Downloading and starting nifi > - Building a simple flow > - What Processors are available > - Working with Attributes > - Working with Templates > - Data Provenance > - Monitoring nifi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-942) Create RouteText processor
[ https://issues.apache.org/jira/browse/NIFI-942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903166#comment-14903166 ] Mark Payne commented on NIFI-942: - Yup makes sense. > Create RouteText processor > -- > > Key: NIFI-942 > URL: https://issues.apache.org/jira/browse/NIFI-942 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Mark Payne >Assignee: Joseph Percivall > Fix For: 0.4.0 > > > The idea is to route individual lines of a text file to different > relationships. This allows for splitting lines based on some criteria or > filtering out specific lines, and would be a much more convenient alternative > than RouteOnContent for textual data. > A discussion for this took place on the users mailing list > (http://mail-archives.apache.org/mod_mbox/nifi-users/201509.mbox/%3CCAKpk5PxjszdX-NXMMf6Pcet4x7Y5GmrT7_jn9uyzS-h_a9TG3A%40mail.gmail.com%3E) > The way that I could see this working is to have a few different properties: > Routing Strategy: > - Route each line to matching Property Name (default) > - Route matching lines to 'matched' if all match > - Route matching lines to 'matched' if any match > - Route FlowFile to 'matched' if all lines match > - Route FlowFile to 'matched' if any line matches > A Match Strategy > - Starts With > - Ends With > - Contains > - Equals > - Matches Regular Expression > - Contains Regular Expression > And then user-defined properties that indicate what to search each line of > text for. > So to find lines that begin with the < character > You would simply add a property named "Begins with Less Than" and set the > value to : < > Then set the Match Strategy to Starts With > And Routing Strategy to "Route each line to matching Property Name" > Then, any line that begins with < will be routed to the Begins with Less Than > relationship. > This would be a simple way to pull out any particular lines of interest in a > text file. > I can see this being very useful for processing log files, CSV, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-942) Create RouteText processor
[ https://issues.apache.org/jira/browse/NIFI-942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14902593#comment-14902593 ] Mark Payne commented on NIFI-942: - [~JPercivall] - thinking a bit more about this, you may be right in your comments above, about going to 2 different properties, having a "Route Each Line Individually" vs. "Route FlowFile as a whole" -- but i would avoid the "Route FlowFile as a whole for each matched line". I'm having trouble envisioning a use case where you'd want to create many copies of an entire FlowFile just because it contains some set of matches. > Create RouteText processor > -- > > Key: NIFI-942 > URL: https://issues.apache.org/jira/browse/NIFI-942 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Mark Payne >Assignee: Joseph Percivall > Fix For: 0.4.0 > > > The idea is to route individual lines of a text file to different > relationships. This allows for splitting lines based on some criteria or > filtering out specific lines, and would be a much more convenient alternative > than RouteOnContent for textual data. > A discussion for this took place on the users mailing list > (http://mail-archives.apache.org/mod_mbox/nifi-users/201509.mbox/%3CCAKpk5PxjszdX-NXMMf6Pcet4x7Y5GmrT7_jn9uyzS-h_a9TG3A%40mail.gmail.com%3E) > The way that I could see this working is to have a few different properties: > Routing Strategy: > - Route each line to matching Property Name (default) > - Route matching lines to 'matched' if all match > - Route matching lines to 'matched' if any match > - Route FlowFile to 'matched' if all lines match > - Route FlowFile to 'matched' if any line matches > A Match Strategy > - Starts With > - Ends With > - Contains > - Equals > - Matches Regular Expression > - Contains Regular Expression > And then user-defined properties that indicate what to search each line of > text for. > So to find lines that begin with the < character > You would simply add a property named "Begins with Less Than" and set the > value to : < > Then set the Match Strategy to Starts With > And Routing Strategy to "Route each line to matching Property Name" > Then, any line that begins with < will be routed to the Begins with Less Than > relationship. > This would be a simple way to pull out any particular lines of interest in a > text file. > I can see this being very useful for processing log files, CSV, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-934) Image content viewer
[
https://issues.apache.org/jira/browse/NIFI-934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14876295#comment-14876295
]
Mark Payne commented on NIFI-934:
-
[~mcgilman]: When I attempted to apply the patch, I got an error:
{noformat}
nifi $ git apply 0001-NIFI-934.patch
error: patch failed:
nifi-nar-bundles/nifi-image-bundle/nifi-image-nar/pom.xml:32
error: nifi-nar-bundles/nifi-image-bundle/nifi-image-nar/pom.xml: patch does
not apply
{noformat}
Guessing this is related to the changing of the version from 0.3.0-SNAPSHOT to
0.3.1-SNAPSHOT?
> Image content viewer
>
>
> Key: NIFI-934
> URL: https://issues.apache.org/jira/browse/NIFI-934
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Minor
> Fix For: 0.4.0
>
> Attachments: 0001-NIFI-934.patch
>
>
> Add a new content viewer for images. This will allow us to view the images in
> the application when they are the content of a flowfile. Should handle PNG,
> JPEG, and GIF to start with.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-934) Image content viewer
[ https://issues.apache.org/jira/browse/NIFI-934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14876518#comment-14876518 ] Mark Payne commented on NIFI-934: - Much better. Thanks, Matt. I can see where Joe is coming from, but I don't know that I'd want to put a border around it - it may be confusing, leading users to believe that's part of the image. I'd prefer to show the raw image as-is. But I can't argue for it or against it very well. Kind of +0 on the concept, really. I'm +1 for this ticket, though, if you don't want to add the border. > Image content viewer > > > Key: NIFI-934 > URL: https://issues.apache.org/jira/browse/NIFI-934 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Minor > Fix For: 0.4.0 > > Attachments: 0001-NIFI-934.patch > > > Add a new content viewer for images. This will allow us to view the images in > the application when they are the content of a flowfile. Should handle PNG, > JPEG, and GIF to start with. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-973) Create a Getting Started Guide
[ https://issues.apache.org/jira/browse/NIFI-973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14847361#comment-14847361 ] Mark Payne commented on NIFI-973: - [~busbey] - as far as I know there's no reviewboard account anywhere that i have access to - anything like that in apache land that I am unaware of? > Create a Getting Started Guide > -- > > Key: NIFI-973 > URL: https://issues.apache.org/jira/browse/NIFI-973 > Project: Apache NiFi > Issue Type: Task > Components: Documentation & Website >Reporter: Mark Payne > Attachments: 0001-NIFI-973-Created-a-Getting-Started-Guide.patch > > > Currently, we have an overview document that is very high level and is more > of an architectural document, and we also have a user guide that is very > verbose and extensive and several other guides that focus on specific aspects > of nifi in great detail. > We do not have anything that is more brief introduction to the features of > NiFi that a new user would be interested in reading through. The User Guide > is a bit longer than many users want to read and is more of a reference guide. > I'd like to see a guide targeted toward those who are not extremely familiar > with NiFi but don't want to read a 70+ page user guide. > Recommended topics: > - Downloading and starting nifi > - Building a simple flow > - What Processors are available > - Working with Attributes > - Working with Templates > - Data Provenance > - Monitoring nifi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-942) Create RouteText processor
[ https://issues.apache.org/jira/browse/NIFI-942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14875731#comment-14875731 ] Mark Payne commented on NIFI-942: - Joe: finding a regex and matching a regex are two very different things. Take for example the input: The quick brown fox jumps over the lazy dog. This contains the regex: quick.*?fox But it definitely does not MATCH the regex. It would match the regex: .*quick.*?fox.* Matches Regular Expression would indicate that the entire line matches the regex exactly, whereas contains would indicate that some portion of the line matches the regex. In Java land, this would translate to: contains: pattern.matcher(line).find() matches: pattern.matcher(line).matches() While we could just supply the "Matches Regular Expression" and expect the user to do a .* before and after the rest of the regex, that is extremely inefficient in Java Pattern matching land. > Create RouteText processor > -- > > Key: NIFI-942 > URL: https://issues.apache.org/jira/browse/NIFI-942 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Mark Payne >Assignee: Joseph Percivall > Fix For: 0.4.0 > > > The idea is to route individual lines of a text file to different > relationships. This allows for splitting lines based on some criteria or > filtering out specific lines, and would be a much more convenient alternative > than RouteOnContent for textual data. > A discussion for this took place on the users mailing list > (http://mail-archives.apache.org/mod_mbox/nifi-users/201509.mbox/%3CCAKpk5PxjszdX-NXMMf6Pcet4x7Y5GmrT7_jn9uyzS-h_a9TG3A%40mail.gmail.com%3E) > The way that I could see this working is to have a few different properties: > Routing Strategy: > - Route each line to matching Property Name (default) > - Route matching lines to 'matched' if all match > - Route matching lines to 'matched' if any match > - Route FlowFile to 'matched' if all lines match > - Route FlowFile to 'matched' if any line matches > A Match Strategy > - Starts With > - Ends With > - Contains > - Equals > - Matches Regular Expression > - Contains Regular Expression > And then user-defined properties that indicate what to search each line of > text for. > So to find lines that begin with the < character > You would simply add a property named "Begins with Less Than" and set the > value to : < > Then set the Match Strategy to Starts With > And Routing Strategy to "Route each line to matching Property Name" > Then, any line that begins with < will be routed to the Begins with Less Than > relationship. > This would be a simple way to pull out any particular lines of interest in a > text file. > I can see this being very useful for processing log files, CSV, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-942) Create RouteText processor
[ https://issues.apache.org/jira/browse/NIFI-942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14875739#comment-14875739 ] Mark Payne commented on NIFI-942: - Joe - I see what you're saying. I definitely don't think we'd want to Route FlowFile as a whole for each matched line... what if we broke the last 2 options into 4 options? Route FlowFile to 'matched' if all lines match -> Route FlowFIle to 'matched' if all lines match all, Route FLowFIle to 'matched' if all lines match any, and then the last option would become: Route FLowFile to 'matched' if any line matches all, Route FlowFIle to 'matched' if any line matches any Does that seem more reasonable? > Create RouteText processor > -- > > Key: NIFI-942 > URL: https://issues.apache.org/jira/browse/NIFI-942 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Mark Payne >Assignee: Joseph Percivall > Fix For: 0.4.0 > > > The idea is to route individual lines of a text file to different > relationships. This allows for splitting lines based on some criteria or > filtering out specific lines, and would be a much more convenient alternative > than RouteOnContent for textual data. > A discussion for this took place on the users mailing list > (http://mail-archives.apache.org/mod_mbox/nifi-users/201509.mbox/%3CCAKpk5PxjszdX-NXMMf6Pcet4x7Y5GmrT7_jn9uyzS-h_a9TG3A%40mail.gmail.com%3E) > The way that I could see this working is to have a few different properties: > Routing Strategy: > - Route each line to matching Property Name (default) > - Route matching lines to 'matched' if all match > - Route matching lines to 'matched' if any match > - Route FlowFile to 'matched' if all lines match > - Route FlowFile to 'matched' if any line matches > A Match Strategy > - Starts With > - Ends With > - Contains > - Equals > - Matches Regular Expression > - Contains Regular Expression > And then user-defined properties that indicate what to search each line of > text for. > So to find lines that begin with the < character > You would simply add a property named "Begins with Less Than" and set the > value to : < > Then set the Match Strategy to Starts With > And Routing Strategy to "Route each line to matching Property Name" > Then, any line that begins with < will be routed to the Begins with Less Than > relationship. > This would be a simple way to pull out any particular lines of interest in a > text file. > I can see this being very useful for processing log files, CSV, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-732) GetKafka if stopped then started doesn't resume pulling messages
[ https://issues.apache.org/jira/browse/NIFI-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14803294#comment-14803294 ] Mark Payne commented on NIFI-732: - So far today I cannot replicate either. Let's just background this for now... if we run into it again, we will re-investigate. Otherwise, when we are closing out 0.4.0 we can mark as Cannot Reproduce. Sound agreeable to everyone? > GetKafka if stopped then started doesn't resume pulling messages > > > Key: NIFI-732 > URL: https://issues.apache.org/jira/browse/NIFI-732 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions > Environment: linux >Reporter: Joseph Witt >Assignee: Joseph Percivall > Fix For: 0.4.0 > > > A nifi user reported that they had to restart nifi to get the GetKafka > processor to resume pulling data once they had stopped the processor. Upon > restarting it showed that it was started but did not resume pulling data. > Need to attempt to reproduce and resolve. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-973) Create a Getting Started Guide
[ https://issues.apache.org/jira/browse/NIFI-973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-973: Attachment: 0001-NIFI-973-Created-a-Getting-Started-Guide.patch > Create a Getting Started Guide > -- > > Key: NIFI-973 > URL: https://issues.apache.org/jira/browse/NIFI-973 > Project: Apache NiFi > Issue Type: Task > Components: Documentation & Website >Reporter: Mark Payne > Attachments: 0001-NIFI-973-Created-a-Getting-Started-Guide.patch > > > Currently, we have an overview document that is very high level and is more > of an architectural document, and we also have a user guide that is very > verbose and extensive and several other guides that focus on specific aspects > of nifi in great detail. > We do not have anything that is more brief introduction to the features of > NiFi that a new user would be interested in reading through. The User Guide > is a bit longer than many users want to read and is more of a reference guide. > I'd like to see a guide targeted toward those who are not extremely familiar > with NiFi but don't want to read a 70+ page user guide. > Recommended topics: > - Downloading and starting nifi > - Building a simple flow > - What Processors are available > - Working with Attributes > - Working with Templates > - Data Provenance > - Monitoring nifi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-912) Support extracting metadata from Avro file headers
[ https://issues.apache.org/jira/browse/NIFI-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14768968#comment-14768968 ] Mark Payne commented on NIFI-912: - And now that I have found the unit test that tests failure (turns out i'm not very good at reading apparently), all looks great. +1 > Support extracting metadata from Avro file headers > -- > > Key: NIFI-912 > URL: https://issues.apache.org/jira/browse/NIFI-912 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > Fix For: 0.4.0 > > Attachments: NIFI-912-2.patch, NIFI-912-3.patch, NIFI-912.patch > > > Extract metadata from Avro file headers to FlowFile attributes so that > downstream processors can make decisions, such as merging together records of > compatible schemas (i.e. the correlation attribute). > Information to extract: > - Schema definition (full, not fp) > - Schema fingerprint > - Schema root record name (if schema is a record) > - Key/value metadata, like compression codec -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-919) Support Splitting Avro Files
[ https://issues.apache.org/jira/browse/NIFI-919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14768966#comment-14768966 ] Mark Payne commented on NIFI-919: - D'oh! I looked for that unit test but didn't see it. I guess because it was the last unit test before you got into private methods. In that case, I apologize and offer you a +1 :) > Support Splitting Avro Files > > > Key: NIFI-919 > URL: https://issues.apache.org/jira/browse/NIFI-919 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > Fix For: 0.4.0 > > Attachments: NIFI-919-2.patch, NIFI-919.patch > > > Provide a processor that splits an Avro file into multiple smaller files. > Would be nice to have a configurable batch size so a user could produce > single record files and also multi-record files of smaller size than the > original. Also consider making the output format configurable, data file vs > bare record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-912) Support extracting metadata from Avro file headers
[ https://issues.apache.org/jira/browse/NIFI-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14768919#comment-14768919 ] Mark Payne commented on NIFI-912: - [~bende] - doing a code review i'm noticing that you're using validators along with Allowable Values. Not necessarily wrong, but really no need, so I generally avoid it to make sure there's no confusion. Similarly to NIFI-919, I want to make sure that the Processor behaves as expected when it receive a non-avro file. Otherwise, I'll reiterate the same comments again: code looks great, great unit test coverage. > Support extracting metadata from Avro file headers > -- > > Key: NIFI-912 > URL: https://issues.apache.org/jira/browse/NIFI-912 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > Fix For: 0.4.0 > > Attachments: NIFI-912-2.patch, NIFI-912-3.patch, NIFI-912.patch > > > Extract metadata from Avro file headers to FlowFile attributes so that > downstream processors can make decisions, such as merging together records of > compatible schemas (i.e. the correlation attribute). > Information to extract: > - Schema definition (full, not fp) > - Schema fingerprint > - Schema root record name (if schema is a record) > - Key/value metadata, like compression codec -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-631) Create ListFile and FetchFile processors
[ https://issues.apache.org/jira/browse/NIFI-631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14746088#comment-14746088 ] Mark Payne commented on NIFI-631: - [~jskora] - just wanted to touch base with you on this. Is this something you're still actively working? > Create ListFile and FetchFile processors > > > Key: NIFI-631 > URL: https://issues.apache.org/jira/browse/NIFI-631 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mark Payne > > This pair of Processors will provide several benefits over the existing > GetFile processor: > 1. Currently, GetFile will continually pull the same files if the "Keep > Source File" property is set to true. There is no way to pull the file and > leave it in the directory without continually pulling the same file. We could > implement state here, but it would either be a huge amount of state to > remember everything pulled or it would have to always pull the oldest file > first so that we can maintain just the Last Modified Date of the last file > pulled plus all files with the same Last Modified Date that have already been > pulled. > 2. If pulling from a network attached storage such as NFS, this would allow a > single processor to run ListFiles and then distribute those FlowFiles to the > cluster so that the cluster can share the work of pulling the data. > 3. There are use cases when we may want to pull a specific file (for example, > in conjunction with ProcessHttpRequest/ProcessHttpResponse) rather than just > pull all files in a directory. GetFile does not support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-883) HandleHttpRequest starts a web server in the OnScheduled method but should start it in onTrigger
[ https://issues.apache.org/jira/browse/NIFI-883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-883: Assignee: Joseph Percivall > HandleHttpRequest starts a web server in the OnScheduled method but should > start it in onTrigger > > > Key: NIFI-883 > URL: https://issues.apache.org/jira/browse/NIFI-883 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Mark Payne >Assignee: Joseph Percivall > Labels: beginner, newbie > Fix For: 0.4.0 > > > When HandleHttpRequest is scheduled, it creates an embedded jetty web server > and starts it. Unfortunately, if this is run in a clustered environment and > configured to run on Primary Node Only, all nodes still start the web server. > This is very confusing if setting the Hostname property, as other nodes will > complain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-732) GetKafka if stopped then started doesn't resume pulling messages
[ https://issues.apache.org/jira/browse/NIFI-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14746165#comment-14746165 ] Mark Payne commented on NIFI-732: - It turns out that I was able to duplicate this when running against a remote Kafka broker but running on localhost i could not replicate for some reason. Not sure if it's the fact that it was on localhost or if the configuration was just different, but I've definitely been able to replicate now. > GetKafka if stopped then started doesn't resume pulling messages > > > Key: NIFI-732 > URL: https://issues.apache.org/jira/browse/NIFI-732 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions > Environment: linux >Reporter: Joseph Witt >Assignee: Joseph Percivall > Fix For: 0.4.0 > > > A nifi user reported that they had to restart nifi to get the GetKafka > processor to resume pulling data once they had stopped the processor. Upon > restarting it showed that it was started but did not resume pulling data. > Need to attempt to reproduce and resolve. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-949) Allow configuration of cleanup resources
[ https://issues.apache.org/jira/browse/NIFI-949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744044#comment-14744044 ] Mark Payne commented on NIFI-949: - I will be interested to see if this is still a problem after 0.3.0. We made some really big improvements in smarts behind the Content Repository so that it generates far fewer files on disk. This also translates to far less cleanup needed. There were also performance improvements in how the Provenance Repository ages off old data, whereby it avoids ever even updating the Lucene Index (which was about 98% of the time required for cleanup). In order to see the difference, though, you'd have to run 0.3.0 for long enough for any data that you may have archived in the Content Repo to be aged off. After that, though, please let us know if you are still having problems with this. Thanks -Mark > Allow configuration of cleanup resources > > > Key: NIFI-949 > URL: https://issues.apache.org/jira/browse/NIFI-949 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Brandon DeVries >Priority: Minor > > Allow the allocation of resource to the cleanup of the content and provenance > repositories to be configurable. There are times at which the resources > spent performing cleanup activities can degrade the operation of the flow > sufficiently to cause problems. Sometimes this requires removing the > offending node from the cluster to allow it to clean up without taking on new > data (and thus backing up). Making the cleanup resources configurable could > prevent such a situation by allowing them to be reduced under times of heavy > load to keep the node useable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-817) Create Processors to interact with HBase
[ https://issues.apache.org/jira/browse/NIFI-817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14742665#comment-14742665 ] Mark Payne commented on NIFI-817: - Alan, GetHBase is indeed intended for incremental updates. As-is, it is not "cluster friendly" and wouldhave to be scheduled for PrimaryNode Only if run in a cluster. It is not ready for prime-time yet, though,which is why we decided to push it out to a later release instead of pulling it in to 0.3.0. It does persist its state so thatif NiFi is restarted, it will continue where it left off. However, it really should be updated to save its state across thecluster so that other nodes can either pull in parallel or at least pick up where the previous node left off.I would say that the current state is a proof-of-concept, more or less, and in 0.4.0 we should have it ready to go. I do like your idea of allowing the user to choose which columns to pull. PutHBaseCell does exactly as you described. The contents of the FlowFile are pushed into exactly one HBase Cell.We called it PutHBaseCell rather than just PutHBase because it is very reasonable to expect that at some point we mayneed to build another processor that will put some sort of structured data (json, xml, avro, etc.) to HBase, rather thanupdating a single cell. We have been looking at building processors for some other NoSQL databases, as well, and will likely be facingsimilar challenges with those. If you have thoughts on design, please feel free to share, as we want these to be veryuseful and easy-to-use. Thanks-Mark > Create Processors to interact with HBase > > > Key: NIFI-817 > URL: https://issues.apache.org/jira/browse/NIFI-817 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.4.0 > > Attachments: > 0001-NIFI-817-Initial-implementation-of-HBase-processors.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-818) Create Processors for interacting with Accumulo
[ https://issues.apache.org/jira/browse/NIFI-818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14742742#comment-14742742 ] Mark Payne commented on NIFI-818: - [~pinkdevelops] - There is a PutAccumuloCell processor that I have written. It puts the entire content of a FlowFile to a single cell in Accumulo. I imagine there will be other use cases for putting structured data to Accumulo, as well. How does the PutAccumulo processor that you wrote work? I.e., does it put content to a single cell or does it expect some sort of structured data, etc.? If it is not putting to a single cell, then both may well be valuable. I have not written anything to perform a Get operation, so if you are willing to contribute this stuff back, it would be greatly appreciated! :) > Create Processors for interacting with Accumulo > --- > > Key: NIFI-818 > URL: https://issues.apache.org/jira/browse/NIFI-818 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.4.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-948) NiFi doesn't provide any mechanism for alerting users that the wrapper restarted it
[ https://issues.apache.org/jira/browse/NIFI-948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741471#comment-14741471 ] Mark Payne commented on NIFI-948: - I also uploaded another patch that is different from the previous one only in that it adds documentation to the administrators-guide. > NiFi doesn't provide any mechanism for alerting users that the wrapper > restarted it > --- > > Key: NIFI-948 > URL: https://issues.apache.org/jira/browse/NIFI-948 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.3.0 > > Attachments: > 0001-NIFI-948-Provide-capability-to-register-notification.patch, > 0002-NIFI-948-Provide-capability-to-register-notification.patch > > > Testing various edge conditions with out of disk space, out of memory errors, > etc., I'm often causing nifi to die or to fail to start (intentionally). But > the wrapper never notifies anyone that anything has happened. It should > provide a mechanism for alerting interested parties when it restarts nifi. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-948) NiFi doesn't provide any mechanism for alerting users that the wrapper restarted it
[ https://issues.apache.org/jira/browse/NIFI-948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-948: Attachment: 0001-NIFI-948-Provide-capability-to-register-notification.patch > NiFi doesn't provide any mechanism for alerting users that the wrapper > restarted it > --- > > Key: NIFI-948 > URL: https://issues.apache.org/jira/browse/NIFI-948 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.3.0 > > Attachments: > 0001-NIFI-948-Provide-capability-to-register-notification.patch > > > Testing various edge conditions with out of disk space, out of memory errors, > etc., I'm often causing nifi to die or to fail to start (intentionally). But > the wrapper never notifies anyone that anything has happened. It should > provide a mechanism for alerting interested parties when it restarts nifi. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-948) NiFi doesn't provide any mechanism for alerting users that the wrapper restarted it
[ https://issues.apache.org/jira/browse/NIFI-948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-948: Fix Version/s: 0.3.0 > NiFi doesn't provide any mechanism for alerting users that the wrapper > restarted it > --- > > Key: NIFI-948 > URL: https://issues.apache.org/jira/browse/NIFI-948 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.3.0 > > Attachments: > 0001-NIFI-948-Provide-capability-to-register-notification.patch > > > Testing various edge conditions with out of disk space, out of memory errors, > etc., I'm often causing nifi to die or to fail to start (intentionally). But > the wrapper never notifies anyone that anything has happened. It should > provide a mechanism for alerting interested parties when it restarts nifi. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-948) NiFi doesn't provide any mechanism for alerting users that the wrapper restarted it
[ https://issues.apache.org/jira/browse/NIFI-948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-948: Attachment: 0001-NIFI-948-Provide-capability-to-register-notification.patch > NiFi doesn't provide any mechanism for alerting users that the wrapper > restarted it > --- > > Key: NIFI-948 > URL: https://issues.apache.org/jira/browse/NIFI-948 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.3.0 > > Attachments: > 0001-NIFI-948-Provide-capability-to-register-notification.patch > > > Testing various edge conditions with out of disk space, out of memory errors, > etc., I'm often causing nifi to die or to fail to start (intentionally). But > the wrapper never notifies anyone that anything has happened. It should > provide a mechanism for alerting interested parties when it restarts nifi. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-936) Provenance Repository can become corrupt if it runs out of disk space, causing NiFi not to restart
[
https://issues.apache.org/jira/browse/NIFI-936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740969#comment-14740969
]
Mark Payne commented on NIFI-936:
-
[~bende] please see if you can reproduce the first and share more details.
When this happens, there is a good chance that you will not be able to start
NiFi, depending on how much disk space is available. You'll probably need about
2 KB of space available to start... so that probably explains why you
successfully started the 2nd time but not the third.
The goal is that you should have no problem starting up after freeing up disk
space and restarting.
> Provenance Repository can become corrupt if it runs out of disk space,
> causing NiFi not to restart
> --
>
> Key: NIFI-936
> URL: https://issues.apache.org/jira/browse/NIFI-936
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 0.3.0
>
> Attachments:
> 0001-NIFI-936-Ensure-that-if-we-fail-to-write-a-record-to.patch
>
>
> I caused the Provenance Repository to run out of disk space. When I attempted
> to restart, I got a huge stack trace that ended with the following:
> {noformat}
> Caused by: org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name 'flowController': FactoryBean threw exception on
> object creation; nested exception is java.lang.OutOfMemoryError: Java heap
> space
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
> ~[na:na]
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1517)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:314)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
> ~[na:na]
> at
> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
> ~[na:na]
> ... 123 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.nifi.provenance.StandardRecordReader.readLongString(StandardRecordReader.java:427)
> ~[na:na]
> at
> org.apache.nifi.provenance.StandardRecordReader.readAttributes(StandardRecordReader.java:372)
> ~[na:na]
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:314)
> ~[na:na]
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1646)
> ~[na:na]
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository.recoverJournalFiles(PersistentProvenanceRepository.java:1357)
> ~[na:na]
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository.recover(PersistentProvenanceRepository.java:577)
> ~[na:na]
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository.initialize(PersistentProvenanceRepository.java:219)
> ~[na:na]
> at
> org.apache.nifi.controller.FlowController.(FlowController.java:406)
> ~[na:na]
> at
> org.apache.nifi.controller.FlowController.createStandaloneInstance(FlowController.java:349)
> ~[na:na]
> at
> org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:63)
> ~[na:na]
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> ~[na:na]
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1517)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:314)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293)
> ~[na:na]
>
[jira] [Updated] (NIFI-938) Appending to FlowFiles Errors Finding Content - GetKafka
[
https://issues.apache.org/jira/browse/NIFI-938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-938:
Attachment: 0001-NIFI-938-If-ResourceClaim-is-removed-while-a-process.patch
> Appending to FlowFiles Errors Finding Content - GetKafka
>
>
> Key: NIFI-938
> URL: https://issues.apache.org/jira/browse/NIFI-938
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: Brian Ghigiarelli
>Priority: Critical
> Fix For: 0.3.0
>
> Attachments:
> 0001-NIFI-938-If-ResourceClaim-is-removed-while-a-process.patch
>
>
> With the latest 0.3.0-SNAPSHOT on the master branch, GetKafka throws errors
> only when the Batch Size is greater than 1. NiFi is unable to find the
> existing FlowFile in order to append new content to it.
> {noformat}
> 2015-09-08 15:36:36,643 ERROR [Timer-Driven Process Thread-3]
> o.apache.nifi.processors.kafka.GetKafka
> GetKafka[id=6eec6998-c271-4de3-b092-42469d9b494c] Failed to receive FlowFile
> from Kafka due to
> org.apache.nifi.processor.exception.MissingFlowFileException: Unable to find
> content for FlowFile:
> org.apache.nifi.processor.exception.MissingFlowFileException: Unable to find
> content for FlowFile
> 2015-09-08 15:36:36,644 ERROR [Timer-Driven Process Thread-3]
> o.apache.nifi.processors.kafka.GetKafka
> org.apache.nifi.processor.exception.MissingFlowFileException: Unable to find
> content for FlowFile
> at
> org.apache.nifi.controller.repository.StandardProcessSession.handleContentNotFound(StandardProcessSession.java:2333)
> ~[na:na]
> at
> org.apache.nifi.controller.repository.StandardProcessSession.append(StandardProcessSession.java:2028)
> ~[na:na]
> at
> org.apache.nifi.processors.kafka.GetKafka.onTrigger(GetKafka.java:313)
> ~[nifi-kafka-processors-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> [nifi-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1077)
> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:127)
> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:49)
> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119)
> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_45]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_45]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_45]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> Caused by: org.apache.nifi.controller.repository.ContentNotFoundException:
> Could not find content for
> org.apache.nifi.controller.repository.claim.StandardContentClaim@90a96e21
> at
> org.apache.nifi.controller.repository.FileSystemRepository.getPath(FileSystemRepository.java:490)
> ~[na:na]
> at
> org.apache.nifi.controller.repository.FileSystemRepository.read(FileSystemRepository.java:795)
> ~[na:na]
> at
> org.apache.nifi.controller.repository.StandardProcessSession.append(StandardProcessSession.java:1988)
> ~[na:na]
> ... 13 common frames omitted
> 2015-09-08 15:36:36,644 ERROR [Timer-Driven Process Thread-3]
> o.apache.nifi.processors.kafka.GetKafka
> GetKafka[id=6eec6998-c271-4de3-b092-42469d9b494c]
> GetKafka[id=6eec6998-c271-4de3-b092-42469d9b494c] failed to process due to
> org.apache.nifi.processor.exception.FlowFileHandlingException:
> StandardFlowFileRecord[uuid=5ae54a81-1db5-4106-b4f4-dce41771a6b4,claim=org.apache.nifi.controller.repository.claim.StandardContentClaim@90a96e21,offset=0,name=56655239492779,size=3644]
> is not known in this session (StandardProcessSession[id=175]); rolling back
> session: org.apache.nifi.processor.exception.FlowFileHandlingException:
>
[jira] [Commented] (NIFI-938) Appending to FlowFiles Errors Finding Content - GetKafka
[
https://issues.apache.org/jira/browse/NIFI-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741002#comment-14741002
]
Mark Payne commented on NIFI-938:
-
Although the ticket for NIFI-939 did resolve the issue described here, it left
a bit of an addressed problem, which was that the Content Repository did allow
a Content Claim to be deleted even when it was still in use. The fix in
NIFI-939 made sure that the finished using the claim before deleting it. The
patch here ensures that even if we do attempt to delete the Content Claim while
it is in use, the Content Repo wont actually delete the data. Both fixes are
important and both should address this issue.
Thanks
-Mark
> Appending to FlowFiles Errors Finding Content - GetKafka
>
>
> Key: NIFI-938
> URL: https://issues.apache.org/jira/browse/NIFI-938
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: Brian Ghigiarelli
>Priority: Critical
> Fix For: 0.3.0
>
> Attachments:
> 0001-NIFI-938-If-ResourceClaim-is-removed-while-a-process.patch
>
>
> With the latest 0.3.0-SNAPSHOT on the master branch, GetKafka throws errors
> only when the Batch Size is greater than 1. NiFi is unable to find the
> existing FlowFile in order to append new content to it.
> {noformat}
> 2015-09-08 15:36:36,643 ERROR [Timer-Driven Process Thread-3]
> o.apache.nifi.processors.kafka.GetKafka
> GetKafka[id=6eec6998-c271-4de3-b092-42469d9b494c] Failed to receive FlowFile
> from Kafka due to
> org.apache.nifi.processor.exception.MissingFlowFileException: Unable to find
> content for FlowFile:
> org.apache.nifi.processor.exception.MissingFlowFileException: Unable to find
> content for FlowFile
> 2015-09-08 15:36:36,644 ERROR [Timer-Driven Process Thread-3]
> o.apache.nifi.processors.kafka.GetKafka
> org.apache.nifi.processor.exception.MissingFlowFileException: Unable to find
> content for FlowFile
> at
> org.apache.nifi.controller.repository.StandardProcessSession.handleContentNotFound(StandardProcessSession.java:2333)
> ~[na:na]
> at
> org.apache.nifi.controller.repository.StandardProcessSession.append(StandardProcessSession.java:2028)
> ~[na:na]
> at
> org.apache.nifi.processors.kafka.GetKafka.onTrigger(GetKafka.java:313)
> ~[nifi-kafka-processors-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> [nifi-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1077)
> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:127)
> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:49)
> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119)
> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_45]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_45]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_45]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> Caused by: org.apache.nifi.controller.repository.ContentNotFoundException:
> Could not find content for
> org.apache.nifi.controller.repository.claim.StandardContentClaim@90a96e21
> at
> org.apache.nifi.controller.repository.FileSystemRepository.getPath(FileSystemRepository.java:490)
> ~[na:na]
> at
> org.apache.nifi.controller.repository.FileSystemRepository.read(FileSystemRepository.java:795)
> ~[na:na]
> at
> org.apache.nifi.controller.repository.StandardProcessSession.append(StandardProcessSession.java:1988)
> ~[na:na]
> ... 13 common frames omitted
> 2015-09-08 15:36:36,644 ERROR [Timer-Driven Process Thread-3]
> o.apache.nifi.processors.kafka.GetKafka
> GetKafka[id=6eec6998-c271-4de3-b092-42469d9b494c]
> GetKafka[id=6eec6998-c271-4de3-b092-42469d9b494c]
[jira] [Created] (NIFI-948) NiFi doesn't provide any mechanism for alerting users that the wrapper restarted it
Mark Payne created NIFI-948: --- Summary: NiFi doesn't provide any mechanism for alerting users that the wrapper restarted it Key: NIFI-948 URL: https://issues.apache.org/jira/browse/NIFI-948 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Testing various edge conditions with out of disk space, out of memory errors, etc., I'm often causing nifi to die or to fail to start (intentionally). But the wrapper never notifies anyone that anything has happened. It should provide a mechanism for alerting interested parties when it restarts nifi. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-936) Provenance Repository can become corrupt if it runs out of disk space, causing NiFi not to restart
[
https://issues.apache.org/jira/browse/NIFI-936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-936:
Attachment: 0001-NIFI-936-Ensure-that-if-we-fail-to-write-a-record-to.patch
> Provenance Repository can become corrupt if it runs out of disk space,
> causing NiFi not to restart
> --
>
> Key: NIFI-936
> URL: https://issues.apache.org/jira/browse/NIFI-936
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
> Fix For: 0.3.0
>
> Attachments:
> 0001-NIFI-936-Ensure-that-if-we-fail-to-write-a-record-to.patch
>
>
> I caused the Provenance Repository to run out of disk space. When I attempted
> to restart, I got a huge stack trace that ended with the following:
> {noformat}
> Caused by: org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name 'flowController': FactoryBean threw exception on
> object creation; nested exception is java.lang.OutOfMemoryError: Java heap
> space
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
> ~[na:na]
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1517)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:314)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
> ~[na:na]
> at
> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
> ~[na:na]
> ... 123 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.nifi.provenance.StandardRecordReader.readLongString(StandardRecordReader.java:427)
> ~[na:na]
> at
> org.apache.nifi.provenance.StandardRecordReader.readAttributes(StandardRecordReader.java:372)
> ~[na:na]
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:314)
> ~[na:na]
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1646)
> ~[na:na]
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository.recoverJournalFiles(PersistentProvenanceRepository.java:1357)
> ~[na:na]
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository.recover(PersistentProvenanceRepository.java:577)
> ~[na:na]
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository.initialize(PersistentProvenanceRepository.java:219)
> ~[na:na]
> at
> org.apache.nifi.controller.FlowController.(FlowController.java:406)
> ~[na:na]
> at
> org.apache.nifi.controller.FlowController.createStandaloneInstance(FlowController.java:349)
> ~[na:na]
> at
> org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:63)
> ~[na:na]
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> ~[na:na]
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1517)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:314)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
> ~[na:na]
> at
> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
> ~[na:na]
> at
> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
> ~[na:na]
> at
>
[jira] [Commented] (NIFI-938) Appending to FlowFiles Errors Finding Content - GetKafka
[
https://issues.apache.org/jira/browse/NIFI-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737386#comment-14737386
]
Mark Payne commented on NIFI-938:
-
So I was just able to replicate this issue on my Macbook Pro. I cannot
replicate running on Windows. I will try debugging from there and see if I can
tell what's going on, whether this is the same as NIFI-939 or not.
> Appending to FlowFiles Errors Finding Content - GetKafka
>
>
> Key: NIFI-938
> URL: https://issues.apache.org/jira/browse/NIFI-938
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: Brian Ghigiarelli
>Priority: Critical
> Fix For: 0.3.0
>
>
> With the latest 0.3.0-SNAPSHOT on the master branch, GetKafka throws errors
> only when the Batch Size is greater than 1. NiFi is unable to find the
> existing FlowFile in order to append new content to it.
> {noformat}
> 2015-09-08 15:36:36,643 ERROR [Timer-Driven Process Thread-3]
> o.apache.nifi.processors.kafka.GetKafka
> GetKafka[id=6eec6998-c271-4de3-b092-42469d9b494c] Failed to receive FlowFile
> from Kafka due to
> org.apache.nifi.processor.exception.MissingFlowFileException: Unable to find
> content for FlowFile:
> org.apache.nifi.processor.exception.MissingFlowFileException: Unable to find
> content for FlowFile
> 2015-09-08 15:36:36,644 ERROR [Timer-Driven Process Thread-3]
> o.apache.nifi.processors.kafka.GetKafka
> org.apache.nifi.processor.exception.MissingFlowFileException: Unable to find
> content for FlowFile
> at
> org.apache.nifi.controller.repository.StandardProcessSession.handleContentNotFound(StandardProcessSession.java:2333)
> ~[na:na]
> at
> org.apache.nifi.controller.repository.StandardProcessSession.append(StandardProcessSession.java:2028)
> ~[na:na]
> at
> org.apache.nifi.processors.kafka.GetKafka.onTrigger(GetKafka.java:313)
> ~[nifi-kafka-processors-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> [nifi-api-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1077)
> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:127)
> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:49)
> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119)
> [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_45]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_45]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_45]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> Caused by: org.apache.nifi.controller.repository.ContentNotFoundException:
> Could not find content for
> org.apache.nifi.controller.repository.claim.StandardContentClaim@90a96e21
> at
> org.apache.nifi.controller.repository.FileSystemRepository.getPath(FileSystemRepository.java:490)
> ~[na:na]
> at
> org.apache.nifi.controller.repository.FileSystemRepository.read(FileSystemRepository.java:795)
> ~[na:na]
> at
> org.apache.nifi.controller.repository.StandardProcessSession.append(StandardProcessSession.java:1988)
> ~[na:na]
> ... 13 common frames omitted
> 2015-09-08 15:36:36,644 ERROR [Timer-Driven Process Thread-3]
> o.apache.nifi.processors.kafka.GetKafka
> GetKafka[id=6eec6998-c271-4de3-b092-42469d9b494c]
> GetKafka[id=6eec6998-c271-4de3-b092-42469d9b494c] failed to process due to
> org.apache.nifi.processor.exception.FlowFileHandlingException:
> StandardFlowFileRecord[uuid=5ae54a81-1db5-4106-b4f4-dce41771a6b4,claim=org.apache.nifi.controller.repository.claim.StandardContentClaim@90a96e21,offset=0,name=56655239492779,size=3644]
> is not known in this session (StandardProcessSession[id=175]); rolling back
> session:
[jira] [Commented] (NIFI-939) GetKafka shows number of bytes written far greater than number of bytes sent out
[ https://issues.apache.org/jira/browse/NIFI-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737351#comment-14737351 ] Mark Payne commented on NIFI-939: - [~mcgilman] - to be clear, I do not think these tickets are addressing the same issue. I believe the fix for this ticket is in fact masking the issue present for NIFI-938. I have not been able to duplicate NIFI-938, though. > GetKafka shows number of bytes written far greater than number of bytes sent > out > > > Key: NIFI-939 > URL: https://issues.apache.org/jira/browse/NIFI-939 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Extensions >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.3.0 > > Attachments: > 0001-NIFI-939-Fixed-bug-in-equals-and-hashCode-methods-of.patch > > > I have a GetKafka processor. If I set the batch size to a number greater than > 1, I see that the number of bytes written is far greater than the number of > bytes transferred out. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-939) GetKafka shows number of bytes written far greater than number of bytes sent out
[ https://issues.apache.org/jira/browse/NIFI-939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-939: Attachment: 0001-NIFI-939-Fixed-bug-in-equals-and-hashCode-methods-of.patch > GetKafka shows number of bytes written far greater than number of bytes sent > out > > > Key: NIFI-939 > URL: https://issues.apache.org/jira/browse/NIFI-939 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Extensions >Reporter: Mark Payne > Fix For: 0.3.0 > > Attachments: > 0001-NIFI-939-Fixed-bug-in-equals-and-hashCode-methods-of.patch > > > I have a GetKafka processor. If I set the batch size to a number greater than > 1, I see that the number of bytes written is far greater than the number of > bytes transferred out. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-939) GetKafka shows number of bytes written far greater than number of bytes sent out
Mark Payne created NIFI-939: --- Summary: GetKafka shows number of bytes written far greater than number of bytes sent out Key: NIFI-939 URL: https://issues.apache.org/jira/browse/NIFI-939 Project: Apache NiFi Issue Type: Bug Components: Core Framework, Extensions Reporter: Mark Payne Fix For: 0.3.0 I have a GetKafka processor. If I set the batch size to a number greater than 1, I see that the number of bytes written is far greater than the number of bytes transferred out. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-936) Provenance Repository can become corrupt if it runs out of disk space, causing NiFi not to restart
Mark Payne created NIFI-936:
---
Summary: Provenance Repository can become corrupt if it runs out
of disk space, causing NiFi not to restart
Key: NIFI-936
URL: https://issues.apache.org/jira/browse/NIFI-936
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Reporter: Mark Payne
Fix For: 0.3.0
I caused the Provenance Repository to run out of disk space. When I attempted
to restart, I got a huge stack trace that ended with the following:
{noformat}
Caused by: org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'flowController': FactoryBean threw exception on object
creation; nested exception is java.lang.OutOfMemoryError: Java heap space
at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
~[na:na]
at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
~[na:na]
at
org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1517)
~[na:na]
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:314)
~[na:na]
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
~[na:na]
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293)
~[na:na]
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
~[na:na]
at
org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
~[na:na]
... 123 common frames omitted
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.nifi.provenance.StandardRecordReader.readLongString(StandardRecordReader.java:427)
~[na:na]
at
org.apache.nifi.provenance.StandardRecordReader.readAttributes(StandardRecordReader.java:372)
~[na:na]
at
org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:314)
~[na:na]
at
org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1646)
~[na:na]
at
org.apache.nifi.provenance.PersistentProvenanceRepository.recoverJournalFiles(PersistentProvenanceRepository.java:1357)
~[na:na]
at
org.apache.nifi.provenance.PersistentProvenanceRepository.recover(PersistentProvenanceRepository.java:577)
~[na:na]
at
org.apache.nifi.provenance.PersistentProvenanceRepository.initialize(PersistentProvenanceRepository.java:219)
~[na:na]
at
org.apache.nifi.controller.FlowController.(FlowController.java:406)
~[na:na]
at
org.apache.nifi.controller.FlowController.createStandaloneInstance(FlowController.java:349)
~[na:na]
at
org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:63)
~[na:na]
at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
~[na:na]
at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
~[na:na]
at
org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1517)
~[na:na]
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:314)
~[na:na]
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
~[na:na]
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293)
~[na:na]
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
~[na:na]
at
org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
~[na:na]
at
org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
~[na:na]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1477)
~[na:na]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1222)
~[na:na]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:537)
~[na:na]
at
[jira] [Commented] (NIFI-926) Clearing counters can cause a Node to disconnect
[
https://issues.apache.org/jira/browse/NIFI-926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14732418#comment-14732418
]
Mark Payne commented on NIFI-926:
-
Looks good. I applied the patch, verified that it builds with contrib-check.
Setup a cluster as described and reset counters without issue. Continued to
process data through the flow and saw that all works as expected. +1
> Clearing counters can cause a Node to disconnect
>
>
> Key: NIFI-926
> URL: https://issues.apache.org/jira/browse/NIFI-926
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Matt Gilman
>Assignee: Matt Gilman
> Fix For: 0.3.0
>
> Attachments: 0001-NIFI-926.patch
>
>
> When clustered, clearing a counter can cause the a node to disconnect. The
> node disconnects due to a 404 - Resource Not Found. It appears the counters
> do not share a common identifier. This leads to the following message:
> {code}
> 2015-09-03 14:20:15,574 INFO [NiFi Web Server-24]
> o.a.n.w.a.c.ResourceNotFoundExceptionMapper
> org.apache.nifi.web.ResourceNotFoundException: Unable to find Counter with id
> 'f4cf3c66-c0e7-3439-be94-2096f7d4a78e'.. Returning Not Found response.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-447) Add "Replacement Strategy" property to ReplaceText
[ https://issues.apache.org/jira/browse/NIFI-447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14730952#comment-14730952 ] Mark Payne commented on NIFI-447: - [~JPercivall]: I would say yes, it makes sense. Good catch. > Add "Replacement Strategy" property to ReplaceText > -- > > Key: NIFI-447 > URL: https://issues.apache.org/jira/browse/NIFI-447 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne > Fix For: 0.4.0 > > > "Regular Expression" property should have displayName changed to "Search > Value" > ReplaceText should allow for a "Replacement Strategy" with values: > Prepend (Search Value is ignored; replacement value prepended to FlowFile) > Append (Search Value is ignored; replacement value appended to FlowFIle) > Regex Replace (current behavior, so this should be default) > Linewise Regex Replace (like current behavior if Evaluation Mode is set to > Line-by-Line) > Literal Replace (do not interpret the Replacement value > The Line-by-Line property should be removed; this is breaking change! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-927) NiFi Spark Receiver not serializing data appropriately
[ https://issues.apache.org/jira/browse/NIFI-927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-927: Attachment: 0001-NIFI-927-Use-a-serializable-version-of-NiFiDataPacke.patch > NiFi Spark Receiver not serializing data appropriately > -- > > Key: NIFI-927 > URL: https://issues.apache.org/jira/browse/NIFI-927 > Project: Apache NiFi > Issue Type: Bug > Components: Tools and Build >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.3.0 > > Attachments: > 0001-NIFI-927-Use-a-serializable-version-of-NiFiDataPacke.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-932) Allow ExecuteSQL to run periodically without any input FlowFile
Mark Payne created NIFI-932: --- Summary: Allow ExecuteSQL to run periodically without any input FlowFile Key: NIFI-932 URL: https://issues.apache.org/jira/browse/NIFI-932 Project: Apache NiFi Issue Type: Improvement Reporter: Mark Payne Currently, ExecuteSQL will return immediately if run without a FlowFile. Instead, it should be able to be configured to run (say once per hour) without any input data. The framework as-is does not provide this information, so the fact of whether or not incoming connections exist would have to be provided by the ProcessContext object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-810) Create Annotation that indicates that a Processor cannot be scheduled to run without an incoming connection
[ https://issues.apache.org/jira/browse/NIFI-810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731304#comment-14731304 ] Mark Payne commented on NIFI-810: - [~aldrin]: thoughts on using an annotation like "@InputRequirement" that takes an enumeration instead of @IgnoresInput and @RequestInput? Then you could use one of: @InputRequirement(INPUT_REQUIRED) @InputRequirement(INPUT_OPTIONAL) @InputRequirement(INPUT_FORBIDDEN) If the annotation is not present, it would be the same as using INPUT_OPTIONAL, which is how this behaves now. Not sure which way is cleaner: to have explicit annotations for each or to have an annotation that accepts input? > Create Annotation that indicates that a Processor cannot be scheduled to run > without an incoming connection > --- > > Key: NIFI-810 > URL: https://issues.apache.org/jira/browse/NIFI-810 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne > > Currently, if a Processor has no incoming connections but is started, it will > continually without ever accomplishing anything. We should have an > annotation, perhaps @RequiresInput, that indicates that the Processor should > not be scheduled to run unless it has an incoming connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-447) Add "Replacement Strategy" property to ReplaceText
[ https://issues.apache.org/jira/browse/NIFI-447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731240#comment-14731240 ] Mark Payne commented on NIFI-447: - I can't argue that we shouldn't. Though if we are going to allow it for each option.. perhaps that should be separated into a new property? Similar to the way that it is currently configured... Line-by-Line or Entire Text, and then Append, Prepend, Regex, Literal Replacement > Add "Replacement Strategy" property to ReplaceText > -- > > Key: NIFI-447 > URL: https://issues.apache.org/jira/browse/NIFI-447 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne > Fix For: 0.4.0 > > > "Regular Expression" property should have displayName changed to "Search > Value" > ReplaceText should allow for a "Replacement Strategy" with values: > Prepend (Search Value is ignored; replacement value prepended to FlowFile) > Append (Search Value is ignored; replacement value appended to FlowFIle) > Regex Replace (current behavior, so this should be default) > Linewise Regex Replace (like current behavior if Evaluation Mode is set to > Line-by-Line) > Literal Replace (do not interpret the Replacement value > The Line-by-Line property should be removed; this is breaking change! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-932) Allow ExecuteSQL to run periodically without any input FlowFile
[ https://issues.apache.org/jira/browse/NIFI-932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-932: Fix Version/s: 0.4.0 > Allow ExecuteSQL to run periodically without any input FlowFile > --- > > Key: NIFI-932 > URL: https://issues.apache.org/jira/browse/NIFI-932 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mark Payne > Fix For: 0.4.0 > > > Currently, ExecuteSQL will return immediately if run without a FlowFile. > Instead, it should be able to be configured to run (say once per hour) > without any input data. > The framework as-is does not provide this information, so the fact of whether > or not incoming connections exist would have to be provided by the > ProcessContext object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-932) Allow ExecuteSQL to run periodically without any input FlowFile
[ https://issues.apache.org/jira/browse/NIFI-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731209#comment-14731209 ] Mark Payne commented on NIFI-932: - I agree. This should certainly be in the documentation. > Allow ExecuteSQL to run periodically without any input FlowFile > --- > > Key: NIFI-932 > URL: https://issues.apache.org/jira/browse/NIFI-932 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mark Payne > Fix For: 0.4.0 > > > Currently, ExecuteSQL will return immediately if run without a FlowFile. > Instead, it should be able to be configured to run (say once per hour) > without any input data. > The framework as-is does not provide this information, so the fact of whether > or not incoming connections exist would have to be provided by the > ProcessContext object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-447) Add "Replacement Strategy" property to ReplaceText
[ https://issues.apache.org/jira/browse/NIFI-447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731330#comment-14731330 ] Mark Payne commented on NIFI-447: - Agreed. That's the route that I would take. > Add "Replacement Strategy" property to ReplaceText > -- > > Key: NIFI-447 > URL: https://issues.apache.org/jira/browse/NIFI-447 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne > Fix For: 0.4.0 > > > "Regular Expression" property should have displayName changed to "Search > Value" > ReplaceText should allow for a "Replacement Strategy" with values: > Prepend (Search Value is ignored; replacement value prepended to FlowFile) > Append (Search Value is ignored; replacement value appended to FlowFIle) > Regex Replace (current behavior, so this should be default) > Linewise Regex Replace (like current behavior if Evaluation Mode is set to > Line-by-Line) > Literal Replace (do not interpret the Replacement value > The Line-by-Line property should be removed; this is breaking change! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-916) NiFi stop command occasionally results in Socket Timeout exception
[
https://issues.apache.org/jira/browse/NIFI-916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731336#comment-14731336
]
Mark Payne commented on NIFI-916:
-
[~bende] - yeah this is okay. It happens sometimes during shutdown. This is
caused because a session commit is updating the FlowFile Repository and the
repo has already been shutdown. This is OK because the session will just be
rolled back, essentially, on restart.
> NiFi stop command occasionally results in Socket Timeout exception
> --
>
> Key: NIFI-916
> URL: https://issues.apache.org/jira/browse/NIFI-916
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Bryan Bende
> Fix For: 0.3.0
>
> Attachments: NIFI-916.patch
>
>
> When I issue the bin/nifi.sh stop command, running in CentOS 7 in my VM, I
> occasionally get a timeout exception. If this happens, it will continue to
> happen until I manually kill the process.
> Below is the output that I receive, as well as the results of running netstat
> to verify that NiFi is in fact listening on the port reported.
> {noformat}
> [mark@centos-7 nifi-0.3.0-SNAPSHOT]$ bin/nifi.sh stop
> Java home: /usr/java/latest
> NiFi home: /opt/nifi-0.3.0-SNAPSHOT
> Bootstrap Config File: /opt/nifi-0.3.0-SNAPSHOT/conf/bootstrap.conf
> 2015-08-31 16:05:38,454 ERROR [main] org.apache.nifi.bootstrap.Command Failed
> to send shutdown command to port 49560 due to
> java.net.SocketTimeoutException: Read timed out
> java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0_45]
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> ~[na:1.8.0_45]
> at java.net.SocketInputStream.read(SocketInputStream.java:170)
> ~[na:1.8.0_45]
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> ~[na:1.8.0_45]
> at java.net.SocketInputStream.read(SocketInputStream.java:223)
> ~[na:1.8.0_45]
> at org.apache.nifi.bootstrap.RunNiFi.stop(RunNiFi.java:536)
> [nifi-bootstrap-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:208)
> [nifi-bootstrap-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> [mark@centos-7 nifi-0.3.0-SNAPSHOT]$
> [mark@centos-7 nifi-0.3.0-SNAPSHOT]$
> [mark@centos-7 nifi-0.3.0-SNAPSHOT]$
> [mark@centos-7 nifi-0.3.0-SNAPSHOT]$ netstat -anp | grep 49560
> (Not all processes could be identified, non-owned process info
> will not be shown, you would have to be root to see it all.)
> tcp0 0 127.0.0.1:49560 0.0.0.0:* LISTEN
> 6873/java
> tcp0 0 127.0.0.1:49560 127.0.0.1:37907
> CLOSE_WAIT 6873/java
> tcp0 0 127.0.0.1:49560 127.0.0.1:37905
> CLOSE_WAIT 6873/java
> tcp6 0 0 127.0.0.1:37907 127.0.0.1:49560 FIN_WAIT2
> -
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-903) If provenance repository runs out of disk space, it will throw a NullPointerException
[
https://issues.apache.org/jira/browse/NIFI-903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-903:
Attachment: 0001-NIFI-903-Check-for-null-value-before-trying-to-close.patch
> If provenance repository runs out of disk space, it will throw a
> NullPointerException
> -
>
> Key: NIFI-903
> URL: https://issues.apache.org/jira/browse/NIFI-903
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 0.3.0
>
> Attachments:
> 0001-NIFI-903-Check-for-null-value-before-trying-to-close.patch
>
>
> I allowed my Provenance Repository to run out of disk space. When it attempts
> to roll over, I started getting the following stack trace:
> {noformat}
> 2015-08-27 16:00:04,880 ERROR [Provenance Repository Rollover Thread-2]
> o.a.n.p.PersistentProvenanceRepository
> java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.8.0_45]
> at java.io.FileOutputStream.write(FileOutputStream.java:326)
> ~[na:1.8.0_45]
> at
> org.apache.nifi.stream.io.ByteCountingOutputStream.write(ByteCountingOutputStream.java:49)
> ~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.stream.io.ByteCountingOutputStream.write(ByteCountingOutputStream.java:44)
> ~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.stream.io.NonCloseableOutputStream.write(NonCloseableOutputStream.java:34)
> ~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> java.util.zip.GZIPOutputStream.writeHeader(GZIPOutputStream.java:182)
> ~[na:1.8.0_45]
> at java.util.zip.GZIPOutputStream.(GZIPOutputStream.java:94)
> ~[na:1.8.0_45]
> at java.util.zip.GZIPOutputStream.(GZIPOutputStream.java:109)
> ~[na:1.8.0_45]
> at
> org.apache.nifi.stream.io.GZIPOutputStream.(GZIPOutputStream.java:36)
> ~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.provenance.StandardRecordWriter.resetWriteStream(StandardRecordWriter.java:126)
> ~[nifi-persistent-provenance-repository-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.provenance.StandardRecordWriter.writeHeader(StandardRecordWriter.java:93)
> ~[nifi-persistent-provenance-repository-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1563)
> ~[nifi-persistent-provenance-repository-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository$8.run(PersistentProvenanceRepository.java:1196)
> ~[nifi-persistent-provenance-repository-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_45]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_45]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_45]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> Suppressed: java.lang.NullPointerException: null
> at
> org.apache.nifi.provenance.StandardRecordWriter.close(StandardRecordWriter.java:277)
> ~[nifi-persistent-provenance-repository-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1672)
> ~[nifi-persistent-provenance-repository-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
> ... 8 common frames omitted
> {noformat}
> While the IOException indicating out of disk space is expected, the
> suppressed NullPointerException is not and needs to be addressed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-911) ReplaceText, with default configuration, will cause the replacement to be added twice
[ https://issues.apache.org/jira/browse/NIFI-911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723698#comment-14723698 ] Mark Payne commented on NIFI-911: - [~mcgilman]: Testing this out, most seems to work okay. I did run into an issue, though, where if the content has new-lines, the new default value doesn't match. Perhaps we should instead use a default value of (?s:^.*$) This will allow the new-lines to be matched as well. Thoughts? > ReplaceText, with default configuration, will cause the replacement to be > added twice > - > > Key: NIFI-911 > URL: https://issues.apache.org/jira/browse/NIFI-911 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne >Assignee: Matt Gilman > Labels: beginner, newbie > Fix For: 0.3.0 > > Attachments: 0001-NIFI-911.patch > > > Currently, the default value for the Regular Expression property is (.\*) > which is intended to match any text. However, due to the way that the regular > expression is interpreted, it first matches the entire content of the > FlowFile and then again matches 0 characters at the end of the content > (because .* will match 0 or more characters of any character). > This results in the replacement value being added to the FlowFile twice. For > example, if the replacement value is "Hello World", the output FlowFile will > contain the text "Hello WorldHello World". > We should instead change the regex pattern to (^.*$) to in order to ensure > that we do not duplicate the output -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-911) ReplaceText, with default configuration, will cause the replacement to be added twice
[ https://issues.apache.org/jira/browse/NIFI-911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723752#comment-14723752 ] Mark Payne commented on NIFI-911: - Sounds good. Thanks. > ReplaceText, with default configuration, will cause the replacement to be > added twice > - > > Key: NIFI-911 > URL: https://issues.apache.org/jira/browse/NIFI-911 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne >Assignee: Matt Gilman > Labels: beginner, newbie > Fix For: 0.3.0 > > > Currently, the default value for the Regular Expression property is (.\*) > which is intended to match any text. However, due to the way that the regular > expression is interpreted, it first matches the entire content of the > FlowFile and then again matches 0 characters at the end of the content > (because .* will match 0 or more characters of any character). > This results in the replacement value being added to the FlowFile twice. For > example, if the replacement value is "Hello World", the output FlowFile will > contain the text "Hello WorldHello World". > We should instead change the regex pattern to (^.*$) to in order to ensure > that we do not duplicate the output -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-911) ReplaceText, with default configuration, will cause the replacement to be added twice
[ https://issues.apache.org/jira/browse/NIFI-911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723865#comment-14723865 ] Mark Payne commented on NIFI-911: - [~mcgilman] - Looks good. Behaves as expected. Unit tests pass, compiles & passes contrib-check. +1 > ReplaceText, with default configuration, will cause the replacement to be > added twice > - > > Key: NIFI-911 > URL: https://issues.apache.org/jira/browse/NIFI-911 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne >Assignee: Matt Gilman > Labels: beginner, newbie > Fix For: 0.3.0 > > Attachments: 0001-NIFI-911.patch > > > Currently, the default value for the Regular Expression property is (.\*) > which is intended to match any text. However, due to the way that the regular > expression is interpreted, it first matches the entire content of the > FlowFile and then again matches 0 characters at the end of the content > (because .* will match 0 or more characters of any character). > This results in the replacement value being added to the FlowFile twice. For > example, if the replacement value is "Hello World", the output FlowFile will > contain the text "Hello WorldHello World". > We should instead change the regex pattern to (^.*$) to in order to ensure > that we do not duplicate the output -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-914) NiFi fails to startup when upgrading to newest 0.3.0-SNAPSHOT
[ https://issues.apache.org/jira/browse/NIFI-914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-914: Attachment: 0001-NIFI-914-If-no-krb5-file-set-in-nifi-properties-trea.patch > NiFi fails to startup when upgrading to newest 0.3.0-SNAPSHOT > - > > Key: NIFI-914 > URL: https://issues.apache.org/jira/browse/NIFI-914 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 0.3.0 >Reporter: Mark Payne >Priority: Critical > Labels: beginner, newbie > Fix For: 0.3.0 > > Attachments: > 0001-NIFI-914-If-no-krb5-file-set-in-nifi-properties-trea.patch > > > I built a new version of NiFi from the master branch and copied the files > from my conf/ directory from my old instance of NiFi to the new instance. > When I tried to start up, I got the following stack trace: > 2015-08-31 12:14:12,662 ERROR [main] org.apache.nifi.NiFi Failure to launch > NiFi due to java.util.ServiceConfigurationError: > org.apache.nifi.processor.Processor: Provider > org.apache.nifi.processors.hadoop.CreateHadoopSequenceFile could not be > instantiated > java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: > Provider org.apache.nifi.processors.hadoop.CreateHadoopSequenceFile could not > be instantiated > at java.util.ServiceLoader.fail(ServiceLoader.java:232) ~[na:1.8.0_45] > at java.util.ServiceLoader.access$100(ServiceLoader.java:185) > ~[na:1.8.0_45] > at > java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) > ~[na:1.8.0_45] > at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) > ~[na:1.8.0_45] > at java.util.ServiceLoader$1.next(ServiceLoader.java:480) > ~[na:1.8.0_45] > at > org.apache.nifi.nar.ExtensionManager.loadExtensions(ExtensionManager.java:109) > ~[nifi-nar-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] > at > org.apache.nifi.nar.ExtensionManager.discoverExtensions(ExtensionManager.java:90) > ~[nifi-nar-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] > at org.apache.nifi.NiFi.(NiFi.java:120) > ~[nifi-runtime-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] > at org.apache.nifi.NiFi.main(NiFi.java:227) > ~[nifi-runtime-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] > Caused by: java.lang.ExceptionInInitializerError: null > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) ~[na:1.8.0_45] > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > ~[na:1.8.0_45] > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > ~[na:1.8.0_45] > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > ~[na:1.8.0_45] > at java.lang.Class.newInstance(Class.java:442) ~[na:1.8.0_45] > at > java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) > ~[na:1.8.0_45] > ... 6 common frames omitted > Caused by: java.lang.NullPointerException: null > at > org.apache.nifi.util.NiFiProperties.getKerberosConfigurationFile(NiFiProperties.java:806) > ~[nifi-properties-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] > at > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.(AbstractHadoopProcessor.java:124) > ~[nifi-hdfs-processors-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] > ... 12 common frames omitted > NiFiProperties is assuming that the kerberos configuration file is specified > in the properties file but a common practice is to copy over the old conf/ > directory from a previous installation. In this case, NiFiProperties should > return null for the configuration file if the property is not present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-916) NiFi stop command occasionally results in Socket Timeout exception
[
https://issues.apache.org/jira/browse/NIFI-916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723977#comment-14723977
]
Mark Payne commented on NIFI-916:
-
When I issue the command with DEBUG log status set, I see the following logs on
the nifi side:
{noformat}
2015-08-31 16:18:43,335 DEBUG [Listen to Bootstrap]
org.apache.nifi.BootstrapListener Listening for Bootstrap Requests
2015-08-31 16:18:45,337 DEBUG [Listen to Bootstrap]
org.apache.nifi.BootstrapListener Listening for Bootstrap Requests
2015-08-31 16:18:46,754 DEBUG [Listen to Bootstrap]
org.apache.nifi.BootstrapListener Received connection from Bootstrap
2015-08-31 16:18:46,754 DEBUG [Listen to Bootstrap]
org.apache.nifi.BootstrapListener Listening for Bootstrap Requests
2015-08-31 16:18:48,757 DEBUG [Listen to Bootstrap]
org.apache.nifi.BootstrapListener Listening for Bootstrap Requests
2015-08-31 16:18:50,760 DEBUG [Listen to Bootstrap]
org.apache.nifi.BootstrapListener Listening for Bootstrap Requests
2015-08-31 16:18:51,851 DEBUG [Listen to Bootstrap]
org.apache.nifi.BootstrapListener Received connection from Bootstrap
2015-08-31 16:18:51,851 DEBUG [Listen to Bootstrap]
org.apache.nifi.BootstrapListener Listening for Bootstrap Requests
2015-08-31 16:18:53,854 DEBUG [Listen to Bootstrap]
org.apache.nifi.BootstrapListener Listening for Bootstrap Requests
2015-08-31 16:18:55,857 DEBUG [Listen to Bootstrap]
org.apache.nifi.BootstrapListener Listening for Bootstrap Requests
2015-08-31 16:18:57,860 DEBUG [Listen to Bootstrap]
org.apache.nifi.BootstrapListener Listening for Bootstrap Requests
2015-08-31 16:18:59,863 DEBUG [Listen to Bootstrap]
org.apache.nifi.BootstrapListener Listening for Bootstrap Requests
2015-08-31 16:19:01,865 DEBUG [Listen to Bootstrap]
org.apache.nifi.BootstrapListener Listening for Bootstrap Requests
{noformat}
and from the bootstrap side, I see:
{noformat}
[mark@centos-7 nifi-0.3.0-SNAPSHOT]$ bin/nifi.sh stop
Java home: /usr/java/latest
NiFi home: /opt/nifi-0.3.0-SNAPSHOT
Bootstrap Config File: /opt/nifi-0.3.0-SNAPSHOT/conf/bootstrap.conf
2015-08-31 16:18:46,745 DEBUG [main] org.apache.nifi.bootstrap.Command Status
File: /opt/nifi-0.3.0-SNAPSHOT/bin/nifi.pid
2015-08-31 16:18:46,749 DEBUG [main] org.apache.nifi.bootstrap.Command Status
File: /opt/nifi-0.3.0-SNAPSHOT/bin/nifi.pid
2015-08-31 16:18:46,749 DEBUG [main] org.apache.nifi.bootstrap.Command
Properties: {pid=6873, port=49560}
2015-08-31 16:18:46,749 DEBUG [main] org.apache.nifi.bootstrap.Command Port
defined in status file: 49560
2015-08-31 16:18:46,750 DEBUG [main] org.apache.nifi.bootstrap.Command Pinging
49560
2015-08-31 16:18:46,754 DEBUG [main] org.apache.nifi.bootstrap.Command Sent
PING command
2015-08-31 16:18:51,759 DEBUG [main] org.apache.nifi.bootstrap.Command PID in
status file is 6873
2015-08-31 16:18:51,844 DEBUG [main] org.apache.nifi.bootstrap.Command Process
with PID 6873 is running
2015-08-31 16:18:51,844 DEBUG [main] org.apache.nifi.bootstrap.Command Lock
File: /opt/nifi-0.3.0-SNAPSHOT/bin/nifi.lock
2015-08-31 16:18:51,845 DEBUG [main] org.apache.nifi.bootstrap.Command Status
File: /opt/nifi-0.3.0-SNAPSHOT/bin/nifi.pid
2015-08-31 16:18:51,850 DEBUG [main] org.apache.nifi.bootstrap.Command Status
File: /opt/nifi-0.3.0-SNAPSHOT/bin/nifi.pid
2015-08-31 16:18:51,851 DEBUG [main] org.apache.nifi.bootstrap.Command
Properties: {pid=6873, port=49560}
2015-08-31 16:18:51,851 DEBUG [main] org.apache.nifi.bootstrap.Command
Connecting to NiFi instance
2015-08-31 16:18:51,851 DEBUG [main] org.apache.nifi.bootstrap.Command
Established connection to NiFi instance.
2015-08-31 16:18:51,851 DEBUG [main] org.apache.nifi.bootstrap.Command Sending
SHUTDOWN Command to port 49560
2015-08-31 16:19:51,874 ERROR [main] org.apache.nifi.bootstrap.Command Failed
to send shutdown command to port 49560 due to java.net.SocketTimeoutException:
Read timed out
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0_45]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
~[na:1.8.0_45]
at java.net.SocketInputStream.read(SocketInputStream.java:170)
~[na:1.8.0_45]
at java.net.SocketInputStream.read(SocketInputStream.java:141)
~[na:1.8.0_45]
at java.net.SocketInputStream.read(SocketInputStream.java:223)
~[na:1.8.0_45]
at org.apache.nifi.bootstrap.RunNiFi.stop(RunNiFi.java:536)
[nifi-bootstrap-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:208)
[nifi-bootstrap-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
{noformat}
So it certainly appears as if NiFi is receiving the connection but not reading
from the socket. Reading from the socket is done in another thread.
> NiFi stop command occasionally results in Socket Timeout exception
> --
>
>
[jira] [Commented] (NIFI-916) NiFi stop command occasionally results in Socket Timeout exception
[
https://issues.apache.org/jira/browse/NIFI-916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723981#comment-14723981
]
Mark Payne commented on NIFI-916:
-
And finally, running a thread dump on nifi provides the following information:
{noformat}
2015-08-31 16:23:26,715 INFO [NiFi logging handler] org.apache.nifi.StdOut
"pool-1-thread-2" #63116 prio=5 os_prio=0 tid=0x7f64fc016800 nid=0x4404
waiting on condition [0x7f64a5feb000]
2015-08-31 16:23:26,715 INFO [NiFi logging handler] org.apache.nifi.StdOut
java.lang.Thread.State: WAITING (parking)
2015-08-31 16:23:26,715 INFO [NiFi logging handler] org.apache.nifi.StdOut
at sun.misc.Unsafe.park(Native Method)
2015-08-31 16:23:26,715 INFO [NiFi logging handler] org.apache.nifi.StdOut
- parking to wait for <0xe0f5dd18> (a
java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
2015-08-31 16:23:26,715 INFO [NiFi logging handler] org.apache.nifi.StdOut
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
2015-08-31 16:23:26,715 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
2015-08-31 16:23:26,715 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
2015-08-31 16:23:26,715 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
2015-08-31 16:23:26,715 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
2015-08-31 16:23:26,715 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.apache.nifi.provenance.PersistentProvenanceRepository.close(PersistentProvenanceRepository.java:633)
2015-08-31 16:23:26,715 INFO [NiFi logging handler] org.apache.nifi.StdOut
- locked <0xe0f5dc20> (a
org.apache.nifi.provenance.PersistentProvenanceRepository)
2015-08-31 16:23:26,715 INFO [NiFi logging handler] org.apache.nifi.StdOut
at org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1114)
2015-08-31 16:23:26,715 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:283)
2015-08-31 16:23:26,715 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextDestroyed(ApplicationStartupContextListener.java:132)
2015-08-31 16:23:26,715 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.eclipse.jetty.server.handler.ContextHandler.callContextDestroyed(ContextHandler.java:808)
2015-08-31 16:23:26,715 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.eclipse.jetty.servlet.ServletContextHandler.callContextDestroyed(ServletContextHandler.java:457)
2015-08-31 16:23:26,715 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.eclipse.jetty.server.handler.ContextHandler.doStop(ContextHandler.java:842)
2015-08-31 16:23:26,715 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.eclipse.jetty.servlet.ServletContextHandler.doStop(ServletContextHandler.java:215)
2015-08-31 16:23:26,715 INFO [NiFi logging handler] org.apache.nifi.StdOut
at org.eclipse.jetty.webapp.WebAppContext.doStop(WebAppContext.java:529)
2015-08-31 16:23:26,716 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
2015-08-31 16:23:26,716 INFO [NiFi logging handler] org.apache.nifi.StdOut
- locked <0xe07530a0> (a java.lang.Object)
2015-08-31 16:23:26,716 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143)
2015-08-31 16:23:26,716 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:162)
2015-08-31 16:23:26,716 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73)
2015-08-31 16:23:26,716 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
2015-08-31 16:23:26,716 INFO [NiFi logging handler] org.apache.nifi.StdOut
- locked <0xe0753708> (a java.lang.Object)
2015-08-31 16:23:26,716 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143)
2015-08-31 16:23:26,716 INFO [NiFi logging handler] org.apache.nifi.StdOut
at
[jira] [Updated] (NIFI-911) ReplaceText, with default configuration, will cause the replacement to be added twice
[ https://issues.apache.org/jira/browse/NIFI-911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-911: Description: Currently, the default value for the Regular Expression property is (.\*) which is intended to match any text. However, due to the way that the regular expression is interpreted, it first matches the entire content of the FlowFile and then again matches 0 characters at the end of the content (because .* will match 0 or more characters of any character). This results in the replacement value being added to the FlowFile twice. For example, if the replacement value is "Hello World", the output FlowFile will contain the text "Hello WorldHello World". We should instead change the regex pattern to (^.*$) to in order to ensure that we do not duplicate the output was: Currently, the default value for the Regular Expression property is (.*) which is intended to match any text. However, due to the way that the regular expression is interpreted, it first matches the entire content of the FlowFile and then again matches 0 characters at the end of the content (because .* will match 0 or more characters of any character). This results in the replacement value being added to the FlowFile twice. For example, if the replacement value is "Hello World", the output FlowFile will contain the text "Hello WorldHello World". We should instead change the regex pattern to (^.*$) to in order to ensure that we do not duplicate the output > ReplaceText, with default configuration, will cause the replacement to be > added twice > - > > Key: NIFI-911 > URL: https://issues.apache.org/jira/browse/NIFI-911 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne > Labels: beginner, newbie > Fix For: 0.3.0 > > > Currently, the default value for the Regular Expression property is (.\*) > which is intended to match any text. However, due to the way that the regular > expression is interpreted, it first matches the entire content of the > FlowFile and then again matches 0 characters at the end of the content > (because .* will match 0 or more characters of any character). > This results in the replacement value being added to the FlowFile twice. For > example, if the replacement value is "Hello World", the output FlowFile will > contain the text "Hello WorldHello World". > We should instead change the regex pattern to (^.*$) to in order to ensure > that we do not duplicate the output -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-911) ReplaceText, with default configuration, will cause the replacement to be added twice
Mark Payne created NIFI-911: --- Summary: ReplaceText, with default configuration, will cause the replacement to be added twice Key: NIFI-911 URL: https://issues.apache.org/jira/browse/NIFI-911 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Mark Payne Fix For: 0.3.0 Currently, the default value for the Regular Expression property is (.*) which is intended to match any text. However, due to the way that the regular expression is interpreted, it first matches the entire content of the FlowFile and then again matches 0 characters at the end of the content (because .* will match 0 or more characters of any character). This results in the replacement value being added to the FlowFile twice. For example, if the replacement value is "Hello World", the output FlowFile will contain the text "Hello WorldHello World". We should instead change the regex pattern to (^.*$) to in order to ensure that we do not duplicate the output -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-902) NiFi does not properly handle error conditions if writing to FlowFile repository fails
[
https://issues.apache.org/jira/browse/NIFI-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-902:
Attachment: (was:
0001-NIFI-902-Ensure-that-if-FlowFile-Repo-runs-out-of-di.patch)
NiFi does not properly handle error conditions if writing to FlowFile
repository fails
--
Key: NIFI-902
URL: https://issues.apache.org/jira/browse/NIFI-902
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Reporter: Mark Payne
Assignee: Mark Payne
Priority: Critical
Fix For: 0.3.0
Attachments:
0001-NIFI-902-Ensure-that-if-we-get-an-IOException-during.patch,
0002-NIFI-902-Ensure-that-we-close-the-underlying-file-st.patch
I ran out of disk space on my FlowFile Repository. After cleaning up some
files on the partition, I was able to continue running but when the FlowFile
Repo attempted to checkpoint, I got the following Exception in the logs:
{noformat}
2015-08-27 15:59:58,398 ERROR [pool-20-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Unable to checkpoint FlowFile
Repository due to java.io.IOException: Stream Closed
java.io.IOException: Stream Closed
at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.8.0_45]
at java.io.FileOutputStream.write(FileOutputStream.java:326)
~[na:1.8.0_45]
at
org.apache.nifi.stream.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.stream.io.BufferedOutputStream.flush(BufferedOutputStream.java:126)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
~[na:1.8.0_45]
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
~[na:1.8.0_45]
at
org.wali.MinimalLockingWriteAheadLog$Partition.rollover(MinimalLockingWriteAheadLog.java:723)
~[nifi-write-ahead-log-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.wali.MinimalLockingWriteAheadLog.checkpoint(MinimalLockingWriteAheadLog.java:512)
~[nifi-write-ahead-log-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.checkpoint(WriteAheadFlowFileRepository.java:392)
~[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.controller.repository.WriteAheadFlowFileRepository$1.run(WriteAheadFlowFileRepository.java:364)
~[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Suppressed: java.io.IOException: Stream Closed
at java.io.FileOutputStream.writeBytes(Native Method)
~[na:1.8.0_45]
at java.io.FileOutputStream.write(FileOutputStream.java:326)
~[na:1.8.0_45]
at
org.apache.nifi.stream.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.stream.io.BufferedOutputStream.flush(BufferedOutputStream.java:126)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
java.io.FilterOutputStream.close(FilterOutputStream.java:158) ~[na:1.8.0_45]
at
java.io.FilterOutputStream.close(FilterOutputStream.java:159) ~[na:1.8.0_45]
... 11 common frames omitted
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (NIFI-902) NiFi does not properly handle error conditions if writing to FlowFile repository fails
[
https://issues.apache.org/jira/browse/NIFI-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne resolved NIFI-902.
-
Resolution: Fixed
NiFi does not properly handle error conditions if writing to FlowFile
repository fails
--
Key: NIFI-902
URL: https://issues.apache.org/jira/browse/NIFI-902
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Reporter: Mark Payne
Assignee: Mark Payne
Priority: Critical
Fix For: 0.3.0
Attachments:
0001-NIFI-902-Ensure-that-if-we-get-an-IOException-during.patch,
0002-NIFI-902-Ensure-that-we-close-the-underlying-file-st.patch
I ran out of disk space on my FlowFile Repository. After cleaning up some
files on the partition, I was able to continue running but when the FlowFile
Repo attempted to checkpoint, I got the following Exception in the logs:
{noformat}
2015-08-27 15:59:58,398 ERROR [pool-20-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Unable to checkpoint FlowFile
Repository due to java.io.IOException: Stream Closed
java.io.IOException: Stream Closed
at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.8.0_45]
at java.io.FileOutputStream.write(FileOutputStream.java:326)
~[na:1.8.0_45]
at
org.apache.nifi.stream.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.stream.io.BufferedOutputStream.flush(BufferedOutputStream.java:126)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
~[na:1.8.0_45]
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
~[na:1.8.0_45]
at
org.wali.MinimalLockingWriteAheadLog$Partition.rollover(MinimalLockingWriteAheadLog.java:723)
~[nifi-write-ahead-log-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.wali.MinimalLockingWriteAheadLog.checkpoint(MinimalLockingWriteAheadLog.java:512)
~[nifi-write-ahead-log-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.checkpoint(WriteAheadFlowFileRepository.java:392)
~[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.controller.repository.WriteAheadFlowFileRepository$1.run(WriteAheadFlowFileRepository.java:364)
~[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Suppressed: java.io.IOException: Stream Closed
at java.io.FileOutputStream.writeBytes(Native Method)
~[na:1.8.0_45]
at java.io.FileOutputStream.write(FileOutputStream.java:326)
~[na:1.8.0_45]
at
org.apache.nifi.stream.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.stream.io.BufferedOutputStream.flush(BufferedOutputStream.java:126)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
java.io.FilterOutputStream.close(FilterOutputStream.java:158) ~[na:1.8.0_45]
at
java.io.FilterOutputStream.close(FilterOutputStream.java:159) ~[na:1.8.0_45]
... 11 common frames omitted
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-902) NiFi does not properly handle error conditions if writing to FlowFile repository fails
[
https://issues.apache.org/jira/browse/NIFI-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-902:
Attachment: 0002-NIFI-902-Ensure-that-we-close-the-underlying-file-st.patch
0001-NIFI-902-Ensure-that-if-we-get-an-IOException-during.patch
NiFi does not properly handle error conditions if writing to FlowFile
repository fails
--
Key: NIFI-902
URL: https://issues.apache.org/jira/browse/NIFI-902
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Reporter: Mark Payne
Assignee: Mark Payne
Priority: Critical
Fix For: 0.3.0
Attachments:
0001-NIFI-902-Ensure-that-if-FlowFile-Repo-runs-out-of-di.patch,
0001-NIFI-902-Ensure-that-if-we-get-an-IOException-during.patch,
0002-NIFI-902-Ensure-that-we-close-the-underlying-file-st.patch
I ran out of disk space on my FlowFile Repository. After cleaning up some
files on the partition, I was able to continue running but when the FlowFile
Repo attempted to checkpoint, I got the following Exception in the logs:
{noformat}
2015-08-27 15:59:58,398 ERROR [pool-20-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Unable to checkpoint FlowFile
Repository due to java.io.IOException: Stream Closed
java.io.IOException: Stream Closed
at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.8.0_45]
at java.io.FileOutputStream.write(FileOutputStream.java:326)
~[na:1.8.0_45]
at
org.apache.nifi.stream.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.stream.io.BufferedOutputStream.flush(BufferedOutputStream.java:126)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
~[na:1.8.0_45]
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
~[na:1.8.0_45]
at
org.wali.MinimalLockingWriteAheadLog$Partition.rollover(MinimalLockingWriteAheadLog.java:723)
~[nifi-write-ahead-log-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.wali.MinimalLockingWriteAheadLog.checkpoint(MinimalLockingWriteAheadLog.java:512)
~[nifi-write-ahead-log-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.checkpoint(WriteAheadFlowFileRepository.java:392)
~[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.controller.repository.WriteAheadFlowFileRepository$1.run(WriteAheadFlowFileRepository.java:364)
~[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Suppressed: java.io.IOException: Stream Closed
at java.io.FileOutputStream.writeBytes(Native Method)
~[na:1.8.0_45]
at java.io.FileOutputStream.write(FileOutputStream.java:326)
~[na:1.8.0_45]
at
org.apache.nifi.stream.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.stream.io.BufferedOutputStream.flush(BufferedOutputStream.java:126)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
java.io.FilterOutputStream.close(FilterOutputStream.java:158) ~[na:1.8.0_45]
at
java.io.FilterOutputStream.close(FilterOutputStream.java:159) ~[na:1.8.0_45]
... 11 common frames omitted
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-902) NiFi does not properly handle error conditions if writing to FlowFile repository fails
[
https://issues.apache.org/jira/browse/NIFI-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-902:
Summary: NiFi does not properly handle error conditions if writing to
FlowFile repository fails (was: If FlowFile repository runs out of disk space
and is then cleaned up, NiFi does not continue to process data)
NiFi does not properly handle error conditions if writing to FlowFile
repository fails
--
Key: NIFI-902
URL: https://issues.apache.org/jira/browse/NIFI-902
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Reporter: Mark Payne
Assignee: Mark Payne
Priority: Critical
Fix For: 0.3.0
Attachments:
0001-NIFI-902-Ensure-that-if-FlowFile-Repo-runs-out-of-di.patch
I ran out of disk space on my FlowFile Repository. After cleaning up some
files on the partition, I was able to continue running but when the FlowFile
Repo attempted to checkpoint, I got the following Exception in the logs:
{noformat}
2015-08-27 15:59:58,398 ERROR [pool-20-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Unable to checkpoint FlowFile
Repository due to java.io.IOException: Stream Closed
java.io.IOException: Stream Closed
at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.8.0_45]
at java.io.FileOutputStream.write(FileOutputStream.java:326)
~[na:1.8.0_45]
at
org.apache.nifi.stream.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.stream.io.BufferedOutputStream.flush(BufferedOutputStream.java:126)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
~[na:1.8.0_45]
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
~[na:1.8.0_45]
at
org.wali.MinimalLockingWriteAheadLog$Partition.rollover(MinimalLockingWriteAheadLog.java:723)
~[nifi-write-ahead-log-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.wali.MinimalLockingWriteAheadLog.checkpoint(MinimalLockingWriteAheadLog.java:512)
~[nifi-write-ahead-log-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.checkpoint(WriteAheadFlowFileRepository.java:392)
~[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.controller.repository.WriteAheadFlowFileRepository$1.run(WriteAheadFlowFileRepository.java:364)
~[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Suppressed: java.io.IOException: Stream Closed
at java.io.FileOutputStream.writeBytes(Native Method)
~[na:1.8.0_45]
at java.io.FileOutputStream.write(FileOutputStream.java:326)
~[na:1.8.0_45]
at
org.apache.nifi.stream.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.stream.io.BufferedOutputStream.flush(BufferedOutputStream.java:126)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
java.io.FilterOutputStream.close(FilterOutputStream.java:158) ~[na:1.8.0_45]
at
java.io.FilterOutputStream.close(FilterOutputStream.java:159) ~[na:1.8.0_45]
... 11 common frames omitted
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-902) NiFi does not properly handle error conditions if writing to FlowFile repository fails
[
https://issues.apache.org/jira/browse/NIFI-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14721150#comment-14721150
]
Mark Payne commented on NIFI-902:
-
[~bende]: While I believe that the supplied patch addressed the problem as it
was stated in the ticket, I was testing more thoroughly and ran into another
issue that occurred when the FlowFile repo ran out of disk space. The repo
ended up becoming corrupt and when I tried to restart NiFi, I got the following
stack trace:
{noformat}
2015-08-28 16:13:35,486 WARN [main] org.apache.nifi.web.server.JettyServer
Failed to start web server... shutting down.
org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller.
at
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:99)
~[na:na]
at
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800)
~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444)
~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
at
org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791)
~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294)
~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
at
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349)
~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
at
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)
~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505)
~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.Server.start(Server.java:387)
~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.Server.doStart(Server.java:354)
~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:658)
~[nifi-jetty-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at org.apache.nifi.NiFi.init(NiFi.java:137)
[nifi-runtime-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at org.apache.nifi.NiFi.main(NiFi.java:227)
[nifi-runtime-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
Caused by: java.io.IOException: Claim Existence Qualifier not found in stream;
found value: 224 after successfully restoring 111070 records
at
org.apache.nifi.controller.repository.WriteAheadFlowFileRepository$WriteAheadRecordSerde.deserializeClaim(WriteAheadFlowFileRepository.java:787)
~[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.controller.repository.WriteAheadFlowFileRepository$WriteAheadRecordSerde.deserializeEdit(WriteAheadFlowFileRepository.java:572)
~[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.controller.repository.WriteAheadFlowFileRepository$WriteAheadRecordSerde.deserializeEdit(WriteAheadFlowFileRepository.java:395)
~[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.wali.MinimalLockingWriteAheadLog$Partition.recoverNextTransaction(MinimalLockingWriteAheadLog.java:977)
[jira] [Updated] (NIFI-902) If FlowFile repository runs out of disk space and is then cleaned up, NiFi does not continue to process data
[
https://issues.apache.org/jira/browse/NIFI-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-902:
Attachment: 0001-NIFI-902-Ensure-that-if-FlowFile-Repo-runs-out-of-di.patch
If FlowFile repository runs out of disk space and is then cleaned up, NiFi
does not continue to process data
Key: NIFI-902
URL: https://issues.apache.org/jira/browse/NIFI-902
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Reporter: Mark Payne
Assignee: Mark Payne
Priority: Critical
Fix For: 0.3.0
Attachments:
0001-NIFI-902-Ensure-that-if-FlowFile-Repo-runs-out-of-di.patch
I ran out of disk space on my FlowFile Repository. After cleaning up some
files on the partition, I was able to continue running but when the FlowFile
Repo attempted to checkpoint, I got the following Exception in the logs:
{noformat}
2015-08-27 15:59:58,398 ERROR [pool-20-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Unable to checkpoint FlowFile
Repository due to java.io.IOException: Stream Closed
java.io.IOException: Stream Closed
at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.8.0_45]
at java.io.FileOutputStream.write(FileOutputStream.java:326)
~[na:1.8.0_45]
at
org.apache.nifi.stream.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.stream.io.BufferedOutputStream.flush(BufferedOutputStream.java:126)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
~[na:1.8.0_45]
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
~[na:1.8.0_45]
at
org.wali.MinimalLockingWriteAheadLog$Partition.rollover(MinimalLockingWriteAheadLog.java:723)
~[nifi-write-ahead-log-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.wali.MinimalLockingWriteAheadLog.checkpoint(MinimalLockingWriteAheadLog.java:512)
~[nifi-write-ahead-log-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.checkpoint(WriteAheadFlowFileRepository.java:392)
~[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.controller.repository.WriteAheadFlowFileRepository$1.run(WriteAheadFlowFileRepository.java:364)
~[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Suppressed: java.io.IOException: Stream Closed
at java.io.FileOutputStream.writeBytes(Native Method)
~[na:1.8.0_45]
at java.io.FileOutputStream.write(FileOutputStream.java:326)
~[na:1.8.0_45]
at
org.apache.nifi.stream.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.stream.io.BufferedOutputStream.flush(BufferedOutputStream.java:126)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
java.io.FilterOutputStream.close(FilterOutputStream.java:158) ~[na:1.8.0_45]
at
java.io.FilterOutputStream.close(FilterOutputStream.java:159) ~[na:1.8.0_45]
... 11 common frames omitted
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-902) If FlowFile repository runs out of disk space and is then cleaned up, NiFi does not continue to process data
[
https://issues.apache.org/jira/browse/NIFI-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-902:
Summary: If FlowFile repository runs out of disk space and is then cleaned
up, NiFi does not continue to process data (was: If FlowFile repository runs
out of disk space and is then cleaned up, it will not checkpoint)
If FlowFile repository runs out of disk space and is then cleaned up, NiFi
does not continue to process data
Key: NIFI-902
URL: https://issues.apache.org/jira/browse/NIFI-902
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Reporter: Mark Payne
Assignee: Mark Payne
Priority: Critical
Fix For: 0.3.0
I ran out of disk space on my FlowFile Repository. After cleaning up some
files on the partition, I was able to continue running but when the FlowFile
Repo attempted to checkpoint, I got the following Exception in the logs:
{noformat}
2015-08-27 15:59:58,398 ERROR [pool-20-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Unable to checkpoint FlowFile
Repository due to java.io.IOException: Stream Closed
java.io.IOException: Stream Closed
at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.8.0_45]
at java.io.FileOutputStream.write(FileOutputStream.java:326)
~[na:1.8.0_45]
at
org.apache.nifi.stream.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.stream.io.BufferedOutputStream.flush(BufferedOutputStream.java:126)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
~[na:1.8.0_45]
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
~[na:1.8.0_45]
at
org.wali.MinimalLockingWriteAheadLog$Partition.rollover(MinimalLockingWriteAheadLog.java:723)
~[nifi-write-ahead-log-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.wali.MinimalLockingWriteAheadLog.checkpoint(MinimalLockingWriteAheadLog.java:512)
~[nifi-write-ahead-log-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.checkpoint(WriteAheadFlowFileRepository.java:392)
~[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.controller.repository.WriteAheadFlowFileRepository$1.run(WriteAheadFlowFileRepository.java:364)
~[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Suppressed: java.io.IOException: Stream Closed
at java.io.FileOutputStream.writeBytes(Native Method)
~[na:1.8.0_45]
at java.io.FileOutputStream.write(FileOutputStream.java:326)
~[na:1.8.0_45]
at
org.apache.nifi.stream.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.stream.io.BufferedOutputStream.flush(BufferedOutputStream.java:126)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
java.io.FilterOutputStream.close(FilterOutputStream.java:158) ~[na:1.8.0_45]
at
java.io.FilterOutputStream.close(FilterOutputStream.java:159) ~[na:1.8.0_45]
... 11 common frames omitted
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-893) If FlowFile Repository runs out of disk space, NiFi stops processing until restarted
[ https://issues.apache.org/jira/browse/NIFI-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14718577#comment-14718577 ] Mark Payne commented on NIFI-893: - I created this ticket and then created NIFI-902 as well, because the symptoms that are described by the two tickets are different. However, the two tickets really are all symptoms of the same underlying issue of NiFi failing to continue processing when the FlowFile Repository runs out of disk space. NIFI-902 is a broader ticket that describes the actual issue: NiFi doesn't handle the condition properly. As a result, I want to just close this as a duplicate of NIFI-902. If FlowFile Repository runs out of disk space, NiFi stops processing until restarted Key: NIFI-893 URL: https://issues.apache.org/jira/browse/NIFI-893 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Fix For: 0.3.0 Attachments: 0001-NIFI-893-Ensure-that-if-session.commit-fails-previou.patch If the FlowFile Repository runs out of disk space, NiFi stops processing any data. Once space is recovered on the partition, NiFi never resumes processing until it is restarted. A restart appears to have recovered all data without issue and all appears to work as expected after restarting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-905) Clean up not occurring when content repository reaches max usage percentage
[
https://issues.apache.org/jira/browse/NIFI-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14720055#comment-14720055
]
Mark Payne commented on NIFI-905:
-
Actually I may have spoken too soon. Based on the thread dump that you provided
above, the thread is blocking when it is trying to archive. I'll dig a bit more
:)
Clean up not occurring when content repository reaches max usage percentage
---
Key: NIFI-905
URL: https://issues.apache.org/jira/browse/NIFI-905
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 0.3.0
Reporter: Bryan Bende
Fix For: 0.3.0
Attachments: nifi.dump
Created a 500MB partition and set the content repository to use that
partition, then created a simple Flow with GenerateFlowFile -
UpdateAttribute, using 10kb FlowFiles.
When the content repository reached approx 224MB it started logging:
Unable to write to container default to archive file size constraints;
waiting for archive cleanup
It appears that the clean up was never occurring and a thread dump shows a
blocked thread:
{code}
FileSystemRepository Workers Thread-2 daemon prio=10 tid=0x7f78c266
nid=0x2ae7 waiting for monitor entry [0x7f78a907d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.nifi.controller.repository.FileSystemRepository.archive(FileSystemRepository.java:1095)
- waiting to lock 0xe193ae00 (a
java.util.concurrent.LinkedBlockingQueue)
at
org.apache.nifi.controller.repository.FileSystemRepository.access$1200(FileSystemRepository.java:83)
at
org.apache.nifi.controller.repository.FileSystemRepository$ArchiveOrDestroyDestructableClaims.run(FileSystemRepository.java:1357)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-905) Clean up not occurring when content repository reaches max usage percentage
[
https://issues.apache.org/jira/browse/NIFI-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14720047#comment-14720047
]
Mark Payne commented on NIFI-905:
-
[~bende] I am able to duplicate this as well. Great find.
Looks like the issue is that the content repo is waiting for the archive to be
cleaned up... but there is nothing in the archive. I think we need to ensure
that if the archive is empty we trigger things to start moving again.
Clean up not occurring when content repository reaches max usage percentage
---
Key: NIFI-905
URL: https://issues.apache.org/jira/browse/NIFI-905
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 0.3.0
Reporter: Bryan Bende
Fix For: 0.3.0
Attachments: nifi.dump
Created a 500MB partition and set the content repository to use that
partition, then created a simple Flow with GenerateFlowFile -
UpdateAttribute, using 10kb FlowFiles.
When the content repository reached approx 224MB it started logging:
Unable to write to container default to archive file size constraints;
waiting for archive cleanup
It appears that the clean up was never occurring and a thread dump shows a
blocked thread:
{code}
FileSystemRepository Workers Thread-2 daemon prio=10 tid=0x7f78c266
nid=0x2ae7 waiting for monitor entry [0x7f78a907d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.nifi.controller.repository.FileSystemRepository.archive(FileSystemRepository.java:1095)
- waiting to lock 0xe193ae00 (a
java.util.concurrent.LinkedBlockingQueue)
at
org.apache.nifi.controller.repository.FileSystemRepository.access$1200(FileSystemRepository.java:83)
at
org.apache.nifi.controller.repository.FileSystemRepository$ArchiveOrDestroyDestructableClaims.run(FileSystemRepository.java:1357)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-905) Clean up not occurring when content repository reaches max usage percentage
[
https://issues.apache.org/jira/browse/NIFI-905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-905:
Attachment: 0001-NIFI-905-Ensure-that-when-archive-threshold-is-hit-a.patch
Clean up not occurring when content repository reaches max usage percentage
---
Key: NIFI-905
URL: https://issues.apache.org/jira/browse/NIFI-905
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 0.3.0
Reporter: Bryan Bende
Fix For: 0.3.0
Attachments:
0001-NIFI-905-Ensure-that-when-archive-threshold-is-hit-a.patch, nifi.dump
Created a 500MB partition and set the content repository to use that
partition, then created a simple Flow with GenerateFlowFile -
UpdateAttribute, using 10kb FlowFiles.
When the content repository reached approx 224MB it started logging:
Unable to write to container default to archive file size constraints;
waiting for archive cleanup
It appears that the clean up was never occurring and a thread dump shows a
blocked thread:
{code}
FileSystemRepository Workers Thread-2 daemon prio=10 tid=0x7f78c266
nid=0x2ae7 waiting for monitor entry [0x7f78a907d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.nifi.controller.repository.FileSystemRepository.archive(FileSystemRepository.java:1095)
- waiting to lock 0xe193ae00 (a
java.util.concurrent.LinkedBlockingQueue)
at
org.apache.nifi.controller.repository.FileSystemRepository.access$1200(FileSystemRepository.java:83)
at
org.apache.nifi.controller.repository.FileSystemRepository$ArchiveOrDestroyDestructableClaims.run(FileSystemRepository.java:1357)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-902) If FlowFile repository runs out of disk space and is then cleaned up, NiFi does not continue to process data
[
https://issues.apache.org/jira/browse/NIFI-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14720372#comment-14720372
]
Mark Payne commented on NIFI-902:
-
[~bende] yup, that's the expected behavior. Once we run out of disk space and
then the disk space is cleared up, you will see that pause that you noted. This
is because the flowfile repo has partial records written to it, so we have to
wait until it is checkpointed so that new journal files are created.
If FlowFile repository runs out of disk space and is then cleaned up, NiFi
does not continue to process data
Key: NIFI-902
URL: https://issues.apache.org/jira/browse/NIFI-902
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Reporter: Mark Payne
Assignee: Mark Payne
Priority: Critical
Fix For: 0.3.0
Attachments:
0001-NIFI-902-Ensure-that-if-FlowFile-Repo-runs-out-of-di.patch
I ran out of disk space on my FlowFile Repository. After cleaning up some
files on the partition, I was able to continue running but when the FlowFile
Repo attempted to checkpoint, I got the following Exception in the logs:
{noformat}
2015-08-27 15:59:58,398 ERROR [pool-20-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Unable to checkpoint FlowFile
Repository due to java.io.IOException: Stream Closed
java.io.IOException: Stream Closed
at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.8.0_45]
at java.io.FileOutputStream.write(FileOutputStream.java:326)
~[na:1.8.0_45]
at
org.apache.nifi.stream.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.stream.io.BufferedOutputStream.flush(BufferedOutputStream.java:126)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
~[na:1.8.0_45]
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
~[na:1.8.0_45]
at
org.wali.MinimalLockingWriteAheadLog$Partition.rollover(MinimalLockingWriteAheadLog.java:723)
~[nifi-write-ahead-log-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.wali.MinimalLockingWriteAheadLog.checkpoint(MinimalLockingWriteAheadLog.java:512)
~[nifi-write-ahead-log-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.checkpoint(WriteAheadFlowFileRepository.java:392)
~[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.controller.repository.WriteAheadFlowFileRepository$1.run(WriteAheadFlowFileRepository.java:364)
~[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Suppressed: java.io.IOException: Stream Closed
at java.io.FileOutputStream.writeBytes(Native Method)
~[na:1.8.0_45]
at java.io.FileOutputStream.write(FileOutputStream.java:326)
~[na:1.8.0_45]
at
org.apache.nifi.stream.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.stream.io.BufferedOutputStream.flush(BufferedOutputStream.java:126)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
java.io.FilterOutputStream.close(FilterOutputStream.java:158) ~[na:1.8.0_45]
at
java.io.FilterOutputStream.close(FilterOutputStream.java:159) ~[na:1.8.0_45]
... 11 common frames omitted
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-905) Clean up not occurring when content repository reaches max usage percentage
[
https://issues.apache.org/jira/browse/NIFI-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14720411#comment-14720411
]
Mark Payne commented on NIFI-905:
-
[~bende]: The reason that you are hitting the 100% full now is because we write
up to a MB of content to a single file before we move on. The repo has 1024
partitions. So you'll get to 1024 MB before it actually starts archiving. We
may want to rethink the idea of using 1 MB as the point at which we move to
another file. Perhaps we should write up to (max size of partition) / 1024 *
50%? So that way we don't fill up the partition without archiving what is
available?
Clean up not occurring when content repository reaches max usage percentage
---
Key: NIFI-905
URL: https://issues.apache.org/jira/browse/NIFI-905
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 0.3.0
Reporter: Bryan Bende
Assignee: Mark Payne
Fix For: 0.3.0
Attachments:
0001-NIFI-905-Ensure-that-when-archive-threshold-is-hit-a.patch, nifi.dump
Created a 500MB partition and set the content repository to use that
partition, then created a simple Flow with GenerateFlowFile -
UpdateAttribute, using 10kb FlowFiles.
When the content repository reached approx 224MB it started logging:
Unable to write to container default to archive file size constraints;
waiting for archive cleanup
It appears that the clean up was never occurring and a thread dump shows a
blocked thread:
{code}
FileSystemRepository Workers Thread-2 daemon prio=10 tid=0x7f78c266
nid=0x2ae7 waiting for monitor entry [0x7f78a907d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.nifi.controller.repository.FileSystemRepository.archive(FileSystemRepository.java:1095)
- waiting to lock 0xe193ae00 (a
java.util.concurrent.LinkedBlockingQueue)
at
org.apache.nifi.controller.repository.FileSystemRepository.access$1200(FileSystemRepository.java:83)
at
org.apache.nifi.controller.repository.FileSystemRepository$ArchiveOrDestroyDestructableClaims.run(FileSystemRepository.java:1357)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (NIFI-902) If FlowFile repository runs out of disk space and is then cleaned up, it will not checkpoint
Mark Payne created NIFI-902: --- Summary: If FlowFile repository runs out of disk space and is then cleaned up, it will not checkpoint Key: NIFI-902 URL: https://issues.apache.org/jira/browse/NIFI-902 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Priority: Critical Fix For: 0.3.0 I ran out of disk space on my FlowFile Repository. After cleaning up some files on the partition, I was able to continue running but when the FlowFile Repo attempted to checkpoint, I got the following Exception in the logs: 2015-08-27 15:59:58,398 ERROR [pool-20-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Unable to checkpoint FlowFile Repository due to java.io.IOException: Stream Closed java.io.IOException: Stream Closed at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.8.0_45] at java.io.FileOutputStream.write(FileOutputStream.java:326) ~[na:1.8.0_45] at org.apache.nifi.stream.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69) ~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] at org.apache.nifi.stream.io.BufferedOutputStream.flush(BufferedOutputStream.java:126) ~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] at java.io.DataOutputStream.flush(DataOutputStream.java:123) ~[na:1.8.0_45] at java.io.FilterOutputStream.close(FilterOutputStream.java:158) ~[na:1.8.0_45] at org.wali.MinimalLockingWriteAheadLog$Partition.rollover(MinimalLockingWriteAheadLog.java:723) ~[nifi-write-ahead-log-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] at org.wali.MinimalLockingWriteAheadLog.checkpoint(MinimalLockingWriteAheadLog.java:512) ~[nifi-write-ahead-log-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.checkpoint(WriteAheadFlowFileRepository.java:392) ~[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository$1.run(WriteAheadFlowFileRepository.java:364) ~[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_45] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_45] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_45] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45] Suppressed: java.io.IOException: Stream Closed at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.8.0_45] at java.io.FileOutputStream.write(FileOutputStream.java:326) ~[na:1.8.0_45] at org.apache.nifi.stream.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69) ~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] at org.apache.nifi.stream.io.BufferedOutputStream.flush(BufferedOutputStream.java:126) ~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT] at java.io.FilterOutputStream.close(FilterOutputStream.java:158) ~[na:1.8.0_45] at java.io.FilterOutputStream.close(FilterOutputStream.java:159) ~[na:1.8.0_45] ... 11 common frames omitted -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-893) If FlowFile Repository runs out of disk space, NiFi stops processing until restarted
[ https://issues.apache.org/jira/browse/NIFI-893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-893: Attachment: 0001-NIFI-893-Ensure-that-if-session.commit-fails-previou.patch If FlowFile Repository runs out of disk space, NiFi stops processing until restarted Key: NIFI-893 URL: https://issues.apache.org/jira/browse/NIFI-893 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Fix For: 0.3.0 Attachments: 0001-NIFI-893-Ensure-that-if-session.commit-fails-previou.patch If the FlowFile Repository runs out of disk space, NiFi stops processing any data. Once space is recovered on the partition, NiFi never resumes processing until it is restarted. A restart appears to have recovered all data without issue and all appears to work as expected after restarting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-902) If FlowFile repository runs out of disk space and is then cleaned up, it will not checkpoint
[
https://issues.apache.org/jira/browse/NIFI-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-902:
Description:
I ran out of disk space on my FlowFile Repository. After cleaning up some files
on the partition, I was able to continue running but when the FlowFile Repo
attempted to checkpoint, I got the following Exception in the logs:
{noformat}
2015-08-27 15:59:58,398 ERROR [pool-20-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Unable to checkpoint FlowFile Repository
due to java.io.IOException: Stream Closed
java.io.IOException: Stream Closed
at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.8.0_45]
at java.io.FileOutputStream.write(FileOutputStream.java:326)
~[na:1.8.0_45]
at
org.apache.nifi.stream.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.stream.io.BufferedOutputStream.flush(BufferedOutputStream.java:126)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
~[na:1.8.0_45]
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
~[na:1.8.0_45]
at
org.wali.MinimalLockingWriteAheadLog$Partition.rollover(MinimalLockingWriteAheadLog.java:723)
~[nifi-write-ahead-log-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.wali.MinimalLockingWriteAheadLog.checkpoint(MinimalLockingWriteAheadLog.java:512)
~[nifi-write-ahead-log-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.checkpoint(WriteAheadFlowFileRepository.java:392)
~[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.controller.repository.WriteAheadFlowFileRepository$1.run(WriteAheadFlowFileRepository.java:364)
~[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Suppressed: java.io.IOException: Stream Closed
at java.io.FileOutputStream.writeBytes(Native Method)
~[na:1.8.0_45]
at java.io.FileOutputStream.write(FileOutputStream.java:326)
~[na:1.8.0_45]
at
org.apache.nifi.stream.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.stream.io.BufferedOutputStream.flush(BufferedOutputStream.java:126)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
java.io.FilterOutputStream.close(FilterOutputStream.java:158) ~[na:1.8.0_45]
at
java.io.FilterOutputStream.close(FilterOutputStream.java:159) ~[na:1.8.0_45]
... 11 common frames omitted
{noformat}
was:
I ran out of disk space on my FlowFile Repository. After cleaning up some files
on the partition, I was able to continue running but when the FlowFile Repo
attempted to checkpoint, I got the following Exception in the logs:
2015-08-27 15:59:58,398 ERROR [pool-20-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Unable to checkpoint FlowFile Repository
due to java.io.IOException: Stream Closed
java.io.IOException: Stream Closed
at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.8.0_45]
at java.io.FileOutputStream.write(FileOutputStream.java:326)
~[na:1.8.0_45]
at
org.apache.nifi.stream.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.stream.io.BufferedOutputStream.flush(BufferedOutputStream.java:126)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
~[na:1.8.0_45]
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
~[na:1.8.0_45]
at
org.wali.MinimalLockingWriteAheadLog$Partition.rollover(MinimalLockingWriteAheadLog.java:723)
~[nifi-write-ahead-log-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.wali.MinimalLockingWriteAheadLog.checkpoint(MinimalLockingWriteAheadLog.java:512)
~[nifi-write-ahead-log-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
[jira] [Created] (NIFI-903) If provenance repository runs out of disk space, it will throw a NullPointerException
Mark Payne created NIFI-903:
---
Summary: If provenance repository runs out of disk space, it will
throw a NullPointerException
Key: NIFI-903
URL: https://issues.apache.org/jira/browse/NIFI-903
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Reporter: Mark Payne
Assignee: Mark Payne
Fix For: 0.3.0
I allowed my Provenance Repository to run out of disk space. When it attempts
to roll over, I started getting the following stack trace:
{noformat}
2015-08-27 16:00:04,880 ERROR [Provenance Repository Rollover Thread-2]
o.a.n.p.PersistentProvenanceRepository
java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.8.0_45]
at java.io.FileOutputStream.write(FileOutputStream.java:326)
~[na:1.8.0_45]
at
org.apache.nifi.stream.io.ByteCountingOutputStream.write(ByteCountingOutputStream.java:49)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.stream.io.ByteCountingOutputStream.write(ByteCountingOutputStream.java:44)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.stream.io.NonCloseableOutputStream.write(NonCloseableOutputStream.java:34)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
java.util.zip.GZIPOutputStream.writeHeader(GZIPOutputStream.java:182)
~[na:1.8.0_45]
at java.util.zip.GZIPOutputStream.init(GZIPOutputStream.java:94)
~[na:1.8.0_45]
at java.util.zip.GZIPOutputStream.init(GZIPOutputStream.java:109)
~[na:1.8.0_45]
at
org.apache.nifi.stream.io.GZIPOutputStream.init(GZIPOutputStream.java:36)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.provenance.StandardRecordWriter.resetWriteStream(StandardRecordWriter.java:126)
~[nifi-persistent-provenance-repository-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.provenance.StandardRecordWriter.writeHeader(StandardRecordWriter.java:93)
~[nifi-persistent-provenance-repository-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1563)
~[nifi-persistent-provenance-repository-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.provenance.PersistentProvenanceRepository$8.run(PersistentProvenanceRepository.java:1196)
~[nifi-persistent-provenance-repository-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Suppressed: java.lang.NullPointerException: null
at
org.apache.nifi.provenance.StandardRecordWriter.close(StandardRecordWriter.java:277)
~[nifi-persistent-provenance-repository-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at
org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1672)
~[nifi-persistent-provenance-repository-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
... 8 common frames omitted
{noformat}
While the IOException indicating out of disk space is expected, the suppressed
NullPointerException is not and needs to be addressed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-893) If FlowFile Repository runs out of disk space, NiFi stops processing until restarted
[ https://issues.apache.org/jira/browse/NIFI-893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-893: Fix Version/s: (was: 0.4.0) 0.3.0 If FlowFile Repository runs out of disk space, NiFi stops processing until restarted Key: NIFI-893 URL: https://issues.apache.org/jira/browse/NIFI-893 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Fix For: 0.3.0 If the FlowFile Repository runs out of disk space, NiFi stops processing any data. Once space is recovered on the partition, NiFi never resumes processing until it is restarted. A restart appears to have recovered all data without issue and all appears to work as expected after restarting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-893) If FlowFile Repository runs out of disk space, NiFi stops processing until restarted
[ https://issues.apache.org/jira/browse/NIFI-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711318#comment-14711318 ] Mark Payne commented on NIFI-893: - [~joewitt] my thinking for putting the fix version as 0.4.0 was that if disk space runs out, it generally will require manual intervention, so at that point I decided 0.4.0 made the most sense. You do have a point, though - if the flowfile repo is not on its own partition, the disk space may clear up itself via some sort of background process or what have you. It was honestly a bit of a toss-up for me which version to use and I certainly wouldn't put out 0.4.0 without doing this ticket. I am okay moving this to 0.3.0 if you think it's critical enough that we need it for 0.3.0. If FlowFile Repository runs out of disk space, NiFi stops processing until restarted Key: NIFI-893 URL: https://issues.apache.org/jira/browse/NIFI-893 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Fix For: 0.4.0 If the FlowFile Repository runs out of disk space, NiFi stops processing any data. Once space is recovered on the partition, NiFi never resumes processing until it is restarted. A restart appears to have recovered all data without issue and all appears to work as expected after restarting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
