[jira] [Commented] (NIFI-731) If content repo is unable to destroy content as fast as it is generated, nifi performance becomes very sporadic
[ https://issues.apache.org/jira/browse/NIFI-731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14610193#comment-14610193 ] Mark Payne commented on NIFI-731: - The patch supplied here provides a few improvements. It allows the user to synchronize individual partitions of the flowfile repo on regular intervals, which will allow some content claims to start being archived/desctroyed immediately. Currently, we wait until the repo is checkpointed and start destroying all content claims, so this will provide a smoother performance. Additionally, it allows the user to change the number of partitions used by the FlowFile Repo. This is done because experimentation shows that 16 partitions is generally enough and results in much better performance than 256 - so the default was also changed from 256 to 16. A better but much more involved solution is to allow the Content Repository to append to an existing Content Claim, as described in NIFI-744. This will result in far fewer files to be deleted, and this will very much alleviate this problem. If content repo is unable to destroy content as fast as it is generated, nifi performance becomes very sporadic --- Key: NIFI-731 URL: https://issues.apache.org/jira/browse/NIFI-731 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Fix For: 0.2.0 Attachments: 0001-NIFI-731-Updated-admin-guide-to-explain-the-flowfile.patch When the FlowFile Repository marks claims as destructable, it puts the notification on a queue that the content repo pulls from. If the content repo cannot keep up, the queue will fill, resulting in backpressure, that prevents the FlowFile repository from being updated. This, in turn, causes Processors to block, waiting on space to become available. This is by design. However, the capacity of this queue is quite large, and the content repo drains the entire queue, then destroys all content claims that are on it. As a result, this act of destroying claims can be quite long, and Processors can block for quite a period of time, leading to very sporadic performance. Instead, the content repo should pull from the queue and destroy the claims one at a time or in small batches, instead of draining the entire queue each time. This should result in much less sporadic behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-731) If content repo is unable to destroy content as fast as it is generated, nifi performance becomes very sporadic
[ https://issues.apache.org/jira/browse/NIFI-731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14608514#comment-14608514 ] ASF subversion and git services commented on NIFI-731: -- Commit 4b5ba3c4e9bbb2be5bd3f9bdf40159179a3b30e2 in incubator-nifi's branch refs/heads/NIFI-731 from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;h=4b5ba3c ] NIFI-731: Try using a Stack to hold FLowFIle Repo partitions so that we tend to use the same partition to avoid disk seeks/thrashing If content repo is unable to destroy content as fast as it is generated, nifi performance becomes very sporadic --- Key: NIFI-731 URL: https://issues.apache.org/jira/browse/NIFI-731 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Fix For: 0.2.0 When the FlowFile Repository marks claims as destructable, it puts the notification on a queue that the content repo pulls from. If the content repo cannot keep up, the queue will fill, resulting in backpressure, that prevents the FlowFile repository from being updated. This, in turn, causes Processors to block, waiting on space to become available. This is by design. However, the capacity of this queue is quite large, and the content repo drains the entire queue, then destroys all content claims that are on it. As a result, this act of destroying claims can be quite long, and Processors can block for quite a period of time, leading to very sporadic performance. Instead, the content repo should pull from the queue and destroy the claims one at a time or in small batches, instead of draining the entire queue each time. This should result in much less sporadic behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-731) If content repo is unable to destroy content as fast as it is generated, nifi performance becomes very sporadic
[ https://issues.apache.org/jira/browse/NIFI-731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14608507#comment-14608507 ] Mark Payne commented on NIFI-731: - I have made some changes to the way that the FlowFileRepository and ContentRepository interact with one another. I created some benchmarks to compare the results before and after the changes. The changes include a new property in the nifi.properties file to configure how often the FlowFile Repo performs a 'sync' (and a good bit of documentation added to the Admin Guide about what this means). Benchmark was performed against both my desktop and my laptop. Note that the flow used is designed specifically to ensure that this issue arises by creating Content Claims that are exactly 1 byte in size, so that massive stress is put on deleting tons of files. It is not intended to mimick a typical flow. After the change -- FlowFile Repo Settings: 8 partitions Sync every 100 updates Hardware: Laptop: 1 drive, 5400 RPM Desktop: 2 drives, 7200 RPM (1 for Content, 1 for FlowFile) Flow: GenerateFlowFile - LogAttribute GenerateFlowFile set to 1 byte files, batch size of 1, 0 ms run duration. So 1 byte per Content Claim/File on Disk LogAttribute set to 'debug' level so it doesn't actually log. 25 ms run duration. With Content Repo's archive disabled: Laptop: 125,000 FlowFiles / 5 min. Warns about backpressure Desktop: 1.03 million FlowFiles / 5 min. Does not warn about backpressure With archive enabled: Laptop: 25,000 FlowFIles / 5 min. Warns about backpressure Desktop: 115,000 FlowFiles / 5 min. Warns about backpressure - Changed Batch Size property of GenerateFlowFile to 5 FlowFiles per Content Claim. Got 435,000 FlowFiles - about 5 times as much, which is what I expected. But a good sanity check. Baseline to compare against, before the patch was applied Laptop: Reached 60,000 FlowFiles/5 mins, then saw very long pause as the Content Repo destroyed content. FlowFiles per 5 mins dropped from 60K to 30K and eventually to under 15K and then back up and down and up and down. Pauses were very noticeable in the UI. Desktop: 481,000 FlowFiles/5 mins, then saw very long pause as the Content Repo destroyed content. FlowFiles per 5 mins then dropped and fluctuated similarly to Laptop's results. If content repo is unable to destroy content as fast as it is generated, nifi performance becomes very sporadic --- Key: NIFI-731 URL: https://issues.apache.org/jira/browse/NIFI-731 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Fix For: 0.2.0 When the FlowFile Repository marks claims as destructable, it puts the notification on a queue that the content repo pulls from. If the content repo cannot keep up, the queue will fill, resulting in backpressure, that prevents the FlowFile repository from being updated. This, in turn, causes Processors to block, waiting on space to become available. This is by design. However, the capacity of this queue is quite large, and the content repo drains the entire queue, then destroys all content claims that are on it. As a result, this act of destroying claims can be quite long, and Processors can block for quite a period of time, leading to very sporadic performance. Instead, the content repo should pull from the queue and destroy the claims one at a time or in small batches, instead of draining the entire queue each time. This should result in much less sporadic behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-731) If content repo is unable to destroy content as fast as it is generated, nifi performance becomes very sporadic
[ https://issues.apache.org/jira/browse/NIFI-731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14608777#comment-14608777 ] ASF subversion and git services commented on NIFI-731: -- Commit 94b4eddb00055fccbe57c5e3fb2ed93c0e91c153 in incubator-nifi's branch refs/heads/NIFI-731 from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;h=94b4edd ] NIFI-731: fixed issue with how bulletins were emitted by content claim manager If content repo is unable to destroy content as fast as it is generated, nifi performance becomes very sporadic --- Key: NIFI-731 URL: https://issues.apache.org/jira/browse/NIFI-731 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Fix For: 0.2.0 Attachments: 0001-NIFI-731-Refactored-how-Content-and-FlowFile-Repos-i.patch When the FlowFile Repository marks claims as destructable, it puts the notification on a queue that the content repo pulls from. If the content repo cannot keep up, the queue will fill, resulting in backpressure, that prevents the FlowFile repository from being updated. This, in turn, causes Processors to block, waiting on space to become available. This is by design. However, the capacity of this queue is quite large, and the content repo drains the entire queue, then destroys all content claims that are on it. As a result, this act of destroying claims can be quite long, and Processors can block for quite a period of time, leading to very sporadic performance. Instead, the content repo should pull from the queue and destroy the claims one at a time or in small batches, instead of draining the entire queue each time. This should result in much less sporadic behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-731) If content repo is unable to destroy content as fast as it is generated, nifi performance becomes very sporadic
[ https://issues.apache.org/jira/browse/NIFI-731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14608776#comment-14608776 ] ASF subversion and git services commented on NIFI-731: -- Commit b3eb4a1a0f29d0f94639174dfbd7ea05307cb99a in incubator-nifi's branch refs/heads/NIFI-731 from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;h=b3eb4a1 ] Revert NIFI-731: Try using a Stack to hold FLowFIle Repo partitions so that we tend to use the same partition to avoid disk seeks/thrashing This reverts commit 4b5ba3c4e9bbb2be5bd3f9bdf40159179a3b30e2. If content repo is unable to destroy content as fast as it is generated, nifi performance becomes very sporadic --- Key: NIFI-731 URL: https://issues.apache.org/jira/browse/NIFI-731 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Fix For: 0.2.0 Attachments: 0001-NIFI-731-Refactored-how-Content-and-FlowFile-Repos-i.patch When the FlowFile Repository marks claims as destructable, it puts the notification on a queue that the content repo pulls from. If the content repo cannot keep up, the queue will fill, resulting in backpressure, that prevents the FlowFile repository from being updated. This, in turn, causes Processors to block, waiting on space to become available. This is by design. However, the capacity of this queue is quite large, and the content repo drains the entire queue, then destroys all content claims that are on it. As a result, this act of destroying claims can be quite long, and Processors can block for quite a period of time, leading to very sporadic performance. Instead, the content repo should pull from the queue and destroy the claims one at a time or in small batches, instead of draining the entire queue each time. This should result in much less sporadic behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-731) If content repo is unable to destroy content as fast as it is generated, nifi performance becomes very sporadic
[ https://issues.apache.org/jira/browse/NIFI-731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14608775#comment-14608775 ] ASF subversion and git services commented on NIFI-731: -- Commit 1677db6261c2c77df15c2e46a54d7428fc91fc57 in incubator-nifi's branch refs/heads/NIFI-731 from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;h=1677db6 ] NIFI-731: Deleted unused method If content repo is unable to destroy content as fast as it is generated, nifi performance becomes very sporadic --- Key: NIFI-731 URL: https://issues.apache.org/jira/browse/NIFI-731 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Fix For: 0.2.0 Attachments: 0001-NIFI-731-Refactored-how-Content-and-FlowFile-Repos-i.patch When the FlowFile Repository marks claims as destructable, it puts the notification on a queue that the content repo pulls from. If the content repo cannot keep up, the queue will fill, resulting in backpressure, that prevents the FlowFile repository from being updated. This, in turn, causes Processors to block, waiting on space to become available. This is by design. However, the capacity of this queue is quite large, and the content repo drains the entire queue, then destroys all content claims that are on it. As a result, this act of destroying claims can be quite long, and Processors can block for quite a period of time, leading to very sporadic performance. Instead, the content repo should pull from the queue and destroy the claims one at a time or in small batches, instead of draining the entire queue each time. This should result in much less sporadic behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-731) If content repo is unable to destroy content as fast as it is generated, nifi performance becomes very sporadic
[ https://issues.apache.org/jira/browse/NIFI-731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14604801#comment-14604801 ] ASF subversion and git services commented on NIFI-731: -- Commit bb5128be2246dc48d5200e98572c47ff7cdfb905 in incubator-nifi's branch refs/heads/NIFI-731 from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;h=bb5128b ] NIFI-731: Updated admin guide to explain the flowfile repo properties more; added new property for now often to 'sync' the flowfile repo; allowed number of partitions in flowfile repo to be changed after repo created. Minor tweaks to content repo and its communication with flowfile repo via ContentClaimManager If content repo is unable to destroy content as fast as it is generated, nifi performance becomes very sporadic --- Key: NIFI-731 URL: https://issues.apache.org/jira/browse/NIFI-731 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Fix For: 0.2.0 When the FlowFile Repository marks claims as destructable, it puts the notification on a queue that the content repo pulls from. If the content repo cannot keep up, the queue will fill, resulting in backpressure, that prevents the FlowFile repository from being updated. This, in turn, causes Processors to block, waiting on space to become available. This is by design. However, the capacity of this queue is quite large, and the content repo drains the entire queue, then destroys all content claims that are on it. As a result, this act of destroying claims can be quite long, and Processors can block for quite a period of time, leading to very sporadic performance. Instead, the content repo should pull from the queue and destroy the claims one at a time or in small batches, instead of draining the entire queue each time. This should result in much less sporadic behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-731) If content repo is unable to destroy content as fast as it is generated, nifi performance becomes very sporadic
[ https://issues.apache.org/jira/browse/NIFI-731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14604800#comment-14604800 ] ASF subversion and git services commented on NIFI-731: -- Commit fb0a1a76f431148fa9664e565c94cf334498dbf8 in incubator-nifi's branch refs/heads/NIFI-731 from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;h=fb0a1a7 ] NIFI-731: Merged develop to branch If content repo is unable to destroy content as fast as it is generated, nifi performance becomes very sporadic --- Key: NIFI-731 URL: https://issues.apache.org/jira/browse/NIFI-731 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Fix For: 0.2.0 When the FlowFile Repository marks claims as destructable, it puts the notification on a queue that the content repo pulls from. If the content repo cannot keep up, the queue will fill, resulting in backpressure, that prevents the FlowFile repository from being updated. This, in turn, causes Processors to block, waiting on space to become available. This is by design. However, the capacity of this queue is quite large, and the content repo drains the entire queue, then destroys all content claims that are on it. As a result, this act of destroying claims can be quite long, and Processors can block for quite a period of time, leading to very sporadic performance. Instead, the content repo should pull from the queue and destroy the claims one at a time or in small batches, instead of draining the entire queue each time. This should result in much less sporadic behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
