[jira] [Commented] (NIFI-905) Clean up not occurring when content repository reaches max usage percentage
[
https://issues.apache.org/jira/browse/NIFI-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723532#comment-14723532
]
Bryan Bende commented on NIFI-905:
--
I'm a +1 here, passes contrib-check, resolves previous issue of blocked thread.
> Clean up not occurring when content repository reaches max usage percentage
> ---
>
> Key: NIFI-905
> URL: https://issues.apache.org/jira/browse/NIFI-905
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: Bryan Bende
>Assignee: Mark Payne
> Fix For: 0.3.0
>
> Attachments:
> 0001-NIFI-905-Ensure-that-when-archive-threshold-is-hit-a.patch, nifi.dump
>
>
> Created a 500MB partition and set the content repository to use that
> partition, then created a simple Flow with GenerateFlowFile ->
> UpdateAttribute, using 10kb FlowFiles.
> When the content repository reached approx 224MB it started logging:
> "Unable to write to container default to archive file size constraints;
> waiting for archive cleanup"
> It appears that the clean up was never occurring and a thread dump shows a
> blocked thread:
> {code}
> "FileSystemRepository Workers Thread-2" daemon prio=10 tid=0x7f78c266
> nid=0x2ae7 waiting for monitor entry [0x7f78a907d000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.nifi.controller.repository.FileSystemRepository.archive(FileSystemRepository.java:1095)
> - waiting to lock <0xe193ae00> (a
> java.util.concurrent.LinkedBlockingQueue)
> at
> org.apache.nifi.controller.repository.FileSystemRepository.access$1200(FileSystemRepository.java:83)
> at
> org.apache.nifi.controller.repository.FileSystemRepository$ArchiveOrDestroyDestructableClaims.run(FileSystemRepository.java:1357)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-905) Clean up not occurring when content repository reaches max usage percentage
[
https://issues.apache.org/jira/browse/NIFI-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723577#comment-14723577
]
ASF subversion and git services commented on NIFI-905:
--
Commit 3159cec7825a4085898b64f29a0dc4cc7d803f17 in nifi's branch
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=3159cec ]
NIFI-905: Ensure that when archive threshold is hit, archived data is destroyed
and if no archived data exists that Processors aren't blocked from updating
content repo
> Clean up not occurring when content repository reaches max usage percentage
> ---
>
> Key: NIFI-905
> URL: https://issues.apache.org/jira/browse/NIFI-905
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: Bryan Bende
>Assignee: Mark Payne
> Fix For: 0.3.0
>
> Attachments:
> 0001-NIFI-905-Ensure-that-when-archive-threshold-is-hit-a.patch, nifi.dump
>
>
> Created a 500MB partition and set the content repository to use that
> partition, then created a simple Flow with GenerateFlowFile ->
> UpdateAttribute, using 10kb FlowFiles.
> When the content repository reached approx 224MB it started logging:
> "Unable to write to container default to archive file size constraints;
> waiting for archive cleanup"
> It appears that the clean up was never occurring and a thread dump shows a
> blocked thread:
> {code}
> "FileSystemRepository Workers Thread-2" daemon prio=10 tid=0x7f78c266
> nid=0x2ae7 waiting for monitor entry [0x7f78a907d000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.nifi.controller.repository.FileSystemRepository.archive(FileSystemRepository.java:1095)
> - waiting to lock <0xe193ae00> (a
> java.util.concurrent.LinkedBlockingQueue)
> at
> org.apache.nifi.controller.repository.FileSystemRepository.access$1200(FileSystemRepository.java:83)
> at
> org.apache.nifi.controller.repository.FileSystemRepository$ArchiveOrDestroyDestructableClaims.run(FileSystemRepository.java:1357)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-905) Clean up not occurring when content repository reaches max usage percentage
[
https://issues.apache.org/jira/browse/NIFI-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723579#comment-14723579
]
ASF subversion and git services commented on NIFI-905:
--
Commit 2bb7853001c680b54282c0720ceba4ae9c477556 in nifi's branch
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=2bb7853 ]
Merge branch 'NIFI-905'
> Clean up not occurring when content repository reaches max usage percentage
> ---
>
> Key: NIFI-905
> URL: https://issues.apache.org/jira/browse/NIFI-905
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: Bryan Bende
>Assignee: Mark Payne
> Fix For: 0.3.0
>
> Attachments:
> 0001-NIFI-905-Ensure-that-when-archive-threshold-is-hit-a.patch, nifi.dump
>
>
> Created a 500MB partition and set the content repository to use that
> partition, then created a simple Flow with GenerateFlowFile ->
> UpdateAttribute, using 10kb FlowFiles.
> When the content repository reached approx 224MB it started logging:
> "Unable to write to container default to archive file size constraints;
> waiting for archive cleanup"
> It appears that the clean up was never occurring and a thread dump shows a
> blocked thread:
> {code}
> "FileSystemRepository Workers Thread-2" daemon prio=10 tid=0x7f78c266
> nid=0x2ae7 waiting for monitor entry [0x7f78a907d000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.nifi.controller.repository.FileSystemRepository.archive(FileSystemRepository.java:1095)
> - waiting to lock <0xe193ae00> (a
> java.util.concurrent.LinkedBlockingQueue)
> at
> org.apache.nifi.controller.repository.FileSystemRepository.access$1200(FileSystemRepository.java:83)
> at
> org.apache.nifi.controller.repository.FileSystemRepository$ArchiveOrDestroyDestructableClaims.run(FileSystemRepository.java:1357)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-905) Clean up not occurring when content repository reaches max usage percentage
[
https://issues.apache.org/jira/browse/NIFI-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14720055#comment-14720055
]
Mark Payne commented on NIFI-905:
-
Actually I may have spoken too soon. Based on the thread dump that you provided
above, the thread is blocking when it is trying to archive. I'll dig a bit more
:)
Clean up not occurring when content repository reaches max usage percentage
---
Key: NIFI-905
URL: https://issues.apache.org/jira/browse/NIFI-905
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 0.3.0
Reporter: Bryan Bende
Fix For: 0.3.0
Attachments: nifi.dump
Created a 500MB partition and set the content repository to use that
partition, then created a simple Flow with GenerateFlowFile -
UpdateAttribute, using 10kb FlowFiles.
When the content repository reached approx 224MB it started logging:
Unable to write to container default to archive file size constraints;
waiting for archive cleanup
It appears that the clean up was never occurring and a thread dump shows a
blocked thread:
{code}
FileSystemRepository Workers Thread-2 daemon prio=10 tid=0x7f78c266
nid=0x2ae7 waiting for monitor entry [0x7f78a907d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.nifi.controller.repository.FileSystemRepository.archive(FileSystemRepository.java:1095)
- waiting to lock 0xe193ae00 (a
java.util.concurrent.LinkedBlockingQueue)
at
org.apache.nifi.controller.repository.FileSystemRepository.access$1200(FileSystemRepository.java:83)
at
org.apache.nifi.controller.repository.FileSystemRepository$ArchiveOrDestroyDestructableClaims.run(FileSystemRepository.java:1357)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-905) Clean up not occurring when content repository reaches max usage percentage
[
https://issues.apache.org/jira/browse/NIFI-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14720047#comment-14720047
]
Mark Payne commented on NIFI-905:
-
[~bende] I am able to duplicate this as well. Great find.
Looks like the issue is that the content repo is waiting for the archive to be
cleaned up... but there is nothing in the archive. I think we need to ensure
that if the archive is empty we trigger things to start moving again.
Clean up not occurring when content repository reaches max usage percentage
---
Key: NIFI-905
URL: https://issues.apache.org/jira/browse/NIFI-905
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 0.3.0
Reporter: Bryan Bende
Fix For: 0.3.0
Attachments: nifi.dump
Created a 500MB partition and set the content repository to use that
partition, then created a simple Flow with GenerateFlowFile -
UpdateAttribute, using 10kb FlowFiles.
When the content repository reached approx 224MB it started logging:
Unable to write to container default to archive file size constraints;
waiting for archive cleanup
It appears that the clean up was never occurring and a thread dump shows a
blocked thread:
{code}
FileSystemRepository Workers Thread-2 daemon prio=10 tid=0x7f78c266
nid=0x2ae7 waiting for monitor entry [0x7f78a907d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.nifi.controller.repository.FileSystemRepository.archive(FileSystemRepository.java:1095)
- waiting to lock 0xe193ae00 (a
java.util.concurrent.LinkedBlockingQueue)
at
org.apache.nifi.controller.repository.FileSystemRepository.access$1200(FileSystemRepository.java:83)
at
org.apache.nifi.controller.repository.FileSystemRepository$ArchiveOrDestroyDestructableClaims.run(FileSystemRepository.java:1357)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-905) Clean up not occurring when content repository reaches max usage percentage
[
https://issues.apache.org/jira/browse/NIFI-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14720404#comment-14720404
]
Bryan Bende commented on NIFI-905:
--
Applied this patch to master and ran the same test, it seems to keep blowing
past the default 50% limit and ends up filling up the partition. I was
expecting it to still stop around 50% and then wait for the archiving, and then
because of the patch the archiving would happen now, right?
Clean up not occurring when content repository reaches max usage percentage
---
Key: NIFI-905
URL: https://issues.apache.org/jira/browse/NIFI-905
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 0.3.0
Reporter: Bryan Bende
Assignee: Mark Payne
Fix For: 0.3.0
Attachments:
0001-NIFI-905-Ensure-that-when-archive-threshold-is-hit-a.patch, nifi.dump
Created a 500MB partition and set the content repository to use that
partition, then created a simple Flow with GenerateFlowFile -
UpdateAttribute, using 10kb FlowFiles.
When the content repository reached approx 224MB it started logging:
Unable to write to container default to archive file size constraints;
waiting for archive cleanup
It appears that the clean up was never occurring and a thread dump shows a
blocked thread:
{code}
FileSystemRepository Workers Thread-2 daemon prio=10 tid=0x7f78c266
nid=0x2ae7 waiting for monitor entry [0x7f78a907d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.nifi.controller.repository.FileSystemRepository.archive(FileSystemRepository.java:1095)
- waiting to lock 0xe193ae00 (a
java.util.concurrent.LinkedBlockingQueue)
at
org.apache.nifi.controller.repository.FileSystemRepository.access$1200(FileSystemRepository.java:83)
at
org.apache.nifi.controller.repository.FileSystemRepository$ArchiveOrDestroyDestructableClaims.run(FileSystemRepository.java:1357)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-905) Clean up not occurring when content repository reaches max usage percentage
[
https://issues.apache.org/jira/browse/NIFI-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14720411#comment-14720411
]
Mark Payne commented on NIFI-905:
-
[~bende]: The reason that you are hitting the 100% full now is because we write
up to a MB of content to a single file before we move on. The repo has 1024
partitions. So you'll get to 1024 MB before it actually starts archiving. We
may want to rethink the idea of using 1 MB as the point at which we move to
another file. Perhaps we should write up to (max size of partition) / 1024 *
50%? So that way we don't fill up the partition without archiving what is
available?
Clean up not occurring when content repository reaches max usage percentage
---
Key: NIFI-905
URL: https://issues.apache.org/jira/browse/NIFI-905
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 0.3.0
Reporter: Bryan Bende
Assignee: Mark Payne
Fix For: 0.3.0
Attachments:
0001-NIFI-905-Ensure-that-when-archive-threshold-is-hit-a.patch, nifi.dump
Created a 500MB partition and set the content repository to use that
partition, then created a simple Flow with GenerateFlowFile -
UpdateAttribute, using 10kb FlowFiles.
When the content repository reached approx 224MB it started logging:
Unable to write to container default to archive file size constraints;
waiting for archive cleanup
It appears that the clean up was never occurring and a thread dump shows a
blocked thread:
{code}
FileSystemRepository Workers Thread-2 daemon prio=10 tid=0x7f78c266
nid=0x2ae7 waiting for monitor entry [0x7f78a907d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.nifi.controller.repository.FileSystemRepository.archive(FileSystemRepository.java:1095)
- waiting to lock 0xe193ae00 (a
java.util.concurrent.LinkedBlockingQueue)
at
org.apache.nifi.controller.repository.FileSystemRepository.access$1200(FileSystemRepository.java:83)
at
org.apache.nifi.controller.repository.FileSystemRepository$ArchiveOrDestroyDestructableClaims.run(FileSystemRepository.java:1357)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-905) Clean up not occurring when content repository reaches max usage percentage
[
https://issues.apache.org/jira/browse/NIFI-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14720442#comment-14720442
]
Bryan Bende commented on NIFI-905:
--
Ah I see what you are saying now, so my scenario will never trigger the
archiving because the partition is too small. I think your idea of how to
calculate the point of moving to another file makes sense, but probably not
required to address the intent of this ticket. I'll let that be your call, but
we could create another ticket Adjust max append claim length based on max
partition size if that makes sense.
In any case, given the above I was able to test a new scenario which is filling
up the content partition. Verified that processing resumed when I removed other
miscellaneous files to free up space.
Clean up not occurring when content repository reaches max usage percentage
---
Key: NIFI-905
URL: https://issues.apache.org/jira/browse/NIFI-905
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 0.3.0
Reporter: Bryan Bende
Assignee: Mark Payne
Fix For: 0.3.0
Attachments:
0001-NIFI-905-Ensure-that-when-archive-threshold-is-hit-a.patch, nifi.dump
Created a 500MB partition and set the content repository to use that
partition, then created a simple Flow with GenerateFlowFile -
UpdateAttribute, using 10kb FlowFiles.
When the content repository reached approx 224MB it started logging:
Unable to write to container default to archive file size constraints;
waiting for archive cleanup
It appears that the clean up was never occurring and a thread dump shows a
blocked thread:
{code}
FileSystemRepository Workers Thread-2 daemon prio=10 tid=0x7f78c266
nid=0x2ae7 waiting for monitor entry [0x7f78a907d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.nifi.controller.repository.FileSystemRepository.archive(FileSystemRepository.java:1095)
- waiting to lock 0xe193ae00 (a
java.util.concurrent.LinkedBlockingQueue)
at
org.apache.nifi.controller.repository.FileSystemRepository.access$1200(FileSystemRepository.java:83)
at
org.apache.nifi.controller.repository.FileSystemRepository$ArchiveOrDestroyDestructableClaims.run(FileSystemRepository.java:1357)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
