[
https://issues.apache.org/jira/browse/NIFI-9464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17804441#comment-17804441
]
Mark Payne commented on NIFI-9464:
----------------------------------
[~tpalfy] I got you. Makes sense. I did a quick look over this again to make
sure that I fully understand what's happening here. It looks like this was
actually designed to work as you've proposed in the PR. But when the Encrypted
Prov Repo was introduced, the base class's init() method was changed to start
creating its own `EventFileManager`. As a result, the base class has a
different instance than the concrete class is using. So this change fixes that
to ensure that both the base class and the concrete class are sharing the same
instance. Makes perfect sense. Great catch! Thanks for running that down and
fixing. I'm a +1 will merge.
> Provenance Events files corrupted
> ---------------------------------
>
> Key: NIFI-9464
> URL: https://issues.apache.org/jira/browse/NIFI-9464
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.11.0, 1.15.0
> Environment: java 11, centos 7, nifi standalone
> Reporter: Wiktor Kubicki
> Assignee: Tamas Palfy
> Priority: Minor
> Fix For: 1.25.0, 2.0.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In my logs i found:
> {code:java}
> SiteToSiteProvenanceReportingTask[id=b209c0ae-016e-1000-ae39-301c9dcfc544]
> Failed to retrieve Provenance Events from repository due to: Attempted to
> skip to byte offset 9149491 for 1125432890.prov.gz but file does not have
> that many bytes (TOC
> Reader=StandardTocReader[file=/..../provenance_repository/toc/1125432890.toc,
> compressed=false]): java.io.EOFException: Attempted to skip to byte offset
> 9149491 for 1125432890.prov.gz but file does not have that many bytes (TOC
> Reader=StandardTocReader[file=/.../provenance_repository/toc/1125432890.toc,
> compressed=false])
> {code}
> It is criticaly important for me to have 100% sure of my logs. It happened
> about 100 times in last 1 year for 15 *.prov.gz files:
> {code:java}
> -rw-rw-rw-. 1 user user 1013923 Oct 17 21:17 1075441276.prov.gz
> -rw-rw-rw-. 1 user user 1345431 Oct 24 13:06 1083362251.prov.gz
> -rw-rw-rw-. 1 user user 1359282 Oct 25 13:07 1084546392.prov.gz
> -rw-rw-rw-. 1 user user 1155791 Nov 2 17:08 1094516954.prov.gz
> -rw-rw-r--. 1 user user 974136 Nov 18 22:07 1113402183.prov.gz
> -rw-rw-r--. 1 user user 1125608 Nov 28 22:00 1125097576.prov.gz
> -rw-rw-r--. 1 user user 1248319 Nov 29 04:30 1125432890.prov.gz
> -rw-rw-r--. 1 user user 832120 Feb 2 2021 661957813.prov.gz
> -rw-rw-r--. 1 user user 1110978 Mar 17 2021 734807613.prov.gz
> -rw-rw-r--. 1 user user 1506819 Apr 16 2021 786154249.prov.gz
> -rw-rw-r--. 1 user user 1763198 May 25 2021 852626782.prov.gz
> -rw-rw-r--. 1 user user 1580598 Jun 15 08:32 891934274.prov.gz
> -rw-rw-r--. 1 user user 2960296 Jun 28 17:07 917991812.prov.gz
> -rw-rw-r--. 1 user user 1808037 Jun 28 17:37 918051650.prov.gz
> -rw-rw-rw-. 1 user user 765924 Aug 14 13:09 991505484.prov.gz
> {code}
> BTW it's interesting why thera ere different chmods
> My config for provenance (BTW if you see posibbility for tune it, please tell
> me):
> {code:java}
> nifi.provenance.repository.directory.default=/....../provenance_repository
> nifi.provenance.repository.max.storage.time=730 days
> nifi.provenance.repository.max.storage.size=512 GB
> nifi.provenance.repository.rollover.time=10 mins
> nifi.provenance.repository.rollover.size=100 MB
> nifi.provenance.repository.query.threads=2
> nifi.provenance.repository.index.threads=1
> nifi.provenance.repository.compress.on.rollover=true
> nifi.provenance.repository.always.sync=false
> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename,
> ProcessorID
> nifi.provenance.repository.indexed.attributes=
> nifi.provenance.repository.index.shard.size=1 GB
> nifi.provenance.repository.max.attribute.length=65536
> nifi.provenance.repository.concurrent.merge.threads=1
> nifi.provenance.repository.buffer.size=100000
> {code}
> Now my provenance repo has 140GB of data.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
