At time of writing in my test env I have: - 132 .txn files
- Most of them are 16777216 bytes - I have 0 ledgers - journalMaxSizeMB=1048 - logSizeLimit (entryLogSizeLimit)= 2147483648 - size of directory referred by ledgerDirNames is 2.8MB honestly I can’t estimate the effective size of data which was written to my bookie I do not have any automatic test case useful to reproduce the problem The procedure is simple, the are two main use cases: 1) Open ledger and write…..close. drop it , close le client (most common case of course) 2) Open ledger and write, if an error occurs (on other resource or during writes to BK), try to close the ledger, try to close the client, then from another procedure (usually on another machine) open the ledger (with fencing), read all the entries, close the ledger, drop the ledger a. Important Subcase: errors during the “recovery”…then we need to retry recovery after some time Da: Sijie Guo [mailto:[email protected]] Inviato: giovedì 9 luglio 2015 18:30 A: [email protected]; Enrico Olivelli - Diennea Oggetto: Re: Journal file never deleted The org.apache.bookkeeper.bookie.CheckpointSource$Checkpoint are the inner classes 'MAX' and 'MIN'. They are expected if entry log file isn't rotated. As the checkpoint logic in bookie is only checkpoint when entry log file rotated, for performance consideration. I am not sure if we are on same page. deleting ledgers & stopping writing entries won't help deleting journal files. The way how journal files rotated and deleted works as below: - all entries are appended in journal - entries are added into ledger storage and periodic flushed to disk. - once the ledger storage is flushed, the log mark will be updated, indicating entries added before the mark are persisted. then the journals before the mark is able to be deleted. - so if there is only one journal, it won't be deleted. if there are more journal files, it would only delete journals that before log mark and keep #getMaxBackupJournals journals before log mark. so it would be good that you could share more details on the problem you encountered: - how many journal files you have? - how many entry log files (.log) you have? - what is the your journal max size? 'journalMaxSizeMB' - what is your size of entry log file? 'logSizeLimit' - how many data you have written? - how's your procedure on reproducing this behavior? - Sijie On Thu, Jul 9, 2015 at 12:13 AM, Enrico Olivelli - Diennea <[email protected]<mailto:[email protected]>> wrote: I reproduced the same problem in a simple staging environment, I deleted every ledger and stopped writing new entries. I’m trying to do a little debug and I see that in Journal.java there is this line if (!(checkpoint instanceof LogMarkCheckpoint)) { return; // we didn't create this checkpoint, so dont do anything with it } Actually my test bookie gets a org.apache.bookkeeper.bookie.CheckpointSource$Checkpoint$1@98aabc2<mailto:org.apache.bookkeeper.bookie.CheckpointSource$Checkpoint$1@98aabc2> class instead of a LogMarkCheclpoint, when the checkpoint is done in the SynchThread#checkpoint method at every FlushInterval hope this can help Enrico Da: Sijie Guo [mailto:[email protected]<mailto:[email protected]>] Inviato: mercoledì 8 luglio 2015 19:06 A: [email protected]<mailto:[email protected]> Oggetto: Re: Journal file never deleted The journal files are only deleted when its corresponding data in ledger storage is flushed. What is your rate on adding entries? How many .txn files you have? On Wed, Jul 8, 2015 at 3:27 AM, Enrico Olivelli - Diennea <[email protected]<mailto:[email protected]>> wrote: I’m running all the default values, except from maxbackups which I set to 1 Even if a delete every ledger the logs never get deleted. I enabled level DEBUG for every logger but I get no useful hint. Normally at which rate logs file should be deleted ? Thanks Enrico Da: Flavio Junqueira [mailto:[email protected]<mailto:[email protected]>] Inviato: mercoledì 8 luglio 2015 12:17 A: [email protected]<mailto:[email protected]> Oggetto: Re: Journal file never deleted Hi Enrico, Have you had a look at the journal settings here? http://bookkeeper.apache.org/docs/r4.3.1/bookieConfigParams.html -Flavio On 08 Jul 2015, at 10:37, Enrico Olivelli - Diennea <[email protected]<mailto:[email protected]>> wrote: Hi, I’m running BookKeeper 4.3.0 (and 4.3.1). I noticed that journal files (*.txn files) never get deleted. I’m using ServerConfiguration.setMaxBackupJournals(1) and default flushinterval of 1 second. I’m running “embedded” bookies, that is that I launch the bookie inside another JVM, with the following code: ServerConfiguration conf = new ServerConfiguration(); conf.setBookiePort(port); conf.setUseHostNameAsBookieID(true); conf.setJournalDirName(logDir.getAbsolutePath()); conf.setZkServers(zookeeperServer); conf.setLedgerDirNames(Arrays.stream(dataDirs).map(File::getAbsolutePath).collect(Collectors.toList()).toArray(new String[dataDirs.length])); conf.setFlushInterval(1000); conf.setAutoRecoveryDaemonEnabled(false); … conf.setMaxBackupJournals(1); … this.bookie = new BookieServer(conf); this.bookee.start(); any suggestion ? thanks Enrico Olivelli Software Development Manager @Diennea Tel.: (+39) 0546 066100<tel:%28%2B39%29%C2%A00546%C2%A0066100> - Int. 925 Viale G.Marconi 30/14 - 48018 Faenza (RA) MagNews - E-mail Marketing Solutions http://www.magnews.it<http://www.magnews.it/> Diennea - Digital Marketing Solutions http://www.diennea.com<http://www.diennea.com/> ________________________________ Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing! http://www.magnews.it/newsletter/ ________________________________ Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing! http://www.magnews.it/newsletter/ ________________________________ Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing! http://www.magnews.it/newsletter/ ________________________________ Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing! http://www.magnews.it/newsletter/
