Re: Journal Corruption

2019-10-12 Thread Venkateswara Rao Jujjuri
may lose important bits, as > 'fence' bits > > Enrico > > Il sab 12 ott 2019, 19:18 Venkateswara Rao Jujjuri ha > scritto: > > > We have observed Journal Corruption while replaying the journal log. > > I could be missing something but I don't see we

Re: Journal Corruption

2019-10-12 Thread Enrico Olivelli
the end of the journal but if the file was corrupted we may lose important bits, as 'fence' bits Enrico Il sab 12 ott 2019, 19:18 Venkateswara Rao Jujjuri ha scritto: > We have observed Journal Corruption while replaying the journal log. > I could be missing something but I don

Journal Corruption

2019-10-12 Thread Venkateswara Rao Jujjuri
We have observed Journal Corruption while replaying the journal log. I could be missing something but I don't see we have clear indication of where journal file ends. In the Journal.java:scanJournal() we have multiple breaks but mostly we assume either the journal file ends in the right pla

Re: Journal Corruption when disk is full

2018-02-17 Thread Ivan Kelly
On Fri, Feb 16, 2018 at 9:48 PM, Matteo Merli wrote: >> One thing that should circumvent this is that the bookie should go >> into readonly mode when it hits 95% full disk. > > I think this is only applying to the ledgers disk, but not for the journal. That doesn't seem like the right thing to do

Re: Journal Corruption when disk is full

2018-02-16 Thread Matteo Merli
> One thing that should circumvent this is that the bookie should go > into readonly mode when it hits 95% full disk. I think this is only applying to the ledgers disk, but not for the journal. And, to answer to Bobby, the switch to read-only mode feature was already present in 4.3 (again, just f

Re: Journal Corruption when disk is full

2018-02-15 Thread Ivan Kelly
On Thu, Feb 15, 2018 at 9:49 PM, Bobby Evans wrote: > I don't have the read only mode on disk full feature yet. I will look at > pulling it back to our fork, but I will also look at fixing the journaling > in general. Having spoken with the HDFS team here, they have seen a lot of > scary things

Re: Journal Corruption when disk is full

2018-02-15 Thread Bobby Evans
I don't have the read only mode on disk full feature yet. I will look at pulling it back to our fork, but I will also look at fixing the journaling in general. Having spoken with the HDFS team here, they have seen a lot of scary things that appear similar to this situation when a disk starts to g

Re: Journal Corruption when disk is full

2018-02-15 Thread Enrico Olivelli
Il gio 15 feb 2018, 18:45 Ivan Kelly ha scritto: > Hi Bobby, > > One thing that should circumvent this is that the bookie should go > into readonly mode when it hits 95% full disk. > > This change is in your branch[1]. Have you disabled it? > > Regards, > Ivan > > [1] > https://github.com/yahoo/b

Re: Journal Corruption when disk is full

2018-02-15 Thread Ivan Kelly
Hi Bobby, One thing that should circumvent this is that the bookie should go into readonly mode when it hits 95% full disk. This change is in your branch[1]. Have you disabled it? Regards, Ivan [1] https://github.com/yahoo/bookkeeper/blob/7fb556fa2dbc1d308b5d7ec3e2676b8b11704698/bookkeeper-ser

Journal Corruption when disk is full

2018-02-15 Thread Bobby Evans
We recently had a situation where the disk a journal is on filled up and we ended up with a partial edit in the journal. This ended up being a METAENTRY_ID_LEDGER_KEY so only a partial master key was output and this ended up making the bookie fail on startup. Because we are running a forked versi