This issue is introduced by LedgerCacheImpl#deleteLedger. It doesn't return the ledger pages that deleted ledgers occupied back to LedgerCache. So the ledger cache pages are exhausted.
I attached a patch to fix it on https://issues.apache.org/jira/browse/BOOKKEEPER-287 . Thanks John for reporting this issue. - Sijie On Wed, Jun 6, 2012 at 6:57 AM, John Nagro <[email protected]> wrote: > I'm sorry, but I don't think so.... basically, the server was running fine > (4.1), there were fencing messages but that seems to be normal, and then > the exceptions started: > > 2012-06-05 15:12:38,107 - WARN [NIOServerFactory-3181:BookieServer@389] > - Ledger 759036 fenced by > Socket[addr=/***REDACTED***,port=54294,localport=3181] > 2012-06-05 15:12:43,818 - WARN [NIOServerFactory-3181:BookieServer@389] > - Ledger 759043 fenced by > Socket[addr=/***REDACTED***,port=54294,localport=3181] > 2012-06-05 15:12:44,672 - WARN [NIOServerFactory-3181:BookieServer@389] > - Ledger 759057 fenced by > Socket[addr=/***REDACTED***,port=54294,localport=3181] > 2012-06-05 15:12:54,018 - WARN [NIOServerFactory-3181:BookieServer@389] > - Ledger 759066 fenced by > Socket[addr=/***REDACTED***,port=54294,localport=3181] > 2012-06-05 15:12:54,124 - WARN [NIOServerFactory-3181:BookieServer@389] > - Ledger 759068 fenced by > Socket[addr=/***REDACTED***,port=54294,localport=3181] > 2012-06-05 15:13:02,857 - WARN [NIOServerFactory-3181:NIOServerFactory@128] > - Exception in server socket loop: /0.0.0.0 > java.util.NoSuchElementException > at java.util.LinkedList.getFirst(LinkedList.java:109) > at > org.apache.bookkeeper.bookie.LedgerCacheImpl.grabCleanPage(LedgerCacheImpl.java:478) > at > org.apache.bookkeeper.bookie.LedgerCacheImpl.grabLedgerEntryPage(LedgerCacheImpl.java:169) > at > org.apache.bookkeeper.bookie.LedgerCacheImpl.putEntryOffset(LedgerCacheImpl.java:199) > at > org.apache.bookkeeper.bookie.InterleavedLedgerStorage.addEntry(InterleavedLedgerStorage.java:109) > at > org.apache.bookkeeper.bookie.LedgerDescriptorImpl.addEntry(LedgerDescriptorImpl.java:81) > at > org.apache.bookkeeper.bookie.Bookie.addEntryInternal(Bookie.java:656) > at org.apache.bookkeeper.bookie.Bookie.addEntry(Bookie.java:691) > at > org.apache.bookkeeper.proto.BookieServer.processPacket(BookieServer.java:368) > at > org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.readRequest(NIOServerFactory.java:310) > at > org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.doIO(NIOServerFactory.java:208) > at > org.apache.bookkeeper.proto.NIOServerFactory.run(NIOServerFactory.java:123) > 2012-06-05 15:13:04,033 - WARN [NIOServerFactory-3181:BookieServer@389] > - Ledger 759078 fenced by > Socket[addr=/***REDACTED***,port=54294,localport=3181] > 2012-06-05 15:13:04,034 - WARN [NIOServerFactory-3181:NIOServerFactory@128] > - Exception in server socket loop: /0.0.0.0 > java.util.NoSuchElementException > at java.util.LinkedList.getFirst(LinkedList.java:109) > at > org.apache.bookkeeper.bookie.LedgerCacheImpl.grabCleanPage(LedgerCacheImpl.java:478) > at > org.apache.bookkeeper.bookie.LedgerCacheImpl.grabLedgerEntryPage(LedgerCacheImpl.java:169) > at > org.apache.bookkeeper.bookie.LedgerCacheImpl.getEntryOffset(LedgerCacheImpl.java:217) > at > org.apache.bookkeeper.bookie.InterleavedLedgerStorage.getEntry(InterleavedLedgerStorage.java:126) > at > org.apache.bookkeeper.bookie.LedgerDescriptorImpl.readEntry(LedgerDescriptorImpl.java:86) > at org.apache.bookkeeper.bookie.Bookie.readEntry(Bookie.java:715) > at > org.apache.bookkeeper.proto.BookieServer.processPacket(BookieServer.java:400) > at > org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.readRequest(NIOServerFactory.java:310) > at > org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.doIO(NIOServerFactory.java:208) > at > org.apache.bookkeeper.proto.NIOServerFactory.run(NIOServerFactory.java:123) > 2012-06-05 15:13:08,868 - WARN [NIOServerFactory-3181:NIOServerFactory@128] > - Exception in server socket loop: /0.0.0.0 > java.util.NoSuchElementException > at java.util.LinkedList.getFirst(LinkedList.java:109) > at > org.apache.bookkeeper.bookie.LedgerCacheImpl.grabCleanPage(LedgerCacheImpl.java:478) > at > org.apache.bookkeeper.bookie.LedgerCacheImpl.grabLedgerEntryPage(LedgerCacheImpl.java:169) > at > org.apache.bookkeeper.bookie.LedgerCacheImpl.putEntryOffset(LedgerCacheImpl.java:199) > at > org.apache.bookkeeper.bookie.InterleavedLedgerStorage.addEntry(InterleavedLedgerStorage.java:109) > at > org.apache.bookkeeper.bookie.LedgerDescriptorImpl.addEntry(LedgerDescriptorImpl.java:81) > at > org.apache.bookkeeper.bookie.Bookie.addEntryInternal(Bookie.java:656) > at org.apache.bookkeeper.bookie.Bookie.addEntry(Bookie.java:691) > at > org.apache.bookkeeper.proto.BookieServer.processPacket(BookieServer.java:368) > at > org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.readRequest(NIOServerFactory.java:310) > at > org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.doIO(NIOServerFactory.java:208) > at > org.apache.bookkeeper.proto.NIOServerFactory.run(NIOServerFactory.java:123) > 2012-06-05 15:13:10,950 - WARN [NIOServerFactory-3181:NIOServerFactory@128] > - Exception in server socket loop: /0.0.0.0 > java.util.NoSuchElementException > at java.util.LinkedList.getFirst(LinkedList.java:109) > at > org.apache.bookkeeper.bookie.LedgerCacheImpl.grabCleanPage(LedgerCacheImpl.java:478) > at > org.apache.bookkeeper.bookie.LedgerCacheImpl.grabLedgerEntryPage(LedgerCacheImpl.java:169) > at > org.apache.bookkeeper.bookie.LedgerCacheImpl.putEntryOffset(LedgerCacheImpl.java:199) > at > org.apache.bookkeeper.bookie.InterleavedLedgerStorage.addEntry(InterleavedLedgerStorage.java:109) > at > org.apache.bookkeeper.bookie.LedgerDescriptorImpl.addEntry(LedgerDescriptorImpl.java:81) > at > org.apache.bookkeeper.bookie.Bookie.addEntryInternal(Bookie.java:656) > at org.apache.bookkeeper.bookie.Bookie.addEntry(Bookie.java:691) > at > org.apache.bookkeeper.proto.BookieServer.processPacket(BookieServer.java:368) > at > org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.readRequest(NIOServerFactory.java:310) > at > org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.doIO(NIOServerFactory.java:208) > at > org.apache.bookkeeper.proto.NIOServerFactory.run(NIOServerFactory.java:123) > 2012-06-05 15:13:14,649 - WARN [NIOServerFactory-3181:BookieServer@389] > - Ledger 759087 fenced by > Socket[addr=/***REDACTED***,port=54294,localport=3181] > 2012-06-05 15:13:14,649 - WARN [NIOServerFactory-3181:NIOServerFactory@128] > - Exception in server socket loop: /0.0.0.0 > java.util.NoSuchElementException > at java.util.LinkedList.getFirst(LinkedList.java:109) > at > org.apache.bookkeeper.bookie.LedgerCacheImpl.grabCleanPage(LedgerCacheImpl.java:478) > at > org.apache.bookkeeper.bookie.LedgerCacheImpl.grabLedgerEntryPage(LedgerCacheImpl.java:169) > at > org.apache.bookkeeper.bookie.LedgerCacheImpl.getEntryOffset(LedgerCacheImpl.java:217) > at > org.apache.bookkeeper.bookie.InterleavedLedgerStorage.getEntry(InterleavedLedgerStorage.java:126) > at > org.apache.bookkeeper.bookie.LedgerDescriptorImpl.readEntry(LedgerDescriptorImpl.java:86) > at org.apache.bookkeeper.bookie.Bookie.readEntry(Bookie.java:715) > at > org.apache.bookkeeper.proto.BookieServer.processPacket(BookieServer.java:400) > at > org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.readRequest(NIOServerFactory.java:310) > at > org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.doIO(NIOServerFactory.java:208) > at > org.apache.bookkeeper.proto.NIOServerFactory.run(NIOServerFactory.java:123) > > On Tue, Jun 5, 2012 at 6:27 PM, Flavio Junqueira <[email protected]>wrote: > >> Thanks for reporting, John. It is a write: at >> org.apache.bookkeeper.bookie.Bookie.addEntry(Bookie.java:691) >> >> Is this all you have or you have more logs/information, John? >> >> -Flavio >> >> >> On Jun 5, 2012, at 11:05 PM, John Nagro wrote: >> >> 2012-06-05 16:24:29,596 - WARN >> [NIOServerFactory-3181:NIOServerFactory@128] - Exception in server >> socket loop: /0.0.0.0 >> java.util.NoSuchElementException >> at java.util.LinkedList.getFirst(LinkedList.java:109) >> at >> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabCleanPage(LedgerCacheImpl.java:478) >> at >> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabLedgerEntryPage(LedgerCacheImpl.java:169) >> at >> org.apache.bookkeeper.bookie.LedgerCacheImpl.putEntryOffset(LedgerCacheImpl.java:199) >> at >> org.apache.bookkeeper.bookie.InterleavedLedgerStorage.addEntry(InterleavedLedgerStorage.java:109) >> at >> org.apache.bookkeeper.bookie.LedgerDescriptorImpl.addEntry(LedgerDescriptorImpl.java:81) >> at >> org.apache.bookkeeper.bookie.Bookie.addEntryInternal(Bookie.java:656) >> at org.apache.bookkeeper.bookie.Bookie.addEntry(Bookie.java:691) >> at >> org.apache.bookkeeper.proto.BookieServer.processPacket(BookieServer.java:368) >> at >> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.readRequest(NIOServerFactory.java:310) >> at >> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.doIO(NIOServerFactory.java:208) >> at >> org.apache.bookkeeper.proto.NIOServerFactory.run(NIOServerFactory.java:123) >> >> unfortunately, i am unsure about what causes this (a read or a write). >> >> >> >> >
