Thanks for looking into it. -John
On Wed, Jun 6, 2012 at 1:13 AM, Sijie Guo <[email protected]> wrote: > This issue is introduced by LedgerCacheImpl#deleteLedger. It doesn't > return the ledger pages that deleted ledgers occupied back to LedgerCache. > So the ledger cache pages are exhausted. > > I attached a patch to fix it on > https://issues.apache.org/jira/browse/BOOKKEEPER-287 . > > Thanks John for reporting this issue. > > - > Sijie > > > On Wed, Jun 6, 2012 at 6:57 AM, John Nagro <[email protected]> wrote: > >> I'm sorry, but I don't think so.... basically, the server was running >> fine (4.1), there were fencing messages but that seems to be normal, and >> then the exceptions started: >> >> 2012-06-05 15:12:38,107 - WARN [NIOServerFactory-3181:BookieServer@389] >> - Ledger 759036 fenced by >> Socket[addr=/***REDACTED***,port=54294,localport=3181] >> 2012-06-05 15:12:43,818 - WARN [NIOServerFactory-3181:BookieServer@389] >> - Ledger 759043 fenced by >> Socket[addr=/***REDACTED***,port=54294,localport=3181] >> 2012-06-05 15:12:44,672 - WARN [NIOServerFactory-3181:BookieServer@389] >> - Ledger 759057 fenced by >> Socket[addr=/***REDACTED***,port=54294,localport=3181] >> 2012-06-05 15:12:54,018 - WARN [NIOServerFactory-3181:BookieServer@389] >> - Ledger 759066 fenced by >> Socket[addr=/***REDACTED***,port=54294,localport=3181] >> 2012-06-05 15:12:54,124 - WARN [NIOServerFactory-3181:BookieServer@389] >> - Ledger 759068 fenced by >> Socket[addr=/***REDACTED***,port=54294,localport=3181] >> 2012-06-05 15:13:02,857 - WARN >> [NIOServerFactory-3181:NIOServerFactory@128] - Exception in server >> socket loop: /0.0.0.0 >> java.util.NoSuchElementException >> at java.util.LinkedList.getFirst(LinkedList.java:109) >> at >> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabCleanPage(LedgerCacheImpl.java:478) >> at >> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabLedgerEntryPage(LedgerCacheImpl.java:169) >> at >> org.apache.bookkeeper.bookie.LedgerCacheImpl.putEntryOffset(LedgerCacheImpl.java:199) >> at >> org.apache.bookkeeper.bookie.InterleavedLedgerStorage.addEntry(InterleavedLedgerStorage.java:109) >> at >> org.apache.bookkeeper.bookie.LedgerDescriptorImpl.addEntry(LedgerDescriptorImpl.java:81) >> at >> org.apache.bookkeeper.bookie.Bookie.addEntryInternal(Bookie.java:656) >> at org.apache.bookkeeper.bookie.Bookie.addEntry(Bookie.java:691) >> at >> org.apache.bookkeeper.proto.BookieServer.processPacket(BookieServer.java:368) >> at >> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.readRequest(NIOServerFactory.java:310) >> at >> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.doIO(NIOServerFactory.java:208) >> at >> org.apache.bookkeeper.proto.NIOServerFactory.run(NIOServerFactory.java:123) >> 2012-06-05 15:13:04,033 - WARN [NIOServerFactory-3181:BookieServer@389] >> - Ledger 759078 fenced by >> Socket[addr=/***REDACTED***,port=54294,localport=3181] >> 2012-06-05 15:13:04,034 - WARN >> [NIOServerFactory-3181:NIOServerFactory@128] - Exception in server >> socket loop: /0.0.0.0 >> java.util.NoSuchElementException >> at java.util.LinkedList.getFirst(LinkedList.java:109) >> at >> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabCleanPage(LedgerCacheImpl.java:478) >> at >> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabLedgerEntryPage(LedgerCacheImpl.java:169) >> at >> org.apache.bookkeeper.bookie.LedgerCacheImpl.getEntryOffset(LedgerCacheImpl.java:217) >> at >> org.apache.bookkeeper.bookie.InterleavedLedgerStorage.getEntry(InterleavedLedgerStorage.java:126) >> at >> org.apache.bookkeeper.bookie.LedgerDescriptorImpl.readEntry(LedgerDescriptorImpl.java:86) >> at org.apache.bookkeeper.bookie.Bookie.readEntry(Bookie.java:715) >> at >> org.apache.bookkeeper.proto.BookieServer.processPacket(BookieServer.java:400) >> at >> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.readRequest(NIOServerFactory.java:310) >> at >> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.doIO(NIOServerFactory.java:208) >> at >> org.apache.bookkeeper.proto.NIOServerFactory.run(NIOServerFactory.java:123) >> 2012-06-05 15:13:08,868 - WARN >> [NIOServerFactory-3181:NIOServerFactory@128] - Exception in server >> socket loop: /0.0.0.0 >> java.util.NoSuchElementException >> at java.util.LinkedList.getFirst(LinkedList.java:109) >> at >> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabCleanPage(LedgerCacheImpl.java:478) >> at >> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabLedgerEntryPage(LedgerCacheImpl.java:169) >> at >> org.apache.bookkeeper.bookie.LedgerCacheImpl.putEntryOffset(LedgerCacheImpl.java:199) >> at >> org.apache.bookkeeper.bookie.InterleavedLedgerStorage.addEntry(InterleavedLedgerStorage.java:109) >> at >> org.apache.bookkeeper.bookie.LedgerDescriptorImpl.addEntry(LedgerDescriptorImpl.java:81) >> at >> org.apache.bookkeeper.bookie.Bookie.addEntryInternal(Bookie.java:656) >> at org.apache.bookkeeper.bookie.Bookie.addEntry(Bookie.java:691) >> at >> org.apache.bookkeeper.proto.BookieServer.processPacket(BookieServer.java:368) >> at >> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.readRequest(NIOServerFactory.java:310) >> at >> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.doIO(NIOServerFactory.java:208) >> at >> org.apache.bookkeeper.proto.NIOServerFactory.run(NIOServerFactory.java:123) >> 2012-06-05 15:13:10,950 - WARN >> [NIOServerFactory-3181:NIOServerFactory@128] - Exception in server >> socket loop: /0.0.0.0 >> java.util.NoSuchElementException >> at java.util.LinkedList.getFirst(LinkedList.java:109) >> at >> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabCleanPage(LedgerCacheImpl.java:478) >> at >> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabLedgerEntryPage(LedgerCacheImpl.java:169) >> at >> org.apache.bookkeeper.bookie.LedgerCacheImpl.putEntryOffset(LedgerCacheImpl.java:199) >> at >> org.apache.bookkeeper.bookie.InterleavedLedgerStorage.addEntry(InterleavedLedgerStorage.java:109) >> at >> org.apache.bookkeeper.bookie.LedgerDescriptorImpl.addEntry(LedgerDescriptorImpl.java:81) >> at >> org.apache.bookkeeper.bookie.Bookie.addEntryInternal(Bookie.java:656) >> at org.apache.bookkeeper.bookie.Bookie.addEntry(Bookie.java:691) >> at >> org.apache.bookkeeper.proto.BookieServer.processPacket(BookieServer.java:368) >> at >> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.readRequest(NIOServerFactory.java:310) >> at >> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.doIO(NIOServerFactory.java:208) >> at >> org.apache.bookkeeper.proto.NIOServerFactory.run(NIOServerFactory.java:123) >> 2012-06-05 15:13:14,649 - WARN [NIOServerFactory-3181:BookieServer@389] >> - Ledger 759087 fenced by >> Socket[addr=/***REDACTED***,port=54294,localport=3181] >> 2012-06-05 15:13:14,649 - WARN >> [NIOServerFactory-3181:NIOServerFactory@128] - Exception in server >> socket loop: /0.0.0.0 >> java.util.NoSuchElementException >> at java.util.LinkedList.getFirst(LinkedList.java:109) >> at >> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabCleanPage(LedgerCacheImpl.java:478) >> at >> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabLedgerEntryPage(LedgerCacheImpl.java:169) >> at >> org.apache.bookkeeper.bookie.LedgerCacheImpl.getEntryOffset(LedgerCacheImpl.java:217) >> at >> org.apache.bookkeeper.bookie.InterleavedLedgerStorage.getEntry(InterleavedLedgerStorage.java:126) >> at >> org.apache.bookkeeper.bookie.LedgerDescriptorImpl.readEntry(LedgerDescriptorImpl.java:86) >> at org.apache.bookkeeper.bookie.Bookie.readEntry(Bookie.java:715) >> at >> org.apache.bookkeeper.proto.BookieServer.processPacket(BookieServer.java:400) >> at >> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.readRequest(NIOServerFactory.java:310) >> at >> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.doIO(NIOServerFactory.java:208) >> at >> org.apache.bookkeeper.proto.NIOServerFactory.run(NIOServerFactory.java:123) >> >> On Tue, Jun 5, 2012 at 6:27 PM, Flavio Junqueira <[email protected]>wrote: >> >>> Thanks for reporting, John. It is a write: at >>> org.apache.bookkeeper.bookie.Bookie.addEntry(Bookie.java:691) >>> >>> Is this all you have or you have more logs/information, John? >>> >>> -Flavio >>> >>> >>> On Jun 5, 2012, at 11:05 PM, John Nagro wrote: >>> >>> 2012-06-05 16:24:29,596 - WARN >>> [NIOServerFactory-3181:NIOServerFactory@128] - Exception in server >>> socket loop: /0.0.0.0 >>> java.util.NoSuchElementException >>> at java.util.LinkedList.getFirst(LinkedList.java:109) >>> at >>> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabCleanPage(LedgerCacheImpl.java:478) >>> at >>> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabLedgerEntryPage(LedgerCacheImpl.java:169) >>> at >>> org.apache.bookkeeper.bookie.LedgerCacheImpl.putEntryOffset(LedgerCacheImpl.java:199) >>> at >>> org.apache.bookkeeper.bookie.InterleavedLedgerStorage.addEntry(InterleavedLedgerStorage.java:109) >>> at >>> org.apache.bookkeeper.bookie.LedgerDescriptorImpl.addEntry(LedgerDescriptorImpl.java:81) >>> at >>> org.apache.bookkeeper.bookie.Bookie.addEntryInternal(Bookie.java:656) >>> at org.apache.bookkeeper.bookie.Bookie.addEntry(Bookie.java:691) >>> at >>> org.apache.bookkeeper.proto.BookieServer.processPacket(BookieServer.java:368) >>> at >>> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.readRequest(NIOServerFactory.java:310) >>> at >>> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.doIO(NIOServerFactory.java:208) >>> at >>> org.apache.bookkeeper.proto.NIOServerFactory.run(NIOServerFactory.java:123) >>> >>> unfortunately, i am unsure about what causes this (a read or a write). >>> >>> >>> >>> >> >
