[ https://issues.apache.org/jira/browse/HBASE-21575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sergey Shelukhin updated HBASE-21575: ------------------------------------- Description: 100s of Mb of logs like this, in a tight loop: {noformat} 2018-12-08 10:27:00,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3646ms 2018-12-08 10:27:00,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3647ms 2018-12-08 10:27:00,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3647ms 2018-12-08 10:27:00,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3648ms 2018-12-08 10:27:00,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3648ms 2018-12-08 10:27:00,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3649ms 2018-12-08 10:27:00,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3649ms 2018-12-08 10:27:00,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3650ms 2018-12-08 10:27:00,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3650ms 2018-12-08 10:27:00,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3651ms 2018-12-08 10:27:00,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3653ms 2018-12-08 10:27:00,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3654ms 2018-12-08 10:27:00,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3654ms 2018-12-08 10:27:00,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3655ms 2018-12-08 10:27:00,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3655ms 2018-12-08 10:27:00,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3656ms 2018-12-08 10:27:00,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3656ms 2018-12-08 10:27:00,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3657ms 2018-12-08 10:27:00,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3658ms 2018-12-08 10:27:00,475 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3659ms 2018-12-08 10:27:00,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3660ms 2018-12-08 10:27:00,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3660ms 2018-12-08 10:27:00,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3661ms 2018-12-08 10:27:00,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3661ms 2018-12-08 10:27:00,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3662ms 2018-12-08 10:27:00,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3663ms 2018-12-08 10:27:00,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3663ms 2018-12-08 10:27:00,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3664ms 2018-12-08 10:27:00,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3664ms 2018-12-08 10:27:00,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3665ms 2018-12-08 10:27:00,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3666ms {noformat} was: 100s of Mb of logs like this: {noformat} 2018-12-08 10:27:00,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3646ms 2018-12-08 10:27:00,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3647ms 2018-12-08 10:27:00,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3647ms 2018-12-08 10:27:00,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3648ms 2018-12-08 10:27:00,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3648ms 2018-12-08 10:27:00,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3649ms 2018-12-08 10:27:00,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3649ms 2018-12-08 10:27:00,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3650ms 2018-12-08 10:27:00,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3650ms 2018-12-08 10:27:00,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3651ms 2018-12-08 10:27:00,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3653ms 2018-12-08 10:27:00,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3654ms 2018-12-08 10:27:00,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3654ms 2018-12-08 10:27:00,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3655ms 2018-12-08 10:27:00,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3655ms 2018-12-08 10:27:00,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3656ms 2018-12-08 10:27:00,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3656ms 2018-12-08 10:27:00,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3657ms 2018-12-08 10:27:00,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3658ms 2018-12-08 10:27:00,475 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3659ms 2018-12-08 10:27:00,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3660ms 2018-12-08 10:27:00,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3660ms 2018-12-08 10:27:00,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3661ms 2018-12-08 10:27:00,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3661ms 2018-12-08 10:27:00,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3662ms 2018-12-08 10:27:00,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3663ms 2018-12-08 10:27:00,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3663ms 2018-12-08 10:27:00,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3664ms 2018-12-08 10:27:00,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3664ms 2018-12-08 10:27:00,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3665ms 2018-12-08 10:27:00,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3666ms {noformat} > memstore above high watermark message is logged too much > -------------------------------------------------------- > > Key: HBASE-21575 > URL: https://issues.apache.org/jira/browse/HBASE-21575 > Project: HBase > Issue Type: Bug > Reporter: Sergey Shelukhin > Assignee: Sergey Shelukhin > Priority: Major > > 100s of Mb of logs like this, in a tight loop: > {noformat} > 2018-12-08 10:27:00,462 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3646ms > 2018-12-08 10:27:00,463 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3647ms > 2018-12-08 10:27:00,463 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3647ms > 2018-12-08 10:27:00,464 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3648ms > 2018-12-08 10:27:00,464 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3648ms > 2018-12-08 10:27:00,465 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3649ms > 2018-12-08 10:27:00,465 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3649ms > 2018-12-08 10:27:00,466 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3650ms > 2018-12-08 10:27:00,466 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3650ms > 2018-12-08 10:27:00,467 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3651ms > 2018-12-08 10:27:00,469 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3653ms > 2018-12-08 10:27:00,470 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3654ms > 2018-12-08 10:27:00,470 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3654ms > 2018-12-08 10:27:00,471 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3655ms > 2018-12-08 10:27:00,471 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3655ms > 2018-12-08 10:27:00,472 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3656ms > 2018-12-08 10:27:00,472 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3656ms > 2018-12-08 10:27:00,473 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3657ms > 2018-12-08 10:27:00,474 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3658ms > 2018-12-08 10:27:00,475 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3659ms > 2018-12-08 10:27:00,476 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3660ms > 2018-12-08 10:27:00,476 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3660ms > 2018-12-08 10:27:00,477 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3661ms > 2018-12-08 10:27:00,477 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3661ms > 2018-12-08 10:27:00,478 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3662ms > 2018-12-08 10:27:00,479 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3663ms > 2018-12-08 10:27:00,479 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3663ms > 2018-12-08 10:27:00,480 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3664ms > 2018-12-08 10:27:00,480 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3664ms > 2018-12-08 10:27:00,481 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3665ms > 2018-12-08 10:27:00,482 WARN > [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] > regionserver.MemStoreFlusher: Memstore is above high water mark and block > 3666ms > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)