We are running a job that does heavy writes into a new table. The table is not pre-split so it has 1 region. I know this is not recommended; we were doing it partially to test this particular case.
Here's what we're seeing: 1. Reads are entirely blocked. No reads to any region on that server make it through. 2. Writes are insanely slow. Some writes appear to be taking over 10 minutes. 3. All of the box's resources are quiet: Around < 20% CPU usage, plenty of memory to spare, iostat looked normal 4. ngrep showed only writes coming through. no reads 5. The logs showed lots of WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call multi(org.apache.hadoop.hbase.client.MultiAction@714cd947) from 10.211.117.161:34380: output error Any ideas what's up? Is there some sort of global lock that might halt reads during heavy writes? Anything else we can look for during this? We can rerun the job to reproduce this, as this is a test cluster which can afford to be brought down.
