Actually... I believe that is expected behavior. Since your CPU is pegged at 100% you're not going to be IO bound. Typically jobs tend to be CPU bound or IO bound. If you're CPU bound you expect to see low IO throughput. If you're IO bound, you expect to see low CPU usage.
On Thu, Oct 10, 2013 at 11:05 AM, Xuri Nagarin <secs...@gmail.com> wrote: > Hi, > > I have a simple Grep job (from bundled examples) that I am running on a > 11-node cluster. Each node is 2x8-core Intel Xeons (shows 32 CPUs with HT > on), 64GB RAM and 8 x 1TB disks. I have mappers set to 20 per node. > > When I run the Grep job, I notice that CPU gets pegged to 100% on multiple > cores but disk throughput remains a dismal 1-2 Mbytes/sec on a single disk > on each node. So I guess, the cluster is poorly performing in terms of disk > IO. Running Terasort, I see each disk puts out 25-35 Mbytes/sec with a > total cluster throughput of above 1.5 Gbytes/sec. > > How do I go about re-configuring or re-writing the job to utilize maximum > disk IO? > > TIA, > > Xuri > > >
