t the time when this happens no other job is running
> on cluster. All mappers (3) has already finished and we have single reduce
> task which fails at ~ 70% of its progress on virtual memory consumption.
> Dataset which is processing is 500MB of avro data file compressed. Reducer
> doesn&
single reduce
task which fails at ~ 70% of its progress on virtual memory consumption.
Dataset which is processing is 500MB of avro data file compressed. Reducer
doesn't cache anything intentionally, just divide a records in various
folders dynamically.
>From RM console I clearly see that there i
Your physical memory is 1GB on this node.
What are the other containers (map tasks) running on this?
You have given map memory as 768M and reduce memory as 1024M and am as 1024M.
With AM and a single map task it is 1.7M and cannot start another
container for reducer.
Reduce these values and chec
Hello hadoop users,
I am facing following issue when running M/R job during a reduce phase:
Container [pid=22961,containerID=container_1409834588043_0080_01_10] is
running beyond virtual memory limits. Current usage: 636.6 MB of 1 GB
physical memory used; 2.1 GB of 2.1 GB virtual memory used.
