In hadoop-env.sh, I have set
export HADOOP_HEAPSIZE_MAX=30000
export HADOOP_HEAPSIZE_MIN=30000
And then restarted my Hadoop cluster with stop-all.sh and start-all.sh
When I print my Hadoop environment variables with hadoop envvar, the above
two variables aren't printed out.
I have a small program -
int main() {
hdfsFS x2 = hdfsConnect("127.0.0.1", 9000);
hdfsFile x3 = hdfsOpenFile(x2, "hdfs://127.0.0.1:9000/10G.txt", 0, 0, 0,
0); // A 10GB file
int64_t size = hdfsAvailable(x2, x3);
char *x5 = malloc(sizeof(char) * size);
hdfsPread(x2, x3, 0, x5, size);
free(x5);
hdfsCloseFile(x2, x3);
return 1;
}
that errors on hdfsPread with the error log -
hdfsPread: NewByteArray error:
OutOfMemoryError: Requestedarray size exceeds VM
limitjava.lang.OutOfMemoryError: Requested array size exceeds VM limit
It looks like the HEAPSIZE I set is not being respected. Could someone
please help me with this?
Do I need to format namenode for the new environment variables to be used?
--
Pratyush Das
