Hi Guys One quick question. I've read that when pig is running in TEZ mode, intermediate files are not stored in HDFS, are kept in memory instead. So that it's available to next task which is going to reuse our container. My question is how does TEZ prevent a case of OOME, cause not writing to disk and keeping everything in memory might cause OOME. Is there any threshold after which files are written to HDFS? Am i missing something here?
Thanks -- Sachin Sabbarwal Linkedin: https://www.linkedin.com/profile?viewProfile=&key=95777265 Facebook: facebook.com/sachinsabbarwal Quora: http://www.quora.com/Sachin-Sabbarwal Blog: http://sachinsabbarwal.tumblr.com/
