There is no need to set Xmx manually since Tez will automatically assign an Xmx value based on the memory actually allocated by YARN for the container (which may be higher than requested memory).
Bikas From: [email protected] [mailto:[email protected]] Sent: Monday, October 26, 2015 7:01 PM To: user <[email protected]> Subject: Re: Re: Constant Full GC making Tez Hive job take almost forever To change Tez memory footprints through Hive, you need to set the following configuration parameters: l SET hive.tez.container.size=<numerical memory value> Sets the size of the container spawned by YARN. l SET hive.tez.java.opts=-Xmx<numerical max heap size> Java command line options for Tez. For example: SET hive.tez.container.size=5120 SET hive.tez.java.opts=-Xmx4096m "hive.tez.container.size" and "hive.tez.java.opts" are the parameters that alter Tez memory settings in Hive. If "hive.tez.container.size" is set to "-1" (default value), it picks the value of "mapreduce.map.memory.mb". If "hive.tez.java.opts" is not specified, it relies on the "mapreduce.map.java.opts". Thus, if Tez specific memory settings are left as default values, memory sizes are picked from mapreduce mapper memory settings "mapreduce.map.memory.mb". Please note that the setting for "hive.tez.java.opts" must be smaller than the size specified for "hive.tez.container.size", or "mapreduce.{map|reduce}.memory.mb" if "hive.tez.container.size" is not specified. Don't forget to review both of them when setting both or either one of them to ensure "hive.tez.java.opts" is smaller then "hive.tez.container.size" or "mapreduce.{map|reduce}.java.opts" is smaller then "mapreduce.{map|reduce}.memory.mb". ________________________________ [email protected]<mailto:[email protected]> From: Juho Autio<mailto:[email protected]> Date: 2015-10-26 18:46 To: user<mailto:[email protected]> Subject: Re: Constant Full GC making Tez Hive job take almost forever Thanks Jeff! I did as you suggested and the job went through in a reasonable time. However the performance is not perfect. With mr engine the performance was better. I didn't set hive.tez.container.size before. The default seems to be -1. I wonder what that means in practice. Any way seems like quite many containers were spawned, so it must be using something rather small value. SET hive.tez.container.size; hive.tez.container.size=-1 Also I was running with this default value: hive> SET hive.auto.convert.join.noconditionaltask.size; hive.auto.convert.join.noconditionaltask.size=10000000 So I changed them to*: SET hive.tez.container.size=4096; SET hive.auto.convert.join.noconditionaltask.size=1252698795; This seems to mean that tez container size needs to be configured depending on yarn container configuration (so basically need to provide different values depending on the cluster size). If the relationship is really so strict, couldn't Tez pick a suitable container size automatically to avoid running out of memory? Any way, thanks a lot for solving my problem! Juho *) P.S. I tried to follow this guide basically: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_performance_tuning/content/hive_perf_best_pract_config_tez.html Do you know if it's with some special hive release that you can specify 4096MB (and ..noconditionaltask.size=1370MB), because for me Hive didn't accept MB suffix. Also this seems to be more or less the only document I've found about tuning Tez for Hive. I wonder if there are some other options and if I could find extensive documentation somewhere. On Fri, Oct 23, 2015 at 11:25 AM, Jianfeng (Jeff) Zhang <[email protected]<mailto:[email protected]>> wrote: Maybe your tez container size is less than container of mr. You can try increasing the container size of tez (You can check the container size of your job in RM WebUI) e.g. Set hive.tez.container.size=4096 BTW, not sure your cluster environment. I meet the similar issue under the cluster managed by ambari. By default in ambari the hive container size is less than the container size of reducer in MR. This may cause performance issue in the reduce stage when input data size is much large. Best Regard, Jeff Zhang From: Juho Autio <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Friday, October 23, 2015 at 4:08 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Constant Full GC making Tez Hive job take almost forever Hi, I'm running a Hive script with tez-0.7.0. The progress is real slow and in the container logs I'm seeing constant Full GC lines, so that there doesn't seem to be no time for the JVM to actually execute anything between the GC pauses. When running the same Hive script with mr execution engine, the job goes through normally. So there's something specific to Tez's memory usage that causes the Full GC issue. Also with similar clusters & configuration other Hive jobs have gone through with Tez just fine. This issue happens when I just add a little more data to be processed by the script. With a smaller workload it goes through also with Tez engine with the expected execution time. For example an extract from one of the container logs: application_1445328511212_0001/container_1445328511212_0001_01_000292/stdout.gz 791.208: [Full GC [PSYoungGen: 58368K->56830K(116736K)] [ParOldGen: 348914K->348909K(349184K)] 407282K->405740K(465920K) [PSPermGen: 43413K->43413K(43520K)], 1.4063790 secs] [Times: user=5.22 sys=0.04, real=1.40 secs] Heap PSYoungGen total 116736K, used 58000K [0x00000000f5500000, 0x0000000100000000, 0x0000000100000000) eden space 58368K, 99% used [0x00000000f5500000,0x00000000f8da41a0,0x00000000f8e00000) from space 58368K, 0% used [0x00000000f8e00000,0x00000000f8e00000,0x00000000fc700000) to space 58368K, 0% used [0x00000000fc700000,0x00000000fc700000,0x0000000100000000) ParOldGen total 349184K, used 348909K [0x00000000e0000000, 0x00000000f5500000, 0x00000000f5500000) object space 349184K, 99% used [0x00000000e0000000,0x00000000f54bb4b0,0x00000000f5500000) PSPermGen total 43520K, used 43413K [0x00000000d5a00000, 0x00000000d8480000, 0x00000000e0000000) object space 43520K, 99% used [0x00000000d5a00000,0x00000000d84657a8,0x00000000d8480000) If I understand the GC log correctly, it seems like ParOldGen is full and Full GC doesn't manage to free space from there. So maybe Tez has created too many objects that can't be released. It could be a memory leak. Or maybe this is just not big enough minimum heap for Tez in general? I could probably fix the problem by changing configuration somehow to simply have less containers and thus bigger heap size per container? Still, changing to bigger nodes doesn't seem like a solution that would eventually scale, so I would prefer to resolve this properly. Please, could you help me with how to troubleshoot & fix this issue? Cheers, Juho -- Juho Autio Analytics Developer Hatch Rovio Entertainment Ltd Mobile: + 358 (0)45 313 0122 [email protected]<mailto:[email protected]> www.rovio.com<http://www.rovio.com> This message and its attachments may contain confidential information and is intended solely for the attention and use of the named addressee(s). If you are not the intended recipient and / or you have received this message in error, please contact the sender immediately and delete all material you have received in this message. You are hereby notified that any use of the information, which you have received in error in whatsoever form, is strictly prohibited. Thank you for your co-operation.
