That might be the truth. When I switched to a 4gb node it just worked. Thanks
On Tue, Mar 29, 2022 at 8:49 PM Pau Tallada <tall...@pic.es> wrote: > I don't know what to say. > It it fails of OutOfMemory, then you have to assign more memory to it. > > Also a, 2GB VM for a hadoop node is too tiny. Hadoop ecosystem is usually > memory-intensive > > Missatge de Bitfox <bit...@bitfox.top> del dia dt., 29 de març 2022 a les > 14:46: > >> Yes, a quite small table with 10000 rows for test purposes. >> >> Thanks >> >> On Tue, Mar 29, 2022 at 8:43 PM Pau Tallada <tall...@pic.es> wrote: >> >>> Hi, >>> >>> I think it depends a lot on the data volume you are trying to process. >>> Does it work with a smaller table? >>> >>> Missatge de Bitfox <bit...@bitfox.top> del dia dt., 29 de març 2022 a >>> les 14:39: >>> >>>> 0: jdbc:hive2://localhost:10000/default> set >>>> hive.tez.container.size=1024; >>>> >>>> No rows affected (0.027 seconds) >>>> >>>> >>>> 0: jdbc:hive2://localhost:10000/default> set hive.execution.engine; >>>> >>>> +---------------------------+ >>>> >>>> | set | >>>> >>>> +---------------------------+ >>>> >>>> | hive.execution.engine=mr | >>>> >>>> +---------------------------+ >>>> >>>> 1 row selected (0.048 seconds) >>>> >>>> >>>> 0: jdbc:hive2://localhost:10000/default> set >>>> mapreduce.map.memory.mb=1024; >>>> >>>> No rows affected (0.032 seconds) >>>> >>>> 0: jdbc:hive2://localhost:10000/default> set >>>> mapreduce.map.java.opts=-Xmx1024m; >>>> >>>> No rows affected (0.01 seconds) >>>> >>>> 0: jdbc:hive2://localhost:10000/default> set >>>> mapreduce.reduce.memory.mb=1024; >>>> >>>> No rows affected (0.014 seconds) >>>> >>>> 0: jdbc:hive2://localhost:10000/default> set >>>> mapreduce.reduce.java.opts=-Xmx1024m; >>>> >>>> No rows affected (0.015 seconds) >>>> >>>> >>>> 0: jdbc:hive2://localhost:10000/default> select job,count(*) as dd from >>>> ppl group by job limit 10; >>>> >>>> Error: Error while processing statement: FAILED: Execution Error, >>>> return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask >>>> (state=08S01,code=2) >>>> >>>> >>>> >>>> >>>> Sorry my test VM has 2gb Ram only. So I set all the above memory size >>>> to 1GB. >>>> >>>> But it still gets the same error. >>>> >>>> >>>> >>>> please help. thanks. >>>> >>>> >>>> >>>> On Tue, Mar 29, 2022 at 8:32 PM Pau Tallada <tall...@pic.es> wrote: >>>> >>>>> I assume you have to increase container size (if using tez/yarn) >>>>> >>>>> Missatge de Bitfox <bit...@bitfox.top> del dia dt., 29 de març 2022 a >>>>> les 14:30: >>>>> >>>>>> My hive run out of memory even for a small query: >>>>>> >>>>>> 2022-03-29T20:26:51,440 WARN [Thread-1329] mapred.LocalJobRunner: >>>>>> job_local300585280_0011 >>>>>> >>>>>> java.lang.Exception: java.lang.OutOfMemoryError: Java heap space >>>>>> >>>>>> at >>>>>> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492) >>>>>> ~[hadoop-mapreduce-client-common-3.3.2.jar:?] >>>>>> >>>>>> at >>>>>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552) >>>>>> ~[hadoop-mapreduce-client-common-3.3.2.jar:?] >>>>>> >>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space >>>>>> >>>>>> >>>>>> >>>>>> hadoop-3.3.2 >>>>>> >>>>>> hive-3.1.2 >>>>>> >>>>>> java version "1.8.0_321" >>>>>> >>>>>> >>>>>> >>>>>> How to fix this? thanks. >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> ---------------------------------- >>>>> Pau Tallada Crespí >>>>> Departament de Serveis >>>>> Port d'Informació Científica (PIC) >>>>> Tel: +34 93 170 2729 >>>>> ---------------------------------- >>>>> >>>>> >>> >>> -- >>> ---------------------------------- >>> Pau Tallada Crespí >>> Departament de Serveis >>> Port d'Informació Científica (PIC) >>> Tel: +34 93 170 2729 >>> ---------------------------------- >>> >>> > > -- > ---------------------------------- > Pau Tallada Crespí > Departament de Serveis > Port d'Informació Científica (PIC) > Tel: +34 93 170 2729 > ---------------------------------- > >
