Hi Rodrick, I had tried increasing memory from 6G to 9G to 12G but still I am getting the same error. The size of dataframe I am trying to write is around 6-7 G and the Hive table is Parquet format.
Thanks, Bijay On Mon, Apr 11, 2016 at 4:03 AM, Rodrick Brown <rodr...@orchardplatform.com> wrote: > Try increasing the memory allocated for this job. > > Sent from Outlook for iPhone <https://aka.ms/wp8k5y> > > > > > On Sun, Apr 10, 2016 at 9:12 PM -0700, "Bijay Kumar Pathak" < > bkpat...@mtu.edu> wrote: > > Hi, >> >> I am running Spark 1.6 on EMR. I have workflow which does the following >> things: >> >> 1. Read the 2 flat file, create the data frame and join it. >> 2. Read the particular partition from the hive table and joins the >> dataframe from 1 with it. >> 3. Finally, insert overwrite into hive table which is partitioned >> into two fields. >> >> The stout log message in terminal when I submit the jobs show the below >> message. >> # >> # java.lang.OutOfMemoryError: Java heap space >> # -XX:OnOutOfMemoryError="kill -9 %p" >> # Executing /bin/sh -c "kill -9 30149"... >> Killed >> >> And while I check the YARN logs it shows the below error. The Spark UI >> doesn't show any failure stages or tasks but the jobs get stuck in the >> middle without completing all the stages. Did anyone come across similar >> issues? What could be the reason behind it and how could I troubleshoot it? >> >> >> 16/04/11 00:19:38 ERROR client.TransportResponseHandler: Still have 1 >> requests outstanding when connection from ip-10-184-195-29.ec2.internal/ >> 10.184.195.29:43162 is closed >> 16/04/11 00:19:38 WARN executor.CoarseGrainedExecutorBackend: An unknown >> (ip-10-184-195-29.ec2.internal:43162) driver disconnected. >> 16/04/11 00:19:38 ERROR executor.CoarseGrainedExecutorBackend: Driver >> 10.184.195.29:43162 disassociated! Shutting down. >> 16/04/11 00:19:38 WARN netty.NettyRpcEndpointRef: Error sending message >> [message = Heartbeat(12,[Lscala.Tuple2;@6545df9a,BlockManagerId(12, >> ip-10-184-194-43.ec2.internal, 43867))] in 1 attempts >> java.io.IOException: Connection from ip-10-184-195-29.ec2.internal/ >> 10.184.195.29:43162 closed >> at >> org.apache.spark.network.client.TransportResponseHandler.channelUnregistered(TransportResponseHandler.java:124) >> at >> org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:94) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144) >> at >> io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144) >> at >> io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144) >> at >> io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144) >> at >> io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:739) >> at >> io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:659) >> at >> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) >> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) >> at >> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) >> at java.lang.Thread.run(Thread.java:745) >> 16/04/11 00:19:38 INFO storage.DiskBlockManager: Shutdown hook called >> 16/04/11 00:19:38 INFO util.ShutdownHookManager: Shutdown hook called >> >> >> > *NOTICE TO RECIPIENTS*: This communication is confidential and intended > for the use of the addressee only. If you are not an intended recipient of > this communication, please delete it immediately and notify the sender by > return email. Unauthorized reading, dissemination, distribution or copying > of this communication is prohibited. This communication does not constitute > an offer to sell or a solicitation of an indication of interest to purchase > any loan, security or any other financial product or instrument, nor is it > an offer to sell or a solicitation of an indication of interest to purchase > any products or services to any persons who are prohibited from receiving > such information under applicable law. The contents of this communication > may not be accurate or complete and are subject to change without notice. > As such, Orchard App, Inc. (including its subsidiaries and affiliates, > "Orchard") makes no representation regarding the accuracy or completeness > of the information contained herein. The intended recipient is advised to > consult its own professional advisors, including those specializing in > legal, tax and accounting matters. Orchard does not provide legal, tax or > accounting advice. >
