Possibly. In that case maybe you should step back from spark and see if there are OS-level tools to understand what's going on, like looking for evidence of the OOM killer - https://docs.memset.com/other/linux-s-oom-process-killer
On Thu, Dec 17, 2020 at 1:45 PM Vikas Garg <sperry...@gmail.com> wrote: > I am running code in a local machine that is single node machine. > > Getting into logs, it looked like the host is killed. This is happening > very frequently an I am unable to find the reason of this. > > Could low memory be the reason? > > On Fri, 18 Dec 2020, 00:11 Patrick McCarthy, <pmccar...@dstillery.com> > wrote: > >> 'Job aborted due to stage failure: Task 1 in stage 39.0 failed 1 times' >> >> You may want to change the number of failures to a higher number like 4. >> A single failure on a task should be able to be tolerated, especially if >> you're on a shared cluster where resources can be preempted. >> >> It seems that a node dies or goes off the network, so perhaps you can >> also debug the logs on the failing node to see why it disappears and >> prevent the failures in the first place. >> >> On Thu, Dec 17, 2020 at 1:27 PM Vikas Garg <sperry...@gmail.com> wrote: >> >>> Mydomain is named by me while pasting the logs >>> >>> Also, there are multiple class files in my project, if I run any 1 or 2 >>> at a time, then they run fine, sometimes they too give this error. But >>> running all the classes at the same time always give this error. >>> >>> Once this error come, I can't run any program and on restarting the >>> system, program starts running fine. >>> This error goes away on >>> >>> On Thu, 17 Dec 2020, 23:50 Patrick McCarthy, <pmccar...@dstillery.com> >>> wrote: >>> >>>> my-domain.com/192.168.166.8:63534 probably isn't a valid address on >>>> your network, is it? >>>> >>>> On Thu, Dec 17, 2020 at 3:03 AM Vikas Garg <sperry...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> Since last few days, I am getting error message while running my >>>>> project. I have searched Google for the solution but didn't got any help. >>>>> >>>>> Can someone help me to figure out how I could mitigate this issue? >>>>> >>>>> >>>>> 20/12/17 13:26:57 ERROR RetryingBlockFetcher: Exception while >>>>> beginning fetch of 1 outstanding blocks >>>>> >>>>> *java.io.IOException*: Failed to connect to >>>>> my-domain.com/192.168.166.8:63534 >>>>> >>>>> at >>>>> org.apache.spark.network.client.TransportClientFactory.createClient( >>>>> *TransportClientFactory.java:253*) >>>>> >>>>> at >>>>> org.apache.spark.network.client.TransportClientFactory.createClient( >>>>> *TransportClientFactory.java:195*) >>>>> >>>>> at >>>>> org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart( >>>>> *NettyBlockTransferService.scala:122*) >>>>> >>>>> at >>>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding( >>>>> *RetryingBlockFetcher.java:141*) >>>>> >>>>> at org.apache.spark.network.shuffle.RetryingBlockFetcher.start( >>>>> *RetryingBlockFetcher.java:121*) >>>>> >>>>> at >>>>> org.apache.spark.network.netty.NettyBlockTransferService.fetchBlocks( >>>>> *NettyBlockTransferService.scala:143*) >>>>> >>>>> at >>>>> org.apache.spark.network.BlockTransferService.fetchBlockSync( >>>>> *BlockTransferService.scala:103*) >>>>> >>>>> at >>>>> org.apache.spark.storage.BlockManager.fetchRemoteManagedBuffer( >>>>> *BlockManager.scala:1010*) >>>>> >>>>> at >>>>> org.apache.spark.storage.BlockManager.$anonfun$getRemoteBlock$8( >>>>> *BlockManager.scala:954*) >>>>> >>>>> at scala.Option.orElse(*Option.scala:447*) >>>>> >>>>> at org.apache.spark.storage.BlockManager.getRemoteBlock( >>>>> *BlockManager.scala:954*) >>>>> >>>>> at org.apache.spark.storage.BlockManager.getRemoteBytes( >>>>> *BlockManager.scala:1092*) >>>>> >>>>> at >>>>> org.apache.spark.scheduler.TaskResultGetter$$anon$3.$anonfun$run$1( >>>>> *TaskResultGetter.scala:88*) >>>>> >>>>> at >>>>> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) >>>>> >>>>> at org.apache.spark.util.Utils$.logUncaughtExceptions( >>>>> *Utils.scala:1932*) >>>>> >>>>> at org.apache.spark.scheduler.TaskResultGetter$$anon$3.run( >>>>> *TaskResultGetter.scala:63*) >>>>> >>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>> *ThreadPoolExecutor.java:1149*) >>>>> >>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>> *ThreadPoolExecutor.java:624*) >>>>> >>>>> at java.lang.Thread.run(*Thread.java:748*) >>>>> >>>>> Caused by: *io.netty.channel.AbstractChannel$AnnotatedSocketException*: >>>>> Permission denied: no further information: >>>>> my-domain.com/192.168.166.8:63534 >>>>> >>>>> Caused by: *java.net.SocketException*: Permission denied: no further >>>>> information >>>>> >>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(*Native Method*) >>>>> >>>>> at sun.nio.ch.SocketChannelImpl.finishConnect( >>>>> *SocketChannelImpl.java:715*) >>>>> >>>>> at >>>>> io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect( >>>>> *NioSocketChannel.java:330*) >>>>> >>>>> at >>>>> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect( >>>>> *AbstractNioChannel.java:334*) >>>>> >>>>> at io.netty.channel.nio.NioEventLoop.processSelectedKey( >>>>> *NioEventLoop.java:702*) >>>>> >>>>> at >>>>> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized( >>>>> *NioEventLoop.java:650*) >>>>> >>>>> at io.netty.channel.nio.NioEventLoop.processSelectedKeys( >>>>> *NioEventLoop.java:576*) >>>>> >>>>> at io.netty.channel.nio.NioEventLoop.run( >>>>> *NioEventLoop.java:493*) >>>>> >>>>> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run( >>>>> *SingleThreadEventExecutor.java:989*) >>>>> >>>>> at io.netty.util.internal.ThreadExecutorMap$2.run( >>>>> *ThreadExecutorMap.java:74*) >>>>> >>>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run( >>>>> *FastThreadLocalRunnable.java:30*) >>>>> >>>>> at java.lang.Thread.run(*Thread.java:748*) >>>>> >>>>> 20/12/17 13:27:02 ERROR RetryingBlockFetcher: Exception while >>>>> beginning fetch of 1 outstanding blocks (after 1 retries) >>>>> >>>>> *java.io.IOException*: Failed to connect to >>>>> my-domain.com/192.168.166.8:63534 >>>>> >>>>> at >>>>> org.apache.spark.network.client.TransportClientFactory.createClient( >>>>> *TransportClientFactory.java:253*) >>>>> >>>>> at >>>>> org.apache.spark.network.client.TransportClientFactory.createClient( >>>>> *TransportClientFactory.java:195*) >>>>> >>>>> at >>>>> org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart( >>>>> *NettyBlockTransferService.scala:122*) >>>>> >>>>> at >>>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding( >>>>> *RetryingBlockFetcher.java:141*) >>>>> >>>>> at >>>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.lambda$initiateRetry$0( >>>>> *RetryingBlockFetcher.java:169*) >>>>> >>>>> at java.util.concurrent.Executors$RunnableAdapter.call( >>>>> *Executors.java:511*) >>>>> >>>>> at java.util.concurrent.FutureTask.run(*FutureTask.java:266*) >>>>> >>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>> *ThreadPoolExecutor.java:1149*) >>>>> >>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>> *ThreadPoolExecutor.java:624*) >>>>> >>>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run( >>>>> *FastThreadLocalRunnable.java:30*) >>>>> >>>>> at java.lang.Thread.run(*Thread.java:748*) >>>>> >>>>> Caused by: *io.netty.channel.AbstractChannel$AnnotatedSocketException*: >>>>> Permission denied: no further information: >>>>> my-domain.com/192.168.166.8:63534 >>>>> >>>>> Caused by: *java.net.SocketException*: Permission denied: no further >>>>> information >>>>> >>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(*Native Method*) >>>>> >>>>> at sun.nio.ch.SocketChannelImpl.finishConnect( >>>>> *SocketChannelImpl.java:715*) >>>>> >>>>> at >>>>> io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect( >>>>> *NioSocketChannel.java:330*) >>>>> >>>>> at >>>>> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect( >>>>> *AbstractNioChannel.java:334*) >>>>> >>>>> at io.netty.channel.nio.NioEventLoop.processSelectedKey( >>>>> *NioEventLoop.java:702*) >>>>> >>>>> at >>>>> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized( >>>>> *NioEventLoop.java:650*) >>>>> >>>>> at io.netty.channel.nio.NioEventLoop.processSelectedKeys( >>>>> *NioEventLoop.java:576*) >>>>> >>>>> at io.netty.channel.nio.NioEventLoop.run( >>>>> *NioEventLoop.java:493*) >>>>> >>>>> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run( >>>>> *SingleThreadEventExecutor.java:989*) >>>>> >>>>> at io.netty.util.internal.ThreadExecutorMap$2.run( >>>>> *ThreadExecutorMap.java:74*) >>>>> >>>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run( >>>>> *FastThreadLocalRunnable.java:30*) >>>>> >>>>> at java.lang.Thread.run(*Thread.java:748*) >>>>> >>>>> 20/12/17 13:27:07 ERROR RetryingBlockFetcher: Exception while >>>>> beginning fetch of 1 outstanding blocks (after 2 retries) >>>>> >>>>> *java.io.IOException*: Failed to connect to >>>>> my-domain.com/192.168.166.8:63534 >>>>> >>>>> at >>>>> org.apache.spark.network.client.TransportClientFactory.createClient( >>>>> *TransportClientFactory.java:253*) >>>>> >>>>> at >>>>> org.apache.spark.network.client.TransportClientFactory.createClient( >>>>> *TransportClientFactory.java:195*) >>>>> >>>>> at >>>>> org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart( >>>>> *NettyBlockTransferService.scala:122*) >>>>> >>>>> at >>>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding( >>>>> *RetryingBlockFetcher.java:141*) >>>>> >>>>> at >>>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.lambda$initiateRetry$0( >>>>> *RetryingBlockFetcher.java:169*) >>>>> >>>>> at java.util.concurrent.Executors$RunnableAdapter.call( >>>>> *Executors.java:511*) >>>>> >>>>> at java.util.concurrent.FutureTask.run(*FutureTask.java:266*) >>>>> >>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>> *ThreadPoolExecutor.java:1149*) >>>>> >>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>> *ThreadPoolExecutor.java:624*) >>>>> >>>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run( >>>>> *FastThreadLocalRunnable.java:30*) >>>>> >>>>> at java.lang.Thread.run(*Thread.java:748*) >>>>> >>>>> Caused by: *io.netty.channel.AbstractChannel$AnnotatedSocketException*: >>>>> Permission denied: no further information: >>>>> my-domain.com/192.168.166.8:63534 >>>>> >>>>> Caused by: *java.net.SocketException*: Permission denied: no further >>>>> information >>>>> >>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(*Native Method*) >>>>> >>>>> at sun.nio.ch.SocketChannelImpl.finishConnect( >>>>> *SocketChannelImpl.java:715*) >>>>> >>>>> at >>>>> io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect( >>>>> *NioSocketChannel.java:330*) >>>>> >>>>> at >>>>> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect( >>>>> *AbstractNioChannel.java:334*) >>>>> >>>>> at io.netty.channel.nio.NioEventLoop.processSelectedKey( >>>>> *NioEventLoop.java:702*) >>>>> >>>>> at >>>>> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized( >>>>> *NioEventLoop.java:650*) >>>>> >>>>> at io.netty.channel.nio.NioEventLoop.processSelectedKeys( >>>>> *NioEventLoop.java:576*) >>>>> >>>>> at io.netty.channel.nio.NioEventLoop.run( >>>>> *NioEventLoop.java:493*) >>>>> >>>>> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run( >>>>> *SingleThreadEventExecutor.java:989*) >>>>> >>>>> at io.netty.util.internal.ThreadExecutorMap$2.run( >>>>> *ThreadExecutorMap.java:74*) >>>>> >>>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run( >>>>> *FastThreadLocalRunnable.java:30*) >>>>> >>>>> at java.lang.Thread.run(*Thread.java:748*) >>>>> >>>>> 20/12/17 13:27:12 ERROR RetryingBlockFetcher: Exception while >>>>> beginning fetch of 1 outstanding blocks (after 3 retries) >>>>> >>>>> *java.io.IOException*: Failed to connect to >>>>> my-domain.com/192.168.166.8:63534 >>>>> >>>>> at >>>>> org.apache.spark.network.client.TransportClientFactory.createClient( >>>>> *TransportClientFactory.java:253*) >>>>> >>>>> at >>>>> org.apache.spark.network.client.TransportClientFactory.createClient( >>>>> *TransportClientFactory.java:195*) >>>>> >>>>> at >>>>> org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart( >>>>> *NettyBlockTransferService.scala:122*) >>>>> >>>>> at >>>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding( >>>>> *RetryingBlockFetcher.java:141*) >>>>> >>>>> at >>>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.lambda$initiateRetry$0( >>>>> *RetryingBlockFetcher.java:169*) >>>>> >>>>> at java.util.concurrent.Executors$RunnableAdapter.call( >>>>> *Executors.java:511*) >>>>> >>>>> at java.util.concurrent.FutureTask.run(*FutureTask.java:266*) >>>>> >>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>> *ThreadPoolExecutor.java:1149*) >>>>> >>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>> *ThreadPoolExecutor.java:624*) >>>>> >>>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run( >>>>> *FastThreadLocalRunnable.java:30*) >>>>> >>>>> at java.lang.Thread.run(*Thread.java:748*) >>>>> >>>>> Caused by: *io.netty.channel.AbstractChannel$AnnotatedSocketException*: >>>>> Permission denied: no further information: >>>>> my-domain.com/192.168.166.8:63534 >>>>> >>>>> Caused by: *java.net.SocketException*: Permission denied: no further >>>>> information >>>>> >>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(*Native Method*) >>>>> >>>>> at sun.nio.ch.SocketChannelImpl.finishConnect( >>>>> *SocketChannelImpl.java:715*) >>>>> >>>>> at >>>>> io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect( >>>>> *NioSocketChannel.java:330*) >>>>> >>>>> at >>>>> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect( >>>>> *AbstractNioChannel.java:334*) >>>>> >>>>> at io.netty.channel.nio.NioEventLoop.processSelectedKey( >>>>> *NioEventLoop.java:702*) >>>>> >>>>> at >>>>> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized( >>>>> *NioEventLoop.java:650*) >>>>> >>>>> at io.netty.channel.nio.NioEventLoop.processSelectedKeys( >>>>> *NioEventLoop.java:576*) >>>>> >>>>> at io.netty.channel.nio.NioEventLoop.run( >>>>> *NioEventLoop.java:493*) >>>>> >>>>> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run( >>>>> *SingleThreadEventExecutor.java:989*) >>>>> >>>>> at io.netty.util.internal.ThreadExecutorMap$2.run( >>>>> *ThreadExecutorMap.java:74*) >>>>> >>>>> at io.netty.util.concurrent.FastThreadLocalRunnable.run( >>>>> *FastThreadLocalRunnable.java:30*) >>>>> >>>>> at java.lang.Thread.run(*Thread.java:748*) >>>>> >>>>> 20/12/17 13:27:12 ERROR TaskSetManager: Task 1 in stage 39.0 failed 1 >>>>> times; aborting job >>>>> >>>>> 20/12/17 13:27:12 ERROR FileFormatWriter: Aborting job >>>>> 0db8a16a-e1d3-4091-b725-0a8bdc17b0b1. >>>>> >>>>> *java.util.concurrent.ExecutionException*: >>>>> *org.apache.spark.SparkException*: Job aborted due to stage failure: >>>>> Task 1 in stage 39.0 failed 1 times, most recent failure: Lost task 1.0 in >>>>> stage 39.0 (TID 655, my-domain.com, executor driver): TaskResultLost >>>>> (result lost from block manager) >>>>> >>>>> Driver stacktrace: >>>>> >>>>> at java.util.concurrent.FutureTask.report( >>>>> *FutureTask.java:122*) >>>>> >>>>> at java.util.concurrent.FutureTask.get(*FutureTask.java:206*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.exchange.BroadcastExchangeExec.doExecuteBroadcast( >>>>> *BroadcastExchangeExec.scala:195*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.InputAdapter.doExecuteBroadcast( >>>>> *WholeStageCodegenExec.scala:515*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.SparkPlan.$anonfun$executeBroadcast$1( >>>>> *SparkPlan.scala:188*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1( >>>>> *SparkPlan.scala:213*) >>>>> >>>>> at org.apache.spark.rdd.RDDOperationScope$.withScope( >>>>> *RDDOperationScope.scala:151*) >>>>> >>>>> at org.apache.spark.sql.execution.SparkPlan.executeQuery( >>>>> *SparkPlan.scala:210*) >>>>> >>>>> at org.apache.spark.sql.execution.SparkPlan.executeBroadcast( >>>>> *SparkPlan.scala:184*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.joins.BroadcastHashJoinExec.prepareBroadcast( >>>>> *BroadcastHashJoinExec.scala:116*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.joins.BroadcastHashJoinExec.codegenInner( >>>>> *BroadcastHashJoinExec.scala:210*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.joins.BroadcastHashJoinExec.doConsume( >>>>> *BroadcastHashJoinExec.scala:100*) >>>>> >>>>> at org.apache.spark.sql.execution.CodegenSupport.consume( >>>>> *WholeStageCodegenExec.scala:194*) >>>>> >>>>> at org.apache.spark.sql.execution.CodegenSupport.consume$( >>>>> *WholeStageCodegenExec.scala:149*) >>>>> >>>>> at org.apache.spark.sql.execution.ProjectExec.consume( >>>>> *basicPhysicalOperators.scala:41*) >>>>> >>>>> at org.apache.spark.sql.execution.ProjectExec.doConsume( >>>>> *basicPhysicalOperators.scala:71*) >>>>> >>>>> at org.apache.spark.sql.execution.CodegenSupport.consume( >>>>> *WholeStageCodegenExec.scala:194*) >>>>> >>>>> at org.apache.spark.sql.execution.CodegenSupport.consume$( >>>>> *WholeStageCodegenExec.scala:149*) >>>>> >>>>> at org.apache.spark.sql.execution.FilterExec.consume( >>>>> *basicPhysicalOperators.scala:97*) >>>>> >>>>> at org.apache.spark.sql.execution.FilterExec.doConsume( >>>>> *basicPhysicalOperators.scala:222*) >>>>> >>>>> at org.apache.spark.sql.execution.CodegenSupport.consume( >>>>> *WholeStageCodegenExec.scala:194*) >>>>> >>>>> at org.apache.spark.sql.execution.CodegenSupport.consume$( >>>>> *WholeStageCodegenExec.scala:149*) >>>>> >>>>> at org.apache.spark.sql.execution.ColumnarToRowExec.consume( >>>>> *Columnar.scala:60*) >>>>> >>>>> at org.apache.spark.sql.execution.ColumnarToRowExec.doProduce( >>>>> *Columnar.scala:185*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.CodegenSupport.$anonfun$produce$1( >>>>> *WholeStageCodegenExec.scala:95*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1( >>>>> *SparkPlan.scala:213*) >>>>> >>>>> at org.apache.spark.rdd.RDDOperationScope$.withScope( >>>>> *RDDOperationScope.scala:151*) >>>>> >>>>> at org.apache.spark.sql.execution.SparkPlan.executeQuery( >>>>> *SparkPlan.scala:210*) >>>>> >>>>> at org.apache.spark.sql.execution.CodegenSupport.produce( >>>>> *WholeStageCodegenExec.scala:90*) >>>>> >>>>> at org.apache.spark.sql.execution.CodegenSupport.produce$( >>>>> *WholeStageCodegenExec.scala:90*) >>>>> >>>>> at org.apache.spark.sql.execution.ColumnarToRowExec.produce( >>>>> *Columnar.scala:60*) >>>>> >>>>> at org.apache.spark.sql.execution.FilterExec.doProduce( >>>>> *basicPhysicalOperators.scala:137*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.CodegenSupport.$anonfun$produce$1( >>>>> *WholeStageCodegenExec.scala:95*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1( >>>>> *SparkPlan.scala:213*) >>>>> >>>>> at org.apache.spark.rdd.RDDOperationScope$.withScope( >>>>> *RDDOperationScope.scala:151*) >>>>> >>>>> at org.apache.spark.sql.execution.SparkPlan.executeQuery( >>>>> *SparkPlan.scala:210*) >>>>> >>>>> at org.apache.spark.sql.execution.CodegenSupport.produce( >>>>> *WholeStageCodegenExec.scala:90*) >>>>> >>>>> at org.apache.spark.sql.execution.CodegenSupport.produce$( >>>>> *WholeStageCodegenExec.scala:90*) >>>>> >>>>> at org.apache.spark.sql.execution.FilterExec.produce( >>>>> *basicPhysicalOperators.scala:97*) >>>>> >>>>> at org.apache.spark.sql.execution.ProjectExec.doProduce( >>>>> *basicPhysicalOperators.scala:51*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.CodegenSupport.$anonfun$produce$1( >>>>> *WholeStageCodegenExec.scala:95*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1( >>>>> *SparkPlan.scala:213*) >>>>> >>>>> at org.apache.spark.rdd.RDDOperationScope$.withScope( >>>>> *RDDOperationScope.scala:151*) >>>>> >>>>> at org.apache.spark.sql.execution.SparkPlan.executeQuery( >>>>> *SparkPlan.scala:210*) >>>>> >>>>> at org.apache.spark.sql.execution.CodegenSupport.produce( >>>>> *WholeStageCodegenExec.scala:90*) >>>>> >>>>> at org.apache.spark.sql.execution.CodegenSupport.produce$( >>>>> *WholeStageCodegenExec.scala:90*) >>>>> >>>>> at org.apache.spark.sql.execution.ProjectExec.produce( >>>>> *basicPhysicalOperators.scala:41*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.joins.BroadcastHashJoinExec.doProduce( >>>>> *BroadcastHashJoinExec.scala:95*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.CodegenSupport.$anonfun$produce$1( >>>>> *WholeStageCodegenExec.scala:95*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1( >>>>> *SparkPlan.scala:213*) >>>>> >>>>> at org.apache.spark.rdd.RDDOperationScope$.withScope( >>>>> *RDDOperationScope.scala:151*) >>>>> >>>>> at org.apache.spark.sql.execution.SparkPlan.executeQuery( >>>>> *SparkPlan.scala:210*) >>>>> >>>>> at org.apache.spark.sql.execution.CodegenSupport.produce( >>>>> *WholeStageCodegenExec.scala:90*) >>>>> >>>>> at org.apache.spark.sql.execution.CodegenSupport.produce$( >>>>> *WholeStageCodegenExec.scala:90*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.joins.BroadcastHashJoinExec.produce( >>>>> *BroadcastHashJoinExec.scala:39*) >>>>> >>>>> at org.apache.spark.sql.execution.ProjectExec.doProduce( >>>>> *basicPhysicalOperators.scala:51*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.CodegenSupport.$anonfun$produce$1( >>>>> *WholeStageCodegenExec.scala:95*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1( >>>>> *SparkPlan.scala:213*) >>>>> >>>>> at org.apache.spark.rdd.RDDOperationScope$.withScope( >>>>> *RDDOperationScope.scala:151*) >>>>> >>>>> at org.apache.spark.sql.execution.SparkPlan.executeQuery( >>>>> *SparkPlan.scala:210*) >>>>> >>>>> at org.apache.spark.sql.execution.CodegenSupport.produce( >>>>> *WholeStageCodegenExec.scala:90*) >>>>> >>>>> at org.apache.spark.sql.execution.CodegenSupport.produce$( >>>>> *WholeStageCodegenExec.scala:90*) >>>>> >>>>> at org.apache.spark.sql.execution.ProjectExec.produce( >>>>> *basicPhysicalOperators.scala:41*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.WholeStageCodegenExec.doCodeGen( >>>>> *WholeStageCodegenExec.scala:632*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute( >>>>> *WholeStageCodegenExec.scala:692*) >>>>> >>>>> at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1( >>>>> *SparkPlan.scala:175*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1( >>>>> *SparkPlan.scala:213*) >>>>> >>>>> at org.apache.spark.rdd.RDDOperationScope$.withScope( >>>>> *RDDOperationScope.scala:151*) >>>>> >>>>> at org.apache.spark.sql.execution.SparkPlan.executeQuery( >>>>> *SparkPlan.scala:210*) >>>>> >>>>> at org.apache.spark.sql.execution.SparkPlan.execute( >>>>> *SparkPlan.scala:171*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.datasources.FileFormatWriter$.write( >>>>> *FileFormatWriter.scala:172*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run( >>>>> *InsertIntoHadoopFsRelationCommand.scala:178*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute( >>>>> *commands.scala:108*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult( >>>>> *commands.scala:106*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute( >>>>> *commands.scala:131*) >>>>> >>>>> at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1( >>>>> *SparkPlan.scala:175*) >>>>> >>>>> at >>>>> org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1( >>>>> *SparkPlan.scala:213*) >>>>> >>>>> at org.apache.spark.rdd.RDDOperationScope$.withScope( >>>>> *RDDOperationScope.scala:151*) >>>>> >>>>> at org.apache.spark.sql.execu >>>>> >>>> >> >> -- >> >> >> *Patrick McCarthy * >> >> Senior Data Scientist, Machine Learning Engineering >> >> Dstillery >> >> 470 Park Ave South, 17th Floor, NYC 10016 >> > -- *Patrick McCarthy * Senior Data Scientist, Machine Learning Engineering Dstillery 470 Park Ave South, 17th Floor, NYC 10016
