[
https://issues.apache.org/jira/browse/SPARK-13525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hardik Gupta updated SPARK-13525:
---------------------------------
Comment: was deleted
(was: Even I am facing this issue when am running in cluster of 5 nodes.
My printschema works but head fails
------------------------------------------------
My code:
library(SparkR, lib.loc = "/opt/BIG-DATA/spark-1.5.0-bin-hadoop2.6/R/lib")
sc <- sparkR.init(master = "spark://10.103.25.39:7077", appName = "demo",
sparkHome = "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/spark",
sparkEnvir = list(spark.executor.memory = '16m'))
sqlContext <- sparkRSQL.init(sc)
df <- createDataFrame(sqlContext, iris)
head(df)
printSchema(df)
sparkR.stop()
------------------------------------
I get the following logs
# spark-submit --master spark://10.103.25.39:7077
/opt/BIG-DATA/spark-1.5.0-bin-hadoop2.6/sample2.R
Loading required package: methods
Attaching package: ‘SparkR’
The following objects are masked from ‘package:stats’:
filter, na.omit
The following objects are masked from ‘package:base’:
intersect, rbind, sample, subset, summary, table, transform
Warning message:
package ‘SparkR’ was built under R version 3.2.1
16/12/15 16:55:12 INFO SparkContext: Running Spark version 1.5.0-cdh5.5.1
16/12/15 16:55:13 INFO SecurityManager: Changing view acls to: root
16/12/15 16:55:13 INFO SecurityManager: Changing modify acls to: root
16/12/15 16:55:13 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(root); users with
modify permissions: Set(root)
16/12/15 16:55:13 INFO Slf4jLogger: Slf4jLogger started
16/12/15 16:55:14 INFO Remoting: Starting remoting
16/12/15 16:55:14 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkDriver@10.103.40.40:38177]
16/12/15 16:55:14 INFO Remoting: Remoting now listens on addresses:
[akka.tcp://sparkDriver@10.103.40.40:38177]
16/12/15 16:55:14 INFO Utils: Successfully started service 'sparkDriver' on
port 38177.
16/12/15 16:55:14 INFO SparkEnv: Registering MapOutputTracker
16/12/15 16:55:14 INFO SparkEnv: Registering BlockManagerMaster
16/12/15 16:55:14 INFO DiskBlockManager: Created local directory at
/tmp/blockmgr-54d59b34-eb65-4af4-9f02-07590932fc4f
16/12/15 16:55:14 INFO MemoryStore: MemoryStore started with capacity 530.0 MB
16/12/15 16:55:14 INFO HttpFileServer: HTTP File server directory is
/tmp/spark-38ff69cb-a858-430e-9104-7ea7b7a4b3be/httpd-7d4b481b-c3b2-43e4-b2bc-bf499798b1fa
16/12/15 16:55:14 INFO HttpServer: Starting HTTP Server
16/12/15 16:55:14 INFO Server: jetty-8.y.z-SNAPSHOT
16/12/15 16:55:14 INFO AbstractConnector: Started SocketConnector@0.0.0.0:33746
16/12/15 16:55:14 INFO Utils: Successfully started service 'HTTP file server'
on port 33746.
16/12/15 16:55:14 INFO SparkEnv: Registering OutputCommitCoordinator
16/12/15 16:55:14 INFO Server: jetty-8.y.z-SNAPSHOT
16/12/15 16:55:14 INFO AbstractConnector: Started
SelectChannelConnector@0.0.0.0:4040
16/12/15 16:55:14 INFO Utils: Successfully started service 'SparkUI' on port
4040.
16/12/15 16:55:14 INFO SparkUI: Started SparkUI at http://10.103.40.40:4040
16/12/15 16:55:14 INFO Utils: Copying
/opt/BIG-DATA/spark-1.5.0-bin-hadoop2.6/sample2.R to
/tmp/spark-38ff69cb-a858-430e-9104-7ea7b7a4b3be/userFiles-57695a67-ea9b-484c-af69-94298e02997b/sample2.R
16/12/15 16:55:14 INFO SparkContext: Added file
file:/opt/BIG-DATA/spark-1.5.0-bin-hadoop2.6/sample2.R at
http://10.103.40.40:33746/files/sample2.R with timestamp 1481801114815
16/12/15 16:55:14 WARN MetricsSystem: Using default name DAGScheduler for
source because spark.app.id is not set.
16/12/15 16:55:14 INFO AppClient$ClientEndpoint: Connecting to master
spark://10.103.25.39:7077...
16/12/15 16:55:15 INFO SparkDeploySchedulerBackend: Connected to Spark cluster
with app ID app-20161215165352-0123
16/12/15 16:55:15 INFO AppClient$ClientEndpoint: Executor added:
app-20161215165352-0123/0 on worker-20161208195327-10.103.40.207-7078
(10.103.40.207:7078) with 4 cores
16/12/15 16:55:15 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20161215165352-0123/0 on hostPort 10.103.40.207:7078 with 4 cores, 16.0 MB
RAM
16/12/15 16:55:15 INFO AppClient$ClientEndpoint: Executor added:
app-20161215165352-0123/1 on worker-20161208184218-10.103.25.39-7078
(10.103.25.39:7078) with 4 cores
16/12/15 16:55:15 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20161215165352-0123/1 on hostPort 10.103.25.39:7078 with 4 cores, 16.0 MB
RAM
16/12/15 16:55:15 INFO AppClient$ClientEndpoint: Executor added:
app-20161215165352-0123/2 on worker-20161208195437-10.103.40.186-7078
(10.103.40.186:7078) with 4 cores
16/12/15 16:55:15 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20161215165352-0123/2 on hostPort 10.103.40.186:7078 with 4 cores, 16.0 MB
RAM
16/12/15 16:55:15 INFO Utils: Successfully started service
'org.apache.spark.network.netty.NettyBlockTransferService' on port 34755.
16/12/15 16:55:15 INFO NettyBlockTransferService: Server created on 34755
16/12/15 16:55:15 INFO BlockManagerMaster: Trying to register BlockManager
16/12/15 16:55:15 INFO BlockManagerMasterEndpoint: Registering block manager
10.103.40.40:34755 with 530.0 MB RAM, BlockManagerId(driver, 10.103.40.40,
34755)
16/12/15 16:55:15 INFO BlockManagerMaster: Registered BlockManager
16/12/15 16:55:15 INFO AppClient$ClientEndpoint: Executor updated:
app-20161215165352-0123/2 is now LOADING
16/12/15 16:55:15 INFO AppClient$ClientEndpoint: Executor updated:
app-20161215165352-0123/0 is now LOADING
16/12/15 16:55:15 INFO AppClient$ClientEndpoint: Executor updated:
app-20161215165352-0123/1 is now LOADING
16/12/15 16:55:15 INFO AppClient$ClientEndpoint: Executor updated:
app-20161215165352-0123/0 is now RUNNING
16/12/15 16:55:15 INFO AppClient$ClientEndpoint: Executor updated:
app-20161215165352-0123/1 is now RUNNING
16/12/15 16:55:15 INFO AppClient$ClientEndpoint: Executor updated:
app-20161215165352-0123/2 is now RUNNING
16/12/15 16:55:16 INFO EventLoggingListener: Logging events to
hdfs://bigdata:8020/user/spark/applicationHistory/app-20161215165352-0123
16/12/15 16:55:16 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready
for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
16/12/15 16:55:16 INFO SparkContext: Starting job: collectPartitions at
NativeMethodAccessorImpl.java:-2
16/12/15 16:55:16 INFO DAGScheduler: Got job 0 (collectPartitions at
NativeMethodAccessorImpl.java:-2) with 1 output partitions
16/12/15 16:55:16 INFO DAGScheduler: Final stage: ResultStage
0(collectPartitions at NativeMethodAccessorImpl.java:-2)
16/12/15 16:55:16 INFO DAGScheduler: Parents of final stage: List()
16/12/15 16:55:16 INFO DAGScheduler: Missing parents: List()
16/12/15 16:55:16 INFO DAGScheduler: Submitting ResultStage 0
(ParallelCollectionRDD[0] at parallelize at RRDD.scala:454), which has no
missing parents
16/12/15 16:55:16 INFO MemoryStore: ensureFreeSpace(1280) called with curMem=0,
maxMem=555755765
16/12/15 16:55:16 INFO MemoryStore: Block broadcast_0 stored as values in
memory (estimated size 1280.0 B, free 530.0 MB)
16/12/15 16:55:16 INFO MemoryStore: ensureFreeSpace(854) called with
curMem=1280, maxMem=555755765
16/12/15 16:55:16 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in
memory (estimated size 854.0 B, free 530.0 MB)
16/12/15 16:55:16 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on
10.103.40.40:34755 (size: 854.0 B, free: 530.0 MB)
16/12/15 16:55:16 INFO SparkContext: Created broadcast 0 from broadcast at
DAGScheduler.scala:861
16/12/15 16:55:16 INFO DAGScheduler: Submitting 1 missing tasks from
ResultStage 0 (ParallelCollectionRDD[0] at parallelize at RRDD.scala:454)
16/12/15 16:55:16 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
16/12/15 16:55:17 ERROR ErrorMonitor: AssociationError
[akka.tcp://sparkDriver@10.103.40.40:38177] <-
[akka.tcp://driverPropsFetcher@10.103.25.39:24380]: Error [Shut down address:
akka.tcp://driverPropsFetcher@10.103.25.39:24380] [
akka.remote.ShutDownAssociation: Shut down address:
akka.tcp://driverPropsFetcher@10.103.25.39:24380
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The
remote system terminated the association because it is shutting down.
]
akka.event.Logging$Error$NoCause$
16/12/15 16:55:17 ERROR ErrorMonitor: AssociationError
[akka.tcp://sparkDriver@10.103.40.40:38177] <-
[akka.tcp://driverPropsFetcher@10.103.40.207:50921]: Error [Shut down address:
akka.tcp://driverPropsFetcher@10.103.40.207:50921] [
akka.remote.ShutDownAssociation: Shut down address:
akka.tcp://driverPropsFetcher@10.103.40.207:50921
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The
remote system terminated the association because it is shutting down.
]
akka.event.Logging$Error$NoCause$
16/12/15 16:55:17 INFO SparkDeploySchedulerBackend: Registered executor:
AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@10.103.25.39:9011/user/Executor#1942442265])
with ID 1
16/12/15 16:55:17 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0,
10.103.25.39, partition 0,PROCESS_LOCAL, 16604 bytes)
16/12/15 16:55:17 INFO BlockManagerMasterEndpoint: Registering block manager
10.103.25.39:15957 with 8.1 MB RAM, BlockManagerId(1, 10.103.25.39, 15957)
16/12/15 16:55:18 INFO SparkDeploySchedulerBackend: Registered executor:
AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@10.103.40.207:60679/user/Executor#-1306991379])
with ID 0
16/12/15 16:55:18 INFO BlockManagerMasterEndpoint: Registering block manager
10.103.40.207:41038 with 8.1 MB RAM, BlockManagerId(0, 10.103.40.207, 41038)
16/12/15 16:55:18 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on
10.103.25.39:15957 (size: 854.0 B, free: 8.1 MB)
16/12/15 16:55:18 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0)
in 1063 ms on 10.103.25.39 (1/1)
16/12/15 16:55:18 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have
all completed, from pool
16/12/15 16:55:18 INFO DAGScheduler: ResultStage 0 (collectPartitions at
NativeMethodAccessorImpl.java:-2) finished in 1.993 s
16/12/15 16:55:18 INFO DAGScheduler: Job 0 finished: collectPartitions at
NativeMethodAccessorImpl.java:-2, took 2.225532 s
16/12/15 16:55:19 INFO BlockManagerInfo: Removed broadcast_0_piece0 on
10.103.40.40:34755 in memory (size: 854.0 B, free: 530.0 MB)
16/12/15 16:55:19 INFO BlockManagerInfo: Removed broadcast_0_piece0 on
10.103.25.39:15957 in memory (size: 854.0 B, free: 8.1 MB)
16/12/15 16:55:19 INFO ContextCleaner: Cleaned accumulator 1
Warning messages:
1: In FUN(X[[i]], ...) :
Use Sepal_Length instead of Sepal.Length as column name
2: In FUN(X[[i]], ...) :
Use Sepal_Width instead of Sepal.Width as column name
3: In FUN(X[[i]], ...) :
Use Petal_Length instead of Petal.Length as column name
4: In FUN(X[[i]], ...) :
Use Petal_Width instead of Petal.Width as column name
16/12/15 16:55:19 INFO SparkContext: Starting job: dfToCols at
NativeMethodAccessorImpl.java:-2
16/12/15 16:55:19 INFO DAGScheduler: Got job 1 (dfToCols at
NativeMethodAccessorImpl.java:-2) with 1 output partitions
16/12/15 16:55:19 INFO DAGScheduler: Final stage: ResultStage 1(dfToCols at
NativeMethodAccessorImpl.java:-2)
16/12/15 16:55:19 INFO DAGScheduler: Parents of final stage: List()
16/12/15 16:55:19 INFO DAGScheduler: Missing parents: List()
16/12/15 16:55:19 INFO DAGScheduler: Submitting ResultStage 1
(MapPartitionsRDD[4] at dfToCols at NativeMethodAccessorImpl.java:-2), which
has no missing parents
16/12/15 16:55:19 INFO MemoryStore: ensureFreeSpace(9176) called with curMem=0,
maxMem=555755765
16/12/15 16:55:19 INFO MemoryStore: Block broadcast_1 stored as values in
memory (estimated size 9.0 KB, free 530.0 MB)
16/12/15 16:55:19 INFO MemoryStore: ensureFreeSpace(3690) called with
curMem=9176, maxMem=555755765
16/12/15 16:55:19 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in
memory (estimated size 3.6 KB, free 530.0 MB)
16/12/15 16:55:19 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on
10.103.40.40:34755 (size: 3.6 KB, free: 530.0 MB)
16/12/15 16:55:19 INFO SparkContext: Created broadcast 1 from broadcast at
DAGScheduler.scala:861
16/12/15 16:55:19 INFO DAGScheduler: Submitting 1 missing tasks from
ResultStage 1 (MapPartitionsRDD[4] at dfToCols at
NativeMethodAccessorImpl.java:-2)
16/12/15 16:55:19 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
16/12/15 16:55:19 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1,
10.103.25.39, partition 0,PROCESS_LOCAL, 16604 bytes)
16/12/15 16:55:19 ERROR ErrorMonitor: AssociationError
[akka.tcp://sparkDriver@10.103.40.40:38177] <-
[akka.tcp://driverPropsFetcher@10.103.40.186:58345]: Error [Shut down address:
akka.tcp://driverPropsFetcher@10.103.40.186:58345] [
akka.remote.ShutDownAssociation: Shut down address:
akka.tcp://driverPropsFetcher@10.103.40.186:58345
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The
remote system terminated the association because it is shutting down.
]
akka.event.Logging$Error$NoCause$
16/12/15 16:55:19 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on
10.103.25.39:15957 (size: 3.6 KB, free: 8.1 MB)
16/12/15 16:55:20 INFO SparkDeploySchedulerBackend: Registered executor:
AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@10.103.40.186:38037/user/Executor#998426628])
with ID 2
16/12/15 16:55:21 INFO BlockManagerMasterEndpoint: Registering block manager
10.103.40.186:19641 with 7.8 MB RAM, BlockManagerId(2, 10.103.40.186, 19641)
16/12/15 16:55:31 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 1,
10.103.25.39): java.net.SocketTimeoutException: Accept timed out
at java.net.PlainSocketImpl.socketAccept(Native Method)
at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.spark.api.r.RRDD$.createRWorker(RRDD.scala:426)
at org.apache.spark.api.r.BaseRRDD.compute(RRDD.scala:62)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
)
> SparkR: java.net.SocketTimeoutException: Accept timed out when running any
> dataframe function
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-13525
> URL: https://issues.apache.org/jira/browse/SPARK-13525
> Project: Spark
> Issue Type: Bug
> Components: SparkR
> Reporter: Shubhanshu Mishra
> Labels: sparkr
>
> I am following the code steps from this example:
> https://spark.apache.org/docs/1.6.0/sparkr.html
> There are multiple issues:
> 1. The head and summary and filter methods are not overridden by spark. Hence
> I need to call them using `SparkR::` namespace.
> 2. When I try to execute the following, I get errors:
> {code}
> $> $R_HOME/bin/R
> R version 3.2.3 (2015-12-10) -- "Wooden Christmas-Tree"
> Copyright (C) 2015 The R Foundation for Statistical Computing
> Platform: x86_64-pc-linux-gnu (64-bit)
> R is free software and comes with ABSOLUTELY NO WARRANTY.
> You are welcome to redistribute it under certain conditions.
> Type 'license()' or 'licence()' for distribution details.
> Natural language support but running in an English locale
> R is a collaborative project with many contributors.
> Type 'contributors()' for more information and
> 'citation()' on how to cite R or R packages in publications.
> Type 'demo()' for some demos, 'help()' for on-line help, or
> 'help.start()' for an HTML browser interface to help.
> Type 'q()' to quit R.
> Welcome at Fri Feb 26 16:19:35 2016
> Attaching package: ‘SparkR’
> The following objects are masked from ‘package:base’:
> colnames, colnames<-, drop, intersect, rank, rbind, sample, subset,
> summary, transform
> Launching java with spark-submit command
> /content/smishra8/SOFTWARE/spark/bin/spark-submit --driver-memory "50g"
> sparkr-shell /tmp/RtmpfBQRg6/backend_portc3bc16f09b1b
> > df <- createDataFrame(sqlContext, iris)
> Warning messages:
> 1: In FUN(X[[i]], ...) :
> Use Sepal_Length instead of Sepal.Length as column name
> 2: In FUN(X[[i]], ...) :
> Use Sepal_Width instead of Sepal.Width as column name
> 3: In FUN(X[[i]], ...) :
> Use Petal_Length instead of Petal.Length as column name
> 4: In FUN(X[[i]], ...) :
> Use Petal_Width instead of Petal.Width as column name
> > training <- filter(df, df$Species != "setosa")
> Error in filter(df, df$Species != "setosa") :
> no method for coercing this S4 class to a vector
> > training <- SparkR::filter(df, df$Species != "setosa")
> > model <- SparkR::glm(Species ~ Sepal_Length + Sepal_Width, data = training,
> > family = "binomial")
> 16/02/26 16:26:46 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1)
> java.net.SocketTimeoutException: Accept timed out
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
> at java.net.ServerSocket.implAccept(ServerSocket.java:530)
> at java.net.ServerSocket.accept(ServerSocket.java:498)
> at org.apache.spark.api.r.RRDD$.createRWorker(RRDD.scala:431)
> at org.apache.spark.api.r.BaseRRDD.compute(RRDD.scala:62)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:77)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:45)
> at org.apache.spark.scheduler.Task.run(Task.scala:81)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 16/02/26 16:26:46 ERROR TaskSetManager: Task 0 in stage 1.0 failed 1 times;
> aborting job
> 16/02/26 16:26:46 ERROR RBackendHandler: fitRModelFormula on
> org.apache.spark.ml.api.r.SparkRWrappers failed
> Error in invokeJava(isStatic = TRUE, className, methodName, ...) :
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0
> in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0
> (TID 1, localhost): java.net.SocketTimeoutException: Accept timed out
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
> at java.net.ServerSocket.implAccept(ServerSocket.java:530)
> at java.net.ServerSocket.accept(ServerSocket.java:498)
> at org.apache.spark.api.r.RRDD$.createRWorker(RRDD.scala:431)
> at org.apache.spark.api.r.BaseRRDD.compute(RRDD.scala:62)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> >
> {code}
> Even, when I try to run the head command on the dataframe, I get similar
> error:
> {code}
> > SparkR::head(df)
> 16/02/26 16:32:05 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID 2)
> java.net.SocketTimeoutException: Accept timed out
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
> at java.net.ServerSocket.implAccept(ServerSocket.java:530)
> at java.net.ServerSocket.accept(ServerSocket.java:498)
> at org.apache.spark.api.r.RRDD$.createRWorker(RRDD.scala:431)
> at org.apache.spark.api.r.BaseRRDD.compute(RRDD.scala:62)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:69)
> at org.apache.spark.scheduler.Task.run(Task.scala:81)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 16/02/26 16:32:05 ERROR TaskSetManager: Task 0 in stage 3.0 failed 1 times;
> aborting job
> 16/02/26 16:32:05 ERROR RBackendHandler: dfToCols on
> org.apache.spark.sql.api.r.SQLUtils failed
> Error in invokeJava(isStatic = TRUE, className, methodName, ...) :
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0
> in stage 3.0 failed 1 times, most recent failure: Lost task 0.0 in stage 3.0
> (TID 2, localhost): java.net.SocketTimeoutException: Accept timed out
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
> at java.net.ServerSocket.implAccept(ServerSocket.java:530)
> at java.net.ServerSocket.accept(ServerSocket.java:498)
> at org.apache.spark.api.r.RRDD$.createRWorker(RRDD.scala:431)
> at org.apache.spark.api.r.BaseRRDD.compute(RRDD.scala:62)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> {code}
> I have a .Rprofile file in my directory which looks like the following:
> {code:title=.Rprofile|borderStyle=solid}
> # Sample Rprofile.site file
> # Things you might want to change
> .First <- function(){
> cat("\nWelcome at", date(), "\n")
> SPARK_HOME <- "/content/user/SOFTWARE/spark"
> .libPaths(c(file.path(SPARK_HOME, "R", "lib"), .libPaths()))
> library(SparkR)
> sc <<- sparkR.init(master="local[20]", appName="Model SparkR",
> sparkHome=SPARK_HOME,
> sparkEnvir=list(spark.local.dir="./tmp",
> spark.executor.memory="50g",
> spark.driver.maxResultSize="50g",
> spark.driver.memory="50g"))
> sqlContext <<- sparkRSQL.init(sc)
> }
> .Last <- function(){
> cat("\nGoodbye at ", date(), "\n")
> }
> {code}
> I am using the master branch of Spark since the following commit:
> {code}
> commit 35316cb0b744bef9bcb390411ddc321167f953be
> Author: Yu ISHIKAWA <yuu.ishik...@gmail.com>
> Date: Thu Feb 25 13:29:10 2016 -0800
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
