[ https://issues.apache.org/jira/browse/SPARK-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Davidson resolved SPARK-2403. ----------------------------------- Resolution: Fixed Fix Version/s: 1.0.2 1.1.0 > Spark stuck when class is not registered with Kryo > -------------------------------------------------- > > Key: SPARK-2403 > URL: https://issues.apache.org/jira/browse/SPARK-2403 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.0.0 > Reporter: Daniel Darabos > Fix For: 1.1.0, 1.0.2 > > > We are using Kryo and require registering classes. When trying to serialize > something containing an unregistered class, Kryo will raise an exception. > DAGScheduler.submitMissingTasks runs in the scheduler thread and checks if > the contents of the task can be serialized by trying to serialize it: > https://github.com/apache/spark/blob/v1.0.0/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L767 > It catches NotSerializableException and aborts the task with an error when > this happens. > The problem is, Kryo does not raise NotSerializableException for unregistered > classes. It raises IllegalArgumentException instead. This exception is not > caught and kills the scheduler thread. The application then hangs, waiting > indefinitely for the job to finish. > Catching IllegalArgumentException also is a quick fix. I'll send a pull > request for it if you agree. Thanks! -- This message was sent by Atlassian JIRA (v6.2#6252)