[
https://issues.apache.org/jira/browse/SPARK-14693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15251354#comment-15251354
]
Sean Owen commented on SPARK-14693:
-----------------------------------
The question is what the state of the application is, since all you say is that
it hangs. Does anything execute on executors? what does the UI show? sometimes
using the "Thread Dump" feature in the executors (or if needed, "kill -QUIT
[pid]" on a Java process) is helpful since it shows you what threads are
running, and you can scan for warnings about a literal deadlock, or at least
find where the key thread is waiting.
> Spark Streaming Context Hangs on Start
> --------------------------------------
>
> Key: SPARK-14693
> URL: https://issues.apache.org/jira/browse/SPARK-14693
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.6.0, 1.6.1
> Environment: Databricks Cloud
> Reporter: Evan Oman
>
> All,
> I am trying to use Kinesis with Spark Streaming on Spark 1.6.0 via Databricks
> and my `ssc.start()` command is hanging.
> I am using the following function (based on [this
> guide|http://spark.apache.org/docs/latest/streaming-kinesis-integration.html],
> which, as an aside, contains some broken Github links) to make my Spark
> Streaming Context:
> {code:borderStyle=solid}
> def creatingFunc(sc: SparkContext): StreamingContext =
> {
> // Create a StreamingContext
> val ssc = new StreamingContext(sc,
> Seconds(batchIntervalSeconds))
> // Creata a Kinesis stream
> val kinesisStream = KinesisUtils.createStream(ssc,
> kinesisAppName, kinesisStreamName,
> kinesisEndpointUrl,
> RegionUtils.getRegionByEndpoint(kinesisEndpointUrl).getName,
> InitialPositionInStream.LATEST,
> Seconds(kinesisCheckpointIntervalSeconds),
> StorageLevel.MEMORY_AND_DISK_SER_2,
> config.awsAccessKeyId, config.awsSecretKey)
> kinesisStream.print()
> ssc.remember(Minutes(1))
> ssc.checkpoint(checkpointDir)
> ssc
> }
> {code}
> However when I run the following to start the streaming context:
> {code:borderStyle=solid}
> // Stop any existing StreamingContext
> val stopActiveContext = true
> if (stopActiveContext) {
> StreamingContext.getActive.foreach { _.stop(stopSparkContext = false) }
> }
> // Get or create a streaming context.
> val ssc = StreamingContext.getActiveOrCreate(() => main.creatingFunc(sc))
> // This starts the streaming context in the background.
> ssc.start()
> {code}
> The last bit, `ssc.start()`, hangs indefinitely without issuing any log
> messages. I am running this on a freshly spun up cluster with no other
> notebooks attached so there aren't any other streaming contexts running.
> Any thoughts?
> Additionally, here are the libraries I am using (from my build.sbt file):
> {code:borderStyle=solid}
> "org.apache.spark" % "spark-core_2.10" % "1.6.0"
> "org.apache.spark" % "spark-sql_2.10" % "1.6.0"
> "org.apache.spark" % "spark-streaming-kinesis-asl_2.10" % "1.6.0"
> "org.apache.spark" % "spark-streaming_2.10" % "1.6.0"
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
