[ https://issues.apache.org/jira/browse/SPARK-14693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15256007#comment-15256007 ]
Sean Owen commented on SPARK-14693: ----------------------------------- Given the end of your stack trace, it seems like it could be an issue with the underlying file system? but that implementation is not part of Spark. > Spark Streaming Context Hangs on Start > -------------------------------------- > > Key: SPARK-14693 > URL: https://issues.apache.org/jira/browse/SPARK-14693 > Project: Spark > Issue Type: Bug > Components: Streaming > Affects Versions: 1.6.0, 1.6.1 > Environment: Databricks Cloud > Reporter: Evan Oman > > All, > I am trying to use Kinesis with Spark Streaming on Spark 1.6.0 via Databricks > and my `ssc.start()` command is hanging. > I am using the following function (based on [this > guide|http://spark.apache.org/docs/latest/streaming-kinesis-integration.html], > which, as an aside, contains some broken Github links) to make my Spark > Streaming Context: > {code:borderStyle=solid} > def creatingFunc(sc: SparkContext): StreamingContext = > { > // Create a StreamingContext > val ssc = new StreamingContext(sc, > Seconds(batchIntervalSeconds)) > // Creata a Kinesis stream > val kinesisStream = KinesisUtils.createStream(ssc, > kinesisAppName, kinesisStreamName, > kinesisEndpointUrl, > RegionUtils.getRegionByEndpoint(kinesisEndpointUrl).getName, > InitialPositionInStream.LATEST, > Seconds(kinesisCheckpointIntervalSeconds), > StorageLevel.MEMORY_AND_DISK_SER_2, > config.awsAccessKeyId, config.awsSecretKey) > kinesisStream.print() > ssc.remember(Minutes(1)) > ssc.checkpoint(checkpointDir) > ssc > } > {code} > However when I run the following to start the streaming context: > {code:borderStyle=solid} > // Stop any existing StreamingContext > val stopActiveContext = true > if (stopActiveContext) { > StreamingContext.getActive.foreach { _.stop(stopSparkContext = false) } > } > // Get or create a streaming context. > val ssc = StreamingContext.getActiveOrCreate(() => main.creatingFunc(sc)) > // This starts the streaming context in the background. > ssc.start() > {code} > The last bit, `ssc.start()`, hangs indefinitely without issuing any log > messages. I am running this on a freshly spun up cluster with no other > notebooks attached so there aren't any other streaming contexts running. > Any thoughts? > Additionally, here are the libraries I am using (from my build.sbt file): > {code:borderStyle=solid} > "org.apache.spark" % "spark-core_2.10" % "1.6.0" > "org.apache.spark" % "spark-sql_2.10" % "1.6.0" > "org.apache.spark" % "spark-streaming-kinesis-asl_2.10" % "1.6.0" > "org.apache.spark" % "spark-streaming_2.10" % "1.6.0" > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org