[jira] [Commented] (SPARK-24086) Exception while executing spark streaming examples
[
https://issues.apache.org/jira/browse/SPARK-24086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16456003#comment-16456003
]
Chandra Hasan commented on SPARK-24086:
---
[~hyukjin.kwon] Thanks mate, i included necessary dependencies while executing
and its working now.
If someone is facing same issue here is the solution
{code:java}
spark-submit --jars
kafka-clients-1.1.0.jar,spark-streaming_2.11-2.3.0.jar,spark-streaming-kafka-0-10_2.11-2.3.0.jar
--class org.apache.spark.examples.streaming.JavaDirectKafkaWordCount
target/original-spark-examples_2.11-2.4.0-SNAPSHOT.jar
{code}
Also [~hyukjin.kwon] I would like to inform that the consumer properties
mentioned in the example file JavaDirectKafkaWordCount example isn't updated
which throws configuration missing error and i need to rewrite the code as below
{{}}
{code:java}
kafkaParams.put("bootstrap.servers", brokers);
kafkaParams.put("key.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
kafkaParams.put("value.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
kafkaParams.put("group.id", "");{code}
{{What do you say, Is it fine or need to open a bug for this?}}
> Exception while executing spark streaming examples
> --
>
> Key: SPARK-24086
> URL: https://issues.apache.org/jira/browse/SPARK-24086
> Project: Spark
> Issue Type: Bug
> Components: Examples
>Affects Versions: 2.3.0
>Reporter: Chandra Hasan
>Priority: Major
>
> After running mvn clean package, I tried to execute one of the spark example
> program JavaDirectKafkaWordCount.java but throws following exeception.
> {code:java}
> [cloud-user@server-2 examples]$ run-example
> streaming.JavaDirectKafkaWordCount 192.168.0.4:9092 msu
> 2018-04-25 09:39:22 WARN NativeCodeLoader:62 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 2018-04-25 09:39:22 INFO SparkContext:54 - Running Spark version 2.3.0
> 2018-04-25 09:39:22 INFO SparkContext:54 - Submitted application:
> JavaDirectKafkaWordCount
> 2018-04-25 09:39:22 INFO SecurityManager:54 - Changing view acls to:
> cloud-user
> 2018-04-25 09:39:22 INFO SecurityManager:54 - Changing modify acls to:
> cloud-user
> 2018-04-25 09:39:22 INFO SecurityManager:54 - Changing view acls groups to:
> 2018-04-25 09:39:22 INFO SecurityManager:54 - Changing modify acls groups to:
> 2018-04-25 09:39:22 INFO SecurityManager:54 - SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(cloud-user);
> groups with view permissions: Set(); users with modify permissions:
> Set(cloud-user); groups with modify permissions: Set()
> 2018-04-25 09:39:23 INFO Utils:54 - Successfully started service
> 'sparkDriver' on port 59333.
> 2018-04-25 09:39:23 INFO SparkEnv:54 - Registering MapOutputTracker
> 2018-04-25 09:39:23 INFO SparkEnv:54 - Registering BlockManagerMaster
> 2018-04-25 09:39:23 INFO BlockManagerMasterEndpoint:54 - Using
> org.apache.spark.storage.DefaultTopologyMapper for getting topology
> information
> 2018-04-25 09:39:23 INFO BlockManagerMasterEndpoint:54 -
> BlockManagerMasterEndpoint up
> 2018-04-25 09:39:23 INFO DiskBlockManager:54 - Created local directory at
> /tmp/blockmgr-6fc11fc1-f638-42ea-a9df-dc01fb81b7b6
> 2018-04-25 09:39:23 INFO MemoryStore:54 - MemoryStore started with capacity
> 366.3 MB
> 2018-04-25 09:39:23 INFO SparkEnv:54 - Registering OutputCommitCoordinator
> 2018-04-25 09:39:23 INFO log:192 - Logging initialized @1825ms
> 2018-04-25 09:39:23 INFO Server:346 - jetty-9.3.z-SNAPSHOT
> 2018-04-25 09:39:23 INFO Server:414 - Started @1900ms
> 2018-04-25 09:39:23 INFO AbstractConnector:278 - Started
> ServerConnector@6813a331{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
> 2018-04-25 09:39:23 INFO Utils:54 - Successfully started service 'SparkUI' on
> port 4040.
> 2018-04-25 09:39:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@4f7c0be3{/jobs,null,AVAILABLE,@Spark}
> 2018-04-25 09:39:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@4cfbaf4{/jobs/json,null,AVAILABLE,@Spark}
> 2018-04-25 09:39:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@58faa93b{/jobs/job,null,AVAILABLE,@Spark}
> 2018-04-25 09:39:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@127d7908{/jobs/job/json,null,AVAILABLE,@Spark}
> 2018-04-25 09:39:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@6b9c69a9{/stages,null,AVAILABLE,@Spark}
> 2018-04-25 09:39:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@6622a690{/stages/json,null,AVAILABLE,@Spark}
> 2018-04-25 09:39:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@30b9eadd{/stages/stage,null,AVAILABLE,@Spar
[jira] [Commented] (SPARK-24086) Exception while executing spark streaming examples
[
https://issues.apache.org/jira/browse/SPARK-24086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16456127#comment-16456127
]
Hyukjin Kwon commented on SPARK-24086:
--
Yup, providing details is helpful to the community. If someone faces the same
issue, the details should be helpful. Thank you.
Not sure yet. I usually encourage people to open a JIRA when it looks quite
clear it's an issue to let the contributors focus on issues not the questions
here.
Otherwise, I would debug it by myself or ask this first to the mailing list and
see if it's really an issue. In my experience, mailing list is pretty
responsive when the question or issue is well descriptive.
> Exception while executing spark streaming examples
> --
>
> Key: SPARK-24086
> URL: https://issues.apache.org/jira/browse/SPARK-24086
> Project: Spark
> Issue Type: Bug
> Components: Examples
>Affects Versions: 2.3.0
>Reporter: Chandra Hasan
>Priority: Major
>
> After running mvn clean package, I tried to execute one of the spark example
> program JavaDirectKafkaWordCount.java but throws following exeception.
> {code:java}
> [cloud-user@server-2 examples]$ run-example
> streaming.JavaDirectKafkaWordCount 192.168.0.4:9092 msu
> 2018-04-25 09:39:22 WARN NativeCodeLoader:62 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 2018-04-25 09:39:22 INFO SparkContext:54 - Running Spark version 2.3.0
> 2018-04-25 09:39:22 INFO SparkContext:54 - Submitted application:
> JavaDirectKafkaWordCount
> 2018-04-25 09:39:22 INFO SecurityManager:54 - Changing view acls to:
> cloud-user
> 2018-04-25 09:39:22 INFO SecurityManager:54 - Changing modify acls to:
> cloud-user
> 2018-04-25 09:39:22 INFO SecurityManager:54 - Changing view acls groups to:
> 2018-04-25 09:39:22 INFO SecurityManager:54 - Changing modify acls groups to:
> 2018-04-25 09:39:22 INFO SecurityManager:54 - SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(cloud-user);
> groups with view permissions: Set(); users with modify permissions:
> Set(cloud-user); groups with modify permissions: Set()
> 2018-04-25 09:39:23 INFO Utils:54 - Successfully started service
> 'sparkDriver' on port 59333.
> 2018-04-25 09:39:23 INFO SparkEnv:54 - Registering MapOutputTracker
> 2018-04-25 09:39:23 INFO SparkEnv:54 - Registering BlockManagerMaster
> 2018-04-25 09:39:23 INFO BlockManagerMasterEndpoint:54 - Using
> org.apache.spark.storage.DefaultTopologyMapper for getting topology
> information
> 2018-04-25 09:39:23 INFO BlockManagerMasterEndpoint:54 -
> BlockManagerMasterEndpoint up
> 2018-04-25 09:39:23 INFO DiskBlockManager:54 - Created local directory at
> /tmp/blockmgr-6fc11fc1-f638-42ea-a9df-dc01fb81b7b6
> 2018-04-25 09:39:23 INFO MemoryStore:54 - MemoryStore started with capacity
> 366.3 MB
> 2018-04-25 09:39:23 INFO SparkEnv:54 - Registering OutputCommitCoordinator
> 2018-04-25 09:39:23 INFO log:192 - Logging initialized @1825ms
> 2018-04-25 09:39:23 INFO Server:346 - jetty-9.3.z-SNAPSHOT
> 2018-04-25 09:39:23 INFO Server:414 - Started @1900ms
> 2018-04-25 09:39:23 INFO AbstractConnector:278 - Started
> ServerConnector@6813a331{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
> 2018-04-25 09:39:23 INFO Utils:54 - Successfully started service 'SparkUI' on
> port 4040.
> 2018-04-25 09:39:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@4f7c0be3{/jobs,null,AVAILABLE,@Spark}
> 2018-04-25 09:39:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@4cfbaf4{/jobs/json,null,AVAILABLE,@Spark}
> 2018-04-25 09:39:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@58faa93b{/jobs/job,null,AVAILABLE,@Spark}
> 2018-04-25 09:39:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@127d7908{/jobs/job/json,null,AVAILABLE,@Spark}
> 2018-04-25 09:39:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@6b9c69a9{/stages,null,AVAILABLE,@Spark}
> 2018-04-25 09:39:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@6622a690{/stages/json,null,AVAILABLE,@Spark}
> 2018-04-25 09:39:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@30b9eadd{/stages/stage,null,AVAILABLE,@Spark}
> 2018-04-25 09:39:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@3249a1ce{/stages/stage/json,null,AVAILABLE,@Spark}
> 2018-04-25 09:39:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@4dd94a58{/stages/pool,null,AVAILABLE,@Spark}
> 2018-04-25 09:39:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@2f4919b0{/stages/pool/json,null,AVAILABLE,@Spark}
> 2018-04-25 09:39:23 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@a8a8b75{/storage,null,AVA
