[jira] [Created] (SPARK-8412) java#KafkaUtils.createDirectStream Java(Pair)RDDs do not implement HasOffsetRanges
jweinste created SPARK-8412: --- Summary: java#KafkaUtils.createDirectStream Java(Pair)RDDs do not implement HasOffsetRanges Key: SPARK-8412 URL: https://issues.apache.org/jira/browse/SPARK-8412 Project: Spark Issue Type: Bug Components: Streaming Affects Versions: 1.3.0 Reporter: jweinste Priority: Critical // Create direct kafka stream with brokers and topics final JavaPairInputDStream messages = KafkaUtils.createDirectStream(jssc, String.class, String.class, StringDecoder.class, StringDecoder.class, kafkaParams, topics); messages.foreachRDD(new Function, Void>() { @Override public Void call(final JavaPairRDD rdd) throws Exception { if (rdd instanceof HasOffsetRanges) { //will never happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-8417) spark-class has illegal statement
jweinste created SPARK-8417: --- Summary: spark-class has illegal statement Key: SPARK-8417 URL: https://issues.apache.org/jira/browse/SPARK-8417 Project: Spark Issue Type: Bug Components: Deploy Affects Versions: 1.4.0 Reporter: jweinste Priority: Blocker spark-class There is an illegal statement. done < <("$RUNNER" -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@") Complaint is ./bin/spark-class: line 100: syntax error near unexpected token `<' -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Reopened] (SPARK-8412) java#KafkaUtils.createDirectStream Java(Pair)RDDs do not implement HasOffsetRanges
[ https://issues.apache.org/jira/browse/SPARK-8412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jweinste reopened SPARK-8412: - This is improperly implemented or improperly documented. Either way something needs to be corrected. http://spark.apache.org/docs/latest/streaming-kafka-integration.html You'll have to scan down to http://spark.apache.org/docs/latest/streaming-kafka-integration.html #tab_java_2 directKafkaStream.foreachRDD( new Function, Void>() { @Override public Void call(JavaPairRDD rdd) throws IOException { OffsetRange[] offsetRanges = ((HasOffsetRanges)rdd).offsetRanges // offsetRanges.length = # of Kafka partitions being consumed ... return null; } } ); > java#KafkaUtils.createDirectStream Java(Pair)RDDs do not implement > HasOffsetRanges > -- > > Key: SPARK-8412 > URL: https://issues.apache.org/jira/browse/SPARK-8412 > Project: Spark > Issue Type: Bug > Components: Streaming >Affects Versions: 1.3.0 >Reporter: jweinste >Priority: Critical > > // Create direct kafka stream with brokers and topics > final JavaPairInputDStream messages = > KafkaUtils.createDirectStream(jssc, String.class, String.class, > StringDecoder.class, > StringDecoder.class, kafkaParams, topics); > messages.foreachRDD(new Function, > Void>() { > @Override > public Void call(final JavaPairRDD rdd) > throws Exception { > if (rdd instanceof HasOffsetRanges) { > //will never happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8417) spark-class has illegal statement
[ https://issues.apache.org/jira/browse/SPARK-8417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14593650#comment-14593650 ] jweinste commented on SPARK-8417: - I saw this issue when running local on mac.. But the command itself looks rather suspect. My feeling is there is a more simple and straight forward way to accomplish the command creation. I tried few variations on the current script using things like <<< and "$(...)" to no avail. > spark-class has illegal statement > - > > Key: SPARK-8417 > URL: https://issues.apache.org/jira/browse/SPARK-8417 > Project: Spark > Issue Type: Bug > Components: Deploy >Affects Versions: 1.4.0 >Reporter: jweinste > > spark-class > There is an illegal statement. > done < <("$RUNNER" -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main > "$@") > Complaint is > ./bin/spark-class: line 100: syntax error near unexpected token `<' -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-8417) spark-class has illegal statement
[ https://issues.apache.org/jira/browse/SPARK-8417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14593650#comment-14593650 ] jweinste edited comment on SPARK-8417 at 6/19/15 5:13 PM: -- I saw this issue when running local on mac.. But the command itself looks rather suspect. My feeling is there is a more simple and straight forward way to accomplish the command creation. I tried a few variations on the current script using things like <<< and "$(...)" to no avail. was (Author: blipe): I saw this issue when running local on mac.. But the command itself looks rather suspect. My feeling is there is a more simple and straight forward way to accomplish the command creation. I tried few variations on the current script using things like <<< and "$(...)" to no avail. > spark-class has illegal statement > - > > Key: SPARK-8417 > URL: https://issues.apache.org/jira/browse/SPARK-8417 > Project: Spark > Issue Type: Bug > Components: Deploy >Affects Versions: 1.4.0 >Reporter: jweinste > > spark-class > There is an illegal statement. > done < <("$RUNNER" -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main > "$@") > Complaint is > ./bin/spark-class: line 100: syntax error near unexpected token `<' -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org