Re: Spark-shell doesn't see changes coming from Kafka topic

2016-12-05 Thread Otávio Carvalho
k at some point or is it > continuously running but not finding latest offsets? > > > On Thu, Dec 1, 2016 at 6:31 AM, Otávio Carvalho <otav...@gmail.com> wrote: > >> Hello hivemind, >> >> I am trying to connect my Spark 2.0.2 cluster to an Apache Kafka 0.10 &

Spark-shell doesn't see changes coming from Kafka topic

2016-12-01 Thread Otávio Carvalho
ssing rate: 0.0 rows/sec* *Trigger details:* *triggerId: -1* *Sink status - org.apache.spark.sql.execution.streaming.ConsoleSink@54d5b6cb* *Committed offsets: [-]* I am starting the spark-shell as follows: /root/spark/bin/spark-shell --packages org.apache.spark:spark-sql-kafka-0-10_2.10:2.0.2 Tha

Re: Spark Performace Comparison Spark on YARN vs Spark Standalone

2014-05-23 Thread Otávio Carvalho
I have been analyzing Storm performance and there's no significant overhead added to the processing nodes. I'm interested in those results over Spark as well. Thanks in advance, Otávio Carvalho. Undergrad. CompSci Student at UFRGS Porto Alegre, Brazil. 2014-05-20 18:46 GMT-03:00 anishs

Re: Use SparkListener to get overall progress of an action

2014-05-23 Thread Otávio Carvalho
Mayur, I'm interested on it as well. Can you send me? Cheers, Otávio Carvalho. Undergrad. Student at Federal University of Rio Grande do Sul Porto Alegre, Brazil. 2014-05-23 11:00 GMT-03:00 Pierre Borckmans pierre.borckm...@realimpactanalytics.com: That would be great, Mayur, thanks