Re: use kafka streams API aggregate ?

2018-01-30 Thread 郭鹏飞 

hi,
Today I do it too.


check your kafka version, then follow  one of the guides below.


http://spark.apache.org/docs/latest/streaming-kafka-0-8-integration.html

http://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html


http://spark.apache.org/docs/latest/streaming-kafka-integration.html







> 在 2018年1月30日，下午10:48，446463...@qq.com 写道：
> 
> Hi
> I am new to kafka.
> today I use kafka streams API for real timing process data
> and I have no idea with this
> Can someone help me ?
> 
> 446463...@qq.com

Re: jar file problem

2017-10-19 Thread 郭鹏飞 
You can use bin/spark-submit tool to submit you jar to the cluster.

> 在 2017年10月19日，下午11:24，Uğur Sopaoğlu  写道：
> 
> Hello, 
> 
> I have a very easy problem. How I run a spark job, I must copy jar file to 
> all worker nodes. Is there any way to do simple?.
> 
> -- 
> Uğur Sopaoğlu


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Hive From Spark: Jdbc VS sparkContext

2017-10-10 Thread 郭鹏飞 

> 在 2017年10月4日，上午2:08，Nicolas Paris  写道：
> 
> Hi
> 
> I wonder the differences accessing HIVE tables in two different ways:
> - with jdbc access
> - with sparkContext
> 
> I would say that jdbc is better since it uses HIVE that is based on
> map-reduce / TEZ and then works on disk. 
> Using spark rdd can lead to memory errors on very huge datasets.
> 
> 
> Anybody knows or can point me to relevant documentation ?
> 
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org


The jdbc will load data into the driver node, this may slow down the speed,and 
may OOM.


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Unsubscribe

2017-08-06 Thread 郭鹏飞 
Unsubscribe


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org