Unsubscribe

[郭]

Unsubscribe


thanks!




郭
祝工作顺利、万事胜意

Re: use kafka streams API aggregate ?

[郭鹏飞]

hi,
Today I do it too.


check your kafka version, then follow  one of the guides below.


http://spark.apache.org/docs/latest/streaming-kafka-0-8-integration.html

http://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html


http://spark.apache.org/docs/latest/streaming-kafka-integration.html







> 在 2018年1月30日，下午10:48，446463...@qq.com 写道：
> 
> Hi
> I am new to kafka.
> today I use kafka streams API for real timing process data
> and I have no idea with this
> Can someone help me ?
> 
> 446463...@qq.com 

Re: jar file problem

[郭鹏飞]

You can use bin/spark-submit tool to submit you jar to the cluster.

> 在 2017年10月19日，下午11:24，Uğur Sopaoğlu  写道：
> 
> Hello, 
> 
> I have a very easy problem. How I run a spark job, I must copy jar file to 
> all worker nodes. Is there any way to do simple?.
> 
> -- 
> Uğur Sopaoğlu


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Hive From Spark: Jdbc VS sparkContext

[郭鹏飞]

> 在 2017年10月4日，上午2:08，Nicolas Paris  写道：
> 
> Hi
> 
> I wonder the differences accessing HIVE tables in two different ways:
> - with jdbc access
> - with sparkContext
> 
> I would say that jdbc is better since it uses HIVE that is based on
> map-reduce / TEZ and then works on disk. 
> Using spark rdd can lead to memory errors on very huge datasets.
> 
> 
> Anybody knows or can point me to relevant documentation ?
> 
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org


The jdbc will load data into the driver node, this may slow down the speed,and 
may OOM.


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Unsubscribe

[郭鹏飞]

Unsubscribe


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

problem for submitting job

[郭谦]

HI,

I'm a junior user of spark from China.

I have a problem about submit spark job right now. I want to submit job
from code.

In other words ,How to submit spark job from within java program to yarn
cluster without using spark-submit


   I've learnt from official site
http://spark.apache.org/docs/latest/submitting-applications.html

that using  bin/spark-submit script to submit a job to cluster is easy .


   Because the script may does lots of complex work such as setting up
the classpath with Spark and its dependencies.

If I don't use the script ,I have to deal with all complex work by
myself.It makes me feel really frustrated.


   I have search this problem from Google,but the answers may not suit
for me .


   In hadoop developing ,I know that after setting up Configuration
,Job and resources ,

we can submit hadoop job by coding like this:

job.waitForCompletion

It is convenient for users to submit job programmatically


I want to know if there is a schedule( may be in spark 1.5+?)that provide
users variety ways of submitting job like hadoop .

Like monitoring ,In the recent release spark(1.4.0) We can get statements
about spark applications by REST API right now.


Thanks  Regards

GUO QIAN