Marco, Ted, thanks for your time. I am sorry if I wasn't clear enough. We have two sources,
1) sql server 2) files are pushed onto edge node by upstreams on a daily basis. Point 1 has been achieved by using JDBC format in spark sql. Point 2 has been achieved by using shell script. My only concern is about point 2. To see if there is any way I can do it in my spark app instead os shell script. Thanks. On Tuesday, June 7, 2016, Ted Yu <yuzhih...@gmail.com> wrote: > bq. load the data from edge node to hdfs > > Does the loading involve accessing sqlserver ? > > Please take a look at > https://spark.apache.org/docs/latest/sql-programming-guide.html > > On Tue, Jun 7, 2016 at 7:19 AM, Marco Mistroni <mmistr...@gmail.com > <javascript:_e(%7B%7D,'cvml','mmistr...@gmail.com');>> wrote: > >> Hi >> how about >> >> 1. have a process that read the data from your sqlserver and dumps it as >> a file into a directory on your hd >> 2. use spark-streanming to read data from that directory and store it >> into hdfs >> >> perhaps there is some sort of spark 'connectors' that allows you to read >> data from a db directly so you dont need to go via spk streaming? >> >> >> hth >> >> >> >> >> >> >> >> >> >> >> On Tue, Jun 7, 2016 at 3:09 PM, Ajay Chander <itsche...@gmail.com >> <javascript:_e(%7B%7D,'cvml','itsche...@gmail.com');>> wrote: >> >>> Hi Spark users, >>> >>> Right now we are using spark for everything(loading the data from >>> sqlserver, apply transformations, save it as permanent tables in >>> hive) in our environment. Everything is being done in one spark application. >>> >>> The only thing we do before we launch our spark application through >>> oozie is, to load the data from edge node to hdfs(it is being triggered >>> through a ssh action from oozie to run shell script on edge node). >>> >>> My question is, there's any way we can accomplish edge-to-hdfs copy >>> through spark ? So that everything is done in one spark DAG and lineage >>> graph? >>> >>> Any pointers are highly appreciated. Thanks >>> >>> Regards, >>> Aj >>> >> >> >
