Re: How to write a Job for importing Files from an external Rest API into Hadoop

Hi Ravi, thanks a lot for your response and the code example! I think this will help me a lot to get started .I am glad to see that my idea is not to exotic. I will report if I can adapt the solution for my problem. best regards Ralph On 31.07.2017 22:05, Ravi Prakash wrote: Hi Ralph! Alth

Re: How to write a Job for importing Files from an external Rest API into Hadoop

Hi Ralph! Although not totally similar to your use case, DistCp may be the closest thing to what you want. https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java . The client builds a file list, and then submits an MR job to copy ov

How to write a Job for importing Files from an external Rest API into Hadoop

Hi, I want to ask, what's the best way implementing a Job which is importing files into the HDFS? I have an external System offering data accessible through a Rest API. My goal is to have a job running in Hadoop which is periodical (maybe started by chron?) looking into the Rest API if new d