Thanks Mark for those tips.. I was mostly trying around the third option but it didn¹t work well as the data we are playing with is huge..
The problem with 1 and 2 options is we cannot move or update that directory. On 22/05/15 7:25 pm, "Mark Payne" <[email protected]> wrote: >The List/Fetch HDFS would allow you to pull new data from HDFS without >destroying it. > ________________________________ The information contained in this message may be confidential and legally protected under applicable law. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction of this message is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message.
