Re: parallel cat

2011-07-07 Thread Rita
Thanks again Steve. I will try to implement it with thrift. On Thu, Jul 7, 2011 at 5:35 AM, Steve Loughran wrote: > On 07/07/11 08:22, Rita wrote: > >> Thanks Steve. This is exactly what I was looking for. Unfortunately, I don >> see any example code for the implementation. >> >> > No. I think

Re: parallel cat

2011-07-07 Thread Steve Loughran
On 07/07/11 08:22, Rita wrote: Thanks Steve. This is exactly what I was looking for. Unfortunately, I don see any example code for the implementation. No. I think I have access to russ's source somewhere, but there'd be paperwork in getting it released. Russ said it wasn't too hard to do, he

Re: parallel cat

2011-07-07 Thread Rita
Thanks Steve. This is exactly what I was looking for. Unfortunately, I don see any example code for the implementation. On Wed, Jul 6, 2011 at 7:35 AM, Steve Loughran wrote: > On 06/07/11 11:08, Rita wrote: > >> I have many large files ranging from 2gb to 800gb and I use hadoop fs -cat >> a >>

Re: parallel cat

2011-07-06 Thread Steve Loughran
On 06/07/11 11:08, Rita wrote: I have many large files ranging from 2gb to 800gb and I use hadoop fs -cat a lot to pipe to various programs. I was wondering if its possible to prefetch the data for clients with more bandwidth. Most of my clients have 10g interface and datanodes are 1g. I was th