Re: libhdfs install dep

I'd recommend using the packages for Apache Hadoop from Apache Bigtop (https://cwiki.apache.org/confluence/display/BIGTOP). The ones upstream (here) aren't maintained as much these days. On Tue, Sep 25, 2012 at 6:27 PM, Pastrana, Rodrigo (RIS-BCT) wrote: > Leo, yes I'm working with hadoop-1.0.1-1

Re: Python + hdfs written thrift sequence files: lots of moving parts!

Thanks harsh: In any case, I'm really curious about how it is that sequence file headers are formatted, as the documentation in the SequenceFile javadocs seems to be very generic. To make my questions more concrete: 1) I notice that the FileSplit class has a getStart() function. It is documented

Re: Python + hdfs written thrift sequence files: lots of moving parts!

Hi Jay, This may be off-topic to you, but I feel its related: Use Avro DataFiles. There's Python support already available, as well as several other languages. On Tue, Sep 25, 2012 at 10:57 PM, Jay Vyas wrote: > Hi guys! > > Im trying to read some hadoop outputted thrift files in plain old java

RE: libhdfs install dep

I see, The apache RPM distro does include the libhdfs* binaries. (And it is self-contained) For the Apache Distro (once installed) Yum,zipper,KPack* only knows about hadoop-1.0.1-1.amd* as the provider for the libhdfs (check with rpm -qlp hadoop.rpm -- this will give you all of the f

Python + hdfs written thrift sequence files: lots of moving parts!

Hi guys! Im trying to read some hadoop outputted thrift files in plain old java (without using SequenceFile.Reader). The reason for this is that I (1) want to understand the sequence file format better and (2) would like to be able to port this code to a language which doesnt have robust hadoop

Re: libhdfs install dep

Hi Rodrigo, The hadoop RPMs are a bit deficient compared to those you would find from your Linux distribution. For example, look at the Apache RPM you used: [bbockelm@rcf-bockelman ~]$ rpm -qp http://mirrors.sonic.net/apache/hadoop/common/hadoop-1.0.3/hadoop-1.0.3-1.x86_64.rpm --provides hado

Re: Hadoop and Cuda , JCuda (CPU+GPU architecture)

Hi Sudha Good question. First of all, you need to specify clearly about your Hadoop environment, (pseudo distributed or real cluster) Secondly, you need to clearly understand how hadoop load job's jar file to all worker nodes, it only copy the jar file to worker nodes. It does not contain the jc

Re: Passing Command-line Parameters to the Job Submit Command

You could always write your own properties file and read it as resource. On Tue, Sep 25, 2012 at 12:10 AM, Hemanth Yamijala wrote: > By java environment variables, do you mean the ones passed as > -Dkey=value ? That's one way of passing them. I suppose another way is > to have a client side site

RE: libhdfs install dep

Leo, yes I'm working with hadoop-1.0.1-1.amd64.rpm from Apache's download site. The rpm installs libhdfs in /usr/lib64 so I'm not sure why I would need the hadoop-<*>libhdfs* rpm. Any idea why the installed /usr/lib64/libhdfs.so is not detected by the package managers? Thanks, Rodrigo. -Or

Re: Passing Command-line Parameters to the Job Submit Command

Building on Hemanth answer : at the end your variables should be in the job.xml (the second file needed with the jar to run a job). Building this job.xml can be done in various way but it does inherit from your local configuration and you can change it using the java API but at the end it is only a

Re: Passing Command-line Parameters to the Job Submit Command

By java environment variables, do you mean the ones passed as -Dkey=value ? That's one way of passing them. I suppose another way is to have a client side site configuration (like mapred-site.xml) that is in the classpath of the client app. Thanks Hemanth On Tue, Sep 25, 2012 at 12:20 AM, Varad M