[ria...@tyu-linux kevinweil-hadoop-lzo-916aeae]$ ./a.out Hello World! [ria...@tyu-linux kevinweil-hadoop-lzo-916aeae]$ gcc -v Using built-in specs. Target: x86_64-redhat-linux Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-libgcj-multifile --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --enable-plugin --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic --host=x86_64-redhat-linux Thread model: posix gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
On Fri, Jan 15, 2010 at 1:47 PM, Alex Kozlov <ale...@cloudera.com> wrote: > Looks like your compiler is misconfigured. Can you try and compile a > simple > "Hello World!" program: > > [ale...@monster01 ~]$ cat Hello.c > #include <stdio.h> > int main() { > printf("Hello World!\n"); > } > [ale...@monster01 ~]$ gcc Hello.c > [ale...@monster01 ~]$ ./a.out > Hello World! > [ale...@monster01 ~]$ gcc -v > Using built-in specs. > Target: x86_64-redhat-linux > Configured with: ../configure --prefix=/usr --mandir=/usr/share/man > --infodir=/usr/share/info --enable-shared --enable-threads=posix > --enable-checking=release --with-system-zlib --enable-__cxa_atexit > --disable-libunwind-exceptions --enable-libgcj-multifile > --enable-languages=c,c++,objc,obj-c++,java,fortran,ada > --enable-java-awt=gtk > --disable-dssi --enable-plugin > --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic > --host=x86_64-redhat-linux > Thread model: posix > gcc version 4.1.2 20080704 (Red Hat 4.1.2-46) > > Alex > > On Fri, Jan 15, 2010 at 1:34 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > I tried to set LD_LIBRARY_PATH to point to where crt1.o resides: > > export LD_LIBRARY_PATH=/usr/lib64 > > > > Still the same issue. > > This is snippet of config.log: > > > > Configured with: ../configure --prefix=/usr --mandir=/usr/share/man > > --infodir=/usr/share/info --enable-shared --enable-threads=posix > > --enable-checking=release --with-system-zlib --enable-__cxa_atexit > > --disable-libunwind-exceptions --enable-libgcj-multifile > > --enable-languages=c,c++,objc,obj-c++,java,fortran,ada > > --enable-java-awt=gtk > > --disable-dssi --enable-plugin > > --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre > --with-cpu=generic > > --host=x86_64-redhat-linux > > Thread model: posix > > gcc version 4.1.2 20080704 (Red Hat 4.1.2-46) > > configure:2781: $? = 0 > > configure:2788: gcc -V >&5 > > gcc: '-V' option must have argument > > configure:2791: $? = 1 > > configure:2814: checking for C compiler default output file name > > configure:2841: gcc -m32 conftest.c >&5 > > /usr/bin/ld: crt1.o: No such file: No such file or directory > > collect2: ld returned 1 exit status > > configure:2844: $? = 1 > > configure:2882: result: > > configure: failed program was: > > | /* confdefs.h. */ > > | #define PACKAGE_NAME "gpl-compression" > > | #define PACKAGE_TARNAME "gpl-compression" > > | #define PACKAGE_VERSION "0.1.0" > > | #define PACKAGE_STRING "gpl-compression 0.1.0" > > | #define PACKAGE_BUGREPORT "omal...@apache.org" > > | #define PACKAGE "gpl-compression" > > | #define VERSION "0.1.0" > > | #define _GNU_SOURCE 1 > > | /* end confdefs.h. */ > > | > > | int > > | main () > > | { > > | > > | ; > > | return 0; > > | } > > configure:2889: error: C compiler cannot create executables > > See `config.log' for more details. > > > > > > On Fri, Jan 15, 2010 at 1:17 PM, Todd Lipcon <t...@cloudera.com> wrote: > > > > > Are you starting from a clean tarball of the lzo stuff? > > > > > > Can you make sure your /tmp/ partition isn't full? > > > > > > There should be a config.log file hanging around somewhere also - > perhaps > > > this would be helpful. > > > > > > -Todd > > > > > > On Fri, Jan 15, 2010 at 1:14 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > > > > > Todd: > > > > Thanks for the continued support. > > > > > > > > I installed lzo-devel: > > > > [r...@tyu-linux batchclient]# rpm -ivh > > > > /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm > > > > warning: /opt/kindsight/lzo-devel-2.02-2.el5.1.x86_64.rpm: Header V3 > > DSA > > > > signature: NOKEY, key ID 217521f6 > > > > Preparing... > ########################################### > > > > [100%] > > > > 1:lzo-devel > ########################################### > > > > [100%] > > > > > > > > But I still see: > > > > [exec] Try `/usr/bin/install --help' for more information. > > > > [exec] checking for gcc... gcc > > > > [exec] checking for C compiler default output file name... > > > > [exec] configure: error: C compiler cannot create executables > > > > > > > > > > > > On Fri, Jan 15, 2010 at 12:09 PM, Todd Lipcon <t...@cloudera.com> > > wrote: > > > > > > > > > Hi Ted, > > > > > > > > > > Did you also install liblzo-devel? > > > > > > > > > > Here are the packages I install for LZO: > > > > > > > > > > lzo-2.02-2.el5.1 > > > > > lzo-devel-2.02-2.el5.1 > > > > > > > > > > -Todd > > > > > > > > > > On Tue, Jan 12, 2010 at 4:28 PM, Ted Yu <yuzhih...@gmail.com> > wrote: > > > > > > > > > > > Package gcc-4.1.2-46.el5_4.1.x86_64 already installed and latest > > > > version > > > > > > Nothing to do > > > > > > > > > > > > [r...@tyu-linux batchclient]# yum install gcc-c++ > > > > > > Loaded plugins: fastestmirror > > > > > > Loading mirror speeds from cached hostfile > > > > > > * base: mirrors.kernel.org > > > > > > * updates: updates.interworx.info > > > > > > * addons: mirrors.kernel.org > > > > > > * extras: mirrors.netdna.com > > > > > > Setting up Install Process > > > > > > Parsing package install arguments > > > > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and > > latest > > > > > > version > > > > > > > > > > > > > > > > > > On Tue, Jan 12, 2010 at 2:32 PM, Todd Lipcon <t...@cloudera.com> > > > > wrote: > > > > > > > > > > > > > Make sure you have gcc, g++ installed? > > > > > > > > > > > > > > -Todd > > > > > > > > > > > > > > On Tue, Jan 12, 2010 at 10:57 AM, Ted Yu <yuzhih...@gmail.com> > > > > wrote: > > > > > > > > > > > > > > > I installed > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ftp://fr.rpmfind.net/linux/EPEL/5/x86_64/lzo-2.02-2.el5.1.x86_64.rpmyesterday > > > > > > > > . > > > > > > > > > > > > > > > > [r...@tyu-linux software]# rpm -e lzo > > > > > > > > [r...@tyu-linux software]# rpm -ivh > > > > > > ~rialto/lzo-2.02-2.el5.1.x86_64.rpm > > > > > > > > warning: /home/rialto/lzo-2.02-2.el5.1.x86_64.rpm: Header V3 > > DSA > > > > > > > signature: > > > > > > > > NOKEY, key ID 217521f6 > > > > > > > > Preparing... > > > > > ########################################### > > > > > > > > [100%] > > > > > > > > 1:lzo > > > > > ########################################### > > > > > > > > [100%] > > > > > > > > > > > > > > > > [ria...@tyu-linux java]$ ls -l /usr/lib64/liblzo2.so.2* > > > > > > > > lrwxrwxrwx 1 root root 16 Jan 12 10:29 > > > /usr/lib64/liblzo2.so.2 > > > > -> > > > > > > > > liblzo2.so.2.0.0 > > > > > > > > -rwxr-xr-x 1 root root 123408 May 6 2007 > > > > > /usr/lib64/liblzo2.so.2.0.0 > > > > > > > > > > > > > > > > I still see this error: > > > > > > > > [exec] configure: error: C compiler cannot create > > > executables > > > > > > > > [exec] See `config.log' for more details. > > > > > > > > > > > > > > > > BUILD FAILED > > > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec > > > > > returned: > > > > > > > 77 > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jan 12, 2010 at 10:25 AM, Todd Lipcon < > > t...@cloudera.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Ted, > > > > > > > > > > > > > > > > > > Please make sure you have version 2.02 of liblzo installed. > > > > There's > > > > > > an > > > > > > > > > earlier version in RHEL that doesn't fail properly at the > > > > configure > > > > > > > stage > > > > > > > > > and I've seen it produce the error you pasted below. > > > > > > > > > > > > > > > > > > -Todd > > > > > > > > > > > > > > > > > > On Tue, Jan 12, 2010 at 10:15 AM, Ted Yu < > > yuzhih...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > I followed > > > > > > http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ > > > > > > > > > > > > > > > > > > > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed > > and > > > > > > latest > > > > > > > > > > version > > > > > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14 > > 06:36:37 > > > > EDT > > > > > > > 2009 > > > > > > > > > > x86_64 x86_64 x86_64 GNU/Linux > > > > > > > > > > > > > > > > > > > > For 32 bit build, I got: > > > > > > > > > > [exec] checking for C compiler default output file > > > name... > > > > > > > > > > [exec] configure: error: C compiler cannot create > > > > executables > > > > > > > > > > [exec] See `config.log' for more details. > > > > > > > > > > > > > > > > > > > > BUILD FAILED > > > > > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: > > exec > > > > > > > returned: > > > > > > > > > 77 > > > > > > > > > > > > > > > > > > > > For 64 bit build, I got: > > > > > > > > > > [exec] gcc -DHAVE_CONFIG_H -I. > > > > > > > > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native > > > -I./impl > > > > > > > > > > -I/usr/java/default/include > > -I/usr/java/default/include/linux > > > > > > > > > > > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl > > > > > > > > > > -Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2 -m64 > > -m64 > > > > -MT > > > > > > > > > > impl/lzo/LzoCompressor.lo -MD -MP -MF > > > > > > > impl/lzo/.deps/LzoCompressor.Tpo > > > > > > > > -c > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c > > > > > > > > > > -fPIC -DPIC -o impl/lzo/.libs/LzoCompressor.o > > > > > > > > > > [exec] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c: > > > > > > > > > > In function > > > > > > 'Java_com_hadoop_compression_lzo_LzoCompressor_initIDs': > > > > > > > > > > [exec] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:123: > > > > > > > > > > error: expected expression before ',' token > > > > > > > > > > [exec] make: *** [impl/lzo/LzoCompressor.lo] Error 1 > > > > > > > > > > > > > > > > > > > > BUILD FAILED > > > > > > > > > > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:251: > > exec > > > > > > > returned: > > > > > > > > 2 > > > > > > > > > > > > > > > > > > > > Has anybody seen the above ? > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > On Mon, Jan 11, 2010 at 3:34 PM, Todd Lipcon < > > > > t...@cloudera.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Ted, > > > > > > > > > > > > > > > > > > > > > > You need to install liblzo from EPEL: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://fr.rpmfind.net/linux/RPM/Extras_Packages_for_Enterprise_Linux.html > > > > > > > > > > > > > > > > > > > > > > -Todd > > > > > > > > > > > > > > > > > > > > > > On Mon, Jan 11, 2010 at 3:21 PM, Ted Yu < > > > yuzhih...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Can someone tell me how I can install liblzo ? > > > > > > > > > > > > > > > > > > > > > > > > [r...@tyu-linux lzo-2.03]# uname -a > > > > > > > > > > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14 > > > > 06:36:37 > > > > > > EDT > > > > > > > > > 2009 > > > > > > > > > > > > x86_64 x86_64 x86_64 GNU/Linux > > > > > > > > > > > > [r...@tyu-linux lzo-2.03]# yum install liblzo-devel > > > > > > > > > > > > Loaded plugins: fastestmirror > > > > > > > > > > > > Loading mirror speeds from cached hostfile > > > > > > > > > > > > * base: mirrors.usc.edu > > > > > > > > > > > > * updates: mirror.san.fastserv.com > > > > > > > > > > > > * addons: centos.promopeddler.com > > > > > > > > > > > > * extras: mirrors.versaweb.com > > > > > > > > > > > > Setting up Install Process > > > > > > > > > > > > Parsing package install arguments > > > > > > > > > > > > No package liblzo-devel available. > > > > > > > > > > > > Nothing to do > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jan 11, 2010 at 12:45 PM, Steve Kuo < > > > > > > kuosen...@gmail.com > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Ted, > > > > > > > > > > > > > > > > > > > > > > > > > > You may want to consider LZO compression, which > > allows > > > > > > > splitting > > > > > > > > > for > > > > > > > > > > a > > > > > > > > > > > > > comporessed file for Map jobs. On the other hand, > > gzip > > > > is > > > > > > not > > > > > > > > > > > > splittable. > > > > > > > > > > > > > > > > > > > > > > > > > > Check out these links. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/ > > > > > > > > > > > > > http://wiki.apache.org/hadoop/UsingLzoCompression > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jan 8, 2010 at 1:13 PM, Ted Yu < > > > > > yuzhih...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > The input file is in .gz format > > > > > > > > > > > > > > FYI > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jan 8, 2010 at 11:08 AM, Ted Yu < > > > > > > yuzhih...@gmail.com > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > My current project processes input file of size > > > > > 333302161 > > > > > > > > > bytes. > > > > > > > > > > > > > > > What I plan to do is to split the file into > equal > > > > size > > > > > > > pieces > > > > > > > > > > (and > > > > > > > > > > > on > > > > > > > > > > > > > > blank > > > > > > > > > > > > > > > line boundary) to improve performance. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I found 12 classes in 0.20.1 source code which > > > > > implement > > > > > > > > > > > InputSplit. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If someone has written code similar to what I > > plan > > > to > > > > > do, > > > > > > > > > please > > > > > > > > > > > > share > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > hint. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jan 8, 2010 at 2:27 AM, Amogh Vasekar < > > > > > > > > > > am...@yahoo-inc.com > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hi, > > > > > > > > > > > > > > >> The deprecation is due to the new evolving > > > mapreduce > > > > ( > > > > > > > > > > > > o.a.h.mapreduce > > > > > > > > > > > > > ) > > > > > > > > > > > > > > >> APIs. Old APIs are supported for available > > > > > > distributions. > > > > > > > > The > > > > > > > > > > > > > equivalent > > > > > > > > > > > > > > of > > > > > > > > > > > > > > >> TextInputFormat is available in new API : > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> Thanks, > > > > > > > > > > > > > > >> Amogh > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> On 1/8/10 3:47 AM, "Ted Yu" < > > yuzhih...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> According to: > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29 > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> isSplitable() is deprecated. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> Which method should I use to replace it ? > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> Thanks > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >