Min, can you check the default compression codec in your hadoop conf? The 8-byte file must be a compressed file using the codec which represents 0-length file.
It seems that codec was not able to decompress the stream. Zheng On Sun, Nov 1, 2009 at 10:49 PM, Min Zhou <coderp...@gmail.com> wrote: > I think there may be a bug still in this release. > > hive>select stuff_status from auctions where auction_id='2591238417' > and pt='20091027'; > > auctions is a table partitioned by date, it stored as a textfile w/o > compression. The query above should return 0 rows. > but when hive.exec.compress.output=true, hive will crash with a > StackOverflowError > > java.lang.StackOverflowError > at java.lang.ref.FinalReference.<init>(FinalReference.java:16) > at java.lang.ref.Finalizer.<init>(Finalizer.java:66) > at java.lang.ref.Finalizer.register(Finalizer.java:72) > at java.lang.Object.<init>(Object.java:20) > at java.net.SocketImpl.<init>(SocketImpl.java:27) > at java.net.PlainSocketImpl.<init>(PlainSocketImpl.java:90) > at java.net.SocksSocketImpl.<init>(SocksSocketImpl.java:33) > at java.net.Socket.setImpl(Socket.java:434) > at java.net.Socket.<init>(Socket.java:68) > at sun.nio.ch.SocketAdaptor.<init>(SocketAdaptor.java:50) > at sun.nio.ch.SocketAdaptor.create(SocketAdaptor.java:55) > at sun.nio.ch.SocketChannelImpl.socket(SocketChannelImpl.java:105) > at > org.apache.hadoop.net.StandardSocketFactory.createSocket(StandardSocketFactory.java:58) > at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1540) > at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1662) > at java.io.DataInputStream.read(DataInputStream.java:132) > at > org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:96) > at > org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:86) > at > org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74) > at java.io.InputStream.read(InputStream.java:85) > at org.apache.hadoop.util.LineReader.backfill(LineReader.java:82) > at org.apache.hadoop.util.LineReader.readLine(LineReader.java:112) > at > org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:134) > at > org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:39) > at > org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:256) > at > org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:272) > > Each mapper will produce a 8 bytes deflate file on hdfs(we set > hive.merge.mapfiles=false), their hex representation is like below: > > 78 9C 03 00 00 00 00 01 > > This is the reason why FetchOperator:272 is called recursively, and > caused a stack overflow error. > > Regards, > Min > > > On Mon, Nov 2, 2009 at 6:34 AM, Zheng Shao <zsh...@gmail.com> wrote: >> I have made a release candidate 0.4.1-rc0. >> >> We've fixed several critical bugs to hive release 0.4.0. We need hive >> release 0.4.1 out asap. >> >> Here are the list of changes: >> >> HIVE-884. Metastore Server should call System.exit() on error. >> (Zheng Shao via pchakka) >> >> HIVE-864. Fix map-join memory-leak. >> (Namit Jain via zshao) >> >> HIVE-878. Update the hash table entry before flushing in Group By >> hash aggregation (Zheng Shao via namit) >> >> HIVE-882. Create a new directory every time for scratch. >> (Namit Jain via zshao) >> >> HIVE-890. Fix cli.sh for detecting Hadoop versions. (Paul Huff via zshao) >> >> HIVE-892. Hive to kill hadoop jobs using POST. (Dhruba Borthakur via >> zshao) >> >> HIVE-883. URISyntaxException when partition value contains special chars. >> (Zheng Shao via namit) >> >> >> Please vote. >> >> -- >> Yours, >> Zheng >> > > > > -- > My research interests are distributed systems, parallel computing and > bytecode based virtual machine. > > My profile: > http://www.linkedin.com/in/coderplay > My blog: > http://coderplay.javaeye.com > -- Yours, Zheng
