Yeah, the bmz itself is tested better than the block compressor
itself, which is just covered by a simple regression. The optimization
is copied from zlib's block compressor but missed a output.reserve...
for some reason. Looks like we should handle checksum and NONE in a
different layer and reduce the complexity of the per codec
implementation. OTOH, what kind of data you're inserting, Josh?
uncompressible random bits?
On Aug 28, 9:21 am, "Doug Judd" <[EMAIL PROTECTED]> wrote:
> I can see the bug in the code snippet you provided! :) Basically, there's
> an optimization where if the size of the compressed buffer is not
> significantly less than the uncompressed buffer, then the block compressor
> will store the uncompressed data as-is (e.g. compression_type == NONE). It
> looks like there is a bug inside the BMZ decompression logic where it is not
> properly handling this case. Luke wrote the BMZ compressor, so he'd be the
> best person to take care of this one.
>
> - Doug
>
> On Wed, Aug 27, 2008 at 7:33 PM, Joshua Taylor <[EMAIL PROTECTED]>wrote:
>
> > Is the BMZ compressor supported? The RangeServer crashed in the BMZ codec
> > when I was dumping 500k random cells into a table using COMPRESSOR="bmz":
>
> > Core was generated by
> > `/data1/home/cosmix/hypertable/bin/Hypertable.RangeServer
> > --pidfile=/data1/home/'.
> > Program terminated with signal 11, Segmentation fault.
> > #0 0x0000000000608723 in Hypertable::BlockCompressionCodecBmz::inflate
> > (this=0x174db00, [EMAIL PROTECTED], [EMAIL PROTECTED], [EMAIL PROTECTED])
> > at
> > /home/josh/hypertable/src/cc/Hypertable/Lib/BlockCompressionCodecBmz.cc:114
> > 114
> > /home/josh/hypertable/src/cc/Hypertable/Lib/BlockCompressionCodecBmz.cc: No
> > such file or directory.
> > in
> > /home/josh/hypertable/src/cc/Hypertable/Lib/BlockCompressionCodecBmz.cc
> > (gdb) directory hypertable/src/cc/Hypertable/Lib
> > Source directories searched:
> > /data1/home/cosmix/hypertable/src/cc/Hypertable/Lib:$cdir:$cwd
> > (gdb) list
> > 109 Error::BLOCK_COMPRESSOR_CHECKSUM_MISMATCH);
> > 110
> > 111 size_t outlen = header.get_data_length();
> > 112
> > 113 if (header.get_compression_type() == NONE)
> > 114 memcpy(output.base, ip, outlen);
> > 115 else {
> > 116 output.reserve(outlen);
> > 117 m_workmem.reserve(bmz_unpack_worklen(outlen), true);
> > 118
> > (gdb) print output
> > $1 = (Hypertable::DynamicBuffer &) @0x1e0aac0: {base = 0x0, ptr = 0x0, size
> > = 0, own = true}
> > (gdb) print ip
> > $2 = (const uint8_t *) 0xbf281a ""
> > (gdb) print outlen
> > $3 = 76
> > (gdb) bt
> > #0 0x0000000000608723 in Hypertable::BlockCompressionCodecBmz::inflate
> > (this=0x174db00, [EMAIL PROTECTED], [EMAIL PROTECTED], [EMAIL PROTECTED])
> > at
> > /home/josh/hypertable/src/cc/Hypertable/Lib/BlockCompressionCodecBmz.cc:114
> > #1 0x000000000059f859 in Hypertable::CellStoreV0::load_index
> > (this=0x1e0aa00) at
> > /home/josh/hypertable/src/cc/Hypertable/RangeServer/CellStoreV0.cc:474
> > #2 0x000000000058b0f0 in Hypertable::AccessGroup::shrink (this=0xeefa00,
> > [EMAIL PROTECTED]) at
> > /home/josh/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:489
> > #3 0x000000000057911f in Hypertable::Range::split_compact_and_shrink
> > (this=0xeffc80) at
> > /home/josh/hypertable/src/cc/Hypertable/RangeServer/Range.cc:563
> > #4 0x000000000057ad95 in Hypertable::Range::split (this=0xeffc80) at
> > /home/josh/hypertable/src/cc/Hypertable/RangeServer/Range.cc:341
> > #5 0x0000000000576aad in Hypertable::MaintenanceTaskSplit::execute
> > (this=0x1c2e9c0) at
> > /home/josh/hypertable/src/cc/Hypertable/RangeServer/MaintenanceTaskSplit.cc:40
> > #6 0x0000000000561d0a in Hypertable::MaintenanceQueue::Worker::operator()
> > (this=0x4f017158) at
> > /home/josh/hypertable/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:108
> > #7 0x0000000000561e73 in
> > boost::detail::function::void_function_obj_invoker0<Hypertable::MaintenanceQueue::Worker,
> > void>::invoke ([EMAIL PROTECTED])
> > at
> > /home/josh/hypertable/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> > #8 0x00002aaaab62551e in boost::function0<void,
> > std::allocator<boost::function_base> >::operator() () from
> > /usr/lib64/libboost_thread-mt.so.3
> > #9 0x00002aaaab624f32 in boost::thread_group::join_all () from
> > /usr/lib64/libboost_thread-mt.so.3
> > #10 0x0000003986106337 in start_thread () from /lib64/libpthread.so.0
> > #11 0x0000003790bcc38d in clone () from /lib64/libc.so.6
> > #12 0x0000000000000000 in ?? ()
> > (gdb)
>
> > Looks like a memcpy to a null output buffer. Here's the tail of the
> > RangeServer log:
>
> > 1219889447 INFO Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:700)
> > Successfully fetched 65453 bytes of scan data
> > RangeServer::fetch_scanblock
> > Scanner ID = 1
> > 1219889447 INFO Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:700)
> > Successfully fetched 65453 bytes of scan data
> > RangeServer::fetch_scanblock
> > Scanner ID = 1
> > 1219889447 INFO Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:700)
> > Successfully fetched 46970 bytes of scan data
> > 1219889447 INFO Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:656)
> > destroying scanner id=1
> > 1219889447 INFO Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:177)
> > Event: type=DISCONNECT from=10.10.1.162:43904
> > 1219889453 INFO Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1883)
> > Cleaning log (threshold=200000000)
> > 1219889456 INFO Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/AsyncComm/IOHandler.h:81) Event:
> > type=CONNECTION_ESTABLISHED from=10.10.1.162:43909
> > 1219889456 INFO Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:167)
> > Event: type=CONNECTION_ESTABLISHED from=10.10.1.162:43909
> > RangeServer::update
> > {TableIdentifier: name='perf' id='19' generation='1'}1219889457 INFO
> > Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1188)
> > Added 16394 (0 split off) updates to 'perf'
> > 1219889457 INFO Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1286)
> > drj mem=4123295 items=67595 vm-est12234695
> > RangeServer::update
> > {TableIdentifier: name='perf' id='19' generation='1'}1219889457 INFO
> > Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1188)
> > Added 16394 (0 split off) updates to 'perf'
> > 1219889457 INFO Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1286)
> > drj mem=5123329 items=83989 vm-est15202009
> > RangeServer::update
> > {TableIdentifier: name='perf' id='19' generation='1'}1219889457 INFO
> > Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1188)
> > Added 16394 (0 split off) updates to 'perf'
> > 1219889457 INFO Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1286)
> > drj mem=6123363 items=100383 vm-est18169323
> > 1219889457 INFO Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:79) RollLimit =
> > 100000000
> > 1219889457 DEBUG Hypertable.RangeServer : write
> > (/home/josh/hypertable/src/cc/Hypertable/Lib/MetaLogDfsBase.cc:117):
> > checksum=12916318 timestamp=1219889457515443000 type=1 payload=149
> > 1219889457 INFO Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:288)
> > Starting Major Compaction of perf[..<FF><FF>](default)
> > RangeServer::update
> > {TableIdentifier: name='perf' id='19' generation='1'}1219889457 INFO
> > Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1188)
> > Added 16394 (8518 split off) updates to 'perf'
> > 1219889457 INFO Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1286)
> > drj mem=7123397 items=116777 vm-est21136637
> > RangeServer::update
> > {TableIdentifier: name='perf' id='19' generation='1'}1219889457 INFO
> > Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1188)
> > Added 16394 (9950 split off) updates to 'perf'
> > 1219889457 INFO Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1286)
> > drj mem=8123431 items=133171 vm-est24103951
> > RangeServer::update
> > {TableIdentifier: name='perf' id='19' generation='1'}1219889458 INFO
> > Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1188)
> > Added 16394 (8246 split off) updates to 'perf'
> > 1219889458 INFO Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1286)
> > drj mem=9123465 items=149565 vm-est27071265
> > RangeServer::update
> > {TableIdentifier: name='perf' id='19' generation='1'}1219889458 INFO
> > Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1188)
> > Added 16394 (7358 split off) updates to 'perf'
> > 1219889458 INFO Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1286)
> > drj mem=10123499 items=165959 vm-est30038579
> > RangeServer::update
> > {TableIdentifier: name='perf' id='19' generation='1'}1219889458 INFO
> > Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1188)
> > Added 16394 (10666 split off) updates to 'perf'
> > 1219889458 INFO Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1286)
> > drj mem=11123533 items=182353 vm-est33005893
> > RangeServer::update
> > {TableIdentifier: name='perf' id='19' generation='1'}1219889458 INFO
> > Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1188)
> > Added 16394 (7086 split off) updates to 'perf'
> > 1219889458 INFO Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1286)
> > drj mem=12123567 items=198747 vm-est35973207
> > RangeServer::update
> > {TableIdentifier: name='perf' id='19' generation='1'}1219889459 INFO
> > Hypertable.RangeServer :
> > (/home/josh/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:430)
> > Finished Compaction of perf[..<FF><FF>](default)
> > 1219889459 INFO Hypertable.RangeServer :
>
> ...
>
> read more »
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Hypertable Development" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/hypertable-dev?hl=en
-~----------~----~----~----~------~----~------~--~---
