The DFS BROKER i/o error certainly would have generated an exception
message in the DfsBroker.hadoop.log file. I think maybe you didn't search
the log archives? You can grep through the archives on one of the machines
as follows:
cd /opt/hypertable/current/log/archive
find. -name DfsBroker.hadoop.log -exec grep -H "Exception" {} \;
Give that a try on all of your machines and let me know what Exceptions (if
any) are reported.
- Doug
On Wed, Dec 11, 2013 at 11:31 PM, Hatice Çobanoğlu <
[email protected]> wrote:
> hypertable 0.9.7.13
>
>
> 2013/12/12 Doug Judd <[email protected]>
>
>> What version of Hypertable are you running?
>>
>>
>> On Tue, Dec 10, 2013 at 5:03 AM, Hatice Çobanoğlu <
>> [email protected]> wrote:
>>
>>> Hi Doug,
>>> i check the DfsBroker.hadoop.log files for the word "Exception" and not
>>> see anything about this word.
>>>
>>> i use version of CDH4 (4.6.0), hadoop 2.0.0, ubuntu 12.04
>>>
>>>
>>> 2013/12/9 Doug Judd <[email protected]>
>>>
>>>> Hi Hatice,
>>>>
>>>> The following error could be the culprit:
>>>>
>>>>
>>>> 1386592901 ERROR ThriftBroker : flush (/root/src/hypertable/src/cc/
>>>> Hypertable/Lib/TableMutator.cc:183): Hypertable::Exception: - DFS
>>>> BROKER i/o error
>>>>
>>>> Please grep through the DfsBroker.hadoop.log files for the word
>>>> "Exception" to see what kind of errors the DfsBroker is encountering.
>>>> Also, what version of Hadoop are you running?
>>>>
>>>> - Doug
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Dec 9, 2013 at 6:57 AM, Hatice Çobanoğlu <
>>>> [email protected]> wrote:
>>>>
>>>>> hi doug,
>>>>> i restart ht (cap stop / cap start), i can select or insert.
>>>>> everything is ok. but thriftbroker log write a lot. one of the thrift log
>>>>> as following:
>>>>>
>>>>> 1386600835.533673000 API mutator_set_cell: API mutator_set_cell:
>>>>> mutator=63393536 cell={Cell: {Key: row='
>>>>> http://allfree-4u.com/fs_gay/gay-twink-porn-movies/link' cf='guid'
>>>>> flag=255} value='031f1713-f57e-4591-8283-bfb445f9a634'} latency=0
>>>>> 1386600835.534667000 API namespace_open: API namespace_open: namespace
>>>>> name=crawler id=1142522706 latency=0
>>>>> 1386600835.534874000 API mutator_set_cell: API mutator_set_cell:
>>>>> mutator=43919680 cell={Cell: {Key: row='929999-fetshop.com'
>>>>> cf='oncelik' cq='b' flag=255} value='0'} latency=0
>>>>> 1386600835.535570000 API hql_exec: API hql_exec: namespace=2860583658
>>>>> hql=select link FROM links where
>>>>> row='designsponge.com_designspongeonline.com_
>>>>> http://designspongeonline.com/category/diy-projects?technique=carpentry&showposts=20&materials=leather'
>>>>> cell_limit 1 noflush=0 unbuffered=0 latency=0
>>>>> 1386600835.536486000 API mutator_set_cell: API mutator_set_cell:
>>>>> mutator=43922560 cell={Cell: {Key:
>>>>> row='so_worked_up_and_horny####999997.009661####d90e93a8-36a3-4f8e-8a40-f2a39325c4f5'
>>>>> cf='urlguid' flag=255} value='tons of pics of hot girls squirting
>>>>> everywhere we have nothing but sexy girls squirting in videos and pictures
>>>>> watch as these hotties get so worked up and horny that their pussies
>>>>> squirt
>>>>> many feet in the air come and see her squirt her load all over herself the
>>>>> cameras and even into some hungry mouths not all girls can squirt like
>>>>> this
>>>>> so we thought we would make a huge collection of only squirting these
>>>>> chicks love getting wet and wild in all the videos in our galleries if she
>>>>> doesnt squirt you wont find her here we know how many guys have a
>>>>> squirting
>>>>> fetish so we bring it to life for you watch for new squirting updates
>>>>> every
>>>>> week'} latency=0
>>>>> 1386600835.536602000 API mutator_open: API mutator_open:
>>>>> namespace=1142522706table=result flags=0 flush_interval=0
>>>>> async_mutator=68997312 latency=0
>>>>> 1386600835.536150000 API hql_exec: API hql_exec:
>>>>> namespace=3019239709hql=select link FROM links where
>>>>> row='designspongeonline.com_
>>>>> http://designspongeonline.com/category/diy-projects?technique=carpentry&showposts=20&materials=leather_designsponge.com'
>>>>> cell_limit 1 noflush=0 unbuffered=0 latency=0
>>>>> 1386600835.536691000 API namespace_close: API namespace_close:
>>>>> namespace=2860583658 latency=0
>>>>> 1386600835.518349000 API mutator_flush: API mutator_flush:
>>>>> mutator=63400256 done latency=19
>>>>> 1386600835.526528000 API mutator_flush: API mutator_flush:
>>>>> mutator=63396416 done latency=11
>>>>> 1386600835.538701000 API namespace_close: API namespace_close:
>>>>> namespace=3019239709 latency=0
>>>>> 1386600835.539385000 API mutator_flush: API mutator_flush:
>>>>> mutator=63400256 done latency=0
>>>>> 1386600835.539378000 API mutator_close: API mutator_close:
>>>>> mutator=63400256 latency=0
>>>>> 1386600835.540058000 API namespace_close: API namespace_close:
>>>>> namespace=3256284611 latency=0
>>>>> 1386600835.534128000 API mutator_flush: API mutator_flush:
>>>>> mutator=63393536 done latency=6
>>>>> 1386600835.540847000 API mutator_flush: API mutator_flush:
>>>>> mutator=63393536 done latency=0
>>>>> 1386600835.540841000 API mutator_close: API mutator_close:
>>>>> mutator=63393536 latency=0
>>>>> 1386600835.541401000 API namespace_close: API namespace_close:
>>>>> namespace=4251420742 latency=0
>>>>> 1386600835.542443000 API namespace_open: API namespace_open: namespace
>>>>> name=crawler id=1156035969 latency=0
>>>>> 1386600835.542945000 API namespace_open: API namespace_open: namespace
>>>>> name=crawler id=2980656141 latency=0
>>>>> 1386600835.543333000 API mutator_set_cell: API mutator_set_cell:
>>>>> mutator=68997312 cell={Cell: {Key:
>>>>> row='the_month####999997.585027####8d975b47-c5c6-41e3-a1f1-80fbb7d2c7f3'
>>>>> cf='urlguid' flag=255} value='well ı am finally done look ı am even done
>>>>> before the month of april ı spent pretty much all day making this unit
>>>>> normally it only takes me a couple hours not this one not even sure why it
>>>>> took all day'} latency=0
>>>>> 1386600835.544141000 API namespace_open: API namespace_open: namespace
>>>>> name=crawler id=1774589716 latency=0
>>>>> 1386600835.544478000 API namespace_open: API namespace_open: namespace
>>>>> name=crawler id=1052633510 latency=0
>>>>> 1386600835.1386600835542867000.543350000 API hql_exec: API hql_exec:
>>>>> namespace=2980656141 hql=select guid FROM urls where row='
>>>>> http://www.gayqualitysites.com/wellhungmodelboys/index.htm'
>>>>> cell_limit 1 noflush=0 unbuffered=0 latency=1
>>>>> API mutator_flush: API mutator_flush: mutator=43919680 done latency=2
>>>>> 1386600835.545779000 API mutator_flush: API mutator_flush:
>>>>> mutator=43919680 done latency=0
>>>>> 1386600835.545772000 API mutator_close: API mutator_close:
>>>>> mutator=43919680 latency=0
>>>>> 1386600835.546109000 API namespace_close: API namespace_close:
>>>>> namespace=1984545983 latency=0
>>>>> 1386600835.546454000 API namespace_open: API namespace_open: namespace
>>>>> name=crawler id=2689713404 latency=0
>>>>> 1386600835.548529000 API namespace_open: API namespace_open: namespace
>>>>> name=crawler id=3127981135 latency=0
>>>>> 1386600835.550342000 API mutator_open: API mutator_open:
>>>>> namespace=305562593table=links flags=0 flush_interval=0
>>>>> async_mutator=43912000 latency=0
>>>>> 1386600835.550619000 API mutator_open: API mutator_open:
>>>>> namespace=1052633510table=icerik flags=0 flush_interval=0
>>>>> async_mutator=63610368 latency=0
>>>>> 1386600835.550759000 API hql_exec: API hql_exec: namespace=1774589716
>>>>> hql=select link FROM links where row='designspongeonline.com_
>>>>> http://designspongeonline.com/category/diy-projects?technique=carpentry&showposts=20&materials=leather_designsponge.com'
>>>>> cell_limit 1 noflush=0 unbuffered=0 latency=0
>>>>> 1386600835.551684000 API mutator_flush: API mutator_flush:
>>>>> mutator=63396416 done latency=0
>>>>> 1386600835.551677000 API mutator_close: API mutator_close:
>>>>> mutator=63396416 latency=0
>>>>> 1386600835.552156000 API namespace_open: API namespace_open: namespace
>>>>> name=crawler id=1162137437 latency=0
>>>>> 1386600835.552195000 API namespace_close: API namespace_close:
>>>>> namespace=1591027820 latency=0
>>>>> 1386600835.552614000 API namespace_close: API namespace_close:
>>>>> namespace=1774589716 latency=0
>>>>> 1386600835.552643000 API namespace_open: API namespace_open: namespace
>>>>> name=crawler id=3720199858 latency=0
>>>>> 1386600835.526241000 API mutator_flush: API mutator_flush:
>>>>> mutator=69000192 done latency=26
>>>>> 1386600835.553304000 API mutator_open: API mutator_open:
>>>>> namespace=2689713404table=urls flags=0 flush_interval=0
>>>>> async_mutator=63393536 latency=0
>>>>> 1386600835.553693000 API mutator_flush: API mutator_flush:
>>>>> mutator=69000192 done latency=0
>>>>> 1386600835.553687000 API mutator_close: API mutator_close:
>>>>> mutator=69000192 latency=0
>>>>> 1386600835.554217000 API namespace_close: API namespace_close:
>>>>> namespace=3838016151 latency=0
>>>>> 1386600835.554592000 API namespace_open: API namespace_open: namespace
>>>>> name=crawler id=3383002693 latency=0
>>>>> 1386600835.554945000 API namespace_open: API namespace_open: namespace
>>>>> name=crawler id=3841379418 latency=0
>>>>> 1386600835.555626000 API mutator_set_cell: API mutator_set_cell:
>>>>> mutator=63393536 cell={Cell: {Key: row='
>>>>> http://www.gayqualitysites.com/wellhungmodelboys/index.htm' cf='guid'
>>>>> flag=255} value='246ea99e-4b66-4ea4-8116-400791e203ff'} latency=0
>>>>> 1386600835.555285000 API hql_exec: API hql_exec: namespace=3841379418
>>>>> hql=select link FROM links where row='picshag.com_
>>>>> http://picshag.com/envelope-warning.html_orgagsm.com' cell_limit 1
>>>>> noflush=0 unbuffered=0 latency=0
>>>>> 1386600835.556588000 API mutator_open: API mutator_open:
>>>>> namespace=3841379418table=links flags=0 flush_interval=0
>>>>> async_mutator=69000192 latency=0
>>>>> 1386600835.557017000 API mutator_set_cell: API mutator_set_cell:
>>>>> mutator=69000192 cell={Cell: {Key: row='picshag.com_
>>>>> http://picshag.com/envelope-warning.html_orgagsm.com' cf='link'
>>>>> cq='d' flag=255} value='http://orgagsm.com/photos/perfec
>>>>>
>>>>>
>>>>>
>>>>> do any problems have, or not?
>>>>>
>>>>>
>>>>> 2013/12/9 Hatice Çobanoğlu <[email protected]>
>>>>>
>>>>>> Hi Doug,
>>>>>> i added ThriftBroker.API.Logging=true to hypertable.cfg file and cap
>>>>>> push_config. and then restart the ThriftBrokers. ThriftBroker.log
>>>>>> file as following,
>>>>>>
>>>>>> hadoop59 's ThriftBroker log:
>>>>>>
>>>>>> tcmalloc: large alloc 1668055040 bytes == 0x34ce000 @
>>>>>> tcmalloc: large alloc 1668055040 bytes == 0x34ce000 @
>>>>>> 1386592292.534763000 API namespace_open: API namespace_open:
>>>>>> namespace name=crawler id=3625439156 latency=0
>>>>>> 1386592892 ERROR ThriftBroker : handle_error
>>>>>> (/root/src/hypertable/src/cc/Hypertable/Lib/TableScannerAsync.cc:385):
>>>>>> Received error: is_create=1 - Hypertable::Exception: Event: type=ERROR
>>>>>> "HYPERTABLE request timeout" from=172.16.200.153:38060 - HYPERTABLE
>>>>>> request timeout
>>>>>> 1386592892 ERROR ThriftBroker : handle_error
>>>>>> (/root/src/hypertable/src/cc/Hypertable/Lib/TableScannerAsync.cc:402):
>>>>>> Hypertable::Exception: Event: type=ERROR "HYPERTABLE request timeout"
>>>>>> from=
>>>>>> 172.16.200.153:38060 - HYPERTABLE request timeout
>>>>>> 1386592892 ERROR ThriftBroker : hql_exec
>>>>>> (/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:584): virtual
>>>>>> void
>>>>>> Hypertable::ThriftBroker::ServerHandler::hql_exec(Hypertable::ThriftGen::HqlResult&,
>>>>>> Hypertable::ThriftGen::Namespace, const Hypertable::String&, bool, bool)
>>>>>> namespace=3625439156 hql=select guid FROM urls where row='
>>>>>> http://washingtonpost.com/business/capitalbusiness/kastle-systems-has-seen-video-mobile-technology-transform-its-security-business/2013/12/07/62a071c4-5de8-11e3-95c2-13623eb2b0e1_story.html?tid=hpModule_1728cf4a-8a79-11e2-98d9-3012c1cd8d1e'
>>>>>> cell_limit 1 noflush=0 unbuffered=0: Hypertable::Exception: Event:
>>>>>> type=ERROR "HYPERTABLE request timeout" from=172.16.200.153:38060 -
>>>>>> HYPERTABLE request timeout
>>>>>> at bool Hypertable::TableScanner::next(Hypertable::Cell&)
>>>>>> (/root/src/hypertable/src/cc/Hypertable/Lib/TableScanner.cc:82)
>>>>>> 1386592892.639819000 API namespace_close: API namespace_close:
>>>>>> namespace=3625439156 latency=0
>>>>>> 1386592892.640398000 API namespace_open: API namespace_open:
>>>>>> namespace name=crawler id=2342619381 latency=0
>>>>>> 1386592902.650809000 API namespace_open: API namespace_open:
>>>>>> namespace name=crawler id=970823060 latency=0
>>>>>>
>>>>>>
>>>>>> hadoop152 ThriftBroker log:
>>>>>>
>>>>>> 1386592891.786474000 API namespace_close: API namespace_close:
>>>>>> namespace=2282353234 latency=0
>>>>>> 1386592891 ERROR ThriftBroker : TSocket::peek() recv() <Host:
>>>>>> 172.16.200.30 Port: 56469>Connection reset by peer
>>>>>> 1386592891 ERROR ThriftBroker : TThreadedServer client died: recv():
>>>>>> Connection reset by peer
>>>>>> 1386592901 ERROR ThriftBroker : flush
>>>>>> (/root/src/hypertable/src/cc/Hypertable/Lib/TableMutator.cc:183):
>>>>>> Hypertable::Exception: - DFS BROKER i/o error
>>>>>> at void
>>>>>> Hypertable::TableMutator::wait_for_flush_completion(Hypertable::TableMutatorAsync*)
>>>>>> (/root/src/hypertable/src/cc/Hypertable/Lib/TableMutator.cc:210)
>>>>>> Failed: (2013-12-09#ali_ata_bak,parametre,-9223372036854775806) - DFS
>>>>>> BROKER i/o error
>>>>>> 1386592901 ERROR ThriftBroker : hql_exec
>>>>>> (/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:584): virtual
>>>>>> void
>>>>>> Hypertable::ThriftBroker::ServerHandler::hql_exec(Hypertable::ThriftGen::HqlResult&,
>>>>>> Hypertable::ThriftGen::Namespace, const Hypertable::String&, bool, bool)
>>>>>> namespace=1660590572 hql=insert into sorgukayit values
>>>>>> ('2013-12-09#ali_ata_bak','parametre','172.16.59.67') noflush=0
>>>>>> unbuffered=0: Hypertable::Exception: - DFS BROKER i/o error
>>>>>> at void
>>>>>> Hypertable::TableMutator::wait_for_flush_completion(Hypertable::TableMutatorAsync*)
>>>>>> (/root/src/hypertable/src/cc/Hypertable/Lib/TableMutator.cc:210)
>>>>>> 1386593491 WARN ThriftBroker :
>>>>>> (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:596) Received
>>>>>> response for non-pending event (id=19,version=1,total_len=91)
>>>>>>
>>>>>>
>>>>>>
>>>>>> and now we can not insert or select on ht shell.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2013/12/6 Doug Judd <[email protected]>
>>>>>>
>>>>>>> Hi Hatice,
>>>>>>>
>>>>>>> Add the following property to your hypertable.cfg file:
>>>>>>>
>>>>>>> ThriftBroker.API.Logging=true
>>>>>>>
>>>>>>> and then restart the ThriftBrokers and wait for the problem to
>>>>>>> happen again. Once it does, send me the ThriftBroker.log file of the
>>>>>>> ThriftBroker that crashed. It will include more information that will
>>>>>>> help
>>>>>>> me diagnose the problem. Thanks!
>>>>>>>
>>>>>>> - Doug
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Dec 6, 2013 at 6:09 AM, Hatice Çobanoğlu <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> Hi everyone,
>>>>>>>> i have 7 thriftboker in cluster. and i've problem oen of them as
>>>>>>>> following.
>>>>>>>>
>>>>>>>> 1386334385 ERROR ThriftBroker : get_namespace
>>>>>>>> (/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:2177): Bad
>>>>>>>> namespace id - 0
>>>>>>>> 1386334385 ERROR ThriftBroker : get_namespace
>>>>>>>> (/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:2177): Bad
>>>>>>>> namespace id - 0
>>>>>>>> 1386334385 ERROR ThriftBroker : get_namespace
>>>>>>>> (/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:2177): Bad
>>>>>>>> namespace id - 0
>>>>>>>> 1386334385 ERROR ThriftBroker : get_namespace
>>>>>>>> (/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:2177): Bad
>>>>>>>> namespace id - 0
>>>>>>>> 1386334385 ERROR ThriftBroker : get_namespace
>>>>>>>> (/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:2177): Bad
>>>>>>>> namespace id - 0
>>>>>>>> 1386334385 ERROR ThriftBroker : get_namespace
>>>>>>>> (/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:2177): Bad
>>>>>>>> namespace id - 0
>>>>>>>> 1386334385 ERROR ThriftBroker : get_namespace
>>>>>>>> (/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:2177): Bad
>>>>>>>> namespace id - 0
>>>>>>>> 1386335515 ERROR ThriftBroker : TThreadedServer exception:
>>>>>>>> St9bad_alloc: std::bad_alloc
>>>>>>>> 1386335532 ERROR ThriftBroker : TThreadedServer exception:
>>>>>>>> St9bad_alloc: std::bad_alloc
>>>>>>>> 1386335593 ERROR ThriftBroker : TThreadedServer exception:
>>>>>>>> St9bad_alloc: std::bad_alloc
>>>>>>>> 1386335653 ERROR ThriftBroker : TThreadedServer exception:
>>>>>>>> St9bad_alloc: std::bad_alloc
>>>>>>>> 1386336456 INFO ThriftBroker :
>>>>>>>> (/root/src/hypertable/src/cc/Common/Config.cc:591) Initializing
>>>>>>>> ThriftBroker (Hypertable 0.9.7.13 (v0.9$
>>>>>>>> CPU cores count=24
>>>>>>>>
>>>>>>>> when have these problems, restart thriftbroker which closed and
>>>>>>>> system continue. why thrift give this error log?
>>>>>>>> not: connect to thrift with 32 client.
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "Hypertable Development" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to [email protected].
>>>>>>>> To post to this group, send email to
>>>>>>>> [email protected].
>>>>>>>> Visit this group at http://groups.google.com/group/hypertable-dev.
>>>>>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Doug Judd
>>>>>>> CEO, Hypertable Inc.
>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "Hypertable Development" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to [email protected].
>>>>>>> To post to this group, send email to [email protected]
>>>>>>> .
>>>>>>> Visit this group at http://groups.google.com/group/hypertable-dev.
>>>>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>>>>
>>>>>>
>>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Hypertable Development" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to [email protected].
>>>>> To post to this group, send email to [email protected].
>>>>> Visit this group at http://groups.google.com/group/hypertable-dev.
>>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Doug Judd
>>>> CEO, Hypertable Inc.
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Hypertable Development" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To post to this group, send email to [email protected].
>>>> Visit this group at http://groups.google.com/group/hypertable-dev.
>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Hypertable Development" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> To post to this group, send email to [email protected].
>>> Visit this group at http://groups.google.com/group/hypertable-dev.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>
>>
>>
>> --
>> Doug Judd
>> CEO, Hypertable Inc.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Hypertable Development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To post to this group, send email to [email protected].
>> Visit this group at http://groups.google.com/group/hypertable-dev.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Hypertable Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at http://groups.google.com/group/hypertable-dev.
> For more options, visit https://groups.google.com/groups/opt_out.
>
--
Doug Judd
CEO, Hypertable Inc.
--
You received this message because you are subscribed to the Google Groups
"Hypertable Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/hypertable-dev.
For more options, visit https://groups.google.com/groups/opt_out.
