hi GuoWei, The current offset of the adaptor is one way to track how much data are sent. If your data is not stored in local disk of the agent, then data can be lost when sending buffer is full. Collector buffers data in memory, it does not store in collector disks by default. There is a local writer which buffers data in disk and upload to hdfs. This is not turn on by default. Hence, you can utilize local writer or implement your own writer to achieve the same result. See: http://incubator.apache.org/chukwa/docs/r0.5.0/collector.html for more information about local writer.
We are planning to merge collector and agent together to provide ability to spill to local disk, and reduce deployment complexity. This is in: https://issues.apache.org/jira/browse/CHUKWA-674 If you are interested. regards, Eric On Mon, Dec 24, 2012 at 9:52 PM, GuoWei <[email protected]> wrote: > Dear all, > > Recently, I write a my custom adaptor and map processor to store data to > Hbase. But when I shut down the Hbase, I found many many error logs in > Collector.log. My questions are : > > 1. When my custom adaptor can not connect to collector, do I need to store > data temporary myself ? Or chukwa can do this for me ? > 2. When my custom demux map processor can not connect to Hbase, do I > need to store data temporary my self at collector side ? Or chukwa can do > this for me ? > > Thanks in advance. > > *Best Regards / **商祺** > *郭伟 Guo Wei > ----------------------------------------------------- > 南京西桥科技有限公司 > Western Bridge Tech Ltd., Nanjing > > 南京市玄武区花园路8号一号楼511 > No. 511, Building 1, No. 8, Hua Yuan Road > > Xuanwu District, Nanjing, PR China > > Email: [email protected] > > Tel: +86 25 8528 4900 (Operator) > Mobile: +86 138 1589 8257 > > Fax: +86 25 8528 4980 > > Web: http://www.wbkit.com > > ----------------------------------------------------- ******* > **WesternBridge Tech: Professional software service provider. > Professional is MANNER as well CAPABILITY.* > > * > * >
