Hi Doug!, Hypertable works fine )
On Tuesday, August 21, 2012 7:25:51 PM UTC+3, Doug Judd wrote: > > Hi Kenny, > > So that looks pretty good. It looks like it solved your problem, correct? > If so, we'll probably change the block cache default to disabled. > > - Doug > > On Tue, Aug 21, 2012 at 12:44 AM, Kenny F. <[email protected] <javascript:> > > wrote: > >> Hi Doug, >> >> Here some results: >> RangeServer works for about 24 hours without crash. >> >> "Block Cache Fill", "Block Cache Hit Rate", "Block Cache Max Memory" are >> zero. >> >> "Virtual Memory" slowly rose up, but it still under the "cap" level, so >> my be it fall down but a bit later. >> At the picture we see restart ~10AM. >> >> Or you already found the reason of crash.) >> >> Regards, >> Kenny F. >> >> >> >> On Friday, August 17, 2012 7:58:05 PM UTC+3, Doug Judd wrote: >> >>> Hi Kenny, >>> >>> I think I may know what's going on here. Can you try your test again >>> with the following changes: >>> >>> 1. Revert the Hypertable.RangeServer.**MemoryLimit changes. In other >>> words, leave the RangeServer memory properties at their defaults (60% of >>> physical RAM). >>> 2. Add the following property to your hypertable.cfg to disable the >>> block cache: >>> >>> Hypertable.RangeServer.**BlockCache.MaxMemory=0 >>> >>> I suspect that the RangeServer is fighting with the OS file cache for >>> RAM. Disabling the block cache is not as bad as you might think. The OS >>> will cache the compressed blocks in its file cache. >>> >>> - Doug >>> >>> On Fri, Aug 17, 2012 at 7:24 AM, Kenny F. <[email protected]> wrote: >>> >>>> )) >>>> I hope this issue help you to locate the problem. >>>> Hypertable's RangeServers use ALL memory(not all FREE memory) when have >>>> no memory limit (or high memory limit). >>>> It is easy to watch "Block Cache Max Memory" at RangeServer Statistics >>>> (monitoring). >>>> This issue appeared between Hypertable 0.9.5.0 and Hypertable 0.9.5.4. >>>> I hope this bug will be fix. >>>> >>>> >>>> On Thursday, August 16, 2012 4:19:53 PM UTC+3, Christoph Rupp wrote: >>>> >>>>> Well, by default the RangeServer uses 60%, but that memory is used >>>>> for caching CellStores. >>>>> >>>>> If there's additional memory required to fetch data for a single >>>>> request, then it can exceed the limit for a short while. It seems that >>>>> this >>>>> might have happened here, If you have very large scans with lots of data >>>>> (and in addition some of the memory was allocated by apache). >>>>> >>>>> But i am glad that we worked it out :) >>>>> >>>>> bye >>>>> Christoph >>>>> >>>>> 2012/8/16 Kenny F. <[email protected]> >>>>> >>>>> Thanks! >>>>>> >>>>>> I've already done it, it helped and post graphs (today) for you to >>>>>> interpret them. >>>>>> Earlier I used Hypertable.RangeServer.**MemoryL****imit.Percentage=20 >>>>>> or 30 or 40 and it didn't help me at all. >>>>>> Now I use Hypertable.RangeServer.**MemoryL****imit=1900Mb (see my >>>>>> post today with graphs) >>>>>> >>>>>> I think, the reason is when we have no limit or high limit, >>>>>> RangeServer use ALL memory (see graph "Block Cache Max Memory"), even if >>>>>> it >>>>>> is fill at 95% or more. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thursday, August 16, 2012 4:04:31 PM UTC+3, Christoph Rupp wrote: >>>>>> >>>>>>> Hi Kenny, >>>>>>> >>>>>>> can you try reducing the memory usage of the RangeServer? By default >>>>>>> the RangeServer uses 60% of the memory. >>>>>>> >>>>>>> There are two properties: >>>>>>> >>>>>>> Hypertable.RangeServer.**MemoryL****imit or Hypertable.RangeServer.* >>>>>>> *MemoryL****imit.Percentage. >>>>>>> >>>>>>> You can use either of them. >>>>>>> >>>>>>> Bye >>>>>>> Christoph >>>>>>> >>>>>>> 2012/8/16 Kenny F. <[email protected]> >>>>>>> >>>>>>> >Is it possible that there is some other process (or set of >>>>>>>> processes) on the machine at the time of the crash that is using up >>>>>>>> all >>>>>>>> virtual memory? >>>>>>>> I don't think so. >>>>>>>> First, I have 2 memory greed processes only: Hypertable and Apache. >>>>>>>> Second, I didn't change the environment. And restarting the system >>>>>>>> didn't solve the issue. >>>>>>>> Third, RangeServer didn't live more then several hours. >>>>>>>> And I don't remember, when RangeServer crashes immediately. >>>>>>>> >>>>>>>> >>>>>>>> On Thursday, August 16, 2012 3:02:39 PM UTC+3, Doug Judd wrote: >>>>>>>> >>>>>>>>> Hi Kenny, >>>>>>>>> >>>>>>>>> One other thought. Is it possible that there is some other >>>>>>>>> process (or set of processes) on the machine at the time of the crash >>>>>>>>> that >>>>>>>>> is using up all virtual memory? Try running 'free' at regular >>>>>>>>> intervals >>>>>>>>> during your test to see if the system is running out of swap space. >>>>>>>>> >>>>>>>>> - Doug >>>>>>>>> >>>>>>>>> >>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Hypertable Development" group. >>>>>> To view this discussion on the web visit https://groups.google.com/d/ >>>>>> **ms**g/hypertable-dev/-/**CWqXo6VjsGQ**J<https://groups.google.com/d/msg/hypertable-dev/-/CWqXo6VjsGQJ> >>>>>> . >>>>>> To post to this group, send email to hyperta...@googlegroups.**com. >>>>>> To unsubscribe from this group, send email to hypertable-de...@** >>>>>> googlegroups.**com. >>>>>> >>>>>> For more options, visit this group at http://groups.google.com/** >>>>>> group**/hypertable-dev?hl=en<http://groups.google.com/group/hypertable-dev?hl=en> >>>>>> . >>>>>> >>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Hypertable Development" group. >>>> To view this discussion on the web visit https://groups.google.com/d/** >>>> msg/hypertable-dev/-/**WDrfBDUZHZMJ<https://groups.google.com/d/msg/hypertable-dev/-/WDrfBDUZHZMJ> >>>> . >>>> >>>> To post to this group, send email to hyperta...@googlegroups.**com. >>>> To unsubscribe from this group, send email to hypertable-de...@** >>>> googlegroups.com. >>>> For more options, visit this group at http://groups.google.com/** >>>> group/hypertable-dev?hl=en<http://groups.google.com/group/hypertable-dev?hl=en> >>>> . >>>> >>> >>> >>> >>> -- >>> Doug Judd >>> CEO, Hypertable Inc. >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Hypertable Development" group. >> To view this discussion on the web visit >> https://groups.google.com/d/msg/hypertable-dev/-/5aqMdL8wbTIJ. >> >> To post to this group, send email to [email protected]<javascript:> >> . >> To unsubscribe from this group, send email to >> [email protected] <javascript:>. >> For more options, visit this group at >> http://groups.google.com/group/hypertable-dev?hl=en. >> > > > > -- > Doug Judd > CEO, Hypertable Inc. > > -- You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/I4jl00vOqoYJ. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.
