Hi Kenny, So that looks pretty good. It looks like it solved your problem, correct? If so, we'll probably change the block cache default to disabled.
- Doug On Tue, Aug 21, 2012 at 12:44 AM, Kenny F. <[email protected]> wrote: > Hi Doug, > > Here some results: > RangeServer works for about 24 hours without crash. > > "Block Cache Fill", "Block Cache Hit Rate", "Block Cache Max Memory" are > zero. > > "Virtual Memory" slowly rose up, but it still under the "cap" level, so my > be it fall down but a bit later. > At the picture we see restart ~10AM. > > Or you already found the reason of crash.) > > Regards, > Kenny F. > > > > On Friday, August 17, 2012 7:58:05 PM UTC+3, Doug Judd wrote: > >> Hi Kenny, >> >> I think I may know what's going on here. Can you try your test again >> with the following changes: >> >> 1. Revert the Hypertable.RangeServer.**MemoryLimit changes. In other >> words, leave the RangeServer memory properties at their defaults (60% of >> physical RAM). >> 2. Add the following property to your hypertable.cfg to disable the block >> cache: >> >> Hypertable.RangeServer.**BlockCache.MaxMemory=0 >> >> I suspect that the RangeServer is fighting with the OS file cache for >> RAM. Disabling the block cache is not as bad as you might think. The OS >> will cache the compressed blocks in its file cache. >> >> - Doug >> >> On Fri, Aug 17, 2012 at 7:24 AM, Kenny F. <[email protected]> wrote: >> >>> )) >>> I hope this issue help you to locate the problem. >>> Hypertable's RangeServers use ALL memory(not all FREE memory) when have >>> no memory limit (or high memory limit). >>> It is easy to watch "Block Cache Max Memory" at RangeServer Statistics >>> (monitoring). >>> This issue appeared between Hypertable 0.9.5.0 and Hypertable 0.9.5.4. >>> I hope this bug will be fix. >>> >>> >>> On Thursday, August 16, 2012 4:19:53 PM UTC+3, Christoph Rupp wrote: >>> >>>> Well, by default the RangeServer uses 60%, but that memory is used for >>>> caching CellStores. >>>> >>>> If there's additional memory required to fetch data for a single >>>> request, then it can exceed the limit for a short while. It seems that this >>>> might have happened here, If you have very large scans with lots of data >>>> (and in addition some of the memory was allocated by apache). >>>> >>>> But i am glad that we worked it out :) >>>> >>>> bye >>>> Christoph >>>> >>>> 2012/8/16 Kenny F. <[email protected]> >>>> >>>> Thanks! >>>>> >>>>> I've already done it, it helped and post graphs (today) for you to >>>>> interpret them. >>>>> Earlier I used Hypertable.RangeServer.**MemoryL****imit.Percentage=20 >>>>> or 30 or 40 and it didn't help me at all. >>>>> Now I use Hypertable.RangeServer.**MemoryL****imit=1900Mb (see my >>>>> post today with graphs) >>>>> >>>>> I think, the reason is when we have no limit or high limit, >>>>> RangeServer use ALL memory (see graph "Block Cache Max Memory"), even if >>>>> it >>>>> is fill at 95% or more. >>>>> >>>>> >>>>> >>>>> >>>>> On Thursday, August 16, 2012 4:04:31 PM UTC+3, Christoph Rupp wrote: >>>>> >>>>>> Hi Kenny, >>>>>> >>>>>> can you try reducing the memory usage of the RangeServer? By default >>>>>> the RangeServer uses 60% of the memory. >>>>>> >>>>>> There are two properties: >>>>>> >>>>>> Hypertable.RangeServer.**MemoryL****imit or Hypertable.RangeServer.** >>>>>> MemoryL****imit.Percentage. >>>>>> >>>>>> You can use either of them. >>>>>> >>>>>> Bye >>>>>> Christoph >>>>>> >>>>>> 2012/8/16 Kenny F. <[email protected]> >>>>>> >>>>>> >Is it possible that there is some other process (or set of >>>>>>> processes) on the machine at the time of the crash that is using up all >>>>>>> virtual memory? >>>>>>> I don't think so. >>>>>>> First, I have 2 memory greed processes only: Hypertable and Apache. >>>>>>> Second, I didn't change the environment. And restarting the system >>>>>>> didn't solve the issue. >>>>>>> Third, RangeServer didn't live more then several hours. >>>>>>> And I don't remember, when RangeServer crashes immediately. >>>>>>> >>>>>>> >>>>>>> On Thursday, August 16, 2012 3:02:39 PM UTC+3, Doug Judd wrote: >>>>>>> >>>>>>>> Hi Kenny, >>>>>>>> >>>>>>>> One other thought. Is it possible that there is some other process >>>>>>>> (or set of processes) on the machine at the time of the crash that is >>>>>>>> using >>>>>>>> up all virtual memory? Try running 'free' at regular intervals during >>>>>>>> your >>>>>>>> test to see if the system is running out of swap space. >>>>>>>> >>>>>>>> - Doug >>>>>>>> >>>>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Hypertable Development" group. >>>>> To view this discussion on the web visit https://groups.google.com/d/* >>>>> *ms**g/hypertable-dev/-/**CWqXo6VjsGQ**J<https://groups.google.com/d/msg/hypertable-dev/-/CWqXo6VjsGQJ> >>>>> . >>>>> To post to this group, send email to hyperta...@googlegroups.**com. >>>>> To unsubscribe from this group, send email to hypertable-de...@** >>>>> googlegroups.**com. >>>>> >>>>> For more options, visit this group at http://groups.google.com/**group >>>>> **/hypertable-dev?hl=en<http://groups.google.com/group/hypertable-dev?hl=en> >>>>> . >>>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Hypertable Development" group. >>> To view this discussion on the web visit https://groups.google.com/d/** >>> msg/hypertable-dev/-/**WDrfBDUZHZMJ<https://groups.google.com/d/msg/hypertable-dev/-/WDrfBDUZHZMJ> >>> . >>> >>> To post to this group, send email to hyperta...@googlegroups.**com. >>> To unsubscribe from this group, send email to hypertable-de...@** >>> googlegroups.com. >>> For more options, visit this group at http://groups.google.com/** >>> group/hypertable-dev?hl=en<http://groups.google.com/group/hypertable-dev?hl=en> >>> . >>> >> >> >> >> -- >> Doug Judd >> CEO, Hypertable Inc. >> >> -- > You received this message because you are subscribed to the Google Groups > "Hypertable Development" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/hypertable-dev/-/5aqMdL8wbTIJ. > > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/hypertable-dev?hl=en. > -- Doug Judd CEO, Hypertable Inc. -- You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.
