Re: Memcache 1.4.24 + stunnel crashes without any notifications

[dormando]

That's unknown territory unfortunately. You'll have to debug sasl a bit on
your own :( getting a backtrace in the way I said could be helpful still.

On Wed, 30 Dec 2015, Prasad Prabhu wrote:

> I looked in /var/log/messages and i see that there are segfaults at the exact 
> time we saw the crashes:
> Dec 30 14:41:08 ip-10-82-116-125 kernel: memcached[10838]: segfault at 10 ip 
> 7f23e26f689e sp 7f23defb5b60 error 4 in 
> libsasl2.so.2.0.23.#prelink#.LawU2D (deleted)[7f23e26e7000+19000]
>
>
> Dec 30 16:01:02 ip-10-82-116-125 kernel: memcached[14329]: segfault at 10 ip 
> 0037b4a0f89e sp 7fa7e6fb8b60 error 4 in 
> libsasl2.so.2.0.23[37b4a0+19000]
> Looks like its memcache + sasl issue.
>
> Regards,
> Prasad
>
> On Wed, Dec 30, 2015 at 4:43 PM, dormando  wrote:
>   Hi,
>
>   Do you have a segfault listed in 'dmesg'?
>
>   You can also try: 1.4.25, which had some bugfixes.
>   Or: the memcached-debug binary from a .24 or .25 compile, which has
>   assert()'s in to give better crashes
>   Or: Attach your running process to gdb, wait for it to crash, then get 
> the
>   backtrace.
>
>   If you're doing the latter two options I'd still recommend using .25 if 
> at
>   all possible.
>
>   On Wed, 30 Dec 2015, Prasad Prabhu wrote:
>
>   > We're running memcache v 1.4.24 with stunnel and we're seeing the 
> process crash without any logs or notifications. We've performance tested it 
> with different profiles and are not able to get a consistent cause of 
> failure.It sometimes crashes during a perf test and at other times, dies on 
> the second or third test after passing the first test
>   with no
>   > problems. One odd thing we saw from the memcache stats is that we saw 
> current connections count staying at a high number (about 6000) even after we 
> shut down the services using memcache. 
>   > We dont see any specific pattern in the memory utilization, number of 
> objects or any other details in the stats. Is there any known issue or a 
> problem we are?
>   > We've run memcache with the -v option and didnt see any additional 
> data.
>   >
>   > Memcache configuration:
>   > PORT="1234"
>   > USER="memcached"
>   > MAXCONN="2"
>   > CACHESIZE="1"
>   > OPTIONS="-S -v >> /var/log/memcached 2>&1"
>   >
>   > Anyone have any idea why this might be happening?
>   >
>   > Regards,
>   > Prasad
>   >
>   > --
>   >
>   > ---
>   > You received this message because you are subscribed to the Google 
> Groups "memcached" group.
>   > To unsubscribe from this group and stop receiving emails from it, 
> send an email to memcached+unsubscr...@googlegroups.com.
>   > For more options, visit https://groups.google.com/d/optout.
>   >
>   >
>
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups 
> "memcached" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to memcached+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>

Re: Memcache 1.4.24 + stunnel crashes without any notifications

[Prasad Prabhu]

I looked in /var/log/messages and i see that there are segfaults at the
exact time we saw the crashes:
Dec 30 14:41:08 ip-10-82-116-125 kernel: memcached[10838]: segfault at 10
ip 7f23e26f689e sp 7f23defb5b60 error 4 in
libsasl2.so.2.0.23.#prelink#.LawU2D (deleted)[7f23e26e7000+19000]


Dec 30 16:01:02 ip-10-82-116-125 kernel: memcached[14329]: segfault at 10
ip 0037b4a0f89e sp 7fa7e6fb8b60 error 4 in
libsasl2.so.2.0.23[37b4a0+19000]
Looks like its memcache + sasl issue.

Regards,
Prasad

On Wed, Dec 30, 2015 at 4:43 PM, dormando  wrote:

> Hi,
>
> Do you have a segfault listed in 'dmesg'?
>
> You can also try: 1.4.25, which had some bugfixes.
> Or: the memcached-debug binary from a .24 or .25 compile, which has
> assert()'s in to give better crashes
> Or: Attach your running process to gdb, wait for it to crash, then get the
> backtrace.
>
> If you're doing the latter two options I'd still recommend using .25 if at
> all possible.
>
> On Wed, 30 Dec 2015, Prasad Prabhu wrote:
>
> > We're running memcache v 1.4.24 with stunnel and we're seeing the
> process crash without any logs or notifications. We've performance tested
> it with different profiles and are not able to get a consistent cause of
> failure.It sometimes crashes during a perf test and at other times, dies on
> the second or third test after passing the first test with no
> > problems. One odd thing we saw from the memcache stats is that we saw
> current connections count staying at a high number (about 6000) even after
> we shut down the services using memcache.
> > We dont see any specific pattern in the memory utilization, number of
> objects or any other details in the stats. Is there any known issue or a
> problem we are?
> > We've run memcache with the -v option and didnt see any additional data.
> >
> > Memcache configuration:
> > PORT="1234"
> > USER="memcached"
> > MAXCONN="2"
> > CACHESIZE="1"
> > OPTIONS="-S -v >> /var/log/memcached 2>&1"
> >
> > Anyone have any idea why this might be happening?
> >
> > Regards,
> > Prasad
> >
> > --
> >
> > ---
> > You received this message because you are subscribed to the Google
> Groups "memcached" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to memcached+unsubscr...@googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
> >
> >
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"memcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to memcached+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.