I may have misread. When you said the server was sitting at 100% CPU, what exactly was using all of the CPU? memcached? perl?
On Wed, 28 May 2014, Alex Gemmell wrote: > Yep, it's 1.4.20. I followed the instructions here > http://memcached.org/downloads and ran "wget http://memcached.org/latest";. > Just to be sure, this morning I ran "wget > http://memcached.org/files/memcached-1.4.20.tar.gz"; and tried to compile it > and got exactly the same > problem. > > I followed your instructions and here's the output (I hope I did this right?) > > ====================== > (gdb) thread apply all bt > > Thread 7 (Thread 0x7fffe7fff700 (LWP 8785)): > #0 0x00000036dfc0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x0000000000416bc9 in item_crawler_thread (arg=<value optimized out>) at > items.c:772 > #2 0x00000036dfc079d1 in start_thread () from /lib64/libpthread.so.0 > #3 0x00000036df8e8b7d in clone () from /lib64/libc.so.6 > > Thread 6 (Thread 0x7ffff52dd700 (LWP 8773)): > #0 0x00000036dfc0b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x000000000041860d in assoc_maintenance_thread (arg=<value optimized > out>) at assoc.c:251 > #2 0x00000036dfc079d1 in start_thread () from /lib64/libpthread.so.0 > #3 0x00000036df8e8b7d in clone () from /lib64/libc.so.6 > > Thread 5 (Thread 0x7ffff5cde700 (LWP 8772)): > #0 0x00000036df8e9173 in epoll_wait () from /lib64/libc.so.6 > #1 0x00007ffff7bb47e6 in ?? () from /usr/lib64/libevent-2.0.so.5 > #2 0x00007ffff7ba2c46 in event_base_loop () from /usr/lib64/libevent-2.0.so.5 > #3 0x00000000004197b5 in worker_libevent (arg=0x645f30) at thread.c:386 > #4 0x00000036dfc079d1 in start_thread () from /lib64/libpthread.so.0 > #5 0x00000036df8e8b7d in clone () from /lib64/libc.so.6 > > Thread 4 (Thread 0x7ffff66df700 (LWP 8771)): > #0 0x00000036df8e9173 in epoll_wait () from /lib64/libc.so.6 > #1 0x00007ffff7bb47e6 in ?? () from /usr/lib64/libevent-2.0.so.5 > #2 0x00007ffff7ba2c46 in event_base_loop () from /usr/lib64/libevent-2.0.so.5 > #3 0x00000000004197b5 in worker_libevent (arg=0x642ba0) at thread.c:386 > ---Type <return> to continue, or q <return> to quit--- > #4 0x00000036dfc079d1 in start_thread () from /lib64/libpthread.so.0 > #5 0x00000036df8e8b7d in clone () from /lib64/libc.so.6 > > Thread 3 (Thread 0x7ffff70e0700 (LWP 8770)): > #0 0x00000036df8e9173 in epoll_wait () from /lib64/libc.so.6 > #1 0x00007ffff7bb47e6 in ?? () from /usr/lib64/libevent-2.0.so.5 > #2 0x00007ffff7ba2c46 in event_base_loop () from /usr/lib64/libevent-2.0.so.5 > #3 0x00000000004197b5 in worker_libevent (arg=0x63f810) at thread.c:386 > #4 0x00000036dfc079d1 in start_thread () from /lib64/libpthread.so.0 > #5 0x00000036df8e8b7d in clone () from /lib64/libc.so.6 > > Thread 2 (Thread 0x7ffff7ae1700 (LWP 8769)): > #0 0x00000036df8e9173 in epoll_wait () from /lib64/libc.so.6 > #1 0x00007ffff7bb47e6 in ?? () from /usr/lib64/libevent-2.0.so.5 > #2 0x00007ffff7ba2c46 in event_base_loop () from /usr/lib64/libevent-2.0.so.5 > #3 0x00000000004197b5 in worker_libevent (arg=0x63c480) at thread.c:386 > #4 0x00000036dfc079d1 in start_thread () from /lib64/libpthread.so.0 > #5 0x00000036df8e8b7d in clone () from /lib64/libc.so.6 > > Thread 1 (Thread 0x7ffff7b8d700 (LWP 8766)): > #0 0x00000036df8e9173 in epoll_wait () from /lib64/libc.so.6 > #1 0x00007ffff7bb47e6 in ?? () from /usr/lib64/libevent-2.0.so.5 > #2 0x00007ffff7ba2c46 in event_base_loop () from /usr/lib64/libevent-2.0.so.5 > #3 0x0000000000408a25 in main (argc=<value optimized out>, argv=<value > optimized out>) at memcached.c:5628 > ====================== > > > On Tuesday, 27 May 2014 19:09:56 UTC-7, Dormando wrote: > You're completely sure that's the 1.4.20 source tree? > > That bug was pretty well fixed... > > If you are definitely testing a 1.4.20 binary, here's the way to grab a > trace: > > start memcached-debug under gdb: > > gdb ./memcached-debug > > handle SIGPIPE nostop noprint pass > > r > > T_MEMD_USE_DAEMON="127.0.0.1:11211" prove -v t/lru-crawler.t > > ... wait until it's been spinning cpu for a few seconds. Then ^C the GDB > window and run "thread apply all bt" > .. and send me that info. > > On Tue, 27 May 2014, Alex Gemmell wrote: > > > Hello Dormando, > > I am having exactly the same issue but with Memcached 1.4.20. > > > > My server specs are: RHEL 6 (Linux 2.6.32-358.23.2.el6.x86_64), > 1880MB RAM, single core :( > > > > Here are the results of me running "prove -v t/lru-crawler.t". It > took exactly 10m 15s to run before it timed out. I watched htop > while it was > > running and the single CPU sat at 100% (which is to be expected I > guess) but the total server memory barely changed and never rose > above 330MB. > > > > ============= > > > prove -v t/lru-crawler.t > > t/lru-crawler.t .. > > 1..189 > > ok 1 > > ok 2 - stored key > > ok 3 - stored key > > ok 4 - stored key > > ok 5 - stored key > > ok 6 - stored key > > ok 7 - stored key > > ok 8 - stored key > > ok 9 - stored key > > ok 10 - stored key > > ok 11 - stored key > > ok 12 - stored key > > ok 13 - stored key > > ok 14 - stored key > > ok 15 - stored key > > ok 16 - stored key > > ok 17 - stored key > > ok 18 - stored key > > ok 19 - stored key > > ok 20 - stored key > > ok 21 - stored key > > ok 22 - stored key > > ok 23 - stored key > > ok 24 - stored key > > ok 25 - stored key > > ok 26 - stored key > > ok 27 - stored key > > ok 28 - stored key > > ok 29 - stored key > > ok 30 - stored key > > ok 31 - stored key > > ok 32 - stored key > > ok 33 - stored key > > ok 34 - stored key > > ok 35 - stored key > > ok 36 - stored key > > ok 37 - stored key > > ok 38 - stored key > > ok 39 - stored key > > ok 40 - stored key > > ok 41 - stored key > > ok 42 - stored key > > ok 43 - stored key > > ok 44 - stored key > > ok 45 - stored key > > ok 46 - stored key > > ok 47 - stored key > > ok 48 - stored key > > ok 49 - stored key > > ok 50 - stored key > > ok 51 - stored key > > ok 52 - stored key > > ok 53 - stored key > > ok 54 - stored key > > ok 55 - stored key > > ok 56 - stored key > > ok 57 - stored key > > ok 58 - stored key > > ok 59 - stored key > > ok 60 - stored key > > ok 61 - stored key > > ok 62 - stored key > > ok 63 - stored key > > ok 64 - stored key > > ok 65 - stored key > > ok 66 - stored key > > ok 67 - stored key > > ok 68 - stored key > > ok 69 - stored key > > ok 70 - stored key > > ok 71 - stored key > > ok 72 - stored key > > ok 73 - stored key > > ok 74 - stored key > > ok 75 - stored key > > ok 76 - stored key > > ok 77 - stored key > > ok 78 - stored key > > ok 79 - stored key > > ok 80 - stored key > > ok 81 - stored key > > ok 82 - stored key > > ok 83 - stored key > > ok 84 - stored key > > ok 85 - stored key > > ok 86 - stored key > > ok 87 - stored key > > ok 88 - stored key > > ok 89 - stored key > > ok 90 - stored key > > ok 91 - stored key > > ok 92 - slab1 has 90 used chunks > > ok 93 - enabled lru crawler > > ok 94 > > ok 95 - kicked lru crawler > > Timeout.. killing the process > > Failed 94/189 subtests > > > > Test Summary Report > > ------------------- > > t/lru-crawler.t (Wstat: 13 Tests: 95 Failed: 0) > > Non-zero wait status: 13 > > Parse errors: Bad plan. You planned 189 tests but ran 95. > > Files=1, Tests=95, 600 wallclock secs ( 0.09 usr 0.01 sys + 352.24 > cusr 61.28 csys = 413.62 CPU) > > Result: FAIL > > ============= > > > > Any ideas? > > > > > > On Thursday, 1 May 2014 18:28:57 UTC-7, Dormando wrote: > > What's the output of: > > > > $ prove -v t/lru-crawler.t > > > > How long are the tests taking to run? This has definitely been > tested on > > ubuntu 12.04 (which is what I assume you meant?), but not > something with > > so little RAM. > > > > On Thu, 1 May 2014, Wilfred Khalik wrote: > > > > > Hi guys, > > > > > > I get the below failure error when I run the make test > command: > > > > > > Any help would be appreciated.I am running this on 512MB > Digital Ocean VPS by the way on Linux 12.0.4.4 LTS. > > > > > > Slab Stats 64 > > > Thread stats 200 > > > Global stats 208 > > > Settings 124 > > > Item (no cas) 32 > > > Item (cas) 40 > > > Libevent thread 100 > > > Connection 340 > > > ---------------------------------------- > > > libevent thread cumulative 13100 > > > Thread stats cumulative 13000 > > > ./testapp > > > 1..48 > > > ok 1 - cache_create > > > ok 2 - cache_constructor > > > ok 3 - cache_constructor_fail > > > ok 4 - cache_destructor > > > ok 5 - cache_reuse > > > ok 6 - cache_redzone > > > ok 7 - issue_161 > > > ok 8 - strtol > > > ok 9 - strtoll > > > ok 10 - strtoul > > > ok 11 - strtoull > > > ok 12 - issue_44 > > > ok 13 - vperror > > > ok 14 - issue_101 > > > ok 15 - start_server > > > ok 16 - issue_92 > > > ok 17 - issue_102 > > > ok 18 - binary_noop > > > ok 19 - binary_quit > > > ok 20 - binary_quitq > > > ok 21 - binary_set > > > ok 22 - binary_setq > > > ok 23 - binary_add > > > ok 24 - binary_addq > > > ok 25 - binary_replace > > > ok 26 - binary_replaceq > > > ok 27 - binary_delete > > > ok 28 - binary_deleteq > > > ok 29 - binary_get > > > ok 30 - binary_getq > > > ok 31 - binary_getk > > > ok 32 - binary_getkq > > > ok 33 - binary_incr > > > ok 34 - binary_incrq > > > ok 35 - binary_decr > > > ok 36 - binary_decrq > > > ok 37 - binary_version > > > ok 38 - binary_flush > > > ok 39 - binary_flushq > > > ok 40 - binary_append > > > ok 41 - binary_appendq > > > ok 42 - binary_prepend > > > ok 43 - binary_prependq > > > ok 44 - binary_stat > > > ok 45 - binary_illegal > > > ok 46 - binary_pipeline_hickup > > > SIGINT handled. > > > ok 47 - shutdown > > > ok 48 - stop_server > > > prove ./t > > > t/00-startup.t ....... 1/18 getaddrinfo(): Name or service > not known > > > failed to listen on TCP port 38181: Success > > > t/00-startup.t ....... 13/18 slab class 1: chunk size > 80 perslab 13107 > > > slab class 2: chunk size 104 perslab 10082 > > > slab class 3: chunk size 136 perslab 7710 > > > slab class 4: chunk size 176 perslab 5957 > > > slab class 5: chunk size 224 perslab 4681 > > > slab class 6: chunk size 280 perslab 3744 > > > slab class 7: chunk size 352 perslab 2978 > > > slab class 8: chunk size 440 perslab 2383 > > > slab class 9: chunk size 552 perslab 1899 > > > slab class 10: chunk size 696 perslab 1506 > > > slab class 11: chunk size 872 perslab 1202 > > > slab class 12: chunk size 1096 perslab 956 > > > slab class 13: chunk size 1376 perslab 762 > > > slab class 14: chunk size 1720 perslab 609 > > > slab class 15: chunk size 2152 perslab 487 > > > slab class 16: chunk size 2696 perslab 388 > > > slab class 17: chunk size 3376 perslab 310 > > > slab class 18: chunk size 4224 perslab 248 > > > slab class 19: chunk size 5280 perslab 198 > > > slab class 20: chunk size 6600 perslab 158 > > > slab class 21: chunk size 8256 perslab 127 > > > slab class 22: chunk size 10320 perslab 101 > > > slab class 23: chunk size 12904 perslab 81 > > > slab class 24: chunk size 16136 perslab 64 > > > slab class 25: chunk size 20176 perslab 51 > > > slab class 26: chunk size 25224 perslab 41 > > > slab class 27: chunk size 31536 perslab 33 > > > slab class 28: chunk size 39424 perslab 26 > > > slab class 29: chunk size 49280 perslab 21 > > > slab class 30: chunk size 61600 perslab 17 > > > slab class 31: chunk size 77000 perslab 13 > > > slab class 32: chunk size 96256 perslab 10 > > > slab class 33: chunk size 120320 perslab 8 > > > slab class 34: chunk size 150400 perslab 6 > > > slab class 35: chunk size 188000 perslab 5 > > > slab class 36: chunk size 235000 perslab 4 > > > slab class 37: chunk size 293752 perslab 3 > > > slab class 38: chunk size 367192 perslab 2 > > > slab class 39: chunk size 458992 perslab 2 > > > slab class 40: chunk size 573744 perslab 1 > > > slab class 41: chunk size 717184 perslab 1 > > > slab class 42: chunk size 1048576 perslab 1 > > > <26 server listening (auto-negotiate) > > > <27 server listening (auto-negotiate) > > > <28 send buffer was 180224, now 268435456 > > > <32 send buffer was 180224, now 268435456 > > > <31 server listening (udp) > > > <35 server listening (udp) > > > <30 server listening (udp) > > > <34 server listening (udp) > > > <29 server listening (udp) > > > <33 server listening (udp) > > > <28 server listening (udp) > > > <32 server listening (udp) > > > slab class 1: chunk size 80 perslab 13107 > > > slab class 2: chunk size 104 perslab 10082 > > > slab class 3: chunk size 136 perslab 7710 > > > slab class 4: chunk size 176 perslab 5957 > > > slab class 5: chunk size 224 perslab 4681 > > > slab class 6: chunk size 280 perslab 3744 > > > slab class 7: chunk size 352 perslab 2978 > > > slab class 8: chunk size 440 perslab 2383 > > > slab class 9: chunk size 552 perslab 1899 > > > slab class 10: chunk size 696 perslab 1506 > > > slab class 11: chunk size 872 perslab 1202 > > > slab class 12: chunk size 1096 perslab 956 > > > slab class 13: chunk size 1376 perslab 762 > > > slab class 14: chunk size 1720 perslab 609 > > > slab class 15: chunk size 2152 perslab 487 > > > slab class 16: chunk size 2696 perslab 388 > > > slab class 17: chunk size 3376 perslab 310 > > > slab class 18: chunk size 4224 perslab 248 > > > slab class 19: chunk size 5280 perslab 198 > > > slab class 20: chunk size 6600 perslab 158 > > > slab class 21: chunk size 8256 perslab 127 > > > slab class 22: chunk size 10320 perslab 101 > > > slab class 23: chunk size 12904 perslab 81 > > > slab class 24: chunk size 16136 perslab 64 > > > slab class 25: chunk size 20176 perslab 51 > > > slab class 26: chunk size 25224 perslab 41 > > > slab class 27: chunk size 31536 perslab 33 > > > slab class 28: chunk size 39424 perslab 26 > > > slab class 29: chunk size 49280 perslab 21 > > > slab class 30: chunk size 61600 perslab 17 > > > slab class 31: chunk size 77000 perslab 13 > > > slab class 32: chunk size 96256 perslab 10 > > > slab class 33: chunk size 120320 perslab 8 > > > slab class 34: chunk size 150400 perslab 6 > > > slab class 35: chunk size 188000 perslab 5 > > > slab class 36: chunk size 235000 perslab 4 > > > slab class 37: chunk size 293752 perslab 3 > > > slab class 38: chunk size 367192 perslab 2 > > > slab class 39: chunk size 458992 perslab 2 > > > slab class 40: chunk size 573744 perslab 1 > > > slab class 41: chunk size 717184 perslab 1 > > > slab class 42: chunk size 1048576 perslab 1 > > > <26 server listening (ascii) > > > <27 server listening (ascii) > > > <28 send buffer was 180224, now 268435456 > > > <32 send buffer was 180224, now 268435456 > > > <31 server listening (udp) > > > <35 server listening (udp) > > > <30 server listening (udp) > > > <34 server listening (udp) > > > <29 server listening (udp) > > > <33 server listening (udp) > > > <28 server listening (udp) > > > <32 server listening (udp) > > > <36 new ascii client connection. > > > <36 connection closed. > > > slab class 1: chunk size 80 perslab 13107 > > > slab class 2: chunk size 104 perslab 10082 > > > slab class 3: chunk size 136 perslab 7710 > > > slab class 4: chunk size 176 perslab 5957 > > > slab class 5: chunk size 224 perslab 4681 > > > slab class 6: chunk size 280 perslab 3744 > > > slab class 7: chunk size 352 perslab 2978 > > > slab class 8: chunk size 440 perslab 2383 > > > slab class 9: chunk size 552 perslab 1899 > > > slab class 10: chunk size 696 perslab 1506 > > > slab class 11: chunk size 872 perslab 1202 > > > slab class 12: chunk size 1096 perslab 956 > > > slab class 13: chunk size 1376 perslab 762 > > > slab class 14: chunk size 1720 perslab 609 > > > slab class 15: chunk size 2152 perslab 487 > > > slab class 16: chunk size 2696 perslab 388 > > > slab class 17: chunk size 3376 perslab 310 > > > slab class 18: chunk size 4224 perslab 248 > > > slab class 19: chunk size 5280 perslab 198 > > > slab class 20: chunk size 6600 perslab 158 > > > slab class 21: chunk size 8256 perslab 127 > > > slab class 22: chunk size 10320 perslab 101 > > > slab class 23: chunk size 12904 perslab 81 > > > slab class 24: chunk size 16136 perslab 64 > > > slab class 25: chunk size 20176 perslab 51 > > > slab class 26: chunk size 25224 perslab 41 > > > slab class 27: chunk size 31536 perslab 33 > > > slab class 28: chunk size 39424 perslab 26 > > > slab class 29: chunk size 49280 perslab 21 > > > slab class 30: chunk size 61600 perslab 17 > > > slab class 31: chunk size 77000 perslab 13 > > > slab class 32: chunk size 96256 perslab 10 > > > slab class 33: chunk size 120320 perslab 8 > > > slab class 34: chunk size 150400 perslab 6 > > > slab class 35: chunk size 188000 perslab 5 > > > slab class 36: chunk size 235000 perslab 4 > > > slab class 37: chunk size 293752 perslab 3 > > > slab class 38: chunk size 367192 perslab 2 > > > slab class 39: chunk size 458992 perslab 2 > > > slab class 40: chunk size 573744 perslab 1 > > > slab class 41: chunk size 717184 perslab 1 > > > slab class 42: chunk size 1048576 perslab 1 > > > <26 server listening (binary) > > > <27 server listening (binary) > > > <28 send buffer was 180224, now 268435456 > > > <32 send buffer was 180224, now 268435456 > > > <31 server listening (udp) > > > <35 server listening (udp) > > > <30 server listening (udp) > > > <34 server listening (udp) > > > <29 server listening (udp) > > > <33 server listening (udp) > > > <28 server listening (udp) > > > <32 server listening (udp) > > > Invalid value for binding protocol: http > > > -- should be one of auto, binary, or ascii > > > t/00-startup.t ....... 17/18 Number of threads must be > greater than 0 > > > t/00-startup.t ....... ok > > > t/64bit.t ............ skipped: Skipping 64-bit tests on > 32-bit build > > > t/binary-get.t ....... ok > > > t/binary-sasl.t ...... This server is not built with SASL > support. > > > t/binary-sasl.t ...... ok > > > t/binary.t ........... ok > > > t/bogus-commands.t ... ok > > > t/cas.t .............. ok > > > t/daemonize.t ........ ok > > > t/dash-M.t ........... ok > > > t/evictions.t ........ ok > > > t/expirations.t ...... ok > > > t/flags.t ............ ok > > > t/flush-all.t ........ ok > > > t/getset.t ........... ok > > > t/incrdecr.t ......... ok > > > t/issue_104.t ........ ok > > > t/issue_108.t ........ ok > > > t/issue_14.t ......... ok > > > t/issue_140.t ........ skipped: Fix for Issue 140 was only an > illusion > > > t/issue_152.t ........ ok > > > t/issue_163.t ........ ok > > > t/issue_183.t ........ ok > > > t/issue_192.t ........ ok > > > t/issue_22.t ......... ok > > > t/issue_260.t ........ skipped: Only possible to test #260 > under artificial conditions > > > t/issue_29.t ......... ok > > > t/issue_3.t .......... ok > > > t/issue_41.t ......... ok > > > t/issue_42.t ......... ok > > > t/issue_50.t ......... ok > > > t/issue_61.t ......... ok > > > t/issue_67.t ......... ok > > > t/issue_68.t ......... ok > > > t/issue_70.t ......... ok > > > t/item_size_max.t .... 1/7 Item max size cannot be less than > 1024 bytes. > > > t/item_size_max.t .... 2/7 Cannot set item size limit higher > than 128 mb. > > > t/item_size_max.t .... 3/7 WARNING: Setting item max size > above 1MB is not recommended! > > > Raising this limit increases the minimum memory requirements > > > and will decrease your memory efficiency. > > > WARNING: Setting item max size above 1MB is not recommended! > > > Raising this limit increases the minimum memory requirements > > > and will decrease your memory efficiency. > > > t/item_size_max.t .... ok > > > t/line-lengths.t ..... ok > > > t/lru-crawler.t ...... 93/189 Timeout.. killing the process > > > t/lru-crawler.t ...... Failed 94/189 subtests > > > t/lru.t .............. ok > > > t/maxconns.t ......... ok > > > t/multiversioning.t .. ok > > > t/noreply.t .......... ok > > > t/slabs_reassign.t ... ok > > > t/stats-conns.t ...... ok > > > t/stats-detail.t ..... ok > > > t/stats.t ............ ok > > > t/touch.t ............ ok > > > t/udp.t .............. ok > > > t/unixsocket.t ....... ok > > > t/whitespace.t ....... skipped: Skipping tests probably > because you don't have git. > > > > > > Test Summary Report > > > ------------------- > > > t/lru-crawler.t (Wstat: 13 Tests: 95 Failed: 0) > > > Non-zero wait status: 13 > > > Parse errors: Bad plan. You planned 189 tests but ran 95. > > > Files=49, Tests=6983, 735 wallclock secs ( 1.96 usr 0.23 sys > + 356.49 cusr 49.42 csys = 408.10 CPU) > > > Result: FAIL > > > make: *** [test] Error 1 > > > > > > -- > > > > > > --- > > > You received this message because you are subscribed to the > Google Groups "memcached" group. > > > To unsubscribe from this group and stop receiving emails from > it, send an email to memcached+...@googlegroups.com. > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > -- > > > > --- > > You received this message because you are subscribed to the Google > Groups "memcached" group. > > To unsubscribe from this group and stop receiving emails from it, > send an email to memcached+...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > --- > You received this message because you are subscribed to the Google Groups > "memcached" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to memcached+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > -- --- You received this message because you are subscribed to the Google Groups "memcached" group. To unsubscribe from this group and stop receiving emails from it, send an email to memcached+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
