Package: redis-server Version: 5:6.0.9-4 Severity: severe Tags: upstream patch Forwarded: https://github.com/redis/redis/pull/7958 Control: block 982209 with -1
Discoved due to autopkgtest timeouts on the python-limits package (#982209). redis-server crashes with a SIGBUS when you start clustering, due to an alignment error. armhf on arm64 kernels has hard alignment requirements. === REDIS BUG REPORT START: Cut & paste starting from here === 16647:M 10 Feb 2021 21:34:54.656 # Redis 6.0.9 crashed by signal: 7, si_code: 1 16647:M 10 Feb 2021 21:34:54.656 # Crashed running the instruction at: 0xbc9446 16647:M 10 Feb 2021 21:34:54.656 # Accessing address: 0x1b133c5 16647:M 10 Feb 2021 21:34:54.656 # Killed by PID: 28390341, UID: 0 16647:M 10 Feb 2021 21:34:54.657 # Failed assertion: <no assertion failed> (<no file>:0) ------ STACK TRACE ------ EIP: redis-server *:7001 [cluster](clusterProcessPacket+0x491)[0xbc9446] Backtrace: redis-server *:7001 [cluster](logStackTrace+0x39)[0xbc263a] redis-server *:7001 [cluster](sigsegvHandler+0x93)[0xbc2ca4] /lib/arm-linux-gnueabihf/libc.so.6(+0x2acc0)[0xf7986cc0] redis-server *:7001 [cluster](clusterProcessPacket+0x491)[0xbc9446] redis-server *:7001 [cluster](clusterReadHandler+0x9b)[0xbc9b88] redis-server *:7001 [cluster](+0x9fb64)[0xbfeb64] redis-server *:7001 [cluster](aeProcessEvents+0x1ef)[0xb85b88] redis-server *:7001 [cluster](aeMain+0xf)[0xb85e44] redis-server *:7001 [cluster](main+0x35d)[0xb819e6] /lib/arm-linux-gnueabihf/libc.so.6(__libc_start_main+0x97)[0xf7976a20] Full log attached. Looks like this has been addressed upstream, and that patch resolves the issue for me. SR
16647:C 10 Feb 2021 21:19:56.163 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 16647:C 10 Feb 2021 21:19:56.163 # Redis version=6.0.9, bits=32, commit=00000000, modified=0, pid=16647, just started 16647:C 10 Feb 2021 21:19:56.163 # Configuration loaded 16647:M 10 Feb 2021 21:19:56.165 * Increased maximum number of open files to 10032 (it was originally set to 1024). 16647:M 10 Feb 2021 21:19:56.165 # Warning: 32 bit instance detected but no memory limit set. Setting 3 GB maxmemory limit with 'noeviction' policy now. 16647:M 10 Feb 2021 21:19:56.165 * No cluster configuration found, I'm ee0dfd182ce28b79f27190c079dbae115bb9d31a 16647:M 10 Feb 2021 21:19:56.211 * Running mode=cluster, port=7001. 16647:M 10 Feb 2021 21:19:56.211 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 16647:M 10 Feb 2021 21:19:56.211 # Server initialized 16647:M 10 Feb 2021 21:19:56.211 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. 16647:M 10 Feb 2021 21:19:56.211 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never'). 16647:M 10 Feb 2021 21:19:56.212 * Ready to accept connections 16647:M 10 Feb 2021 21:20:27.306 # configEpoch set to 2 via CLUSTER SET-CONFIG-EPOCH === REDIS BUG REPORT START: Cut & paste starting from here === 16647:M 10 Feb 2021 21:34:54.656 # Redis 6.0.9 crashed by signal: 7, si_code: 1 16647:M 10 Feb 2021 21:34:54.656 # Crashed running the instruction at: 0xbc9446 16647:M 10 Feb 2021 21:34:54.656 # Accessing address: 0x1b133c5 16647:M 10 Feb 2021 21:34:54.656 # Killed by PID: 28390341, UID: 0 16647:M 10 Feb 2021 21:34:54.657 # Failed assertion: <no assertion failed> (<no file>:0) ------ STACK TRACE ------ EIP: redis-server *:7001 [cluster](clusterProcessPacket+0x491)[0xbc9446] Backtrace: redis-server *:7001 [cluster](logStackTrace+0x39)[0xbc263a] redis-server *:7001 [cluster](sigsegvHandler+0x93)[0xbc2ca4] /lib/arm-linux-gnueabihf/libc.so.6(+0x2acc0)[0xf7986cc0] redis-server *:7001 [cluster](clusterProcessPacket+0x491)[0xbc9446] redis-server *:7001 [cluster](clusterReadHandler+0x9b)[0xbc9b88] redis-server *:7001 [cluster](+0x9fb64)[0xbfeb64] redis-server *:7001 [cluster](aeProcessEvents+0x1ef)[0xb85b88] redis-server *:7001 [cluster](aeMain+0xf)[0xb85e44] redis-server *:7001 [cluster](main+0x35d)[0xb819e6] /lib/arm-linux-gnueabihf/libc.so.6(__libc_start_main+0x97)[0xf7976a20] ------ INFO OUTPUT ------ # Server redis_version:6.0.9 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:74d695619ee2c9cb redis_mode:cluster os:Linux 4.19.0-14-arm64 armv8l arch_bits:32 multiplexing_api:epoll atomicvar_api:atomic-builtin gcc_version:10.2.1 process_id:16647 run_id:46f70fb862cefd4e2c1f068385d388a6258005ff tcp_port:7001 uptime_in_seconds:31 uptime_in_days:0 hz:10 configured_hz:10 lru_clock:2379291 executable:/tmp/mantest/redis-server config_file:/tmp/mantest/tests/redis-configurations/cluster/redis-1.conf io_threads_active:0 # Clients connected_clients:1 client_recent_max_input_buffer:12 client_recent_max_output_buffer:0 blocked_clients:0 tracking_clients:0 clients_in_timeout_table:0 # Memory used_memory:978644 used_memory_human:955.71K used_memory_rss:4382720 used_memory_rss_human:4.18M used_memory_peak:1136932 used_memory_peak_human:1.08M used_memory_peak_perc:86.08% used_memory_overhead:965356 used_memory_startup:948472 used_memory_dataset:13288 used_memory_dataset_perc:44.04% allocator_allocated:1000760 allocator_active:4344832 allocator_resident:4344832 total_system_memory:4003082240 total_system_memory_human:3.73G used_memory_lua:37888 used_memory_lua_human:37.00K used_memory_scripts:0 used_memory_scripts_human:0B number_of_cached_scripts:0 maxmemory:3221225472 maxmemory_human:3.00G maxmemory_policy:noeviction allocator_frag_ratio:4.34 allocator_frag_bytes:3344072 allocator_rss_ratio:1.00 allocator_rss_bytes:0 rss_overhead_ratio:1.01 rss_overhead_bytes:37888 mem_fragmentation_ratio:4.38 mem_fragmentation_bytes:3381960 mem_not_counted_for_evict:0 mem_replication_backlog:0 mem_clients_slaves:0 mem_clients_normal:16884 mem_aof_buffer:0 mem_allocator:libc active_defrag_running:0 lazyfree_pending_objects:0 # Persistence loading:0 rdb_changes_since_last_save:0 rdb_bgsave_in_progress:0 rdb_last_save_time:1612991996 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:-1 rdb_current_bgsave_time_sec:-1 rdb_last_cow_size:0 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_last_cow_size:0 module_fork_in_progress:0 module_fork_last_cow_size:0 # Stats total_connections_received:1 total_commands_processed:7 instantaneous_ops_per_sec:4 total_net_input_bytes:55759 total_net_output_bytes:7729 instantaneous_input_kbps:34.03 instantaneous_output_kbps:4.72 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:0 expired_stale_perc:0.00 expired_time_cap_reached_count:0 expire_cycle_cpu_milliseconds:0 evicted_keys:0 keyspace_hits:0 keyspace_misses:0 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:0 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0 tracking_total_keys:0 tracking_total_items:0 tracking_total_prefixes:0 unexpected_error_replies:0 total_reads_processed:10 total_writes_processed:7 io_threaded_reads_processed:0 io_threaded_writes_processed:0 # Replication role:master connected_slaves:0 master_replid:0a04e2b499e454933581fef2007dccdefe468d16 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:0 second_repl_offset:-1 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0 # CPU used_cpu_sys:0.018198 used_cpu_user:0.024264 used_cpu_sys_children:0.000000 used_cpu_user_children:0.000000 # Modules # Commandstats cmdstat_info:calls=2,usec=227,usec_per_call=113.50 cmdstat_cluster:calls=5,usec=888,usec_per_call=177.60 # Cluster cluster_enabled:1 # Keyspace ------ CLIENT LIST OUTPUT ------ id=4 addr=?:0 fd=11 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=16884 events=r cmd=cluster user=default ------ REGISTERS ------ 16647:M 10 Feb 2021 21:34:54.658 # R10:0000000000c17a30 R9 :000000008dd11af8 R8 :0000000000000177 R7 :0000000001ae3633 R6 :0000000001b10820 R5 :0000000001b133b5 R4 :0000000001b10800 R3 :0000000000000000 R2 :0000000000000001 R1 :0000000000000000 R0 :0000000004000000 EC :0000000001a73008 fp: 0000000000000000 ip:0000000000c42830 16647:M 10 Feb 2021 21:34:54.658 # (ff9c82ef) -> 00000008 16647:M 10 Feb 2021 21:34:54.658 # (ff9c82ee) -> 000008c8 16647:M 10 Feb 2021 21:34:54.658 # (ff9c82ed) -> 000011a0 16647:M 10 Feb 2021 21:34:54.658 # (ff9c82ec) -> 0000000c 16647:M 10 Feb 2021 21:34:54.658 # (ff9c82eb) -> a3e91a00 16647:M 10 Feb 2021 21:34:54.658 # (ff9c82ea) -> 01ada380 16647:M 10 Feb 2021 21:34:54.658 # (ff9c82e9) -> f79b71ff 16647:M 10 Feb 2021 21:34:54.658 # (ff9c82e8) -> 00bfea95 16647:M 10 Feb 2021 21:34:54.658 # (ff9c82e7) -> 00000005 16647:M 10 Feb 2021 21:34:54.658 # (ff9c82e6) -> f7a57000 16647:M 10 Feb 2021 21:34:54.658 # (ff9c82e5) -> 00001100 16647:M 10 Feb 2021 21:34:54.658 # (ff9c82e4) -> f7e7f270 16647:M 10 Feb 2021 21:34:54.658 # (ff9c82e3) -> f7e7f304 16647:M 10 Feb 2021 21:34:54.658 # (ff9c82e2) -> 000011a6 16647:M 10 Feb 2021 21:34:54.658 # (ff9c82e1) -> f7a5776c 16647:M 10 Feb 2021 21:34:54.658 # (ff9c82e0) -> 00bfea95 ------ MODULES INFO OUTPUT ------ ------ FAST MEMORY TEST ------ 16647:M 10 Feb 2021 21:34:54.658 # Bio thread for job type #0 terminated 16647:M 10 Feb 2021 21:34:54.659 # Bio thread for job type #1 terminated 16647:M 10 Feb 2021 21:34:54.659 # Bio thread for job type #2 terminated *** Preparing to test memory region c43000 (69632 bytes) *** Preparing to test memory region 1a73000 (270336 bytes) *** Preparing to test memory region 1ab5000 (536576 bytes) *** Preparing to test memory region f5ef6000 (8388608 bytes) *** Preparing to test memory region f66f7000 (8388608 bytes) *** Preparing to test memory region f6ef8000 (9048064 bytes) *** Preparing to test memory region f7a59000 (8192 bytes) *** Preparing to test memory region f7a7f000 (8192 bytes) *** Preparing to test memory region f7c60000 (16384 bytes) *** Preparing to test memory region f7d53000 (4096 bytes) *** Preparing to test memory region f7e4b000 (4096 bytes) *** Preparing to test memory region f7e7e000 (24576 bytes) .O.O.O.O.O.O.O.O.O.O.O.O Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible. ------ DUMPING CODE AROUND EIP ------ Symbol: clusterProcessPacket (base: 0xbc8fb5) Module: redis-server *:7001 [cluster] (base 0xb5f000) $ xxd -r -p /tmp/dump.hex /tmp/dump.bin $ objdump --adjust-vma=0xbc8fb5 -D -b binary -m i386:x86-64 /tmp/dump.bin ------ 16647:M 10 Feb 2021 21:34:55.002 # dump of function (hexdump of 1297 bytes): f8cc2a2de9f04f05460669dff8c43a7a442ded028b99b0b489dff8bc7ad35864ba7f441b6817934ff000037368a4b293fa83f8bff78cff092c8246894613d8dff8983afb58d3f8dc3a03ebc40303f5a023d3f8d020d3f8d4100132c3f8d02041f10001c3f8d410dff8741a4346224600207944bff7a4fcb8f10f0f11d9296911f8013c03f00702042a00f21e85dfe802f09496999c1d0040f6f803984500f007814ff00108dff83c2adff8283a7a44d3581a68179b5a404ff0000340f00085404619b0bdec028bbde8f08f51f8113c9845e6d83389b3f5807fe2d1022c77d8f38968225bba9bb202fb03f303f50d634345d6d106f128032821184608ee103ab6f8ca380593c6f75bfadff8d0398046014657f803b0dbf8dc3a9869bef7e6fa07464046c6f7fefa002f00f0ad81bf68002f00f0a9813b6bc7f860a8c7f86498990640f1a6810023cde90633cde908334ff6fd70204000284ad0022c65d9032c00f0cc80042c00f07e82052c00f05483062c00f08a83082c00f02f84072c00f06d84092c40f05c84002f8ad006f50d62d6f8d83807f108004ff001081bbad2e90045029306f6dd03019322462b4696f8dc18009121f04eff75e7db0888e711f8033c85e731f8053c82e751f8093c7fe7032c3ff461af042c5ed0631f082c18bf012b00f2b681b8f50d6f7ff45aaf81e7dff8f4186a697944bff7e2fb022c00f05d82dff8e8387b441b6893f8903823b9dbf8f03a002b00f0948301212846fcf73dfddff8cc287a4414b1dff8c8287a44059b0020dff8c0185bba79449bb208936b69bff7bdfb6b690593002b00f095801b6b13f0200800f08b81002f00f09d82dff8981807f1080201207944bff7a8fb324629463846fcf72ff938b1dbf8dc3a03f5a0239a6f42f006029a6768694ff00008fef76bf806e7d6f8d038d6f8d4281bba12ba134403f6d80343457ff4f9ae06f128032821184608ee103ab6f8ca380593c6f77df9dff814388046014657f803b0dbf8dc3a9869bef708fa07464046c6f720fa002f7ff422af032c08bf06f50d6600f0bf82cde9087733e706f50d66002f00f0b78228213046c6f759f9dbf8dc3a044621469869bef7e8f9024620461446c6f7fff9002c3ff4bbaea468002c3ff4b7ae236b13f0180f7ff4b2aedff8c01718ee102a3346022079444ff00108bff736fbdbf8dc2a236b02f5a022c4f868a823f00403c4f86c9843f008032363936f43f00603936795e6002f00f05a83089b03f400723b6b23f4007313433b63002c40f04b82990640f156824ff00108dff8601706f50564282220467944b8f798e8002840f00f813846fcf751f93b6bdc0740f1e48106f150094ff40062494607f14000b8f784e8e0b3dde906233846cdf80090fef754fadbf8dc0a0024dde906ce00f50030203003e00134b4f5804f28d0e31004f0070219f803301341db07f3d550f82430002b18bfbb42edd0d3e90e128c457eeb0202e7d2dff8dc160120083307f108027944bff7c2fad7f8c80840b104f50044dbf8dc3a083453f82410fcf7acfadff8b4367b441b681a6bd0070bd53a6bd10708d5d3e90e23dde90610834208bf8a4200f0ed82294630464ff00108fef73dfc0ee6cde90677cde908775ee6d6e9040104f09df9884608900991d6e9060104f096f9dbf8dcca4246cde906010898dce902138142934138bfcce90208d7e90e1206981346079a8142934107d20cf5a023c7e90e029a6f42f00c029a67d6e9080104f075f9dbf8dc3a8c46c7f88c1803f5a023c7f88808c7f878a8c7f87c98d3e914120a433ff421aedff80026 === REDIS BUG REPORT END. Make sure to include from START to END. === Please report the crash by opening an issue on github: http://github.com/redis/redis/issues Suspect RAM error? Use redis-server --test-memory to verify it.