Package: redis-server
Version: 5:6.0.9-4
Severity: severe
Tags: upstream patch
Forwarded: https://github.com/redis/redis/pull/7958
Control: block 982209 with -1
Discoved due to autopkgtest timeouts on the python-limits package
(#982209).
redis-server crashes with a SIGBUS when you start clustering, due to an
alignment error. armhf on arm64 kernels has hard alignment requirements.
=== REDIS BUG REPORT START: Cut & paste starting from here ===
16647:M 10 Feb 2021 21:34:54.656 # Redis 6.0.9 crashed by signal: 7, si_code: 1
16647:M 10 Feb 2021 21:34:54.656 # Crashed running the instruction at: 0xbc9446
16647:M 10 Feb 2021 21:34:54.656 # Accessing address: 0x1b133c5
16647:M 10 Feb 2021 21:34:54.656 # Killed by PID: 28390341, UID: 0
16647:M 10 Feb 2021 21:34:54.657 # Failed assertion: <no assertion failed> (<no
file>:0)
------ STACK TRACE ------
EIP:
redis-server *:7001 [cluster](clusterProcessPacket+0x491)[0xbc9446]
Backtrace:
redis-server *:7001 [cluster](logStackTrace+0x39)[0xbc263a]
redis-server *:7001 [cluster](sigsegvHandler+0x93)[0xbc2ca4]
/lib/arm-linux-gnueabihf/libc.so.6(+0x2acc0)[0xf7986cc0]
redis-server *:7001 [cluster](clusterProcessPacket+0x491)[0xbc9446]
redis-server *:7001 [cluster](clusterReadHandler+0x9b)[0xbc9b88]
redis-server *:7001 [cluster](+0x9fb64)[0xbfeb64]
redis-server *:7001 [cluster](aeProcessEvents+0x1ef)[0xb85b88]
redis-server *:7001 [cluster](aeMain+0xf)[0xb85e44]
redis-server *:7001 [cluster](main+0x35d)[0xb819e6]
/lib/arm-linux-gnueabihf/libc.so.6(__libc_start_main+0x97)[0xf7976a20]
Full log attached.
Looks like this has been addressed upstream, and that patch resolves the
issue for me.
SR
16647:C 10 Feb 2021 21:19:56.163 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
16647:C 10 Feb 2021 21:19:56.163 # Redis version=6.0.9, bits=32,
commit=00000000, modified=0, pid=16647, just started
16647:C 10 Feb 2021 21:19:56.163 # Configuration loaded
16647:M 10 Feb 2021 21:19:56.165 * Increased maximum number of open files to
10032 (it was originally set to 1024).
16647:M 10 Feb 2021 21:19:56.165 # Warning: 32 bit instance detected but no
memory limit set. Setting 3 GB maxmemory limit with 'noeviction' policy now.
16647:M 10 Feb 2021 21:19:56.165 * No cluster configuration found, I'm
ee0dfd182ce28b79f27190c079dbae115bb9d31a
16647:M 10 Feb 2021 21:19:56.211 * Running mode=cluster, port=7001.
16647:M 10 Feb 2021 21:19:56.211 # WARNING: The TCP backlog setting of 511
cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower
value of 128.
16647:M 10 Feb 2021 21:19:56.211 # Server initialized
16647:M 10 Feb 2021 21:19:56.211 # WARNING overcommit_memory is set to 0!
Background save may fail under low memory condition. To fix this issue add
'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the
command 'sysctl vm.overcommit_memory=1' for this to take effect.
16647:M 10 Feb 2021 21:19:56.211 # WARNING you have Transparent Huge Pages
(THP) support enabled in your kernel. This will create latency and memory usage
issues with Redis. To fix this issue run the command 'echo madvise >
/sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your
/etc/rc.local in order to retain the setting after a reboot. Redis must be
restarted after THP is disabled (set to 'madvise' or 'never').
16647:M 10 Feb 2021 21:19:56.212 * Ready to accept connections
16647:M 10 Feb 2021 21:20:27.306 # configEpoch set to 2 via CLUSTER
SET-CONFIG-EPOCH
=== REDIS BUG REPORT START: Cut & paste starting from here ===
16647:M 10 Feb 2021 21:34:54.656 # Redis 6.0.9 crashed by signal: 7, si_code: 1
16647:M 10 Feb 2021 21:34:54.656 # Crashed running the instruction at: 0xbc9446
16647:M 10 Feb 2021 21:34:54.656 # Accessing address: 0x1b133c5
16647:M 10 Feb 2021 21:34:54.656 # Killed by PID: 28390341, UID: 0
16647:M 10 Feb 2021 21:34:54.657 # Failed assertion: <no assertion failed> (<no
file>:0)
------ STACK TRACE ------
EIP:
redis-server *:7001 [cluster](clusterProcessPacket+0x491)[0xbc9446]
Backtrace:
redis-server *:7001 [cluster](logStackTrace+0x39)[0xbc263a]
redis-server *:7001 [cluster](sigsegvHandler+0x93)[0xbc2ca4]
/lib/arm-linux-gnueabihf/libc.so.6(+0x2acc0)[0xf7986cc0]
redis-server *:7001 [cluster](clusterProcessPacket+0x491)[0xbc9446]
redis-server *:7001 [cluster](clusterReadHandler+0x9b)[0xbc9b88]
redis-server *:7001 [cluster](+0x9fb64)[0xbfeb64]
redis-server *:7001 [cluster](aeProcessEvents+0x1ef)[0xb85b88]
redis-server *:7001 [cluster](aeMain+0xf)[0xb85e44]
redis-server *:7001 [cluster](main+0x35d)[0xb819e6]
/lib/arm-linux-gnueabihf/libc.so.6(__libc_start_main+0x97)[0xf7976a20]
------ INFO OUTPUT ------
# Server
redis_version:6.0.9
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:74d695619ee2c9cb
redis_mode:cluster
os:Linux 4.19.0-14-arm64 armv8l
arch_bits:32
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:10.2.1
process_id:16647
run_id:46f70fb862cefd4e2c1f068385d388a6258005ff
tcp_port:7001
uptime_in_seconds:31
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:2379291
executable:/tmp/mantest/redis-server
config_file:/tmp/mantest/tests/redis-configurations/cluster/redis-1.conf
io_threads_active:0
# Clients
connected_clients:1
client_recent_max_input_buffer:12
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
# Memory
used_memory:978644
used_memory_human:955.71K
used_memory_rss:4382720
used_memory_rss_human:4.18M
used_memory_peak:1136932
used_memory_peak_human:1.08M
used_memory_peak_perc:86.08%
used_memory_overhead:965356
used_memory_startup:948472
used_memory_dataset:13288
used_memory_dataset_perc:44.04%
allocator_allocated:1000760
allocator_active:4344832
allocator_resident:4344832
total_system_memory:4003082240
total_system_memory_human:3.73G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:3221225472
maxmemory_human:3.00G
maxmemory_policy:noeviction
allocator_frag_ratio:4.34
allocator_frag_bytes:3344072
allocator_rss_ratio:1.00
allocator_rss_bytes:0
rss_overhead_ratio:1.01
rss_overhead_bytes:37888
mem_fragmentation_ratio:4.38
mem_fragmentation_bytes:3381960
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:16884
mem_aof_buffer:0
mem_allocator:libc
active_defrag_running:0
lazyfree_pending_objects:0
# Persistence
loading:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1612991996
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
# Stats
total_connections_received:1
total_commands_processed:7
instantaneous_ops_per_sec:4
total_net_input_bytes:55759
total_net_output_bytes:7729
instantaneous_input_kbps:34.03
instantaneous_output_kbps:4.72
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_reads_processed:10
total_writes_processed:7
io_threaded_reads_processed:0
io_threaded_writes_processed:0
# Replication
role:master
connected_slaves:0
master_replid:0a04e2b499e454933581fef2007dccdefe468d16
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:0.018198
used_cpu_user:0.024264
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
# Modules
# Commandstats
cmdstat_info:calls=2,usec=227,usec_per_call=113.50
cmdstat_cluster:calls=5,usec=888,usec_per_call=177.60
# Cluster
cluster_enabled:1
# Keyspace
------ CLIENT LIST OUTPUT ------
id=4 addr=?:0 fd=11 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1
qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=16884 events=r
cmd=cluster user=default
------ REGISTERS ------
16647:M 10 Feb 2021 21:34:54.658 #
R10:0000000000c17a30 R9 :000000008dd11af8
R8 :0000000000000177 R7 :0000000001ae3633
R6 :0000000001b10820 R5 :0000000001b133b5
R4 :0000000001b10800 R3 :0000000000000000
R2 :0000000000000001 R1 :0000000000000000
R0 :0000000004000000 EC :0000000001a73008
fp: 0000000000000000 ip:0000000000c42830
16647:M 10 Feb 2021 21:34:54.658 # (ff9c82ef) -> 00000008
16647:M 10 Feb 2021 21:34:54.658 # (ff9c82ee) -> 000008c8
16647:M 10 Feb 2021 21:34:54.658 # (ff9c82ed) -> 000011a0
16647:M 10 Feb 2021 21:34:54.658 # (ff9c82ec) -> 0000000c
16647:M 10 Feb 2021 21:34:54.658 # (ff9c82eb) -> a3e91a00
16647:M 10 Feb 2021 21:34:54.658 # (ff9c82ea) -> 01ada380
16647:M 10 Feb 2021 21:34:54.658 # (ff9c82e9) -> f79b71ff
16647:M 10 Feb 2021 21:34:54.658 # (ff9c82e8) -> 00bfea95
16647:M 10 Feb 2021 21:34:54.658 # (ff9c82e7) -> 00000005
16647:M 10 Feb 2021 21:34:54.658 # (ff9c82e6) -> f7a57000
16647:M 10 Feb 2021 21:34:54.658 # (ff9c82e5) -> 00001100
16647:M 10 Feb 2021 21:34:54.658 # (ff9c82e4) -> f7e7f270
16647:M 10 Feb 2021 21:34:54.658 # (ff9c82e3) -> f7e7f304
16647:M 10 Feb 2021 21:34:54.658 # (ff9c82e2) -> 000011a6
16647:M 10 Feb 2021 21:34:54.658 # (ff9c82e1) -> f7a5776c
16647:M 10 Feb 2021 21:34:54.658 # (ff9c82e0) -> 00bfea95
------ MODULES INFO OUTPUT ------
------ FAST MEMORY TEST ------
16647:M 10 Feb 2021 21:34:54.658 # Bio thread for job type #0 terminated
16647:M 10 Feb 2021 21:34:54.659 # Bio thread for job type #1 terminated
16647:M 10 Feb 2021 21:34:54.659 # Bio thread for job type #2 terminated
*** Preparing to test memory region c43000 (69632 bytes)
*** Preparing to test memory region 1a73000 (270336 bytes)
*** Preparing to test memory region 1ab5000 (536576 bytes)
*** Preparing to test memory region f5ef6000 (8388608 bytes)
*** Preparing to test memory region f66f7000 (8388608 bytes)
*** Preparing to test memory region f6ef8000 (9048064 bytes)
*** Preparing to test memory region f7a59000 (8192 bytes)
*** Preparing to test memory region f7a7f000 (8192 bytes)
*** Preparing to test memory region f7c60000 (16384 bytes)
*** Preparing to test memory region f7d53000 (4096 bytes)
*** Preparing to test memory region f7e4b000 (4096 bytes)
*** Preparing to test memory region f7e7e000 (24576 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a
memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------
Symbol: clusterProcessPacket (base: 0xbc8fb5)
Module: redis-server *:7001 [cluster] (base 0xb5f000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0xbc8fb5 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
16647:M 10 Feb 2021 21:34:55.002 # dump of function (hexdump of 1297 bytes):
f8cc2a2de9f04f05460669dff8c43a7a442ded028b99b0b489dff8bc7ad35864ba7f441b6817934ff000037368a4b293fa83f8bff78cff092c8246894613d8dff8983afb58d3f8dc3a03ebc40303f5a023d3f8d020d3f8d4100132c3f8d02041f10001c3f8d410dff8741a4346224600207944bff7a4fcb8f10f0f11d9296911f8013c03f00702042a00f21e85dfe802f09496999c1d0040f6f803984500f007814ff00108dff83c2adff8283a7a44d3581a68179b5a404ff0000340f00085404619b0bdec028bbde8f08f51f8113c9845e6d83389b3f5807fe2d1022c77d8f38968225bba9bb202fb03f303f50d634345d6d106f128032821184608ee103ab6f8ca380593c6f75bfadff8d0398046014657f803b0dbf8dc3a9869bef7e6fa07464046c6f7fefa002f00f0ad81bf68002f00f0a9813b6bc7f860a8c7f86498990640f1a6810023cde90633cde908334ff6fd70204000284ad0022c65d9032c00f0cc80042c00f07e82052c00f05483062c00f08a83082c00f02f84072c00f06d84092c40f05c84002f8ad006f50d62d6f8d83807f108004ff001081bbad2e90045029306f6dd03019322462b4696f8dc18009121f04eff75e7db0888e711f8033c85e731f8053c82e751f8093c7fe7032c3ff461af042c5ed0631f082c18bf012b00f2b681b8f50d6f7ff45aaf81e7dff8f4186a697944bff7e2fb022c00f05d82dff8e8387b441b6893f8903823b9dbf8f03a002b00f0948301212846fcf73dfddff8cc287a4414b1dff8c8287a44059b0020dff8c0185bba79449bb208936b69bff7bdfb6b690593002b00f095801b6b13f0200800f08b81002f00f09d82dff8981807f1080201207944bff7a8fb324629463846fcf72ff938b1dbf8dc3a03f5a0239a6f42f006029a6768694ff00008fef76bf806e7d6f8d038d6f8d4281bba12ba134403f6d80343457ff4f9ae06f128032821184608ee103ab6f8ca380593c6f77df9dff814388046014657f803b0dbf8dc3a9869bef708fa07464046c6f720fa002f7ff422af032c08bf06f50d6600f0bf82cde9087733e706f50d66002f00f0b78228213046c6f759f9dbf8dc3a044621469869bef7e8f9024620461446c6f7fff9002c3ff4bbaea468002c3ff4b7ae236b13f0180f7ff4b2aedff8c01718ee102a3346022079444ff00108bff736fbdbf8dc2a236b02f5a022c4f868a823f00403c4f86c9843f008032363936f43f00603936795e6002f00f05a83089b03f400723b6b23f4007313433b63002c40f04b82990640f156824ff00108dff8601706f50564282220467944b8f798e8002840f00f813846fcf751f93b6bdc0740f1e48106f150094ff40062494607f14000b8f784e8e0b3dde906233846cdf80090fef754fadbf8dc0a0024dde906ce00f50030203003e00134b4f5804f28d0e31004f0070219f803301341db07f3d550f82430002b18bfbb42edd0d3e90e128c457eeb0202e7d2dff8dc160120083307f108027944bff7c2fad7f8c80840b104f50044dbf8dc3a083453f82410fcf7acfadff8b4367b441b681a6bd0070bd53a6bd10708d5d3e90e23dde90610834208bf8a4200f0ed82294630464ff00108fef73dfc0ee6cde90677cde908775ee6d6e9040104f09df9884608900991d6e9060104f096f9dbf8dcca4246cde906010898dce902138142934138bfcce90208d7e90e1206981346079a8142934107d20cf5a023c7e90e029a6f42f00c029a67d6e9080104f075f9dbf8dc3a8c46c7f88c1803f5a023c7f88808c7f878a8c7f87c98d3e914120a433ff421aedff80026
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Please report the crash by opening an issue on github:
http://github.com/redis/redis/issues
Suspect RAM error? Use redis-server --test-memory to verify it.
