Hello, my gigabyte brix system (Celeron J3160) worked rock solid 24/7 up to now (stretch install, upgrade to buster) as a small server with zabbix/mariadb, some kvm virtual machines and docker containers. Mass storage is at 2 usb3 2T disks. After updating the kernel to 4.19.0-10-amd64 (Debian 4.19.132-1) the system froze after around 2-3 days. I rebooted and tried again with the same result. I did a fallback to 4.19.0-9-amd64 (4.19.118-2+deb10u1) and the system again runs stable for a month now.
Unfortunately no information about the reason for the freeze could be found in the logs. But around one day after boot I found a similar warning in the log like it is reported in this report. Here is the message of the first try: Aug 9 00:00:02 brix kernel: [273566.195095] ------------[ cut here ]------------ Aug 9 00:00:02 brix kernel: [273566.195107] percpu ref (css_release) <= 0 (-399677) after switching to atomic Aug 9 00:00:02 brix kernel: [273566.195127] WARNING: CPU: 1 PID: 0 at lib/percpu-refcount.c:155 percpu_ref_switch_to_atomic_rcu+0xf8/0x120 Aug 9 00:00:02 brix kernel: [273566.195128] Modules linked in: fuse btrfs zstd_compress zstd_decompress xxhash xor raid6_pq ufs qnx4 hfsplus hfs minix vfat msdos fat jfs xfs dm_mod vhost_net vhost tap tun devlink ipt_MASQUERADE nf_conntrack_netlink xfrm_user xfrm_algo nft_counter nft_chain_nat_ipv4 nf_nat_ipv4 xt_addrtype nft_compat nf_tables nfnetlink xt_conntrack nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c br_netfilter nls_utf8 overlay isofs loop bridge stp llc snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic btusb intel_rapl intel_powerclamp coretemp arc4 kvm_intel snd_hda_intel i915 snd_hda_codec iwlmvm kvm hci_uart mac80211 snd_hda_core btqca evdev snd_hwdep btrtl irqbypass snd_pcm btbcm drm_kms_helper crct10dif_pclmul btintel crc32_pclmul iwlwifi rtsx_pci_ms ghash_clmulni_intel snd_timer intel_cstate Aug 9 00:00:02 brix kernel: [273566.195176] bluetooth memstick cfg80211 iTCO_wdt pcspkr iTCO_vendor_support drm snd soundcore i2c_algo_bit drbg ansi_cprng ecdh_generic pcc_cpufreq video rfkill pwm_lpss_platform pwm_lpss acpi_pad intel_int0002_vgpio sg button ib_iser rdma_cm iw_cm ib_cm ib_core configfs iscsi_tcp nfsd libiscsi_tcp libiscsi scsi_transport_iscsi auth_rpcgss nfs_acl lockd grace sunrpc ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic fscrypto ecb uas usb_storage hid_generic usbhid sd_mod crc32c_intel rtsx_pci_sdmmc ahci libahci aesni_intel xhci_pci xhci_hcd libata i2c_i801 r8169 sdhci_pci aes_x86_64 crypto_simd cryptd glue_helper cqhci sdhci usbcore lpc_ich realtek scsi_mod libphy rtsx_pci mmc_core mfd_core usb_common thermal fan i2c_hid hid Aug 9 00:00:02 brix kernel: [273566.195225] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.19.0-10-amd64 #1 Debian 4.19.132-1 Aug 9 00:00:02 brix kernel: [273566.195227] Hardware name: GIGABYTE GB-BACE-3160/MZBSWMP-00, BIOS F6 06/13/2018 Aug 9 00:00:02 brix kernel: [273566.195230] RIP: 0010:percpu_ref_switch_to_atomic_rcu+0xf8/0x120 Aug 9 00:00:02 brix kernel: [273566.195232] Code: 78 ff ff ff 80 3d 2b 71 d2 00 00 75 8c 49 8b 54 24 d8 49 8b 74 24 e8 48 c7 c7 68 98 29 a4 c6 05 11 71 d2 00 01 e8 f2 ae ca ff <0f> 0b e9 68 ff ff ff f0 49 83 6c 24 d8 01 75 9c 49 8b 44 24 e8 48 Aug 9 00:00:02 brix kernel: [273566.195234] RSP: 0018:ffff9b017bc83ee0 EFLAGS: 00010282 Aug 9 00:00:02 brix kernel: [273566.195236] RAX: 0000000000000000 RBX: 7ffffffffff9e6c3 RCX: 0000000000000000 Aug 9 00:00:02 brix kernel: [273566.195238] RDX: 0000000000000041 RSI: ffffffffa49f7b21 RDI: 0000000000000246 Aug 9 00:00:02 brix kernel: [273566.195239] RBP: 000042a50401ac08 R08: 0000000000000000 R09: 0000000000021980 Aug 9 00:00:02 brix kernel: [273566.195241] R10: 00018e1d3146539c R11: ffffffffa49f7b06 R12: ffff9b0176c3e838 Aug 9 00:00:02 brix kernel: [273566.195242] R13: ffffffffa452dce0 R14: 7fffffffffffffff R15: 0000000000000202 Aug 9 00:00:02 brix kernel: [273566.195244] FS: 0000000000000000(0000) GS:ffff9b017bc80000(0000) knlGS:0000000000000000 Aug 9 00:00:02 brix kernel: [273566.195245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Aug 9 00:00:02 brix kernel: [273566.195247] CR2: 00007fce0683e9a0 CR3: 000000015a20a000 CR4: 00000000001026e0 Aug 9 00:00:02 brix kernel: [273566.195248] Call Trace: Aug 9 00:00:02 brix kernel: [273566.195252] <IRQ> Aug 9 00:00:02 brix kernel: [273566.195259] rcu_process_callbacks+0x218/0x4b0 Aug 9 00:00:02 brix kernel: [273566.195264] ? rebalance_domains+0x274/0x2c0 Aug 9 00:00:02 brix kernel: [273566.195268] __do_softirq+0xde/0x2d8 Aug 9 00:00:02 brix kernel: [273566.195273] irq_exit+0xba/0xc0 Aug 9 00:00:02 brix kernel: [273566.195276] smp_apic_timer_interrupt+0x74/0x140 Aug 9 00:00:02 brix kernel: [273566.195279] apic_timer_interrupt+0xf/0x20 Aug 9 00:00:02 brix kernel: [273566.195281] </IRQ> Aug 9 00:00:02 brix kernel: [273566.195284] RIP: 0010:cpuidle_enter_state+0xb9/0x320 Aug 9 00:00:02 brix kernel: [273566.195286] Code: e8 4c b9 b0 ff 80 7c 24 0b 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 3b 02 00 00 31 ff e8 1e a9 b6 ff fb 66 0f 1f 44 00 00 <48> b8 ff ff ff ff f3 01 00 00 48 2b 1c 24 ba ff ff ff 7f 48 39 c3 Aug 9 00:00:02 brix kernel: [273566.195287] RSP: 0018:ffffbda6806afe90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 Aug 9 00:00:02 brix kernel: [273566.195290] RAX: ffff9b017bca20c0 RBX: 0000f8ce97e0d4ed RCX: 000000000000001f Aug 9 00:00:02 brix kernel: [273566.195291] RDX: 0000f8ce97e0d4ed RSI: 0000000050000000 RDI: 0000000000000000 Aug 9 00:00:02 brix kernel: [273566.195292] RBP: ffff9b017bcaa300 R08: 0000000000000000 R09: 0000000000021980 Aug 9 00:00:02 brix kernel: [273566.195293] R10: 00018e1d3144d260 R11: ffff9b017bca10a8 R12: 0000000000000003 Aug 9 00:00:02 brix kernel: [273566.195295] R13: ffffffffa44b7138 R14: 0000000000000003 R15: 0000000000000000 Aug 9 00:00:02 brix kernel: [273566.195299] do_idle+0x228/0x270 Aug 9 00:00:02 brix kernel: [273566.195302] cpu_startup_entry+0x6f/0x80 Aug 9 00:00:02 brix kernel: [273566.195305] start_secondary+0x1a4/0x200 Aug 9 00:00:02 brix kernel: [273566.195309] secondary_startup_64+0xa4/0xb0 Aug 9 00:00:02 brix kernel: [273566.195312] ---[ end trace e76a8f0ad6b82761 ]--- and the message of the second try: Aug 16 00:00:02 brix kernel: [282158.027005] ------------[ cut here ]------------ Aug 16 00:00:02 brix kernel: [282158.027015] percpu ref (css_release) <= 0 (-412250) after switching to atomic Aug 16 00:00:02 brix kernel: [282158.027035] WARNING: CPU: 0 PID: 9 at lib/percpu-refcount.c:155 percpu_ref_switch_to_atomic_rcu+0xf8/0x120 Aug 16 00:00:02 brix kernel: [282158.027037] Modules linked in: nls_utf8 isofs loop tcp_diag inet_diag hid_generic usbhid vhost_net vhost tap tun ipt_MASQUERADE nf_conntrack_netlink xfrm_user xfrm_algo nft_counter nft_chain_nat_ipv4 nf_nat_ipv4 xt_addrtype nft_compat xt_conntrack nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c br_netfilter devlink overlay nf_tables nfnetlink bridge stp llc snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic arc4 intel_rapl iwlmvm snd_hda_intel intel_powerclamp i915 snd_hda_codec mac80211 coretemp evdev kvm_intel kvm iwlwifi snd_hda_core drm_kms_helper irqbypass hci_uart snd_hwdep crct10dif_pclmul crc32_pclmul btqca snd_pcm btusb btrtl ghash_clmulni_intel btbcm drm intel_cstate pcspkr cfg80211 btintel iTCO_wdt snd_timer iTCO_vendor_support rtsx_pci_ms bluetooth i2c_algo_bit Aug 16 00:00:02 brix kernel: [282158.027080] sg snd soundcore memstick drbg ansi_cprng ecdh_generic video acpi_pad pcc_cpufreq rfkill intel_int0002_vgpio pwm_lpss_platform pwm_lpss button ib_iser rdma_cm iw_cm ib_cm nfsd ib_core auth_rpcgss configfs nfs_acl lockd grace iscsi_tcp sunrpc libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic fscrypto ecb uas usb_storage sd_mod crc32c_intel rtsx_pci_sdmmc xhci_pci sdhci_pci xhci_hcd cqhci sdhci ahci libahci aesni_intel r8169 libata realtek usbcore aes_x86_64 crypto_simd cryptd glue_helper i2c_i801 lpc_ich mmc_core libphy scsi_mod usb_common rtsx_pci mfd_core thermal fan i2c_hid hid Aug 16 00:00:02 brix kernel: [282158.027126] CPU: 0 PID: 9 Comm: ksoftirqd/0 Not tainted 4.19.0-10-amd64 #1 Debian 4.19.132-1 Aug 16 00:00:02 brix kernel: [282158.027127] Hardware name: GIGABYTE GB-BACE-3160/MZBSWMP-00, BIOS F6 06/13/2018 Aug 16 00:00:02 brix kernel: [282158.027130] RIP: 0010:percpu_ref_switch_to_atomic_rcu+0xf8/0x120 Aug 16 00:00:02 brix kernel: [282158.027133] Code: 78 ff ff ff 80 3d 2b 71 d2 00 00 75 8c 49 8b 54 24 d8 49 8b 74 24 e8 48 c7 c7 68 98 09 a5 c6 05 11 71 d2 00 01 e8 f2 ae ca ff <0f> 0b e9 68 ff ff ff f0 49 83 6c 24 d8 01 75 9c 49 8b 44 24 e8 48 Aug 16 00:00:02 brix kernel: [282158.027135] RSP: 0000:ffffb925c0687df0 EFLAGS: 00010286 Aug 16 00:00:02 brix kernel: [282158.027137] RAX: 0000000000000000 RBX: 7ffffffffff9b5a8 RCX: 0000000000000000 Aug 16 00:00:02 brix kernel: [282158.027138] RDX: 0000000000000041 RSI: ffffffffa57f7b21 RDI: 0000000000000246 Aug 16 00:00:02 brix kernel: [282158.027139] RBP: 00004d1c4401a178 R08: 0000000000000000 R09: 0000000000021980 Aug 16 00:00:02 brix kernel: [282158.027141] R10: 00019aa08aecbe24 R11: ffffffffa57f7b06 R12: ffff8c097ab2c038 Aug 16 00:00:02 brix kernel: [282158.027142] R13: ffffffffa532dce0 R14: 7fffffffffffffff R15: 0000000000000202 Aug 16 00:00:02 brix kernel: [282158.027144] FS: 0000000000000000(0000) GS:ffff8c097bc00000(0000) knlGS:0000000000000000 Aug 16 00:00:02 brix kernel: [282158.027146] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Aug 16 00:00:02 brix kernel: [282158.027147] CR2: 00007f4986b7fcd0 CR3: 000000007192e000 CR4: 00000000001026f0 Aug 16 00:00:02 brix kernel: [282158.027148] Call Trace: Aug 16 00:00:02 brix kernel: [282158.027159] rcu_process_callbacks+0x218/0x4b0 Aug 16 00:00:02 brix kernel: [282158.027164] __do_softirq+0xde/0x2d8 Aug 16 00:00:02 brix kernel: [282158.027168] ? sort_range+0x20/0x20 Aug 16 00:00:02 brix kernel: [282158.027171] run_ksoftirqd+0x26/0x40 Aug 16 00:00:02 brix kernel: [282158.027174] smpboot_thread_fn+0xc5/0x160 Aug 16 00:00:02 brix kernel: [282158.027177] kthread+0x112/0x130 Aug 16 00:00:02 brix kernel: [282158.027180] ? kthread_bind+0x30/0x30 Aug 16 00:00:02 brix kernel: [282158.027182] ret_from_fork+0x35/0x40 Aug 16 00:00:02 brix kernel: [282158.027185] ---[ end trace 8620e7ca719e6c15 ]--- These warnings do no appear in the logs with the previous kernel 4.19.0-9-amd64 (4.19.118-2+deb10u1). Bye,wahlm
