** Description changed: - State the component where the Bug is occuring: - kernel + SRU Justification: + ================== + + [Impact] + + * The kernel crashed under load with a null pointer issue in nfs code: + [556585.270959] Krnl Code:#0000000000000000: 0000 illegal + >0000000000000002: 0000 illegal + 0000000000000004: 0000 illegal + 0000000000000006: 0000 illegal + 0000000000000008: 0000 illegal + 000000000000000a: 0000 illegal + 000000000000000c: 0000 illegal + 000000000000000e: 0000 illegal + [556585.270967] Call Trace: + [556585.270982] ([<000003ff80d6fb1a>] rpcauth_lookup_credcache+0x5a/0x300 [sunrpc]) + [556585.270993] [<000003ff80e1182c>] nfs_ctx_key_to_expire+0xec/0x130 [nfs] + [556585.271004] [<000003ff80e1189c>] nfs_key_timeout_notify+0x2c/0x70 [nfs] + [556585.271014] [<000003ff80dfdf7e>] nfs_file_write+0x3e/0x320 [nfs] + [556585.271016] [<00000028165944a8>] new_sync_write+0x118/0x1b0 + [556585.271017] [<0000002816594ee0>] vfs_write+0xb0/0x1b0 + [556585.271019] [<0000002816596a1e>] ksys_pwrite64+0x7e/0xc0 + [556585.271021] [<0000002816bb26b2>] system_call+0x2a6/0x2c8 + + * Several dumps were generated and shared with Canonical. + + * Analysis (done by kernel and SEG) point to refcount leaks fixed, + that are already fixed in the following commit/fix: + + [Fix] + + * ca05cbae2a0468e5d78e9b4605936a8bf5da328b ca05cbae2a04 "NFS: Fix up + nfs_ctx_key_to_expire()" + + [Test Case] + + * There is unfortunately no reproducer or trigger available for this + issue. + + * It just happens now and then under higher load. + + * A patched kernel (focal 5.4 and bionic 5.4-hwe) were created and + ran for more than a week in a special staging environment (at IBM) + without further crashes. + + * Hence the test and verification will be done by the IBM Z team. + + [Where problems could occur] + + * The inode handling can become broken, in case the changes + on the pointers are erroneous. + + * Problems with the authentication and/or the credentials could occur + due to the modifications in put_rpccred, rpc_cred and rpc_auth. + + * The expiration of the cached credentials could be harmed as well, + due to the changes in nfs_ctx_key_to_expire. + + * The different pointer arithmetic may cause further issues - wrong + or null pointer references. + + * Positive is that the original commit was brought upstream by nfs + experts. + + * A patched test kernel sustained day long runs under load in a staging + and test environment. + + * The author of the upstream commit/patch is well known in the NFS area. + + [Other] + + * The Salesforce Case Number 00334334 is associated with this bug. + + * Commit ca05cbae2a04 was upstream accepted with 5.16-rc1. + + * But commit ca05cbae2a04 was unfortunately not tagged as stable, + hence it was not picked automatically. + + * Since kinetic's (22.10) target kernel is 5.18, + it will have the patch included, + hence no dedicated PATCH request for kinetic. + + __________ + + State the component where the Bug is occurring: + kernel Indicate the nature of the problem by answering the below questions: - Is this problem reproducible? No No, steps unknown, but we have seen these before - - - Is the system sitting at a debugger (kdb, or xmon)? No - + - Is the system sitting at a debugger (kdb, or xmon)? No - Is the system hung? No No, dumped and rebooted - - - Are there any custom patches installed? Yes + - Are there any custom patches installed? Yes On base system level (CloudAppliance) we are still running with the zfpc_proc module loaded. But no recent changes in the module and is running absolutely stable in HA (same kernel and userspace, Ubuntu 20.04 LTS) - - - Is there any special hardware that may be relevant to this problem? Yes + - Is there any special hardware that may be relevant to this problem? Yes We are running with mlx (cloud network adapters) installed. - - - Is access information for the machine the problem was found on available? Yes - - - - Is the bug occuring in a userspace application? No - + - Is access information for the machine the problem was found on + available? Yes + + - Is the bug occuring in a userspace application? No - Was a stack trace produced? Yes This is what mention in first comment by @Boris Barth - - Did the system produce an Oops message on the console? Yes - [556585.270902] illegal operation: 0001 ilc:1 [#10] SMP + [556585.270902] illegal operation: 0001 ilc:1 [#10] SMP [556585.270905] Modules linked in: vhost_net macvtap macvlan tap rpcsec_gss_krb5 auth_rpcgss nfsv3 nfs_acl nfs lockd grace fscache veth xt_statistic ipt_REJECT nf_reject_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs iptable_mangle xt_mark sunrpc nf_log_ipv6 nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_set ip_set_hash_net ip_set_hash_ip ip_set tcp_diag inet_diag xt_comment xt_nat cls_cgroup sch_htb act_gact sch_multiq act_mirred act_pedit act_tunnel_key cls_flower act_police cls_u32 vxlan ip6_udp_tunnel udp_tunnel dummy nf_tables ebtable_filter ebtables xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key sch_ingress mlx5_ib ib_uverbs ib_core mlx5_core tls mlxfw ptp pps_core dm_integrity async_xor async_tx dm_bufio bonding xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_nat nf_nat br_netfilter bridge vhost_vsock vmw_vsock_virtio_transport_common vhost vsock 8021q garp mrp stp llc xt_multiport xt_tcpudp qeth_l2 lcs ctcm fsm dasd_fba_mod aufs overlay scsi_dh_rdac [556585.270923] scsi_dh_emc s390_trng xt_state xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter bpfilter sch_fq_codel zFPC_proc(OE) zFPC_diag(OE) vfio_ap vfio_mdev drm vfio_iommu_type1 drm_panel_orientation_quirks i2c_core ip_tables x_tables scsi_dh_alua pkey zcrypt ghash_s390 prng aes_s390 des_s390 libdes sha3_512_s390 sha3_256_s390 sha512_s390 sha256_s390 sha1_s390 sha_common chsc_sch qeth ccwgroup eadm_sch vfio_ccw mdev vfio btrfs libcrc32c crc32_vx_s390 xor zstd_compress raid6_pq dm_crypt virtio_blk dm_service_time dm_multipath zfcp scsi_transport_fc qdio dasd_eckd_mod dasd_mod zlib_deflate [last unloaded: tls] [556585.270945] CPU: 28 PID: 217741 Comm: worker Kdump: loaded Tainted: G D OE 5.4.0-90-generic #101-Ubuntu [556585.270947] Hardware name: IBM 8562 GT2 A00 (LPAR) [556585.270948] Krnl PSW : 0704d00180000000 0000000000000002 (0x2) [556585.270951] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3 [556585.270953] Krnl GPRS: 0000000000000000 0000000000000000 000003e010ebbcf8 00000071c45e1ec0 [556585.270954] 0000000000000000 0000002816f7b18c 00000078dd36a4a0 000000713a62f718 [556585.270955] 0000000000000000 000003e010ebbcf8 0000000000000068 00000071c45e1ec0 [556585.270957] 0000006090a12200 0000000000000c40 000003ff80d6fb54 000003e010ebbbf0 - [556585.270959] Krnl Code:#0000000000000000: 0000 illegal - >0000000000000002: 0000 illegal - 0000000000000004: 0000 illegal - 0000000000000006: 0000 illegal - 0000000000000008: 0000 illegal - 000000000000000a: 0000 illegal - 000000000000000c: 0000 illegal - 000000000000000e: 0000 illegal + [556585.270959] Krnl Code:#0000000000000000: 0000 illegal + >0000000000000002: 0000 illegal + 0000000000000004: 0000 illegal + 0000000000000006: 0000 illegal + 0000000000000008: 0000 illegal + 000000000000000a: 0000 illegal + 000000000000000c: 0000 illegal + 000000000000000e: 0000 illegal [556585.270967] Call Trace: [556585.270982] ([<000003ff80d6fb1a>] rpcauth_lookup_credcache+0x5a/0x300 [sunrpc]) - [556585.270993] [<000003ff80e1182c>] nfs_ctx_key_to_expire+0xec/0x130 [nfs] - [556585.271004] [<000003ff80e1189c>] nfs_key_timeout_notify+0x2c/0x70 [nfs] - [556585.271014] [<000003ff80dfdf7e>] nfs_file_write+0x3e/0x320 [nfs] - [556585.271016] [<00000028165944a8>] new_sync_write+0x118/0x1b0 - [556585.271017] [<0000002816594ee0>] vfs_write+0xb0/0x1b0 - [556585.271019] [<0000002816596a1e>] ksys_pwrite64+0x7e/0xc0 - [556585.271021] [<0000002816bb26b2>] system_call+0x2a6/0x2c8 - + [556585.270993] [<000003ff80e1182c>] nfs_ctx_key_to_expire+0xec/0x130 [nfs] + [556585.271004] [<000003ff80e1189c>] nfs_key_timeout_notify+0x2c/0x70 [nfs] + [556585.271014] [<000003ff80dfdf7e>] nfs_file_write+0x3e/0x320 [nfs] + [556585.271016] [<00000028165944a8>] new_sync_write+0x118/0x1b0 + [556585.271017] [<0000002816594ee0>] vfs_write+0xb0/0x1b0 + [556585.271019] [<0000002816596a1e>] ksys_pwrite64+0x7e/0xc0 + [556585.271021] [<0000002816bb26b2>] system_call+0x2a6/0x2c8 - Was a system dump produced ie kdump, netdumpmp, or LKCD? Yes That is the kdump where the stacktrace from. - Enter data below to accurately describe the problem: - Problem description: Null Pointer issue in nfs code running Ubuntu Ubuntu 18.04 with HWE kernel 5.4 on IBM Z - - - Enter uname -a output: - @lon1-qz1-sr4-rk101-s04> uname -a + + - Enter uname -a output: + @lon1-qz1-sr4-rk101-s04> uname -a Linux lon1-qz1-sr4-rk101-s04 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 19:59:45 UTC 2021 s390x s390x s390x GNU/Linux - - - Enter failing machine type and model (ie p520 9111-520 lpar, x336 47U-8637): + - Enter failing machine type and model (ie p520 9111-520 lpar, x336 47U-8637): Manufacturer: IBM Type: 8562 Model: A00 GT2 Model Capacity: A00 00000000 Capacity Adj. Ind.: 100 LPAR CPUs Total: 16 LPAR CPUs Configured: 16 LPAR CPUs Standby: 0 LPAR CPUs Reserved: 0 LPAR CPUs Dedicated: 0 LPAR CPUs Shared: 16 LPAR CPUs G-MTID: 0 LPAR CPUs S-MTID: 1 LPAR CPUs PS-MTID: 1 - - - Enter primary and backup contact information (name/email): + - Enter primary and backup contact information (name/email): Prabhat Ranjan pranj...@in.ibm.com Christoph Schlameu? schlame...@de.ibm.com - Detail the configuration of the additonal hardware - - - Enter common userspace tool name: N/A - - - - Enter name of userspace RPM: N/A - + - Enter common userspace tool name: N/A + + - Enter name of userspace RPM: N/A - If failing tool is obtained from project website vs RPM install, what is the version/release/mod. - If from the project's CVS, what is the branch tag and date of checkout (put "na" if not applicable)? + If from the project's CVS, what is the branch tag and date of checkout (put "na" if not applicable)? N/A - - - Is the failing userspace tool 32-bit, 64-bit, or both? N/A - - - - Describe how unresponsive the system is. What steps have you taken to reclaim the system: + - Is the failing userspace tool 32-bit, 64-bit, or both? N/A + + - Describe how unresponsive the system is. What steps have you taken to reclaim the system: kernel oops was detected and automatically dumped and restarted - - - Is a debugger configured (xmon or kdb enabled)? No - - - - Enter Oops message from console: - [556585.270902] illegal operation: 0001 ilc:1 [#10] SMP + - Is a debugger configured (xmon or kdb enabled)? No + + - Enter Oops message from console: + [556585.270902] illegal operation: 0001 ilc:1 [#10] SMP [556585.270905] Modules linked in: vhost_net macvtap macvlan tap rpcsec_gss_krb5 auth_rpcgss nfsv3 nfs_acl nfs lockd grace fscache veth xt_statistic ipt_REJECT nf_reject_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs iptable_mangle xt_mark sunrpc nf_log_ipv6 nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_set ip_set_hash_net ip_set_hash_ip ip_set tcp_diag inet_diag xt_comment xt_nat cls_cgroup sch_htb act_gact sch_multiq act_mirred act_pedit act_tunnel_key cls_flower act_police cls_u32 vxlan ip6_udp_tunnel udp_tunnel dummy nf_tables ebtable_filter ebtables xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key sch_ingress mlx5_ib ib_uverbs ib_core mlx5_core tls mlxfw ptp pps_core dm_integrity async_xor async_tx dm_bufio bonding xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_nat nf_nat br_netfilter bridge vhost_vsock vmw_vsock_virtio_transport_common vhost vsock 8021q garp mrp stp llc xt_multiport xt_tcpudp qeth_l2 lcs ctcm fsm dasd_fba_mod aufs overlay scsi_dh_rdac [556585.270923] scsi_dh_emc s390_trng xt_state xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter bpfilter sch_fq_codel zFPC_proc(OE) zFPC_diag(OE) vfio_ap vfio_mdev drm vfio_iommu_type1 drm_panel_orientation_quirks i2c_core ip_tables x_tables scsi_dh_alua pkey zcrypt ghash_s390 prng aes_s390 des_s390 libdes sha3_512_s390 sha3_256_s390 sha512_s390 sha256_s390 sha1_s390 sha_common chsc_sch qeth ccwgroup eadm_sch vfio_ccw mdev vfio btrfs libcrc32c crc32_vx_s390 xor zstd_compress raid6_pq dm_crypt virtio_blk dm_service_time dm_multipath zfcp scsi_transport_fc qdio dasd_eckd_mod dasd_mod zlib_deflate [last unloaded: tls] [556585.270945] CPU: 28 PID: 217741 Comm: worker Kdump: loaded Tainted: G D OE 5.4.0-90-generic #101-Ubuntu [556585.270947] Hardware name: IBM 8562 GT2 A00 (LPAR) [556585.270948] Krnl PSW : 0704d00180000000 0000000000000002 (0x2) [556585.270951] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3 [556585.270953] Krnl GPRS: 0000000000000000 0000000000000000 000003e010ebbcf8 00000071c45e1ec0 [556585.270954] 0000000000000000 0000002816f7b18c 00000078dd36a4a0 000000713a62f718 [556585.270955] 0000000000000000 000003e010ebbcf8 0000000000000068 00000071c45e1ec0 [556585.270957] 0000006090a12200 0000000000000c40 000003ff80d6fb54 000003e010ebbbf0 - [556585.270959] Krnl Code:#0000000000000000: 0000 illegal - >0000000000000002: 0000 illegal - 0000000000000004: 0000 illegal - 0000000000000006: 0000 illegal - 0000000000000008: 0000 illegal - 000000000000000a: 0000 illegal - 000000000000000c: 0000 illegal - 000000000000000e: 0000 illegal + [556585.270959] Krnl Code:#0000000000000000: 0000 illegal + >0000000000000002: 0000 illegal + 0000000000000004: 0000 illegal + 0000000000000006: 0000 illegal + 0000000000000008: 0000 illegal + 000000000000000a: 0000 illegal + 000000000000000c: 0000 illegal + 000000000000000e: 0000 illegal [556585.270967] Call Trace: [556585.270982] ([<000003ff80d6fb1a>] rpcauth_lookup_credcache+0x5a/0x300 [sunrpc]) - [556585.270993] [<000003ff80e1182c>] nfs_ctx_key_to_expire+0xec/0x130 [nfs] - [556585.271004] [<000003ff80e1189c>] nfs_key_timeout_notify+0x2c/0x70 [nfs] - [556585.271014] [<000003ff80dfdf7e>] nfs_file_write+0x3e/0x320 [nfs] - [556585.271016] [<00000028165944a8>] new_sync_write+0x118/0x1b0 - [556585.271017] [<0000002816594ee0>] vfs_write+0xb0/0x1b0 - [556585.271019] [<0000002816596a1e>] ksys_pwrite64+0x7e/0xc0 - [556585.271021] [<0000002816bb26b2>] system_call+0x2a6/0x2c8 - - - - Detail the steps to reproduce this problem: unknown - - - - Was the system configured to capture a system dump? Yes + [556585.270993] [<000003ff80e1182c>] nfs_ctx_key_to_expire+0xec/0x130 [nfs] + [556585.271004] [<000003ff80e1189c>] nfs_key_timeout_notify+0x2c/0x70 [nfs] + [556585.271014] [<000003ff80dfdf7e>] nfs_file_write+0x3e/0x320 [nfs] + [556585.271016] [<00000028165944a8>] new_sync_write+0x118/0x1b0 + [556585.271017] [<0000002816594ee0>] vfs_write+0xb0/0x1b0 + [556585.271019] [<0000002816596a1e>] ksys_pwrite64+0x7e/0xc0 + [556585.271021] [<0000002816bb26b2>] system_call+0x2a6/0x2c8 + + - Detail the steps to reproduce this problem: unknown + + - Was the system configured to capture a system dump? Yes
-- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1968096 Title: [UBUNTU 20.04] Null Pointer issue in nfs code running Ubuntu on IBM Z Status in Ubuntu on IBM z Systems: In Progress Status in linux package in Ubuntu: New Status in linux source package in Focal: New Status in linux source package in Impish: New Status in linux source package in Jammy: New Bug description: SRU Justification: ================== [Impact] * The kernel crashed under load with a null pointer issue in nfs code: [556585.270959] Krnl Code:#0000000000000000: 0000 illegal >0000000000000002: 0000 illegal 0000000000000004: 0000 illegal 0000000000000006: 0000 illegal 0000000000000008: 0000 illegal 000000000000000a: 0000 illegal 000000000000000c: 0000 illegal 000000000000000e: 0000 illegal [556585.270967] Call Trace: [556585.270982] ([<000003ff80d6fb1a>] rpcauth_lookup_credcache+0x5a/0x300 [sunrpc]) [556585.270993] [<000003ff80e1182c>] nfs_ctx_key_to_expire+0xec/0x130 [nfs] [556585.271004] [<000003ff80e1189c>] nfs_key_timeout_notify+0x2c/0x70 [nfs] [556585.271014] [<000003ff80dfdf7e>] nfs_file_write+0x3e/0x320 [nfs] [556585.271016] [<00000028165944a8>] new_sync_write+0x118/0x1b0 [556585.271017] [<0000002816594ee0>] vfs_write+0xb0/0x1b0 [556585.271019] [<0000002816596a1e>] ksys_pwrite64+0x7e/0xc0 [556585.271021] [<0000002816bb26b2>] system_call+0x2a6/0x2c8 * Several dumps were generated and shared with Canonical. * Analysis (done by kernel and SEG) point to refcount leaks fixed, that are already fixed in the following commit/fix: [Fix] * ca05cbae2a0468e5d78e9b4605936a8bf5da328b ca05cbae2a04 "NFS: Fix up nfs_ctx_key_to_expire()" [Test Case] * There is unfortunately no reproducer or trigger available for this issue. * It just happens now and then under higher load. * A patched kernel (focal 5.4 and bionic 5.4-hwe) were created and ran for more than a week in a special staging environment (at IBM) without further crashes. * Hence the test and verification will be done by the IBM Z team. [Where problems could occur] * The inode handling can become broken, in case the changes on the pointers are erroneous. * Problems with the authentication and/or the credentials could occur due to the modifications in put_rpccred, rpc_cred and rpc_auth. * The expiration of the cached credentials could be harmed as well, due to the changes in nfs_ctx_key_to_expire. * The different pointer arithmetic may cause further issues - wrong or null pointer references. * Positive is that the original commit was brought upstream by nfs experts. * A patched test kernel sustained day long runs under load in a staging and test environment. * The author of the upstream commit/patch is well known in the NFS area. [Other] * The Salesforce Case Number 00334334 is associated with this bug. * Commit ca05cbae2a04 was upstream accepted with 5.16-rc1. * But commit ca05cbae2a04 was unfortunately not tagged as stable, hence it was not picked automatically. * Since kinetic's (22.10) target kernel is 5.18, it will have the patch included, hence no dedicated PATCH request for kinetic. __________ State the component where the Bug is occurring: kernel Indicate the nature of the problem by answering the below questions: - Is this problem reproducible? No No, steps unknown, but we have seen these before - Is the system sitting at a debugger (kdb, or xmon)? No - Is the system hung? No No, dumped and rebooted - Are there any custom patches installed? Yes On base system level (CloudAppliance) we are still running with the zfpc_proc module loaded. But no recent changes in the module and is running absolutely stable in HA (same kernel and userspace, Ubuntu 20.04 LTS) - Is there any special hardware that may be relevant to this problem? Yes We are running with mlx (cloud network adapters) installed. - Is access information for the machine the problem was found on available? Yes - Is the bug occuring in a userspace application? No - Was a stack trace produced? Yes This is what mention in first comment by @Boris Barth - Did the system produce an Oops message on the console? Yes [556585.270902] illegal operation: 0001 ilc:1 [#10] SMP [556585.270905] Modules linked in: vhost_net macvtap macvlan tap rpcsec_gss_krb5 auth_rpcgss nfsv3 nfs_acl nfs lockd grace fscache veth xt_statistic ipt_REJECT nf_reject_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs iptable_mangle xt_mark sunrpc nf_log_ipv6 nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_set ip_set_hash_net ip_set_hash_ip ip_set tcp_diag inet_diag xt_comment xt_nat cls_cgroup sch_htb act_gact sch_multiq act_mirred act_pedit act_tunnel_key cls_flower act_police cls_u32 vxlan ip6_udp_tunnel udp_tunnel dummy nf_tables ebtable_filter ebtables xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key sch_ingress mlx5_ib ib_uverbs ib_core mlx5_core tls mlxfw ptp pps_core dm_integrity async_xor async_tx dm_bufio bonding xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_nat nf_nat br_netfilter bridge vhost_vsock vmw_vsock_virtio_transport_common vhost vsock 8021q garp mrp stp llc xt_multiport xt_tcpudp qeth_l2 lcs ctcm fsm dasd_fba_mod aufs overlay scsi_dh_rdac [556585.270923] scsi_dh_emc s390_trng xt_state xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter bpfilter sch_fq_codel zFPC_proc(OE) zFPC_diag(OE) vfio_ap vfio_mdev drm vfio_iommu_type1 drm_panel_orientation_quirks i2c_core ip_tables x_tables scsi_dh_alua pkey zcrypt ghash_s390 prng aes_s390 des_s390 libdes sha3_512_s390 sha3_256_s390 sha512_s390 sha256_s390 sha1_s390 sha_common chsc_sch qeth ccwgroup eadm_sch vfio_ccw mdev vfio btrfs libcrc32c crc32_vx_s390 xor zstd_compress raid6_pq dm_crypt virtio_blk dm_service_time dm_multipath zfcp scsi_transport_fc qdio dasd_eckd_mod dasd_mod zlib_deflate [last unloaded: tls] [556585.270945] CPU: 28 PID: 217741 Comm: worker Kdump: loaded Tainted: G D OE 5.4.0-90-generic #101-Ubuntu [556585.270947] Hardware name: IBM 8562 GT2 A00 (LPAR) [556585.270948] Krnl PSW : 0704d00180000000 0000000000000002 (0x2) [556585.270951] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3 [556585.270953] Krnl GPRS: 0000000000000000 0000000000000000 000003e010ebbcf8 00000071c45e1ec0 [556585.270954] 0000000000000000 0000002816f7b18c 00000078dd36a4a0 000000713a62f718 [556585.270955] 0000000000000000 000003e010ebbcf8 0000000000000068 00000071c45e1ec0 [556585.270957] 0000006090a12200 0000000000000c40 000003ff80d6fb54 000003e010ebbbf0 [556585.270959] Krnl Code:#0000000000000000: 0000 illegal >0000000000000002: 0000 illegal 0000000000000004: 0000 illegal 0000000000000006: 0000 illegal 0000000000000008: 0000 illegal 000000000000000a: 0000 illegal 000000000000000c: 0000 illegal 000000000000000e: 0000 illegal [556585.270967] Call Trace: [556585.270982] ([<000003ff80d6fb1a>] rpcauth_lookup_credcache+0x5a/0x300 [sunrpc]) [556585.270993] [<000003ff80e1182c>] nfs_ctx_key_to_expire+0xec/0x130 [nfs] [556585.271004] [<000003ff80e1189c>] nfs_key_timeout_notify+0x2c/0x70 [nfs] [556585.271014] [<000003ff80dfdf7e>] nfs_file_write+0x3e/0x320 [nfs] [556585.271016] [<00000028165944a8>] new_sync_write+0x118/0x1b0 [556585.271017] [<0000002816594ee0>] vfs_write+0xb0/0x1b0 [556585.271019] [<0000002816596a1e>] ksys_pwrite64+0x7e/0xc0 [556585.271021] [<0000002816bb26b2>] system_call+0x2a6/0x2c8 - Was a system dump produced ie kdump, netdumpmp, or LKCD? Yes That is the kdump where the stacktrace from. Enter data below to accurately describe the problem: - Problem description: Null Pointer issue in nfs code running Ubuntu Ubuntu 18.04 with HWE kernel 5.4 on IBM Z - Enter uname -a output: @lon1-qz1-sr4-rk101-s04> uname -a Linux lon1-qz1-sr4-rk101-s04 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 19:59:45 UTC 2021 s390x s390x s390x GNU/Linux - Enter failing machine type and model (ie p520 9111-520 lpar, x336 47U-8637): Manufacturer: IBM Type: 8562 Model: A00 GT2 Model Capacity: A00 00000000 Capacity Adj. Ind.: 100 LPAR CPUs Total: 16 LPAR CPUs Configured: 16 LPAR CPUs Standby: 0 LPAR CPUs Reserved: 0 LPAR CPUs Dedicated: 0 LPAR CPUs Shared: 16 LPAR CPUs G-MTID: 0 LPAR CPUs S-MTID: 1 LPAR CPUs PS-MTID: 1 - Enter primary and backup contact information (name/email): Prabhat Ranjan pranj...@in.ibm.com Christoph Schlameu? schlame...@de.ibm.com - Detail the configuration of the additonal hardware - Enter common userspace tool name: N/A - Enter name of userspace RPM: N/A - If failing tool is obtained from project website vs RPM install, what is the version/release/mod. If from the project's CVS, what is the branch tag and date of checkout (put "na" if not applicable)? N/A - Is the failing userspace tool 32-bit, 64-bit, or both? N/A - Describe how unresponsive the system is. What steps have you taken to reclaim the system: kernel oops was detected and automatically dumped and restarted - Is a debugger configured (xmon or kdb enabled)? No - Enter Oops message from console: [556585.270902] illegal operation: 0001 ilc:1 [#10] SMP [556585.270905] Modules linked in: vhost_net macvtap macvlan tap rpcsec_gss_krb5 auth_rpcgss nfsv3 nfs_acl nfs lockd grace fscache veth xt_statistic ipt_REJECT nf_reject_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs iptable_mangle xt_mark sunrpc nf_log_ipv6 nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_set ip_set_hash_net ip_set_hash_ip ip_set tcp_diag inet_diag xt_comment xt_nat cls_cgroup sch_htb act_gact sch_multiq act_mirred act_pedit act_tunnel_key cls_flower act_police cls_u32 vxlan ip6_udp_tunnel udp_tunnel dummy nf_tables ebtable_filter ebtables xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key sch_ingress mlx5_ib ib_uverbs ib_core mlx5_core tls mlxfw ptp pps_core dm_integrity async_xor async_tx dm_bufio bonding xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_nat nf_nat br_netfilter bridge vhost_vsock vmw_vsock_virtio_transport_common vhost vsock 8021q garp mrp stp llc xt_multiport xt_tcpudp qeth_l2 lcs ctcm fsm dasd_fba_mod aufs overlay scsi_dh_rdac [556585.270923] scsi_dh_emc s390_trng xt_state xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter bpfilter sch_fq_codel zFPC_proc(OE) zFPC_diag(OE) vfio_ap vfio_mdev drm vfio_iommu_type1 drm_panel_orientation_quirks i2c_core ip_tables x_tables scsi_dh_alua pkey zcrypt ghash_s390 prng aes_s390 des_s390 libdes sha3_512_s390 sha3_256_s390 sha512_s390 sha256_s390 sha1_s390 sha_common chsc_sch qeth ccwgroup eadm_sch vfio_ccw mdev vfio btrfs libcrc32c crc32_vx_s390 xor zstd_compress raid6_pq dm_crypt virtio_blk dm_service_time dm_multipath zfcp scsi_transport_fc qdio dasd_eckd_mod dasd_mod zlib_deflate [last unloaded: tls] [556585.270945] CPU: 28 PID: 217741 Comm: worker Kdump: loaded Tainted: G D OE 5.4.0-90-generic #101-Ubuntu [556585.270947] Hardware name: IBM 8562 GT2 A00 (LPAR) [556585.270948] Krnl PSW : 0704d00180000000 0000000000000002 (0x2) [556585.270951] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3 [556585.270953] Krnl GPRS: 0000000000000000 0000000000000000 000003e010ebbcf8 00000071c45e1ec0 [556585.270954] 0000000000000000 0000002816f7b18c 00000078dd36a4a0 000000713a62f718 [556585.270955] 0000000000000000 000003e010ebbcf8 0000000000000068 00000071c45e1ec0 [556585.270957] 0000006090a12200 0000000000000c40 000003ff80d6fb54 000003e010ebbbf0 [556585.270959] Krnl Code:#0000000000000000: 0000 illegal >0000000000000002: 0000 illegal 0000000000000004: 0000 illegal 0000000000000006: 0000 illegal 0000000000000008: 0000 illegal 000000000000000a: 0000 illegal 000000000000000c: 0000 illegal 000000000000000e: 0000 illegal [556585.270967] Call Trace: [556585.270982] ([<000003ff80d6fb1a>] rpcauth_lookup_credcache+0x5a/0x300 [sunrpc]) [556585.270993] [<000003ff80e1182c>] nfs_ctx_key_to_expire+0xec/0x130 [nfs] [556585.271004] [<000003ff80e1189c>] nfs_key_timeout_notify+0x2c/0x70 [nfs] [556585.271014] [<000003ff80dfdf7e>] nfs_file_write+0x3e/0x320 [nfs] [556585.271016] [<00000028165944a8>] new_sync_write+0x118/0x1b0 [556585.271017] [<0000002816594ee0>] vfs_write+0xb0/0x1b0 [556585.271019] [<0000002816596a1e>] ksys_pwrite64+0x7e/0xc0 [556585.271021] [<0000002816bb26b2>] system_call+0x2a6/0x2c8 - Detail the steps to reproduce this problem: unknown - Was the system configured to capture a system dump? Yes To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1968096/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
