Hello, On Tue, Jun 06, 2017 at 11:18:36AM -0500, Michael Bringmann wrote: > On 05/25/2017 10:30 AM, Michael Bringmann wrote: > > I will try that patch shortly. I also updated my patch to be conditional > > on whether the pool's cpumask attribute was empty. You should have received > > V2 of that patch by now. > > Let's try this again. > > The hotplug problem goes away with the changes that you provided earlier, and
So, that means we're ending up in situations where NUMA online is a proper superset of NUMA possible. > shown in the patch below. I kept this change to get_unbound_pool' as a just > in case to explain the crash in the event that it occurs again: > > if (!cpumask_weight(pool->attrs->cpumask)) > cpumask_copy(pool->attrs->cpumask, cpumask_of(smp_processor_id())); > > I could also insert > > BUG(!cpumask_weight(pool->attrs->cpumask, cpumask_of(smp_processor_id())); > > at that place, but I really prefer not to crash the system if there is a > workaround. I'm not sure because it doesn't make any logical sense and it's not right in terms of correctness. The above would be able to enable CPUs which are explicitly excluded from a workqueue. The only fallback which makes sense is falling back to the default pwq. > > Can you please post the messages with the debug patch from the prev > > thread? In fact, let's please continue on that thread. I'm having a > > hard time following what's going wrong with the code. > > Are these the failure logs that you requested? > > > Red Hat Enterprise Linux Server 7.3 (Maipo) > Kernel 4.12.0-rc1.wi91275_debug_03.ppc64le+ on an ppc64le > > ltcalpine2-lp20 login: root > Password: > Last login: Wed May 24 18:45:40 from oc1554177480.austin.ibm.com > [root@ltcalpine2-lp20 ~]# numactl -H > available: 2 nodes (0,6) > node 0 cpus: > node 0 size: 0 MB > node 0 free: 0 MB > node 6 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 > 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 > 51 52 53 54 55 56 57 58 59 60 61 62 63 > node 6 size: 19858 MB > node 6 free: 16920 MB > node distances: > node 0 6 > 0: 10 40 > 6: 40 10 > [root@ltcalpine2-lp20 ~]# numactl -H > available: 2 nodes (0,6) > node 0 cpus: > node 0 size: 0 MB > node 0 free: 0 MB > node 6 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 > 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 > 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 > 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 > 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 > 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 > 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 > 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 > 178 179 180 181 182 183 184 185 186 187 188 189 190 191 > node 6 size: 19858 MB > node 6 free: 16362 MB > node distances: > node 0 6 > 0: 10 40 > 6: 40 10 > [root@ltcalpine2-lp20 ~]# [ 321.310943] workqueue:get_unbound_pool has empty > cpumask for pool attrs > [ 321.310961] ------------[ cut here ]------------ > [ 321.310997] WARNING: CPU: 184 PID: 13201 at kernel/workqueue.c:3375 > alloc_unbound_pwq+0x5c0/0x5e0 > [ 321.311005] Modules linked in: rpadlpar_io rpaphp dccp_diag dccp tcp_diag > udp_diag inet_diag unix_diag af_packet_diag netlink_diag sg pseries_rng > ghash_generic gf128mul xts vmx_crypto binfmt_misc ip_tables xfs libcrc32c > sd_mod ibmvscsi ibmveth scsi_transport_srp dm_mirror dm_region_hash dm_log > dm_mod > [ 321.311097] CPU: 184 PID: 13201 Comm: cpuhp/184 Not tainted > 4.12.0-rc1.wi91275_debug_03.ppc64le+ #8 > [ 321.311106] task: c000000408961080 task.stack: c000000406394000 > [ 321.311113] NIP: c000000000116c80 LR: c000000000116c7c CTR: > 0000000000000000 > [ 321.311121] REGS: c0000004063977b0 TRAP: 0700 Not tainted > (4.12.0-rc1.wi91275_debug_03.ppc64le+) > [ 321.311128] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> > [ 321.311150] CR: 28000082 XER: 00000000 > [ 321.311159] CFAR: c000000000a2dc80 SOFTE: 1 > [ 321.311159] GPR00: c000000000116c7c c000000406397a30 c0000000013ae900 > 000000000000003b > [ 321.311159] GPR04: c000000408961a38 0000000000000006 00000000a49e41e5 > ffffffffa4a5a483 > [ 321.311159] GPR08: 00000000000062cc 0000000000000000 0000000000000000 > c000000408961a38 > [ 321.311159] GPR12: 0000000000000000 c00000000fb38c00 c00000000011e858 > c00000040a902ac0 > [ 321.311159] GPR16: 0000000000000000 0000000000000000 0000000000000000 > 0000000000000000 > [ 321.311159] GPR20: c000000406394000 0000000000000002 c000000406394000 > 0000000000000000 > [ 321.311159] GPR24: c000000405075400 c000000404fc0000 0000000000000110 > c0000000015a4c88 > [ 321.311159] GPR28: 0000000000000000 c0000004fe256000 c0000004fe256008 > c0000004fe052800 > [ 321.311290] NIP [c000000000116c80] alloc_unbound_pwq+0x5c0/0x5e0 > [ 321.311298] LR [c000000000116c7c] alloc_unbound_pwq+0x5bc/0x5e0 > [ 321.311305] Call Trace: > [ 321.311310] [c000000406397a30] [c000000000116c7c] > alloc_unbound_pwq+0x5bc/0x5e0 (unreliable) > [ 321.311323] [c000000406397ad0] [c000000000116e30] > wq_update_unbound_numa+0x190/0x270 > [ 321.311334] [c000000406397b60] [c000000000118eb0] > workqueue_offline_cpu+0xe0/0x130 > [ 321.311345] [c000000406397bf0] [c0000000000e9f20] > cpuhp_invoke_callback+0x240/0xcd0 > [ 321.311355] [c000000406397cb0] [c0000000000eab28] > cpuhp_down_callbacks+0x78/0xf0 > [ 321.311365] [c000000406397d00] [c0000000000eae6c] > cpuhp_thread_fun+0x18c/0x1a0 > [ 321.311376] [c000000406397d30] [c0000000001251cc] > smpboot_thread_fn+0x2fc/0x3b0 > [ 321.311386] [c000000406397dc0] [c00000000011e9c0] kthread+0x170/0x1b0 > [ 321.311397] [c000000406397e30] [c00000000000b4f4] > ret_from_kernel_thread+0x5c/0x68 > [ 321.311406] Instruction dump: > [ 321.311413] 3d42fff0 892ac565 2f890000 40fefd98 39200001 3c62ff89 3c82ff6c > 3863d590 > [ 321.311437] 38847cb0 992ac565 48916fc9 60000000 <0fe00000> 4bfffd70 > 60000000 60420000 The only way offlining can lead to this failure is when wq numa possible cpu mask is a proper subset of the matching online mask. Can you please print out the numa online cpu and wq_numa_possible_cpumask masks and verify that online stays within the possible for each node? If not, the ppc arch init code needs to be updated so that cpu <-> node binding is establish for all possible cpus on boot. Note that this isn't a requirement coming solely from wq. All node affine (thus percpu) allocations depend on that. Thanks. -- tejun
