Re: [RFC PATCH] MIPS: Oprofile: Drop support
Hi Robert, On 04/05/18 13:27, Robert Richter wrote: On 04.05.18 12:03:12, Matt Redfearn wrote: As said, oprofile version 0.9.x is still available for cpus that do not support perf. What is the breakage? The breakage I originally set out to fix was the MT support in perf. https://www.linux-mips.org/archives/linux-mips/2018-04/msg00259.html Since the perf code shares so much copied code from oprofile, those same issues exist in oprofile and ought to be addressed. But as newer oprofile userspace does not use the (MIPS) kernel oprofile code, then we could, perhaps, just remove it (as per the RFC). That would break legacy tools (0.9.x) though... Those support perf: (CPU_MIPS32 || CPU_MIPS64 || CPU_R1 || CPU_SB1 || CPU_CAVIUM_OCTEON || CPU_XLP || CPU_LOONGSON3) Here is the total list of CPU_*: $ git grep -h config.CPU_ arch/mips/ | sort -u | wc -l 79 To be fair, that list for oprofile is not much different: arch/mips/oprofile/Makefile: oprofile-$(CONFIG_CPU_MIPS32) += op_model_mipsxx.o oprofile-$(CONFIG_CPU_MIPS64) += op_model_mipsxx.o oprofile-$(CONFIG_CPU_R1) += op_model_mipsxx.o oprofile-$(CONFIG_CPU_SB1) += op_model_mipsxx.o oprofile-$(CONFIG_CPU_XLR) += op_model_mipsxx.o oprofile-$(CONFIG_CPU_LOONGSON2)+= op_model_loongson2.o oprofile-$(CONFIG_CPU_LOONGSON3)+= op_model_loongson3.o However, since those are generally CPU families rather than individual CPUs, the number of models supported by each framework tells a different story: git grep -h ops.cpu_type arch/mips/oprofile | wc -l 20 git grep -h pmu.name arch/mips/kernel/perf_event* | wc -l 17 The difference is mainly older CPUs - M14Kc, 20K, loongson1, etc. But yes you are right dropping it would kill profiling for them - that being the case I guess oprofile should remain and instead just remove support for the MT capable CPUs (34K, interAptiv) which are all supported by perf. Thanks, Matt The comparisation might not be accurate, but at least gives a hint that there are many cpus not supporting perf. You would drop profiling support at al to them. If it is too hard to also fix the oprofile code (code duplication seems the main issue here), then it would be also ok to blacklist newer cpus to enable oprofile kernel code (where it is broken). -Robert
Re: [RFC PATCH] MIPS: Oprofile: Drop support
On Fri, May 04, 2018 at 02:27:51PM +0200, Robert Richter wrote: > On 04.05.18 12:03:12, Matt Redfearn wrote: > > >As said, oprofile version 0.9.x is still available for cpus that do > > >not support perf. What is the breakage? > > > > The breakage I originally set out to fix was the MT support in perf. > > https://www.linux-mips.org/archives/linux-mips/2018-04/msg00259.html > > > > Since the perf code shares so much copied code from oprofile, those same > > issues exist in oprofile and ought to be addressed. But as newer oprofile > > userspace does not use the (MIPS) kernel oprofile code, then we could, > > perhaps, just remove it (as per the RFC). That would break legacy tools > > (0.9.x) though... > > Those support perf: > > (CPU_MIPS32 || CPU_MIPS64 || CPU_R1 || CPU_SB1 || CPU_CAVIUM_OCTEON || > CPU_XLP || CPU_LOONGSON3) > > Here is the total list of CPU_*: > > $ git grep -h config.CPU_ arch/mips/ | sort -u | wc -l > 79 > > The comparisation might not be accurate, but at least gives a hint > that there are many cpus not supporting perf. You would drop profiling > support at al to them. The grep results are a bit missleading. We have many symbols such as CPU_MIPS32_R1 which describe a particular architecture revision supported by a CPU. Others CPU_* symbols are describing particular features or options so in reality there are far fewer CPU to be supported. Also many of the older processors or embedded cores don't have performance counters at all; for yet other cores presence, number and a few other details can be configured when synthesizing the RTL so relying on the CONFIG_* alone won't help. > If it is too hard to also fix the oprofile code (code duplication > seems the main issue here), then it would be also ok to blacklist > newer cpus to enable oprofile kernel code (where it is broken). Ralf
Re: [RFC PATCH] MIPS: Oprofile: Drop support
On 04.05.18 12:03:12, Matt Redfearn wrote: > >As said, oprofile version 0.9.x is still available for cpus that do > >not support perf. What is the breakage? > > The breakage I originally set out to fix was the MT support in perf. > https://www.linux-mips.org/archives/linux-mips/2018-04/msg00259.html > > Since the perf code shares so much copied code from oprofile, those same > issues exist in oprofile and ought to be addressed. But as newer oprofile > userspace does not use the (MIPS) kernel oprofile code, then we could, > perhaps, just remove it (as per the RFC). That would break legacy tools > (0.9.x) though... Those support perf: (CPU_MIPS32 || CPU_MIPS64 || CPU_R1 || CPU_SB1 || CPU_CAVIUM_OCTEON || CPU_XLP || CPU_LOONGSON3) Here is the total list of CPU_*: $ git grep -h config.CPU_ arch/mips/ | sort -u | wc -l 79 The comparisation might not be accurate, but at least gives a hint that there are many cpus not supporting perf. You would drop profiling support at al to them. If it is too hard to also fix the oprofile code (code duplication seems the main issue here), then it would be also ok to blacklist newer cpus to enable oprofile kernel code (where it is broken). -Robert
Re: [RFC PATCH] MIPS: Oprofile: Drop support
Hi Robert, On 04/05/18 11:26, Robert Richter wrote: On 04.05.18 10:54:32, Matt Redfearn wrote: perf is available for MIPS and supports many more CPU types than oprofile. oprofile userspace seemingly has been broken since 1.0.0 - removing oprofile support from the MIPS kernel would not break it more thatn it already is, What do you mean with "oprofile is broken"? It looks like you modified Kconfig to enable oprofile and perf in parallel, which is not intended to work. Have you tried a kernel with oprofile disabled and perf enabled? Oh I see what you mean - previously I was trying v1.1.0 of the userspace with a kernel that has perf disabled - and that did not work (I assumed, naively, that the kernel oprofile code was required to run the oprofile userspace). Thanks for the pointer - I confirmed that oprofile 1.1.0 userspace tools work with a kernel with "CONFIG_OPROFILE is not set", and "CONFIG_HW_PERF_EVENTS=y". As said, oprofile version 0.9.x is still available for cpus that do not support perf. What is the breakage? The breakage I originally set out to fix was the MT support in perf. https://www.linux-mips.org/archives/linux-mips/2018-04/msg00259.html Since the perf code shares so much copied code from oprofile, those same issues exist in oprofile and ought to be addressed. But as newer oprofile userspace does not use the (MIPS) kernel oprofile code, then we could, perhaps, just remove it (as per the RFC). That would break legacy tools (0.9.x) though... Thanks, Matt Thanks, -Robert
Re: [RFC PATCH] MIPS: Oprofile: Drop support
On 04.05.18 10:54:32, Matt Redfearn wrote: > perf is available for MIPS and supports many more CPU types than oprofile. > oprofile userspace seemingly has been broken since 1.0.0 - removing oprofile > support from the MIPS kernel would not break it more thatn it already is, What do you mean with "oprofile is broken"? It looks like you modified Kconfig to enable oprofile and perf in parallel, which is not intended to work. Have you tried a kernel with oprofile disabled and perf enabled? As said, oprofile version 0.9.x is still available for cpus that do not support perf. What is the breakage? Thanks, -Robert
Re: [RFC PATCH] MIPS: Oprofile: Drop support
Hi Robert, On 04/05/18 10:30, Robert Richter wrote: On 24.04.18 14:15:58, Matt Redfearn wrote: On 24/04/18 14:05, James Hogan wrote: On Tue, Apr 24, 2018 at 01:55:54PM +0100, Matt Redfearn wrote: Since it appears that MIPS oprofile support is currently broken, core oprofile is not getting many updates and not as many architectures implement support for it compared to perf, remove the MIPS support. That sounds reasonable to me. Any idea how long its been broken? Sorry, not yet. I haven't yet looked into where/how it's broken that would narrow that down... oprofile moved to perf syscall as kernel i/f with version 1.0.0. The OK interesting. I guess this was the point at which MIPS' current Kconfig rule which only allows building oprofile or perf into a kernel broke oprofile userspace. opcontrol script that was using the oprofile kernel i/f was removed: https://sourceforge.net/p/oprofile/oprofile/ci/0c142c3a096d3e9ec42cc9b0ddad994fea60d135/ Thus, cpus that do not support the perf syscall are no longer supported by 1.x releases. https://sourceforge.net/p/oprofile/oprofile/ci/797d01dea0b82dbbdb0c21112a3de75990e011d2/ For those remainings there is still version 0.9.x available (tagged PRE_RELEASE_1_0). I am undecided whether removing oprofile kernel i/f falls under the rule of "never break user space" here. Strictly seen, yes it breaks those remainings. So if the perf syscall is not available as an alternative, the oprofile kernel support shouldn't be removed. perf is available for MIPS and supports many more CPU types than oprofile. oprofile userspace seemingly has been broken since 1.0.0 - removing oprofile support from the MIPS kernel would not break it more thatn it already is, but of course it would be better to fix it - if it is still useful and people still use it. That is the question that I was looking for answers for with this RFC - whether to spend the time & effort to fix oprofile, or if it can be removed since everyone uses perf. Thanks, Matt -Robert
Re: [RFC PATCH] MIPS: Oprofile: Drop support
On 24.04.18 14:15:58, Matt Redfearn wrote: > On 24/04/18 14:05, James Hogan wrote: > >On Tue, Apr 24, 2018 at 01:55:54PM +0100, Matt Redfearn wrote: > >>Since it appears that MIPS oprofile support is currently broken, core > >>oprofile is not getting many updates and not as many architectures > >>implement support for it compared to perf, remove the MIPS support. > > > >That sounds reasonable to me. Any idea how long its been broken? > > Sorry, not yet. I haven't yet looked into where/how it's broken that would > narrow that down... oprofile moved to perf syscall as kernel i/f with version 1.0.0. The opcontrol script that was using the oprofile kernel i/f was removed: https://sourceforge.net/p/oprofile/oprofile/ci/0c142c3a096d3e9ec42cc9b0ddad994fea60d135/ Thus, cpus that do not support the perf syscall are no longer supported by 1.x releases. https://sourceforge.net/p/oprofile/oprofile/ci/797d01dea0b82dbbdb0c21112a3de75990e011d2/ For those remainings there is still version 0.9.x available (tagged PRE_RELEASE_1_0). I am undecided whether removing oprofile kernel i/f falls under the rule of "never break user space" here. Strictly seen, yes it breaks those remainings. So if the perf syscall is not available as an alternative, the oprofile kernel support shouldn't be removed. -Robert
Re: [RFC PATCH] MIPS: Oprofile: Drop support
On Tue, Apr 24, 2018 at 01:55:54PM +0100, Matt Redfearn wrote: > The core oprofile code in drivers/oprofile/ has not seeen significant > maintenance other than fixes to changes in other parts of the tree for > the last 5 years at least. It looks as through the perf tool has > more or less superceeded it's functionality. > Additionally the MIPS architecture support has bitrotted to an extent > meaning it is not currently functional. I wonder if time has come to drop oprofile support entirely. (in addition to your patch which seems more urgent)
Re: [RFC PATCH] MIPS: Oprofile: Drop support
On 24/04/18 14:05, James Hogan wrote: On Tue, Apr 24, 2018 at 01:55:54PM +0100, Matt Redfearn wrote: Since it appears that MIPS oprofile support is currently broken, core oprofile is not getting many updates and not as many architectures implement support for it compared to perf, remove the MIPS support. That sounds reasonable to me. Any idea how long its been broken? Sorry, not yet. I haven't yet looked into where/how it's broken that would narrow that down... The other thing to bear in mind is that the userspace tools have not seen any MIPS additions since 2010, the last commit being to add 1004K and 34K support: https://sourceforge.net/p/oprofile/oprofile/ci/master/tree/events/mips/ Though I'm not sure if anyone is maintaining a vendor specific fork containing further support. I'll let it sit on the list for a bit in case anybody does object and wants to fix it instead. Cool, sounds good. Thanks, Matt Thanks James
Re: [RFC PATCH] MIPS: Oprofile: Drop support
On Tue, Apr 24, 2018 at 01:55:54PM +0100, Matt Redfearn wrote: > Since it appears that MIPS oprofile support is currently broken, core > oprofile is not getting many updates and not as many architectures > implement support for it compared to perf, remove the MIPS support. That sounds reasonable to me. Any idea how long its been broken? I'll let it sit on the list for a bit in case anybody does object and wants to fix it instead. Thanks James signature.asc Description: Digital signature
[RFC PATCH] MIPS: Oprofile: Drop support
The core oprofile code in drivers/oprofile/ has not seeen significant maintenance other than fixes to changes in other parts of the tree for the last 5 years at least. It looks as through the perf tool has more or less superceeded it's functionality. Additionally the MIPS architecture support has bitrotted to an extent meaning it is not currently functional. For example the current Kconfig rule disallows HW_PERF from being enabled in a kernel supporting OPROFILE, but attempting to use oprofile on such a kernel results in: $ op-check-perfevents -v perf_event_open syscall returned No such file or directory Due to missing perf support If this dependency is removed such that oprofile can be started, very quickly an oops is triggered which appears to be a result of the oprofile buffer having not been set up correctly: $ operf: Profiler started [ 96.950415] CPU 1 Unable to handle kernel paging request at virtual address 0008, epc == 804fcb14, ra == 809b6424 [ 96.962266] Oops[#1]: [ 96.964821] CPU: 1 PID: 145 Comm: operf Not tainted 4.16.0+ #235 [ 96.971516] $ 0 : 809b6500 d179d1bb [ 96.977367] $ 4 : 000c 0001 8efab000 [ 96.983215] $ 8 : 80e0 007f 002887fc [ 96.989062] $12 : 8f495de4 8131c400 80e01000 [ 96.994916] $16 : 8f495dfc 80d1 8efab000 [ 97.000781] $20 : 80cf8aa0 8f495ef8 80d7 [ 97.006630] $24 : [ 97.012477] $28 : 8e936000 8f495d68 0001 809b6424 [ 97.018329] Hi: 00210c94 [ 97.021551] Lo: 4a54aef0 [ 97.024810] epc : 804fcb14 ring_buffer_lock_reserve+0x38/0x6fc [ 97.031526] ra: 809b6424 op_cpu_buffer_write_reserve+0x3c/0x78 [ 97.038417] Status: 1100fc02KERNEL EXL [ 97.042820] Cause : 04808008 (ExcCode 02) [ 97.047286] BadVA : 0008 [ 97.050503] PrId : 0001a120 (MIPS interAptiv (multi)) [ 97.056225] Modules linked in: [ 97.059667] Process operf (pid: 145, threadinfo=e9a0abb0, task=3ddccdc2, tls=77fcd490) [ 97.068484] Stack : 80e0 8132d400 0001 8efab000 8132d400 80d1 80d0de64 0001 [ 97.077836] 3973 8047763c 8293d343 80d7 8f495d98 d179d1bb 0001 8f495dfc [ 97.087190] 0001 0001 8efab000 80cf8aa0 8f495ef8 80d7 809b6424 [ 97.096542] 8293e5d2 0016 8293d343 80d7 804c9c38 000b 80d1 809b6500 [ 97.105897] 81329ae8 8e937a90 8293e5d2 0016 8293d343 81329ae8 81329ae8 804b4cf4 [ 97.115251] ... [ 97.117992] Call Trace: [ 97.120743] [<804fcb14>] ring_buffer_lock_reserve+0x38/0x6fc [ 97.127080] [<809b6424>] op_cpu_buffer_write_reserve+0x3c/0x78 [ 97.133604] [<809b6500>] op_add_code+0x80/0x114 [ 97.138675] [<809b6610>] log_sample+0x7c/0xf0 [ 97.143545] [<809b6928>] oprofile_add_sample+0x90/0xd4 [ 97.149292] [<809ba2f0>] mipsxx_perfcount_handler+0x408/0x474 [ 97.155728] [<80492c4c>] __handle_irq_event_percpu+0xbc/0x288 [ 97.162141] [<80492e58>] handle_irq_event_percpu+0x40/0x98 [ 97.168276] [<80498700>] handle_percpu_irq+0x98/0xc8 [ 97.173833] [<8049204c>] generic_handle_irq+0x38/0x48 [ 97.179504] [<807da298>] gic_handle_local_int+0xb0/0x108 [ 97.185438] [<807da498>] gic_irq_dispatch+0x20/0x30 [ 97.190887] [<8049204c>] generic_handle_irq+0x38/0x48 [ 97.196539] [<80b7e098>] do_IRQ+0x28/0x34 [ 97.201024] [<807d8ef0>] plat_irq_dispatch+0xf0/0x11c [ 97.206677] [<804060a8>] except_vec_vi_end+0xb8/0xc4 [ 97.212227] [<80b7dc2c>] _raw_spin_unlock_irq+0x28/0x34 [ 97.218067] [<80544428>] __add_to_page_cache_locked.part.44+0xe4/0x218 [ 97.225353] [<80544930>] add_to_page_cache_lru+0x88/0x1bc [ 97.231419] [<805563e0>] read_cache_pages+0x9c/0x1bc [ 97.236981] [<806d4b38>] nfs_readpages+0x124/0x234 [ 97.242344] [<805566e8>] __do_page_cache_readahead+0x1e8/0x344 [ 97.248877] [<80556e70>] page_cache_sync_readahead+0x68/0x74 [ 97.255199] [<80547354>] generic_file_read_iter+0x340/0xd50 [ 97.261449] [<806c5a9c>] nfs_file_read+0x84/0x120 [ 97.266726] [<805a8d28>] __vfs_read+0x144/0x198 [ 97.271801] [<805a8e34>] vfs_read+0xb8/0x144 [ 97.276589] [<805a8ee8>] kernel_read+0x28/0x3c [ 97.281585] [<805b00f4>] prepare_binprm+0x18c/0x1c4 [ 97.287042] [<805b21c8>] do_execveat_common+0x3fc/0x6d8 [ 97.292878] [<805b24dc>] do_execve+0x38/0x44 [ 97.297669] [<80415a58>] syscall_common+0x34/0x58 [ 97.302924] Code: afb0003c 8e22d4e0 afa20034 <8c820008> 14400170 24030020 8f82000c 26460010 00a09825 Since it appears that MIPS oprofile support is currently broken, core oprofile is not getting many updates and not as many architectures implement support for it compared to perf, remove the MIPS support. Signed-off-by: Matt Redfearn --- This is meant as an RFC to determine if people are still using oprofile. If they are, then let's get it fixed such that it works correctly and perhaps make the oprofile support more closely tie in with perf, sharing the code as much as pos
