Re: Regarding your thread on LKML - drm_radeon spamming alloc_contig_range [WAS: Re: PROBLEM-PERSISTS: dmesg spam: alloc_contig_range: [XX, YY) PFNs busy]
[CC Vlastimil, Joonsoo]
On Thu 29-06-17 17:47:05, Robin H. Johnson wrote:
> CC'd back to LKML.
>
> On Thu, Jun 29, 2017 at 06:11:00PM +0530, Kumar Abhishek wrote:
> > Hi Robin,
> >
> > I am an independent developer who stumbled upon your thread on the LKML
> > after facing a similar issue - my kernel log being spammed by
> > alloc_contig_range messages. I am running Linux on an ARM system
> > (specifically the BeagleBoard-X15) and am on kernel version 4.9.33 with TI
> > patches on top of it.
> >
> > I am running Debian Stretch (9.0) on the system.
> >
> > Here's what my stack trace looks like:
> ..
> >
> > It's somewhat similar to your stack trace, but this here happens on an
> > etnaviv GPU (Vivante GCxx).
> >
> > In my case if I do 'sudo service lightdm stop', these messages stop too.
> > This seems to suggest that the problem may be in the X server rather than
> > the kernel? I seem to think this because I replicated this on an entirely
> > different set of hardware than yours.
> >
> > I just wanted to bring this to your notice, and also ask you if you managed
> > to solve it for yourself.
> >
> > One solution could be to demote the pr_info in alloc_contig_range to
> > pr_debug or to do away with the message altogether, but this would be
> > suppressing the issue instead of really knowing what it is about.
> >
> > Let me know how I could further investigate this.
> The problem, as far as I got diagnosed on LKML, is that some of the GPUs
> have a bunch of non-fatal contiguous memory allocation requests: they
> have a meaningful fallback path on the allocation, so 'PFNs busy' is a
> false busy for their case.
Well, later on we found out that a change to the compaction has changed
a picture and 424f6c4818bb ("mm: alloc_contig: re-allow CMA to compact
FS pages") fixed that issue. It went to 4.10 and I do not see it in 4.9
stable tree. Maybe it can help in this case as well.
> However, if there was a another consumer that does NOT have a fallback,
> the output would still be crucially useful.
>
> Attached is the patch that I unsuccessfully proposed on LKML to
> rate-limit the messages, with the last revision to only dump_stack() if
> CONFIG_CMA_DEBUG was set.
The patch makes some sense to me in general. Try to repost it.
> The path that LKML wanted was to add a new parameter to suppress or at
> least demote the failure message, and update all of the callers: but it
> means that many of the indirect callers need that added parameter as
> well.
>
> mm/cma.c:cma_alloc this call can suppress the error, you can see it retry.
> mm/hugetlb.c: These callers should get the error message.
>
> The error message DOES still have a good general use in notifying you
> that something is going wrong. There was noticeable performance slowdown
> in my case when it was trying hard to allocate.
>
> --
> Robin Hugh Johnson
> E-Mail : robb...@orbis-terrarum.net
> Home Page : http://www.orbis-terrarum.net/?l=people.robbat2
> ICQ# : 30269588 or 41961639
> GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
> commit 808c209dc82ce79147122ca78e7047bc74a16149
> Author: Robin H. Johnson
> Date: Wed Nov 30 10:32:57 2016 -0800
>
> mm: ratelimit & trace PFNs busy.
>
> Signed-off-by: Robin H. Johnson
> Acked-by: Michal Nazarewicz
>
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 6de9440e3ae2..3c28ec3d18f8 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -7289,8 +7289,16 @@ int alloc_contig_range(unsigned long start, unsigned
> long end,
>
> /* Make sure the range is really isolated. */
> if (test_pages_isolated(outer_start, end, false)) {
> - pr_info("%s: [%lx, %lx) PFNs busy\n",
> - __func__, outer_start, end);
> + static DEFINE_RATELIMIT_STATE(ratelimit_pfn_busy,
> + DEFAULT_RATELIMIT_INTERVAL,
> + DEFAULT_RATELIMIT_BURST);
> + if (__ratelimit(&ratelimit_pfn_busy)) {
> + pr_info("%s: [%lx, %lx) PFNs busy\n",
> + __func__, outer_start, end);
> + if (IS_ENABLED(CONFIG_CMA_DEBUG))
> + dump_stack();
> + }
> +
> ret = -EBUSY;
> goto done;
> }
--
Michal Hocko
SUSE Labs
Re: Regarding your thread on LKML - drm_radeon spamming alloc_contig_range [WAS: Re: PROBLEM-PERSISTS: dmesg spam: alloc_contig_range: [XX, YY) PFNs busy]
CC'd back to LKML.
On Thu, Jun 29, 2017 at 06:11:00PM +0530, Kumar Abhishek wrote:
> Hi Robin,
>
> I am an independent developer who stumbled upon your thread on the LKML
> after facing a similar issue - my kernel log being spammed by
> alloc_contig_range messages. I am running Linux on an ARM system
> (specifically the BeagleBoard-X15) and am on kernel version 4.9.33 with TI
> patches on top of it.
>
> I am running Debian Stretch (9.0) on the system.
>
> Here's what my stack trace looks like:
..
>
> It's somewhat similar to your stack trace, but this here happens on an
> etnaviv GPU (Vivante GCxx).
>
> In my case if I do 'sudo service lightdm stop', these messages stop too.
> This seems to suggest that the problem may be in the X server rather than
> the kernel? I seem to think this because I replicated this on an entirely
> different set of hardware than yours.
>
> I just wanted to bring this to your notice, and also ask you if you managed
> to solve it for yourself.
>
> One solution could be to demote the pr_info in alloc_contig_range to
> pr_debug or to do away with the message altogether, but this would be
> suppressing the issue instead of really knowing what it is about.
>
> Let me know how I could further investigate this.
The problem, as far as I got diagnosed on LKML, is that some of the GPUs
have a bunch of non-fatal contiguous memory allocation requests: they
have a meaningful fallback path on the allocation, so 'PFNs busy' is a
false busy for their case.
However, if there was a another consumer that does NOT have a fallback,
the output would still be crucially useful.
Attached is the patch that I unsuccessfully proposed on LKML to
rate-limit the messages, with the last revision to only dump_stack() if
CONFIG_CMA_DEBUG was set.
The path that LKML wanted was to add a new parameter to suppress or at
least demote the failure message, and update all of the callers: but it
means that many of the indirect callers need that added parameter as
well.
mm/cma.c:cma_alloc this call can suppress the error, you can see it retry.
mm/hugetlb.c: These callers should get the error message.
The error message DOES still have a good general use in notifying you
that something is going wrong. There was noticeable performance slowdown
in my case when it was trying hard to allocate.
--
Robin Hugh Johnson
E-Mail : robb...@orbis-terrarum.net
Home Page : http://www.orbis-terrarum.net/?l=people.robbat2
ICQ# : 30269588 or 41961639
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
commit 808c209dc82ce79147122ca78e7047bc74a16149
Author: Robin H. Johnson
Date: Wed Nov 30 10:32:57 2016 -0800
mm: ratelimit & trace PFNs busy.
Signed-off-by: Robin H. Johnson
Acked-by: Michal Nazarewicz
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 6de9440e3ae2..3c28ec3d18f8 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -7289,8 +7289,16 @@ int alloc_contig_range(unsigned long start, unsigned long end,
/* Make sure the range is really isolated. */
if (test_pages_isolated(outer_start, end, false)) {
- pr_info("%s: [%lx, %lx) PFNs busy\n",
- __func__, outer_start, end);
+ static DEFINE_RATELIMIT_STATE(ratelimit_pfn_busy,
+ DEFAULT_RATELIMIT_INTERVAL,
+ DEFAULT_RATELIMIT_BURST);
+ if (__ratelimit(&ratelimit_pfn_busy)) {
+ pr_info("%s: [%lx, %lx) PFNs busy\n",
+__func__, outer_start, end);
+ if (IS_ENABLED(CONFIG_CMA_DEBUG))
+dump_stack();
+ }
+
ret = -EBUSY;
goto done;
}
signature.asc
Description: Digital signature
Re: PROBLEM-PERSISTS: dmesg spam: alloc_contig_range: [XX, YY) PFNs busy
On Wed, Nov 30 2016, Robin H. Johnson wrote:
> (I'm going to respond directly to this email with the stack trace.)
>
> On Wed, Nov 30, 2016 at 02:28:49PM +0100, Michal Hocko wrote:
>> > On the other hand, if this didn’t happen and now happens all the time,
>> > this indicates a regression in CMA’s capability to allocate pages so
>> > just rate limiting the output would hide the potential actual issue.
>>
>> Or there might be just a much larger demand on those large blocks, no?
>> But seriously, dumping those message again and again into the low (see
>> the 2.5_GB_/h to the log is just insane. So there really should be some
>> throttling.
>>
>> Does the following help you Robin. At least to not get swamped by those
>> message.
> Here's what I whipped up based on that, to ensure that dump_stack got
> rate-limited at the same pass as PFNs-busy. It dropped the dmesg spew to
> ~25MB/hour (and is suppressing ~43 entries/second right now).
>
> commit 6ad4037e18ec2199f8755274d8a745a9904241a1
> Author: Robin H. Johnson
> Date: Wed Nov 30 10:32:57 2016 -0800
>
> mm: ratelimit & trace PFNs busy.
>
> Signed-off-by: Robin H. Johnson
Acked-by: Michal Nazarewicz
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 6de9440e3ae2..3c28ec3d18f8 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -7289,8 +7289,15 @@ int alloc_contig_range(unsigned long start, unsigned
> long end,
>
> /* Make sure the range is really isolated. */
> if (test_pages_isolated(outer_start, end, false)) {
> - pr_info("%s: [%lx, %lx) PFNs busy\n",
> - __func__, outer_start, end);
> + static DEFINE_RATELIMIT_STATE(ratelimit_pfn_busy,
> + DEFAULT_RATELIMIT_INTERVAL,
> + DEFAULT_RATELIMIT_BURST);
> + if (__ratelimit(&ratelimit_pfn_busy)) {
> + pr_info("%s: [%lx, %lx) PFNs busy\n",
> + __func__, outer_start, end);
I’m thinking out loud here, but maybe it would be useful to include
a count of how many times this message has been suppressed?
> + dump_stack();
Perhaps do it only if CMA_DEBUG?
+ if (IS_ENABLED(CONFIG_CMA_DEBUG))
+ dump_stack();
> + }
> +
> ret = -EBUSY;
> goto done;
> }
>
> --
> Robin Hugh Johnson
> Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer
> E-Mail : robb...@gentoo.org
> GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
> GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
--
Best regards
ミハウ “𝓶𝓲𝓷𝓪86” ナザレヴイツ
«If at first you don’t succeed, give up skydiving»
Re: PROBLEM-PERSISTS: dmesg spam: alloc_contig_range: [XX, YY) PFNs busy
(I'm going to respond directly to this email with the stack trace.)
On Wed, Nov 30, 2016 at 02:28:49PM +0100, Michal Hocko wrote:
> > On the other hand, if this didn’t happen and now happens all the time,
> > this indicates a regression in CMA’s capability to allocate pages so
> > just rate limiting the output would hide the potential actual issue.
>
> Or there might be just a much larger demand on those large blocks, no?
> But seriously, dumping those message again and again into the low (see
> the 2.5_GB_/h to the log is just insane. So there really should be some
> throttling.
>
> Does the following help you Robin. At least to not get swamped by those
> message.
Here's what I whipped up based on that, to ensure that dump_stack got
rate-limited at the same pass as PFNs-busy. It dropped the dmesg spew to
~25MB/hour (and is suppressing ~43 entries/second right now).
commit 6ad4037e18ec2199f8755274d8a745a9904241a1
Author: Robin H. Johnson
Date: Wed Nov 30 10:32:57 2016 -0800
mm: ratelimit & trace PFNs busy.
Signed-off-by: Robin H. Johnson
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 6de9440e3ae2..3c28ec3d18f8 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -7289,8 +7289,15 @@ int alloc_contig_range(unsigned long start, unsigned
long end,
/* Make sure the range is really isolated. */
if (test_pages_isolated(outer_start, end, false)) {
- pr_info("%s: [%lx, %lx) PFNs busy\n",
- __func__, outer_start, end);
+ static DEFINE_RATELIMIT_STATE(ratelimit_pfn_busy,
+ DEFAULT_RATELIMIT_INTERVAL,
+ DEFAULT_RATELIMIT_BURST);
+ if (__ratelimit(&ratelimit_pfn_busy)) {
+ pr_info("%s: [%lx, %lx) PFNs busy\n",
+ __func__, outer_start, end);
+ dump_stack();
+ }
+
ret = -EBUSY;
goto done;
}
--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer
E-Mail : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
Re: PROBLEM-PERSISTS: dmesg spam: alloc_contig_range: [XX, YY) PFNs busy
On Wed 30-11-16 14:08:00, Michal Nazarewicz wrote:
> On Wed, Nov 30 2016, Michal Hocko wrote:
> > [Let's CC linux-mm and Michal]
> >
> > On Tue 29-11-16 22:43:08, Robin H. Johnson wrote:
> >> I didn't get any responses to this.
> >>
> >> git bisect shows that the problem did actually exist in 4.5.0-rc6, but
> >> has gotten worse by many orders of magnitude (< 1/week to ~20M/hour).
> >>
> >> Presently with 4.9-rc5, it's now writing ~2.5GB/hour to syslog.
> >
> > This is really not helpful. I think we should simply make it pr_debug or
> > need some ratelimitting. AFAIU the message is far from serious
>
> On the other hand, if this didn’t happen and now happens all the time,
> this indicates a regression in CMA’s capability to allocate pages so
> just rate limiting the output would hide the potential actual issue.
Or there might be just a much larger demand on those large blocks, no?
But seriously, dumping those message again and again into the low (see
the 2.5_GB_/h to the log is just insane. So there really should be some
throttling.
Does the following help you Robin. At least to not get swamped by those
message.
---
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 0fbfead6aa7d..96eb8d107582 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -7424,7 +7424,7 @@ int alloc_contig_range(unsigned long start, unsigned long
end,
/* Make sure the range is really isolated. */
if (test_pages_isolated(outer_start, end, false)) {
- pr_info("%s: [%lx, %lx) PFNs busy\n",
+ printk_ratelimited(KERN_DEBUG "%s: [%lx, %lx) PFNs busy\n",
__func__, outer_start, end);
ret = -EBUSY;
goto done;
I would also suggest to add dump_stack() to that path to see who is
actually demanding so much large continuous blocks.
--
Michal Hocko
SUSE Labs
Re: PROBLEM-PERSISTS: dmesg spam: alloc_contig_range: [XX, YY) PFNs busy
On Wed, Nov 30 2016, Michal Hocko wrote: > [Let's CC linux-mm and Michal] > > On Tue 29-11-16 22:43:08, Robin H. Johnson wrote: >> I didn't get any responses to this. >> >> git bisect shows that the problem did actually exist in 4.5.0-rc6, but >> has gotten worse by many orders of magnitude (< 1/week to ~20M/hour). >> >> Presently with 4.9-rc5, it's now writing ~2.5GB/hour to syslog. > > This is really not helpful. I think we should simply make it pr_debug or > need some ratelimitting. AFAIU the message is far from serious On the other hand, if this didn’t happen and now happens all the time, this indicates a regression in CMA’s capability to allocate pages so just rate limiting the output would hide the potential actual issue. > >> The list of addresses in that time is only ~80 unique ranges, each >> appearing ~320K times. They don't appear exactly in order, so the kernel >> does not squelch the log message for appearing too frequently. >> >> Could somebody at least make a suggestion on how to trace the printed >> range to somewhere in the kernel? >> >> On Sat, Nov 19, 2016 at 03:25:32AM +, Robin H. Johnson wrote: >> > (Replies CC to list and direct to me please) >> > >> > Summary: >> > >> > dmesg spammed with alloc_contig_range: [XX, YY) PFNs busy >> > >> > Description: >> > >> > I recently upgrading 4.9-rc5, (previous kernel 4.5.0-rc6-00141-g6794402), >> > and since then my dmesg has been absolutely flooded with 'PFNs busy' >> > (>3GiB/day). My config did not change (all new options =n). >> > >> > It's not consistent addresses, so the squelch of identical printk lines >> > hasn't helped. >> > Eg output: >> > [187487.621916] alloc_contig_range: [83f0a9, 83f0aa) PFNs busy >> > [187487.621924] alloc_contig_range: [83f0ce, 83f0cf) PFNs busy >> > [187487.621976] alloc_contig_range: [83f125, 83f126) PFNs busy >> > [187487.622013] alloc_contig_range: [83f127, 83f128) PFNs busy >> > >> > Keywords: >> > - >> > mm, alloc_contig_range, CMA >> > >> > Most recent kernel version which did not have the bug: >> > -- >> > Known 4.5.0-rc6-00141-g6794402 >> > >> > ver_linux: >> > -- >> > Linux bohr-int 4.9.0-rc5-00177-g81bcfe5 #12 SMP Wed Nov 16 13:16:32 PST >> > 2016 x86_64 Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz GenuineIntel >> > GNU/Linux >> > >> > GNU C 5.3.0 >> > GNU Make 4.2.1 >> > Binutils 2.25.1 >> > Util-linux 2.29 >> > Mount 2.29 >> > Quota-tools4.03 >> > Linux C Library2.23 >> > Dynamic linker (ldd) 2.23 >> > readlink: missing operand >> > Try 'readlink --help' for more information. >> > Procps 3.3.12 >> > Net-tools 1.60 >> > Kbd2.0.3 >> > Console-tools 2.0.3 >> > Sh-utils 8.25 >> > Udev 230 >> > Modules Loaded 3w_sas 3w_ ablk_helper aesni_intel >> > aes_x86_64 af_packet ahci aic79xx amdgpu async_memcpy async_pq >> > async_raid6_recov async_tx async_xor ata_piix auth_rpcgss binfmt_misc >> > bluetooth bnep bnx2 bonding btbcm btintel btrfs btrtl btusb button cdrom >> > cn configs coretemp crc32c_intel crc32_pclmul crc_ccitt crc_itu_t >> > crct10dif_pclmul cryptd dca dm_bio_prison dm_bufio dm_cache dm_cache_smq >> > dm_crypt dm_delay dm_flakey dm_log dm_log_userspace dm_mirror dm_mod >> > dm_multipath dm_persistent_data dm_queue_length dm_raid dm_region_hash >> > dm_round_robin dm_service_time dm_snapshot dm_thin_pool dm_zero drm >> > drm_kms_helper dummy e1000 e1000e evdev ext2 fat fb_sys_fops >> > firewire_core firewire_ohci fjes fscache fuse ghash_clmulni_intel >> > glue_helper grace hangcheck_timer hid_a4tech hid_apple hid_belkin >> > hid_cherry hid_chicony hid_cypress hid_ezkey hid_generic hid_gyration >> > hid_logitech hid_logitech_dj hid_microsoft hid_monterey hid_petalynx >> > hid_pl hid_samsung hid_sony hid_sunplus hwmon_vid i2c_algo_bit i2c_i801 >> > i2c_smbus igb input_leds intel_rapl ip6_udp_tunnel ipv6 irqbypass >> > iscsi_tcp iTCO_vendor_support iTCO_wdt ixgb ixgbe jfs kvm kvm_intel >> > libahci libata libcrc32c libiscsi libiscsi_tcp linear lockd lpc_ich lpfc >> > lrw macvlan mdio md_mod megaraid_mbox megaraid_mm megaraid_sas mii >> > mptbase mptfc mptsas mptscsih mptspi multipath nfs nfs_acl nfsd >> > nls_cp437 nls_iso8859_1 nvram ohci_hcd pata_jmicron pata_marvell >> > pata_platform pcspkr psmouse qla1280 qla2xxx r8169 radeon raid0 raid10 >> > raid1 raid456 raid6_pq reiserfs rfkill sata_mv sata_sil24 >> > scsi_transport_fc scsi_transport_iscsi scsi_transport_sas >> > scsi_transport_spi sd_mod sg sky2 snd snd_hda_codec >> > snd_hda_codec_generic snd_hda_codec_hdmi snd_hda
Re: PROBLEM-PERSISTS: dmesg spam: alloc_contig_range: [XX, YY) PFNs busy
[Let's CC linux-mm and Michal] On Tue 29-11-16 22:43:08, Robin H. Johnson wrote: > I didn't get any responses to this. > > git bisect shows that the problem did actually exist in 4.5.0-rc6, but > has gotten worse by many orders of magnitude (< 1/week to ~20M/hour). > > Presently with 4.9-rc5, it's now writing ~2.5GB/hour to syslog. This is really not helpful. I think we should simply make it pr_debug or need some ratelimitting. AFAIU the message is far from serious > The list of addresses in that time is only ~80 unique ranges, each > appearing ~320K times. They don't appear exactly in order, so the kernel > does not squelch the log message for appearing too frequently. > > Could somebody at least make a suggestion on how to trace the printed > range to somewhere in the kernel? > > On Sat, Nov 19, 2016 at 03:25:32AM +, Robin H. Johnson wrote: > > (Replies CC to list and direct to me please) > > > > Summary: > > > > dmesg spammed with alloc_contig_range: [XX, YY) PFNs busy > > > > Description: > > > > I recently upgrading 4.9-rc5, (previous kernel 4.5.0-rc6-00141-g6794402), > > and since then my dmesg has been absolutely flooded with 'PFNs busy' > > (>3GiB/day). My config did not change (all new options =n). > > > > It's not consistent addresses, so the squelch of identical printk lines > > hasn't helped. > > Eg output: > > [187487.621916] alloc_contig_range: [83f0a9, 83f0aa) PFNs busy > > [187487.621924] alloc_contig_range: [83f0ce, 83f0cf) PFNs busy > > [187487.621976] alloc_contig_range: [83f125, 83f126) PFNs busy > > [187487.622013] alloc_contig_range: [83f127, 83f128) PFNs busy > > > > Keywords: > > - > > mm, alloc_contig_range, CMA > > > > Most recent kernel version which did not have the bug: > > -- > > Known 4.5.0-rc6-00141-g6794402 > > > > ver_linux: > > -- > > Linux bohr-int 4.9.0-rc5-00177-g81bcfe5 #12 SMP Wed Nov 16 13:16:32 PST > > 2016 x86_64 Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz GenuineIntel > > GNU/Linux > > > > GNU C 5.3.0 > > GNU Make4.2.1 > > Binutils2.25.1 > > Util-linux 2.29 > > Mount 2.29 > > Quota-tools 4.03 > > Linux C Library 2.23 > > Dynamic linker (ldd)2.23 > > readlink: missing operand > > Try 'readlink --help' for more information. > > Procps 3.3.12 > > Net-tools 1.60 > > Kbd 2.0.3 > > Console-tools 2.0.3 > > Sh-utils8.25 > > Udev230 > > Modules Loaded 3w_sas 3w_ ablk_helper aesni_intel > > aes_x86_64 af_packet ahci aic79xx amdgpu async_memcpy async_pq > > async_raid6_recov async_tx async_xor ata_piix auth_rpcgss binfmt_misc > > bluetooth bnep bnx2 bonding btbcm btintel btrfs btrtl btusb button cdrom > > cn configs coretemp crc32c_intel crc32_pclmul crc_ccitt crc_itu_t > > crct10dif_pclmul cryptd dca dm_bio_prison dm_bufio dm_cache dm_cache_smq > > dm_crypt dm_delay dm_flakey dm_log dm_log_userspace dm_mirror dm_mod > > dm_multipath dm_persistent_data dm_queue_length dm_raid dm_region_hash > > dm_round_robin dm_service_time dm_snapshot dm_thin_pool dm_zero drm > > drm_kms_helper dummy e1000 e1000e evdev ext2 fat fb_sys_fops > > firewire_core firewire_ohci fjes fscache fuse ghash_clmulni_intel > > glue_helper grace hangcheck_timer hid_a4tech hid_apple hid_belkin > > hid_cherry hid_chicony hid_cypress hid_ezkey hid_generic hid_gyration > > hid_logitech hid_logitech_dj hid_microsoft hid_monterey hid_petalynx > > hid_pl hid_samsung hid_sony hid_sunplus hwmon_vid i2c_algo_bit i2c_i801 > > i2c_smbus igb input_leds intel_rapl ip6_udp_tunnel ipv6 irqbypass > > iscsi_tcp iTCO_vendor_support iTCO_wdt ixgb ixgbe jfs kvm kvm_intel > > libahci libata libcrc32c libiscsi libiscsi_tcp linear lockd lpc_ich lpfc > > lrw macvlan mdio md_mod megaraid_mbox megaraid_mm megaraid_sas mii > > mptbase mptfc mptsas mptscsih mptspi multipath nfs nfs_acl nfsd > > nls_cp437 nls_iso8859_1 nvram ohci_hcd pata_jmicron pata_marvell > > pata_platform pcspkr psmouse qla1280 qla2xxx r8169 radeon raid0 raid10 > > raid1 raid456 raid6_pq reiserfs rfkill sata_mv sata_sil24 > > scsi_transport_fc scsi_transport_iscsi scsi_transport_sas > > scsi_transport_spi sd_mod sg sky2 snd snd_hda_codec > > snd_hda_codec_generic snd_hda_codec_hdmi snd_hda_codec_realtek > > snd_hda_core snd_hda_intel snd_hwdep snd_pcm snd_timer soundcore sr_mod > > sunrpc syscopyarea sysfillrect sysimgblt tg3 ttm uas udp_tunnel > > usb_storage vfat virtio virtio_net virtio_ring vxlan w83627ehf > > x86_pkg_temp_thermal xfs xhci_hcd xhci_pci xor zlib_deflate > > -- > Robin Hugh Johnson > E-Mail : robb...@orbis-terrar
