Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot
disabled CONFIG_FORCE_NR_CPUS option for 6.9.5 but the trace + panic still exists. So that one didn't help. I've also been bisecting the trace but have not finished it yet as the last half dozen builds produced non-bootable kernels. Anyway, I will continue it soon(ish) when I have a bit more free time. --Ilkka On Tue, Jun 18, 2024 at 5:52 PM Steven Rostedt wrote: > > On Thu, 13 Jun 2024 10:32:24 +0300 > Ilkka Naulapää wrote: > > > ok, so if you don't have any idea where this bug is after those debug > > patches, I'll try to find some time to bisect it as a last resort. > > Stay tuned. > > FYI, > > I just debugged a strange crash that was caused by my config having > something leftover from your config. Specifically, that was: > > CONFIG_FORCE_NR_CPUS > > Do you get any warning about nr cpus not matching at boot up? > > Regardless, can you disable that and see if you still get the same > crash. > > Thanks, > > -- Steve
Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot
On Thu, 13 Jun 2024 10:32:24 +0300 Ilkka Naulapää wrote: > ok, so if you don't have any idea where this bug is after those debug > patches, I'll try to find some time to bisect it as a last resort. > Stay tuned. FYI, I just debugged a strange crash that was caused by my config having something leftover from your config. Specifically, that was: CONFIG_FORCE_NR_CPUS Do you get any warning about nr cpus not matching at boot up? Regardless, can you disable that and see if you still get the same crash. Thanks, -- Steve
Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot
On 13.06.24 09:32, Ilkka Naulapää wrote:
> On Wed, Jun 12, 2024 at 6:56 PM Steven Rostedt wrote:
>> On Wed, 12 Jun 2024 15:36:22 +0200
>> "Linux regression tracking (Thorsten Leemhuis)"
>> wrote:
>>>
>>> Ilkka or Steven, what happened to this? This thread looks stalled. I
>>> also was unsuccessful when looking for other threads related to this
>>> report or the culprit. Did it fall through the cracks or am I missing
>>> something here?
>
>> Honesty, I have no idea where the bug is. I can't reproduce it. [...]
Steven, thx for the update! And yeah, that's how it sometimes is. Given
that we haven't seen similar reports (at least afaics) it's nothing I
worry much about.
> ok, so if you don't have any idea where this bug is after those debug
> patches, I'll try to find some time to bisect it as a last resort.
> Stay tuned.
Yeah, that would be great help. Thank you, too!
Ciao, Thorsten
>>> On 02.06.24 09:32, Ilkka Naulapää wrote:
sorry longer delay, been a bit busy but here is the result from that
new patch. Only applied this patch so if the previous one is needed
also, let me know and I'll rerun it.
--Ilkka
On Thu, May 30, 2024 at 5:00 PM Steven Rostedt wrote:
>
> On Thu, 30 May 2024 16:02:37 +0300
> Ilkka Naulapää wrote:
>
>> applied your patch and here's the output.
>>
>
> Unfortunately, it doesn't give me any new information. I added one more
> BUG on, want to try this? Otherwise, I'm pretty much at a lost. :-/
>
> -- Steve
>
> diff --git a/fs/tracefs/inode.c b/fs/tracefs/inode.c
> index de5b72216b1a..a090495e78c9 100644
> --- a/fs/tracefs/inode.c
> +++ b/fs/tracefs/inode.c
> @@ -39,13 +39,17 @@ static struct inode *tracefs_alloc_inode(struct
> super_block *sb)
> return NULL;
>
> ti->flags = 0;
> + ti->magic = 20240823;
>
> return &ti->vfs_inode;
> }
>
> static void tracefs_free_inode(struct inode *inode)
> {
> - kmem_cache_free(tracefs_inode_cachep, get_tracefs(inode));
> + struct tracefs_inode *ti = get_tracefs(inode);
> +
> + BUG_ON(ti->magic != 20240823);
> + kmem_cache_free(tracefs_inode_cachep, ti);
> }
>
> static ssize_t default_read_file(struct file *file, char __user *buf,
> @@ -147,16 +151,6 @@ static const struct inode_operations
> tracefs_dir_inode_operations = {
> .rmdir = tracefs_syscall_rmdir,
> };
>
> -struct inode *tracefs_get_inode(struct super_block *sb)
> -{
> - struct inode *inode = new_inode(sb);
> - if (inode) {
> - inode->i_ino = get_next_ino();
> - inode->i_atime = inode->i_mtime =
> inode_set_ctime_current(inode);
> - }
> - return inode;
> -}
> -
> struct tracefs_mount_opts {
> kuid_t uid;
> kgid_t gid;
> @@ -384,6 +378,7 @@ static void tracefs_dentry_iput(struct dentry
> *dentry, struct inode *inode)
> return;
>
> ti = get_tracefs(inode);
> + BUG_ON(ti->magic != 20240823);
> if (ti && ti->flags & TRACEFS_EVENT_INODE)
> eventfs_set_ef_status_free(dentry);
> iput(inode);
> @@ -568,6 +563,18 @@ struct dentry *eventfs_end_creating(struct dentry
> *dentry)
> return dentry;
> }
>
> +struct inode *tracefs_get_inode(struct super_block *sb)
> +{
> + struct inode *inode = new_inode(sb);
> +
> + BUG_ON(sb->s_op != &tracefs_super_operations);
> + if (inode) {
> + inode->i_ino = get_next_ino();
> + inode->i_atime = inode->i_mtime =
> inode_set_ctime_current(inode);
> + }
> + return inode;
> +}
> +
> /**
> * tracefs_create_file - create a file in the tracefs filesystem
> * @name: a pointer to a string containing the name of the file to
> create.
> diff --git a/fs/tracefs/internal.h b/fs/tracefs/internal.h
> index 69c2b1d87c46..9059b8b11bb6 100644
> --- a/fs/tracefs/internal.h
> +++ b/fs/tracefs/internal.h
> @@ -9,12 +9,15 @@ enum {
> struct tracefs_inode {
> unsigned long flags;
> void*private;
> + unsigned long magic;
> struct inodevfs_inode;
> };
>
> static inline struct tracefs_inode *get_tracefs(const struct inode
> *inode)
> {
> - return container_of(inode, struct tracefs_inode, vfs_inode);
> + struct tracefs_inode *ti = container_of(inode, struct
> tracefs_inode, vfs_inode);
> + BUG_ON(ti->magic != 20240823);
> + return ti;
> }
>
> struct dentry *tracefs_start_creating(const char *name, struct dentry
> *parent);
Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot
ok, so if you don't have any idea where this bug is after those debug
patches, I'll try to find some time to bisect it as a last resort.
Stay tuned.
--Ilkka
On Wed, Jun 12, 2024 at 6:56 PM Steven Rostedt wrote:
>
> On Wed, 12 Jun 2024 15:36:22 +0200
> "Linux regression tracking (Thorsten Leemhuis)"
> wrote:
>
> > Hi, Thorsten here, the Linux kernel's regression tracker. Top-posting
> > for once, to make this easily accessible to everyone.
> >
> > Ilkka or Steven, what happened to this? This thread looks stalled. I
> > also was unsuccessful when looking for other threads related to this
> > report or the culprit. Did it fall through the cracks or am I missing
> > something here?
>
> Honesty, I have no idea where the bug is. I can't reproduce it. These
> patches I sent would check all the places that add to the list to make
> sure the proper trace_inode was being added, and the output shows that
> they are all correct. Then suddenly, something that came from the
> inode cache is calling the tracefs inode cache to free it, and that's
> where the bug is happening.
>
> This really looks like another bug that the recent changes have made
> more predominate.
>
> -- Steve
>
>
> >
> > Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
> > --
> > Everything you wanna know about Linux kernel regression tracking:
> > https://linux-regtracking.leemhuis.info/about/#tldr
> > If I did something stupid, please tell me, as explained on that page.
> >
> > #regzbot poke
> >
> > On 02.06.24 09:32, Ilkka Naulapää wrote:
> > > sorry longer delay, been a bit busy but here is the result from that
> > > new patch. Only applied this patch so if the previous one is needed
> > > also, let me know and I'll rerun it.
> > >
> > > --Ilkka
> > >
> > > On Thu, May 30, 2024 at 5:00 PM Steven Rostedt
> > > wrote:
> > >>
> > >> On Thu, 30 May 2024 16:02:37 +0300
> > >> Ilkka Naulapää wrote:
> > >>
> > >>> applied your patch and here's the output.
> > >>>
> > >>
> > >> Unfortunately, it doesn't give me any new information. I added one more
> > >> BUG on, want to try this? Otherwise, I'm pretty much at a lost. :-/
> > >>
> > >> -- Steve
> > >>
> > >> diff --git a/fs/tracefs/inode.c b/fs/tracefs/inode.c
> > >> index de5b72216b1a..a090495e78c9 100644
> > >> --- a/fs/tracefs/inode.c
> > >> +++ b/fs/tracefs/inode.c
> > >> @@ -39,13 +39,17 @@ static struct inode *tracefs_alloc_inode(struct
> > >> super_block *sb)
> > >> return NULL;
> > >>
> > >> ti->flags = 0;
> > >> + ti->magic = 20240823;
> > >>
> > >> return &ti->vfs_inode;
> > >> }
> > >>
> > >> static void tracefs_free_inode(struct inode *inode)
> > >> {
> > >> - kmem_cache_free(tracefs_inode_cachep, get_tracefs(inode));
> > >> + struct tracefs_inode *ti = get_tracefs(inode);
> > >> +
> > >> + BUG_ON(ti->magic != 20240823);
> > >> + kmem_cache_free(tracefs_inode_cachep, ti);
> > >> }
> > >>
> > >> static ssize_t default_read_file(struct file *file, char __user *buf,
> > >> @@ -147,16 +151,6 @@ static const struct inode_operations
> > >> tracefs_dir_inode_operations = {
> > >> .rmdir = tracefs_syscall_rmdir,
> > >> };
> > >>
> > >> -struct inode *tracefs_get_inode(struct super_block *sb)
> > >> -{
> > >> - struct inode *inode = new_inode(sb);
> > >> - if (inode) {
> > >> - inode->i_ino = get_next_ino();
> > >> - inode->i_atime = inode->i_mtime =
> > >> inode_set_ctime_current(inode);
> > >> - }
> > >> - return inode;
> > >> -}
> > >> -
> > >> struct tracefs_mount_opts {
> > >> kuid_t uid;
> > >> kgid_t gid;
> > >> @@ -384,6 +378,7 @@ static void tracefs_dentry_iput(struct dentry
> > >> *dentry, struct inode *inode)
> > >> return;
> > >>
> > >> ti = get_tracefs(inode);
> > >> + BUG_ON(ti->magic != 20240823);
> > >> if (ti && ti->flags & TRACEFS_EVENT_INODE)
> > >> eventfs_set_ef_status_free(dentry);
> > >> iput(inode);
> > >> @@ -568,6 +563,18 @@ struct dentry *eventfs_end_creating(struct dentry
> > >> *dentry)
> > >> return dentry;
> > >> }
> > >>
> > >> +struct inode *tracefs_get_inode(struct super_block *sb)
> > >> +{
> > >> + struct inode *inode = new_inode(sb);
> > >> +
> > >> + BUG_ON(sb->s_op != &tracefs_super_operations);
> > >> + if (inode) {
> > >> + inode->i_ino = get_next_ino();
> > >> + inode->i_atime = inode->i_mtime =
> > >> inode_set_ctime_current(inode);
> > >> + }
> > >> + return inode;
> > >> +}
> > >> +
> > >> /**
> > >> * tracefs_create_file - create a file in the tracefs filesystem
> > >> * @name: a pointer to a string containing the name of the file to
> > >> create.
> > >> diff --git a/fs/tracefs/internal.h b/fs/tracefs/internal.h
> > >> index 69c2b1d87c46..9059b8b11bb6 100644
> > >> --- a/fs/tracefs/internal.h
> > >> +++ b/fs/tracefs/internal.h
> > >> @
Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot
On Wed, 12 Jun 2024 15:36:22 +0200
"Linux regression tracking (Thorsten Leemhuis)"
wrote:
> Hi, Thorsten here, the Linux kernel's regression tracker. Top-posting
> for once, to make this easily accessible to everyone.
>
> Ilkka or Steven, what happened to this? This thread looks stalled. I
> also was unsuccessful when looking for other threads related to this
> report or the culprit. Did it fall through the cracks or am I missing
> something here?
Honesty, I have no idea where the bug is. I can't reproduce it. These
patches I sent would check all the places that add to the list to make
sure the proper trace_inode was being added, and the output shows that
they are all correct. Then suddenly, something that came from the
inode cache is calling the tracefs inode cache to free it, and that's
where the bug is happening.
This really looks like another bug that the recent changes have made
more predominate.
-- Steve
>
> Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
> --
> Everything you wanna know about Linux kernel regression tracking:
> https://linux-regtracking.leemhuis.info/about/#tldr
> If I did something stupid, please tell me, as explained on that page.
>
> #regzbot poke
>
> On 02.06.24 09:32, Ilkka Naulapää wrote:
> > sorry longer delay, been a bit busy but here is the result from that
> > new patch. Only applied this patch so if the previous one is needed
> > also, let me know and I'll rerun it.
> >
> > --Ilkka
> >
> > On Thu, May 30, 2024 at 5:00 PM Steven Rostedt wrote:
> >
> >>
> >> On Thu, 30 May 2024 16:02:37 +0300
> >> Ilkka Naulapää wrote:
> >>
> >>> applied your patch and here's the output.
> >>>
> >>
> >> Unfortunately, it doesn't give me any new information. I added one more
> >> BUG on, want to try this? Otherwise, I'm pretty much at a lost. :-/
> >>
> >> -- Steve
> >>
> >> diff --git a/fs/tracefs/inode.c b/fs/tracefs/inode.c
> >> index de5b72216b1a..a090495e78c9 100644
> >> --- a/fs/tracefs/inode.c
> >> +++ b/fs/tracefs/inode.c
> >> @@ -39,13 +39,17 @@ static struct inode *tracefs_alloc_inode(struct
> >> super_block *sb)
> >> return NULL;
> >>
> >> ti->flags = 0;
> >> + ti->magic = 20240823;
> >>
> >> return &ti->vfs_inode;
> >> }
> >>
> >> static void tracefs_free_inode(struct inode *inode)
> >> {
> >> - kmem_cache_free(tracefs_inode_cachep, get_tracefs(inode));
> >> + struct tracefs_inode *ti = get_tracefs(inode);
> >> +
> >> + BUG_ON(ti->magic != 20240823);
> >> + kmem_cache_free(tracefs_inode_cachep, ti);
> >> }
> >>
> >> static ssize_t default_read_file(struct file *file, char __user *buf,
> >> @@ -147,16 +151,6 @@ static const struct inode_operations
> >> tracefs_dir_inode_operations = {
> >> .rmdir = tracefs_syscall_rmdir,
> >> };
> >>
> >> -struct inode *tracefs_get_inode(struct super_block *sb)
> >> -{
> >> - struct inode *inode = new_inode(sb);
> >> - if (inode) {
> >> - inode->i_ino = get_next_ino();
> >> - inode->i_atime = inode->i_mtime =
> >> inode_set_ctime_current(inode);
> >> - }
> >> - return inode;
> >> -}
> >> -
> >> struct tracefs_mount_opts {
> >> kuid_t uid;
> >> kgid_t gid;
> >> @@ -384,6 +378,7 @@ static void tracefs_dentry_iput(struct dentry *dentry,
> >> struct inode *inode)
> >> return;
> >>
> >> ti = get_tracefs(inode);
> >> + BUG_ON(ti->magic != 20240823);
> >> if (ti && ti->flags & TRACEFS_EVENT_INODE)
> >> eventfs_set_ef_status_free(dentry);
> >> iput(inode);
> >> @@ -568,6 +563,18 @@ struct dentry *eventfs_end_creating(struct dentry
> >> *dentry)
> >> return dentry;
> >> }
> >>
> >> +struct inode *tracefs_get_inode(struct super_block *sb)
> >> +{
> >> + struct inode *inode = new_inode(sb);
> >> +
> >> + BUG_ON(sb->s_op != &tracefs_super_operations);
> >> + if (inode) {
> >> + inode->i_ino = get_next_ino();
> >> + inode->i_atime = inode->i_mtime =
> >> inode_set_ctime_current(inode);
> >> + }
> >> + return inode;
> >> +}
> >> +
> >> /**
> >> * tracefs_create_file - create a file in the tracefs filesystem
> >> * @name: a pointer to a string containing the name of the file to create.
> >> diff --git a/fs/tracefs/internal.h b/fs/tracefs/internal.h
> >> index 69c2b1d87c46..9059b8b11bb6 100644
> >> --- a/fs/tracefs/internal.h
> >> +++ b/fs/tracefs/internal.h
> >> @@ -9,12 +9,15 @@ enum {
> >> struct tracefs_inode {
> >> unsigned long flags;
> >> void*private;
> >> + unsigned long magic;
> >> struct inodevfs_inode;
> >> };
> >>
> >> static inline struct tracefs_inode *get_tracefs(const struct inode *inode)
> >> {
> >> - return container_of(inode, struct tracefs_inode, vfs_inode);
> >> + struct tracefs_inode *ti = container_of(inode,
Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot
Hi, Thorsten here, the Linux kernel's regression tracker. Top-posting
for once, to make this easily accessible to everyone.
Ilkka or Steven, what happened to this? This thread looks stalled. I
also was unsuccessful when looking for other threads related to this
report or the culprit. Did it fall through the cracks or am I missing
something here?
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.
#regzbot poke
On 02.06.24 09:32, Ilkka Naulapää wrote:
> sorry longer delay, been a bit busy but here is the result from that
> new patch. Only applied this patch so if the previous one is needed
> also, let me know and I'll rerun it.
>
> --Ilkka
>
> On Thu, May 30, 2024 at 5:00 PM Steven Rostedt wrote:
>>
>> On Thu, 30 May 2024 16:02:37 +0300
>> Ilkka Naulapää wrote:
>>
>>> applied your patch and here's the output.
>>>
>>
>> Unfortunately, it doesn't give me any new information. I added one more
>> BUG on, want to try this? Otherwise, I'm pretty much at a lost. :-/
>>
>> -- Steve
>>
>> diff --git a/fs/tracefs/inode.c b/fs/tracefs/inode.c
>> index de5b72216b1a..a090495e78c9 100644
>> --- a/fs/tracefs/inode.c
>> +++ b/fs/tracefs/inode.c
>> @@ -39,13 +39,17 @@ static struct inode *tracefs_alloc_inode(struct
>> super_block *sb)
>> return NULL;
>>
>> ti->flags = 0;
>> + ti->magic = 20240823;
>>
>> return &ti->vfs_inode;
>> }
>>
>> static void tracefs_free_inode(struct inode *inode)
>> {
>> - kmem_cache_free(tracefs_inode_cachep, get_tracefs(inode));
>> + struct tracefs_inode *ti = get_tracefs(inode);
>> +
>> + BUG_ON(ti->magic != 20240823);
>> + kmem_cache_free(tracefs_inode_cachep, ti);
>> }
>>
>> static ssize_t default_read_file(struct file *file, char __user *buf,
>> @@ -147,16 +151,6 @@ static const struct inode_operations
>> tracefs_dir_inode_operations = {
>> .rmdir = tracefs_syscall_rmdir,
>> };
>>
>> -struct inode *tracefs_get_inode(struct super_block *sb)
>> -{
>> - struct inode *inode = new_inode(sb);
>> - if (inode) {
>> - inode->i_ino = get_next_ino();
>> - inode->i_atime = inode->i_mtime =
>> inode_set_ctime_current(inode);
>> - }
>> - return inode;
>> -}
>> -
>> struct tracefs_mount_opts {
>> kuid_t uid;
>> kgid_t gid;
>> @@ -384,6 +378,7 @@ static void tracefs_dentry_iput(struct dentry *dentry,
>> struct inode *inode)
>> return;
>>
>> ti = get_tracefs(inode);
>> + BUG_ON(ti->magic != 20240823);
>> if (ti && ti->flags & TRACEFS_EVENT_INODE)
>> eventfs_set_ef_status_free(dentry);
>> iput(inode);
>> @@ -568,6 +563,18 @@ struct dentry *eventfs_end_creating(struct dentry
>> *dentry)
>> return dentry;
>> }
>>
>> +struct inode *tracefs_get_inode(struct super_block *sb)
>> +{
>> + struct inode *inode = new_inode(sb);
>> +
>> + BUG_ON(sb->s_op != &tracefs_super_operations);
>> + if (inode) {
>> + inode->i_ino = get_next_ino();
>> + inode->i_atime = inode->i_mtime =
>> inode_set_ctime_current(inode);
>> + }
>> + return inode;
>> +}
>> +
>> /**
>> * tracefs_create_file - create a file in the tracefs filesystem
>> * @name: a pointer to a string containing the name of the file to create.
>> diff --git a/fs/tracefs/internal.h b/fs/tracefs/internal.h
>> index 69c2b1d87c46..9059b8b11bb6 100644
>> --- a/fs/tracefs/internal.h
>> +++ b/fs/tracefs/internal.h
>> @@ -9,12 +9,15 @@ enum {
>> struct tracefs_inode {
>> unsigned long flags;
>> void*private;
>> + unsigned long magic;
>> struct inodevfs_inode;
>> };
>>
>> static inline struct tracefs_inode *get_tracefs(const struct inode *inode)
>> {
>> - return container_of(inode, struct tracefs_inode, vfs_inode);
>> + struct tracefs_inode *ti = container_of(inode, struct tracefs_inode,
>> vfs_inode);
>> + BUG_ON(ti->magic != 20240823);
>> + return ti;
>> }
>>
>> struct dentry *tracefs_start_creating(const char *name, struct dentry
>> *parent);
Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot
On Thu, 30 May 2024 16:02:37 +0300
Ilkka Naulapää wrote:
> applied your patch and here's the output.
>
Unfortunately, it doesn't give me any new information. I added one more
BUG on, want to try this? Otherwise, I'm pretty much at a lost. :-/
-- Steve
diff --git a/fs/tracefs/inode.c b/fs/tracefs/inode.c
index de5b72216b1a..a090495e78c9 100644
--- a/fs/tracefs/inode.c
+++ b/fs/tracefs/inode.c
@@ -39,13 +39,17 @@ static struct inode *tracefs_alloc_inode(struct super_block
*sb)
return NULL;
ti->flags = 0;
+ ti->magic = 20240823;
return &ti->vfs_inode;
}
static void tracefs_free_inode(struct inode *inode)
{
- kmem_cache_free(tracefs_inode_cachep, get_tracefs(inode));
+ struct tracefs_inode *ti = get_tracefs(inode);
+
+ BUG_ON(ti->magic != 20240823);
+ kmem_cache_free(tracefs_inode_cachep, ti);
}
static ssize_t default_read_file(struct file *file, char __user *buf,
@@ -147,16 +151,6 @@ static const struct inode_operations
tracefs_dir_inode_operations = {
.rmdir = tracefs_syscall_rmdir,
};
-struct inode *tracefs_get_inode(struct super_block *sb)
-{
- struct inode *inode = new_inode(sb);
- if (inode) {
- inode->i_ino = get_next_ino();
- inode->i_atime = inode->i_mtime =
inode_set_ctime_current(inode);
- }
- return inode;
-}
-
struct tracefs_mount_opts {
kuid_t uid;
kgid_t gid;
@@ -384,6 +378,7 @@ static void tracefs_dentry_iput(struct dentry *dentry,
struct inode *inode)
return;
ti = get_tracefs(inode);
+ BUG_ON(ti->magic != 20240823);
if (ti && ti->flags & TRACEFS_EVENT_INODE)
eventfs_set_ef_status_free(dentry);
iput(inode);
@@ -568,6 +563,18 @@ struct dentry *eventfs_end_creating(struct dentry *dentry)
return dentry;
}
+struct inode *tracefs_get_inode(struct super_block *sb)
+{
+ struct inode *inode = new_inode(sb);
+
+ BUG_ON(sb->s_op != &tracefs_super_operations);
+ if (inode) {
+ inode->i_ino = get_next_ino();
+ inode->i_atime = inode->i_mtime =
inode_set_ctime_current(inode);
+ }
+ return inode;
+}
+
/**
* tracefs_create_file - create a file in the tracefs filesystem
* @name: a pointer to a string containing the name of the file to create.
diff --git a/fs/tracefs/internal.h b/fs/tracefs/internal.h
index 69c2b1d87c46..9059b8b11bb6 100644
--- a/fs/tracefs/internal.h
+++ b/fs/tracefs/internal.h
@@ -9,12 +9,15 @@ enum {
struct tracefs_inode {
unsigned long flags;
void*private;
+ unsigned long magic;
struct inodevfs_inode;
};
static inline struct tracefs_inode *get_tracefs(const struct inode *inode)
{
- return container_of(inode, struct tracefs_inode, vfs_inode);
+ struct tracefs_inode *ti = container_of(inode, struct tracefs_inode,
vfs_inode);
+ BUG_ON(ti->magic != 20240823);
+ return ti;
}
struct dentry *tracefs_start_creating(const char *name, struct dentry *parent);
Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot
On Wed, 29 May 2024 14:47:57 -0400
Steven Rostedt wrote:
> Let me make a debug patch (that crashes on this issue) for that kernel,
> and perhaps you could bisect it?
Can you try this on 6.6-rc1 and see if it gives you any other splats?
Hmm, you can switch it to WARN_ON and that way it may not crash the
machine, and you can use dmesg to get the output.
Thanks,
-- Steve
diff --git a/fs/tracefs/inode.c b/fs/tracefs/inode.c
index de5b72216b1a..a090495e78c9 100644
--- a/fs/tracefs/inode.c
+++ b/fs/tracefs/inode.c
@@ -39,13 +39,17 @@ static struct inode *tracefs_alloc_inode(struct super_block
*sb)
return NULL;
ti->flags = 0;
+ ti->magic = 20240823;
return &ti->vfs_inode;
}
static void tracefs_free_inode(struct inode *inode)
{
- kmem_cache_free(tracefs_inode_cachep, get_tracefs(inode));
+ struct tracefs_inode *ti = get_tracefs(inode);
+
+ BUG_ON(ti->magic != 20240823);
+ kmem_cache_free(tracefs_inode_cachep, ti);
}
static ssize_t default_read_file(struct file *file, char __user *buf,
@@ -147,16 +151,6 @@ static const struct inode_operations
tracefs_dir_inode_operations = {
.rmdir = tracefs_syscall_rmdir,
};
-struct inode *tracefs_get_inode(struct super_block *sb)
-{
- struct inode *inode = new_inode(sb);
- if (inode) {
- inode->i_ino = get_next_ino();
- inode->i_atime = inode->i_mtime =
inode_set_ctime_current(inode);
- }
- return inode;
-}
-
struct tracefs_mount_opts {
kuid_t uid;
kgid_t gid;
@@ -384,6 +378,7 @@ static void tracefs_dentry_iput(struct dentry *dentry,
struct inode *inode)
return;
ti = get_tracefs(inode);
+ BUG_ON(ti->magic != 20240823);
if (ti && ti->flags & TRACEFS_EVENT_INODE)
eventfs_set_ef_status_free(dentry);
iput(inode);
@@ -568,6 +563,18 @@ struct dentry *eventfs_end_creating(struct dentry *dentry)
return dentry;
}
+struct inode *tracefs_get_inode(struct super_block *sb)
+{
+ struct inode *inode = new_inode(sb);
+
+ BUG_ON(sb->s_op != &tracefs_super_operations);
+ if (inode) {
+ inode->i_ino = get_next_ino();
+ inode->i_atime = inode->i_mtime =
inode_set_ctime_current(inode);
+ }
+ return inode;
+}
+
/**
* tracefs_create_file - create a file in the tracefs filesystem
* @name: a pointer to a string containing the name of the file to create.
diff --git a/fs/tracefs/internal.h b/fs/tracefs/internal.h
index 69c2b1d87c46..9f6f303a9e58 100644
--- a/fs/tracefs/internal.h
+++ b/fs/tracefs/internal.h
@@ -9,6 +9,7 @@ enum {
struct tracefs_inode {
unsigned long flags;
void*private;
+ unsigned long magic;
struct inodevfs_inode;
};
Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot
On Wed, 29 May 2024 21:36:08 +0300 Ilkka Naulapää wrote: > applied your patch without others, so trace and panic there. > Screenshot attached. Also tested kernels backward and found out that Bah, it's still in an RCU callback, which doesn't tell us why a normal inode is being sent to the trace inode free list. > this trace bug first triggered on 6.6-rc1. Hmm, that's when eventfs was added. > > Let me know if you need more assistance with this. Let me make a debug patch (that crashes on this issue) for that kernel, and perhaps you could bisect it? Thanks! -- Steve
Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot
On Tue, 28 May 2024 07:51:30 +0300
Ilkka Naulapää wrote:
> yeah, the cache_from_obj tracing bug (without panic) has been
> displayed quite some time now - maybe even since 6.7.x or so. I could
> try checking a few versions back for this and try bisecting it if I
> can find when this started.
>
OK, so I don't think the commit your last bisect hit is the cause of
the bug. It added a delay (via RCU) and is causing the real bug to blow
up more.
Can you add this patch to v6.9.2 and hopefully it crashes in a better
location that we can find where the mixup happened.
You may need to add the other commit (too if this doesn't trigger.
Thanks,
-- Steve
diff --git a/fs/tracefs/inode.c b/fs/tracefs/inode.c
index 417c840e6403..7af3f696696d 100644
--- a/fs/tracefs/inode.c
+++ b/fs/tracefs/inode.c
@@ -50,6 +50,7 @@ static struct inode *tracefs_alloc_inode(struct super_block
*sb)
list_add_rcu(&ti->list, &tracefs_inodes);
spin_unlock_irqrestore(&tracefs_inode_lock, flags);
+ ti->magic = 20240823;
return &ti->vfs_inode;
}
@@ -66,6 +67,7 @@ static void tracefs_free_inode(struct inode *inode)
struct tracefs_inode *ti = get_tracefs(inode);
unsigned long flags;
+ BUG_ON(ti->magic != 20240823);
spin_lock_irqsave(&tracefs_inode_lock, flags);
list_del_rcu(&ti->list);
spin_unlock_irqrestore(&tracefs_inode_lock, flags);
@@ -271,16 +273,6 @@ static const struct inode_operations
tracefs_file_inode_operations = {
.setattr= tracefs_setattr,
};
-struct inode *tracefs_get_inode(struct super_block *sb)
-{
- struct inode *inode = new_inode(sb);
- if (inode) {
- inode->i_ino = get_next_ino();
- simple_inode_init_ts(inode);
- }
- return inode;
-}
-
struct tracefs_mount_opts {
kuid_t uid;
kgid_t gid;
@@ -448,6 +440,17 @@ static const struct super_operations
tracefs_super_operations = {
.show_options = tracefs_show_options,
};
+struct inode *tracefs_get_inode(struct super_block *sb)
+{
+ struct inode *inode = new_inode(sb);
+ BUG_ON(sb->s_op != &tracefs_super_operations);
+ if (inode) {
+ inode->i_ino = get_next_ino();
+ simple_inode_init_ts(inode);
+ }
+ return inode;
+}
+
/*
* It would be cleaner if eventfs had its own dentry ops.
*
diff --git a/fs/tracefs/internal.h b/fs/tracefs/internal.h
index f704d8348357..dda7d2708e30 100644
--- a/fs/tracefs/internal.h
+++ b/fs/tracefs/internal.h
@@ -16,6 +16,7 @@ struct tracefs_inode {
};
/* The below gets initialized with memset_after(ti, 0, vfs_inode) */
struct list_headlist;
+ unsigned long magic;
unsigned long flags;
void*private;
};
Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot
yeah, the cache_from_obj tracing bug (without panic) has been displayed quite some time now - maybe even since 6.7.x or so. I could try checking a few versions back for this and try bisecting it if I can find when this started. --Ilkka On Tue, May 28, 2024 at 1:31 AM Steven Rostedt wrote: > > On Fri, 24 May 2024 12:50:08 +0200 > "Linux regression tracking (Thorsten Leemhuis)" > wrote: > > > > - Affected Versions: Before kernel version 6.8.10, the bug caused a > > > quick display of a kernel trace dump before the shutdown/reboot > > > completed. Starting from version 6.8.10 and continuing into version > > > 6.9.0 and 6.9.1, this issue has escalated to a kernel panic, > > > preventing the shutdown or reboot from completing and leaving the > > > machine stuck. > > You state "Before kernel version 6.8.10, the bug caused ...". Does that > mean that a bug was happening before v6.8.10? But did not cause a panic? > > I just noticed your second screen shot from your report, and it has: > > "cache_from_obj: Wrong slab cache, tracefs_inode_cache but object is from > inode_cache" > > So somehow an tracefs_inode was allocated from the inode_cache and is > now being freed by the tracefs_inode logic? Did this happen before > 6.8.10? If so, this code could just be triggering an issue from an > unrelated bug. > > Thanks, > > -- Steve
Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot
I tried 6.10-rc1 and it still ends up to panic --Ilkka On Tue, May 28, 2024 at 12:44 AM Steven Rostedt wrote: > > On Mon, 27 May 2024 20:14:42 +0200 > Greg KH wrote: > > > On Mon, May 27, 2024 at 07:40:21PM +0300, Ilkka Naulapää wrote: > > > Hi Steven, > > > > > > I took some time and bisected the 6.8.9 - 6.8.10 and git gave the > > > panic inducing commit: > > > > > > 414fb08628143 (tracefs: Reset permissions on remount if permissions are > > > options) > > > > > > I reverted that commit to 6.9.2 and now it only serves the trace but > > > the panic is gone. But I can live with it. > > > > Steven, should we revert that? > > > > Or is there some other change that we should take to resolve this? > > > > Before we revert it (as it may be a bug in mainline), Ilkka, can you > test v6.10-rc1? If it exists there, it will let me know whether or not > I missed something. > > Thanks, > > -- Steve
Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot
On Fri, 24 May 2024 12:50:08 +0200 "Linux regression tracking (Thorsten Leemhuis)" wrote: > > - Affected Versions: Before kernel version 6.8.10, the bug caused a > > quick display of a kernel trace dump before the shutdown/reboot > > completed. Starting from version 6.8.10 and continuing into version > > 6.9.0 and 6.9.1, this issue has escalated to a kernel panic, > > preventing the shutdown or reboot from completing and leaving the > > machine stuck. You state "Before kernel version 6.8.10, the bug caused ...". Does that mean that a bug was happening before v6.8.10? But did not cause a panic? I just noticed your second screen shot from your report, and it has: "cache_from_obj: Wrong slab cache, tracefs_inode_cache but object is from inode_cache" So somehow an tracefs_inode was allocated from the inode_cache and is now being freed by the tracefs_inode logic? Did this happen before 6.8.10? If so, this code could just be triggering an issue from an unrelated bug. Thanks, -- Steve
Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot
On Mon, 27 May 2024 20:14:42 +0200 Greg KH wrote: > On Mon, May 27, 2024 at 07:40:21PM +0300, Ilkka Naulapää wrote: > > Hi Steven, > > > > I took some time and bisected the 6.8.9 - 6.8.10 and git gave the > > panic inducing commit: > > > > 414fb08628143 (tracefs: Reset permissions on remount if permissions are > > options) > > > > I reverted that commit to 6.9.2 and now it only serves the trace but > > the panic is gone. But I can live with it. > > Steven, should we revert that? > > Or is there some other change that we should take to resolve this? > Before we revert it (as it may be a bug in mainline), Ilkka, can you test v6.10-rc1? If it exists there, it will let me know whether or not I missed something. Thanks, -- Steve
Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot
On Mon, May 27, 2024 at 07:40:21PM +0300, Ilkka Naulapää wrote: > Hi Steven, > > I took some time and bisected the 6.8.9 - 6.8.10 and git gave the > panic inducing commit: > > 414fb08628143 (tracefs: Reset permissions on remount if permissions are > options) > > I reverted that commit to 6.9.2 and now it only serves the trace but > the panic is gone. But I can live with it. Steven, should we revert that? Or is there some other change that we should take to resolve this? thanks, greg k-h
Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot
Hi Steven,
I took some time and bisected the 6.8.9 - 6.8.10 and git gave the
panic inducing commit:
414fb08628143 (tracefs: Reset permissions on remount if permissions are options)
I reverted that commit to 6.9.2 and now it only serves the trace but
the panic is gone. But I can live with it.
--Ilkka
On Sun, May 26, 2024 at 8:42 PM Ilkka Naulapää wrote:
>
> hi,
>
> I took 6.9.2 and applied that 0bcfd9aa4dafa to it. Now the kernel is
> serving me both problems; the trace and the panic as the pic shows.
>
> > To understand this, did you do anything with tracing? Before shutting down,
> > is there anything in /sys/kernel/tracing/instances directory?
> > Were any of the files/directories permissions in /sys/kernel/tracing
> > changed?
>
> And to answer your question, I did not do any tracing or so and the
> /sys/kernel/tracing is empty.
> Just plain boot-up, no matter if in full desktop or in bare rescue
> mode, ends up the same way.
>
> --Ilkka
>
> On Fri, May 24, 2024 at 8:19 PM Steven Rostedt wrote:
> >
> > On Fri, 24 May 2024 12:50:08 +0200
> > "Linux regression tracking (Thorsten Leemhuis)"
> > wrote:
> >
> > > > - Affected Versions: Before kernel version 6.8.10, the bug caused a
> > > > quick display of a kernel trace dump before the shutdown/reboot
> > > > completed. Starting from version 6.8.10 and continuing into version
> > > > 6.9.0 and 6.9.1, this issue has escalated to a kernel panic,
> > > > preventing the shutdown or reboot from completing and leaving the
> > > > machine stuck.
> >
> > Ah, I bet it was this commit: baa23a8d4360d ("tracefs: Reset permissions on
> > remount if permissions are options"), which added a "iput" callback to the
> > dentry without calling iput, leaving stale inodes around.
> >
> > This is fixed with:
> >
> > 0bcfd9aa4dafa ("tracefs: Clear EVENT_INODE flag in tracefs_drop_inode()")
> >
> > Try adding just that patch. It will at least make it go back to what was
> > happening before 6.8.10 (I hope!).
> >
> > -- Steve
Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot
On Fri, 24 May 2024 12:50:08 +0200
"Linux regression tracking (Thorsten Leemhuis)"
wrote:
> > - Affected Versions: Before kernel version 6.8.10, the bug caused a
> > quick display of a kernel trace dump before the shutdown/reboot
> > completed. Starting from version 6.8.10 and continuing into version
> > 6.9.0 and 6.9.1, this issue has escalated to a kernel panic,
> > preventing the shutdown or reboot from completing and leaving the
> > machine stuck.
Ah, I bet it was this commit: baa23a8d4360d ("tracefs: Reset permissions on
remount if permissions are options"), which added a "iput" callback to the
dentry without calling iput, leaving stale inodes around.
This is fixed with:
0bcfd9aa4dafa ("tracefs: Clear EVENT_INODE flag in tracefs_drop_inode()")
Try adding just that patch. It will at least make it go back to what was
happening before 6.8.10 (I hope!).
-- Steve
Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot
On Fri, 24 May 2024 12:50:08 +0200
"Linux regression tracking (Thorsten Leemhuis)"
wrote:
> [CCing a few people]
>
Thanks for the Cc.
> On 24.05.24 12:31, Ilkka Naulapää wrote:
> >
> > I have encountered a critical bug in the Linux vanilla kernel that
> > leads to a kernel panic during the shutdown or reboot process. The
> > issue arises after all services, including `journald`, have been
> > stopped. As a result, the machine fails to complete the shutdown or
> > reboot procedure, effectively causing the system to hang and not shut
> > down or reboot.
To understand this, did you do anything with tracing? Before shutting down,
is there anything in /sys/kernel/tracing/instances directory?
Were any of the files/directories permissions in /sys/kernel/tracing changed?
>
> Thx for the report. Not my area of expertise, so take this with a gain
> of salt. But given the versions your mention in your report and the
> screenshot that mentioned tracefs_free_inode I suspect this is caused by
> baa23a8d4360d ("tracefs: Reset permissions on remount if permissions are
> options"). A few fixes for it will soon hit mainline and are meant to be
> backported to affected stable trees:
>
> https://lore.kernel.org/all/20240523212406.254317...@goodmis.org/
> https://lore.kernel.org/all/20240523174419.1e588...@gandalf.local.home/
>
> You might want to try them – or recheck once they hit the stable trees
> you are about. If they don't work, please report back.
There's been quite a bit of updates in this code, but this looks new to me.
I have more fixes that were just pulled by Linus today.
https://git.kernel.org/torvalds/c/0eb03c7e8e2a4cc3653eb5eeb2d2001182071215
I'm not sure how relevant that is for this. But if you can reproduce it
with that commit, then this is a new bug.
-- Steve
Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot
[CCing a few people]
On 24.05.24 12:31, Ilkka Naulapää wrote:
>
> I have encountered a critical bug in the Linux vanilla kernel that
> leads to a kernel panic during the shutdown or reboot process. The
> issue arises after all services, including `journald`, have been
> stopped. As a result, the machine fails to complete the shutdown or
> reboot procedure, effectively causing the system to hang and not shut
> down or reboot.
Thx for the report. Not my area of expertise, so take this with a gain
of salt. But given the versions your mention in your report and the
screenshot that mentioned tracefs_free_inode I suspect this is caused by
baa23a8d4360d ("tracefs: Reset permissions on remount if permissions are
options"). A few fixes for it will soon hit mainline and are meant to be
backported to affected stable trees:
https://lore.kernel.org/all/20240523212406.254317...@goodmis.org/
https://lore.kernel.org/all/20240523174419.1e588...@gandalf.local.home/
You might want to try them – or recheck once they hit the stable trees
you are about. If they don't work, please report back.
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.
> Here are the details of the issue:
>
> - Affected Versions: Before kernel version 6.8.10, the bug caused a
> quick display of a kernel trace dump before the shutdown/reboot
> completed. Starting from version 6.8.10 and continuing into version
> 6.9.0 and 6.9.1, this issue has escalated to a kernel panic,
> preventing the shutdown or reboot from completing and leaving the
> machine stuck.
>
> - Symptoms:
> - In normal shutdown/reboot scenarios, the kernel trace dump briefly
> appears as the last message on the screen.
> - In rescue mode, the kernel panic message is displayed. Normally it
> is not shown.
>
> Since `journald` is stopped before this issue occurs, no textual logs
> are available. However, I have captured two pictures illustrating
> these related issues, which I am attaching to this email for your
> reference. Also added my custom kernel config.
>
> Thank you for your attention to this matter. Please let me know if any
> additional information is required to assist in diagnosing and
> resolving this bug.
>
> Best regards,
>
> Ilkka Naulapää
