Re: 4.4.0 - no space left with >1.7 TB free space left

[Roman Mamedov]

On Fri, 8 Apr 2016 16:53:32 +0500
Roman Mamedov  wrote:

> On Fri, 08 Apr 2016 20:36:26 +0900
> Tomasz Chmielewski  wrote:
> 
> > On 2016-02-08 20:24, Roman Mamedov wrote:
> > 
> > >> Linux 4.4.0 - btrfs is mainly used to host lots of test containers,
> > >> often snapshots, and at times, there is heavy IO in many of them for
> > >> extended periods of time. btrfs is on HDDs.
> > >> 
> > >> 
> > >> Every few days I'm getting "no space left" in a container running 
> > >> mongo
> > >> 3.2.1 database. Interestingly, haven't seen this issue in containers
> > >> with MySQL. All databases have chattr +C set on their directories.
> > > 
> > > Hello,
> > > 
> > > Do you snapshot the parent subvolume which holds the databases? Can you
> > > correlate that perhaps ENOSPC occurs at the time of snapshotting? If 
> > > yes, then
> > > you should try the patch https://patchwork.kernel.org/patch/7967161/
> > > 
> > > (Too bad this was not included into 4.4.1.)
> > 
> > By the way - was it included in any later kernel? I'm running 4.4.5 on 
> > that server, but still hitting the same issue.
> 
> It's not in 4.4.6 either. I don't know why it doesn't get included, or what
> we need to do. Last time I asked, it was queued:
> http://www.spinics.net/lists/linux-btrfs/msg52478.html
> But maybe that meant 4.5 or 4.6 only? While the bug is affecting people on
> 4.4.x today.

This got applied now in 4.4.21, thanks.

-- 
With respect,
Roman


pgpqOFdEk406P.pgp
Description: OpenPGP digital signature

Re: 4.4.0 - no space left with >1.7 TB free space left

[Tomasz Chmielewski]

On 2016-05-12 15:03, Tomasz Chmielewski wrote:


FYI, I'm still getting this with 4.5.3, which probably means the fix
was not yet included ("No space left" at snapshot time):

/var/log/postgresql/postgresql-9.3-main.log:2016-05-11 06:06:10 UTC
LOG:  could not close temporary statistics file
"pg_stat_tmp/db_0.tmp": No space left on device
/var/log/postgresql/postgresql-9.3-main.log:2016-05-11 06:06:10 UTC
LOG:  could not close temporary statistics file
"pg_stat_tmp/global.tmp": No space left on device
/var/log/postgresql/postgresql-9.3-main.log:2016-05-11 06:06:10 UTC
LOG:  could not close temporary statistics file
"pg_stat_tmp/db_0.tmp": No space left on device
/var/log/postgresql/postgresql-9.3-main.log:2016-05-11 06:06:10 UTC
LOG:  could not close temporary statistics file
"pg_stat_tmp/global.tmp": No space left on device


I've tried mounting with space_cache=v2, but it didn't help.


On the good side, I see it's in 4.6-rc7.


Tomasz Chmielewski
http://wpkg.org

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 4.4.0 - no space left with >1.7 TB free space left

[Tomasz Chmielewski]

On 2016-04-08 20:53, Roman Mamedov wrote:


> Do you snapshot the parent subvolume which holds the databases? Can you
> correlate that perhaps ENOSPC occurs at the time of snapshotting? If
> yes, then
> you should try the patch https://patchwork.kernel.org/patch/7967161/
>
> (Too bad this was not included into 4.4.1.)

By the way - was it included in any later kernel? I'm running 4.4.5 on
that server, but still hitting the same issue.


It's not in 4.4.6 either. I don't know why it doesn't get included, or 
what

we need to do. Last time I asked, it was queued:
http://www.spinics.net/lists/linux-btrfs/msg52478.html
But maybe that meant 4.5 or 4.6 only? While the bug is affecting people 
on

4.4.x today.


FYI, I'm still getting this with 4.5.3, which probably means the fix was 
not yet included ("No space left" at snapshot time):


/var/log/postgresql/postgresql-9.3-main.log:2016-05-11 06:06:10 UTC LOG: 
 could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No 
space left on device
/var/log/postgresql/postgresql-9.3-main.log:2016-05-11 06:06:10 UTC LOG: 
 could not close temporary statistics file "pg_stat_tmp/global.tmp": No 
space left on device
/var/log/postgresql/postgresql-9.3-main.log:2016-05-11 06:06:10 UTC LOG: 
 could not close temporary statistics file "pg_stat_tmp/db_0.tmp": No 
space left on device
/var/log/postgresql/postgresql-9.3-main.log:2016-05-11 06:06:10 UTC LOG: 
 could not close temporary statistics file "pg_stat_tmp/global.tmp": No 
space left on device



I've tried mounting with space_cache=v2, but it didn't help.



Tomasz Chmielewski
http://wpkg.org

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 4.4.0 - no space left with >1.7 TB free space left

[Duncan]

Roman Mamedov posted on Fri, 08 Apr 2016 16:53:32 +0500 as excerpted:

> It's not in 4.4.6 either. I don't know why it doesn't get included, or
> what we need to do. Last time I asked, it was queued:
> http://www.spinics.net/lists/linux-btrfs/msg52478.html But maybe that
> meant 4.5 or 4.6 only? While the bug is affecting people on 4.4.x today.

Patches must make it to the current development kernel before they're 
eligible for stable.  Additionally, they need to be cced to stable as 
well, in ordered to be queued there.

So check 4.5 and 4.6-rc.  If it's in neither of those, it's not going to 
be in stable yet.  Once it's in the development kernel, see if it was cced 
to stable and if needed, ask the author and btrfs devs to cc it to stable.

Tho sometimes stable can get a backlog as well.  I know earlier this year 
they were dealing with one, but I follow release or development, not 
stable, and don't know what stable's current status is.

If it gets to stable, and it wasn't for a bug introduced /after/ 4.4, it 
should eventually get into 4.4, as that's an LTS kernel.  But it might 
take awhile, as the above discussion hints.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 4.4.0 - no space left with >1.7 TB free space left

[Tomasz Chmielewski]

On 2016-04-08 20:53, Roman Mamedov wrote:


> Do you snapshot the parent subvolume which holds the databases? Can you
> correlate that perhaps ENOSPC occurs at the time of snapshotting? If
> yes, then
> you should try the patch https://patchwork.kernel.org/patch/7967161/
>
> (Too bad this was not included into 4.4.1.)

By the way - was it included in any later kernel? I'm running 4.4.5 on
that server, but still hitting the same issue.


It's not in 4.4.6 either. I don't know why it doesn't get included, or 
what

we need to do. Last time I asked, it was queued:
http://www.spinics.net/lists/linux-btrfs/msg52478.html
But maybe that meant 4.5 or 4.6 only? While the bug is affecting people 
on

4.4.x today.


Does it mean 4.5 also doesn't have it yet?


Tomasz Chmielewski
http://wpkg.org

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 4.4.0 - no space left with >1.7 TB free space left

[Roman Mamedov]

On Fri, 08 Apr 2016 20:36:26 +0900
Tomasz Chmielewski  wrote:

> On 2016-02-08 20:24, Roman Mamedov wrote:
> 
> >> Linux 4.4.0 - btrfs is mainly used to host lots of test containers,
> >> often snapshots, and at times, there is heavy IO in many of them for
> >> extended periods of time. btrfs is on HDDs.
> >> 
> >> 
> >> Every few days I'm getting "no space left" in a container running 
> >> mongo
> >> 3.2.1 database. Interestingly, haven't seen this issue in containers
> >> with MySQL. All databases have chattr +C set on their directories.
> > 
> > Hello,
> > 
> > Do you snapshot the parent subvolume which holds the databases? Can you
> > correlate that perhaps ENOSPC occurs at the time of snapshotting? If 
> > yes, then
> > you should try the patch https://patchwork.kernel.org/patch/7967161/
> > 
> > (Too bad this was not included into 4.4.1.)
> 
> By the way - was it included in any later kernel? I'm running 4.4.5 on 
> that server, but still hitting the same issue.

It's not in 4.4.6 either. I don't know why it doesn't get included, or what
we need to do. Last time I asked, it was queued:
http://www.spinics.net/lists/linux-btrfs/msg52478.html
But maybe that meant 4.5 or 4.6 only? While the bug is affecting people on
4.4.x today.

Thanks

-- 
With respect,
Roman


pgpxDETSxWzY9.pgp
Description: OpenPGP digital signature

Re: 4.4.0 - no space left with >1.7 TB free space left

[Tomasz Chmielewski]

On 2016-02-08 20:24, Roman Mamedov wrote:


Linux 4.4.0 - btrfs is mainly used to host lots of test containers,
often snapshots, and at times, there is heavy IO in many of them for
extended periods of time. btrfs is on HDDs.


Every few days I'm getting "no space left" in a container running 
mongo

3.2.1 database. Interestingly, haven't seen this issue in containers
with MySQL. All databases have chattr +C set on their directories.


Hello,

Do you snapshot the parent subvolume which holds the databases? Can you
correlate that perhaps ENOSPC occurs at the time of snapshotting? If 
yes, then

you should try the patch https://patchwork.kernel.org/patch/7967161/

(Too bad this was not included into 4.4.1.)


By the way - was it included in any later kernel? I'm running 4.4.5 on 
that server, but still hitting the same issue.



Tomasz Chmielewski
http://wpkg.org

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 4.4.0 - no space left with >1.7 TB free space left

[Tomasz Chmielewski]

On 2016-02-08 20:24, Roman Mamedov wrote:

On Mon, 08 Feb 2016 18:22:34 +0900
Tomasz Chmielewski  wrote:


Linux 4.4.0 - btrfs is mainly used to host lots of test containers,
often snapshots, and at times, there is heavy IO in many of them for
extended periods of time. btrfs is on HDDs.


Every few days I'm getting "no space left" in a container running 
mongo

3.2.1 database. Interestingly, haven't seen this issue in containers
with MySQL. All databases have chattr +C set on their directories.


Hello,

Do you snapshot the parent subvolume which holds the databases? Can you
correlate that perhaps ENOSPC occurs at the time of snapshotting?


Not sure.

With the last error, a snapshot was made at around 06:06, while "no 
space left" was reported on 06:14. Suspiciously close to each other, but 
still, a few minutes away.


Unfortunately I don't have error log for previous cases.



If yes, then
you should try the patch https://patchwork.kernel.org/patch/7967161/

(Too bad this was not included into 4.4.1.)


I'll keep an eye on it, thanks.


Tomasz Chmielewski
http://www.ptraveler.com

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 4.4.0 - no space left with >1.7 TB free space left

[Roman Mamedov]

On Mon, 08 Feb 2016 21:15:38 +0900
Tomasz Chmielewski  wrote:

> With the last error, a snapshot was made at around 06:06
> "no space left" was reported on 06:14.

If you mean the log that you have posted in your original message, the ENOSPC
happened at 06:06 and 14 seconds, not 06:14.

-- 
With respect,
Roman


pgpGvudF_ZoCz.pgp
Description: OpenPGP digital signature

Re: 4.4.0 - no space left with >1.7 TB free space left

[Roman Mamedov]

On Mon, 08 Feb 2016 18:22:34 +0900
Tomasz Chmielewski  wrote:

> Linux 4.4.0 - btrfs is mainly used to host lots of test containers, 
> often snapshots, and at times, there is heavy IO in many of them for 
> extended periods of time. btrfs is on HDDs.
> 
> 
> Every few days I'm getting "no space left" in a container running mongo 
> 3.2.1 database. Interestingly, haven't seen this issue in containers 
> with MySQL. All databases have chattr +C set on their directories.

Hello,

Do you snapshot the parent subvolume which holds the databases? Can you
correlate that perhaps ENOSPC occurs at the time of snapshotting? If yes, then
you should try the patch https://patchwork.kernel.org/patch/7967161/

(Too bad this was not included into 4.4.1.)

-- 
With respect,
Roman


pgpQUIIInnFRo.pgp
Description: OpenPGP digital signature