Re: [yocto] ROOTFS_RPM_DEBUG undocumented

[Alexander Kanavin]

On 05/08/2018 12:33 PM, Peter Kjellerstedt wrote:

FYI, the already applied patch only solves half the problem. I have an
updated version that solves all of it that I am preparing for publication.
I only need to make some verifications first...


I just found that out as well. Lua is not relevant, we don't even enable 
or build it. Anyway, I've been fighting*scratch*convincing upstream all 
morning that fcntl bombs are a real problem under Docker, you are 
welcome to join :)


https://bugzilla.redhat.com/show_bug.cgi?id=1537564

Alex
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] ROOTFS_RPM_DEBUG undocumented

[Peter Kjellerstedt]

> -Original Message-
> From: yocto-boun...@yoctoproject.org [mailto:yocto-
> boun...@yoctoproject.org] On Behalf Of Alexander Kanavin
> Sent: den 7 maj 2018 12:12
> To: Mirza Krak <mirza.k...@gmail.com>
> Cc: Yocto list discussion <yocto@yoctoproject.org>
> Subject: Re: [yocto] ROOTFS_RPM_DEBUG undocumented
> 
> On 05/07/2018 12:03 PM, Mirza Krak wrote:
> > I have also been bitten by rpm performance issues in docker and glad
> > that I found this. I am in the process of verifying above (on pyro
> > branch).
> >
> > Is this patch on its way to pyro and rocko branches?
> 
> As far as I know, no. You are welcome to backport and submit.
> 
> Alex

FYI, the already applied patch only solves half the problem. I have an 
updated version that solves all of it that I am preparing for publication. 
I only need to make some verifications first...

//Peter

-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] ROOTFS_RPM_DEBUG undocumented

[Alexander Kanavin]

On 05/07/2018 12:03 PM, Mirza Krak wrote:

I have also been bitten by rpm performance issues in docker and glad
that I found this. I am in the process of verifying above (on pyro
branch).

Is this patch on its way to pyro and rocko branches?


As far as I know, no. You are welcome to backport and submit.

Alex
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] ROOTFS_RPM_DEBUG undocumented

[Mirza Krak]

On 4 May 2018 at 14:41, Alexander Kanavin
 wrote:
> On 05/04/2018 03:42 PM, Paulo Neves wrote:
>>
>> I will propose a patch with a default bb.debug, and always have the
>> verbose on dnf. Let's see how it affects the performance. I cannot
>> test this in a docker container because of the problems described
>> below:
>>
>> In the mean time I found what was happening with my slow do_rootfs. I
>> was running bitbake with RPM packaging in a docker instance. It seems
>> there is a problem with RPM/dnf in bitbake where it is extremely slow
>> inside a docker container (1 file per second). After changing to IPK
>> packages everything went smoothly.
>>
>> Others have had the same problem[1]. I will later report this to
>> openembedded mailing list as this effectively makes RPM unusable in a
>> container. I will also provide a Dockerfile to reproduce this problem.
>>
>> [1] https://github.com/moby/moby/issues/23137#issuecomment-359097008
>
>
> No need to report. We have already fixed this and reported upstream:
>
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-devtools/rpm/files/0001-Revert-Set-FD_CLOEXEC-on-opened-files-before-exec-fr.patch

I have also been bitten by rpm performance issues in docker and glad
that I found this. I am in the process of verifying above (on pyro
branch).

Is this patch on its way to pyro and rocko branches?

-- 
Med Vänliga Hälsningar / Best Regards

Mirza Krak
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] ROOTFS_RPM_DEBUG undocumented

[Paulo Neves]

Ouch you are right. Damn me for using pyro still?
After applying this patch I will report on the remaining issues.

On Fri, May 4, 2018 at 2:41 PM, Alexander Kanavin
 wrote:
> On 05/04/2018 03:42 PM, Paulo Neves wrote:
>>
>> I will propose a patch with a default bb.debug, and always have the
>> verbose on dnf. Let's see how it affects the performance. I cannot
>> test this in a docker container because of the problems described
>> below:
>>
>> In the mean time I found what was happening with my slow do_rootfs. I
>> was running bitbake with RPM packaging in a docker instance. It seems
>> there is a problem with RPM/dnf in bitbake where it is extremely slow
>> inside a docker container (1 file per second). After changing to IPK
>> packages everything went smoothly.
>>
>> Others have had the same problem[1]. I will later report this to
>> openembedded mailing list as this effectively makes RPM unusable in a
>> container. I will also provide a Dockerfile to reproduce this problem.
>>
>> [1] https://github.com/moby/moby/issues/23137#issuecomment-359097008
>
>
> No need to report. We have already fixed this and reported upstream:
>
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-devtools/rpm/files/0001-Revert-Set-FD_CLOEXEC-on-opened-files-before-exec-fr.patch
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1537564
>
>
> Alex
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] ROOTFS_RPM_DEBUG undocumented

[Alexander Kanavin]

On 05/04/2018 03:50 PM, Scott Rifenbark wrote:
If ROOTFS_RPM_DEBUG should be a documented variable in the Yocto Project 
Reference Manual, could someone please provide me with some base 
explanation of the variable and any usage specifics?


No need, we're going to remove it.

Alex
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] ROOTFS_RPM_DEBUG undocumented

[Scott Rifenbark]

If ROOTFS_RPM_DEBUG should be a documented variable in the Yocto Project
Reference Manual, could someone please provide me with some base
explanation of the variable and any usage specifics?

Thanks,
Scott

On Fri, May 4, 2018 at 5:41 AM, Alexander Kanavin <
alexander.kana...@linux.intel.com> wrote:

> On 05/04/2018 03:42 PM, Paulo Neves wrote:
>
>> I will propose a patch with a default bb.debug, and always have the
>> verbose on dnf. Let's see how it affects the performance. I cannot
>> test this in a docker container because of the problems described
>> below:
>>
>> In the mean time I found what was happening with my slow do_rootfs. I
>> was running bitbake with RPM packaging in a docker instance. It seems
>> there is a problem with RPM/dnf in bitbake where it is extremely slow
>> inside a docker container (1 file per second). After changing to IPK
>> packages everything went smoothly.
>>
>> Others have had the same problem[1]. I will later report this to
>> openembedded mailing list as this effectively makes RPM unusable in a
>> container. I will also provide a Dockerfile to reproduce this problem.
>>
>> [1] https://github.com/moby/moby/issues/23137#issuecomment-359097008
>>
>
> No need to report. We have already fixed this and reported upstream:
>
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/rec
> ipes-devtools/rpm/files/0001-Revert-Set-FD_CLOEXEC-on-opene
> d-files-before-exec-fr.patch
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1537564
>
>
>
> Alex
> --
> ___
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] ROOTFS_RPM_DEBUG undocumented

[Alexander Kanavin]

On 05/04/2018 03:41 PM, Alexander Kanavin wrote:


https://bugzilla.redhat.com/show_bug.cgi?id=1537564


Obviously you and anyone else affected need to make noise in the redhat 
bugzilla, otherwise they're likely to keep this low priority.


Alex
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] ROOTFS_RPM_DEBUG undocumented

[Alexander Kanavin]

On 05/04/2018 03:42 PM, Paulo Neves wrote:

I will propose a patch with a default bb.debug, and always have the
verbose on dnf. Let's see how it affects the performance. I cannot
test this in a docker container because of the problems described
below:

In the mean time I found what was happening with my slow do_rootfs. I
was running bitbake with RPM packaging in a docker instance. It seems
there is a problem with RPM/dnf in bitbake where it is extremely slow
inside a docker container (1 file per second). After changing to IPK
packages everything went smoothly.

Others have had the same problem[1]. I will later report this to
openembedded mailing list as this effectively makes RPM unusable in a
container. I will also provide a Dockerfile to reproduce this problem.

[1] https://github.com/moby/moby/issues/23137#issuecomment-359097008


No need to report. We have already fixed this and reported upstream:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-devtools/rpm/files/0001-Revert-Set-FD_CLOEXEC-on-opened-files-before-exec-fr.patch

https://bugzilla.redhat.com/show_bug.cgi?id=1537564


Alex
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] ROOTFS_RPM_DEBUG undocumented

[Paulo Neves]

I will propose a patch with a default bb.debug, and always have the
verbose on dnf. Let's see how it affects the performance. I cannot
test this in a docker container because of the problems described
below:

In the mean time I found what was happening with my slow do_rootfs. I
was running bitbake with RPM packaging in a docker instance. It seems
there is a problem with RPM/dnf in bitbake where it is extremely slow
inside a docker container (1 file per second). After changing to IPK
packages everything went smoothly.

Others have had the same problem[1]. I will later report this to
openembedded mailing list as this effectively makes RPM unusable in a
container. I will also provide a Dockerfile to reproduce this problem.

[1] https://github.com/moby/moby/issues/23137#issuecomment-359097008

Paulo Neves

On Thu, May 3, 2018 at 3:03 PM, Alexander Kanavin
 wrote:
> On 05/03/2018 01:42 PM, Paulo Neves wrote:
>>
>> # Add ROOTFS_RPM_DEBUG to the documentation;
>
>
> I'd rather get rid of it, the less variables the better :)
>
>> # Detect if we are running with debug output and enable the debugging
>> output. This is the most elegant solution but I do not know how to
>> detect if debug log level is turned on;
>
>
> I don't think you are meant to detect that; bitbake's debug level is not
> meant to affect what is being printed, but merely whether it's printed or
> discarded.
>
>> # Have dnf always print in verbose mode and print the output to
>> bb.debug instead of bb.note.
>
>
> I think this is the best solution actually.
>
>> I am happy to provide a patch upon decision or suggestions.
>
>
> Thank you :) Do measure the performance before and after as we don't want to
> introduce a significant slowdown here.
>
>
> Alex
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] ROOTFS_RPM_DEBUG undocumented

[Alexander Kanavin]

On 05/03/2018 04:37 PM, Peter Kjellerstedt wrote:

# Have dnf always print in verbose mode and print the output to
bb.debug instead of bb.note.


I think this is the best solution actually.


I disagree with changing bb.note to bb.debug for this. It is very good
to be able to look in log.do_rootfs to see what dnf has done (we have
ROOTFS_RPM_DEBUG set by default). Changing the output to debug level
would mean having to run bitbake with -D, which is anything but
desirable.


If I'm reading the bitbake --help right, bb.debug (and bb.note) are 
always written to the log; setting -D level or -v only affects what is 
printed on stdout.



Alex
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] ROOTFS_RPM_DEBUG undocumented

[Peter Kjellerstedt]

> -Original Message-
> From: yocto-boun...@yoctoproject.org [mailto:yocto-
> boun...@yoctoproject.org] On Behalf Of Alexander Kanavin
> Sent: den 3 maj 2018 15:04
> To: Paulo Neves <ptsne...@gmail.com>; Yocto list discussion
> <yocto@yoctoproject.org>
> Subject: Re: [yocto] ROOTFS_RPM_DEBUG undocumented
> 
> On 05/03/2018 01:42 PM, Paulo Neves wrote:
> > # Add ROOTFS_RPM_DEBUG to the documentation;
> 
> I'd rather get rid of it, the less variables the better :)
> 
> > # Detect if we are running with debug output and enable the debugging
> > output. This is the most elegant solution but I do not know how to
> > detect if debug log level is turned on;
> 
> I don't think you are meant to detect that; bitbake's debug level is
> not
> meant to affect what is being printed, but merely whether it's printed
> or discarded.
> 
> > # Have dnf always print in verbose mode and print the output to
> > bb.debug instead of bb.note.
> 
> I think this is the best solution actually.

I disagree with changing bb.note to bb.debug for this. It is very good 
to be able to look in log.do_rootfs to see what dnf has done (we have 
ROOTFS_RPM_DEBUG set by default). Changing the output to debug level 
would mean having to run bitbake with -D, which is anything but 
desirable.

> > I am happy to provide a patch upon decision or suggestions.
> 
> Thank you :) Do measure the performance before and after as we don't
> want to introduce a significant slowdown here.
> 
> Alex

//Peter

-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] ROOTFS_RPM_DEBUG undocumented

[Alexander Kanavin]

On 05/03/2018 01:42 PM, Paulo Neves wrote:

# Add ROOTFS_RPM_DEBUG to the documentation;


I'd rather get rid of it, the less variables the better :)


# Detect if we are running with debug output and enable the debugging
output. This is the most elegant solution but I do not know how to
detect if debug log level is turned on;


I don't think you are meant to detect that; bitbake's debug level is not 
meant to affect what is being printed, but merely whether it's printed 
or discarded.



# Have dnf always print in verbose mode and print the output to
bb.debug instead of bb.note.


I think this is the best solution actually.


I am happy to provide a patch upon decision or suggestions.


Thank you :) Do measure the performance before and after as we don't 
want to introduce a significant slowdown here.



Alex
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto