Re: [OE-core] [PATCH] package: Fix PACKAGELOCK handling

2018-04-20 Thread Martin Jansa 
Now with pseudo log files a bit smaller this package-output.lock seems to
be most common "issue" reported with do_qa_pseudo.

e.g.:
libao-1.2.0: This /work/i586-oe-linux-musl/libao/1.2.0-r0/pseudo/pseudo.log
indicates 16 errors [qa-pseudo]
see grep -e '^creat ignored for existing file' -e '^creat for.*replaces
existing'
/home/jenkins/oe/world/shr-core/tmpfs/work/i586-oe-linux-musl/libao/1.2.0-r0/pseudo/pseudo.log
or grep -v '^path mismatch'
/home/jenkins/oe/world/shr-core/tmpfs/work/i586-oe-linux-musl/libao/1.2.0-r0/pseudo/pseudo.log:
debug_logfile: fd 2
pid 4165 [parent 4160], doing new pid setup and server start
Setup complete, sending SIGUSR1 to pid 4160.
db cleanup for server shutdown, 01:43:56.829
memory-to-file backup complete, 01:43:56.830.
db cleanup finished, 01:43:56.830
debug_logfile: fd 2
pid 11266 [parent 11265], doing new pid setup and server start
Setup complete, sending SIGUSR1 to pid 11265.
inode mismatch:
'/home/jenkins/oe/world/shr-core/tmpfs/sysroots/package-output.lock' ino
346594 in db, 347812 in request.
creat ignored for existing file
'/home/jenkins/oe/world/shr-core/tmpfs/sysroots/package-output.lock'.
inode mismatch:
'/home/jenkins/oe/world/shr-core/tmpfs/sysroots/package-output.lock' ino
346594 in db, 347812 in request.
inode mismatch:
'/home/jenkins/oe/world/shr-core/tmpfs/sysroots/package-output.lock' ino
346594 in db, 347812 in request.
inode mismatch:
'/home/jenkins/oe/world/shr-core/tmpfs/sysroots/package-output.lock' ino
346594 in db, 347812 in request.
inode mismatch:
'/home/jenkins/oe/world/shr-core/tmpfs/sysroots/package-output.lock' ino
346594 in db, 347812 in request.
inode mismatch:
'/home/jenkins/oe/world/shr-core/tmpfs/sysroots/package-output.lock' ino
346594 in db, 347812 in request.
 [qa-pseudo]

shown in last world build:
http://jenkins.nas-admin.org/view/OE/job/oe_world_qemux86/760/console

I have no idea how to resolve this one as it needs to be global and shared
between many pseudo databases.

Any idea?

On Fri, Apr 20, 2018 at 9:25 AM, Richard Purdie <
richard.pur...@linuxfoundation.org> wrote:

> PACKAGELOCK is there to protect readers of PKGDATA_DIR from writes and
> files
> changing whilst they're being read. With various changes to the codebase,
> the lock code has become confused as the files are now written by the
> sstate
> code in do_packagedata, not in do_package directly any longer.
>
> This change cleans up the code so read sites take the shared lock
> (anything in
> do_package), write sites take the full lock (do_packagedata sstate).
>
> The lock from do_package sstate is no longer needed since it doesn't write
> outside WORKDIR.
>
> Signed-off-by: Richard Purdie 
> ---
>  meta/classes/package.bbclass | 11 ++-
>  1 file changed, 6 insertions(+), 5 deletions(-)
>
> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
> index 0436d91..e03d450 100644
> --- a/meta/classes/package.bbclass
> +++ b/meta/classes/package.bbclass
> @@ -1615,7 +1615,7 @@ python package_do_shlibs() {
>  shlibswork_dir = d.getVar('SHLIBSWORKDIR')
>
>  # Take shared lock since we're only reading, not writing
> -lf = bb.utils.lockfile(d.expand("${PACKAGELOCK}"))
> +lf = bb.utils.lockfile(d.expand("${PACKAGELOCK}"), True)
>
>  def linux_so(file, needed, sonames, renames, pkgver):
>  needs_ldconfig = False
> @@ -1900,7 +1900,7 @@ python package_do_pkgconfig () {
>  pkgconfig_needed[pkg] += exp.replace(',',
> ' ').split()
>
>  # Take shared lock since we're only reading, not writing
> -lf = bb.utils.lockfile(d.expand("${PACKAGELOCK}"))
> +lf = bb.utils.lockfile(d.expand("${PACKAGELOCK}"), True)
>
>  for pkg in packages.split():
>  pkgs_file = os.path.join(shlibswork_dir, pkg + ".pclist")
> @@ -2233,11 +2233,9 @@ do_package[dirs] = "${SHLIBSWORKDIR} ${PKGDESTWORK}
> ${D}"
>  do_package[vardeps] += "${PACKAGEBUILDPKGD} ${PACKAGESPLITFUNCS}
> ${PACKAGEFUNCS} ${@gen_packagevar(d)}"
>  addtask package after do_install
>
> -PACKAGELOCK = "${STAGING_DIR}/package-output.lock"
>  SSTATETASKS += "do_package"
>  do_package[cleandirs] = "${PKGDEST} ${PKGDESTWORK}"
>  do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST} ${PKGDESTWORK}"
> -do_package[sstate-lockfile-shared] = "${PACKAGELOCK}"
>  do_package_setscene[dirs] = "${STAGING_DIR}"
>
>  python do_package_setscene () {
> @@ -2252,9 +2250,12 @@ do_packagedata () {
>  addtask packagedata before do_build after do_package
>
>  SSTATETASKS += "do_packagedata"
> +# PACKAGELOCK protects readers of PKGDATA_DIR against writes
> +# whilst code is reading in do_package
> +PACKAGELOCK = "${STAGING_DIR}/package-output.lock"
>  do_packagedata[sstate-inputdirs] = "${PKGDESTWORK}"
>  do_packagedata[sstate-outputdirs] = "${PKGDATA_DIR}"
> -do_packagedata[sstate-lockfile-shared] = "${PACKAGELOCK}"
> +do_packagedata[sstate-lockfile] = "${PACKAGELOCK}"
>  do_packagedata[stamp-extra-info] = "${MACHINE_ARCH}"
>
>

[OE-core] [PATCH] package: Fix PACKAGELOCK handling

2018-04-20 Thread Richard Purdie 
PACKAGELOCK is there to protect readers of PKGDATA_DIR from writes and files
changing whilst they're being read. With various changes to the codebase,
the lock code has become confused as the files are now written by the sstate
code in do_packagedata, not in do_package directly any longer.

This change cleans up the code so read sites take the shared lock (anything in
do_package), write sites take the full lock (do_packagedata sstate).

The lock from do_package sstate is no longer needed since it doesn't write
outside WORKDIR.

Signed-off-by: Richard Purdie 
---
 meta/classes/package.bbclass | 11 ++-
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 0436d91..e03d450 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1615,7 +1615,7 @@ python package_do_shlibs() {
 shlibswork_dir = d.getVar('SHLIBSWORKDIR')
 
 # Take shared lock since we're only reading, not writing
-lf = bb.utils.lockfile(d.expand("${PACKAGELOCK}"))
+lf = bb.utils.lockfile(d.expand("${PACKAGELOCK}"), True)
 
 def linux_so(file, needed, sonames, renames, pkgver):
 needs_ldconfig = False
@@ -1900,7 +1900,7 @@ python package_do_pkgconfig () {
 pkgconfig_needed[pkg] += exp.replace(',', ' 
').split()
 
 # Take shared lock since we're only reading, not writing
-lf = bb.utils.lockfile(d.expand("${PACKAGELOCK}"))
+lf = bb.utils.lockfile(d.expand("${PACKAGELOCK}"), True)
 
 for pkg in packages.split():
 pkgs_file = os.path.join(shlibswork_dir, pkg + ".pclist")
@@ -2233,11 +2233,9 @@ do_package[dirs] = "${SHLIBSWORKDIR} ${PKGDESTWORK} ${D}"
 do_package[vardeps] += "${PACKAGEBUILDPKGD} ${PACKAGESPLITFUNCS} 
${PACKAGEFUNCS} ${@gen_packagevar(d)}"
 addtask package after do_install
 
-PACKAGELOCK = "${STAGING_DIR}/package-output.lock"
 SSTATETASKS += "do_package"
 do_package[cleandirs] = "${PKGDEST} ${PKGDESTWORK}"
 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST} ${PKGDESTWORK}"
-do_package[sstate-lockfile-shared] = "${PACKAGELOCK}"
 do_package_setscene[dirs] = "${STAGING_DIR}"
 
 python do_package_setscene () {
@@ -2252,9 +2250,12 @@ do_packagedata () {
 addtask packagedata before do_build after do_package
 
 SSTATETASKS += "do_packagedata"
+# PACKAGELOCK protects readers of PKGDATA_DIR against writes
+# whilst code is reading in do_package
+PACKAGELOCK = "${STAGING_DIR}/package-output.lock"
 do_packagedata[sstate-inputdirs] = "${PKGDESTWORK}"
 do_packagedata[sstate-outputdirs] = "${PKGDATA_DIR}"
-do_packagedata[sstate-lockfile-shared] = "${PACKAGELOCK}"
+do_packagedata[sstate-lockfile] = "${PACKAGELOCK}"
 do_packagedata[stamp-extra-info] = "${MACHINE_ARCH}"
 
 python do_packagedata_setscene () {
-- 
2.7.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core