Re: [OE-core] [PATCH] devtool: ensure temp directory is created

2017-10-19 Thread Paul Eggleton 
On Wednesday, 18 October 2017 5:34:14 AM NZDT Joshua Lock wrote:
> 
> On 17/10/17 17:12, Burton, Ross wrote:
> > On 17 October 2017 at 17:02, Joshua Lock  > > wrote:
> > 
> > FWIW the failure has been observed on multiple host distros (Ubuntu
> > 16.04, Fedora 26, Tumbleweed) and I could replicate it very easily
> > on my Fedora 26 host with an use of devtool modify -x.
> > 
> > 
> > If you can replicate it on demand how about using a tracing tool to 
> > figure out if a) the mkdir doesn't actually happen or b) something 
> > deletes the directory.
> 
> It seems it's not as easy to reproduce as I thought. I can't reproduce 
> it at all today on either of my two systems.
> 
> I will attempt to gather some logs next time we see it on the autobuilder.

Just to follow up on this thread, creating the temp directory doesn't fix the 
issue. I found and fixed the underlying problem yesterday, details on the bug:

  https://bugzilla.yoctoproject.org/show_bug.cgi?id=12185

I've sent a fix to the bitbake list.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] devtool: ensure temp directory is created

2017-10-17 Thread Joshua Lock 



On 17/10/17 17:12, Burton, Ross wrote:
On 17 October 2017 at 17:02, Joshua Lock > wrote:


FWIW the failure has been observed on multiple host distros (Ubuntu
16.04, Fedora 26, Tumbleweed) and I could replicate it very easily
on my Fedora 26 host with an use of devtool modify -x.


If you can replicate it on demand how about using a tracing tool to 
figure out if a) the mkdir doesn't actually happen or b) something 
deletes the directory.


It seems it's not as easy to reproduce as I thought. I can't reproduce 
it at all today on either of my two systems.


I will attempt to gather some logs next time we see it on the autobuilder.

Joshua
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] devtool: ensure temp directory is created

2017-10-17 Thread Burton, Ross 
On 17 October 2017 at 17:02, Joshua Lock 
wrote:

> FWIW the failure has been observed on multiple host distros (Ubuntu 16.04,
> Fedora 26, Tumbleweed) and I could replicate it very easily on my Fedora 26
> host with an use of devtool modify -x.


If you can replicate it on demand how about using a tracing tool to figure
out if a) the mkdir doesn't actually happen or b) something deletes the
directory.

Ross
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] devtool: ensure temp directory is created

2017-10-17 Thread Joshua Lock 



On 17/10/17 17:00, Leonardo Sandoval wrote:

On Tue, 17 Oct 2017 16:05:06 +0100
Joshua Lock  wrote:

I would be nice that we keep the bugzilla ticket open (and included the tag on 
this patch) and investigate further  this issue. Seems like in some distros, 
the temfile.mkdtemp is not doing what it is supposed thus we have this 
workaround (perhaps strace may lead the real reason)


Agreed. I meant to write more before sending this patch (and copy Paul). 
 It's clearly not the right fix as the tempfile.mkdtemp() call should 
create the directory, but I wanted to share my simple findings (that the 
directory doesn't appear to be reliably created by the mkdtemp call, as 
expected).


FWIW the failure has been observed on multiple host distros (Ubuntu 
16.04, Fedora 26, Tumbleweed) and I could replicate it very easily on my 
Fedora 26 host with an use of devtool modify -x.


Joshua




We often see nightly-oe-selftest on the Yocto Project Autobuilder fail in
test_devtool_modify_native with:

  'devtool modify apt-native -x /tmp/devtoolqahlptvsfq/apt-native' returned
  non-zero exit status 1

and the following traceback:

Traceback (most recent call last):
   File 
"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/devtool",
 line 331, in 
 ret = main()
   File 
"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/devtool",
 line 318, in main
 ret = args.func(args, config, basepath, workspace)
   File 
"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/lib/devtool/standard.py",
 line 714, in modify
 initial_rev = _extract_source(srctree, args.keep_temp, args.branch, False, 
config, basepath, workspace, args.fixed_setup, rd, tinfoil)
   File 
"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/lib/devtool/standard.py",
 line 545, in _extract_source
 with open(os.path.join(tempdir, 'initial_rev'), 'r') as f:

Ensure the tempdir is created before we try and create any files there.

Signed-off-by: Joshua Lock 
---
  scripts/lib/devtool/standard.py | 1 +
  1 file changed, 1 insertion(+)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index beea0d4c274..38197251b42 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -494,6 +494,7 @@ def _extract_source(srctree, keep_temp, devbranch, sync, 
config, basepath, works
  tempbasedir = d.getVar('WORKDIR')
  bb.utils.mkdirhier(tempbasedir)
  tempdir = tempfile.mkdtemp(prefix='devtooltmp-', dir=tempbasedir)
+bb.utils.mkdirhier(tempdir)
  try:
  tinfoil.logger.setLevel(logging.WARNING)
  
--

2.13.6

--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core




--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] devtool: ensure temp directory is created

2017-10-17 Thread Leonardo Sandoval 
On Tue, 17 Oct 2017 16:05:06 +0100
Joshua Lock  wrote:

I would be nice that we keep the bugzilla ticket open (and included the tag on 
this patch) and investigate further  this issue. Seems like in some distros, 
the temfile.mkdtemp is not doing what it is supposed thus we have this 
workaround (perhaps strace may lead the real reason)

> We often see nightly-oe-selftest on the Yocto Project Autobuilder fail in
> test_devtool_modify_native with:
> 
>  'devtool modify apt-native -x /tmp/devtoolqahlptvsfq/apt-native' returned
>  non-zero exit status 1
> 
> and the following traceback:
> 
> Traceback (most recent call last):
>   File 
> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/devtool",
>  line 331, in 
> ret = main()
>   File 
> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/devtool",
>  line 318, in main
> ret = args.func(args, config, basepath, workspace)
>   File 
> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/lib/devtool/standard.py",
>  line 714, in modify
> initial_rev = _extract_source(srctree, args.keep_temp, args.branch, 
> False, config, basepath, workspace, args.fixed_setup, rd, tinfoil)
>   File 
> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/lib/devtool/standard.py",
>  line 545, in _extract_source
> with open(os.path.join(tempdir, 'initial_rev'), 'r') as f:
> 
> Ensure the tempdir is created before we try and create any files there.
> 
> Signed-off-by: Joshua Lock 
> ---
>  scripts/lib/devtool/standard.py | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
> index beea0d4c274..38197251b42 100644
> --- a/scripts/lib/devtool/standard.py
> +++ b/scripts/lib/devtool/standard.py
> @@ -494,6 +494,7 @@ def _extract_source(srctree, keep_temp, devbranch, sync, 
> config, basepath, works
>  tempbasedir = d.getVar('WORKDIR')
>  bb.utils.mkdirhier(tempbasedir)
>  tempdir = tempfile.mkdtemp(prefix='devtooltmp-', dir=tempbasedir)
> +bb.utils.mkdirhier(tempdir)
>  try:
>  tinfoil.logger.setLevel(logging.WARNING)
>  
> -- 
> 2.13.6
> 
> -- 
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core


-- 
Leonardo Sandoval 
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] devtool: ensure temp directory is created

2017-10-17 Thread Joshua Lock 
We often see nightly-oe-selftest on the Yocto Project Autobuilder fail in
test_devtool_modify_native with:

 'devtool modify apt-native -x /tmp/devtoolqahlptvsfq/apt-native' returned
 non-zero exit status 1

and the following traceback:

Traceback (most recent call last):
  File 
"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/devtool",
 line 331, in 
ret = main()
  File 
"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/devtool",
 line 318, in main
ret = args.func(args, config, basepath, workspace)
  File 
"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/lib/devtool/standard.py",
 line 714, in modify
initial_rev = _extract_source(srctree, args.keep_temp, args.branch, False, 
config, basepath, workspace, args.fixed_setup, rd, tinfoil)
  File 
"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/lib/devtool/standard.py",
 line 545, in _extract_source
with open(os.path.join(tempdir, 'initial_rev'), 'r') as f:

Ensure the tempdir is created before we try and create any files there.

Signed-off-by: Joshua Lock 
---
 scripts/lib/devtool/standard.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index beea0d4c274..38197251b42 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -494,6 +494,7 @@ def _extract_source(srctree, keep_temp, devbranch, sync, 
config, basepath, works
 tempbasedir = d.getVar('WORKDIR')
 bb.utils.mkdirhier(tempbasedir)
 tempdir = tempfile.mkdtemp(prefix='devtooltmp-', dir=tempbasedir)
+bb.utils.mkdirhier(tempdir)
 try:
 tinfoil.logger.setLevel(logging.WARNING)
 
-- 
2.13.6

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core