date:20180204

Re: [yocto] [layerindex-web][PATCH 4/5] update_layer.py: use it as a module

2018-02-04 Thread Robert Yang


Hi Paul,

Thank you very much, and please see my comments inline.

On 02/05/2018 07:27 AM, Paul Eggleton wrote:

Hi Robert,

On Wednesday, 3 January 2018 6:42:25 PM NZDT Robert Yang wrote:

It had been split to a separate tool and use subprocess to run it to avoid
tinfoil issues (e.g., can't be shutdown correctly in old version), but the
utils.setup_django() cost a lot of time which made it a little slow, use
mulitprocessing.Process() to run it as function can avoid both issues (tinfoil
and slow issues).

And remove bitbake modules when swith branch since it is may not be compatible
between branches.

This can save about 5 minutes in my testing when "update.py -b ",
with parallel fetch and this, we can reduce the time from more than 9mins to
less than 3 mins.

Signed-off-by: Robert Yang 
---
  layerindex/update.py   | 115 +++--
  layerindex/update_layer.py | 103 +---
  2 files changed, 70 insertions(+), 148 deletions(-)

diff --git a/layerindex/update.py b/layerindex/update.py
index d6488f0..e13c9ab 100755
--- a/layerindex/update.py
+++ b/layerindex/update.py
@@ -21,6 +21,9 @@ import utils
  import operator
  import re
  import multiprocessing
+import update_layer
+import copy
+import imp
  
  import warnings

  warnings.filterwarnings("ignore", category=DeprecationWarning)
@@ -34,61 +37,15 @@ except ImportError:
  logger.error("Please install PythonGit 0.3.1 or later in order to use this 
script")
  sys.exit(1)
  
-

-def reenable_sigint():
-signal.signal(signal.SIGINT, signal.SIG_DFL)
-
-def run_command_interruptible(cmd):
-"""
-Run a command with output displayed on the console, but ensure any Ctrl+C 
is
-processed only by the child process.
-"""
-signal.signal(signal.SIGINT, signal.SIG_IGN)
+def call_update_layer(options, queue):
+ret = 1
+output = ""
  try:
-process = subprocess.Popen(
-cmd, cwd=os.path.dirname(sys.argv[0]), shell=True, 
preexec_fn=reenable_sigint, stdout=subprocess.PIPE, stderr=subprocess.STDOUT
-)
-
-reader = codecs.getreader('utf-8')(process.stdout, 
errors='surrogateescape')
-buf = ''
-while True:
-out = reader.read(1, 1)
-if out:
-sys.stdout.write(out)
-sys.stdout.flush()
-buf += out
-elif out == '' and process.poll() != None:
-break
-
+ret, output = update_layer.main(options, queue)
+except Exception:
+raise
  finally:
-signal.signal(signal.SIGINT, signal.SIG_DFL)
-return process.returncode, buf
-
-
-def prepare_update_layer_command(options, branch, layer, initial=False):
-"""Prepare the update_layer.py command line"""
-if branch.update_environment:
-cmdprefix = branch.update_environment.get_command()
-else:
-cmdprefix = 'python3'
-cmd = '%s update_layer.py -l %s -b %s' % (cmdprefix, layer.name, 
branch.name)
-if options.reload:
-cmd += ' --reload'
-if options.fullreload:
-cmd += ' --fullreload'
-if options.nocheckout:
-cmd += ' --nocheckout'
-if options.dryrun:
-cmd += ' -n'
-if initial:
-cmd += ' -i'
-if options.loglevel == logging.DEBUG:
-cmd += ' -d'
-elif options.loglevel == logging.ERROR:
-cmd += ' -q'
-if options.keep_temp:
-cmd += ' --keep-temp'
-return cmd
+queue.put([ret, output])
  
  def update_actual_branch(layerquery, fetchdir, branch, options, update_bitbake, bitbakepath):

  """Update actual branch for layers and bitbake in database"""
@@ -266,6 +223,7 @@ def main():
  try:
  lockfn = os.path.join(fetchdir, "layerindex.lock")
  lockfile = utils.lock_file(lockfn)
+update_log_path = update_layer.update_log_path
  if not lockfile:
  logger.error("Layer index lock timeout expired")
  sys.exit(1)
@@ -315,8 +273,15 @@ def main():
  # We now do this by calling out to a separate script; doing 
otherwise turned out to be
  # unreliable due to leaking memory (we're using bitbake internals 
in a manner in which
  # they never get used during normal operation).
+update_layer_options = copy.copy(options)
  last_rev = {}
  for branch in branches:
+# The bitbake modules are not compatible between branches, so
+# remove them, they will be imported again when needed.
+for name, module in sys.modules.copy().items():
+if '__file__' in module.__dict__ and 
module.__file__.startswith(bitbakepath):
+logger.debug('Removing module %s' % name)
+del sys.modules[name]


This part worried me, so I did some quick searching and came across this:

Re: [yocto] [layerindex-web] Updating layers?

2018-02-04 Thread Paul Eggleton

Hi Steve

On Friday, 12 January 2018 4:52:23 AM NZDT Steve Bedford wrote:
> Is the layerindex not running the update script?  I see end of September as
> last updates, and a bunch of branches in our layer (meta-timesys) were
> never picked up.

FYI we've fixed this as of a few hours after your email - apologies for the 
downtime, we should now have things in place to avoid that kind of outage.
Let me know if you see any other anomalies.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [layerindex-web][PATCH 0/5] update.py: performance improve and 2 fixes

2018-02-04 Thread Paul Eggleton

Hi Robert

On Thursday, 1 February 2018 5:59:42 PM NZDT Robert Yang wrote:
> On 01/03/2018 01:42 PM, Robert Yang wrote:
> > This email contains two parts:
> > - The first 2 are bug fixes
> > - The other 3 are performance improvements, I have 124 layers,
> >the "update.py -b " needs about 9m20s to finish the running
> >when everything is
> >update, and we have several branches, we need them to run in
> >periodically
> >(e.g., per 30 mins, the short the better), so one branch 9m20s is
> >really a problem.
> >Only about 1m43s are needed with these improvements when set
> >PARALLEL_JOBS to 10:
> >update.py: fetch repos parallelly # Saved 2 mins
> >update_layer.py: use it as a module # Saved 5 mins
> >update_layer.py: only call init_parser when needed # Saved 1 min
>
> Any comments, please ?

My apologies for the delay - I had concerns about the update_layer.py
module stuff but hadn't made the time to review it properly until now.

Aside from that the other patches are fine so I've merged those (with
some minor tweaks to the commit messages).

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [layerindex-web][PATCH 4/5] update_layer.py: use it as a module

2018-02-04 Thread Paul Eggleton

Hi Robert,

On Wednesday, 3 January 2018 6:42:25 PM NZDT Robert Yang wrote:
> It had been split to a separate tool and use subprocess to run it to avoid
> tinfoil issues (e.g., can't be shutdown correctly in old version), but the
> utils.setup_django() cost a lot of time which made it a little slow, use
> mulitprocessing.Process() to run it as function can avoid both issues (tinfoil
> and slow issues).
> 
> And remove bitbake modules when swith branch since it is may not be compatible
> between branches.
> 
> This can save about 5 minutes in my testing when "update.py -b ",
> with parallel fetch and this, we can reduce the time from more than 9mins to
> less than 3 mins.
> 
> Signed-off-by: Robert Yang 
> ---
>  layerindex/update.py   | 115 
> +++--
>  layerindex/update_layer.py | 103 +---
>  2 files changed, 70 insertions(+), 148 deletions(-)
> 
> diff --git a/layerindex/update.py b/layerindex/update.py
> index d6488f0..e13c9ab 100755
> --- a/layerindex/update.py
> +++ b/layerindex/update.py
> @@ -21,6 +21,9 @@ import utils
>  import operator
>  import re
>  import multiprocessing
> +import update_layer
> +import copy
> +import imp
>  
>  import warnings
>  warnings.filterwarnings("ignore", category=DeprecationWarning)
> @@ -34,61 +37,15 @@ except ImportError:
>  logger.error("Please install PythonGit 0.3.1 or later in order to use 
> this script")
>  sys.exit(1)
>  
> -
> -def reenable_sigint():
> -signal.signal(signal.SIGINT, signal.SIG_DFL)
> -
> -def run_command_interruptible(cmd):
> -"""
> -Run a command with output displayed on the console, but ensure any 
> Ctrl+C is
> -processed only by the child process.
> -"""
> -signal.signal(signal.SIGINT, signal.SIG_IGN)
> +def call_update_layer(options, queue):
> +ret = 1
> +output = ""
>  try:
> -process = subprocess.Popen(
> -cmd, cwd=os.path.dirname(sys.argv[0]), shell=True, 
> preexec_fn=reenable_sigint, stdout=subprocess.PIPE, stderr=subprocess.STDOUT
> -)
> -
> -reader = codecs.getreader('utf-8')(process.stdout, 
> errors='surrogateescape')
> -buf = ''
> -while True:
> -out = reader.read(1, 1)
> -if out:
> -sys.stdout.write(out)
> -sys.stdout.flush()
> -buf += out
> -elif out == '' and process.poll() != None:
> -break
> -
> +ret, output = update_layer.main(options, queue)
> +except Exception:
> +raise
>  finally:
> -signal.signal(signal.SIGINT, signal.SIG_DFL)
> -return process.returncode, buf
> -
> -
> -def prepare_update_layer_command(options, branch, layer, initial=False):
> -"""Prepare the update_layer.py command line"""
> -if branch.update_environment:
> -cmdprefix = branch.update_environment.get_command()
> -else:
> -cmdprefix = 'python3'
> -cmd = '%s update_layer.py -l %s -b %s' % (cmdprefix, layer.name, 
> branch.name)
> -if options.reload:
> -cmd += ' --reload'
> -if options.fullreload:
> -cmd += ' --fullreload'
> -if options.nocheckout:
> -cmd += ' --nocheckout'
> -if options.dryrun:
> -cmd += ' -n'
> -if initial:
> -cmd += ' -i'
> -if options.loglevel == logging.DEBUG:
> -cmd += ' -d'
> -elif options.loglevel == logging.ERROR:
> -cmd += ' -q'
> -if options.keep_temp:
> -cmd += ' --keep-temp'
> -return cmd
> +queue.put([ret, output])
>  
>  def update_actual_branch(layerquery, fetchdir, branch, options, 
> update_bitbake, bitbakepath):
>  """Update actual branch for layers and bitbake in database"""
> @@ -266,6 +223,7 @@ def main():
>  try:
>  lockfn = os.path.join(fetchdir, "layerindex.lock")
>  lockfile = utils.lock_file(lockfn)
> +update_log_path = update_layer.update_log_path
>  if not lockfile:
>  logger.error("Layer index lock timeout expired")
>  sys.exit(1)
> @@ -315,8 +273,15 @@ def main():
>  # We now do this by calling out to a separate script; doing 
> otherwise turned out to be
>  # unreliable due to leaking memory (we're using bitbake 
> internals in a manner in which
>  # they never get used during normal operation).
> +update_layer_options = copy.copy(options)
>  last_rev = {}
>  for branch in branches:
> +# The bitbake modules are not compatible between branches, so
> +# remove them, they will be imported again when needed.
> +for name, module in sys.modules.copy().items():
> +if '__file__' in module.__dict__ and 
> module.__file__.startswith(bitbakepath):
> +logger.debug('Removing module %s' % name)
> +del

[yocto] [layerindex-web][PATCH] Handle __isnull in API query filtering

2018-02-04 Thread Paul Eggleton

If you query on a boolean field you can use the string "False" to match
False in the database; however if you try the same with __isnull then
the query will match every record which is obviously undesirable. If
__isnull is being used, then convert the value to a boolean so that the
query works properly.

An example of this type of query:

  
http://127.0.0.1:8000/layerindex/api/layerBranches/?filter=yp_compatible_version__isnull:false

Signed-off-by: Paul Eggleton 
---
 layerindex/querysethelper.py | 24 ++--
 1 file changed, 14 insertions(+), 10 deletions(-)

diff --git a/layerindex/querysethelper.py b/layerindex/querysethelper.py
index b4d30c2..4e7cac5 100644
--- a/layerindex/querysethelper.py
+++ b/layerindex/querysethelper.py
@@ -28,16 +28,16 @@ VALUE_SEPARATOR = "!"
 DESCENDING = "-"
 
 def __get_q_for_val(name, value):
-if "OR" in value:
-return functools.reduce(operator.or_, map(lambda x: 
__get_q_for_val(name, x), [ x for x in value.split("OR") ]))
-if "AND" in value:
-return functools.reduce(operator.and_, map(lambda x: 
__get_q_for_val(name, x), [ x for x in value.split("AND") ]))
-if value.startswith("NOT"):
-kwargs = { name : value.strip("NOT") }
-return ~Q(**kwargs)
-else:
-kwargs = { name : value }
-return Q(**kwargs)
+if isinstance(value, str):
+if "OR" in value:
+return functools.reduce(operator.or_, map(lambda x: 
__get_q_for_val(name, x), [ x for x in value.split("OR") ]))
+if "AND" in value:
+return functools.reduce(operator.and_, map(lambda x: 
__get_q_for_val(name, x), [ x for x in value.split("AND") ]))
+if value.startswith("NOT"):
+kwargs = { name : value.strip("NOT") }
+return ~Q(**kwargs)
+kwargs = { name : value }
+return Q(**kwargs)
 
 def _get_filtering_query(filter_string):
 
@@ -46,6 +46,10 @@ def _get_filtering_query(filter_string):
 values = search_terms[1].split(VALUE_SEPARATOR)
 
 querydict = dict(zip(keys, values))
+for key in keys:
+if key.endswith('__isnull'):
+querydict[key] = (querydict[key].lower() == 'true')
+
 return functools.reduce(operator.and_, map(lambda x: __get_q_for_val(x, 
querydict[x]), [k for k in querydict]))
 
 # we check that the input comes in a valid form that we can recognize
-- 
2.9.5

-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] Yocto Project Technical Team Meeting

2018-02-04 Thread akuster



On 02/03/2018 09:25 PM, Stephano Cetola wrote:
> On 2/2/18 11:04 AM, Cetola, Stephano wrote:
>> Cetola, Stephano has invited you to Yocto Project Technical Team Meeting
>>
>> When:
>>
>> Occurs the first Tuesday of every month effective 2/6/18 from 8:00 AM to
>> 8:30 AM.
>>  
>> Organizer:
>>
>>  Cetola, Stephano 
>>
>> Description:
>>
>> When: Occurs every month on the first Tuesday of the month from 8:00 AM
>> to 8:30 AM effective 2/6/2018. (UTC-08:00) Pacific Time (US & Canada)
> It has been mentioned to me that there is some confusion on when this
> meeting occurs. I sent this invite using Outlook, which is a necessary
> evil here.
>
> The meeting is from 8:00 - 8:30 (UTC -08:00). I'm happy to send a new
> invite if enough people need one, but I want to minimize the noise here.
Same time, Same Bat Channel only hosted by Robin?

Is this link still correct?

https://www.yoctoproject.org/tools-resources/community/monthly-technical-call

- Armin

We
>
> Cheers,
> Stephano


-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

[yocto] runqemu looking for qemuboot.conf file...

2018-02-04 Thread Peter Spierenburg

I've written a bsp for the adzs-sc589-ezlite board from analog devices. I am
trying to use qemu to emulate the board, but I get the following

runqemu - INFO - Can't find qemuboot conf file, DEPLOY_DIR_IMAGE is NULL!
runqemu - INFO - Running MACHINE=adzs-sc589-ezlite bitbake -e...
runqemu - INFO - Overriding conf file setting of STAGING_DIR_NATIVE to
/home/peter/yocto-build/build/tmp/work/armv5e-poky-linux-gnueabi/defaultpkgname/1.0-r0/recipe-sysroot-native
from Bitbake environment
Traceback (most recent call last):
File "/home/peter/yocto-build/poky/scripts/runqemu", line 1243, in
ret = main()
File "/home/peter/yocto-build/poky/scripts/runqemu", line 1230, in main
config.check_and_set()
File "/home/peter/yocto-build/poky/scripts/runqemu", line 652, in
check_and_set
self.check_fstype()
File "/home/peter/yocto-build/poky/scripts/runqemu", line 513, in check_fstype
raise Exception("FSTYPE is NULL!")
Exception: FSTYPE is NULL!

I've tried forcing:

$ bitbake core-image-minimal -c do_write_qemuboot_conf

but with no success.

Peter.

This communication, including any attached documentation, is intended only for
the person or entity to which it is addressed, and may contain confidential,
personal, and/or privileged information. Any unauthorized disclosure, copying,
or taking action on the contents is strictly prohibited. If you have received
this message in error, please contact us immediately so we may correct our
records. Please then delete or destroy the original transmission and any
subsequent reply. Thank you.
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [layerindex-web][PATCH 4/5] update_layer.py: use it as a module

Re: [yocto] [layerindex-web] Updating layers?

Re: [yocto] [layerindex-web][PATCH 0/5] update.py: performance improve and 2 fixes

Re: [yocto] [layerindex-web][PATCH 4/5] update_layer.py: use it as a module

[yocto] [layerindex-web][PATCH] Handle __isnull in API query filtering

Re: [yocto] Yocto Project Technical Team Meeting

[yocto] runqemu looking for qemuboot.conf file...

7 matches

Site Navigation

Mail list logo

Footer information