[yocto] [layerindex-web][PATCH 0/6] Recipe search improvements, misc fixes [cover letter only]

2019-11-20 Thread Paul Eggleton
Some minor improvements to recipe searching, plus fixes for a couple of issues I noticed while doing the RRS rework. Note: this patchset is on top of paule/recipesymbol. The following changes since commit 6f85a1b4589df08e849a106338fe03d72e2394c2: TODO: add some more tasks (2019-11-21

[yocto] [layerindex-web][PATCH 00/29] RRS: rework upgrade history collection [cover letter only]

2019-11-20 Thread Paul Eggleton
Taking a hard look at the recipe upgrade functionality, there were a number of shortcomings in how the data was collected which led to upgrades being missed, history disappearing when recipes were removed or renamed and scenarios such as multiple recipes with the same name not being handled

Re: [yocto] [layerindex-web] Add SPDX Identifier

2019-10-28 Thread Paul Eggleton
Hi Yann On Wednesday, 23 October 2019 2:25:48 AM NZDT Yann CARDAILLAC wrote: > On Mon, Oct 21, 2019 at 9:37 PM Paul Eggleton > wrote: > > On Tuesday, 22 October 2019 4:15:03 AM NZDT Yann CARDAILLAC wrote: > > > https://bugzilla.yoctoproject.org/show_bug.cgi?id=13527 > > > > > > I'm beginning the

[yocto] [layerindex-web][PATCH 2/2] requirements.txt: bump a couple more versions

2019-10-28 Thread Paul Eggleton
Update pytz and beautifulsoup4. Signed-off-by: Paul Eggleton --- requirements.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/requirements.txt b/requirements.txt index 84f2ea54..07d8495e 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,5 +1,5 @@

[yocto] [layerindex-web][PATCH 1/2] requirements.txt: update to fix CVE-2019-16865

2019-10-28 Thread Paul Eggleton
Update Pillow version to incorporate a fix for a denial-of-service vulnerability (which should not affect this application however, as it does not use Pillow to process external images): https://nvd.nist.gov/vuln/detail/CVE-2019-16865 Signed-off-by: Paul Eggleton --- requirements.txt | 2 +-

[yocto] [layerindex-web][PATCH 0/2] dependency updates

2019-10-28 Thread Paul Eggleton
Updates to python dependencies in requirements.txt, including one upstream CVE fix. The following changes since commit fae9771604097896f10a4aec2f2aee7fefd8ec86: Fix cgit commit URL setting (2019-10-21 12:02:44 +1300) are available in the Git repository at:

Re: [yocto] [layerindex-web] Add SPDX Identifier

2019-10-22 Thread Yann CARDAILLAC
On Mon, Oct 21, 2019 at 9:37 PM Paul Eggleton wrote: > On Tuesday, 22 October 2019 4:15:03 AM NZDT Yann CARDAILLAC wrote: > > https://bugzilla.yoctoproject.org/show_bug.cgi?id=13527 > > > > I'm beginning the work of updating the layerindex-web repo, this one is a > > bit tricky, how should I

Re: [yocto] [layerindex-web] Add SPDX Identifier

2019-10-21 Thread Paul Eggleton
Hi Yann Thanks for looking into this. On Tuesday, 22 October 2019 4:15:03 AM NZDT Yann CARDAILLAC wrote: > https://bugzilla.yoctoproject.org/show_bug.cgi?id=13527 > > I'm beginning the work of updating the layerindex-web repo, this one is a > bit tricky, how should I treat the files generated

[yocto] [layerindex-web] Add SPDX Identifier

2019-10-21 Thread Yann CARDAILLAC
Hi, https://bugzilla.yoctoproject.org/show_bug.cgi?id=13527 I'm beginning the work of updating the layerindex-web repo, this one is a bit tricky, how should I treat the files generated using Django ? *For now I've given them MIT as for the other hand written script. However I'm kind of worried

[yocto] [layerindex-web][PATCH 3/3] Fix cgit commit URL setting

2019-10-20 Thread Paul Eggleton
Typo, there has to be a ? in front of the id or otherwise we don't get linked to the right commit. This would have affected recently added layers with a cgit web frontend. Signed-off-by: Paul Eggleton --- templates/layerindex/editlayer.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)

[yocto] [layerindex-web][PATCH 2/3] Add branch comparison function

2019-10-20 Thread Paul Eggleton
Add the ability to compare available recipes and their versions between two branches for a selection of layers (default is just OE-Core). This was mainly intended to help us with the Yocto Project release notes preparation (hence the "Plain text" button at the bottom of the page) but is also

[yocto] [layerindex-web][PATCH 1/3] Track SRCREV for each recipe

2019-10-20 Thread Paul Eggleton
For the purposes of the branch comparison function I'm about to add it would be useful to track the value of SRCREV, so we can see if it has changed even if PV hasn't. Signed-off-by: Paul Eggleton --- layerindex/migrations/0043_recipe_srcrev.py | 20 layerindex/models.py

[yocto] [layerindex-web][PATCH 0/3] Branch comparison functionality

2019-10-20 Thread Paul Eggleton
Add the ability to compare the available recipes and their versions in two different branches, plus one unrelated patch to fix an issue I noticed recently with cgit URLs. The following changes since commit 1ce4a85d257ac928307ac0de6edf4cb80b50adce: editlayer: Be more specific on the searches

Re: [yocto] [layerindex-web] Enabled zeus, not working...

2019-10-14 Thread Mark Hatle
On 10/14/19 5:08 PM, Paul Eggleton wrote: > Hi Mark > > On Tuesday, 15 October 2019 10:21:29 AM NZDT Mark Hatle wrote: >> I added the zeus branch on the layers.openembedded.org today and it's not >> showing up and being indexed. Any idea why? > > The bitbake branch wasn't correctly specified

Re: [yocto] [layerindex-web] Enabled zeus, not working...

2019-10-14 Thread Paul Eggleton
Hi Mark On Tuesday, 15 October 2019 10:21:29 AM NZDT Mark Hatle wrote: > I added the zeus branch on the layers.openembedded.org today and it's not > showing up and being indexed. Any idea why? The bitbake branch wasn't correctly specified - it needed to be set to "1.44" (grabbed from

[yocto] [layerindex-web] Enabled zeus, not working...

2019-10-14 Thread Mark Hatle
I added the zeus branch on the layers.openembedded.org today and it's not showing up and being indexed. Any idea why? On my own personal layer index I did it and it worked fine. So it may be something related to the configuration. Below are the errors from the update log: Oct. 14, 2019, 7:13

Re: [yocto] [layerindex-web] [PATCH 0/3] Some misc changes/fixes..

2019-10-13 Thread Paul Eggleton
Hi Mark On Sunday, 13 October 2019 2:56:30 PM NZDT Mark Hatle wrote: > A few misc changes/fixes. The first two are well tested. However, I suspect > the 3/3 may be incorrect and I've labeled it an RFC due to this. > > 1/3 - '.' wasn't allowed in branch names w/o an error. This turned out >

[yocto] [layerindex-web] [PATCH 3/3] RFC: editlayer: Be more specific on the searches

2019-10-12 Thread Mark Hatle
Just because git.yoctoproject.org is in the URL, doesn't mean we can or should force the vcs_web_url to be a specific value. If it starts with git://git.yoctoproject.org then we can do this. git.openembedded.org already did this. This also changes github, gitlab and bitbucket references.

[yocto] [layerindex-web] [PATCH 2/3] update.py: Allow bitbake to live in a subdirectory of a repository

2019-10-12 Thread Mark Hatle
Add a new BITBAKE_PATH to the settings file to specify the path within the BITBAKE_REPO_URL where bitbake lives. This is useful when using a combined repository, such as poky, that contains bitbake, openembedded-core and other layers. This change also changes the default path, in the fetch

[yocto] [layerindex-web] [PATCH 1/3] layerindex/urls.py: Allow branches with a '.' in the name

2019-10-12 Thread Mark Hatle
Without this change the system will fail parsing various URL components Signed-off-by: Mark Hatle --- layerindex/urls.py | 12 ++-- 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/layerindex/urls.py b/layerindex/urls.py index 7f4e545..89e70a2 100644 ---

[yocto] [layerindex-web] [PATCH 0/3] Some misc changes/fixes..

2019-10-12 Thread Mark Hatle
A few misc changes/fixes. The first two are well tested. However, I suspect the 3/3 may be incorrect and I've labeled it an RFC due to this. 1/3 - '.' wasn't allowed in branch names w/o an error. This turned out to be a fairly simple fix. 2/3 - For people who want to use 'poky' repository and

Re: [yocto] [layerindex-web] Having problems instantiating a docker image

2019-10-09 Thread Paul Eggleton
Hi Mark On Wednesday, 9 October 2019 11:59:35 AM NZDT Mark Hatle wrote: > I've setup the layerindex in the past (without docker).. but I'm attempting to > follow the current instructions w/o much success. > > I'm trying to use: > > ./dockersetup.py -m 8080:80 --no-https > > It asks me for my

[yocto] [layerindex-web] Having problems instantiating a docker image

2019-10-08 Thread Mark Hatle
I've setup the layerindex in the past (without docker).. but I'm attempting to follow the current instructions w/o much success. I'm trying to use: ./dockersetup.py -m 8080:80 --no-https It asks me for my email address and then builds the docker images... then I get a failure connecting to

[yocto] [layerindex-web][PATCH] Track PE and PR for recipes

2019-10-04 Thread Pranay Mankad
This change is to report PE (epoch) and PR (release) values for recipes from the bitbake cooker. These values will be reflected in the Rest API and will not be exposed on the UI. The aim of this change is to enrich metadata availability to consumers of the feed. ` Signed-off-by: Pranay Mankad

[yocto] [layerindex-web][PATCH] These changes will add the metadata for product revision and epoch if present in the bitbake recipe.

2019-10-04 Thread Pranay Mankad
Signed-off-by: Pranay Mankad --- .../migrations/0042_recipe_property_addition.py | 20 layerindex/models.py | 2 ++ layerindex/update_layer.py | 2 ++ 3 files changed, 24 insertions(+) create mode 100644

[yocto] [layerindex-web][PATCH 0/3] More minor fixes [cover letter only]

2019-10-01 Thread Paul Eggleton
A few miscellaneous minor improvements from the Clear Linux Dissector. The following changes since commit db51830ab238ed1a7ce33d43519c630b1c74c70b: update_classic_status: ensure fields are correctly ordered (2019-10-02 14:13:47 +1300) are available in the Git repository at:

Re: [yocto] [layerindex-web][PATCH] These changes will add the metadata for product revision and epoch if present in the bitbake recipe.

2019-09-30 Thread Paul Eggleton
Hi Pranay, On Tuesday, 1 October 2019 3:52:59 PM NZDT Pranay Mankad wrote: > Signed-off-by: Pranay Mankad > --- > .../migrations/0042_recipe_property_addition.py | 20 > > layerindex/models.py | 2 ++ > layerindex/update_layer.py

[yocto] [layerindex-web][PATCH 00/15] More dockersetup/import fixes [cover letter only]

2019-09-29 Thread Paul Eggleton
A few more improvements from the Clear Linux Dissector [1] for dockersetup, import_layers, and update_classic_status. [1] https://github.com/intel/clear-linux-dissector-web The following changes since commit b78e498c2086e4cb6631d73f36db059d37ad70ad: import_layers: fix updating build

[yocto] [layerindex-web]

2019-09-24 Thread Pranay Mankad
Hi All, Thank you for supporting the recipesExtended view over the last couple of days. Here’s a small patch that includes values from two more variables from the Bitbake cooker – PR and PE. I read through Paul’s explanation of why PE made sense but not PR, and thinking about it, the PR in

Re: [yocto] [layerindex-web] PR / PE in layer index

2019-09-23 Thread Paul Eggleton
Hi Pranay On Tuesday, 24 September 2019 10:00:46 AM NZST Pranay Mankad wrote: > Thank you for supporting the recipesExtended view over the last couple of > days. Here’s a small patch that includes values from two more variables from > the Bitbake cooker – PR and PE. > > I read through Paul’s

[yocto] [layerindex-web][PATCH 3/3] import_layers: fix output when importing layers from scratch

2019-09-15 Thread Paul Eggleton
Fixes "Importing None" message since we had not yet initialised the layerbranch variable. Signed-off-by: Paul Eggleton --- layerindex/tools/import_layers.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/layerindex/tools/import_layers.py b/layerindex/tools/import_layers.py

[yocto] [layerindex-web][PATCH 2/3] import_layers: use recipesExtended viewset

2019-09-15 Thread Paul Eggleton
The new viewset has a different URL and uses pagination, so we need to take that into account. Signed-off-by: Paul Eggleton --- layerindex/tools/import_layers.py | 92 ++- 1 file changed, 55 insertions(+), 37 deletions(-) diff --git

[yocto] [layerindex-web][PATCH 1/3] API: fix recipes API performance regression

2019-09-15 Thread Paul Eggleton
Adding these extra child items to the standard "recipes" viewset (which we did recently in 684a06a383fd2a8da80490dff5f0dbc47750934e) means that some current usage is impractical due to the size of the returned list of items. Instead, create a recipesExtended viewset, move the new child items to

[yocto] [layerindex-web][PATCH 0/3] REST API fixes

2019-09-15 Thread Paul Eggleton
Some fixes relating to the REST API performance issues experienced with the recent upgrade on layers.openembedded.org. The following changes since commit 73198bd5483c16b9852121c6855dc6dac96a98a0: README: add reference to using import_layers to update (2019-08-06 12:04:17 +1200) are

[yocto] [layerindex-web][PATCH 0/4] dockersetup fixes (cover letter only)

2019-08-11 Thread Paul Eggleton
Some fixes for the dockersetup.py script from the Clear Linux Dissector on top of the set that was merged recently. The following changes since commit 73198bd5483c16b9852121c6855dc6dac96a98a0: README: add reference to using import_layers to update (2019-08-06 12:04:17 +1200) are available

[yocto] [layerindex-web][PATCH 00/15] REST API / import_layers improvements (cover letter only)

2019-08-05 Thread Paul Eggleton
Add some missing items to the REST API and extend the import_layers script in order to make it possible to create and update a clone of an existing layer index without needing to fetch and parse all of the layer repositories. The following changes since commit

[yocto] [layerindex-web][PATCH v2 000/129] Docker setup / misc fixes (cover letter only)

2019-07-25 Thread Paul Eggleton
The Clear Linux* Dissector [1] is a special-purpose fork of the OE Layer Index codebase. During development a number of general-purpose fixes have been made, so I am now sending a slightly reworked set of these back to be merged into the layer index. Highlights: * Enhanced docker/docker-compose

[yocto] [layerindex-web][PATCH 000/126] Docker setup / misc fixes (cover letter only)

2019-07-17 Thread Paul Eggleton
The Clear Linux* Dissector [1] is a special-purpose fork of the OE Layer Index codebase. During development a number of general-purpose fixes have been made, so I am now sending a slightly reworked set of these back to be merged into the layer index. Highlights: * Enhanced docker/docker-compose

[yocto] [layerindex-web][PATCH 0/7] Recipe reporting system fixes (cover letter only)

2019-07-16 Thread Paul Eggleton
A bunch of fixes mostly for the Recipe Reporting System (RRS) part of the layer index. In particular, 5/7 fixes the RRS for layers in meta-openembedded. The following changes since commit ce5141fc39b2b171383fca002b2bc5cf082b31f4: RRS: add tool to import/export upstream history data

[yocto] [layerindex-web][PATCH 00/30] Misc improvements (cover letter only)

2018-11-19 Thread Paul Eggleton
A bunch of assorted improvements for the layer index. The following changes since commit 24b6e13442c403ae6d4a2b1782a5b5f06b643e22: docker: add TASK_LOG_DIR to docker settings file (2018-10-04 11:15:39 +1300) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web

[yocto] [layerindex-web] update: Parse the layer's compatible version info

2018-10-17 Thread Kevin Hao
We have already set the layer's compatible version info in the layer.conf, but we lack this info in the layerindex database. Add support to parse this info and set it in the corresponding tables. We plan to use this to filter the layerindex results. Signed-off-by: Kevin Hao ---

[yocto] [layerindex-web] Add YPCompatibleVersions to the api page

2018-10-11 Thread Kevin Hao
We plan to use this compatible flag to filter the layer index results, so add it to the api page first. Signed-off-by: Kevin Hao --- layerindex/restviews.py | 4 layerindex/urls.py | 1 + 2 files changed, 5 insertions(+) diff --git a/layerindex/restviews.py b/layerindex/restviews.py

Re: [yocto] [layerindex-web][PATCH 000/242] Integrate and improve Recipe Reporting System (cover letter only)

2018-10-02 Thread Anibal Limon
Hi Paul, Good to know that is integrated now. Cheers, Anibal On Tue, 1 May 2018 at 15:09, Paul Eggleton wrote: > Hi Alex, > > On Tuesday, 1 May 2018 9:58:03 PM NZST Alexander Kanavin wrote: > > On 05/01/2018 05:23 AM, Paul Eggleton wrote: > > > The Recipe Reporting System (RRS - live instance

Re: [yocto] [layerindex-web] RFC: layer index docker fixes

2018-10-02 Thread Konrad Scherer
On 09/05/2018 08:04 AM, Paul Eggleton wrote: On Thursday, 30 August 2018 11:44:10 AM NZST Paul Eggleton wrote: On Wednesday, 29 August 2018 10:46:27 PM NZST Paul Eggleton wrote: I know Konrad promised to update the docker-compose setup to be closer to what we'd done recently with the docker

[yocto] [layerindex-web][PATCH] Fix broken regex for recipes with names containing + signs

2018-10-01 Thread Paul Eggleton
In order to show bbappends on the recipe detail page we are doing a regex query to find any whose names match up with the recipe. In the layer index instance at layers.openembedded.org viewing the recipe detail page for any recipe whose name contains ++ (e.g. libsigc++-2.0 in meta-oe) results in

[yocto] [layerindex-web][PATCH 00/12] More minor fixes (cover letter only)

2018-09-30 Thread Paul Eggleton
A bunch of fixes for minor issues / missing functionality I've noticed since we did the update of layers.openembedded.org last week. The following changes since commit 2554449baa5dece6b3b2ca027f1a368823e43b00: update: fix regression caused by previous temp dir fix (2018-09-27 15:35:38 +1200)

[yocto] [layerindex-web][PATCH] update: fix regression caused by previous temp dir fix

2018-09-26 Thread Paul Eggleton
In c26604146a74149487a1a2dfc40d40d53aa68bdf I made a fix to change where the bitbake code writes out bitbake.lock and other files it creates during parsing, but didn't adequately test it and it turns out our call to delete the temp directory races against bitbake deleting bitbake.lock and

[yocto] [layerindex-web][PATCH 0/5] More minor fixes (cover letter only)

2018-09-25 Thread Paul Eggleton
A tweak for the other distro comparison modals, some fixes for the browse/search forms on the machines/distros/classes pages, and a fix for where bitbake-cookerdaemon.log gets written out when parsing. The following changes since commit 5edeb03cb1a50d68dc2b60c21f807c1f5c2db295: TODO: remove

[yocto] [layerindex-web][PATCH 00/10] Bootstrap 3, update task enhancements, etc. (cover letter only)

2018-09-19 Thread Paul Eggleton
A bit of a mixed bag: one bugfix, move to Bootstrap 3 for the frontend, some minor features and significant work on how update tasks for the other distro comparison functionality works. The following changes since commit 3aa6bf964d81a04b652499069f0454c546a5e296: import_layer.py: add -t option

[yocto] [layerindex-web][V2 PATCH 1/1] import_layer.py: add -t option for layer_type

2018-09-11 Thread Robert Yang
Now the logic is: Use options.layer_type if specified, and guess if not. Default to 'M'. Note choices=['A', 'B', 'S', 'D', 'M', ''], the '' is for default='', we can't use default='M' here, otherwise we don't know whether the 'M' is specified by user or is the default value, we don't guess if it

[yocto] [layerindex-web][V2 PATCH 0/1] import_layer.py: add -t option for layer_type

2018-09-11 Thread Robert Yang
* V2: - Make -t option use LAYER_TYPE_CHOICES from models.py. * V1: - Initial version // Robert The following changes since commit ac32edb5bdbdef30ccf3ef49af0977246571abaa: TODO: add some more items (2018-09-06 11:08:43 +1200) are available in the git repository at:

[yocto] [layerindex-web][PATCH 00/15] More minor fixes (cover letter only)

2018-09-05 Thread Paul Eggleton
The following changes since commit f82f5f2754dd2d41db8d62f455dd380c26635ee1: update: add option to stop immediately on parse errors (2018-08-24 16:41:59 +1200) are available in the Git repository at: git://git.yoctoproject.org/layerindex-web paule/fixes7

Re: [yocto] [layerindex-web] RFC: layer index docker fixes

2018-09-05 Thread Paul Eggleton
On Thursday, 30 August 2018 11:44:10 AM NZST Paul Eggleton wrote: > On Wednesday, 29 August 2018 10:46:27 PM NZST Paul Eggleton wrote: > > I know Konrad promised to update the docker-compose setup to be closer > > to what we'd done recently with the docker setup, but I needed something > >

Re: [yocto] [layerindex-web][PATCH 1/1] import_layer.py: add -t option for layer_type

2018-08-30 Thread Robert Yang
On 08/30/2018 02:20 PM, Paul Eggleton wrote: Hi Robert Sorry I meant to reply earlier. I'm not especially happy with moving the constants to utils.py. Is there any way we can avoid that? I will try to import models.py, but that may make things complicated. // Robert Thanks Paul On

Re: [yocto] [layerindex-web][PATCH 1/1] import_layer.py: add -t option for layer_type

2018-08-30 Thread Paul Eggleton
Hi Robert Sorry I meant to reply earlier. I'm not especially happy with moving the constants to utils.py. Is there any way we can avoid that? Thanks Paul On Thursday, 30 August 2018 2:05:11 PM NZST Robert Yang wrote: > Hi Paul, > > Do you have any comments on this, please ? > > // Robert >

Re: [yocto] [layerindex-web][PATCH 1/1] import_layer.py: add -t option for layer_type

2018-08-29 Thread Robert Yang
Hi Paul, Do you have any comments on this, please ? // Robert On 07/31/2018 03:57 PM, Robert Yang wrote: Hi Paul, On 07/24/2018 04:33 PM, Paul Eggleton wrote: Hi Robert, On Monday, 23 July 2018 12:30:01 PM CEST Robert Yang wrote: Now the logic is: Use options.layer_type if specified,

Re: [yocto] [layerindex-web] RFC: layer index docker fixes

2018-08-29 Thread Paul Eggleton
On Wednesday, 29 August 2018 10:46:27 PM NZST Paul Eggleton wrote: > Hi folks > > I know Konrad promised to update the docker-compose setup to be closer > to what we'd done recently with the docker setup, but I needed something > urgently and wanted to learn how to use docker-compose anyway, so

Re: [yocto] [layerindex-web] RFC: layer index docker fixes

2018-08-29 Thread Paul Eggleton
Hi folks I know Konrad promised to update the docker-compose setup to be closer to what we'd done recently with the docker setup, but I needed something urgently and wanted to learn how to use docker-compose anyway, so I've put this together:

[yocto] [layerindex-web][PATCH 3/3] update: add option to stop immediately on parse errors

2018-08-23 Thread Paul Eggleton
During debugging a parsing issue, we don't really want to continue if a parsing error occurs, and in that situation I usually end up using Ctrl+C to exit early. Add an option to exit immediately upon error to avoid having to do that. Signed-off-by: Paul Eggleton --- layerindex/update.py |

[yocto] [layerindex-web][PATCH 0/3] Minor layer index fixes

2018-08-23 Thread Paul Eggleton
A couple of minor fixes for the recent RRS and other distro integration, plus one additional debugging feature for the update script. The following changes since commit b3e9cb05d22e0961341d87dce3128eeb273544f6: Include errors/warnings in main log in error/warning counts for updates

[yocto] [layerindex-web][PATCH 2/3] Hide navbar on comparison select page

2018-08-23 Thread Paul Eggleton
We don't want to see the navbar on this page because otherwise it lets you select branches and click on Stats etc. which doesn't make any sense in this context. Signed-off-by: Paul Eggleton --- templates/base_toplevel.html | 2 ++

[yocto] [layerindex-web][PATCH 1/3] rrs: fix recipe/maintainer count showing at top

2018-08-23 Thread Paul Eggleton
We were using the main badge class as a selector here to show the recipe or maintainer count, and that meant that it also showed up at the top next to the login button if there were some layers to review. Use a proper id to stop that from happening. Signed-off-by: Paul Eggleton ---

[yocto] [layerindex-web][PATCH 5/5] Include errors/warnings in main log in error/warning counts for updates

2018-08-14 Thread Paul Eggleton
When showing the error/warning counts for update records we need to include any errors/warnings that are shown only in the main update log, so we need to adjust how these are collected. Use a function rather than pure aggregation to give a bit more control, and a {% with ... %} block in the

[yocto] [layerindex-web][PATCH 4/5] update: ensure that exceptions during the update process are logged

2018-08-14 Thread Paul Eggleton
If an exception occurred during the update then we were managing to save the update record, but we did not include the exception traceback in the log for the update. Catch the exception and log it which ensures it gets captured in the update record and still gets printed as well. Signed-off-by:

[yocto] [layerindex-web][PATCH 2/5] update_layer: improve exception handling

2018-08-14 Thread Paul Eggleton
When an exception occurs during the main part of update_layer, we were catching and printing it but that's not enough - we need to do the following as well: * Use logger.error() to print the exception information, so that it gets logged and highlighted as an error in the layer update * Exit

[yocto] [layerindex-web][PATCH 0/5] Update / error handling fixes

2018-08-14 Thread Paul Eggleton
The layer index has recently been failing to update, and that wasn't picked up earlier as the errors weren't actually being collected in the update records. Makesure they are collected and sufficiently highlighted in the UI. The following changes since commit

[yocto] [layerindex-web][PATCH v2 00/31] Distro comparisons (cover letter only)

2018-08-13 Thread Paul Eggleton
After running some more tests with Fedora metadata I've made the following fixes: - Turns out fedora golang rpms already have the processed name so we don't need the stuff in rpmmacros.py, so drop that - Add missing __grep definition - Handle unbracketed expressions e.g. %name instead of

[yocto] [layerindex-web][PATCH 00/31] Distro comparisons (cover letter only)

2018-08-08 Thread Paul Eggleton
This set of changes builds upon the OE-Classic support we had previously to enable importing data from other Linux distributions (or indeed other operating systems) and linking and comparing the packages there to recipes in OE. The initial support is just for importing RPM spec files, using the

[yocto] [layerindex-web][PATCH 0/9] Layer index fixes (cover letter only)

2018-08-06 Thread Paul Eggleton
Some fixes for the Docker setup and bulkchange, plus a couple of unrelated patches. The following changes since commit 65d058237f96a151c913a70ba4d639c7aa0fb0b5: Dockerfile: Reduce image size by merging RUN stages together (2018-07-23 08:40:00 +0200) are available in the Git repository at:

Re: [yocto] [layerindex-web][PATCH 1/1] import_layer.py: add -t option for layer_type

2018-07-31 Thread Robert Yang
Hi Paul, On 07/24/2018 04:33 PM, Paul Eggleton wrote: Hi Robert, On Monday, 23 July 2018 12:30:01 PM CEST Robert Yang wrote: Now the logic is: Use options.layer_type if specified, guess if not, default to 'M'. Note choices=['A', 'B', 'S', 'D', 'M', ''], the '' is for default='', we can't use

Re: [yocto] [layerindex-web][PATCH 1/1] import_layer.py: add -t option for layer_type

2018-07-26 Thread Paul Eggleton
Hi Robert, On Monday, 23 July 2018 12:30:01 PM CEST Robert Yang wrote: > Now the logic is: > Use options.layer_type if specified, guess if not, default to 'M'. > > Note choices=['A', 'B', 'S', 'D', 'M', ''], the '' is for default='', we can't > use default='M' here, otherwise we don't know

[yocto] [layerindex-web][PATCH 1/1] import_layer.py: add -t option for layer_type

2018-07-23 Thread Robert Yang
Now the logic is: Use options.layer_type if specified, guess if not, default to 'M'. Note choices=['A', 'B', 'S', 'D', 'M', ''], the '' is for default='', we can't use default='M' here, otherwise we don't know whether the 'M' is specified by user or not, we don't guess if it is specified by user,

[yocto] [layerindex-web][PATCH 0/1] import_layer.py: add -t option for layer_type

2018-07-23 Thread Robert Yang
The following changes since commit 65d058237f96a151c913a70ba4d639c7aa0fb0b5: Dockerfile: Reduce image size by merging RUN stages together (2018-07-23 08:40:00 +0200) are available in the git repository at: git://git.pokylinux.org/poky-contrib rbt/li

Re: [yocto] [layerindex-web] RFC: layer index docker fixes

2018-07-17 Thread Konrad Scherer
On 07/17/2018 06:16 AM, Paul Eggleton wrote: On Monday, 16 July 2018 10:16:55 PM CEST Paul Eggleton wrote: On Monday, 16 July 2018 8:24:53 PM CEST Konrad Scherer wrote: I recently noticed that if gcc is removed in the 'apt-get purge' step, the bitbake sanity checker will fail due to missing

Re: [yocto] [layerindex-web] RFC: layer index docker fixes

2018-07-17 Thread Paul Eggleton
On Monday, 16 July 2018 10:16:55 PM CEST Paul Eggleton wrote: > On Monday, 16 July 2018 8:24:53 PM CEST Konrad Scherer wrote: > > I recently noticed that if gcc is removed in the 'apt-get purge' step, > > the bitbake sanity checker will fail due to missing 'gcc' when doing a > > layer update. >

Re: [yocto] [layerindex-web] RFC: layer index docker fixes

2018-07-16 Thread Paul Eggleton
On Monday, 16 July 2018 8:24:53 PM CEST Konrad Scherer wrote: > On 07/13/2018 03:29 AM, Paul Eggleton wrote: > > On Thursday, 12 July 2018 10:16:12 PM CEST you wrote: > >> I just noticed that the image doesn't contain git. Was this > >> deliberate or an oversight? > > > > Ah, no, it wasn't

Re: [yocto] [layerindex-web] RFC: layer index docker fixes

2018-07-16 Thread Konrad Scherer
On 07/13/2018 03:29 AM, Paul Eggleton wrote: On Thursday, 12 July 2018 10:16:12 PM CEST you wrote: I just noticed that the image doesn't contain git. Was this deliberate or an oversight? Ah, no, it wasn't deliberate - perhaps it was there earlier with buildpack- deps? It doesn't appear that

Re: [yocto] [layerindex-web] RFC: layer index docker fixes

2018-07-13 Thread Paul Eggleton
On Thursday, 12 July 2018 10:16:12 PM CEST you wrote: > I just noticed that the image doesn't contain git. Was this > deliberate or an oversight? Ah, no, it wasn't deliberate - perhaps it was there earlier with buildpack- deps? It doesn't appear that we ever explicitly had it in there. I've

Re: [yocto] [layerindex-web] RFC: layer index docker fixes

2018-07-12 Thread Konrad Scherer
On 07/12/2018 02:34 AM, Paul Eggleton wrote: Hi Konrad On Wednesday, 11 July 2018 7:03:28 PM CEST Konrad Scherer wrote: I have attached a patch to reduce the layerindex-web image size further. Docker image layers are like git objects and RUN commands can only hide files, not remove them once

Re: [yocto] [layerindex-web] RFC: layer index docker fixes

2018-07-12 Thread Paul Eggleton
Hi Konrad On Wednesday, 11 July 2018 7:03:28 PM CEST Konrad Scherer wrote: > I have attached a patch to reduce the layerindex-web image size further. > Docker image layers are like git objects and RUN commands can only hide > files, not remove them once they have been committed. The pip upgrade

Re: [yocto] [layerindex-web] RFC: layer index docker fixes

2018-07-11 Thread Konrad Scherer
On 07/10/2018 12:17 PM, Paul Eggleton wrote: Hi folks I've been trying to get the OE layer index example docker setup in shape, based upon improvements from myself, Michael and Konrad. For the moment I've taken most of what Michael and I did (with some fixes and tweaks, and broken up into

[yocto] [layerindex-web] RFC: layer index docker fixes

2018-07-10 Thread Paul Eggleton
Hi folks I've been trying to get the OE layer index example docker setup in shape, based upon improvements from myself, Michael and Konrad. For the moment I've taken most of what Michael and I did (with some fixes and tweaks, and broken up into smaller patches) and a few pieces of the Dockerfile

Re: [yocto] [layerindex-web][PATCH 1/6] Add docker-compose file to create full layerindex stack of MariaDB, RabbitMQ and Nginx

2018-07-10 Thread Paul Eggleton
Hi Konrad On Tuesday, 26 June 2018 7:41:30 PM CEST Konrad Scherer wrote: > Lots of new features added: > > - Layerindex runs as unprivileged user inside container > > - Celery worker is started before gunicorn > > - Entrypoint script supports changing RabbitMQ location > > - Entrypoint script

Re: [yocto] [layerindex-web][PATCH 4/4] update.py: check whether branch existed when nocheckout

2018-07-09 Thread Paul Eggleton
On Monday, 9 July 2018 10:45:58 AM CEST Robert Yang wrote: > On 07/09/2018 04:36 PM, Paul Eggleton wrote: > > On Monday, 9 July 2018 10:34:07 AM CEST Paul Eggleton wrote: > >> On Monday, 9 July 2018 6:11:30 AM CEST Robert Yang wrote: > >>> Fixed: > >>> Assume there is no master branch in hello

Re: [yocto] [layerindex-web][PATCH 4/4] update.py: check whether branch existed when nocheckout

2018-07-09 Thread Robert Yang
On 07/09/2018 04:36 PM, Paul Eggleton wrote: On Monday, 9 July 2018 10:34:07 AM CEST Paul Eggleton wrote: Hi Robert, On Monday, 9 July 2018 6:11:30 AM CEST Robert Yang wrote: Fixed: Assume there is no master branch in hello layer: $ update.py -l hello -b master INFO: Skipping update of

Re: [yocto] [layerindex-web][PATCH 4/4] update.py: check whether branch existed when nocheckout

2018-07-09 Thread Robert Yang
Hi Paul, On 07/09/2018 04:34 PM, Paul Eggleton wrote: Hi Robert, On Monday, 9 July 2018 6:11:30 AM CEST Robert Yang wrote: Fixed: Assume there is no master branch in hello layer: $ update.py -l hello -b master INFO: Skipping update of layer hello - branch master doesn't exist This is correct

Re: [yocto] [layerindex-web][PATCH 4/4] update.py: check whether branch existed when nocheckout

2018-07-09 Thread Paul Eggleton
On Monday, 9 July 2018 10:34:07 AM CEST Paul Eggleton wrote: > Hi Robert, > > On Monday, 9 July 2018 6:11:30 AM CEST Robert Yang wrote: > > Fixed: > > Assume there is no master branch in hello layer: > > $ update.py -l hello -b master > > INFO: Skipping update of layer hello - branch master

Re: [yocto] [layerindex-web][PATCH 4/4] update.py: check whether branch existed when nocheckout

2018-07-09 Thread Paul Eggleton
Hi Robert, On Monday, 9 July 2018 6:11:30 AM CEST Robert Yang wrote: > Fixed: > Assume there is no master branch in hello layer: > $ update.py -l hello -b master > INFO: Skipping update of layer hello - branch master doesn't exist > > This is correct since hello layer doesn't have master branch,

Re: [yocto] [layerindex-web][PATCH 5/7] update: ignore recommends when ordering layers

2018-07-08 Thread Robert Yang
Hi Paul, I've sent the patches to mailing list: [yocto] [layerindex-web][PATCH 0/4] update.py: several fixes The one for recommends is: update.py: add layers when RECOMMENDS isn't satisfied Now you can drop [PATCH 5/7], others are still needed. // Robert On 07/06/2018 02:49 PM, Robert

[yocto] [layerindex-web][PATCH 2/4] utils.py: fix checkout_repo when no HEAD

2018-07-08 Thread Robert Yang
Fixed: $ git clone warning: remote HEAD refers to nonexistent ref, unable to checkout. $ git rev-parse HEAD HEAD fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree. Use '--' to separate paths from revisions, like this: 'git [...] -- [...]' Catch the error and

[yocto] [layerindex-web][PATCH 4/4] update.py: check whether branch existed when nocheckout

2018-07-08 Thread Robert Yang
Fixed: Assume there is no master branch in hello layer: $ update.py -l hello -b master INFO: Skipping update of layer hello - branch master doesn't exist This is correct since hello layer doesn't have master branch, but when --nocheckout: $ update.py -l hello -b master --nocheckout [snip] INFO:

[yocto] [layerindex-web][PATCH 3/4] update.py: add layers when RECOMMENDS isn't satisfied

2018-07-08 Thread Robert Yang
When layer_a RECOMMENDS layer_b, try to add layer_b before layer_a, but if layer_b is not found, still add layer_a. And print summary error mesage: $ update.py -b master ERROR: Issues found on branch master: openembedded-core: Added without LAYERRECOMMENDS meta-secure-env: Failed to add

[yocto] [layerindex-web][PATCH 1/4] update_layer.py: avoid calling setup_core_layer_sys_path() when --initial

2018-07-08 Thread Robert Yang
Fixed: $ update.py -b [snip] NOTE: Starting bitbake server... Traceback (most recent call last): File "update_layer.py", line 471, in main utils.setup_core_layer_sys_path(settings, branch.name) File "/buildarea1/lyang1/layerindex-web/layerindex/utils.py", line 376, in

[yocto] [layerindex-web][PATCH 0/4] update.py: several fixes

2018-07-08 Thread Robert Yang
The following changes since commit 5cfdfdca8b9200b5a6e4aa14661b14059a799310: rrs_upstream_history.py: fix set_regexes function (2018-06-06 11:25:46 +1200) are available in the git repository at: git://git.openembedded.org/openembedded-core-contrib rbt/li_recs

Re: [yocto] [layerindex-web][PATCH 5/7] update: ignore recommends when ordering layers

2018-07-06 Thread Robert Yang
On 07/06/2018 01:28 PM, Paul Eggleton wrote: Hi Robert On Wednesday, 4 July 2018 7:52:05 PM NZST you wrote: I'm sorry to say that I met layerindex' loaddata problems yesterday and today, I still didn't find the root cause. Have you tried dumpdata and loaddata recently, please ? What I did

Re: [yocto] [layerindex-web][PATCH 5/7] update: ignore recommends when ordering layers

2018-07-05 Thread Paul Eggleton
Hi Robert On Wednesday, 4 July 2018 7:52:05 PM NZST you wrote: > I'm sorry to say that I met layerindex' loaddata problems yesterday and > today, > I still didn't find the root cause. Have you tried dumpdata and loaddata > recently, please ? > > What I did was: > > $ python3 manage.py dumpdata

Re: [yocto] [layerindex-web][PATCH 5/7] update: ignore recommends when ordering layers

2018-07-04 Thread Robert Yang
Hi Paul, I'm sorry to say that I met layerindex' loaddata problems yesterday and today, I still didn't find the root cause. Have you tried dumpdata and loaddata recently, please ? What I did was: $ python3 manage.py dumpdata --settings settings --exclude=contenttypes --exclude=auth.Permission

Re: [yocto] [layerindex-web][PATCH 5/7] update: ignore recommends when ordering layers

2018-07-02 Thread Robert Yang
On 07/03/2018 10:58 AM, Paul Eggleton wrote: Hi Robert On Tuesday, 3 July 2018 2:45:11 PM NZST Robert Yang wrote: Thanks for let me know this, this patch might be incorrect, suppose we have two layers: core and hello: 1) LAYERRECOMMENDS_core = "hello" 2) $ update.py -l hello,core Then

Re: [yocto] [layerindex-web][PATCH 5/7] update: ignore recommends when ordering layers

2018-07-02 Thread Paul Eggleton
Hi Robert On Tuesday, 3 July 2018 2:45:11 PM NZST Robert Yang wrote: > Thanks for let me know this, this patch might be incorrect, suppose we have > two > layers: core and hello: > > 1) LAYERRECOMMENDS_core = "hello" > 2) $ update.py -l hello,core > > Then core maybe added before hello layer

Re: [yocto] [layerindex-web][PATCH 5/7] update: ignore recommends when ordering layers

2018-07-02 Thread Robert Yang
Hi Paul, Thanks for let me know this, this patch might be incorrect, suppose we have two layers: core and hello: 1) LAYERRECOMMENDS_core = "hello" 2) $ update.py -l hello,core Then core maybe added before hello layer since it ignores recs on hello, and if hello is a new layer, it would not be

  1   2   3   4   >