Re: [yocto] Yocto and NPM issues

[Svein Seldal]

On 21.03.2018 14:54, Alexander Kanavin wrote:
I'm attempting to package a npm-based js application into a Rocko 
image, and I'm having some issues. For testing I use the official poky 
repo and the meta-openembedded repo (for installing nodejs 8.4.0). I'm 
building for qemux86-64.


I know this is not an answer to your specific issues, but npm is a bit 
of maintenance nightmare as its workflow (downloading random stuff off 
the internet as an integral part of the build) clashes badly with how 
bitbake, and embedded world in general handle builds. We've discussed 
what to do about it, with no clear outcome, but we would like to have 
something that doesn't require constant fixing and hacks:


http://lists.openembedded.org/pipermail/openembedded-architecture/2017-March/000480.html 


I haven't read the post in full yet, but the principle of npm is 
inherently different from the philosophies of Yocto, granted.


One approach that we've been discussing as an alternative to the 
yocto+npm approach is to install the js app on a target machine and 
snapshot the whole resulting directory hierarchy. The product validation 
is then made on the modules as a whole. The recipe would need to be a 
simple file-installer.


The problem with that is that one needs a target deployment system that 
runs npm install on the actual target in order to make the correct files 
for the target. The output from that must be plugged into the oe build 
system somehow, so there will be many moving parts in this scheme too.


> Oh, and you might have more success using meta-nodejs, as it doesn't try
> to implement a custom fetcher, and just runs npm directly during the
> build. If you don't mind that that makes the build outcome essentially
> random from one invocation to the next.
>
> https://github.com/imyller/meta-nodejs

The meta-nodejs repo seems to be abandoned, as it does support any newer 
versions of nodejs. I asked on #yocto and I was suggested that this 
layer wasn't really necessary as meta-openembedded supports nodejs.



Best regards,
Svein
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

[yocto] Yocto and NPM issues

[Svein Seldal]

Hi

I'm attempting to package a npm-based js application into a Rocko image, 
and I'm having some issues. For testing I use the official poky repo and 
the meta-openembedded repo (for installing nodejs 8.4.0). I'm building 
for qemux86-64.



Issue 1: NPM builds not working in Rocko

If I'm following the examples outlined in 
https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM, and run devtool 
add "npm://registry.npmjs.org;name=cute-files;version=1.0.2" and then 
bitbake cute-file. It fails on an npm error seen below. I found that 
this patch fixes the problem: 
https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/meta/classes/npm.bbclass?id=d38e1e2c2ea4646b34ea6282d3d7620df5b0374b


Will there be more releases of rocko? Can I request that this patch is 
backported to the rocko branch please?



Issue 2: Workspace npm recipe does not work after bitbake -c cleanall

If using the devtool/workspace for the above recipe, and run bitbake -c 
cleanall cute-files, then try to rerun bitbake cute-files, it will warn 
on missing LICENSE files for the npm packages and fail on 
LIC_FILES_CHKSUM point to an invalid file QA error. E.g. something in 
line of:


WARNING: cute-files-1.0.2-r0 do_populate_lic: Could not copy license 
file 
/home/common/yocto/yocto-poky/build/workspace/sources/cute-files/node_modules/express/node_modules/serve-static/node_modules/send/node_modules/ms/license.md 
to 
/home/common/yocto/yocto-poky/build/tmp/work/core2-64-poky-linux/cute-files/1.0.2-r0/license-destdir/cute-files/license.md: 
[Errno 2] No such file or directory: 
'/home/common/yocto/yocto-poky/build/workspace/sources/cute-files/node_modules/express/node_modules/serve-static/node_modules/send/node_modules/ms/license.md'
ERROR: cute-files-1.0.2-r0 do_populate_lic: QA Issue: cute-files: 
LIC_FILES_CHKSUM points to an invalid file: 
/home/common/yocto/yocto-poky/build/workspace/sources/cute-files/node_modules/express/node_modules/accepts/LICENSE 
[license-checksum]



Issue 3: npm recipe fails to download npm images after -c cleanall

If the recipe is copied into a normal layer and run without externalsrc, 
and runned after a -c cleanall, the following error is returned:


WARNING: cute-files-1.0.2-r0 do_fetch: Checksum failure encountered with 
download of npm://registry.npmjs.org;name=cute-files;version=1.0.2 - 
will attempt other sources if available

ERROR: cute-files-1.0.2-r0 do_fetch: Fetcher failure: Checksum mismatch!
File: 'commander-2.15.1.tgz' has sha1 checksum 
df46e867d0fc2aec66a34662b406a9ccafff5b0f when * was expected
ERROR: cute-files-1.0.2-r0 do_fetch: Fetcher failure for URL: 
'npm://registry.npmjs.org;name=cute-files;version=1.0.2'. Unable to 
fetch URL from any source.

ERROR: cute-files-1.0.2-r0 do_fetch: Function failed: base_do_fetch
ERROR: Logfile of failure stored in: 
/home/common/yocto/yocto-poky/build/tmp/work/core2-64-poky-linux/cute-files/1.0.2-r0/temp/log.do_fetch.20513
ERROR: Task 
(/home/common/yocto/yocto-poky/meta-foobar/recipes/cute-files/cute-files_1.0.2.bb:do_fetch) 
failed with exit code '1'



The only way to circumvent issue 2 and 3 after a -c cleanall is to use 
the devtool to download the packages, and then build the package as 
normal.



Best regards,
Svein




FAILURE Building cute-files:

ERROR: cute-files-1.0.2-r0 do_compile: Function failed: do_compile (log 
file is located at 
/home/common/yocto/yocto-poky/build/tmp/work/core2-64-poky-linux/cute-files/1.0.2-r0/temp/log.do_compile.9536)
ERROR: Logfile of failure stored in: 
/home/common/yocto/yocto-poky/build/tmp/work/core2-64-poky-linux/cute-files/1.0.2-r0/temp/log.do_compile.9536

Log data follows:
| DEBUG: Executing python function externalsrc_compile_prefunc
| NOTE: cute-files: compiling from external source tree 
/home/common/yocto/yocto-poky/build/workspace/sources/cute-files

| DEBUG: Python function externalsrc_compile_prefunc finished
| DEBUG: Executing shell function do_compile
| npm ERR! As of npm@5, the npm cache self-heals from corruption issues 
and data extracted from the cache is guaranteed to be valid. If you want 
to make sure everything is consistent, use 'npm cache verify' instead.

| npm ERR!
| npm ERR! If you're sure you want to delete the entire cache, rerun 
this command with --force.

|
| npm ERR! A complete log of this run can be found in:
| npm ERR! 
/home/common/yocto/yocto-poky/build/tmp/work/core2-64-poky-linux/cute-files/1.0.2-r0/npm_cache/_logs/2018-03-21T12_36_48_034Z-debug.log

| WARNING: exit code 1 from a shell command.
| ERROR: Function failed: do_compile (log file is located at 
/home/common/yocto/yocto-poky/build/tmp/work/core2-64-poky-linux/cute-files/1.0.2-r0/temp/log.do_compile.9536)
ERROR: Task 
(/home/common/yocto/yocto-poky/build/workspace/recipes/cute-files/cute-files_1.0.2.bb:do_compile) 
failed with exit code '1'

--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

[yocto] Workdir of qmake5 recipes in Pyro

[Svein Seldal]

Hi

I have a qt5 recipe which is built for a imx6 system with 
MACHINE=lm_cm-poky-linux-gnueabi). The recipe sp.bb contains:


include qmake5
# PACKAGE_ARCH = "${MACHINE_ARCH}"

When I build this on Krogoth I find the build for my recipe under:

   tmp/work/cortexa9hf-neon-poky-linux-gnueabi/sp/

I notice that qtbase is found under:

   tmp/work/cortexa9hf-neon-mx6qdl-poky-linux-gnueabi/qtbase/

When I've changed Poky to Pyro, I noticed that my recipe is now build in:

   tmp/work/cortexa9hf-neon-mx6qdl-poky-linux-gnueabi/sp/

but I haven't made any changes to my recipe. All of my other non-qt 
recipes still place their build output in the other directory.


Why does this change happen? Am I missing something, e.g. a PACKAGE_ARCH 
statement?



Best regards,
Svein
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] release management

[Svein Seldal]

I have also been slightly puzzled over the meticulousness and 
determinism of Yocto on recipe level, but the left-to-yourself thinking 
for the top-level meta-level and configuration management.


We have chosen to make a custom configuration management (CM) and 
lightweight build system on top of Yocto. Firstly, we have mixed SCM, 
mixing both git and hg, and the CM contains a manifest of the sources 
and can handle proper checkout (which repo does too for git).


Secondly, the CM provides a standardized unified location for setting 
configuration options and control of what artifacts to generate. This is 
then independent of what sub-system, like Yocto, is being run behind the 
scenes. Included in this is organization specific systems, such as 
central defining of system version and build numbers.


The CM also contains some postprocessors for Yocto. In the versions of 
Yocto we use, Yocto cannot generate artifacts for more than one MACHINE 
for each bitbake invocation. We must pack all these output artifacts 
into a common releasable end-user image and upload it to our PLM system. 
This part I have not been able to do (cleanly) from a recipe since it 
spans multiple MACHINES.


So our conclusion is that you need to know what Yocto is good at and 
what it can do, and perhaps rely on other systems where this is 
required. YMMV


Best regards,
Svein


On 19. aug. 2017 00:58, Chris Z. wrote:

Hi,

Regarding submodule for layers. This is good when no one beside you use 
it. Especially  with yocto layers. But when dev starts to use it. You or 
other support team will be faced against wall of problems with basic 
understanding of git submodules and index. But google starts to invest 
some funds in submodules instead repo usage. So it can be a game changer 
in favor of git submodules.


 From own experience (dev team of 80 persons) we decided to leave 
submodules in favor of repo tool. But we had the same conclusion at 
first. Why to use other tool were git provides submodules. Further 
development and expansion of layers are pro the repo tool manifest file. 
But that is from my own exp. And it was good change.


Br,
Chris Z

15.08.2017 15:16 "Joshua Watt" > napisał(a):


On Tue, 2017-08-15 at 09:25 +0200, Mike Looijmans wrote:
 > On 14-08-17 21:10, Gunnar Andersson wrote:
 > >
 > > Hey Russell
 > >
 > > I don't claim to be an authority on best practices but I will share
 > > some
 > > thoughts.
 > >
 > > On Sun, 2017-08-13 at 06:58 -0400, Russell Peterson wrote:
 > > > Hello.
 > > >
 > > > As I learn more about yocto and more importantly gain practical
 > > > experience
 > > > with it I have started to think about my release structure.  Is
 > > > there a
 > > > “best practices” document or something like that that speaks to
 > > > this?
 > > > For example, how does everyone deal with “external” meta layer
 > > > dependencies?  My software uses poky and meta-openembedded, of
 > > > course.
 > >
 > > We simply use a parent project [1] that includes git submodules,
 > > one per
 > > yocto layer.  I'm of the opinion that if git (only) can be used,
 > > why
 > > introduce other tools?   But it requires you to learn and master
 > > that
 > > feature.
 > >
 >
 > We arrived at exactly the same setup. One parent repository that
 > contains the
 > project-unique recipes, and submodules for the layers it needs.
 >
 > It's a lot better than attempting to script things. For one thing,
 > git will
 > tell you the state of the submodules.
 >
 > If I rembemer correctly, Android builds use a tool to manage the
 > layer
 > repositories. I didn't much like it though. And "yet another tool"...

It's called "repo" https://code.google.com/archive/p/git-repo/
.

In my experience git submodules work fine if you follow the core
workflow of adding new submodules and updating them to newer versions.
The workflows for deleting, renaming, changed upstream repostiory, etc.
for submodules is IHMO pretty bad. Pretty much every time I've tried
that it has resulted in a borked local repository that I either had to
manually go into the .git directory and fix, or just start over with a
fresh clone (perhaps I'm just bad at it or newer versions of git are
better?). Not to say you shouldn't use them, just be aware of some of
the catches so you can match the tool to your expected workflow.

For my $0.02: We use git submodules to manage the multiple yocto trees
we are pulling in and also keep "parallel" layers for each with the
local changes we want to make in .bbappends, where possible (following
the strategy of no modifying the original). We do (particularly for
older versions of Yocto) make changes to the core since we 

Re: [yocto] Path to current bb-file or layer

[Svein Seldal]

On 22. sep. 2017 13:42, Richard Purdie wrote:

A better way to get what you're after may be to look at the contents
of BBINCLUDED.


Ah, yes. Perfect. You got me onto a lead: Setting SP_BASEDIR := 
"${LAYERDIR}" in layer.conf and then access this variable from the 
bbclass file.


Although I would like to suggest that bitbake should have a FILE-ish 
variable that points to the actual running current file. Especially when 
bbclass files is thought to act as generic black-box-ish collection of 
funtions.


Thank you Richard


Best regards,
Svein
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] Path to current bb-file or layer

[Svein Seldal]

On 22. sep. 2017 10:30, Richard Purdie wrote:

> Its all about when your code is executed. Try putting:
> 
> SP_BASEVERSION := "${@read_spbaseversion(d)}"
> 
> in the bbclass file. I suspect that will give you the bbclass file
> name.
> 
> By default everything is deferred expansion so your code would give you
> the final .bb as that is the context the code is run in.

In local.conf I put:

 INHERIT += "sp-version"

In classes/sp-version.bbclass

 def read_spbaseversion(d):
# The goal of this function is to read and
# parse a file from the layer
 bb.warn("MYSELF: ", d.getVar('FILE', False))
 return '0'

 SP_BASEVERSION := "${@read_spbaseversion(d)}"

Then both on Krogoth and Pyro, the following command returns:

 $ bitbake -e |tee var.txt |grep MYSELF

WARNING: MYSELF: /home/s/bughunt-pyro/build/conf/bblayers.conf
WARNING: MYSELF: /home/s/bughunt-pyro/build/conf/bblayers.conf
 bb.warn("MYSELF: ", d.getVar('FILE', False))

Likewise:

 $ bitbake -e sp-image |tee sp-image.txt |grep MYSELF

WARNING: MYSELF: /home/s/bughunt-pyro/build/conf/bblayers.conf
WARNING: MYSELF: /home/s/bughunt-pyro/build/conf/bblayers.conf
WARNING: MYSELF: /home/s/bughunt-pyro/build/conf/bblayers.conf
 bb.warn("MYSELF: ", d.getVar('FILE', False))


Is there an alternative to d.getVar() to access the "other" versions of 
the variable?


Best regards,
Svein
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] Path to current bb-file or layer

[Svein Seldal]

On 21. sep. 2017 23:36, Richard Purdie wrote:
> You mean like ${FILE} ?
>
> $ bitbake bash -e | grep ^FILE=
> FILE="/media/build1/poky/meta/recipes-extended/bash/bash_4.4.bb"

To me ${FILE} seems to be pointing towards the receipe, and not the 
current .bbclass file as you'd expect.


If I put this in local.conf

INHERIT += "sp-version"
SP_BASEVERSION = "${@read_spbaseversion(d)}"

And in sp-version.bbclass:

def read_spbaseversion(d):
bb.warn("SVEIN FILE: ", d.getVar('FILE', False))
return '0'

If I do bitbake -e, I get

WARNING: SVEIN FILE: /home/s/build-sp-image/build/conf/bblayers.conf

Later the read_spbaseversion is used from within recipe (although they 
don't need to as this variable goes into the global scope, but for the 
sake of the demonstration). Then the value being prined is:


WARNING: /home/s/build-sp-image/meta-sp/sp/sp.bb: SVEIN OPEN: 
/home/s/build-sp-image/meta-sp/sp/sp.bb


Hence, ${FILE} seems to point to the current file scope, like a recipe, 
and not any class files that the recipe might take use of.



Best regards,
Svein
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

[yocto] Path to current bb-file or layer

[Svein Seldal]

To determine a image version, I'd like to read a VERSION file which is 
located in the root of the layer. (Per our development procedure.) I'd 
like to read it from a bbclass file. However I seem to be unable to find 
any methods or variables to find a useful path to either the current 
bb-file or the root of the layer.


The only way I have found is to parse through BBLAYERS and guess at what 
my own layer is of those, and then use the found to access the file. But 
this feels very wacky.


Is there a reason why bitbake doesn't have a variable path reference to 
the current file?



Best regards,
Svein
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] Sysroot bug in bitbake or wrong configuration?

[Svein Seldal]

Perfect, that worked.

On 21. sep. 2017 03:45, Andre McCurdy wrote:
> Since image recipes don't have a do_configure task (or at least, they
> do their work in tasks such as do_rootfs which don't depend on
> do_configure), using the DEPENDS shorthand for setting dependencies
> for the do_configure task doesn't work.

Technically, this isn't a Yocto image recipe per se, but a general 
recipe for processing some file which happens to have the name "image" 
in it. The naming was perhaps a little misleading.


Notice that this recipe does not disable the default tasks, so all of 
them run, including do_configure. What I learned after some 
experimentation is that DEPENDS seems to works fine if you place your 
custom tasks after the do_configure tasks:


addtask do_spu_rootfs before do_build after do_compile

This construct seems to work even if you disable the confiugre task with 
do_configure[noexec]="1"


But agreed, this works because of the implicit dependency on 
do_populate_* by do_compile. Setting the dep explicitly with the 
"longhand" format is more robust.



Best regards,
Svein
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

[yocto] Sysroot bug in bitbake or wrong configuration?

[Svein Seldal]

I have the spu-image.bb recipe below, and running on Pyro, the recipe 
behaves differently if the recipe is run on a fresh system with no 
sstate elements, compared to a system that has a sstate cache present.


The failure is that the spu-image required the host tool "uuidgen", and 
thus has DEPENDS on "util-linux-native". When the -c cleanall spu-image 
is run prior to building spu-image, the recipe sysroot is properly 
initialized with util-linux-native and uuidgen is available in the task 
functions.


If -c clean is run prior to build, or simply by deleting tmp, the 
sysroot will not be properly initialized and uuidgen is not available 
and the recipe fails


Is this a bug in bitbake or am I missing something in my recipe?


Best regards,
Svein Seldal


# spu-image.bb
DESCRIPTION = "Upgrade Image"

LICENSE = "MIT"
LIC_FILES_CHKSUM = 
"file://${COREBASE}/LICENSE;md5=4d92cd373abda3937c2bc47fbc49d690"


DEPENDS = "util-linux-native"
INHIBIT_DEFAULT_DEPS = "1"

fakeroot do_spu_rootfs() {
uuidgen
}
addtask do_spu_rootfs before do_build

fakeroot do_spu_image () {
uuidgen
}
addtask do_spu_image after do_spu_rootfs before do_build

# It does not matter if these are noexec-ed or not
#do_fetch[noexec] = "1"
#do_unpack[noexec] = "1"
#do_patch[noexec] = "1"
#do_configure[noexec] = "1"
#do_compile[noexec] = "1"
#do_install[noexec] = "1"
#do_package[noexec] = "1"
#do_package_qa[noexec] = "1"
#do_packagedata[noexec] = "1"
#do_package_write_ipk[noexec] = "1"
#do_package_write_deb[noexec] = "1"
#do_package_write_rpm[noexec] = "1"


# 1) Running works fine
#   bitbake -v spu-image |tee log1.txt
#   cat log1.txt | grep -2 uuidgen
#
# 2) Cleaning
#   bitbake -c clean spu-image
#
# 3) Rebuilding -- now fails
#   bitbake -v spu-image |tee log2.txt
#   cat log2.txt | grep -2 uuidgen
#
# 4) Sstate cleaning
#   bitbake -c cleanall spu-image
#
# 5) Works again:
#   bitbake -v spu-image |tee log3.txt
#   cat log3.txt | grep -2 uuidgen

--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

[yocto] Failing EXTERNALSRC and dangling symlinks

[Svein Seldal]

I am running on a customized Yocto tree based off Poky krogoth and I 
think I have stumbled upon a bug in bb.


One of our recipes use EXTERNALSRC and EXTERNALSRC_BUILD, set from 
local.conf. When the recipe is built, two symlinks 
${EXTERNALSRC}/oe-logs and ${EXTERNALSRC}/oe-workdir is created in the 
source tree.


If the build output tmp/ later is deleted, these two symlinks becomes 
dangling links. Re-running bitbake will cause it to fail. First it warns 
of this during parsing, and later it will crash bb (see below).


Removing these symlinks when deleting tmp/ ensures bitbake will not crash.

I suppose this is a bug?

Best regards,
Svein Seldal



(EXTERNALSRC="/srv/user/yocto/sp-core-image/src")

WARNING: Unable to get checksum for sp SRC_URI entry oe-logs: [Errno 2] 
No such file or directory: '/home/user/yocto/sp-core-image/src/oe-logs'
WARNING: Unable to get checksum for sp SRC_URI entry oe-workdir: [Errno 
2] No such file or directory: 
'/home/user/yocto/sp-core-image/src/oe-workdir'



ERROR: sp-99.8.0-r0 do_compile: Build of do_compile failed
ERROR: sp-99.8.0-r0 do_compile: Traceback (most recent call last):
  File 
"/home/user/yocto/sp-core-image/src/poky/bitbake/lib/bb/build.py", line 
567, in exec_task

return _exec_task(fn, task, d, quieterr)
  File 
"/home/user/yocto/sp-core-image/src/poky/bitbake/lib/bb/build.py", line 
541, in _exec_task

event.fire(TaskSucceeded(task, logfn, localdata), localdata)
  File 
"/home/user/yocto/sp-core-image/src/poky/bitbake/lib/bb/event.py", line 
171, in fire

fire_class_handlers(event, d)
  File 
"/home/user/yocto/sp-core-image/src/poky/bitbake/lib/bb/event.py", line 
110, in fire_class_handlers

execute_handler(name, handler, event, d)
  File 
"/home/user/yocto/sp-core-image/src/poky/bitbake/lib/bb/event.py", line 
82, in execute_handler

ret = handler(event)
  File 
"/home/user/yocto/sp-core-image/src/poky/meta/classes/sstate.bbclass", 
line 946, in sstate_eventhandler
bb.siggen.dump_this_task(sstatepkg + '_' + taskname + ".tgz" 
".siginfo", d)
  File 
"/home/user/yocto/sp-core-image/src/poky/bitbake/lib/bb/siggen.py", line 
348, in dump_this_task
bb.parse.siggen.dump_sigtask(fn, task, outfile, "customfile:" + 
referencestamp)
  File 
"/home/user/yocto/sp-core-image/src/poky/meta/lib/oe/sstatesig.py", line 
182, in dump_sigtask
super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigtask(fn, 
task, stampbase, runtime)
  File 
"/home/user/yocto/sp-core-image/src/poky/bitbake/lib/bb/siggen.py", line 
303, in dump_sigtask

computed_taskhash = calc_taskhash(data)
  File 
"/home/user/yocto/sp-core-image/src/poky/bitbake/lib/bb/siggen.py", line 
548, in calc_taskhash

data = data + c[1]
TypeError: cannot concatenate 'str' and 'NoneType' objects
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto