[OE-core] [PATCH 4/4] recipetool: go: apply pylint recommendations

2024-02-26 Thread Lukas Funke 
From: Lukas Funke 

Apply pylint recommendations where reasanable. Not all
recommendations were applied:
 - long regex expressions were not split for a better readability
 - not all (short) variables were renamed in order to be consistent
   with the rest of the framework (e.g. 'd')

Signed-off-by: Lukas Funke 
---
 scripts/lib/recipetool/create_go.py | 357 
 1 file changed, 205 insertions(+), 152 deletions(-)

diff --git a/scripts/lib/recipetool/create_go.py 
b/scripts/lib/recipetool/create_go.py
index cae7175a33..41161b27f5 100644
--- a/scripts/lib/recipetool/create_go.py
+++ b/scripts/lib/recipetool/create_go.py
@@ -9,16 +9,6 @@
 # SPDX-License-Identifier: GPL-2.0-only
 #
 
-
-from collections import namedtuple
-from enum import Enum
-from html.parser import HTMLParser
-from recipetool.create import RecipeHandler, handle_license_vars
-from recipetool.create import guess_license, tidy_licenses, fixup_license
-from recipetool.create import determine_from_url
-from urllib.error import URLError
-
-import bb.utils
 import json
 import logging
 import os
@@ -30,6 +20,17 @@ import tempfile
 import urllib.parse
 import urllib.request
 
+from collections import namedtuple
+from enum import Enum
+from html.parser import HTMLParser
+from urllib.error import URLError
+
+import bb.utils
+
+from recipetool.create import RecipeHandler, handle_license_vars
+from recipetool.create import guess_license, tidy_licenses, fixup_license
+from recipetool.create import determine_from_url
+
 
 GoImport = namedtuple('GoImport', 'root vcs url suffix')
 logger = logging.getLogger('recipetool')
@@ -78,12 +79,14 @@ class GoRecipeHandler(RecipeHandler):
 
 if not os.path.exists(gopath):
 logger.error(
-'%s required to process specified source, but %s did not 
seem to populate it' % 'go', recipe)
+'%s required to process specified source, '
+'but %s did not seem to populate it', 'go', recipe)
 return None
 
 return bindir
 
-def __resolve_repository_static(self, modulepath):
+@staticmethod
+def __resolve_repository_static(modulepath):
 """Resolve the repository in a static manner
 
 The method is based on the go implementation of
@@ -110,83 +113,84 @@ class GoRecipeHandler(RecipeHandler):
 # contains the subdir and major path. Thus,
 # we ignore this error for now
 logger.debug(
-1, "Failed to fetch page from [%s]: %s" % (url, str(url_err)))
+"Failed to fetch page from [%s]: %s", url, str(url_err))
 
 host, _, _ = modulepath.partition('/')
 
-class vcs(Enum):
-pathprefix = "pathprefix"
-regexp = "regexp"
-type = "type"
-repo = "repo"
-check = "check"
-schemelessRepo = "schemelessRepo"
+class Vcs(Enum):
+"""Version control system enum"""
+PATHPREFIX = "pathprefix"
+REGEXP = "regexp"
+TYPE = "type"
+REPO = "repo"
+CHECK = "check"
+SCHEMELESS_REPO = "schemelessRepo"
 
 # GitHub
 vcsGitHub = {}
-vcsGitHub[vcs.pathprefix] = "github.com"
-vcsGitHub[vcs.regexp] = re.compile(
+vcsGitHub[Vcs.PATHPREFIX] = "github.com"
+vcsGitHub[Vcs.REGEXP] = re.compile(
 
r'^(?Pgithub\.com/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(/(?P[A-Za-z0-9_.\-]+))*$')
-vcsGitHub[vcs.type] = "git"
-vcsGitHub[vcs.repo] = "https://\\g"
+vcsGitHub[Vcs.TYPE] = "git"
+vcsGitHub[Vcs.REPO] = "https://\\g"
 
 # Bitbucket
 vcsBitbucket = {}
-vcsBitbucket[vcs.pathprefix] = "bitbucket.org"
-vcsBitbucket[vcs.regexp] = re.compile(
+vcsBitbucket[Vcs.PATHPREFIX] = "bitbucket.org"
+vcsBitbucket[Vcs.REGEXP] = re.compile(
 
r'^(?Pbitbucket\.org/(?P[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+))(/(?P[A-Za-z0-9_.\-]+))*$')
-vcsBitbucket[vcs.type] = "git"
-vcsBitbucket[vcs.repo] = "https://\\g"
+vcsBitbucket[Vcs.TYPE] = "git"
+vcsBitbucket[Vcs.REPO] = "https://\\g"
 
 # IBM DevOps Services (JazzHub)
 vcsIBMDevOps = {}
-vcsIBMDevOps[vcs.pathprefix] = "hub.jazz.net/git"
-vcsIBMDevOps[vcs.regexp] = re.compile(
+vcsIBMDevOps[Vcs.PATHPREFIX] = "hub.jazz.net/git"
+vcsIBMDevOps[Vcs.REGEXP] = re.compile(
 
r'

[OE-core] [PATCH 3/4] recipetool: go: ignore 'go: *' log messages

2024-02-26 Thread Lukas Funke 
From: Lukas Funke 

Go will download verious modules during vendoring and tell us about
it, i.e. "go: downloading foo/bar vX.Y.Z". These lines are mixed with the
relevant modules.txt lines. Thus, ignore log messages starting with
"go: .*".

Signed-off-by: Lukas Funke 
---
 scripts/lib/recipetool/create_go.py | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/recipetool/create_go.py 
b/scripts/lib/recipetool/create_go.py
index abe1f6e0da..cae7175a33 100644
--- a/scripts/lib/recipetool/create_go.py
+++ b/scripts/lib/recipetool/create_go.py
@@ -615,10 +615,15 @@ class GoRecipeHandler(RecipeHandler):
 _, stderr = self.__go_run_cmd(
 "go mod vendor -v -o %s" % (tmp_vendor_dir), srctree, d)
 
+# ignore miscellaneous go log messages
+vedoring_log = [x for x in stderr.splitlines() \
+if not x.startswith("go: ")]
+
 modules_txt_basename = "modules.txt"
 modules_txt_filename = os.path.join(localfilesdir, 
modules_txt_basename)
 with open(modules_txt_filename, "w") as f:
-f.write(stderr)
+f.write('\n'.join(vedoring_log))
+f.write('\n') # don't forget the trailing newline
 
 extravalues['extrafiles'][modules_txt_basename] = modules_txt_filename
 
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#196210): 
https://lists.openembedded.org/g/openembedded-core/message/196210
Mute This Topic: https://lists.openembedded.org/mt/104582951/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH 2/4] recipetool: go: do not restore go.mod/sum file

2024-02-26 Thread Lukas Funke 
From: Lukas Funke 

Do not restore 'go.mod' and 'go.sum' file after the projects go version
is updated to >= 1.17.

When we leave the go.mod file as-is then vendor manifest is created
using the < 1.17 version. This results in an inconsistency between the
patched (newer) 'go.mod' file and the 'older' modules.txt.

Signed-off-by: Lukas Funke 
---
 scripts/lib/recipetool/create_go.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/scripts/lib/recipetool/create_go.py 
b/scripts/lib/recipetool/create_go.py
index c560831442..abe1f6e0da 100644
--- a/scripts/lib/recipetool/create_go.py
+++ b/scripts/lib/recipetool/create_go.py
@@ -598,8 +598,6 @@ class GoRecipeHandler(RecipeHandler):
 
 # Create patch in order to upgrade go version
 self.__go_run_cmd("git diff go.mod > %s" % (patchfilename), srctree, d)
-# Restore original state
-self.__go_run_cmd("git checkout HEAD go.mod go.sum", srctree, d)
 
 go_mod = json.loads(stdout)
 tmpfile = os.path.join(localfilesdir, patchfilename)
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#196209): 
https://lists.openembedded.org/g/openembedded-core/message/196209
Mute This Topic: https://lists.openembedded.org/mt/104582950/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH 0/4] go: improve vendoring

2024-02-26 Thread Lukas Funke 
From: Lukas Funke 

This series is intended to improve 'vendoring' for golang projects. 'Vendoring'
is the golang mechanism to build go-projects in an offline manner. It enables
the fetcher to gather all dependencies during fetching-phase and unpack them
during build into the workdirs 'vendor' folder.

During testing on our go-projects some issues were discovered, that prevent 
clean vendoring:

 - The '//go:embed' directive enables go-files to include additional source
   files. These files/folder are not considered in the manifest created by go.
   Thus, this files are not copied to the vendor folder by yocto, resulting
   in a broken build.
 - Some files contain additional '//go:build' directives, that instruct go to
   ignore this files during vendoring. This case is now covered as well.

Additional changes:

 - fix an issue where some go log messages are included in the vendor manifest
 - run pylint analysis on the 'go' recipetool part and incorporate suggestions

Lukas Funke (4):
  classes: go-vendor: improve handling of go vendoring
  recipetool: go: Do not restore go.mod/sum file
  recipetool: go: ignore 'go: *' log messages
  recipetool: go: apply pylint recommendations

 meta/classes/go-vendor.bbclass  | 324 ++---
 scripts/lib/recipetool/create_go.py | 362 
 2 files changed, 449 insertions(+), 237 deletions(-)

-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#196207): 
https://lists.openembedded.org/g/openembedded-core/message/196207
Mute This Topic: https://lists.openembedded.org/mt/104582948/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH 1/4] classes: go-vendor: improve handling of go vendoring

2024-02-26 Thread Lukas Funke 
From: Lukas Funke 

This commit deals with some specialties around go:

 - Enable the use of //go:embed directives:
 Using this directive it is possible to include additional sournce
 files into your go source-code. Since these are no listed in the
 vendor manifest, we have to check for the pattern in the actual
 source code

 - Do not vendor //go:build ignore files
 Do not vendor source files which should be ignored during the build

In addition to the changes above the behaviour of the vendoring folder
is changed: the vendor folder is not linked any longer into the source
directory, the files are rather copied directly into the projects
vendor folder. Because the link was removed before packaging, yocto was
unable to copie the listed licenses

Signed-off-by: Lukas Funke 
---
 meta/classes/go-vendor.bbclass | 324 -
 1 file changed, 240 insertions(+), 84 deletions(-)

diff --git a/meta/classes/go-vendor.bbclass b/meta/classes/go-vendor.bbclass
index 1bbb99ac79..d4a7d7c224 100644
--- a/meta/classes/go-vendor.bbclass
+++ b/meta/classes/go-vendor.bbclass
@@ -40,18 +40,159 @@ def go_src_uri(repo, version, path=None, subdir=None, \
 
 return src_uri
 
-python do_vendor_unlink() {
-go_import = d.getVar('GO_IMPORT')
-source_dir = d.getVar('S')
-linkname = os.path.join(source_dir, *['src', go_import, 'vendor'])
 
-os.unlink(linkname)
-}
+def read_vendor_manifest(fname):
+vendoredModulePaths = dict()
+replacedPaths = dict()
+
+with open(fname) as _file:
+content = _file.readlines()
+modPath = ""
+version = ""
+for line in content:
+# Modules starts with a single hash tag
+# followed by the modules path
+if line.startswith("# ", 0, 2):
+t = line[2:].strip()
+if "=>" in t:
+lhs, rhs = t.split("=>")
+
+# This module has been replaced, use a local path
+# we parse the line that has a pattern "# module-name 
[module-version] => local-path
+lhs = lhs.strip().split()
+rhs = rhs.strip().split()
+
+# Check for version replacement
+# "# module versionABC => module versionXZY"
+if len(lhs) == 2 and len(rhs) == 2:
+lhsModPath = lhs[0]
+rhsModPath = rhs[0]
+if lhsModPath == rhsModPath:
+modPath = lhsModPath
+version = rhs[1]
+
+elif (len(lhs) == 1 or len(lhs) == 2) \
+and len(rhs) == 1:
+replacedPaths[modPath] = rhs[0]
+
+else:
+modPath, version = t.split()
+if modPath not in vendoredModulePaths:
+vendoredModulePaths[modPath] = {'version': version,
+'pkgs': set()}
+
+if not line.startswith("#"):
+pkg = line.strip()
+bb.debug(2, "manifest: module %s: add pkg %s" % (modPath, pkg))
+vendoredModulePaths[modPath]['pkgs'].add(pkg)
+
+return vendoredModulePaths, replacedPaths
+
+
+def should_build(fname):
+
+with open(fname) as _file:
+goBuildDirectivePos = -1
+endpos = -1
+content = _file.readlines()
+
+for i, line in enumerate(content):
+if len(line.strip()) == 0 \
+or line.startswith("//"):
+continue
+endpos = i
+break
+
+for i, line in enumerate(content):
+if i == endpos:
+break
+if line.startswith("//go:build"):
+goBuildDirectivePos = i
+continue
+if goBuildDirectivePos >= 0 and len(line.strip()) == 0:
+directive = content[goBuildDirectivePos].strip().split()
+if len(directive) > 1 and directive[1] == "ignore":
+return False
+return True
+
+
+def match_potential_source_file(fname):
+
+basename = os.path.basename(fname)
+
+if basename.endswith("_test.go"):
+return False
+
+# We assume that go version >= 1.17
+# (See https://golang.org/issue/42970.)
+if basename == "go.mod" or basename == "go.sum":
+return False
+
+if basename.endswith(".go"):
+if not should_build(fname):
+return False
+
+return True
+
+
+def resolve_embed(fname, dst):
+import glob
+import re
+import shutil
+
+go_embed_re = re.compile("//go:embed (.*)")
+
+b

Re: [OE-core] Patchtest results for [PATCH v2 8/9] oeqa/selftest/recipetool: Move helper function to the class scope

2024-01-22 Thread Lukas Funke 
Is this the reason why I cannot find the series on patchwork? This makes 
it kinda tricky to review/test IMHO.


Please provide a commit message, so patchtest is happy.

On 16.01.2024 09:45, Patchtest via lists.openembedded.org wrote:

Thank you for your submission. Patchtest identified one
or more issues with the patch. Please see the log below for
more information:

---
Testing patch 
/home/patchtest/share/mboxes/v2-8-9-oeqa-selftest-recipetool-Move-helper-function-to-the-class-scope.patch

FAIL: test commit message presence: Please include a commit message on your 
patch explaining the change (test_mbox.TestMbox.test_commit_message_presence)

PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint)
PASS: test Signed-off-by presence 
(test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test max line length (test_metadata.TestMetadata.test_max_line_length)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test pylint (test_python_pylint.PyLint.test_pylint)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)
PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list)

SKIP: pretest src uri left files: Patch cannot be merged 
(test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE check ignore: No modified recipes, skipping test 
(test_metadata.TestMetadata.test_cve_check_ignore)
SKIP: test CVE tag format: No new CVE patches introduced 
(test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced 
(test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced 
(test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found 
(test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, 
skipping test 
(test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test 
(test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test 
(test_metadata.TestMetadata.test_license_presence)
SKIP: test series merge on head: Merge test is disabled for now 
(test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: Patch cannot be merged 
(test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test 
(test_metadata.TestMetadata.test_summary_presence)

---

Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this
email with an explanation of why the patch should be accepted. If you
believe these results are due to an error in patchtest, please submit a
bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category
under 'Yocto Project Subprojects'). For more information on specific
failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank
you!








-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#194177): 
https://lists.openembedded.org/g/openembedded-core/message/194177
Mute This Topic: https://lists.openembedded.org/mt/103758513/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH 1/3] recipetool: Don't fail on local go modules

2024-01-10 Thread Lukas Funke 

On 10.01.2024 14:59, Richard Purdie wrote:

On Wed, 2024-01-10 at 12:53 +0100, Vyacheslav Yurkov wrote:

Local modules are usually referenced with a 'replace' directive in
go.mod file. If that's the case, remove them from populating SRC_URI.

Signed-off-by: Vyacheslav Yurkov 
---
  scripts/lib/recipetool/create_go.py | 32 +
  1 file changed, 19 insertions(+), 13 deletions(-)


Do these changes mean we need to improve the test coverage in oe-
selftest for recipetool?


Test converage needs improvement, yes. I haven't considered the case for 
local modules since this seems to be rare(?). Are there public projects 
which use this mechanism?


Best regards
Lukas



Cheers,

Richard



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#193515): 
https://lists.openembedded.org/g/openembedded-core/message/193515
Mute This Topic: https://lists.openembedded.org/mt/103639058/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH v2 0/2] patch: reduce changes during patch refresh

2023-12-04 Thread Lukas Funke 

Hi Alex,

On 04.12.2023 11:54, Alexander Kanavin wrote:

On Mon, 4 Dec 2023 at 11:48, Lukas Funke
 wrote:

Unfortunately, the patch indeed causes the tests to fail. The
problamatic lines are in the 'lib\devtool\standard.py':

if len(firstlineitems) > 1 and len(firstlineitems[1]) == 40:
 if not firstlineitems[1] in changed_revs:
 continue

When the commit-hash becomes all zero, there is no way for devtools to
detect the change. I'm a little puzzled how to handle this situation.
However, I still think the all zero hash is usefull since it will reduce
the noise when developers review their changed in git(lab|hub). Maybe it
would be for the best to remove the check?


It would help if you provide the context for that code snippet, so we
can see why is it there, and what is it trying to do, without having
to go and separately study the code ourselves. Patch review is hard,
please do not make it harder.


You're probably right. A little context is always helpful.

From the top: the proposed change is setting the commit-hash in 
exported patches to all-zero.


When devtool is updating the recipes using 'devtool update-recipe 
' it figures out (in '_get_patchset_revs()') which commits 
were changed during development (in the workspace).
Now devtool known which commits actually contain changes. Subsequently 
devtool is exporting the patches with all-zero commit hash (in 
'_export_patches()', which contains our all-zero commit hash change).
'_export_patches()' also compares the now exported commit-hash to the 
expected commit-hash which is an argument to the very same function. 
Since it they didn't match (see the code snippet), devtool thinks there 
is no change. Thus, the patch is ignored.


If we want to export patches with all-zero commits we have to either 
remove this check or have to export the original commit id in the patch. 
The latter would again introduce noise.


Best regards
Lukas



Alex



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191735): 
https://lists.openembedded.org/g/openembedded-core/message/191735
Mute This Topic: https://lists.openembedded.org/mt/102748808/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH v2 0/2] patch: reduce changes during patch refresh

2023-12-04 Thread Lukas Funke 

Hi Richard,

On 27.11.2023 14:45, Richard Purdie wrote:

On Wed, 2023-11-22 at 15:08 +0100, Lukas Funke wrote:

From: Lukas Funke 

The patch series aims to reduce the noise in patches created by devtools. Some
diffs are just introduced due to an update in the hash or in the diffstats.
These changes are not important to a reviewer.

Stefan Herbrechtsmeier (2):
   patch: extract patches without diffstats
   patch: extract patches with all-zero hash

  meta/lib/oe/patch.py | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)


I'm fairly sure but haven't conclusively proven this causes these
failures:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/6095/steps/14/logs/stdio

2023-11-23 23:31:16,216 - oe-selftest - INFO - RESULTS - 
devtool.DevtoolUpdateTests.test_devtool_update_recipe_local_patch_gz: FAILED 
(65.60s)
2023-11-23 23:31:16,216 - oe-selftest - INFO - RESULTS - 
devtool.DevtoolUpdateTests.test_devtool_update_recipe_long_filename: FAILED 
(53.35s)
2023-11-23 23:31:16,216 - oe-selftest - INFO - RESULTS - 
devtool.DevtoolUpdateTests.test_devtool_update_recipe_with_gitignore: FAILED 
(44.11s)


Unfortunately, the patch indeed causes the tests to fail. The 
problamatic lines are in the 'lib\devtool\standard.py':


if len(firstlineitems) > 1 and len(firstlineitems[1]) == 40:
if not firstlineitems[1] in changed_revs:
continue

When the commit-hash becomes all zero, there is no way for devtools to 
detect the change. I'm a little puzzled how to handle this situation. 
However, I still think the all zero hash is usefull since it will reduce 
the noise when developers review their changed in git(lab|hub). Maybe it 
would be for the best to remove the check?


Best regards
Lukas



Cheers,

Richard



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191732): 
https://lists.openembedded.org/g/openembedded-core/message/191732
Mute This Topic: https://lists.openembedded.org/mt/102748808/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v3 0/2] wic: extend empty plugin with options to write zeros to partiton

2023-11-28 Thread Lukas Funke 
From: Lukas Funke 

Adds features to explicitly write zeros to the start of the
partition. This is useful to overwrite old content like
filesystem signatures which may be re-recognized otherwise.

The new features can be enabled with
'--soucreparams="[fill|size=[S|s|K|k|M|G]][,][bs=[S|s|K|k|M|G]]"'
Conflicting or missing options throw errors.

The features are:
- fill
  Fill the entire partition with zeros. Requires '--fixed-size' option
  to be set.
- size=[S|s|K|k|M|G]
  Set the first N bytes of the partition to zero. Default unit is 'K'.
- bs=[S|s|K|k|M|G]
  Write at most N bytes at a time during source file creation.
  Defaults to '1M'. Default unit is 'K'.

Changes in v3:
 - Add testcase 
 - Fix interpretation of --fixed-size parameter

Changed in v2:
 - Added SoB

---

Lukas Funke (1):
  selftest: wic: add test for zerorize option of empty plugin

Malte Schmidt (1):
  wic: extend empty plugin with options to write zeros to partiton

 meta/lib/oeqa/selftest/cases/wic.py | 36 
 scripts/lib/wic/plugins/source/empty.py | 57 -
 2 files changed, 92 insertions(+), 1 deletion(-)

-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191372): 
https://lists.openembedded.org/g/openembedded-core/message/191372
Mute This Topic: https://lists.openembedded.org/mt/102850424/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v3 1/2] wic: extend empty plugin with options to write zeros to partiton

2023-11-28 Thread Lukas Funke 
From: Malte Schmidt 

Adds features to explicitly write zeros to the start of the
partition. This is useful to overwrite old content like
filesystem signatures which may be re-recognized otherwise.

The new features can be enabled with
'--soucreparams="[fill|size=[S|s|K|k|M|G]][,][bs=[S|s|K|k|M|G]]"'
Conflicting or missing options throw errors.

The features are:
- fill
  Fill the entire partition with zeros. Requires '--fixed-size' option
  to be set.
- size=[S|s|K|k|M|G]
  Set the first N bytes of the partition to zero. Default unit is 'K'.
- bs=[S|s|K|k|M|G]
  Write at most N bytes at a time during source file creation.
  Defaults to '1M'. Default unit is 'K'.

Signed-off-by: Malte Schmidt 
Signed-off-by: Lukas Funke 
---
 scripts/lib/wic/plugins/source/empty.py | 57 -
 1 file changed, 56 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/wic/plugins/source/empty.py 
b/scripts/lib/wic/plugins/source/empty.py
index 9c492ca206..0a9f5fa27c 100644
--- a/scripts/lib/wic/plugins/source/empty.py
+++ b/scripts/lib/wic/plugins/source/empty.py
@@ -9,9 +9,19 @@
 # To use it you must pass "empty" as argument for the "--source" parameter in
 # the wks file. For example:
 # part foo --source empty --ondisk sda --size="1024" --align 1024
+#
+# The plugin supports writing zeros to the start of the
+# partition. This is useful to overwrite old content like
+# filesystem signatures which may be re-recognized otherwise.
+# This feature can be enabled with
+# '--soucreparams="[fill|size=[S|s|K|k|M|G]][,][bs=[S|s|K|k|M|G]]"'
+# Conflicting or missing options throw errors.
 
 import logging
+import os
 
+from wic import WicError
+from wic.ksparser import sizetype
 from wic.pluginbase import SourcePlugin
 
 logger = logging.getLogger('wic')
@@ -19,6 +29,16 @@ logger = logging.getLogger('wic')
 class EmptyPartitionPlugin(SourcePlugin):
 """
 Populate unformatted empty partition.
+
+The following sourceparams are supported:
+- fill
+  Fill the entire partition with zeros. Requires '--fixed-size' option
+  to be set.
+- size=[S|s|K|k|M|G]
+  Set the first N bytes of the partition to zero. Default unit is 'K'.
+- bs=[S|s|K|k|M|G]
+  Write at most N bytes at a time during source file creation.
+  Defaults to '1M'. Default unit is 'K'.
 """
 
 name = 'empty'
@@ -31,4 +51,39 @@ class EmptyPartitionPlugin(SourcePlugin):
 Called to do the actual content population for a partition i.e. it
 'prepares' the partition to be incorporated into the image.
 """
-return
+get_byte_count = sizetype('K', True)
+size = 0
+
+if 'fill' in source_params and 'size' in source_params:
+raise WicError("Conflicting source parameters 'fill' and 'size' 
specified, exiting.")
+
+# Set the size of the zeros to be written to the partition
+if 'fill' in source_params:
+if part.fixed_size == 0:
+raise WicError("Source parameter 'fill' only works with the 
'--fixed-size' option, exiting.")
+size = get_byte_count(part.fixed_size)
+elif 'size' in source_params:
+size = get_byte_count(source_params['size'])
+
+if size == 0:
+# Nothing to do, create empty partition
+return
+
+if 'bs' in source_params:
+bs = get_byte_count(source_params['bs'])
+else:
+bs = get_byte_count('1M')
+
+# Create a binary file of the requested size filled with zeros
+source_file = os.path.join(cr_workdir, 'empty-plugin-zeros%s.bin' % 
part.lineno)
+if not os.path.exists(os.path.dirname(source_file)):
+os.makedirs(os.path.dirname(source_file))
+
+quotient, remainder = divmod(size, bs)
+with open(source_file, 'wb') as file:
+for _ in range(quotient):
+file.write(bytearray(bs))
+file.write(bytearray(remainder))
+
+part.size = (size + 1024 - 1) // 1024  # size in KB rounded up
+part.source_file = source_file
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191371): 
https://lists.openembedded.org/g/openembedded-core/message/191371
Mute This Topic: https://lists.openembedded.org/mt/102850423/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v3 2/2] selftest: wic: add test for zerorize option of empty plugin

2023-11-28 Thread Lukas Funke 
From: Lukas Funke 

Add test for empty plugin which tests whether the plugin creates
partitions with actual data which is 'zero'.

Signed-off-by: Lukas Funke 
---
 meta/lib/oeqa/selftest/cases/wic.py | 36 +
 1 file changed, 36 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/wic.py 
b/meta/lib/oeqa/selftest/cases/wic.py
index b4866bcb32..9672f3fae4 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -1391,6 +1391,42 @@ class Wic2(WicTestCase):
 result = runCmd("%s/usr/sbin/sfdisk --part-label %s 3" % (sysroot, 
image_path))
 self.assertEqual('ext-space', result.output)
 
+def test_empty_zeroize_plugin(self):
+img = 'core-image-minimal'
+expected_size = [ 1024*1024,# 1M
+  512*1024, # 512K
+  2*1024*1024]  # 2M
+# Check combination of sourceparams
+with NamedTemporaryFile("w", suffix=".wks") as wks:
+wks.writelines(
+['part empty --source empty --sourceparams="fill" --ondisk sda 
--fixed-size 1M\n',
+ 'part empty --source empty --sourceparams="size=512K" 
--ondisk sda --size 1M --align 1024\n',
+ 'part empty --source empty 
--sourceparams="size=2048k,bs=512K" --ondisk sda --size 4M --align 1024\n'
+ ])
+wks.flush()
+cmd = "wic create %s -e %s -o %s" % (wks.name, img, self.resultdir)
+runCmd(cmd)
+wksname = os.path.splitext(os.path.basename(wks.name))[0]
+wicout = glob(os.path.join(self.resultdir, "%s-*direct" % wksname))
+# Skip the complete image and just look at the single partitions
+for idx, value in enumerate(wicout[1:]):
+self.logger.info(wicout[idx])
+# Check if partitions are actually zeroized
+with open(wicout[idx], mode="rb") as fd:
+ba = bytearray(fd.read())
+for b in ba:
+self.assertEqual(b, 0)
+self.assertEqual(expected_size[idx], 
os.path.getsize(wicout[idx]))
+
+# Check inconsistancy check between "fill" and "--size" parameter
+with NamedTemporaryFile("w", suffix=".wks") as wks:
+wks.writelines(['part empty --source empty --sourceparams="fill" 
--ondisk sda --size 1M\n'])
+wks.flush()
+cmd = "wic create %s -e %s -o %s" % (wks.name, img, self.resultdir)
+result = runCmd(cmd, ignore_status=True)
+self.assertIn("Source parameter 'fill' only works with the 
'--fixed-size' option, exiting.", result.output)
+self.assertNotEqual(0, result.status)
+
 class ModifyTests(WicTestCase):
 def test_wic_ls(self):
 """Test listing image content using 'wic ls'"""
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191370): 
https://lists.openembedded.org/g/openembedded-core/message/191370
Mute This Topic: https://lists.openembedded.org/mt/102850422/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v2 1/2] patch: extract patches without diffstats

2023-11-22 Thread Lukas Funke 
From: Stefan Herbrechtsmeier 

Extract patches without diffstats to reduce changes during patch
refresh.

Signed-off-by: Stefan Herbrechtsmeier 
Signed-off-by: Lukas Funke 
---
 meta/lib/oe/patch.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/patch.py b/meta/lib/oe/patch.py
index ff9afc9df9..495a302121 100644
--- a/meta/lib/oe/patch.py
+++ b/meta/lib/oe/patch.py
@@ -466,7 +466,8 @@ class GitApplyTree(PatchTree):
 import shutil
 tempdir = tempfile.mkdtemp(prefix='oepatch')
 try:
-shellcmd = ["git", "format-patch", "--no-signature", 
"--no-numbered", startcommit, "-o", tempdir]
+shellcmd = ["git", "format-patch", "--no-signature", 
"--no-numbered",
+"--no-stat", startcommit, "-o", tempdir]
 if paths:
 shellcmd.append('--')
 shellcmd.extend(paths)
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191106): 
https://lists.openembedded.org/g/openembedded-core/message/191106
Mute This Topic: https://lists.openembedded.org/mt/102748810/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v2 2/2] patch: extract patches with all-zero hash

2023-11-22 Thread Lukas Funke 
From: Stefan Herbrechtsmeier 

Extract patches with all-zero hash in each patch header instead of the
hash of the commit to reduce changes during patch refresh.

Signed-off-by: Stefan Herbrechtsmeier 
Signed-off-by: Lukas Funke 
---
 meta/lib/oe/patch.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oe/patch.py b/meta/lib/oe/patch.py
index 495a302121..6d81bd4f67 100644
--- a/meta/lib/oe/patch.py
+++ b/meta/lib/oe/patch.py
@@ -467,7 +467,7 @@ class GitApplyTree(PatchTree):
 tempdir = tempfile.mkdtemp(prefix='oepatch')
 try:
 shellcmd = ["git", "format-patch", "--no-signature", 
"--no-numbered",
-"--no-stat", startcommit, "-o", tempdir]
+"--no-stat", "--zero-commit", startcommit, "-o", 
tempdir]
 if paths:
 shellcmd.append('--')
 shellcmd.extend(paths)
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191107): 
https://lists.openembedded.org/g/openembedded-core/message/191107
Mute This Topic: https://lists.openembedded.org/mt/102748811/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v2 0/2] patch: reduce changes during patch refresh

2023-11-22 Thread Lukas Funke 
From: Lukas Funke 

The patch series aims to reduce the noise in patches created by devtools. Some
diffs are just introduced due to an update in the hash or in the diffstats.
These changes are not important to a reviewer.

Stefan Herbrechtsmeier (2):
  patch: extract patches without diffstats
  patch: extract patches with all-zero hash

 meta/lib/oe/patch.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191105): 
https://lists.openembedded.org/g/openembedded-core/message/191105
Mute This Topic: https://lists.openembedded.org/mt/102748808/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [meta-oe][PATCH v2 0/1] wic: extend empty plugin with options to write zeros to partiton

2023-11-22 Thread Lukas Funke 

Hi,

On 22.11.2023 11:47, Lukas Funke via lists.openembedded.org wrote:

From: Lukas Funke 

Adds features to explicitly write zeros to the start of the
partition. This is useful to overwrite old content like
filesystem signatures which may be re-recognized otherwise.

The new features can be enabled with
'--soucreparams="[fill|size=[S|s|K|k|M|G]][,][bs=[S|s|K|k|M|G]]"'
Conflicting or missing options throw errors.

The features are:
- fill
   Fill the entire partition with zeros. Requires '--fixed-size' option
   to be set.
- size=[S|s|K|k|M|G]
   Set the first N bytes of the partition to zero. Default unit is 'K'.
- bs=[S|s|K|k|M|G]
   Write at most N bytes at a time during source file creation.
   Defaults to '1M'. Default unit is 'K'.

Changed in v2:
  - Added SoB
---

Malte Schmidt (1):
   wic: extend empty plugin with options to write zeros to partiton

  scripts/lib/wic/plugins/source/empty.py | 57 -
  1 file changed, 56 insertions(+), 1 deletion(-)



I've seen that there is a misstake and the patch was sended with the 
'meta-oe' prefix in the subject. This was not intendet and it belongs to 
the core layer only. I hope that you don't mind, otherwise I'll send a 
v3 version.










-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191100): 
https://lists.openembedded.org/g/openembedded-core/message/191100
Mute This Topic: https://lists.openembedded.org/mt/102746528/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v4] wic: rawcopy: add support for zstd decompression

2023-11-22 Thread Lukas Funke 
From: Malte Schmidt 

Add support for zstd decompression in rawcopy plugin. zstd claims
to reach higher, uniform decompression rates.

Signed-off-by: Malte Schmidt 
Signed-off-by: Lukas Funke 
---
 scripts/lib/wic/plugins/source/rawcopy.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/wic/plugins/source/rawcopy.py 
b/scripts/lib/wic/plugins/source/rawcopy.py
index ccf332554e..21903c2f23 100644
--- a/scripts/lib/wic/plugins/source/rawcopy.py
+++ b/scripts/lib/wic/plugins/source/rawcopy.py
@@ -58,7 +58,8 @@ class RawCopyPlugin(SourcePlugin):
 decompressor = {
 ".bz2": "bzip2",
 ".gz": "gzip",
-".xz": "xz"
+".xz": "xz",
+".zst": "zstd -f",
 }.get(extension)
 if not decompressor:
 raise WicError("Not supported compressor filename extension: %s" % 
extension)
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191073): 
https://lists.openembedded.org/g/openembedded-core/message/191073
Mute This Topic: https://lists.openembedded.org/mt/102747437/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [meta-oe][PATCH v2 0/1] wic: extend empty plugin with options to write zeros to partiton

2023-11-22 Thread Lukas Funke 
From: Lukas Funke 

Adds features to explicitly write zeros to the start of the
partition. This is useful to overwrite old content like
filesystem signatures which may be re-recognized otherwise.

The new features can be enabled with
'--soucreparams="[fill|size=[S|s|K|k|M|G]][,][bs=[S|s|K|k|M|G]]"'
Conflicting or missing options throw errors.

The features are:
- fill
  Fill the entire partition with zeros. Requires '--fixed-size' option
  to be set.
- size=[S|s|K|k|M|G]
  Set the first N bytes of the partition to zero. Default unit is 'K'.
- bs=[S|s|K|k|M|G]
  Write at most N bytes at a time during source file creation.
  Defaults to '1M'. Default unit is 'K'.

Changed in v2:
 - Added SoB
---

Malte Schmidt (1):
  wic: extend empty plugin with options to write zeros to partiton

 scripts/lib/wic/plugins/source/empty.py | 57 -
 1 file changed, 56 insertions(+), 1 deletion(-)

-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191057): 
https://lists.openembedded.org/g/openembedded-core/message/191057
Mute This Topic: https://lists.openembedded.org/mt/102746528/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [meta-oe][PATCH v2 1/1] wic: extend empty plugin with options to write zeros to partiton

2023-11-22 Thread Lukas Funke 
From: Malte Schmidt 

Adds features to explicitly write zeros to the start of the
partition. This is useful to overwrite old content like
filesystem signatures which may be re-recognized otherwise.

The new features can be enabled with
'--soucreparams="[fill|size=[S|s|K|k|M|G]][,][bs=[S|s|K|k|M|G]]"'
Conflicting or missing options throw errors.

The features are:
- fill
  Fill the entire partition with zeros. Requires '--fixed-size' option
  to be set.
- size=[S|s|K|k|M|G]
  Set the first N bytes of the partition to zero. Default unit is 'K'.
- bs=[S|s|K|k|M|G]
  Write at most N bytes at a time during source file creation.
  Defaults to '1M'. Default unit is 'K'.

Signed-off-by: Malte Schmidt 
Signed-off-by: Lukas Funke 
---
 scripts/lib/wic/plugins/source/empty.py | 57 -
 1 file changed, 56 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/wic/plugins/source/empty.py 
b/scripts/lib/wic/plugins/source/empty.py
index 9c492ca206..775631b588 100644
--- a/scripts/lib/wic/plugins/source/empty.py
+++ b/scripts/lib/wic/plugins/source/empty.py
@@ -9,9 +9,19 @@
 # To use it you must pass "empty" as argument for the "--source" parameter in
 # the wks file. For example:
 # part foo --source empty --ondisk sda --size="1024" --align 1024
+#
+# The plugin supports writing zeros to the start of the
+# partition. This is useful to overwrite old content like
+# filesystem signatures which may be re-recognized otherwise.
+# This feature can be enabled with
+# '--soucreparams="[fill|size=[S|s|K|k|M|G]][,][bs=[S|s|K|k|M|G]]"'
+# Conflicting or missing options throw errors.
 
 import logging
+import os
 
+from wic import WicError
+from wic.ksparser import sizetype
 from wic.pluginbase import SourcePlugin
 
 logger = logging.getLogger('wic')
@@ -19,6 +29,16 @@ logger = logging.getLogger('wic')
 class EmptyPartitionPlugin(SourcePlugin):
 """
 Populate unformatted empty partition.
+
+The following sourceparams are supported:
+- fill
+  Fill the entire partition with zeros. Requires '--fixed-size' option
+  to be set.
+- size=[S|s|K|k|M|G]
+  Set the first N bytes of the partition to zero. Default unit is 'K'.
+- bs=[S|s|K|k|M|G]
+  Write at most N bytes at a time during source file creation.
+  Defaults to '1M'. Default unit is 'K'.
 """
 
 name = 'empty'
@@ -31,4 +51,39 @@ class EmptyPartitionPlugin(SourcePlugin):
 Called to do the actual content population for a partition i.e. it
 'prepares' the partition to be incorporated into the image.
 """
-return
+get_byte_count = sizetype('K', True)
+size = 0
+
+if 'fill' in source_params and 'size' in source_params:
+raise WicError("Conflicting source parameters 'fill' and 'size' 
specified, exiting.")
+
+# Set the size of the zeros to be written to the partition
+if 'fill' in source_params:
+if part.fixed_size == 0:
+raise WicError("Source parameter 'fill' only works with the 
'--fixed-size' option, exiting.")
+size = part.fixed_size
+elif 'size' in source_params:
+size = get_byte_count(source_params['size'])
+
+if size == 0:
+# Nothing to do, create empty partition
+return
+
+if 'bs' in source_params:
+bs = get_byte_count(source_params['bs'])
+else:
+bs = get_byte_count('1M')
+
+# Create a binary file of the requested size filled with zeros
+source_file = os.path.join(cr_workdir, 'empty-plugin-zeros%s.bin' % 
part.lineno)
+if not os.path.exists(os.path.dirname(source_file)):
+os.makedirs(os.path.dirname(source_file))
+
+quotient, remainder = divmod(size, bs)
+with open(source_file, 'wb') as file:
+for _ in range(quotient):
+file.write(bytearray(bs))
+file.write(bytearray(remainder))
+
+part.size = (size + 1024 - 1) // 1024  # size in KB rounded up
+part.source_file = source_file
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191058): 
https://lists.openembedded.org/g/openembedded-core/message/191058
Mute This Topic: https://lists.openembedded.org/mt/102746529/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [meta-oe][PATCH] classes: go-mod: do not pack go mod cache

2023-11-16 Thread Lukas Funke 
From: Stefan Herbrechtsmeier 

Clean go module cache from builddir to prevent it of beeing packed.

Signed-off-by: Stefan Herbrechtsmeier 
Signed-off-by: Lukas Funke 
---
 meta/classes-recipe/go-mod.bbclass | 4 
 1 file changed, 4 insertions(+)

diff --git a/meta/classes-recipe/go-mod.bbclass 
b/meta/classes-recipe/go-mod.bbclass
index 61571596bc..ca3a690d05 100644
--- a/meta/classes-recipe/go-mod.bbclass
+++ b/meta/classes-recipe/go-mod.bbclass
@@ -24,3 +24,7 @@ inherit go
 
 GO_WORKDIR ?= "${GO_IMPORT}"
 do_compile[dirs] += "${B}/src/${GO_WORKDIR}"
+
+export GOMODCACHE = "${B}/.mod"
+
+do_compile[cleandirs] += "${B}/.mod"
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190795): 
https://lists.openembedded.org/g/openembedded-core/message/190795
Mute This Topic: https://lists.openembedded.org/mt/102626271/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [meta-oe][PATCH v3] systemd: use nonarch libdir for tmpfiles.d

2023-11-16 Thread Lukas Funke 
From: Malte Schmidt 

The documentation of systemd states that /etc/tmpfiles.d should be
reserved for the local administrator and packages should put their files
in /usr/lib/tmpfiles.d [1].

[1] https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html

Signed-off-by: Malte Schmidt 
Signed-off-by: Stefan Herbrechtsmeier 
Signed-off-by: Lukas Funke 
---
 meta/recipes-core/systemd/systemd_254.4.bb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/systemd/systemd_254.4.bb 
b/meta/recipes-core/systemd/systemd_254.4.bb
index cc0e29fdd2..6d71fbaba2 100644
--- a/meta/recipes-core/systemd/systemd_254.4.bb
+++ b/meta/recipes-core/systemd/systemd_254.4.bb
@@ -282,12 +282,12 @@ do_install() {
[ ! -e ${D}/${base_sbindir}/udevd ] && ln -s 
${rootlibexecdir}/systemd/systemd-udevd ${D}/${base_sbindir}/udevd
 
install -d ${D}${sysconfdir}/udev/rules.d/
-   install -d ${D}${sysconfdir}/tmpfiles.d
+   install -d ${D}${nonarch_libdir}/tmpfiles.d
for rule in $(find ${WORKDIR} -maxdepth 1 -type f -name "*.rules"); do
install -m 0644 $rule ${D}${sysconfdir}/udev/rules.d/
done
 
-   install -m 0644 ${WORKDIR}/00-create-volatile.conf 
${D}${sysconfdir}/tmpfiles.d/
+   install -m 0644 ${WORKDIR}/00-create-volatile.conf 
${D}${nonarch_libdir}/tmpfiles.d/
 
if 
${@bb.utils.contains('DISTRO_FEATURES','sysvinit','true','false',d)}; then
install -d ${D}${sysconfdir}/init.d
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190793): 
https://lists.openembedded.org/g/openembedded-core/message/190793
Mute This Topic: https://lists.openembedded.org/mt/102625692/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [meta-oe][PATCH v2 4/4] glibc: use nonarch libdir for tmpfiles.d

2023-11-16 Thread Lukas Funke 
From: Stefan Herbrechtsmeier 

The documentation of systemd states that /etc/tmpfiles.d should be
reserved for the local administrator and packages should put their files
in /usr/lib/tmpfiles.d [1].

[1] https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html

Signed-off-by: Stefan Herbrechtsmeier 
Signed-off-by: Lukas Funke 
---
 meta/recipes-core/glibc/glibc-package.inc | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/meta/recipes-core/glibc/glibc-package.inc 
b/meta/recipes-core/glibc/glibc-package.inc
index 1d4e4c5274..1ef987be0a 100644
--- a/meta/recipes-core/glibc/glibc-package.inc
+++ b/meta/recipes-core/glibc/glibc-package.inc
@@ -42,7 +42,7 @@ FILES_SOLIBSDEV = "${libdir}/lib*${SOLIBSDEV}"
 FILES:${PN}-dev += "${libdir}/libpthread.a ${libdir}/libdl.a 
${libdir}/libutil.a ${libdir}/libanl.a ${libdir}/*_nonshared.a 
${base_libdir}/*_nonshared.a ${base_libdir}/*.o ${datadir}/aclocal"
 RDEPENDS:${PN}-dev = "linux-libc-headers-dev"
 FILES:${PN}-staticdev += "${libdir}/*.a ${base_libdir}/*.a"
-FILES:nscd = "${sbindir}/nscd* ${sysconfdir}/init.d/nscd 
${systemd_system_unitdir}/nscd* ${sysconfdir}/tmpfiles.d/nscd.conf \
+FILES:nscd = "${sbindir}/nscd* ${sysconfdir}/init.d/nscd 
${systemd_system_unitdir}/nscd* ${nonarch_libdir}/tmpfiles.d/nscd.conf \
   ${sysconfdir}/nscd.conf ${sysconfdir}/default/volatiles/98_nscd 
${localstatedir}/db/nscd"
 FILES:${PN}-mtrace = "${bindir}/mtrace"
 FILES:tzcode = "${bindir}/tzselect ${sbindir}/zic ${bindir}/zdump"
@@ -132,9 +132,9 @@ def get_libc_fpu_setting(bb, d):
 
 do_install:append:class-target() {
if ${@bb.utils.contains('DISTRO_FEATURES', 'systemd', 'true', 'false', 
d)}; then
-   install -d ${D}${sysconfdir}/tmpfiles.d
+   install -d ${D}${nonarch_libdir}/tmpfiles.d
echo "d /run/nscd 755 root root -" \
-   > ${D}${sysconfdir}/tmpfiles.d/nscd.conf
+   > ${D}${nonarch_libdir}/tmpfiles.d/nscd.conf
fi
 
if ${@bb.utils.contains('DISTRO_FEATURES', 'sysvinit', 'true', 'false', 
d)}; then
@@ -280,7 +280,7 @@ python populate_packages:prepend () {
 pkg_postinst:nscd () {
if [ -z "$D" ]; then
if command -v systemd-tmpfiles >/dev/null; then
-   systemd-tmpfiles --create 
${sysconfdir}/tmpfiles.d/nscd.conf
+   systemd-tmpfiles --create 
${nonarch_libdir}/tmpfiles.d/nscd.conf
elif [ -e ${sysconfdir}/init.d/populate-volatile.sh ]; then
${sysconfdir}/init.d/populate-volatile.sh update
fi
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190789): 
https://lists.openembedded.org/g/openembedded-core/message/190789
Mute This Topic: https://lists.openembedded.org/mt/102625108/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [meta-oe][PATCH v2 2/4] pam: use nonarch libdir for tmpfiles.d

2023-11-16 Thread Lukas Funke 
From: Malte Schmidt 

The documentation of systemd states that /etc/tmpfiles.d should be
reserved for the local administrator and packages should put their files
in /usr/lib/tmpfiles.d [1].

[1] https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html

Signed-off-by: Malte Schmidt 
Signed-off-by: Stefan Herbrechtsmeier 
Signed-off-by: Lukas Funke 
---
 meta/recipes-extended/pam/libpam_1.5.3.bb | 9 ++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-extended/pam/libpam_1.5.3.bb 
b/meta/recipes-extended/pam/libpam_1.5.3.bb
index 1aa307af4d..7af3ea99d1 100644
--- a/meta/recipes-extended/pam/libpam_1.5.3.bb
+++ b/meta/recipes-extended/pam/libpam_1.5.3.bb
@@ -49,7 +49,10 @@ PACKAGECONFIG[audit] = 
"--enable-audit,--disable-audit,audit,"
 PACKAGECONFIG[userdb] = "--enable-db=db,--enable-db=no,db,"
 
 PACKAGES += "${PN}-runtime ${PN}-xtests"
-FILES:${PN} = "${base_libdir}/lib*${SOLIBS}"
+FILES:${PN} = " \
+${base_libdir}/lib*${SOLIBS} \
+${nonarch_libdir}/tmpfiles.d/*.conf \
+"
 FILES:${PN}-dev += "${base_libdir}/security/*.la ${base_libdir}/*.la 
${base_libdir}/lib*${SOLIBSDEV}"
 FILES:${PN}-runtime = "${sysconfdir} ${sbindir} ${systemd_system_unitdir}"
 FILES:${PN}-xtests = "${datadir}/Linux-PAM/xtests"
@@ -130,9 +133,9 @@ do_install() {
 if 
${@bb.utils.contains('DISTRO_FEATURES','sysvinit','false','true',d)}; then
 rm -rf ${D}${sysconfdir}/init.d/
 rm -rf ${D}${sysconfdir}/rc*
-install -d ${D}${sysconfdir}/tmpfiles.d
+install -d ${D}${nonarch_libdir}/tmpfiles.d
 install -m 0644 ${WORKDIR}/pam-volatiles.conf \
-${D}${sysconfdir}/tmpfiles.d/pam.conf
+${D}${nonarch_libdir}/tmpfiles.d/pam.conf
 else
 install -d ${D}${sysconfdir}/default/volatiles
 install -m 0644 ${WORKDIR}/99_pam \
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190787): 
https://lists.openembedded.org/g/openembedded-core/message/190787
Mute This Topic: https://lists.openembedded.org/mt/102625106/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [meta-oe][PATCH v2 3/4] sysstat: use nonarch libdir for tmpfiles.d

2023-11-16 Thread Lukas Funke 
From: Malte Schmidt 

The documentation of systemd states that /etc/tmpfiles.d should be
reserved for the local administrator and packages should put their files
in /usr/lib/tmpfiles.d [1].

[1] https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html

Signed-off-by: Malte Schmidt 
Signed-off-by: Stefan Herbrechtsmeier 
Signed-off-by: Lukas Funke 
---
 meta/recipes-extended/sysstat/sysstat_12.7.4.bb | 10 +++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-extended/sysstat/sysstat_12.7.4.bb 
b/meta/recipes-extended/sysstat/sysstat_12.7.4.bb
index 134fd5cf96..660bc634a0 100644
--- a/meta/recipes-extended/sysstat/sysstat_12.7.4.bb
+++ b/meta/recipes-extended/sysstat/sysstat_12.7.4.bb
@@ -48,9 +48,9 @@ do_install() {
install -m 0644 ${WORKDIR}/99_sysstat ${D}/etc/default/volatiles
fi
if ${@bb.utils.contains('DISTRO_FEATURES', 'systemd', 'true', 'false', 
d)}; then
-   install -d ${D}${sysconfdir}/tmpfiles.d
+   install -d ${D}${nonarch_libdir}/tmpfiles.d
echo "d ${localstatedir}/log/sa - - - -" \
-> ${D}${sysconfdir}/tmpfiles.d/sysstat.conf
+> ${D}${nonarch_libdir}/tmpfiles.d/sysstat.conf
 
# Unless both cron and systemd are enabled, install our own
# systemd unit file. Otherwise the package will install one.
@@ -70,7 +70,11 @@ pkg_postinst:${PN} () {
fi
 }
 
-FILES:${PN} += "${systemd_system_unitdir} ${nonarch_base_libdir}/systemd"
+FILES:${PN} += " \
+   ${systemd_system_unitdir} \
+   ${nonarch_base_libdir}/systemd  \
+   ${nonarch_libdir}/tmpfiles.d \
+"
 
 TARGET_CC_ARCH += "${LDFLAGS}"
 
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190788): 
https://lists.openembedded.org/g/openembedded-core/message/190788
Mute This Topic: https://lists.openembedded.org/mt/102625107/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [meta-oe][PATCH v2 1/4] systemd: use nonarch libdir for tmpfiles.d

2023-11-16 Thread Lukas Funke 
From: Malte Schmidt 

The documentation of systemd states that /etc/tmpfiles.d should be
reserved for the local administrator and packages should put their files
in /usr/lib/tmpfiles.d [1].

[1] https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html

Signed-off-by: Malte Schmidt 
Signed-off-by: Stefan Herbrechtsmeier 
Signed-off-by: Lukas Funke 
---
 meta/recipes-core/systemd/systemd_254.4.bb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/systemd/systemd_254.4.bb 
b/meta/recipes-core/systemd/systemd_254.4.bb
index cc0e29fdd2..6d71fbaba2 100644
--- a/meta/recipes-core/systemd/systemd_254.4.bb
+++ b/meta/recipes-core/systemd/systemd_254.4.bb
@@ -282,12 +282,12 @@ do_install() {
[ ! -e ${D}/${base_sbindir}/udevd ] && ln -s 
${rootlibexecdir}/systemd/systemd-udevd ${D}/${base_sbindir}/udevd
 
install -d ${D}${sysconfdir}/udev/rules.d/
-   install -d ${D}${sysconfdir}/tmpfiles.d
+   install -d ${D}${nonarch_libdir}/tmpfiles.d
for rule in $(find ${WORKDIR} -maxdepth 1 -type f -name "*.rules"); do
install -m 0644 $rule ${D}${sysconfdir}/udev/rules.d/
done
 
-   install -m 0644 ${WORKDIR}/00-create-volatile.conf 
${D}${sysconfdir}/tmpfiles.d/
+   install -m 0644 ${WORKDIR}/00-create-volatile.conf 
${D}${exec_prefix}/lib/tmpfiles.d/
 
if 
${@bb.utils.contains('DISTRO_FEATURES','sysvinit','true','false',d)}; then
install -d ${D}${sysconfdir}/init.d
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190785): 
https://lists.openembedded.org/g/openembedded-core/message/190785
Mute This Topic: https://lists.openembedded.org/mt/102625104/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [meta-oe][PATCH v2 0/4] tmpfiles.d: use nonarch libdir

2023-11-16 Thread Lukas Funke 
From: Lukas Funke 

The series intents to move tmpfiles.d configurations from /etc to
/usr/lib.

The documentation of systemd states that /etc/tmpfiles.d should be
reserved for the local administrator and packages should put their files
in /usr/lib/tmpfiles.d [1].

[1] https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html

Signed-off-by: Lukas Funke 

---
Changes in v2:
 - Added Sign-off-by to commit messages
 - Use ${nonarch_libdir} in systemd recipe

Malte Schmidt (3):
  systemd: use nonarch libdir for tmpfiles.d
  pam: use nonarch libdir for tmpfiles.d
  sysstat: use nonarch libdir for tmpfiles.d

Stefan Herbrechtsmeier (1):
  glibc: use nonarch libdir for tmpfiles.d

 meta/recipes-core/glibc/glibc-package.inc   |  8 
 meta/recipes-core/systemd/systemd_254.4.bb  |  4 ++--
 meta/recipes-extended/pam/libpam_1.5.3.bb   |  9 ++---
 meta/recipes-extended/sysstat/sysstat_12.7.4.bb | 10 +++---
 4 files changed, 19 insertions(+), 12 deletions(-)

-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190786): 
https://lists.openembedded.org/g/openembedded-core/message/190786
Mute This Topic: https://lists.openembedded.org/mt/102625105/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-Core][PATCH v3 0/4] recipetool: Add handler to create go recipes

2023-11-14 Thread Lukas Funke 

Hi Slava,

On 15.11.2023 06:59, Vyacheslav Yurkov wrote:

Hi Lukas,
Thanks for the v3. I know I'm late to the party, because it's already 
merged, but my testing results are below anyway.


No problem. Party is still going on. The 'modules.txt' is intendet to be 
in your ${WORKDIR} as it is part of the SRC_URI, generated by
the recipetool. My guess would be, that there is some missconfiguration 
in your SRC_URI?




The series seems to have handled my ssh URL correctly.

I've got an error in do_go_vendor though.
File: ''/meta/classes/go-vendor.bbclass', lineno: 166, function: 
do_go_vendor

  0162:
  0163:    # Copy vendor manifest
  0164:    modules_txt_src = os.path.join(d.getVar('WORKDIR'), 
"modules.txt")

  0165:    bb.debug(1, "cp %s --> %s" % (modules_txt_src, vendor_dir))
  *** 0166:    shutil.copy2(modules_txt_src, vendor_dir)
  0167:
  0168:    # Clean up vendor dir
  0169:    # We only require the modules in the modules_txt file
  0170:    fetched_paths = set([os.path.relpath(x[0], vendor_dir) 
for x in os.walk(vendor_dir)])

File: '/usr/lib64/python3.9/shutil.py', lineno: 444, function: copy2
  0440:    resembles GNU's "cp -P src dst".
  0441:    """
  0442:    if os.path.isdir(dst):
  0443:    dst = os.path.join(dst, os.path.basename(src))
  *** 0444:    copyfile(src, dst, follow_symlinks=follow_symlinks)
  0445:    copystat(src, dst, follow_symlinks=follow_symlinks)
  0446:    return dst
  0447:
  0448:def ignore_patterns(*patterns):
File: '/usr/lib64/python3.9/shutil.py', lineno: 264, function: copyfile
  0260:
  0261:    if not follow_symlinks and _islink(src):
  0262:    os.symlink(os.readlink(src), dst)
  0263:    else:
  *** 0264:    with open(src, 'rb') as fsrc:
  0265:    try:
  0266:    with open(dst, 'wb') as fdst:
  0267:    # macOS
  0268:    if _HAS_FCOPYFILE:
Exception: FileNotFoundError: [Errno 2] No such file or directory: 
'/modules.txt'


I've located "modules.txt" in my $WORKDIR/$GO_IMPORT/vendor directory 
though, which seems to be consistent with 
https://go.dev/ref/mod#go-mod-file-go . Am I missing something or I can 
send a fixup?


Slava

On 02.11.2023 16:53, lukas.funke-...@weidmueller.com wrote:

From: Lukas Funke 

This patch series adds a recipetool handler in order to create 'go' 
recipes.

Each recipe contains a list of dependencies in their SRC_URI.
Dependencies are derived from the projects `go.mod` file. For each
dependency the corresponding license file uri/hash is added.

The recipe may not work ad-hoc, but is a good starting point to create
a working recipe and have a working offline-build.

In addition to the main recipe three additional files will be generated:
   - $pn-modules.inc
   - $pn-license.inc
   - modules.txt

Changes from v2:
   - Generate separate *.inc for go dependencies and licenses
   - Adapted oe-selftest according to change above
   - Incorparate community suggestions

Lukas Funke (4):
   classes: go-vendor: Add go-vendor class
   selftest: recipetool: Add test for go recipe handler
   recipetool: Ignore *.go files while scanning for licenses
   recipetool: Add handler to create go recipes

  meta/classes/go-vendor.bbclass | 135 
  meta/lib/oeqa/selftest/cases/recipetool.py | 163 +
  scripts/lib/recipetool/create.py   |   2 +-
  scripts/lib/recipetool/create_go.py    | 730 +
  4 files changed, 1029 insertions(+), 1 deletion(-)
  create mode 100644 meta/classes/go-vendor.bbclass
  create mode 100644 scripts/lib/recipetool/create_go.py





Best regards
Lukas

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190557): 
https://lists.openembedded.org/g/openembedded-core/message/190557
Mute This Topic: https://lists.openembedded.org/mt/102345308/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-Core][PATCH 1/4] systemd: use nonarch libdir for tmpfiles.d

2023-11-14 Thread Lukas Funke 
From: Malte Schmidt 

The documentation of systemd states that /etc/tmpfiles.d should be
reserved for the local administrator and packages should put their files
in /usr/lib/tmpfiles.d [1].

[1] https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html

Signed-off-by: Malte Schmidt 
Signed-off-by: Stefan Herbrechtsmeier 
---
 meta/recipes-core/systemd/systemd_254.4.bb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/systemd/systemd_254.4.bb 
b/meta/recipes-core/systemd/systemd_254.4.bb
index cc0e29fdd2..48d467dfdd 100644
--- a/meta/recipes-core/systemd/systemd_254.4.bb
+++ b/meta/recipes-core/systemd/systemd_254.4.bb
@@ -282,12 +282,12 @@ do_install() {
[ ! -e ${D}/${base_sbindir}/udevd ] && ln -s 
${rootlibexecdir}/systemd/systemd-udevd ${D}/${base_sbindir}/udevd
 
install -d ${D}${sysconfdir}/udev/rules.d/
-   install -d ${D}${sysconfdir}/tmpfiles.d
+   install -d ${D}${exec_prefix}/lib/tmpfiles.d
for rule in $(find ${WORKDIR} -maxdepth 1 -type f -name "*.rules"); do
install -m 0644 $rule ${D}${sysconfdir}/udev/rules.d/
done
 
-   install -m 0644 ${WORKDIR}/00-create-volatile.conf 
${D}${sysconfdir}/tmpfiles.d/
+   install -m 0644 ${WORKDIR}/00-create-volatile.conf 
${D}${exec_prefix}/lib/tmpfiles.d/
 
if 
${@bb.utils.contains('DISTRO_FEATURES','sysvinit','true','false',d)}; then
install -d ${D}${sysconfdir}/init.d
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190493): 
https://lists.openembedded.org/g/openembedded-core/message/190493
Mute This Topic: https://lists.openembedded.org/mt/102581153/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-Core][PATCH 4/4] glibc: use nonarch libdir for tmpfiles.d

2023-11-14 Thread Lukas Funke 
From: Stefan Herbrechtsmeier 

The documentation of systemd states that /etc/tmpfiles.d should be
reserved for the local administrator and packages should put their files
in /usr/lib/tmpfiles.d [1].

[1] https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html

Signed-off-by: Stefan Herbrechtsmeier 
---
 meta/recipes-core/glibc/glibc-package.inc | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/meta/recipes-core/glibc/glibc-package.inc 
b/meta/recipes-core/glibc/glibc-package.inc
index 1d4e4c5274..1ef987be0a 100644
--- a/meta/recipes-core/glibc/glibc-package.inc
+++ b/meta/recipes-core/glibc/glibc-package.inc
@@ -42,7 +42,7 @@ FILES_SOLIBSDEV = "${libdir}/lib*${SOLIBSDEV}"
 FILES:${PN}-dev += "${libdir}/libpthread.a ${libdir}/libdl.a 
${libdir}/libutil.a ${libdir}/libanl.a ${libdir}/*_nonshared.a 
${base_libdir}/*_nonshared.a ${base_libdir}/*.o ${datadir}/aclocal"
 RDEPENDS:${PN}-dev = "linux-libc-headers-dev"
 FILES:${PN}-staticdev += "${libdir}/*.a ${base_libdir}/*.a"
-FILES:nscd = "${sbindir}/nscd* ${sysconfdir}/init.d/nscd 
${systemd_system_unitdir}/nscd* ${sysconfdir}/tmpfiles.d/nscd.conf \
+FILES:nscd = "${sbindir}/nscd* ${sysconfdir}/init.d/nscd 
${systemd_system_unitdir}/nscd* ${nonarch_libdir}/tmpfiles.d/nscd.conf \
   ${sysconfdir}/nscd.conf ${sysconfdir}/default/volatiles/98_nscd 
${localstatedir}/db/nscd"
 FILES:${PN}-mtrace = "${bindir}/mtrace"
 FILES:tzcode = "${bindir}/tzselect ${sbindir}/zic ${bindir}/zdump"
@@ -132,9 +132,9 @@ def get_libc_fpu_setting(bb, d):
 
 do_install:append:class-target() {
if ${@bb.utils.contains('DISTRO_FEATURES', 'systemd', 'true', 'false', 
d)}; then
-   install -d ${D}${sysconfdir}/tmpfiles.d
+   install -d ${D}${nonarch_libdir}/tmpfiles.d
echo "d /run/nscd 755 root root -" \
-   > ${D}${sysconfdir}/tmpfiles.d/nscd.conf
+   > ${D}${nonarch_libdir}/tmpfiles.d/nscd.conf
fi
 
if ${@bb.utils.contains('DISTRO_FEATURES', 'sysvinit', 'true', 'false', 
d)}; then
@@ -280,7 +280,7 @@ python populate_packages:prepend () {
 pkg_postinst:nscd () {
if [ -z "$D" ]; then
if command -v systemd-tmpfiles >/dev/null; then
-   systemd-tmpfiles --create 
${sysconfdir}/tmpfiles.d/nscd.conf
+   systemd-tmpfiles --create 
${nonarch_libdir}/tmpfiles.d/nscd.conf
elif [ -e ${sysconfdir}/init.d/populate-volatile.sh ]; then
${sysconfdir}/init.d/populate-volatile.sh update
fi
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190496): 
https://lists.openembedded.org/g/openembedded-core/message/190496
Mute This Topic: https://lists.openembedded.org/mt/102581156/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-Core][PATCH 2/4] pam: use nonarch libdir for tmpfiles.d

2023-11-14 Thread Lukas Funke 
From: Malte Schmidt 

The documentation of systemd states that /etc/tmpfiles.d should be
reserved for the local administrator and packages should put their files
in /usr/lib/tmpfiles.d [1].

[1] https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html

Signed-off-by: Malte Schmidt 
Signed-off-by: Stefan Herbrechtsmeier 
---
 meta/recipes-extended/pam/libpam_1.5.3.bb | 9 ++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-extended/pam/libpam_1.5.3.bb 
b/meta/recipes-extended/pam/libpam_1.5.3.bb
index 1aa307af4d..7af3ea99d1 100644
--- a/meta/recipes-extended/pam/libpam_1.5.3.bb
+++ b/meta/recipes-extended/pam/libpam_1.5.3.bb
@@ -49,7 +49,10 @@ PACKAGECONFIG[audit] = 
"--enable-audit,--disable-audit,audit,"
 PACKAGECONFIG[userdb] = "--enable-db=db,--enable-db=no,db,"
 
 PACKAGES += "${PN}-runtime ${PN}-xtests"
-FILES:${PN} = "${base_libdir}/lib*${SOLIBS}"
+FILES:${PN} = " \
+${base_libdir}/lib*${SOLIBS} \
+${nonarch_libdir}/tmpfiles.d/*.conf \
+"
 FILES:${PN}-dev += "${base_libdir}/security/*.la ${base_libdir}/*.la 
${base_libdir}/lib*${SOLIBSDEV}"
 FILES:${PN}-runtime = "${sysconfdir} ${sbindir} ${systemd_system_unitdir}"
 FILES:${PN}-xtests = "${datadir}/Linux-PAM/xtests"
@@ -130,9 +133,9 @@ do_install() {
 if 
${@bb.utils.contains('DISTRO_FEATURES','sysvinit','false','true',d)}; then
 rm -rf ${D}${sysconfdir}/init.d/
 rm -rf ${D}${sysconfdir}/rc*
-install -d ${D}${sysconfdir}/tmpfiles.d
+install -d ${D}${nonarch_libdir}/tmpfiles.d
 install -m 0644 ${WORKDIR}/pam-volatiles.conf \
-${D}${sysconfdir}/tmpfiles.d/pam.conf
+${D}${nonarch_libdir}/tmpfiles.d/pam.conf
 else
 install -d ${D}${sysconfdir}/default/volatiles
 install -m 0644 ${WORKDIR}/99_pam \
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190494): 
https://lists.openembedded.org/g/openembedded-core/message/190494
Mute This Topic: https://lists.openembedded.org/mt/102581154/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-Core][PATCH 0/4] tmpfiles.d: use nonarch libdir

2023-11-14 Thread Lukas Funke 
From: Lukas Funke 

The series intents to move tmpfiles.d configurations from /etc to /usr/lib.

The documentation of systemd states that /etc/tmpfiles.d should be
reserved for the local administrator and packages should put their files
in /usr/lib/tmpfiles.d [1].

[1] https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html

Malte Schmidt (3):
  systemd: use nonarch libdir for tmpfiles.d
  pam: use nonarch libdir for tmpfiles.d
  sysstat: use nonarch libdir for tmpfiles.d

Stefan Herbrechtsmeier (1):
  glibc: use nonarch libdir for tmpfiles.d

 meta/recipes-core/glibc/glibc-package.inc   |  8 
 meta/recipes-core/systemd/systemd_254.4.bb  |  4 ++--
 meta/recipes-extended/pam/libpam_1.5.3.bb   |  9 ++---
 meta/recipes-extended/sysstat/sysstat_12.7.4.bb | 10 +++---
 4 files changed, 19 insertions(+), 12 deletions(-)

-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190492): 
https://lists.openembedded.org/g/openembedded-core/message/190492
Mute This Topic: https://lists.openembedded.org/mt/102581150/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-Core][PATCH 3/4] sysstat: use nonarch libdir for tmpfiles.d

2023-11-14 Thread Lukas Funke 
From: Malte Schmidt 

The documentation of systemd states that /etc/tmpfiles.d should be
reserved for the local administrator and packages should put their files
in /usr/lib/tmpfiles.d [1].

[1] https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html

Signed-off-by: Malte Schmidt 
Signed-off-by: Stefan Herbrechtsmeier 
---
 meta/recipes-extended/sysstat/sysstat_12.7.4.bb | 10 +++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-extended/sysstat/sysstat_12.7.4.bb 
b/meta/recipes-extended/sysstat/sysstat_12.7.4.bb
index 134fd5cf96..660bc634a0 100644
--- a/meta/recipes-extended/sysstat/sysstat_12.7.4.bb
+++ b/meta/recipes-extended/sysstat/sysstat_12.7.4.bb
@@ -48,9 +48,9 @@ do_install() {
install -m 0644 ${WORKDIR}/99_sysstat ${D}/etc/default/volatiles
fi
if ${@bb.utils.contains('DISTRO_FEATURES', 'systemd', 'true', 'false', 
d)}; then
-   install -d ${D}${sysconfdir}/tmpfiles.d
+   install -d ${D}${nonarch_libdir}/tmpfiles.d
echo "d ${localstatedir}/log/sa - - - -" \
-> ${D}${sysconfdir}/tmpfiles.d/sysstat.conf
+> ${D}${nonarch_libdir}/tmpfiles.d/sysstat.conf
 
# Unless both cron and systemd are enabled, install our own
# systemd unit file. Otherwise the package will install one.
@@ -70,7 +70,11 @@ pkg_postinst:${PN} () {
fi
 }
 
-FILES:${PN} += "${systemd_system_unitdir} ${nonarch_base_libdir}/systemd"
+FILES:${PN} += " \
+   ${systemd_system_unitdir} \
+   ${nonarch_base_libdir}/systemd  \
+   ${nonarch_libdir}/tmpfiles.d \
+"
 
 TARGET_CC_ARCH += "${LDFLAGS}"
 
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190495): 
https://lists.openembedded.org/g/openembedded-core/message/190495
Mute This Topic: https://lists.openembedded.org/mt/102581155/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-Core][PATCH 0/1] wic: extend empty plugin with options to write zeros to partiton

2023-11-14 Thread Lukas Funke 
From: Lukas Funke 

Adds features to explicitly write zeros to the start of the
partition. This is useful to overwrite old content like
filesystem signatures which may be re-recognized otherwise.

The new features can be enabled with
'--soucreparams="[fill|size=[S|s|K|k|M|G]][,][bs=[S|s|K|k|M|G]]"'
Conflicting or missing options throw errors.

The features are:
- fill
  Fill the entire partition with zeros. Requires '--fixed-size' option
  to be set.
- size=[S|s|K|k|M|G]
  Set the first N bytes of the partition to zero. Default unit is 'K'.
- bs=[S|s|K|k|M|G]
  Write at most N bytes at a time during source file creation.
  Defaults to '1M'. Default unit is 'K'.

Malte Schmidt (1):
  wic: extend empty plugin with options to write zeros to partiton

 scripts/lib/wic/plugins/source/empty.py | 57 -
 1 file changed, 56 insertions(+), 1 deletion(-)

-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190488): 
https://lists.openembedded.org/g/openembedded-core/message/190488
Mute This Topic: https://lists.openembedded.org/mt/102579764/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-Core][PATCH 1/1] wic: extend empty plugin with options to write zeros to partiton

2023-11-14 Thread Lukas Funke 
From: Malte Schmidt 

Adds features to explicitly write zeros to the start of the
partition. This is useful to overwrite old content like
filesystem signatures which may be re-recognized otherwise.

The new features can be enabled with
'--soucreparams="[fill|size=[S|s|K|k|M|G]][,][bs=[S|s|K|k|M|G]]"'
Conflicting or missing options throw errors.

The features are:
- fill
  Fill the entire partition with zeros. Requires '--fixed-size' option
  to be set.
- size=[S|s|K|k|M|G]
  Set the first N bytes of the partition to zero. Default unit is 'K'.
- bs=[S|s|K|k|M|G]
  Write at most N bytes at a time during source file creation.
  Defaults to '1M'. Default unit is 'K'.

Signed-off-by: Malte Schmidt 
---
 scripts/lib/wic/plugins/source/empty.py | 57 -
 1 file changed, 56 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/wic/plugins/source/empty.py 
b/scripts/lib/wic/plugins/source/empty.py
index 041617d648..f7581a0298 100644
--- a/scripts/lib/wic/plugins/source/empty.py
+++ b/scripts/lib/wic/plugins/source/empty.py
@@ -7,9 +7,19 @@
 # To use it you must pass "empty" as argument for the "--source" parameter in
 # the wks file. For example:
 # part foo --source empty --ondisk sda --size="1024" --align 1024
+#
+# The plugin supports writing zeros to the start of the
+# partition. This is useful to overwrite old content like
+# filesystem signatures which may be re-recognized otherwise.
+# This feature can be enabled with
+# '--soucreparams="[fill|size=[S|s|K|k|M|G]][,][bs=[S|s|K|k|M|G]]"'
+# Conflicting or missing options throw errors.
 
 import logging
+import os
 
+from wic import WicError
+from wic.ksparser import sizetype
 from wic.pluginbase import SourcePlugin
 
 logger = logging.getLogger('wic')
@@ -17,6 +27,16 @@ logger = logging.getLogger('wic')
 class EmptyPartitionPlugin(SourcePlugin):
 """
 Populate unformatted empty partition.
+
+The following sourceparams are supported:
+- fill
+  Fill the entire partition with zeros. Requires '--fixed-size' option
+  to be set.
+- size=[S|s|K|k|M|G]
+  Set the first N bytes of the partition to zero. Default unit is 'K'.
+- bs=[S|s|K|k|M|G]
+  Write at most N bytes at a time during source file creation.
+  Defaults to '1M'. Default unit is 'K'.
 """
 
 name = 'empty'
@@ -29,4 +49,39 @@ class EmptyPartitionPlugin(SourcePlugin):
 Called to do the actual content population for a partition i.e. it
 'prepares' the partition to be incorporated into the image.
 """
-return
+get_byte_count = sizetype('K', True)
+size = 0
+
+if 'fill' in source_params and 'size' in source_params:
+raise WicError("Conflicting source parameters 'fill' and 'size' 
specified, exiting.")
+
+# Set the size of the zeros to be written to the partition
+if 'fill' in source_params:
+if part.fixed_size == 0:
+raise WicError("Source parameter 'fill' only works with the 
'--fixed-size' option, exiting.")
+size = part.fixed_size
+elif 'size' in source_params:
+size = get_byte_count(source_params['size'])
+
+if size == 0:
+# Nothing to do, create empty partition
+return
+
+if 'bs' in source_params:
+bs = get_byte_count(source_params['bs'])
+else:
+bs = get_byte_count('1M')
+
+# Create a binary file of the requested size filled with zeros
+source_file = os.path.join(cr_workdir, 'empty-plugin-zeros%s.bin' % 
part.lineno)
+if not os.path.exists(os.path.dirname(source_file)):
+os.makedirs(os.path.dirname(source_file))
+
+quotient, remainder = divmod(size, bs)
+with open(source_file, 'wb') as file:
+for _ in range(quotient):
+file.write(bytearray(bs))
+file.write(bytearray(remainder))
+
+part.size = (size + 1024 - 1) // 1024  # size in KB rounded up
+part.source_file = source_file
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190489): 
https://lists.openembedded.org/g/openembedded-core/message/190489
Mute This Topic: https://lists.openembedded.org/mt/102579765/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-Core][PATCH v3] wic: rawcopy: add support for zsdt decompression

2023-11-13 Thread Lukas Funke 
From: Malte Schmidt 

---
 scripts/lib/wic/plugins/source/rawcopy.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/wic/plugins/source/rawcopy.py 
b/scripts/lib/wic/plugins/source/rawcopy.py
index 7c90cd3cf8..82d38fbb84 100644
--- a/scripts/lib/wic/plugins/source/rawcopy.py
+++ b/scripts/lib/wic/plugins/source/rawcopy.py
@@ -52,7 +52,8 @@ class RawCopyPlugin(SourcePlugin):
 decompressor = {
 ".bz2": "bzip2",
 ".gz": "gzip",
-".xz": "xz"
+".xz": "xz",
+".zst": "zstd -f",
 }.get(extension)
 if not decompressor:
 raise WicError("Not supported compressor filename extension: %s" % 
extension)
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190473): 
https://lists.openembedded.org/g/openembedded-core/message/190473
Mute This Topic: https://lists.openembedded.org/mt/102561555/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-Core][PATCH 2/2] patch: extract patches with all-zero hash

2023-11-13 Thread Lukas Funke 
From: Stefan Herbrechtsmeier 

Extract patches with all-zero hash in each patch header instead of the
hash of the commit to reduce changes during patch refresh.

Signed-off-by: Stefan Herbrechtsmeier 
---
 meta/lib/oe/patch.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oe/patch.py b/meta/lib/oe/patch.py
index 71cd193afb..cc4c8f4371 100644
--- a/meta/lib/oe/patch.py
+++ b/meta/lib/oe/patch.py
@@ -465,7 +465,7 @@ class GitApplyTree(PatchTree):
 tempdir = tempfile.mkdtemp(prefix='oepatch')
 try:
 shellcmd = ["git", "format-patch", "--no-signature", 
"--no-numbered",
-"--no-stat", startcommit, "-o", tempdir]
+"--no-stat", "--zero-commit", startcommit, "-o", 
tempdir]
 if paths:
 shellcmd.append('--')
 shellcmd.extend(paths)
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190472): 
https://lists.openembedded.org/g/openembedded-core/message/190472
Mute This Topic: https://lists.openembedded.org/mt/102561451/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-Core][PATCH 1/2] patch: extract patches without diffstats

2023-11-13 Thread Lukas Funke 
From: Stefan Herbrechtsmeier 

Extract patches without diffstats to reduce changes during patch
refresh.

Signed-off-by: Stefan Herbrechtsmeier 
---
 meta/lib/oe/patch.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/patch.py b/meta/lib/oe/patch.py
index 4ec9caed45..71cd193afb 100644
--- a/meta/lib/oe/patch.py
+++ b/meta/lib/oe/patch.py
@@ -464,7 +464,8 @@ class GitApplyTree(PatchTree):
 import shutil
 tempdir = tempfile.mkdtemp(prefix='oepatch')
 try:
-shellcmd = ["git", "format-patch", "--no-signature", 
"--no-numbered", startcommit, "-o", tempdir]
+shellcmd = ["git", "format-patch", "--no-signature", 
"--no-numbered",
+"--no-stat", startcommit, "-o", tempdir]
 if paths:
 shellcmd.append('--')
 shellcmd.extend(paths)
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190471): 
https://lists.openembedded.org/g/openembedded-core/message/190471
Mute This Topic: https://lists.openembedded.org/mt/102561450/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-Core][PATCH 0/2] patch: reduce changes during patch refresh

2023-11-13 Thread Lukas Funke 
From: Lukas Funke 

The patch series aims to reduce the noise in patches created by devtools. Some
diffs are just introduced due to an update in the hash or in the diffstats.
These changes are not important to a reviewer.

Stefan Herbrechtsmeier (2):
  patch: extract patches without diffstats
  patch: extract patches with all-zero hash

 meta/lib/oe/patch.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190470): 
https://lists.openembedded.org/g/openembedded-core/message/190470
Mute This Topic: https://lists.openembedded.org/mt/102561449/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-Core][PATCH v2] udev-extraconf: mount.sh: check if filesystem is supported before mounting

2023-11-08 Thread Lukas Funke 
From: Lukas Funke 

Check if the filesystem is supported by the kernel before trying to
mount it. Systemd-mount will mount the directories asynchronously
resulting in stale directories if the devices filesystem is not
supported.

Signed-off-by: Lukas Funke 
---
 meta/recipes-core/udev/udev-extraconf/mount.sh | 15 +++
 1 file changed, 15 insertions(+)

diff --git a/meta/recipes-core/udev/udev-extraconf/mount.sh 
b/meta/recipes-core/udev/udev-extraconf/mount.sh
index 989fabc194..adf0c7aafa 100644
--- a/meta/recipes-core/udev/udev-extraconf/mount.sh
+++ b/meta/recipes-core/udev/udev-extraconf/mount.sh
@@ -36,6 +36,16 @@ do
fi
 done
 
+is_filesystem_supported() {
+while read -r fs; do
+   if [ "${fs#nodev}" = "$1" ];
+   then
+   return 0
+   fi
+done < "/proc/filesystems"
+return 1
+}
+
 automount_systemd() {
 name="`basename "$DEVNAME"`"
 
@@ -64,6 +74,11 @@ automount_systemd() {
 grep "^[[:space:]]*$tmp" /etc/fstab && return
 done
 
+if ! is_filesystem_supported $ID_FS_TYPE; then
+logger "mount.sh/automount" "Filesystem '$ID_FS_TYPE' on '${DEVNAME}' 
is unsupported"
+return
+fi
+
 [ -d "$MOUNT_BASE/$name" ] || mkdir -p "$MOUNT_BASE/$name"
 
 MOUNT="$MOUNT -o silent"
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190308): 
https://lists.openembedded.org/g/openembedded-core/message/190308
Mute This Topic: https://lists.openembedded.org/mt/102460745/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-Core][PATCH] udev-extraconf: mount.sh: check if filesystem is supported before mounting

2023-11-07 Thread Lukas Funke 

Hi Richard,

On 07.11.2023 13:15, Richard Purdie wrote:

On Tue, 2023-11-07 at 13:00 +0100, Lukas Funke wrote:

From: Lukas Funke 

Check if the filesystem is supported by the kernel before trying to
mount it. Systemd-mount will mount the directories asynchronously
resulting in stale directories if the devices filesystem is not
supported.

Signed-off-by: Lukas Funke 
---
  meta/recipes-core/udev/udev-extraconf/mount.sh | 16 
  1 file changed, 16 insertions(+)

diff --git a/meta/recipes-core/udev/udev-extraconf/mount.sh 
b/meta/recipes-core/udev/udev-extraconf/mount.sh
index 989fabc194..59563b7511 100644
--- a/meta/recipes-core/udev/udev-extraconf/mount.sh
+++ b/meta/recipes-core/udev/udev-extraconf/mount.sh
@@ -36,6 +36,17 @@ do
fi
  done
  
+is_filesystem_supported() {

+list_fs=$(cat /proc/filesystems | cut -f 2 | tr '\n' ' ')
+for fs in ${list_fs}; do
+if [ "$fs" == "$ID_FS_TYPE" ]
+then
+return 0
+fi
+done
+return 1
+}
+


We once went through the init scripts and carefully minimised all the
fork() calls in them, since all that process execution overhead does
mount up and delay boot times.

I really don't know what to do, whether I should continue to care or
not. It is easy to say systems have plenty of memory/cpu and therefore,
"who cares?". Equally, systems working efficiently is nice to have...


Not sure if correctly interpreted, but it seems that there are some 
"frustration" vibes in your reply :)


I was not aware of the discussion but understand what you are trying to 
achieve. We can replace the 'cat' with a 'read', which should avoid the 
fork() since it's a shell-builtin (correct me if I'm wrong).


Best regards
Lukas



Cheers,

Richard





-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190280): 
https://lists.openembedded.org/g/openembedded-core/message/190280
Mute This Topic: https://lists.openembedded.org/mt/102440934/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-Core][PATCH] udev-extraconf: mount.sh: check if filesystem is supported before mounting

2023-11-07 Thread Lukas Funke 
From: Lukas Funke 

Check if the filesystem is supported by the kernel before trying to
mount it. Systemd-mount will mount the directories asynchronously
resulting in stale directories if the devices filesystem is not
supported.

Signed-off-by: Lukas Funke 
---
 meta/recipes-core/udev/udev-extraconf/mount.sh | 16 
 1 file changed, 16 insertions(+)

diff --git a/meta/recipes-core/udev/udev-extraconf/mount.sh 
b/meta/recipes-core/udev/udev-extraconf/mount.sh
index 989fabc194..59563b7511 100644
--- a/meta/recipes-core/udev/udev-extraconf/mount.sh
+++ b/meta/recipes-core/udev/udev-extraconf/mount.sh
@@ -36,6 +36,17 @@ do
fi
 done
 
+is_filesystem_supported() {
+list_fs=$(cat /proc/filesystems | cut -f 2 | tr '\n' ' ')
+for fs in ${list_fs}; do
+if [ "$fs" == "$ID_FS_TYPE" ]
+then
+return 0
+fi
+done
+return 1
+}
+
 automount_systemd() {
 name="`basename "$DEVNAME"`"
 
@@ -64,6 +75,11 @@ automount_systemd() {
 grep "^[[:space:]]*$tmp" /etc/fstab && return
 done
 
+if ! is_filesystem_supported; then
+logger "mount.sh/automount" "Filesystem '$ID_FS_TYPE' on '${DEVNAME}' 
is unsupported"
+return
+fi
+
 [ -d "$MOUNT_BASE/$name" ] || mkdir -p "$MOUNT_BASE/$name"
 
 MOUNT="$MOUNT -o silent"
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190272): 
https://lists.openembedded.org/g/openembedded-core/message/190272
Mute This Topic: https://lists.openembedded.org/mt/102440934/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-Core][PATCH v3 2/4] selftest: recipetool: Add test for go recipe handler

2023-11-02 Thread Lukas Funke 
From: Lukas Funke 

This commit adds a test for the go recipetool handler. The choosen go
project to test the created recipe was picked randomly. The SRC_URIs and
the LIC_FILES_CHKSUMs are checked against there reference values.

Signed-off-by: Lukas Funke 
---
 meta/lib/oeqa/selftest/cases/recipetool.py | 185 +
 1 file changed, 185 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py 
b/meta/lib/oeqa/selftest/cases/recipetool.py
index 48661bee6f..73b5ce1d83 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -532,6 +532,191 @@ class RecipetoolTests(RecipetoolBase):
 libpath = os.path.join(get_bb_var('COREBASE'), 'scripts', 'lib', 
'recipetool')
 sys.path.insert(0, libpath)
 
+def test_recipetool_create_go(self):
+# Basic test to check go recipe generation
+def urifiy(url, version, modulepath = None, pathmajor = None, subdir = 
None):
+modulepath = ",path='%s'" % modulepath if len(modulepath) else ''
+pathmajor = ",pathmajor='%s'" % pathmajor if len(pathmajor) else ''
+subdir = ",subdir='%s'" % subdir if len(subdir) else ''
+return "${@go_src_uri('%s','%s'%s%s%s)}" % (url, version, 
modulepath, pathmajor, subdir)
+
+temprecipe = os.path.join(self.tempdir, 'recipe')
+os.makedirs(temprecipe)
+
+recipefile = os.path.join(temprecipe, 'edgex-go_git.bb')
+deps_require_file = os.path.join(temprecipe, 'edgex-go', 
'edgex-go-modules.inc')
+lics_require_file = os.path.join(temprecipe, 'edgex-go', 
'edgex-go-licenses.inc')
+modules_txt_file = os.path.join(temprecipe, 'edgex-go', 'modules.txt')
+
+srcuri = 'https://github.com/edgexfoundry/edgex-go.git'
+srcrev = "v3.0.0"
+srcbranch = "main"
+
+result = runCmd('recipetool create -o %s %s -S %s -B %s' % 
(temprecipe, srcuri, srcrev, srcbranch))
+
+self.maxDiff = None
+inherits = ['go-vendor']
+
+checkvars = {}
+checkvars['GO_IMPORT'] = "github.com/edgexfoundry/edgex-go"
+checkvars['SRC_URI'] = 
{'git://${GO_IMPORT};destsuffix=git/src/${GO_IMPORT};nobranch=1;name=${BPN};protocol=https',
+'file://modules.txt'}
+checkvars['LIC_FILES_CHKSUM'] = 
{'file://src/${GO_IMPORT}/LICENSE;md5=8f8bc924cf73f6a32381e5fd4c58d603'}
+
+self.assertTrue(os.path.isfile(recipefile))
+self._test_recipe_contents(recipefile, checkvars, inherits)
+
+checkvars = {}
+checkvars['VENDORED_LIC_FILES_CHKSUM'] = set(
+ 
['file://src/${GO_IMPORT}/vendor/github.com/Microsoft/go-winio/LICENSE;md5=69205ff73858f2c22b2ca135b557e8ef',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/armon/go-metrics/LICENSE;md5=d2d77030c0183e3d1e66d26dc1f243be',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/cenkalti/backoff/LICENSE;md5=1571d94433e3f3aa05267efd4dbea68b',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/davecgh/go-spew/LICENSE;md5=c06795ed54b2a35ebeeb543cd3a73e56',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/eclipse/paho.mqtt.golang/LICENSE;md5=dcdb33474b60c38efd27356d8f2edec7',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/eclipse/paho.mqtt.golang/edl-v10;md5=3adfcc70f5aeb7a44f3f9b495aa1fbf3',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/edgexfoundry/go-mod-bootstrap/v3/LICENSE;md5=0d6dae39976133b2851fba4c1e1275ff',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/edgexfoundry/go-mod-configuration/v3/LICENSE;md5=0d6dae39976133b2851fba4c1e1275ff',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/edgexfoundry/go-mod-core-contracts/v3/LICENSE;md5=0d6dae39976133b2851fba4c1e1275ff',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/edgexfoundry/go-mod-messaging/v3/LICENSE;md5=0d6dae39976133b2851fba4c1e1275ff',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/edgexfoundry/go-mod-registry/v3/LICENSE;md5=0d6dae39976133b2851fba4c1e1275ff',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/edgexfoundry/go-mod-secrets/v3/LICENSE;md5=f9fa2f4f8e0ef8cc7b5dd150963eb457',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/fatih/color/LICENSE.md;md5=316e6d590bdcde7993fb175662c0dd5a',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/fxamacker/cbor/v2/LICENSE;md5=8

[OE-Core][PATCH v3 3/4] recipetool: Ignore *.go files while scanning for licenses

2023-11-02 Thread Lukas Funke 
From: Lukas Funke 

Signed-off-by: Lukas Funke 
---
 scripts/lib/recipetool/create.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 143bc63e9d..293198d1c8 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1212,7 +1212,7 @@ def guess_license(srctree, d):
 
 licenses = []
 licspecs = ['*LICEN[CS]E*', 'COPYING*', '*[Ll]icense*', 'LEGAL*', 
'[Ll]egal*', '*GPL*', 'README.lic*', 'COPYRIGHT*', '[Cc]opyright*', 
'e[dp]l-v10']
-skip_extensions = (".html", ".js", ".json", ".svg", ".ts")
+skip_extensions = (".html", ".js", ".json", ".svg", ".ts", ".go")
 licfiles = []
 for root, dirs, files in os.walk(srctree):
 for fn in files:
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190083): 
https://lists.openembedded.org/g/openembedded-core/message/190083
Mute This Topic: https://lists.openembedded.org/mt/102345309/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-Core][PATCH v3 1/4] classes: go-vendor: Add go-vendor class

2023-11-02 Thread Lukas Funke 
From: Lukas Funke 

Signed-off-by: Lukas Funke 
---
 meta/classes/go-vendor.bbclass | 200 +
 1 file changed, 200 insertions(+)
 create mode 100644 meta/classes/go-vendor.bbclass

diff --git a/meta/classes/go-vendor.bbclass b/meta/classes/go-vendor.bbclass
new file mode 100644
index 00..5b017b0b9d
--- /dev/null
+++ b/meta/classes/go-vendor.bbclass
@@ -0,0 +1,200 @@
+#
+# Copyright 2023 (C) Weidmueller GmbH & Co KG
+# Author: Lukas Funke 
+#
+# Handle Go vendor support for offline builds
+#
+# When importing Go modules, Go downloads the imported modules using
+# a network (proxy) connection ahead of the compile stage. This contradicts 
+# the yocto build concept of fetching every source ahead of build-time
+# and supporting offline builds.
+#
+# To support offline builds, we use Go 'vendoring': module dependencies are 
+# downloaded during the fetch-phase and unpacked into the modules 'vendor'
+# folder. Additionally a manifest file is generated for the 'vendor' folder
+# 
+
+inherit go-mod
+
+def go_src_uri(repo, version, path=None, subdir=None, \
+vcs='git', replaces=None, pathmajor=None):
+
+destsuffix = "git/src/import/vendor.fetch"
+module_path = repo if not path else path
+
+src_uri = "{}://{};name={}".format(vcs, repo, module_path.replace('/', 
'.'))
+src_uri += ";destsuffix={}/{}@{}".format(destsuffix, repo, version)
+
+if vcs == "git":
+src_uri += ";nobranch=1;protocol=https"
+
+src_uri += ";go_module_path={}".format(module_path)
+
+if replaces:
+src_uri += ";go_module_replacement={}".format(replaces)
+if subdir:
+src_uri += ";go_subdir={}".format(subdir)
+if pathmajor:
+src_uri += ";go_pathmajor={}".format(pathmajor)
+src_uri += ";is_go_dependency=1"
+
+return src_uri
+
+python do_vendor_unlink() {
+
+# We unlink
+
+go_import = d.getVar('GO_IMPORT')
+source_dir = d.getVar('S')
+linkname = os.path.join(source_dir, *['src', go_import, 'vendor'])
+
+os.unlink(linkname)
+}
+
+addtask vendor_unlink before do_install after do_compile
+
+python do_go_vendor() {
+import shutil
+
+src_uri = (d.getVar('SRC_URI') or "").split()
+
+if len(src_uri) == 0:
+bb.error("SRC_URI is empty")
+return
+
+default_destsuffix = "git/src/import/vendor.fetch"
+fetcher = bb.fetch2.Fetch(src_uri, d)
+go_import = d.getVar('GO_IMPORT')
+source_dir = d.getVar('S')
+
+linkname = os.path.join(source_dir, *['src', go_import, 'vendor'])
+vendor_dir = os.path.join(source_dir, *['src', 'import', 'vendor'])
+import_dir = os.path.join(source_dir, *['src', 'import', 'vendor.fetch'])
+
+if os.path.exists(vendor_dir):
+# Nothing to do except re-establish link to actual vendor folder
+if not os.path.exists(linkname):
+os.symlink(vendor_dir, linkname)
+return
+
+bb.utils.mkdirhier(vendor_dir)
+
+modules = {}
+
+for url in fetcher.urls:
+srcuri = fetcher.ud[url].host + fetcher.ud[url].path
+
+# Skip non Go module src uris
+if not fetcher.ud[url].parm.get('is_go_dependency'):
+continue
+
+destsuffix = fetcher.ud[url].parm.get('destsuffix')
+# We derive the module repo / version in the following manner 
(exmaple):
+# 
+# destsuffix = git/src/import/vendor.fetch/github.com/foo/bar@v1.2.3
+# p = github.com/foo/bar@v1.2.3
+# repo = github.com/foo/bar
+# version = v1.2.3
+
+p = destsuffix[len(default_destsuffix)+1:]
+repo, version = p.split('@')
+
+module_path = fetcher.ud[url].parm.get('go_module_path')
+
+subdir = fetcher.ud[url].parm.get('go_subdir')
+subdir = None if not subdir else subdir
+
+pathMajor = fetcher.ud[url].parm.get('go_pathmajor')
+pathMajor = None if not pathMajor else pathMajor.strip('/')
+
+if not repo in modules:
+modules[repo] =   { "version": version,
+"repo_path": os.path.join(import_dir, p),
+"module_path": module_path,
+"subdir": subdir,
+"pathMajor": pathMajor }
+
+for module_key in sorted(modules):
+
+# only take the version which is explicitly listed
+# as a dependency in the go.mod
+module = modules[module_key]
+module_path = module['module_path']
+rootdir

[OE-Core][PATCH v3 4/4] recipetool: Add handler to create go recipes

2023-11-02 Thread Lukas Funke 
From: Lukas Funke 

Signed-off-by: Lukas Funke 
---
 scripts/lib/recipetool/create_go.py | 751 
 1 file changed, 751 insertions(+)
 create mode 100644 scripts/lib/recipetool/create_go.py

diff --git a/scripts/lib/recipetool/create_go.py 
b/scripts/lib/recipetool/create_go.py
new file mode 100644
index 00..c7b2381a33
--- /dev/null
+++ b/scripts/lib/recipetool/create_go.py
@@ -0,0 +1,751 @@
+# Recipe creation tool - go support plugin
+#
+# The code is based on golang internals. See the afftected
+# methods for further reference and information.
+#
+# Copyright (C) 2023 Weidmueller GmbH & Co KG
+# Author: Lukas Funke 
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+
+from collections import namedtuple
+from enum import Enum
+from html.parser import HTMLParser
+from recipetool.create import RecipeHandler, handle_license_vars
+from recipetool.create import guess_license, tidy_licenses, fixup_license
+from recipetool.create import determine_from_url
+from urllib.error import URLError
+
+import bb.utils
+import json
+import logging
+import os
+import re
+import subprocess
+import sys
+import shutil
+import tempfile
+import urllib.parse
+import urllib.request
+
+
+GoImport = namedtuple('GoImport', 'root vcs url suffix')
+logger = logging.getLogger('recipetool')
+CodeRepo = namedtuple(
+'CodeRepo', 'path codeRoot codeDir pathMajor pathPrefix pseudoMajor')
+
+tinfoil = None
+
+# Regular expression to parse pseudo semantic version
+# see https://go.dev/ref/mod#pseudo-versions
+re_pseudo_semver = re.compile(
+
r"^v[0-9]+\.(0\.0-|\d+\.\d+-([^+]*\.)?0\.)(?P\d{14})-(?P[A-Za-z0-9]+)(\+[0-9A-Za-z-]+(\.[0-9A-Za-z-]+)*)?$")
+# Regular expression to parse semantic version
+re_semver = re.compile(
+
r"^v(?P0|[1-9]\d*)\.(?P0|[1-9]\d*)\.(?P0|[1-9]\d*)(?:-(?P(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$")
+
+
+def tinfoil_init(instance):
+global tinfoil
+tinfoil = instance
+
+
+class GoRecipeHandler(RecipeHandler):
+"""Class to handle the go recipe creation"""
+
+@staticmethod
+def __ensure_go():
+"""Check if the 'go' command is available in the recipes"""
+recipe = "go-native"
+if not tinfoil.recipes_parsed:
+tinfoil.parse_recipes()
+try:
+rd = tinfoil.parse_recipe(recipe)
+except bb.providers.NoProvider:
+bb.error(
+"Nothing provides '%s' which is required for the build" % 
(recipe))
+bb.note(
+"You will likely need to add a layer that provides '%s'" % 
(recipe))
+return None
+
+bindir = rd.getVar('STAGING_BINDIR_NATIVE')
+gopath = os.path.join(bindir, 'go')
+
+if not os.path.exists(gopath):
+tinfoil.build_targets(recipe, 'addto_recipe_sysroot')
+
+if not os.path.exists(gopath):
+logger.error(
+'%s required to process specified source, but %s did not 
seem to populate it' % 'go', recipe)
+return None
+
+return bindir
+
+def __resolve_repository_static(self, modulepath):
+"""Resolve the repository in a static manner
+
+The method is based on the go implementation of
+`repoRootFromVCSPaths` in
+
https://github.com/golang/go/blob/master/src/cmd/go/internal/vcs/vcs.go
+"""
+
+url = urllib.parse.urlparse("https://"; + modulepath)
+req = urllib.request.Request(url.geturl())
+
+try:
+resp = urllib.request.urlopen(req)
+# Some modulepath are just redirects to github (or some other vcs
+# hoster). Therefore, we check if this modulepath redirects to
+# somewhere else
+if resp.geturl() != url.geturl():
+bb.debug(1, "%s is redirectred to %s" %
+ (url.geturl(), resp.geturl()))
+url = urllib.parse.urlparse(resp.geturl())
+modulepath = url.netloc + url.path
+
+except URLError as url_err:
+# This is probably because the module path
+# contains the subdir and major path. Thus,
+# we ignore this error for now
+logger.debug(
+1, "Failed to fetch page from [%s]: %s" % (url, str(url_err)))
+
+host, _, _ = modulepath.partition('/')
+
+class vcs(Enum):
+pathprefix = "pathprefix"
+regexp = "regexp"
+type = "type"
+repo = "repo"
+check = "check&qu

[OE-Core][PATCH v3 0/4] recipetool: Add handler to create go recipes

2023-11-02 Thread Lukas Funke 
From: Lukas Funke 

This patch series adds a recipetool handler in order to create 'go' recipes.
Each recipe contains a list of dependencies in their SRC_URI.
Dependencies are derived from the projects `go.mod` file. For each
dependency the corresponding license file uri/hash is added.

The recipe may not work ad-hoc, but is a good starting point to create
a working recipe and have a working offline-build.

In addition to the main recipe three additional files will be generated:
  - $pn-modules.inc
  - $pn-license.inc
  - modules.txt

Changes from v2:
  - Generate separate *.inc for go dependencies and licenses
  - Adapted oe-selftest according to change above
  - Incorparate community suggestions

Lukas Funke (4):
  classes: go-vendor: Add go-vendor class
  selftest: recipetool: Add test for go recipe handler
  recipetool: Ignore *.go files while scanning for licenses
  recipetool: Add handler to create go recipes

 meta/classes/go-vendor.bbclass | 135 
 meta/lib/oeqa/selftest/cases/recipetool.py | 163 +
 scripts/lib/recipetool/create.py   |   2 +-
 scripts/lib/recipetool/create_go.py| 730 +
 4 files changed, 1029 insertions(+), 1 deletion(-)
 create mode 100644 meta/classes/go-vendor.bbclass
 create mode 100644 scripts/lib/recipetool/create_go.py

-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190082): 
https://lists.openembedded.org/g/openembedded-core/message/190082
Mute This Topic: https://lists.openembedded.org/mt/102345308/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-Core][PATCH v2 0/4] recipetool: Add handler to create go recipes

2023-10-24 Thread Lukas Funke 

On 24.10.2023 09:12, Vyacheslav Yurkov wrote:

Hey Lukas

On 24.10.2023 08:33, Lukas Funke wrote:


- I placed the correct URL into SRC_URI, but do_go_vendor still 
failed with following stacktrace:


File: 
'/home/uvv/projects/yocto-lorch-mapro/openembedded-core/meta/classes/go-vendor.bbclass', lineno: 86, function: do_go_vendor

  0082:    # path = github.com/foo/bar
  0083:    # version = v1.2.3
  0084:
  0085:    p = destsuffix[len(default_destsuffix)+1:]
  *** 0086:    path, version = p.split('@')
  0087:
  0088:    subdir = fetcher.ud[url].parm.get('go_subdir')
  0089:    subdir = "" if not subdir else subdir
  0090:
Exception: ValueError: not enough values to unpack (expected 2, got 1)

The reason is that my go.mod name does not have a version component. 
If I understood the convention https://go.dev/ref/mod#introduction, 
it's not a required component, so this should be taken into account.


This error could happen if your dependencies don't have a version. 
I've never seen this in my experiments. Maybe check your go.mod file 
for the missing version info. 


I debugged it a bit and see that the error is actually caused by my URL 
modification. The URL that works for me looks like

SRC_URI = git://git@${GO_IMPORT}.git;...

The parsing expects the version after "@", which is not right anymore.


The problem here is to distiguish between the actual project SRC_URI and 
it's dependencies. This is currently done by comparing the SRC_URI entry 
to the GO_IMPORT variable. If they match then it's not a dependency. But 
you are correct: this can be solved in a more general manner. Good 
catch. I'll try to fix it in the next version.




Slava



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189640): 
https://lists.openembedded.org/g/openembedded-core/message/189640
Mute This Topic: https://lists.openembedded.org/mt/102017388/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-Core][PATCH v2 0/4] recipetool: Add handler to create go recipes

2023-10-23 Thread Lukas Funke 

On 23.10.2023 20:06, Vyacheslav Yurkov wrote:

On 23.10.2023 14:18, Lukas Funke wrote:

Hi Slava,

On 22.10.2023 20:34, Vyacheslav Yurkov wrote:

Hey Lukas,
Thanks a lot for the patch. A few questions/comments from my initial 
test below.


- I tried it with a go-based backend I have by providing ssh URL to 
github. It seems like the GO_IMPORT is set to a module name from 
go.mod of my project, which of course fails to fetch it like that, 
because it's not a valid URL. How is it supposed to be used?


Your assumption is correct: the GO_IMPORT is taken from the go.mod file.

I've not considered the case where the repo URL is unequal to the 
module path. This may require manual rework of the recipe. Another 
solution would be to take the source URL from the recipetool. I think 
there is no correct solution to this problem, because probably most 
people might want to have the original solution, since they are 
creating recipes from oss components (i.e. from github).


Some more results from my tests.

- I refactored module name to contain a valid URL... It seems though 
that current version of go_src_uri does not handle ssh URLs, and all 
required info from URL was lost (git@ component, ssh protocol, .git 
suffix).


Currently only https is handled.



- I placed the correct URL into SRC_URI, but do_go_vendor still failed 
with following stacktrace:


File: 
'/home/uvv/projects/yocto-lorch-mapro/openembedded-core/meta/classes/go-vendor.bbclass', lineno: 86, function: do_go_vendor

  0082:    # path = github.com/foo/bar
  0083:    # version = v1.2.3
  0084:
  0085:    p = destsuffix[len(default_destsuffix)+1:]
  *** 0086:    path, version = p.split('@')
  0087:
  0088:    subdir = fetcher.ud[url].parm.get('go_subdir')
  0089:    subdir = "" if not subdir else subdir
  0090:
Exception: ValueError: not enough values to unpack (expected 2, got 1)

The reason is that my go.mod name does not have a version component. If 
I understood the convention https://go.dev/ref/mod#introduction, it's 
not a required component, so this should be taken into account.


This error could happen if your dependencies don't have a version. I've 
never seen this in my experiments. Maybe check your go.mod file for the 
missing version info.




Thanks,
Slava



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189637): 
https://lists.openembedded.org/g/openembedded-core/message/189637
Mute This Topic: https://lists.openembedded.org/mt/102017388/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-Core][PATCH v2 0/4] recipetool: Add handler to create go recipes

2023-10-23 Thread Lukas Funke 

Hi Slava,

On 23.10.2023 19:05, Vyacheslav Yurkov wrote:

On 23.10.2023 14:18, Lukas Funke wrote:

Hi Slava,

On 22.10.2023 20:34, Vyacheslav Yurkov wrote:

Hey Lukas,
Thanks a lot for the patch. A few questions/comments from my initial 
test below.


- I tried it with a go-based backend I have by providing ssh URL to 
github. It seems like the GO_IMPORT is set to a module name from 
go.mod of my project, which of course fails to fetch it like that, 
because it's not a valid URL. How is it supposed to be used?


Your assumption is correct: the GO_IMPORT is taken from the go.mod file.

I've not considered the case where the repo URL is unequal to the 
module path. This may require manual rework of the recipe. Another 
solution would be to take the source URL from the recipetool. I think 
there is no correct solution to this problem, because probably most 
people might want to have the original solution, since they are 
creating recipes from oss components (i.e. from github).


Ah, it could be that module name is not used correctly in our backend. 
I'll try to put a proper URL there, thanks.





The file will explode anyway. Thanks 'go'.

Adding more intelligence to the underlying license detection could be 
an idea: in your first example, the license file contains actually two 
licenses: MIT and Apache. Yocto has no way to detect those cases.


Yeah, I only mentioned 2 out of 20 in my email to not make it huge ;) 
There are indeed around 20. I'll try to prepare the list and send the 
patch to include them.






- Could please clarify where does the version from go.mod hide? Is it 
taken directly from go.mod? I'm trying to understand what should be 
the workflow when a module version should be bumped up in the go.mod. 
Will that be reflected in the recipe in any way?


No, currently the go-version doesn't play a role ATM. Except one case 
when you have a go.mod file with go < 1.17. These go.mod files don't 
include indirect dependencies.




Still trying to wrap my head around... When there's no version at 
parsing stage, how this will affect reproducibility? If it's not known, 
then whenever the version is bumped up in go.mod, a manual 'clean all' 
will be required? (It's probably the same as now though).


Maybe I don't understand the problem: Is it required for the go module 
to have the *same* version as the golang package in yocto? In my 
understanding, when the golang version is greater-equal to the go.mod 
version we're good?




Thanks,
Slava



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189636): 
https://lists.openembedded.org/g/openembedded-core/message/189636
Mute This Topic: https://lists.openembedded.org/mt/102017388/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-Core][PATCH v2 0/4] recipetool: Add handler to create go recipes

2023-10-23 Thread Lukas Funke 

Hi Slava,

On 22.10.2023 20:34, Vyacheslav Yurkov wrote:

Hey Lukas,
Thanks a lot for the patch. A few questions/comments from my initial 
test below.


- I tried it with a go-based backend I have by providing ssh URL to 
github. It seems like the GO_IMPORT is set to a module name from go.mod 
of my project, which of course fails to fetch it like that, because it's 
not a valid URL. How is it supposed to be used?


Your assumption is correct: the GO_IMPORT is taken from the go.mod file.

I've not considered the case where the repo URL is unequal to the module 
path. This may require manual rework of the recipe. Another solution 
would be to take the source URL from the recipetool. I think there is no 
correct solution to this problem, because probably most people might 
want to have the original solution, since they are creating recipes from 
oss components (i.e. from github).



- I've got about 20 lines like:
INFO: Please add the following line for 
'vendor/google.golang.org/protobuf/LICENSE' to a 
'lib/recipetool/licenses.csv' and replace `Unknown` with the license:

02d4002e9171d41a8fad93aa7faf3956,Unknown
INFO: Please add the following line for 
'vendor/gopkg.in/yaml.v3/LICENSE' to a 'lib/recipetool/licenses.csv' and 
replace `Unknown` with the license:

3c91c17266710e16afdbb2b6d15c761c,Unknown

I believe I have to go through all of them manually and set a proper 
license in the recipe. Still the questions arises, what should be a 
better way to reduce amount of this work? Adding more hashes/licenses to 
lib/recipetool/licenses.csv would be a way to go? I'm just afraid that 
file might explode if we use it like that...


The file will explode anyway. Thanks 'go'.

Adding more intelligence to the underlying license detection could be an 
idea: in your first example, the license file contains actually two 
licenses: MIT and Apache. Yocto has no way to detect those cases.




- Could please clarify where does the version from go.mod hide? Is it 
taken directly from go.mod? I'm trying to understand what should be the 
workflow when a module version should be bumped up in the go.mod. Will 
that be reflected in the recipe in any way?


No, currently the go-version doesn't play a role ATM. Except one case 
when you have a go.mod file with go < 1.17. These go.mod files don't 
include indirect dependencies.


I hope you might find this helpful and thanks for testing!

Cheers,
Lukas



Thanks,
Slava

On 17.10.2023 15:26, lukas.funke-...@weidmueller.com wrote:

From: Lukas Funke 

This patch series adds a recipetool handler in order to create 'go' 
recipes.

Each recipe contains a list of dependencies in their SRC_URI
variable which are derived from the projects `go.mod` file. For each
dependency the corresponding license file uri/hash is added.

The recipe may not work ad-hoc, but is a good starting point to create
a working recipe and have a working offline-build.

Lukas Funke (4):
   classes: go-vendor: Add go-vendor class
   selftest: recipetool: Add test for go recipe handler
   recipetool: Ignore *.go files while scanning for licenses
   recipetool: Add handler to create go recipes

  meta/classes/go-vendor.bbclass | 135 
  meta/lib/oeqa/selftest/cases/recipetool.py | 163 +
  scripts/lib/recipetool/create.py   |   2 +-
  scripts/lib/recipetool/create_go.py    | 730 +
  4 files changed, 1029 insertions(+), 1 deletion(-)
  create mode 100644 meta/classes/go-vendor.bbclass
  create mode 100644 scripts/lib/recipetool/create_go.py






-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189623): 
https://lists.openembedded.org/g/openembedded-core/message/189623
Mute This Topic: https://lists.openembedded.org/mt/102017388/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-Core][PATCH v2 4/4] recipetool: Add handler to create go recipes

2023-10-17 Thread Lukas Funke 

Hi Richard,

On 17.10.2023 15:53, Richard Purdie wrote:

On Tue, 2023-10-17 at 15:26 +0200, Lukas Funke wrote:

From: Lukas Funke 

Signed-off-by: Lukas Funke 
---
  scripts/lib/recipetool/create_go.py | 730 
  1 file changed, 730 insertions(+)
  create mode 100644 scripts/lib/recipetool/create_go.py

diff --git a/scripts/lib/recipetool/create_go.py 
b/scripts/lib/recipetool/create_go.py
new file mode 100644
index 00..e0254f111b
--- /dev/null
+++ b/scripts/lib/recipetool/create_go.py
@@ -0,0 +1,730 @@
+# Recipe creation tool - go support plugin
+#
+# Copyright (C) 2023 Weidmueller GmbH & Co KG
+# Author: Lukas Funke 
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (c) 2009 The Go Authors. All rights reserved.
+#
+#  SPDX-License-Identifier: BSD-3-Clause
+#


Can you clarify what this license information means please? Two
different license identifier lines seems rather confusing and
problematic.

I've not looked into the rest of the patches yet, this just caught my
eye.


Some of the ideas/code was ported from the original golang code to 
python here. Thus, I had to copy the license information as well (I 
guess?). If this is wrong or could be written in another way please 
provide an example how it's done.


Best regards,
Lukas



Cheers,

Richard




-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189336): 
https://lists.openembedded.org/g/openembedded-core/message/189336
Mute This Topic: https://lists.openembedded.org/mt/102017392/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-Core][PATCH v2 2/4] selftest: recipetool: Add test for go recipe handler

2023-10-17 Thread Lukas Funke 
From: Lukas Funke 

This commit adds a test for the go recipetool handler. The choosen go
project to test the created recipe was picked randomly. The SRC_URIs and
the LIC_FILES_CHKSUMs are checked against there reference values.

Signed-off-by: Lukas Funke 
---
 meta/lib/oeqa/selftest/cases/recipetool.py | 163 +
 1 file changed, 163 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py 
b/meta/lib/oeqa/selftest/cases/recipetool.py
index 48661bee6f..6f3d5df50c 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -532,6 +532,169 @@ class RecipetoolTests(RecipetoolBase):
 libpath = os.path.join(get_bb_var('COREBASE'), 'scripts', 'lib', 
'recipetool')
 sys.path.insert(0, libpath)
 
+def test_recipetool_create_go(self):
+# Basic test to check go recipe generation
+def urifiy(url, version, modulepath = None, pathmajor = None, subdir = 
None):
+modulepath = ",path='%s'" % modulepath if len(modulepath) else ''
+pathmajor = ",pathmajor='%s'" % pathmajor if len(pathmajor) else ''
+subdir = ",subdir='%s'" % subdir if len(subdir) else ''
+return "${@go_src_uri('%s','%s'%s%s%s)}" % (url, version, 
modulepath, pathmajor, subdir)
+
+temprecipe = os.path.join(self.tempdir, 'recipe')
+os.makedirs(temprecipe)
+recipefile = os.path.join(temprecipe, 'edgex-go_git.bb')
+srcuri = 'https://github.com/edgexfoundry/edgex-go.git'
+srcrev = "v3.0.0"
+result = runCmd('recipetool create -o %s %s -S %s' % (temprecipe, 
srcuri, srcrev))
+self.assertTrue(os.path.isfile(recipefile))
+checkvars = {}
+src_uri = 
{'git://${GO_IMPORT};destsuffix=git/src/${GO_IMPORT};nobranch=1;name=${BPN};protocol=https',
+   'file://modules.txt'}
+checkvars['LIC_FILES_CHKSUM'] = set(
+
['file://src/${GO_IMPORT}/LICENSE;md5=8f8bc924cf73f6a32381e5fd4c58d603',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/Microsoft/go-winio/LICENSE;md5=69205ff73858f2c22b2ca135b557e8ef',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/armon/go-metrics/LICENSE;md5=d2d77030c0183e3d1e66d26dc1f243be',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/cenkalti/backoff/LICENSE;md5=1571d94433e3f3aa05267efd4dbea68b',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/davecgh/go-spew/LICENSE;md5=c06795ed54b2a35ebeeb543cd3a73e56',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/eclipse/paho.mqtt.golang/LICENSE;md5=dcdb33474b60c38efd27356d8f2edec7',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/eclipse/paho.mqtt.golang/edl-v10;md5=3adfcc70f5aeb7a44f3f9b495aa1fbf3',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/edgexfoundry/go-mod-bootstrap/v3/LICENSE;md5=0d6dae39976133b2851fba4c1e1275ff',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/edgexfoundry/go-mod-configuration/v3/LICENSE;md5=0d6dae39976133b2851fba4c1e1275ff',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/edgexfoundry/go-mod-core-contracts/v3/LICENSE;md5=0d6dae39976133b2851fba4c1e1275ff',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/edgexfoundry/go-mod-messaging/v3/LICENSE;md5=0d6dae39976133b2851fba4c1e1275ff',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/edgexfoundry/go-mod-registry/v3/LICENSE;md5=0d6dae39976133b2851fba4c1e1275ff',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/edgexfoundry/go-mod-secrets/v3/LICENSE;md5=f9fa2f4f8e0ef8cc7b5dd150963eb457',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/fatih/color/LICENSE.md;md5=316e6d590bdcde7993fb175662c0dd5a',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/fxamacker/cbor/v2/LICENSE;md5=827f5a2fa861382d35a3943adf9ebb86',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/go-jose/go-jose/v3/LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/go-jose/go-jose/v3/json/LICENSE;md5=591778525c869cdde0ab5a1bf283cd81',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/go-kit/log/LICENSE;md5=5b7c15ad5fffe2ff6e9d58a6c161f082',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/go-logfmt/logfmt/LICENSE;md5=98e39517c38127f969de33057067091e',
+ 
'file://src/${GO_IMPORT}/vendor/github.com/go-playground/locales/LICENSE;md5=3ccbda375ee345400ad1da85ba522301',
+

[OE-Core][PATCH v2 4/4] recipetool: Add handler to create go recipes

2023-10-17 Thread Lukas Funke 
From: Lukas Funke 

Signed-off-by: Lukas Funke 
---
 scripts/lib/recipetool/create_go.py | 730 
 1 file changed, 730 insertions(+)
 create mode 100644 scripts/lib/recipetool/create_go.py

diff --git a/scripts/lib/recipetool/create_go.py 
b/scripts/lib/recipetool/create_go.py
new file mode 100644
index 00..e0254f111b
--- /dev/null
+++ b/scripts/lib/recipetool/create_go.py
@@ -0,0 +1,730 @@
+# Recipe creation tool - go support plugin
+#
+# Copyright (C) 2023 Weidmueller GmbH & Co KG
+# Author: Lukas Funke 
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (c) 2009 The Go Authors. All rights reserved.
+#
+#  SPDX-License-Identifier: BSD-3-Clause
+#
+
+
+from collections import namedtuple
+from enum import Enum
+from html.parser import HTMLParser
+from recipetool.create import RecipeHandler, handle_license_vars
+from recipetool.create import guess_license, tidy_licenses, fixup_license
+from urllib.error import URLError
+
+import bb.utils
+import json
+import logging
+import os
+import re
+import subprocess
+import sys
+import shutil
+import tempfile
+import urllib.parse
+import urllib.request
+
+
+GoImport = namedtuple('GoImport', 'root vcs url suffix')
+logger = logging.getLogger('recipetool')
+CodeRepo = namedtuple(
+'CodeRepo', 'path codeRoot codeDir pathMajor pathPrefix pseudoMajor')
+
+tinfoil = None
+
+# Regular expression to parse pseudo semantic version
+# see https://go.dev/ref/mod#pseudo-versions
+re_pseudo_semver = re.compile(
+
r"^v[0-9]+\.(0\.0-|\d+\.\d+-([^+]*\.)?0\.)(?P\d{14})-(?P[A-Za-z0-9]+)(\+[0-9A-Za-z-]+(\.[0-9A-Za-z-]+)*)?$")
+# Regular expression to parse semantic version
+re_semver = re.compile(
+
r"^v(?P0|[1-9]\d*)\.(?P0|[1-9]\d*)\.(?P0|[1-9]\d*)(?:-(?P(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$")
+
+
+def tinfoil_init(instance):
+global tinfoil
+tinfoil = instance
+
+
+class GoRecipeHandler(RecipeHandler):
+"""Class to handle the go recipe creation"""
+
+@staticmethod
+def __ensure_go():
+"""Check if the 'go' command is available in the recipes"""
+recipe = "go-native"
+if not tinfoil.recipes_parsed:
+tinfoil.parse_recipes()
+try:
+rd = tinfoil.parse_recipe(recipe)
+except bb.providers.NoProvider:
+bb.error(
+"Nothing provides '%s' which is required for the build" % 
(recipe))
+bb.note(
+"You will likely need to add a layer that provides '%s'" % 
(recipe))
+return None
+
+bindir = rd.getVar('STAGING_BINDIR_NATIVE')
+gopath = os.path.join(bindir, 'go')
+
+if not os.path.exists(gopath):
+tinfoil.build_targets(recipe, 'addto_recipe_sysroot')
+
+if not os.path.exists(gopath):
+logger.error(
+'%s required to process specified source, but %s did not 
seem to populate it' % 'go', recipe)
+return None
+
+return bindir
+
+def __resolve_repository_static(self, modulepath):
+"""Resolve the repository in a static manner
+
+The method is based on the go implementation of
+`repoRootFromVCSPaths` in
+
https://github.com/golang/go/blob/master/src/cmd/go/internal/vcs/vcs.go
+"""
+
+url = urllib.parse.urlparse("https://"; + modulepath)
+req = urllib.request.Request(url.geturl())
+
+try:
+resp = urllib.request.urlopen(req)
+# Some modulepath are just redirects to github (or some other vcs
+# hoster). Therefore, we check if this modulepath redirects to
+# somewhere else
+if resp.geturl() != url.geturl():
+bb.debug(1, "%s is redirectred to %s" %
+ (url.geturl(), resp.geturl()))
+url = urllib.parse.urlparse(resp.geturl())
+modulepath = url.netloc + url.path
+
+except URLError as url_err:
+# This is probably because the module path
+# contains the subdir and major path. Thus,
+# we ignore this error for now
+logger.debug(
+1, "Failed to fetch page from [%s]: %s" % (url, str(url_err)))
+
+host, _, _ = modulepath.partition('/')
+
+class vcs(Enum):
+pathprefix = "pathprefix"
+regexp = "regexp"
+type = "type"
+repo = "repo"
+check = "check"
+schemelessRepo = "scheme

[OE-Core][PATCH v2 1/4] classes: go-vendor: Add go-vendor class

2023-10-17 Thread Lukas Funke 
From: Lukas Funke 

Signed-off-by: Lukas Funke 
---
 meta/classes/go-vendor.bbclass | 135 +
 1 file changed, 135 insertions(+)
 create mode 100644 meta/classes/go-vendor.bbclass

diff --git a/meta/classes/go-vendor.bbclass b/meta/classes/go-vendor.bbclass
new file mode 100644
index 00..13f1b8b2be
--- /dev/null
+++ b/meta/classes/go-vendor.bbclass
@@ -0,0 +1,135 @@
+#
+# Copyright 2023 (C) Weidmueller GmbH & Co KG
+# Author: Lukas Funke 
+#
+# Handle Go vendor support for offline builds
+#
+# When importing Go modules, Go downloads the imported module using
+# a network (proxy) connection ahead of the compile stage. This contradicts 
+# the yocto build concept of fetching every source ahead of build-time
+# and supporting offline builds.
+#
+# To support offline builds, we use Go 'vendoring': module dependencies are 
+# downloaded during the fetch-phase and unpacked into the modules 'vendor'
+# folder. Additinally a manifest file is generated for the 'vendor' folder
+# 
+
+inherit go-mod
+
+def go_src_uri(repo, version, path=None, subdir=None, \
+vcs='git', replaces=None, pathmajor=None):
+
+destsuffix = "git/src/import/vendor.fetch"
+go_module_path = repo if not path else path
+
+src_uri = "{}://{}" \
+  ";name={}" \
+  "".format(vcs, repo, \
+go_module_path.replace('/', '.'))
+
+src_uri += ";destsuffix={}/{}@{}".format(destsuffix, \
+go_module_path, \
+version)
+
+if vcs == "git":
+src_uri += ";nobranch=1;protocol=https"
+if replaces:
+src_uri += ";go_module_replacement={}".format(replaces)
+if subdir:
+src_uri += ";go_subdir={}".format(subdir)
+if pathmajor:
+src_uri += ";go_pathmajor={}".format(pathmajor)
+
+return src_uri
+
+
+python do_go_vendor() {
+import shutil
+
+src_uri = (d.getVar('SRC_URI') or "").split()
+
+if len(src_uri) == 0:
+bb.error("SRC_URI is empty")
+return
+
+default_destsuffix = "git/src/import/vendor.fetch"
+fetcher = bb.fetch2.Fetch(src_uri, d)
+go_import = d.getVar('GO_IMPORT')
+source_dir = d.getVar('S')
+
+vendor_dir = os.path.join(source_dir, *['src', go_import, 'vendor'])
+import_dir = os.path.join(source_dir, *['src', 'import', 'vendor.fetch'])
+
+bb.utils.mkdirhier(vendor_dir)
+modules = {}
+
+for url in fetcher.urls:
+srcuri = fetcher.ud[url].host + fetcher.ud[url].path
+
+# Skip main module for which the recipe is actually created
+if srcuri == go_import:
+continue
+
+# Skip local files
+if fetcher.ud[url].type == 'file':
+continue
+
+destsuffix = fetcher.ud[url].parm.get('destsuffix')
+# We derive the module path / version in the following manner 
(exmaple):
+# 
+# destsuffix = git/src/import/vendor.fetch/github.com/foo/bar@v1.2.3
+# p = github.com/foo/bar@v1.2.3
+# path = github.com/foo/bar
+# version = v1.2.3
+
+p = destsuffix[len(default_destsuffix)+1:]
+path, version = p.split('@')
+
+subdir = fetcher.ud[url].parm.get('go_subdir')
+subdir = "" if not subdir else subdir
+
+pathMajor = fetcher.ud[url].parm.get('go_pathmajor')
+pathMajor = "" if not pathMajor else pathMajor
+
+base = path[:-(len(subdir)+len(pathMajor))-1]
+r = fetcher.ud[url].parm.get('go_module_replacement')
+
+if not path in modules and not r:
+modules[path] =   {
+"version": version,
+"src": os.path.join(import_dir, *[p, subdir]),
+"subdir": subdir,
+"pathMajor": pathMajor,
+  }
+
+for module_key in sorted(modules):
+
+# only take the version which is explicitly listed
+# as a dependency in the go.mod
+module = modules[module_key]
+src = module['src']
+
+# If the module is released at major version 2 or higher, the module
+# path must end with a major version suffix like /v2.
+# This may or may not be part of the subdirectory name
+#
+# https://go.dev/ref/mod#modules-overview
+srcMajorVersion = os.path.join(src, module['pathMajor'].strip('/'))
+if os.path.exists(srcMajorVe

[OE-Core][PATCH v2 0/4] recipetool: Add handler to create go recipes

2023-10-17 Thread Lukas Funke 
From: Lukas Funke 

This patch series adds a recipetool handler in order to create 'go' recipes.
Each recipe contains a list of dependencies in their SRC_URI
variable which are derived from the projects `go.mod` file. For each
dependency the corresponding license file uri/hash is added.

The recipe may not work ad-hoc, but is a good starting point to create
a working recipe and have a working offline-build.

Lukas Funke (4):
  classes: go-vendor: Add go-vendor class
  selftest: recipetool: Add test for go recipe handler
  recipetool: Ignore *.go files while scanning for licenses
  recipetool: Add handler to create go recipes

 meta/classes/go-vendor.bbclass | 135 
 meta/lib/oeqa/selftest/cases/recipetool.py | 163 +
 scripts/lib/recipetool/create.py   |   2 +-
 scripts/lib/recipetool/create_go.py| 730 +
 4 files changed, 1029 insertions(+), 1 deletion(-)
 create mode 100644 meta/classes/go-vendor.bbclass
 create mode 100644 scripts/lib/recipetool/create_go.py

-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189326): 
https://lists.openembedded.org/g/openembedded-core/message/189326
Mute This Topic: https://lists.openembedded.org/mt/102017388/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-Core][PATCH v2 3/4] recipetool: Ignore *.go files while scanning for licenses

2023-10-17 Thread Lukas Funke 
From: Lukas Funke 

Signed-off-by: Lukas Funke 
---
 scripts/lib/recipetool/create.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 143bc63e9d..293198d1c8 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1212,7 +1212,7 @@ def guess_license(srctree, d):
 
 licenses = []
 licspecs = ['*LICEN[CS]E*', 'COPYING*', '*[Ll]icense*', 'LEGAL*', 
'[Ll]egal*', '*GPL*', 'README.lic*', 'COPYRIGHT*', '[Cc]opyright*', 
'e[dp]l-v10']
-skip_extensions = (".html", ".js", ".json", ".svg", ".ts")
+skip_extensions = (".html", ".js", ".json", ".svg", ".ts", ".go")
 licfiles = []
 for root, dirs, files in os.walk(srctree):
 for fn in files:
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189327): 
https://lists.openembedded.org/g/openembedded-core/message/189327
Mute This Topic: https://lists.openembedded.org/mt/102017389/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [langdale][kirkstone][PATCH] go-mod.bbclass: Allow network in do_compile

2023-09-27 Thread Lukas Funke 

On 10.09.2023 23:26, Martin Jansa wrote:

On Sun, Mar 12, 2023 at 10:15 AM Martin Jansa via lists.openembedded.org
 wrote:


On Tue, Jan 3, 2023 at 5:03 PM Lukas Funke <
lukas.funke-...@weidmueller.com> wrote:


Martin,

first of all: thank you for testing the patches. As usual the corner
cases are the most tricky ones.

tl;dr: I'm working on an updated patch series to address the reported
issues.



Hi Lukas, Stefan,

and update on this series?

It doesn't have to be perfect, other people can help with remaining corner
cases and right now there is nothing for this in oe-core, so any version
will be big improvement :) and it will make it easier for others to submit
incremental improvements - I have some as well, but haven't sent them as I
don't know what you've already updated locally, so I'm waiting for v2.

Mickledore is already in feature freeze, but maybe RP will make an
exception as this is new important functionality which isn't likely to
break other existing code.



Hello Lukas,

any progress on updated patch series? Nanbield release is near and I don't
remember seeing the updated patch series in last couple months.

Is there anything I can do to help? The last version looked reasonably well
and IMHO could be used as starting point for everybody to help fixing those
corner cases where it didn't work well.

Regards,


Hello Martin,

I'm currently working on an update, but can't make any promises 
regarding the timeline.


Thanks for offering help! Reviewing and testing is always helpful when 
the updated series is on it's way.


Regards
Lukas






Kind Regards,


I've looked into the issues and would like to give some explanation.


1) the first module you mention ('gonum.org/v1/gonum
<http://gonum.org/v1/gonum>') does not obey the go documentation for
resolving the modules source-code repository (see
https://go.dev/ref/mod#go-mod-download , Section 'Version control
systems'). The documentation states that there should be a html page,
queried by "?go-get=1", with a -tag which contains the original
URL to the source-code repository. For 'gonum' there is only a 404-page.
This page contains javascript-code that redirects to the actual gonum
package page. Unnecessary hard to process. I'll open an issue for that.
The second module 'code.cloudfoundry.org/clock
<http://code.cloudfoundry.org/clock>' has a broken certificate. I
already opend an issue on github for this.

I addressed this issue above by maintaining a list of modules and their
actual repo URLs inside the go-recipetool as an absolute fallback. Would
do you think about this solution?



Yes, that's what I was starting to do in:

https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=b973c7f17c8a613233a1a18de0bf6daa352c47d8

https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=61c6c9962b3b7d4c1cf6e6823de6c32b09998b00



2) yes, my bad :)



as work around I was using

https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=5767819bfe96de29ab751d4a37a431a324f7e547


3) The problem is that some repositories have a prefix in their tags.

The tags usually only contain the semantic version string. I'm curretly
working on this issue.

Best regards

Lukas

Am 22.12.2022 um 20:08 schrieb Martin Jansa via lists.openembedded.org:

On Thu, Dec 22, 2022 at 7:39 PM Vyacheslav Yurkov 
wrote:

 On 22.12.2022 17:48, Bruce Ashfield wrote:
 > On Thu, Dec 22, 2022 at 11:20 AM Richard Purdie
 >  wrote:
 >> This patch is not in master and is not a backport therefore not
 >> eligible for kirkstone/langdale.

 My bad, the intention was to have it in all three branches: master,
 kirkstone, and langdale.

 > 'nor should it go to master.
 >
 > If someone wants to allow go to fetch sources during builds, it
 needs
 > to be done in their own layers.
 >
 > Bruce

 Probably I misunderstood the outcome of the discussion in my first
 link.
 Bruce and Richard, you both suggested to have it in go-mod with the
 warning. I agree that it breaks reproducibility, and is considered
 a bad
 practice in general (I'm aware of a few more build systems with
 the same
 shortcoming).

 My point is, until a proper solution is in place, it should be at
 least
 documented somewhere that this should be done in own layer/recipe.
 Any
 suggestions where this can be documented if not in OE-Core?


Have you tried the changes submitted by Lukas/Stefan in:
https://lists.openembedded.org/g/openembedded-architecture/message/1539
?

It's not perfect, I was testing it on
https://github.com/influxdata/telegraf/blob/master/go.mod and I've
found some corner cases where it failed.

But it seems like very good start and

Re: [OE-core] [langdale][kirkstone][PATCH] go-mod.bbclass: Allow network in do_compile

2023-01-03 Thread Lukas Funke 

Martin,

first of all: thank you for testing the patches. As usual the corner 
cases are the most tricky ones.


tl;dr: I'm working on an updated patch series to address the reported 
issues.



I've looked into the issues and would like to give some explanation.

1) the first module you mention ('gonum.org/v1/gonum 
') does not obey the go documentation for 
resolving the modules source-code repository (see 
https://go.dev/ref/mod#go-mod-download , Section 'Version control 
systems'). The documentation states that there should be a html page, 
queried by "?go-get=1", with a -tag which contains the original 
URL to the source-code repository. For 'gonum' there is only a 404-page. 
This page contains javascript-code that redirects to the actual gonum 
package page. Unnecessary hard to process. I'll open an issue for that. 
The second module 'code.cloudfoundry.org/clock 
' has a broken certificate. I 
already opend an issue on github for this.


I addressed this issue above by maintaining a list of modules and their 
actual repo URLs inside the go-recipetool as an absolute fallback. Would 
do you think about this solution?


2) yes, my bad :)

3) The problem is that some repositories have a prefix in their tags. 
The tags usually only contain the semantic version string. I'm curretly 
working on this issue.


Best regards

Lukas

Am 22.12.2022 um 20:08 schrieb Martin Jansa via lists.openembedded.org:
On Thu, Dec 22, 2022 at 7:39 PM Vyacheslav Yurkov  
wrote:


On 22.12.2022 17:48, Bruce Ashfield wrote:
> On Thu, Dec 22, 2022 at 11:20 AM Richard Purdie
>  wrote:
>> This patch is not in master and is not a backport therefore not
>> eligible for kirkstone/langdale.

My bad, the intention was to have it in all three branches: master,
kirkstone, and langdale.

> 'nor should it go to master.
>
> If someone wants to allow go to fetch sources during builds, it
needs
> to be done in their own layers.
>
> Bruce

Probably I misunderstood the outcome of the discussion in my first
link.
Bruce and Richard, you both suggested to have it in go-mod with the
warning. I agree that it breaks reproducibility, and is considered
a bad
practice in general (I'm aware of a few more build systems with
the same
shortcoming).

My point is, until a proper solution is in place, it should be at
least
documented somewhere that this should be done in own layer/recipe.
Any
suggestions where this can be documented if not in OE-Core?


Have you tried the changes submitted by Lukas/Stefan in:
https://lists.openembedded.org/g/openembedded-architecture/message/1539
?

It's not perfect, I was testing it on 
https://github.com/influxdata/telegraf/blob/master/go.mod and I've 
found some corner cases where it failed.


But it seems like very good start and we should work with Lukas/Stefan 
to get it merged in master. Then all branches could consume recipes 
created in master and only the exceptions would need to have network 
access (instead of the bbclass enabling it for every go-mod user - 
go-vendor inherits go-mod as well).


Lukas/Stefan: is there something we can do to help with v2 of your 
patches?


I wanted to submit better review on submitted patches after more 
debugging, but here it is:


1) to resolve 2 deps:
{'Path': 'gonum.org/v1/gonum ', 'Version': 
'v0.12.0'} due to 404 "ERROR: Error while fetching redirect page: HTTP 
Error 404: Not Found"


{'Path': 'code.cloudfoundry.org/clock 
', 'Version': 'v1.0.0', 
'Indirect': True} due to this being redirect to github.com 
 and failing with "ERROR: Error while fetching 
redirect page: certificate verify failed: Hostname mismatch, certificate is not valid 
for 'code.cloudfoundry.org '. (_ssl.c:996)>"

parse

2) some issues with LICENSEs:
INFO: Please add the following line for 
'docs/LICENSE_OF_DEPENDENCIES.md' to a 'lib/recipetool/licenses.csv' 
and replace `Unknown` with the license:

c4807c5b058a56c02f592207b4e056b1,Unknown
Traceback (most recent call last):
  File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line 
111, in 

    ret = main()
          ^^
  File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line 
100, in main

    ret = args.func(args)
          ^^^
  File 
"/OE/layers/openembedded-core/scripts/lib/recipetool/create.py", line 
746, in create_recipe
    handler.process(srctree_use, classes, lines_before, lines_after, 
handled, extravalues)
  File 
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", 
line 349, in process

    self._rewrite_lic_uri(lines_before)
  File 
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", 
line 380, in _rewrite_lic_uri
    updated, newlines = bb.utils.edit_metadata(lines_before, 
['LIC_FILES_CHKS

Re: [OE-core] [PATCH 1/1] go.bbclass: Allow network in do_compile

2022-03-11 Thread lukas . funke 
On Wed, Mar 9, 2022 at 07:43 PM, Bruce Ashfield wrote:

> 
> On Wed, Mar 9, 2022 at 3:10 AM  wrote:
> 
>> On Thu, Mar 3, 2022 at 05:46 PM, Bruce Ashfield wrote:
>> 
>> On Thu, Mar 3, 2022 at 10:13 AM  wrote:
>> 
>> On Thu, Mar 3, 2022 at 04:34 AM, Bruce Ashfield wrote:
>> 
>> On Wed, Mar 2, 2022 at 4:57 PM Andrei Gherzan  wrote:
>> 
>> 
>> Mar 1, 2022 20:15:52 Bruce Ashfield :
>> 
>> On Tue, Mar 1, 2022 at 10:54 AM  wrote:
>> 
>> On Tue, Mar 1, 2022 at 02:14 PM, Bruce Ashfield wrote:
>> 
>> On Tue, Mar 1, 2022 at 6:42 AM Andrei Gherzan  wrote:
>> 
>> 
>> On Tue, 1 Mar 2022, at 01:55, Bruce Ashfield wrote:
>> 
>> On Mon, Feb 28, 2022 at 8:17 PM Bruce Ashfield via
>> lists.openembedded.org
>>  wrote:
>> 
>> 
>> On Mon, Feb 28, 2022 at 6:54 PM Andrei Gherzan  wrote:
>> 
>> 
>> 
>> From: Andrei Gherzan 
>> 
>> Compile pulls in the go.mod list requiring network. Without this, do
>> compile would fail with a similar error to the following:
>> 
>> dial tcp: lookup proxy.golang.org: Temporary failure in name resolution
>> 
>> This is something that needs to be carried in your own layers, IMHO it
>> isn't appropriate for core.
>> 
>> It isn't about the fetching, it is the entire gap in functionality
>> that we are missing if go starts fetching dependencies during compile.
>> 
>> A further thought is that if this is for go.mod issues, there is the
>> go-mod.bbclass.
>> 
>> Perhaps enabling it in that class and doing a bbwarn about go fetching
>> dependencies would be appropriate ?
>> 
>> Otherwise, someone may not know that this is happening and that a no
>> network configuration has no chance of working.
>> 
>> I reckon that is reasonable. I'll personally go down the recipe level to
>> workaround this change but understanding and agreeing with the reasoning
>> behind this change, I want to invest a bit into trying to find a proper
>> solution in the core. Bruce, I know you invested a fair amount of time
>> into this already. Would you be willing to sync up and see how we can work
>> together in tackling this?
>> 
>> Definitely, more ideas are good. In fact, I think there are probably
>> several approaches that can co-exist, depending on what a
>> recipe/developer needs.
>> 
>> I'm in the Eastern time zone here, and will try and grab folks on IRC
>> to have a level set
>> 
>> Bruce
>> 
>> Added Zyga to CC as he is also interested in this as part of his go
>> development activities.
>> 
>> Thanks,
>> Andrei
>> 
>> 
>> 
>> --
>> - Thou shalt not follow the NULL pointer, for chaos and madness await
>> thee at its end
>> - "Use the force Harry" - Gandalf, Star Trek II
>> 
>> The problem in allowing downloads during compile (e.g. by go) is, that it
>> leads to non-reproducable builds. I'm currently facing the same issue and
>> would like to have a reproducable go *offline* build.
>> I would like to propose two ideas to workaround the go-compile fetching
>> issue:
>> 
>> First:
>> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by
>> writing a seperate go fetcher or a wget-fetcher) and unpack the
>> dependencies into go projects 'vendor' folder. This forces go to compile
>> offline. However, one have to generate the 'modules.txt' file in the
>> vendor folder 'manually' during unpack. This is error prone, as there is
>> no official documentation how this format should look like. Anyway, I've
>> tried this approach and it works for me.
>> 
>> Second:
>> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by
>> writing a seperate go fetcher) and unpack the dependencies into a local
>> (workdir) go-path. This seemed a good solution for me as the go-path is
>> well defined. But for some reason 'go' fetches the zip-files during
>> compile into it's download-cache AGAIN, even if the source is already
>> unpacked in the go-path. I'll assume this is required to verify the source
>> files integrity?! With this approach one have to adapt 'go' to suppress
>> this download behaviour.
>> 
>> I've been doing offline builds using a constructed vendor/ directory
>> and generated modules.txt.
>> 
>> The only difference between what I have working and what you are
>> suggesting (type 1), is that I've gone directly to the sources and
>> constructed the vendor directory using the OE git fetcher. That allows
>> all functionality to continue to work that is part of OEcore, and the
>> build to continue. Switching out the git fetches for tarballs would
>> be possible, I just wasn't sure how to use the proxied modules (and I
>> wanted the history for debug).
>> 
>> I've never had any issues with the modules.txt, as I generate it at
>> the same time as the git fetch lines for the SRC_URI. I've also not
>> been using information from the go.mod directly from go.proxy.org, it
>> is information I've generated from a clone of the project and dumped
>> via go mod. There's likely improvements I can do there, but with what
>> I'm doing, I'm going directly to the source of the projects and doing
>> clones, which

Re: [OE-core] [PATCH 1/1] go.bbclass: Allow network in do_compile

2022-03-09 Thread lukas . funke 
On Thu, Mar 3, 2022 at 05:46 PM, Bruce Ashfield wrote:

> 
> On Thu, Mar 3, 2022 at 10:13 AM  wrote:
> 
>> On Thu, Mar 3, 2022 at 04:34 AM, Bruce Ashfield wrote:
>> 
>> On Wed, Mar 2, 2022 at 4:57 PM Andrei Gherzan  wrote:
>> 
>> 
>> Mar 1, 2022 20:15:52 Bruce Ashfield :
>> 
>> On Tue, Mar 1, 2022 at 10:54 AM  wrote:
>> 
>> On Tue, Mar 1, 2022 at 02:14 PM, Bruce Ashfield wrote:
>> 
>> On Tue, Mar 1, 2022 at 6:42 AM Andrei Gherzan  wrote:
>> 
>> 
>> On Tue, 1 Mar 2022, at 01:55, Bruce Ashfield wrote:
>> 
>> On Mon, Feb 28, 2022 at 8:17 PM Bruce Ashfield via
>> lists.openembedded.org
>>  wrote:
>> 
>> 
>> On Mon, Feb 28, 2022 at 6:54 PM Andrei Gherzan  wrote:
>> 
>> 
>> 
>> From: Andrei Gherzan 
>> 
>> Compile pulls in the go.mod list requiring network. Without this, do
>> compile would fail with a similar error to the following:
>> 
>> dial tcp: lookup proxy.golang.org: Temporary failure in name resolution
>> 
>> This is something that needs to be carried in your own layers, IMHO it
>> isn't appropriate for core.
>> 
>> It isn't about the fetching, it is the entire gap in functionality
>> that we are missing if go starts fetching dependencies during compile.
>> 
>> A further thought is that if this is for go.mod issues, there is the
>> go-mod.bbclass.
>> 
>> Perhaps enabling it in that class and doing a bbwarn about go fetching
>> dependencies would be appropriate ?
>> 
>> Otherwise, someone may not know that this is happening and that a no
>> network configuration has no chance of working.
>> 
>> I reckon that is reasonable. I'll personally go down the recipe level to
>> workaround this change but understanding and agreeing with the reasoning
>> behind this change, I want to invest a bit into trying to find a proper
>> solution in the core. Bruce, I know you invested a fair amount of time
>> into this already. Would you be willing to sync up and see how we can work
>> together in tackling this?
>> 
>> Definitely, more ideas are good. In fact, I think there are probably
>> several approaches that can co-exist, depending on what a
>> recipe/developer needs.
>> 
>> I'm in the Eastern time zone here, and will try and grab folks on IRC
>> to have a level set
>> 
>> Bruce
>> 
>> Added Zyga to CC as he is also interested in this as part of his go
>> development activities.
>> 
>> Thanks,
>> Andrei
>> 
>> 
>> 
>> --
>> - Thou shalt not follow the NULL pointer, for chaos and madness await
>> thee at its end
>> - "Use the force Harry" - Gandalf, Star Trek II
>> 
>> The problem in allowing downloads during compile (e.g. by go) is, that it
>> leads to non-reproducable builds. I'm currently facing the same issue and
>> would like to have a reproducable go *offline* build.
>> I would like to propose two ideas to workaround the go-compile fetching
>> issue:
>> 
>> First:
>> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by
>> writing a seperate go fetcher or a wget-fetcher) and unpack the
>> dependencies into go projects 'vendor' folder. This forces go to compile
>> offline. However, one have to generate the 'modules.txt' file in the
>> vendor folder 'manually' during unpack. This is error prone, as there is
>> no official documentation how this format should look like. Anyway, I've
>> tried this approach and it works for me.
>> 
>> Second:
>> - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by
>> writing a seperate go fetcher) and unpack the dependencies into a local
>> (workdir) go-path. This seemed a good solution for me as the go-path is
>> well defined. But for some reason 'go' fetches the zip-files during
>> compile into it's download-cache AGAIN, even if the source is already
>> unpacked in the go-path. I'll assume this is required to verify the source
>> files integrity?! With this approach one have to adapt 'go' to suppress
>> this download behaviour.
>> 
>> I've been doing offline builds using a constructed vendor/ directory
>> and generated modules.txt.
>> 
>> The only difference between what I have working and what you are
>> suggesting (type 1), is that I've gone directly to the sources and
>> constructed the vendor directory using the OE git fetcher. That allows
>> all functionality to continue to work that is part of OEcore, and the
>> build to continue. Switching out the git fetches for tarballs would
>> be possible, I just wasn't sure how to use the proxied modules (and I
>> wanted the history for debug).
>> 
>> I've never had any issues with the modules.txt, as I generate it at
>> the same time as the git fetch lines for the SRC_URI. I've also not
>> been using information from the go.mod directly from go.proxy.org, it
>> is information I've generated from a clone of the project and dumped
>> via go mod. There's likely improvements I can do there, but with what
>> I'm doing, I'm going directly to the source of the projects and doing
>> clones, which keeps everything clear of the go infrastructure.
>> 
>> I have a utility that I'm still cleaning up that gen

Re: [OE-core] [PATCH 1/1] go.bbclass: Allow network in do_compile

2022-03-03 Thread lukas . funke 
On Thu, Mar 3, 2022 at 04:34 AM, Bruce Ashfield wrote:

> 
> On Wed, Mar 2, 2022 at 4:57 PM Andrei Gherzan  wrote:
> 
>> 
>> Mar 1, 2022 20:15:52 Bruce Ashfield :
>> 
>> 
>>> On Tue, Mar 1, 2022 at 10:54 AM  wrote:
>>> 
 On Tue, Mar 1, 2022 at 02:14 PM, Bruce Ashfield wrote:
 
 On Tue, Mar 1, 2022 at 6:42 AM Andrei Gherzan  wrote:
 
 
 On Tue, 1 Mar 2022, at 01:55, Bruce Ashfield wrote:
 
 On Mon, Feb 28, 2022 at 8:17 PM Bruce Ashfield via
 lists.openembedded.org
  wrote:
 
 
 On Mon, Feb 28, 2022 at 6:54 PM Andrei Gherzan  wrote:
 
 
 
 From: Andrei Gherzan 
 
 Compile pulls in the go.mod list requiring network. Without this, do
 compile would fail with a similar error to the following:
 
 dial tcp: lookup proxy.golang.org: Temporary failure in name resolution
 
 This is something that needs to be carried in your own layers, IMHO it
 isn't appropriate for core.
 
 It isn't about the fetching, it is the entire gap in functionality
 that we are missing if go starts fetching dependencies during compile.
 
 A further thought is that if this is for go.mod issues, there is the
 go-mod.bbclass.
 
 Perhaps enabling it in that class and doing a bbwarn about go fetching
 dependencies would be appropriate ?
 
 Otherwise, someone may not know that this is happening and that a no
 network configuration has no chance of working.
 
 I reckon that is reasonable. I'll personally go down the recipe level to
 workaround this change but understanding and agreeing with the reasoning
 behind this change, I want to invest a bit into trying to find a proper
 solution in the core. Bruce, I know you invested a fair amount of time
 into this already. Would you be willing to sync up and see how we can work
 together in tackling this?
 
 Definitely, more ideas are good. In fact, I think there are probably
 several approaches that can co-exist, depending on what a
 recipe/developer needs.
 
 I'm in the Eastern time zone here, and will try and grab folks on IRC
 to have a level set
 
 Bruce
 
 Added Zyga to CC as he is also interested in this as part of his go
 development activities.
 
 Thanks,
 Andrei
 
 
 
 --
 - Thou shalt not follow the NULL pointer, for chaos and madness await
 thee at its end
 - "Use the force Harry" - Gandalf, Star Trek II
 
 The problem in allowing downloads during compile (e.g. by go) is, that it
 leads to non-reproducable builds. I'm currently facing the same issue and
 would like to have a reproducable go *offline* build.
 I would like to propose two ideas to workaround the go-compile fetching
 issue:
 
 First:
 - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by
 writing a seperate go fetcher or a wget-fetcher) and unpack the
 dependencies into go projects 'vendor' folder. This forces go to compile
 offline. However, one have to generate the 'modules.txt' file in the
 vendor folder 'manually' during unpack. This is error prone, as there is
 no official documentation how this format should look like. Anyway, I've
 tried this approach and it works for me.
 
 Second:
 - Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by
 writing a seperate go fetcher) and unpack the dependencies into a local
 (workdir) go-path. This seemed a good solution for me as the go-path is
 well defined. But for some reason 'go' fetches the zip-files during
 compile into it's download-cache AGAIN, even if the source is already
 unpacked in the go-path. I'll assume this is required to verify the source
 files integrity?! With this approach one have to adapt 'go' to suppress
 this download behaviour.
>>> 
>>> I've been doing offline builds using a constructed vendor/ directory
>>> and generated modules.txt.
>>> 
>>> The only difference between what I have working and what you are
>>> suggesting (type 1), is that I've gone directly to the sources and
>>> constructed the vendor directory using the OE git fetcher. That allows
>>> all functionality to continue to work that is part of OEcore, and the
>>> build to continue. Switching out the git fetches for tarballs would
>>> be possible, I just wasn't sure how to use the proxied modules (and I
>>> wanted the history for debug).
>>> 
>>> I've never had any issues with the modules.txt, as I generate it at
>>> the same time as the git fetch lines for the SRC_URI. I've also not
>>> been using information from the go.mod directly from go.proxy.org, it
>>> is information I've generated from a clone of the project and dumped
>>> via go mod. There's likely improvements I can do there, but with what
>>> I'm doing, I'm going directly to the source of the projects and doing
>>> clones, which keeps ever

Re: [OE-core] [PATCH 1/1] go.bbclass: Allow network in do_compile

2022-03-01 Thread lukas . funke 
On Tue, Mar 1, 2022 at 02:14 PM, Bruce Ashfield wrote:

> 
> On Tue, Mar 1, 2022 at 6:42 AM Andrei Gherzan  wrote:
> 
>> 
>> On Tue, 1 Mar 2022, at 01:55, Bruce Ashfield wrote:
>> 
>>> On Mon, Feb 28, 2022 at 8:17 PM Bruce Ashfield via
>>> lists.openembedded.org
>>>  wrote:
>>> 
 
 On Mon, Feb 28, 2022 at 6:54 PM Andrei Gherzan  wrote:
 
 
> 
> From: Andrei Gherzan 
> 
> Compile pulls in the go.mod list requiring network. Without this, do
> compile would fail with a similar error to the following:
> 
> dial tcp: lookup proxy.golang.org: Temporary failure in name resolution
 
 This is something that needs to be carried in your own layers, IMHO it
 isn't appropriate for core.
 
 It isn't about the fetching, it is the entire gap in functionality
 that we are missing if go starts fetching dependencies during compile.
>>> 
>>> A further thought is that if this is for go.mod issues, there is the
>>> go-mod.bbclass.
>>> 
>>> Perhaps enabling it in that class and doing a bbwarn about go fetching
>>> dependencies would be appropriate ?
>>> 
>>> Otherwise, someone may not know that this is happening and that a no
>>> network configuration has no chance of working.
>> 
>> I reckon that is reasonable. I'll personally go down the recipe level to
>> workaround this change but understanding and agreeing with the reasoning
>> behind this change, I want to invest a bit into trying to find a proper
>> solution in the core. Bruce, I know you invested a fair amount of time
>> into this already. Would you be willing to sync up and see how we can work
>> together in tackling this?
> 
> Definitely, more ideas are good. In fact, I think there are probably
> several approaches that can co-exist, depending on what a
> recipe/developer needs.
> 
> I'm in the Eastern time zone here, and will try and grab folks on IRC
> to have a level set
> 
> Bruce
> 
> 
>> Added Zyga to CC as he is also interested in this as part of his go
>> development activities.
>> 
>> Thanks,
>> Andrei
> 
> 
> 
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II

The problem in allowing downloads during compile (e.g. by go) is, that it leads 
to non-reproducable builds. I'm currently facing the same issue and would like 
to have a reproducable go *offline* build.
I would like to propose two ideas to workaround the go-compile fetching issue:

First:
- Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by 
writing a seperate go fetcher or a wget-fetcher) and unpack the dependencies 
into go projects 'vendor' folder. This forces go to compile offline. However, 
one have to generate the 'modules.txt' file in the vendor folder 'manually' 
during unpack. This is error prone, as there is no official documentation how 
this format should look like. Anyway, I've tried this approach and it works for 
me.

Second:
- Fetch go-dependencies using go.mod file from 'proxy.golang.org' (e.g. by 
writing a seperate go fetcher) and unpack the dependencies into a local 
(workdir) go-path. This seemed a good solution for me as the go-path is well 
defined. But for some reason 'go'  fetches the zip-files during compile into 
it's download-cache AGAIN, even if the source is already unpacked in the 
go-path. I'll assume this is required to verify the source files integrity?! 
With this approach one have to adapt 'go' to suppress this download behaviour.

Please let me know your opinion on these two approaches.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#162562): 
https://lists.openembedded.org/g/openembedded-core/message/162562
Mute This Topic: https://lists.openembedded.org/mt/89464905/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-