Chen Qi via lists.yoctoproject.org <Qi.Chen= windriver....@lists.yoctoproject.org> escreveu no dia terça, 5/09/2023 à(s) 04:22:
> From: Chen Qi <qi.c...@windriver.com> > > oe-go-mod-autogen.py is a helper script for go mod recipes. It follows > Bruce's initiative about how to deal with go mod recipes in OE. > Place the below description on some README file or commented itself inside the script can be useful for others. Jose Example: > cmd: <path_to>/meta-virtualization/scripts/oe-go-mod-autogen.py \ > --repo https://github.com/docker/compose --rev v2.20.3 > output: src_uri.inc, relocation.inc, modules.txt > > Copy these three generated files to replace the original ones, > then we only need update PV and SRCREV, and docker-compose is upgraded. > > Below are some technical details. > > * get module's repo from module name > > This script checks the following two URLs to determine the module's repo. > 1. https://<module_name_tweaked>?=go-get=1 > 2. https://pkg.go.dev/<module_name_tweaked> > > The module_name_tweaked is derived from module_name, with the last > components > removed one by one. Let me use two examples to explain this. > > For module_name sigs.k8s.io/json, the sigs.k8s.io/json is first used as > module_name_tweaked for searching. And we can correctly get the repo > URL, so > the search stops. > > For module_name github.com/k3s-io/etcd/api/v3, the following ones are > used > as module_name_tweaked: > github.com/k3s-io/etcd/api/v3 > github.com/k3s-io/etcd/api > github.com/k3s-io/etcd > And when searching 'github.com/k3s-io/etcd', we get the repo URL, so > the search > stops. > > * determine the srcdir:destdir mapping in 'vendor' creation > > To correctly form the 'vendor' directory, the mapping is critical. > This script makes use of tag matching and path matching to determine > the subpath in the repo for the module. > > * avoid subpath being overriden by parent path > > We need to avoid subpath being overriden by parent path. This is needed > for both SRC_URI ordering in src_uri.inc and the sites mapping ordering > in relocation.inc. This script simply uses the length as the ordering > key, > simply for the reason that if a path is a subpath of another path, it > must > be longer. > > * the .git suffix is removed to sync with each other > > Unlike normal recipes, go mod recipe usually have many SRC_URIs. This > script > remove the '.git' suffix from repo URL so that the repo URLs are in sync > with each. > > * basic directory hierarchy and caching mechanism > > <cwd>/repos: hold the repos downloaded and checked > <cwd>/wget-contents: hold the contents to determine the module's repo > <cwd>/wget-contents/<module_name>.repo_url.cache: the repo value cache > This is to avoid unnecessary URL fetching and repo cloning. > > * the ERROR_OUT_ON_FETCH_AND_CHECKOUT_FAILURE switch in script > > The script must get the correct repo_url, fullsrc_rev and subpath for > each required module in go.mod to correctly generate the src_uri.inc and > relocation.inc files. If this process fails for any required module, this > script stop immediately, as I deliberately set > ERROR_OUT_ON_FETCH_AND_CHECKOUT_FAILURE > to True in this script. The purpose is to encourage people to report > problems to meta-virt so that we can improve this script according to > these feedbacks. But this variable can set to False, then the script > only records the failed modules in self.modules_unhandled with reasons > added, people can modify the generated src_uri.inc and relocation.inc > to manually handle these unhandled modules if they are urgent to > add/upgrade some go mod recipes. > > Signed-off-by: Chen Qi <qi.c...@windriver.com> > --- > scripts/oe-go-mod-autogen.py | 663 +++++++++++++++++++++++++++++++++++ > 1 file changed, 663 insertions(+) > create mode 100755 scripts/oe-go-mod-autogen.py > > diff --git a/scripts/oe-go-mod-autogen.py b/scripts/oe-go-mod-autogen.py > new file mode 100755 > index 00000000..09d6133b > --- /dev/null > +++ b/scripts/oe-go-mod-autogen.py > @@ -0,0 +1,663 @@ > +#!/usr/bin/env python3 > + > +import os > +import sys > +import logging > +import argparse > +from collections import OrderedDict > +import subprocess > + > +# This switch is used to make this script error out ASAP, mainly for > debugging purpose > +ERROR_OUT_ON_FETCH_AND_CHECKOUT_FAILURE = True > + > +logger = logging.getLogger('oe-go-mod-autogen') > +loggerhandler = logging.StreamHandler() > +loggerhandler.setFormatter(logging.Formatter("%(levelname)s: > %(message)s")) > +logger.addHandler(loggerhandler) > +logger.setLevel(logging.INFO) > + > +class GoModTool(object): > + def __init__(self, repo, rev, workdir): > + self.repo = repo > + self.rev = rev > + self.workdir = workdir > + > + # Stores the actual module name and its related information > + # {module: (repo_url, repo_dest_dir, fullsrcrev)} > + self.modules_repoinfo = {} > + > + # {module_name: (url, version, destdir, fullsrcrev)} > + # > + # url: place to get the source codes, we only support git repo > + # version: module version, git tag or git rev > + # destdir: place to put the fetched source codes > + # fullsrcrev: full src rev which is the value of SRC_REV > + # > + # e.g. > + # For 'github.com/Masterminds/semver/v3 v3.1.1' in go.mod: > + # module_name = github.com/Masterminds/semver/v3 > + # url = https://github.com/Masterminds/semver > + # version = v3.1.1 > + # destdir = ${WORKDIR}/${BP}/src/${GO_IMPORT}/vendor/ > github.com/Masterminds/semver/v3 > + # fullsrcrev = d387ce7889a157b19ad7694dba39a562051f41b0 > + self.modules_require = OrderedDict() > + > + # {orig_module: (actual_module, actual_version)} > + self.modules_replace = OrderedDict() > + > + # Unhandled modules > + self.modules_unhandled = OrderedDict() > + > + # store subpaths used to form srcpath > + # {actual_module_name: subpath} > + self.modules_subpaths = OrderedDict() > + > + # modules's actual source paths, record those that are not the > same with the module itself > + self.modules_srcpaths = OrderedDict() > + > + # store lines, comment removed > + self.require_lines = [] > + self.replace_lines = [] > + > + # fetch repo > + self.fetch_and_checkout_repo(self.repo.split('://')[1], > self.repo, self.rev, checkout=True, get_subpath=False) > + > + def show_go_mod_info(self): > + # Print modules_require, modules_replace and modules_unhandled > + print("modules required:") > + for m in self.modules_require: > + url, version, destdir, fullrev = self.modules_require[m] > + print("%s %s %s %s" % (m, version, url, fullrev)) > + > + print("modules replace:") > + for m in self.modules_replace: > + actual_module, actual_version = self.modules_replace[m] > + print("%s => %s %s" % (m, actual_module, actual_version)) > + > + print("modules unhandled:") > + for m in self.modules_unhandled: > + reason = self.modules_unhandled[m] > + print("%s unhandled: %s" % (m, reason)) > + > + def parse(self): > + # check if this repo needs autogen > + repo_url, repo_dest_dir, repo_fullrev = > self.modules_repoinfo[self.repo.split('://')[1]] > + if os.path.isdir(os.path.join(repo_dest_dir, 'vendor')): > + logger.info("vendor direcotry has already existed for %s, no > need to add other repos" % self.repo) > + return > + go_mod_file = os.path.join(repo_dest_dir, 'go.mod') > + if not os.path.exists(go_mod_file): > + logger.info("go.mod file does not exist for %s, no need to > add otehr repos" % self.repo) > + return > + self.parse_go_mod(go_mod_file) > + self.show_go_mod_info() > + > + def fetch_and_checkout_repo(self, module_name, repo_url, rev, > default_protocol='https://', checkout=False, get_subpath=True): > + """ > + Fetch repo_url to <workdir>/repos/repo_base_name > + """ > + protocol = default_protocol > + if '://' in repo_url: > + repo_url_final = repo_url > + else: > + repo_url_final = default_protocol + repo_url > + logger.debug("fetch and checkout %s %s" % (repo_url_final, rev)) > + repos_dir = os.path.join(self.workdir, 'repos') > + if not os.path.exists(repos_dir): > + os.makedirs(repos_dir) > + repo_basename = repo_url.split('/')[-1].split('.git')[0] > + repo_dest_dir = os.path.join(repos_dir, repo_basename) > + module_last_name = module_name.split('/')[-1] > + git_action = "fetch" > + if os.path.exists(repo_dest_dir): > + if checkout: > + # check if current HEAD is rev > + try: > + headrev = subprocess.check_output('git rev-list -1 > HEAD', shell=True, cwd=repo_dest_dir).decode('utf-8').strip() > + requiredrev = subprocess.check_output('git rev-list > -1 %s 2>/dev/null || git rev-list -1 %s/%s' % (rev, module_last_name, rev), > shell=True, cwd=repo_dest_dir).decode('utf-8').strip() > + if headrev == requiredrev: > + logger.info("%s has already been fetched and > checked out as required, skipping" % repo_url) > + self.modules_repoinfo[module_name] = (repo_url, > repo_dest_dir, requiredrev) > + return > + else: > + logger.info("HEAD of %s is not %s, will do a > clean clone" % (repo_dest_dir, requiredrev)) > + git_action = "clone" > + except: > + logger.info("'git rev-list' in %s failed, will do a > clean clone" % repo_dest_dir) > + git_action = "clone" > + else: > + # determine if the current repo points to the desired > remote repo > + try: > + remote_origin_url = subprocess.check_output('git > config --get remote.origin.url', shell=True, > cwd=repo_dest_dir).decode('utf-8').strip() > + if remote_origin_url.endswith('.git'): > + if not repo_url_final.endswith('.git'): > + remote_origin_url = remote_origin_url[:-4] > + else: > + if repo_url_final.endswith('.git'): > + remote_origin_url = remote_origin_url + '.git' > + if remote_origin_url != repo_url_final: > + logger.info("remote.origin.url for %s is not %s, > will do a clean clone" % (repo_dest_dir, repo_url_final)) > + git_action = "clone" > + except: > + logger.info("'git config --get remote.origin.url' in > %s failed, will do a clean clone" % repo_dest_dir) > + git_action = "clone" > + else: > + # No local repo, clone it. > + git_action = "clone" > + > + if git_action == "clone": > + logger.info("Removing %s" % repo_dest_dir) > + subprocess.check_call('rm -rf %s' % repo_dest_dir, shell=True) > + > + # clone/fetch repo > + try: > + git_cwd = repos_dir if git_action == "clone" else > repo_dest_dir > + logger.info("git %s %s in %s" % (git_action, repo_url_final, > git_cwd)) > + subprocess.check_call('git %s %s >/dev/null 2>&1' % > (git_action, repo_url_final), shell=True, cwd=git_cwd) > + except: > + logger.warning("Failed to %s %s in %s" % (git_action, > repo_url_final, git_cwd)) > + return > + > + def get_requiredrev(get_subpath): > + import re > + # check if rev is a revision or a version > + if len(rev) == 12 and re.match('[0-9a-f]+', rev): > + rev_is_version = False > + else: > + rev_is_version = True > + > + # if rev is not a version, 'git rev-list -1 <rev>' should > just succeed! > + if not rev_is_version: > + try: > + rev_return = subprocess.check_output('git rev-list -1 > %s 2>/dev/null' % rev, shell=True, > cwd=repo_dest_dir).decode('utf-8').strip() > + if get_subpath: > + cmd = 'git branch -M toremove && git checkout -b > check_subpath %s && git branch -D toremove' % rev_return > + subprocess.check_call(cmd, shell=True, > cwd=repo_dest_dir) > + # try to get the subpath for this module > + module_name_parts = module_name.split('/') > + while (len(module_name_parts) > 0): > + subpath = '/'.join(module_name_parts) > + dir_to_check = repo_dest_dir + '/' + > '/'.join(module_name_parts) > + if os.path.isdir(dir_to_check): > + self.modules_subpaths[module_name] = > subpath > + break > + else: > + module_name_parts.pop(0) > + return rev_return > + except: > + logger.warning("Revision (%s) not in repo(%s)" % > (rev, repo_dest_dir)) > + return None > + > + # the following codes deals with case where rev is a version > + # determine the longest match tag, in this way, we can get > the current srcpath to be used in relocation.inc > + # we first get the initial tag, which is formed from > module_name and rev > + module_parts = module_name.split('/') > + if rev.startswith(module_parts[-1] + '.'): > + tag = '/'.join(module_parts[:-1]) + '/' + rev > + last_module_part_replaced = True > + else: > + tag = '/'.join(module_parts) + '/' + rev > + last_module_part_replaced = False > + logger.debug("use %s as the initial tag for %s" % (tag, > module_name)) > + tag_parts = tag.split('/') > + while(len(tag_parts) > 0): > + try: > + rev_return = subprocess.check_output('git rev-list -1 > %s 2>/dev/null' % tag, shell=True, > cwd=repo_dest_dir).decode('utf-8').strip() > + if len(tag_parts) > 1: > + # ensure that the subpath exists > + if get_subpath: > + cmd = 'git branch -M toremove && git checkout > -b check_subpath %s && git branch -D toremove' % rev_return > + subprocess.check_call(cmd, shell=True, > cwd=repo_dest_dir) > + # get subpath for the actual_module_name > + if last_module_part_replaced: > + subpath = '/'.join(tag_parts[:-1]) + '/' > + module_parts[-1] > + if not os.path.isdir(repo_dest_dir + '/' > + subpath): > + subpath = '/'.join(tag_parts[:-1]) > + else: > + subpath = '/'.join(tag_parts[:-1]) > + if not os.path.isdir(repo_dest_dir + '/' + > subpath): > + logger.warning("subpath (%s) derived from > tag matching does not exist in %s" % (subpath, repo_dest_dir)) > + return None > + self.modules_subpaths[module_name] = subpath > + logger.info("modules_subpath[%s] = %s" % > (module_name, subpath)) > + return rev_return > + except: > + tag_parts.pop(0) > + tag = '/'.join(tag_parts) > + logger.warning("No tag matching %s" % rev) > + return None > + > + requiredrev = get_requiredrev(get_subpath) > + if requiredrev: > + logger.info("Got module(%s) requiredrev: %s" % (module_name, > requiredrev)) > + if checkout: > + subprocess.check_call('git checkout -b gomodautogen %s' % > requiredrev, shell=True, cwd=repo_dest_dir) > + self.modules_repoinfo[module_name] = (repo_url, > repo_dest_dir, requiredrev) > + else: > + logger.warning("Failed to get requiredrev, repo_url = %s, rev > = %s, module_name = %s" % (repo_url, rev, module_name)) > + > + def parse_go_mod(self, go_mod_path): > + """ > + Parse go.mod file to get the modules info > + """ > + # First we get the require and replace lines > + # The parsing logic assumes the replace lines come *after* the > require lines > + inrequire = False > + inreplace = False > + with open(go_mod_path, 'r') as f: > + lines = f.readlines() > + for line in lines: > + if line.startswith('require ('): > + inrequire = True > + continue > + if line.startswith(')'): > + inrequire = False > + continue > + if line.startswith('require ') or inrequire: > + # we have one line require > + require_line = line.lstrip('require > ').split('//')[0].strip() > + if require_line: > + self.require_lines.append(require_line) > + continue > + # we can deal with requires and replaces separately > because go.mod always writes requires before replaces > + if line.startswith('replace ('): > + inreplace = True > + continue > + if line.startswith(')'): > + inreplace = False > + continue > + if line.startswith('replace ') or inreplace: > + replace_line = line.lstrip('replace > ').split('//')[0].strip() > + if replace_line: > + self.replace_lines.append(replace_line) > + continue > + # > + # parse the require_lines and replace_lines to form > self.modules_require and self.modules_replace > + # > + logger.debug("Parsing require_lines and replace_lines ...") > + # A typical replace line is as below: > + # github.com/hashicorp/golang-lru => > github.com/ktock/golang-lru v0.5.5-0.20211029085301-ec551be6f75c > + # It means that the github.com/hashicorp/golang-lru module is > replaced by github.com/ktock/golang-lru > + # with the version 'v0.5.5-0.20211029085301-ec551be6f75c'. > + # So the destdir is vendor/github.com/hashicorp/golang-lru while > the contents are from github.com/ktock/golang-lru > + for line in self.replace_lines: > + orig_module, actual = line.split('=>') > + actual_module, actual_version = actual.split() > + orig_module = orig_module.strip() > + actual_module = actual_module.strip() > + actual_version = actual_version.strip() > + self.modules_replace[orig_module] = (actual_module, > actual_version) > + # > + # Typical require lines are as below: > + # github.com/Masterminds/semver/v3 v3.1.1 > + # golang.org/x/crypto v0.0.0-20220321153916-2c7772ba3064 > + # > + # We need to first try https://<module_name>?=go-get=1 to see it > contains > + # line starting with '<meta name="go-import" content='. > + # > + # If so, get root-path vcs repo-url from content. See > https://go.dev/ref/mod#vcs-find > + # For example, the above 'wget > https://golang.org/x/crypto?go-get=1' gives you > + # <meta name="go-import" content="golang.org/x/crypto git > https://go.googlesource.com/crypto";> > + # In such case, the self.modules_require has the following > contents: > + # module_name: golang.org/x/crypto > + # url: https://go.googlesource.com/crypto > + # version: v0.0.0-20220321153916-2c7772ba3064 > + # destdir: ${WORKDIR}/${BP}/src/import/vendor.fetch/ > golang.org/x/crypto > + # fullsrcrev: 2c7772ba30643b7a2026cbea938420dce7c6384d (git > rev-list -1 2c7772ba3064) > + # > + # If not, try https://pkg.go.dev/<module_name>, and find the > 'Repository'. > + # For example, 'wget > https://pkg.go.dev/github.com/Masterminds/semver/v3' gives: > + # github.com/Masterminds/semver > + # In such case, the self.modules has the following contents: > + # module_name: github.com/Masterminds/semver/v3 > + # url: https://github.com/Masterminds/semver > + # version: v3.1.1 > + # destdir: ${WORKDIR}/${BP}/src/import/vendor.fetch/ > github.com/Masterminds/semver/v3 > + # fullsrcrev: 7bb0c843b53d6ad21a3f619cb22c4b442bb3ef3e (git > rev-list -1 v3.1.1) > + # > + # As a last resort, if the last component of <module_name> > matches 'v[0-9]+', > + # remove the last component and try wget https:// > <module_name_with_last_component_removed>?go-get=1, > + # then try using the above matching method. > + # > + for line in self.require_lines: > + module_name, version = line.strip().split() > + logger.debug("require line: %s" % line) > + logger.debug("module_name = %s; version = %s" % (module_name, > version)) > + # take the modules_replace into consideration to get the > actual version and actual module name > + # note that the module_name is used in destdir, and the > actual_module_name and actual_version > + # are used to determine the url and fullsrcrev > + destdir = '${WORKDIR}/${BP}/src/import/vendor.fetch/%s' % > module_name > + actual_module_name = module_name > + actual_version = version > + if module_name in self.modules_replace: > + actual_module_name, actual_version = > self.modules_replace[module_name] > + logger.debug("actual_module_name = %s; actual_version = %s" % > (actual_module_name, actual_version)) > + url, fullsrcrev = self.get_url_srcrev(actual_module_name, > actual_version) > + logger.debug("url = %s; fullsrcrev = %s" % (url, fullsrcrev)) > + if url and fullsrcrev: > + self.modules_require[module_name] = (url, version, > destdir, fullsrcrev) > + # form srcpath, actual_module_name/<subpath> > + if actual_module_name in self.modules_subpaths: > + subpath = self.modules_subpaths[actual_module_name] > + srcpath = '%s/%s' % (actual_module_name, subpath) > + self.modules_srcpaths[module_name] = srcpath > + logger.info("self.modules_srcpaths[%s] = %s" % > (module_name, srcpath)) > + else: > + self.modules_srcpaths[module_name] = > actual_module_name > + else: > + logger.warning("get_url_srcrev(%s, %s) failed" % > (actual_module_name, actual_version)) > + if ERROR_OUT_ON_FETCH_AND_CHECKOUT_FAILURE: > + sys.exit(1) > + > + def use_wget_to_get_repo_url(self, wget_content_file, url_cache_file, > module_name): > + """ > + Use wget to get repo_url for module_name, return None if not found > + """ > + try: > + logger.info("wget -O %s https://%s?=go-get=1"; % > (wget_content_file, module_name)) > + subprocess.check_call('wget -O %s https://%s?=go-get=1' % > (wget_content_file, module_name), shell=True) > + with open(wget_content_file, 'r') as f: > + for line in f.readlines(): > + if '<meta name="go-import" content=' in line: > + logger.info("Succeed to find go-import content > for %s" % module_name) > + logger.debug("The line is %s" % line) > + root_path, vcs, repo_url = > line.split('content=')[1].split('"')[1].split() > + logger.info("%s: %s %s %s" % (module_name, > root_path, vcs, repo_url)) > + if vcs != 'git': > + logger.warning('%s unhandled as its vcs is %s > which is not supported by this script.' % (module_name, vcs)) > + unhandled_reason = 'vcs %s is not supported > by this script' % vcs > + self.modules_unhandled[module_name] = > unhandled_reason > + return None > + with open(url_cache_file, 'w') as f: > + f.write(repo_url) > + return repo_url > + except: > + logger.info("wget -O %s https://%s?=go-get=1 failed" % > (wget_content_file, module_name)) > + # if we cannot find repo url from https://<module_name>?=go-get=1, > try https://pkg.go/dev/<module_name> > + try: > + logger.info("wget -O %s https://pkg.go.dev/%s"; % > (wget_content_file, module_name)) > + subprocess.check_call("wget -O %s https://pkg.go.dev/%s"; % > (wget_content_file, module_name), shell=True) > + repo_url_found = False > + with open(wget_content_file, 'r') as f: > + in_repo_section = False > + for line in f.readlines(): > + if '>Repository<' in line: > + in_repo_section = True > + continue > + if in_repo_section: > + newline = line.strip() > + if newline != '' and not newline.startswith('<'): > + repo_url = newline > + repo_url_found = True > + break > + if repo_url_found: > + logger.info("repo url for %s: %s" % (module_name, > repo_url)) > + with open(url_cache_file, 'w') as f: > + f.write(repo_url) > + return repo_url > + else: > + unhandled_reason = 'cannot determine repo_url for %s' % > module_name > + self.modules_unhandled[module_name] = unhandled_reason > + return None > + except: > + logger.info("wget -O %s https://pkg.go.dev/%s failed" % > (wget_content_file, module_name)) > + return None > + > + > + def get_repo_url_rev(self, module_name, version): > + """ > + Return (repo_url, rev) > + """ > + import re > + # First get rev from version > + v = version.split('+incompatible')[0] > + version_components = v.split('-') > + if len(version_components) == 1: > + rev = v > + elif len(version_components) == 3: > + if len(version_components[2]) == 12: > + rev = version_components[2] > + else: > + rev = v > + else: > + rev = v > + > + # > + # Get repo_url > + # We put a cache mechanism here, > <wget_content_file>.repo_url.cache is used to store the repo url fetch > before > + # > + wget_dir = os.path.join(self.workdir, 'wget-contents') > + if not os.path.exists(wget_dir): > + os.makedirs(wget_dir) > + wget_content_file = os.path.join(wget_dir, > module_name.replace('/', '_')) > + url_cache_file = "%s.repo_url.cache" % wget_content_file > + if os.path.exists(url_cache_file): > + with open(url_cache_file, 'r') as f: > + repo_url = f.readline().strip() > + return (repo_url, rev) > + module_name_parts = module_name.split('/') > + while (len(module_name_parts) > 0): > + module_name_to_check = '/'.join(module_name_parts) > + logger.info("module_name_to_check: %s" % > module_name_to_check) > + repo_url = self.use_wget_to_get_repo_url(wget_content_file, > url_cache_file, module_name_to_check) > + if repo_url: > + return (repo_url, rev) > + else: > + if module_name in self.modules_unhandled: > + return (None, rev) > + else: > + module_name_parts.pop(-1) > + > + unhandled_reason = 'cannot determine the repo for %s' % > module_name > + self.modules_unhandled[module_name] = unhandled_reason > + return (None, rev) > + > + def get_url_srcrev(self, module_name, version): > + """ > + Return url and fullsrcrev according to module_name and version > + """ > + repo_url, rev = self.get_repo_url_rev(module_name, version) > + if not repo_url or not rev: > + return (None, None) > + self.fetch_and_checkout_repo(module_name, repo_url, rev) > + if module_name in self.modules_repoinfo: > + repo_url, repo_dest_dir, repo_fullrev = > self.modules_repoinfo[module_name] > + # remove the .git suffix to sync repos across modules with > different versions and across recipes > + if repo_url.endswith('.git'): > + repo_url = repo_url[:-len('.git')] > + return (repo_url, repo_fullrev) > + else: > + unhandled_reason = 'fetch_and_checkout_repo(%s, %s, %s) > failed' % (module_name, repo_url, rev) > + self.modules_unhandled[module_name] = unhandled_reason > + return (None, None) > + > + def gen_src_uri_inc(self): > + """ > + Generate src_uri.inc file containing SRC_URIs > + """ > + src_uri_inc_file = os.path.join(self.workdir, 'src_uri.inc') > + # record the <name> after writting SRCREV_<name>, this is to > avoid modules having the same basename resulting in same SRCREV_xxx > + srcrev_name_recorded = [] > + template = """# %s %s > +# [1] git ls-remote %s %s > +SRCREV_%s="%s" > +SRC_URI += > "git://%s;name=%s;protocol=https;nobranch=1;destsuffix=${WORKDIR}/${BP}/src/import/vendor.fetch/%s" > + > +""" > + # We can't simply write SRC_URIs one by one in the order that > go.mod specify them. > + # Because the latter one might clean things up for the former one > if the former one is a subpath of the latter one. > + def take_first_len(elem): > + return len(elem[0]) > + > + src_uri_contents = [] > + with open(src_uri_inc_file, 'w') as f: > + for module in self.modules_require: > + # {module_name: (url, version, destdir, fullsrcrev)} > + repo_url, version, destdir, fullrev = > self.modules_require[module] > + if module in self.modules_replace: > + actual_module_name, actual_version = > self.modules_replace[module] > + else: > + actual_module_name, actual_version = (module, version) > + if '://' in repo_url: > + repo_url_noprotocol = repo_url.split('://')[1] > + else: > + repo_url_noprotocol = repo_url > + if not repo_url.startswith('https://'): > + repo_url = 'https://' + repo_url > + name = module.split('/')[-1] > + if name in srcrev_name_recorded: > + name = '-'.join(module.split('/')[-2:]) > + src_uri_contents.append((actual_module_name, > actual_version, repo_url, fullrev, name, fullrev, repo_url_noprotocol, > name, actual_module_name)) > + srcrev_name_recorded.append(name) > + # sort the src_uri_contents and then write it > + src_uri_contents.sort(key=take_first_len) > + for content in src_uri_contents: > + f.write(template % content) > + logger.info("%s generated" % src_uri_inc_file) > + > + def gen_relocation_inc(self): > + """ > + Generate relocation.inc file > + """ > + relocation_inc_file = os.path.join(self.workdir, 'relocation.inc') > + template = """export sites="%s" > + > +do_compile:prepend() { > + cd ${S}/src/import > + for s in $sites; do > + site_dest=$(echo $s | cut -d: -f1) > + site_source=$(echo $s | cut -d: -f2) > + force_flag=$(echo $s | cut -d: -f3) > + mkdir -p vendor.copy/$site_dest > + if [ -n "$force_flag" ]; then > + echo "[INFO] $site_dest: force copying .go files" > + rm -rf vendor.copy/$site_dest > + rsync -a --exclude='vendor/' --exclude='.git/' > vendor.fetch/$site_source/ vendor.copy/$site_dest > + else > + [ -n "$(ls -A vendor.copy/$site_dest/*.go 2> /dev/null)" ] && > { echo "[INFO] vendor.fetch/$site_source -> $site_dest: go copy skipped > (files present)" ; true ; } || { echo "[INFO] $site_dest: copying .go > files" ; rsync -a --exclude='vendor/' --exclude='.git/' > vendor.fetch/$site_source/ vendor.copy/$site_dest ; } > + fi > + done > +} > +""" > + sites = [] > + for module in self.modules_require: > + # <dest>:<source>[:force] > + if module in self.modules_srcpaths: > + srcpath = self.modules_srcpaths[module] > + logger.debug("Using %s as srcpath of module (%s)" % > (srcpath, module)) > + else: > + srcpath = module > + sites.append("%s:%s:force" % (module, srcpath)) > + # To avoid the former one being overriden by the latter one when > the former one is a subpath of the latter one, sort sites > + sites.sort(key=len) > + with open(relocation_inc_file, 'w') as f: > + sites_str = ' \\\n '.join(sites) > + f.write(template % sites_str) > + logger.info("%s generated" % relocation_inc_file) > + > + def gen_modules_txt(self): > + """ > + Generate modules.txt file > + """ > + modules_txt_file = os.path.join(self.workdir, 'modules.txt') > + with open(modules_txt_file, 'w') as f: > + for l in self.require_lines: > + f.write('# %s\n' % l) > + f.write('## explicit\n') > + for l in self.replace_lines: > + f.write('# %s\n' %l) > + logger.info("%s generated" % modules_txt_file) > + > + def sanity_check(self): > + """ > + Various anity checks > + """ > + sanity_check_ok = True > + # > + # Sanity Check 1: > + # For modules having the same repo, at most one is allowed to > not have subpath. > + # This check operates on self.modules_repoinfo and > self.modules_subpaths > + # > + repo_modules = {} > + for module in self.modules_repoinfo: > + # first form {repo: [module1, module2, ...]} > + repo_url, repo_dest_dir, fullsrcrev = > self.modules_repoinfo[module] > + if repo_url not in repo_modules: > + repo_modules[repo_url] = [module] > + else: > + repo_modules[repo_url].append(module) > + for repo in repo_modules: > + modules = repo_modules[repo] > + if len(modules) == 1: > + continue > + # for modules sharing the same repo, at most one is allowed > to not have subpath > + nosubpath_modules = [] > + for m in modules: > + if m not in self.modules_subpaths: > + nosubpath_modules.append(m) > + if len(nosubpath_modules) == 0: > + continue > + if len(nosubpath_modules) > 1: > + logger.warning("Multiple modules sharing %s, but they > don't have subpath: %s. Please double check." % (repo, nosubpath_modules)) > + if len(nosubpath_modules) == 1: > + # do further check, OK if the module is the prefix for > other modules sharing the same repo > + module_to_check = nosubpath_modules[0] > + for m in modules: > + if module_to_check == m: > + continue > + if not m.startswith('%s/' % module_to_check): > + logger.warning("%s is sharing repo (%s) with > other modules, and it might need a subpath. Please double check" % > (module_to_check, repo)) > + continue > + > + # > + # End of Sanity Check > + # > + if not sanity_check_ok: > + sys.exit(1) > + return > + > +def main(): > + parser = argparse.ArgumentParser( > + description="oe-go-mod-autogen.py is used to generate > src_uri.inc, relocation.inc and modules.txt to be used by go mod recipes", > + epilog="Use %(prog)s --help to get help") > + parser.add_argument("--repo", help = "Repo for the recipe.", > required=True) > + parser.add_argument("--rev", help = "Revision for the recipe.", > required=True) > + parser.add_argument("--module", help = "Go module name. To be used > with '--test'") > + parser.add_argument("--version", help = "Go module version. To be > used with '--test'") > + parser.add_argument("--test", help = "Test to get repo url and > fullsrcrev, used together with --module and --version.", > action="store_true") > + parser.add_argument("--workdir", help = "Working directory to hold > intermediate results and output.", default=os.getcwd()) > + parser.add_argument("-d", "--debug", > + help = "Enable debug output", > + action="store_const", const=logging.DEBUG, > dest="loglevel", default=logging.INFO) > + parser.add_argument("-q", "--quiet", > + help = "Hide all output except error messages", > + action="store_const", const=logging.ERROR, > dest="loglevel") > + args = parser.parse_args() > + > + logger.setLevel(args.loglevel) > + logger.debug("oe-go-mod-autogen.py running for %s:%s in %s" % > (args.repo, args.rev, args.workdir)) > + gomodtool = GoModTool(args.repo, args.rev, args.workdir) > + if args.test: > + if not args.module or not args.version: > + print("Please specify --module and --version") > + sys.exit(1) > + url, srcrev = gomodtool.get_url_srcrev(args.module, args.version) > + print("url = %s, srcrev = %s" % (url, srcrev)) > + if not url or not srcrev: > + print("Failed to get url & srcrev for %s:%s" % (args.module, > args.version)) > + else: > + gomodtool.parse() > + gomodtool.sanity_check() > + gomodtool.gen_src_uri_inc() > + gomodtool.gen_relocation_inc() > + gomodtool.gen_modules_txt() > + > + > +if __name__ == "__main__": > + try: > + ret = main() > + except Exception as esc: > + ret = 1 > + import traceback > + traceback.print_exc() > + sys.exit(ret) > -- > 2.40.0 > > > > > -- Best regards, José Quaresma
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#8275): https://lists.yoctoproject.org/g/meta-virtualization/message/8275 Mute This Topic: https://lists.yoctoproject.org/mt/101162746/21656 Group Owner: meta-virtualization+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/meta-virtualization/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
