Script 'mail_helper' called by obssrc Hello community, here is the log from the commit of package obs-service-cargo_audit for openSUSE:Factory checked in at 2021-07-05 22:22:51 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/obs-service-cargo_audit (Old) and /work/SRC/openSUSE:Factory/.obs-service-cargo_audit.new.2625 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "obs-service-cargo_audit" Mon Jul 5 22:22:51 2021 rev:5 rq:904002 version:0.1.5~git0.01fea16 Changes: -------- --- /work/SRC/openSUSE:Factory/obs-service-cargo_audit/obs-service-cargo_audit.changes 2021-06-09 21:53:15.322576691 +0200 +++ /work/SRC/openSUSE:Factory/.obs-service-cargo_audit.new.2625/obs-service-cargo_audit.changes 2021-07-05 22:23:17.393578056 +0200 @@ -1,0 +2,6 @@ +Mon Jul 05 03:33:51 UTC 2021 - wbr...@suse.de + +- Update to version 0.1.5~git0.01fea16: + * Allows non-scm packages in obs to be scanned for vulnerabilities + +------------------------------------------------------------------- Old: ---- obs-service-cargo_audit-0.1.4~git0.a3c7623.tar.gz New: ---- obs-service-cargo_audit-0.1.5~git0.01fea16.tar.gz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ obs-service-cargo_audit.spec ++++++ --- /var/tmp/diff_new_pack.UWv1j1/_old 2021-07-05 22:23:17.905574094 +0200 +++ /var/tmp/diff_new_pack.UWv1j1/_new 2021-07-05 22:23:17.905574094 +0200 @@ -1,5 +1,5 @@ # -# spec file for package obs-service-cargo_audit +# spec file # # Copyright (c) 2021 SUSE LLC # @@ -22,7 +22,7 @@ License: MPL-2.0 Group: Development/Tools/Building URL: https://github.com/openSUSE/obs-service-%{service} -Version: 0.1.4~git0.a3c7623 +Version: 0.1.5~git0.01fea16 Release: 0 Source: %{name}-%{version}.tar.gz BuildRequires: python3 ++++++ _service ++++++ --- /var/tmp/diff_new_pack.UWv1j1/_old 2021-07-05 22:23:17.953573723 +0200 +++ /var/tmp/diff_new_pack.UWv1j1/_new 2021-07-05 22:23:17.953573723 +0200 @@ -3,7 +3,7 @@ <param name="url">https://github.com/openSUSE/obs-service-cargo_audit.git</param> <param name="versionformat">@PARENT_TAG@~git@TAG_OFFSET@.%h</param> <param name="scm">git</param> - <param name="revision">v0.1.4</param> + <param name="revision">v0.1.5</param> <param name="match-tag">v*</param> <param name="versionrewrite-pattern">v(\d+\.\d+\.\d+)</param> <param name="versionrewrite-replacement">\1</param> ++++++ obs-service-cargo_audit-0.1.4~git0.a3c7623.tar.gz -> obs-service-cargo_audit-0.1.5~git0.01fea16.tar.gz ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/obs-service-cargo_audit-0.1.4~git0.a3c7623/.gitignore new/obs-service-cargo_audit-0.1.5~git0.01fea16/.gitignore --- old/obs-service-cargo_audit-0.1.4~git0.a3c7623/.gitignore 1970-01-01 01:00:00.000000000 +0100 +++ new/obs-service-cargo_audit-0.1.5~git0.01fea16/.gitignore 2021-07-05 05:26:03.000000000 +0200 @@ -0,0 +1,5 @@ +.DS_Store +*.pyc +*.patch +openSUSE:Factory/ +target/ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/obs-service-cargo_audit-0.1.4~git0.a3c7623/README.md new/obs-service-cargo_audit-0.1.5~git0.01fea16/README.md --- old/obs-service-cargo_audit-0.1.4~git0.a3c7623/README.md 2021-06-08 05:53:44.000000000 +0200 +++ new/obs-service-cargo_audit-0.1.5~git0.01fea16/README.md 2021-07-05 05:26:03.000000000 +0200 @@ -6,7 +6,8 @@ a Rust application have known security vulnerabilities. If vulnerabilities are found, the source service will alert allowing you to update and help upstream update their sources. -This relies on having unpacked sources available, so you should use the SCM module. +It is prefered that you use an SCM service for srcdir to exist, but alternately this can +unpack your sources for scanning. ## Usage for packagers (with SCM service) @@ -29,40 +30,78 @@ $ osc service ra ``` -## Manual (without SCM service) +## Usage for packagers (with OUT SCM service) -If you are not using SCM, you can use the cargo audit service manually. +1. Add to the `_service` file this snippet: + +``` +<services> + <service name="cargo_audit" mode="disabled"> + </service> +</services> +``` + +2. Run `osc` command locally: + +``` +$ osc service ra +``` + +## Manual (without obs services configured) -1. Extract your source archive into your working directory. +1. Run `osc` command locally in your checked out obs package. ``` -$ tar -xv archive.tar.xz +$ osc service lr cargo_audit ``` -2. Examine the folder name it extracted, IE archive-v1.0.0 +## To use a specific lockfile -3. Set srcdir to match +Some projects ship multiple Cargo.lock files, in some cases with intentionally vulnerable content. +Rustsec does this for example to test that they can find these vulnerabilities. + +In these cases you may wish to set a lockfile to scan. This is relative to your unpacked sources +or srcdir. + +### SCM ``` <services> + <service name="obs_scm" mode="disabled"> + ... + </service> <service name="cargo_audit" mode="disabled"> - <param name="srcdir">archive-v1.0.0</param> + <param name="srcdir">projectname</param> + <param name="lockfile">Cargo.lock</param> </service> </services> ``` -4. Run `osc` command locally: +### Source Unpacking ``` -$ osc service ra +<services> + <service name="cargo_audit" mode="disabled"> + <param name="lockfile">projectname-0.7.2/Cargo.lock</param> + </service> +</services> ``` -## Testinc +## Testing To test this module directly by hand: ``` -python3 path/to/cargo_audit --srcdir /path/to/unpacked/sources +python3 path/to/cargo_audit +python3 path/to/cargo_audit --srcdir=/path/to/unpacked/sources +python3 path/to/cargo_audit --srcdir=/path/to/unpacked/sources --lockfile=path/to/Cargo.lock +python3 path/to/cargo_audit --lockfile=path/to/Cargo.lock +``` + +To get further debugging info, run with: + +``` +DEBUG=1 python3 path/to/cargo_audit ``` ## License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/obs-service-cargo_audit-0.1.4~git0.a3c7623/cargo_audit new/obs-service-cargo_audit-0.1.5~git0.01fea16/cargo_audit --- old/obs-service-cargo_audit-0.1.4~git0.a3c7623/cargo_audit 2021-06-08 05:53:44.000000000 +0200 +++ new/obs-service-cargo_audit-0.1.5~git0.01fea16/cargo_audit 2021-07-05 05:26:03.000000000 +0200 @@ -22,6 +22,8 @@ See README.md for additional documentation. """ +import tarfile +import tempfile import logging import argparse import os @@ -35,16 +37,20 @@ service_name = "obs-service-cargo_audit" description = __doc__ -logging.basicConfig(level=logging.INFO) +if os.getenv('DEBUG') is not None: + logging.basicConfig(level=logging.DEBUG) +else: + logging.basicConfig(level=logging.INFO) + log = logging.getLogger(service_name) parser = argparse.ArgumentParser( description=description, formatter_class=argparse.RawDescriptionHelpFormatter ) -parser.add_argument("--srcdir", default=os.getcwd()) +parser.add_argument("--srcdir", default=None) parser.add_argument("--lockfile", default=None) -# We always ignore this parameter. -parser.add_argument("--outdir") +# We always ignore this parameter, but it has to exist. +parser.add_argument("--outdir", default=None) args = parser.parse_args() srcdir = args.srcdir @@ -94,43 +100,70 @@ # Issue may have been found! vuln_count = details["vulnerabilities"]["count"] if vuln_count > 0: - log.error(f" possible vulnerabilties: {vuln_count}") + log.error(f" ???? possible vulnerabilties: {vuln_count}") + log.error(f" {lock_file}") + log.error(f" For more information you SHOULD inspect the output of cargo audit manually") vulns = details["vulnerabilities"]["list"] for vuln in vulns: affects = vuln["advisory"]["package"] cvss = vuln["advisory"]["cvss"] vid = vuln["advisory"]["id"] categories = vuln["advisory"]["categories"] - log.error(f" ???? {vid} -> crate: {affects}, cvss: {cvss}, class: {categories}") - log.error(f" For more information you SHOULD inspect the output of cargo-audit manually for {lock_file}.") + log.error(f" * {vid} -> crate: {affects}, cvss: {cvss}, class: {categories}") return True - log.info(f" ??? No known issues detected in {lock_file}") + log.info(f" ??? No known issues detected in {lock_file}") return False +def do_extract(tgt_dir): + cwd = os.getcwd() + content = os.listdir(cwd) + maybe_src = [ + x for x in content + if '.tar' in x and 'vendor' not in x and not x.endswith('.asc') + ] + for src in maybe_src: + log.debug(f"Unpacking assumed source tar {src} to {tgt_dir}") + with tarfile.open(f"{cwd}/{src}", "r:*") as tar: + tar.extractall(path=tgt_dir) + def main(): log.info(f" Running OBS Source Service ????: {service_name}") log.debug(f" Current working dir: {os.getcwd()}") - cargo_lock_paths = [] - if lockfile: - cargo_lock_paths = [lockfile] - log.info(f" _service configured lock file: {lockfile}") - else: - log.info(f" Searching for Cargo.lock in: {srcdir}") - cargo_lock_paths = find_file(srcdir, "Cargo.lock") - - if not cargo_lock_paths: - log.info(f" No Rust Cargo.lock found under {srcdir}") - log.info(f" Searching for Cargo.toml in: {srcdir}") - if find_file(srcdir, "Cargo.toml"): - generate_lock(srcdir) - else: - log.error(f" No Rust Cargo.toml found under {srcdir}") - exit(1) + lsrcdir = srcdir + status = False + # Setup our temp dir, in case we need it. + with tempfile.TemporaryDirectory() as tmpdirname: + if lsrcdir is None: + # We likely need to unpack sources. + log.debug(f" setting up sources into {tmpdirname}") + lsrcdir = tmpdirname + do_extract(tmpdirname) + + cargo_lock_paths = [] + if lockfile: + log.info(f" _service configured lock file: {lockfile}") + cargo_lock_paths = [os.path.join(lsrcdir, lockfile)] + log.debug(f" {cargo_lock_paths}") else: - log.debug(f" Detected Rust lock files: {cargo_lock_paths}") + log.debug(f" Searching for Cargo.lock in: {lsrcdir}") + cargo_lock_paths = find_file(lsrcdir, "Cargo.lock") + + if not cargo_lock_paths: + log.info(f" No Rust Cargo.lock found under {lsrcdir}") + log.info(f" Searching for Cargo.toml in: {lsrcdir}") + if find_file(lsrcdir, "Cargo.toml"): + generate_lock(lsrcdir) + else: + log.error(f" No Rust Cargo.toml found under {lsrcdir}") + exit(1) + else: + log.debug(f" Detected Rust lock files: {cargo_lock_paths}") + + # Do the scan. + status = any([cargo_audit(cargo_lock_path) for cargo_lock_path in cargo_lock_paths]) - status = any([cargo_audit(cargo_lock_path) for cargo_lock_path in cargo_lock_paths]) + # We exit the context manager, so tempdirname is now removed. if status: log.error(" ?????? Vulnerabilities may have been found. You must review these.") exit(1) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/obs-service-cargo_audit-0.1.4~git0.a3c7623/do_scan.py new/obs-service-cargo_audit-0.1.5~git0.01fea16/do_scan.py --- old/obs-service-cargo_audit-0.1.4~git0.a3c7623/do_scan.py 2021-06-08 05:53:44.000000000 +0200 +++ new/obs-service-cargo_audit-0.1.5~git0.01fea16/do_scan.py 2021-07-05 05:26:03.000000000 +0200 @@ -66,18 +66,7 @@ print(e.stdout) def do_unpack_scan(pkgname): - tgt_dir = f"openSUSE:Factory/{pkgname}" - # List everything in the folder. - content = os.listdir(tgt_dir) - # Find anything that contains .tar - # Exclude vendor.tar.* - maybe_src = [x for x in content if '.tar' in x and 'vendor' not in x] - # Attempt to unpack it into the directory. - for src in maybe_src: - print(f"Unpacking assumed source tar {src} to {tgt_dir}") - with tarfile.open(f"{tgt_dir}/{src}", "r:*") as tar: - tar.extractall(path=tgt_dir) - # Now do osc service lr cargo_audit to run manually. + # This will automatically do the unpack for use due to how cargo_audit as a service works :) try: out = subprocess.check_output(["osc", "service", "lr", "cargo_audit"], cwd=f"openSUSE:Factory/{pkgname}", encoding='UTF-8', stderr=subprocess.STDOUT) print(f"??? -- passed") @@ -85,11 +74,31 @@ print(f"???? -- services failed") print(e.stdout) +# def do_unpack_scan_old(pkgname): +# tgt_dir = f"openSUSE:Factory/{pkgname}" +# # List everything in the folder. +# content = os.listdir(tgt_dir) +# # Find anything that contains .tar +# # Exclude vendor.tar.* +# maybe_src = [x for x in content if '.tar' in x and 'vendor' not in x and not x.endswith('.asc')] +# # Attempt to unpack it into the directory. +# for src in maybe_src: +# print(f"Unpacking assumed source tar {src} to {tgt_dir}") +# with tarfile.open(f"{tgt_dir}/{src}", "r:*") as tar: +# tar.extractall(path=tgt_dir) +# # Now do osc service lr cargo_audit to run manually. +# try: +# out = subprocess.check_output(["osc", "service", "lr", "cargo_audit"], cwd=f"openSUSE:Factory/{pkgname}", encoding='UTF-8', stderr=subprocess.STDOUT) +# print(f"??? -- passed") +# except subprocess.CalledProcessError as e: +# print(f"???? -- services failed") +# print(e.stdout) + if __name__ == '__main__': depends = list_whatdepends() # For testing, we hardcode the list for dev. - depends = ['kanidm', 'librsvg', 'rust-cbindgen'] + # depends = ['kanidm', 'librsvg', 'rust-cbindgen'] # Check them out, or update if they exist. auditable_depends = [] @@ -97,11 +106,11 @@ for pkgname in depends: print("---") checkout_or_update(pkgname) - # do they have cargo_audit as a service? + # do they have cargo_audit as a service? Could we consider adding it? has_audit = does_have_cargo_audit(pkgname) if not has_audit: print(f"?????? https://build.opensuse.org/package/show/openSUSE:Factory/{pkgname} missing cargo_audit service") - print(f"?????? https://build.opensuse.org/package/users/openSUSE:Factory/{pkgname}") + # print(f"?????? https://build.opensuse.org/package/users/openSUSE:Factory/{pkgname}") # If not, we should contact the developers to add this. We can attempt to unpack # and run a scan still though. unpack_depends.append(pkgname)