This is an automated email from the ASF dual-hosted git repository.
cwylie pushed a commit to branch 0.16.0-incubating
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/0.16.0-incubating by this push:
new 43c836e packaging script adjustments (#8436) (#8440)
43c836e is described below
commit 43c836ebb6d18cc903e30c8a44a780cee5286387
Author: Clint Wylie <[email protected]>
AuthorDate: Fri Aug 30 01:30:04 2019 -0700
packaging script adjustments (#8436) (#8440)
* set encoding for license and notice scripts, split generate-license.py
into generate-binary-license.py and check-licenses.py, check-licenses when
-Papache-release is used
* missing docs
* doc fix
* more doc fix
* remove comments
* good catch travis +1
* fix lgtm alerts
---
.travis.yml | 2 +-
distribution/asf-release-process-guide.md | 7 +-
.../bin/{generate-license.py => check-licenses.py} | 230 ++++-----------------
distribution/bin/find-missing-backports.py | 0
distribution/bin/generate-binary-license.py | 184 +++++++++++++++++
...-notice-binary.py => generate-binary-notice.py} | 7 +-
distribution/bin/tag-missing-milestones.py | 0
distribution/pom.xml | 59 +++++-
8 files changed, 277 insertions(+), 212 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index 453cd27..42cb9e9 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -82,7 +82,7 @@ jobs:
#
(https://docs.travis-ci.com/user/reference/overview/#virtualisation-environment-vs-operating-system).
- mkdir -p target
- distribution/bin/generate-license-dependency-reports.py . target
--clean-maven-artifact-transfer --parallel 2
- - distribution/bin/generate-license.py licenses/APACHE2 licenses.yaml
LICENSES.BINARY --dependency-reports target/license-reports
+ - distribution/bin/check-licenses.py licenses.yaml
target/license-reports
- &compile_strict
name: "(openjdk8) strict compilation"
diff --git a/distribution/asf-release-process-guide.md
b/distribution/asf-release-process-guide.md
index 7c161d1..c733e88 100644
--- a/distribution/asf-release-process-guide.md
+++ b/distribution/asf-release-process-guide.md
@@ -152,9 +152,9 @@ If the dependency requires it, copy any licenses to the
`licenses/src` or `licen
| tool | description |
| --- | --- |
-| [generate_license](bin/generate_license.py) | This script is run
_automatically_ when building the distribution package to generate a
`LICENSE.BINARY` file from `licenses.yaml` which is renamed to `LICENSE` in the
binary package. It can also be used to verify that all dependencies are
present and match the versions in `licenses.yaml` (and is run by travis for all
PRs so it _should_ be up to date). |
-| [generate-notice-binary](bin/generate-notice-binary.py) | This script is run
_automatically_ when building the distribution package, and generates a
`NOTICE.BINARY` file by appending the notice content of `licenses.yaml` to the
source `NOTICE` file. This script does _not_ currently verify that all notices
that need to be are present and correct, this must currently be done manually
at release time if not done in the PR that changed a dependency. |
- | [web-console/licenses](../web-console/scripts/licenses) | Updates
`licenses.yaml` with all Druid of the licenses used by the Druid web-console
'binary'. |
+| [generate-binary-license](bin/generate-binary-license.py) | This script is
run _automatically_ when building the distribution package to generate a
`LICENSE.BINARY` file from `licenses.yaml` which is renamed to `LICENSE` in the
binary package. |
+| [generate-binary-notice](bin/generate-binary-notice.py) | This script is run
_automatically_ when building the distribution package, and generates a
`NOTICE.BINARY` file by appending the notice content of `licenses.yaml` to the
source `NOTICE` file. This script does _not_ currently verify that all notices
that need to be are present and correct, this must currently be done manually
at release time if not done in the PR that changed a dependency. |
+| [web-console/licenses](../web-console/script/licenses) | Updates
`licenses.yaml` with all Druid of the licenses used by the Druid web-console
'binary'. |
### Additional tools
@@ -163,6 +163,7 @@ These additional tools were largely used to bootstrap the
initial `LICENSE`, `LI
| tool | description |
| --- | --- |
|
[generate-license-dependency-reports](bin/generate-license-dependency-reports.py)
| Point this to the Druid source root, and give it the location of a temp
scratch directory, and it will output Maven dependency reports for Druid. (I
believe I had to generate Maven dep report separately for hadoop-client) |
+| [check-licenses](bin/check-licenses.py) | Checks `licenses.yaml` against the
output of `generate-license-dependency-reports.py`, used by travis and
`-Papache-release` when building distribution, to verify that all dependencies
are present and match the versions in `licenses.yaml`. |
| [jar-notice-lister](bin/jar-notice-lister.py) | Point this to an extracted
Druid binary distribution, and give it a temp scratch directory, and it will
output NOTICE information for all the Druid JAR files. |
diff --git a/distribution/bin/generate-license.py
b/distribution/bin/check-licenses.py
similarity index 67%
rename from distribution/bin/generate-license.py
rename to distribution/bin/check-licenses.py
index 6c5a643..ea48058 100755
--- a/distribution/bin/generate-license.py
+++ b/distribution/bin/check-licenses.py
@@ -22,7 +22,6 @@ import sys
from html.parser import HTMLParser
import argparse
-
class DependencyReportParser(HTMLParser):
# This class parses the given html file to find all dependency reports
under "Project dependencies"
# and "Projection transparent dependencies" sections.
@@ -60,7 +59,7 @@ class DependencyReportParser(HTMLParser):
if self.state == "none":
if tag == "h2":
self.state = "h2_start"
-
+
if self.state == "h2_start":
if tag == "a":
for attr in attrs:
@@ -71,13 +70,13 @@ class DependencyReportParser(HTMLParser):
if self.state == "h2_end":
if tag == "h3":
self.state = "h3_start"
-
+
if self.state == "h3_start":
if tag == "a":
for attr in attrs:
if attr[0] == "name" and attr[1] == "compile":
self.state = "compile_start"
-
+
if self.state == "h3_end":
if tag == "table":
self.state = "table_start"
@@ -91,7 +90,7 @@ class DependencyReportParser(HTMLParser):
if tag == "tr":
self.state = "row_start"
self.clear_attr()
-
+
if self.state == "row_start":
if tag == "td":
self.state = "td_start"
@@ -101,7 +100,7 @@ class DependencyReportParser(HTMLParser):
if self.state == "th_end":
if tag == "th":
self.state = "th_start"
-
+
if self.state == "td_end":
if tag == "td":
self.state = "td_start"
@@ -115,11 +114,11 @@ class DependencyReportParser(HTMLParser):
if self.state == "h2_start":
if tag == "h2":
self.state = "h2_end"
-
+
if self.state == "project_dependencies_end":
if tag == "h2":
self.state = "h2_end"
-
+
if self.state == "compile_start":
if tag == "a":
self.state = "compile_end"
@@ -127,7 +126,7 @@ class DependencyReportParser(HTMLParser):
if self.state == "compile_end":
if tag == "h3":
self.state = "h3_end"
-
+
if self.state == "table_start":
if tag == "table":
self.state = "none"
@@ -136,7 +135,7 @@ class DependencyReportParser(HTMLParser):
if tag == "td":
self.state = "td_end"
self.attr_index = self.attr_index + 1
-
+
if self.state == "th_start":
if tag == "th":
self.state = "th_end"
@@ -148,14 +147,14 @@ class DependencyReportParser(HTMLParser):
if self.state == "th_end":
if tag == "tr":
self.state = "row_end"
-
+
if self.state == "td_end":
if tag == "tr":
self.state = "row_end"
# print(json.dumps({"groupId": self.group_id, "artifactId":
self.artifact_id, "version": self.version, "classifier": self.classifier,
"type": self.dep_type, "license": self.license}))
if self.group_id.find("org.apache.druid") < 0:
self.dep_to_license[get_dep_key(self.group_id,
self.artifact_id, self.version)] = (self.license, self.druid_module_name)
-
+
if self.state == "row_end":
if tag == "table":
self.state = "none"
@@ -208,11 +207,8 @@ class DependencyReportParser(HTMLParser):
self.license = self.compatible_license_names[data]
-outfile = None
-
-def get_dep_key(group_id, artifact_id, version):
- return (group_id, artifact_id, version)
-
+def print_log_to_stderr(string):
+ print(string, file=sys.stderr)
def build_compatible_license_names():
compatible_licenses = {}
@@ -272,24 +268,8 @@ def build_compatible_license_names():
compatible_licenses['-'] = '-'
return compatible_licenses
-
-def module_to_upper(module):
- extensions_offset = module.lower().find("extensions")
- if extensions_offset < 0:
- return module.upper()
- elif extensions_offset == 0:
- return module[0:len("extensions")].upper() +
module[len("extensions"):len(module)]
- else:
- raise Exception("Expected extensions at 0, but
{}".format(extensions_offset))
-
-
-def print_outfile(string):
- print(string, file=outfile)
-
-
-def print_error(string):
- print(string, file=sys.stderr)
-
+def get_dep_key(group_id, artifact_id, version):
+ return (group_id, artifact_id, version)
def get_version_string(version):
if type(version) == str:
@@ -297,92 +277,6 @@ def get_version_string(version):
else:
return str(version)
-
-def print_license_phrase(license_phrase):
- remaining = license_phrase
- while len(remaining) > 0:
- # print("remaining: {}".format(remaining))
- # print("len: {}".format(len(remaining)))
- if len(remaining) > 120:
- chars_of_200 = remaining[0:120]
- phrase_len = chars_of_200.rfind(" ")
- if phrase_len < 0:
- raise Exception("Can't find whitespace in
{}".format(chars_of_200))
- print_outfile(" {}".format(remaining[0:phrase_len]))
- remaining = remaining[phrase_len:]
- else:
- print_outfile(" {}".format(remaining))
- remaining = ""
-
-
-def is_non_empty(dic, key):
- if key in dic and dic[key] is not None:
- if type(dic[key]) == str:
- return len(dic[key]) > 0
- else:
- return True
- else:
- return False
-
-
-def print_license(license):
- license_phrase = "This product"
- if license['license_category'] == "source":
- license_phrase += " contains"
- elif license['license_category'] == "binary":
- license_phrase += " bundles"
- license_phrase += " {}".format(license['name'])
- if is_non_empty(license, 'version'):
- license_phrase += " version {}".format(license['version'])
- if is_non_empty(license, 'copyright'):
- license_phrase += ", copyright {}".format(license['copyright'])
- if is_non_empty(license, 'additional_copyright_statement'):
- license_phrase += ",
{}".format(license['additional_copyright_statement'])
- if license['license_name'] != 'Apache License version 2.0':
- license_phrase += " which is available under
{}".format(license['license_name'])
- if is_non_empty(license, 'additional_license_statement'):
- license_phrase += ",
{}".format(license['additional_license_statement'])
- if is_non_empty(license, 'license_file_path'):
- license_file_list = []
- if type(license['license_file_path']) == list:
- license_file_list.extend(license['license_file_path'])
- else:
- license_file_list.append(license['license_file_path'])
- if len(license_file_list) == 1:
- license_phrase += ". For details, see
{}".format(license_file_list[0])
- else:
- license_phrase += ". For details, "
- for each_file in license_file_list:
- if each_file == license_file_list[-1]:
- license_phrase += ", and {}".format(each_file)
- elif each_file == license_file_list[0]:
- license_phrase += "see {}".format(each_file)
- else:
- license_phrase += ", {}".format(each_file)
-
- license_phrase += "."
-
- print_license_phrase(license_phrase)
-
- if 'source_paths' in license:
- for source_path in license['source_paths']:
- if type(source_path) is dict:
- for class_name, path in source_path.items():
- print_outfile(" {}:".format(class_name))
- print_outfile(" * {}".format(path))
- else:
- print_outfile(" * {}".format(source_path))
-
- if 'libraries' in license:
- for library in license['libraries']:
- if type(library) is not dict:
- raise Exception("Expected dict but got
{}[{}]".format(type(library), library))
- if len(library) > 1:
- raise Exception("Expected 1 groupId and artifactId, but got
[{}]".format(library))
- for group_id, artifact_id in library.items():
- print_outfile(" * {}:{}".format(group_id, artifact_id))
-
-
def find_druid_module_name(dirpath):
ext_start = dirpath.find("/ext/")
if ext_start > 0:
@@ -397,11 +291,10 @@ def find_druid_module_name(dirpath):
# Druid core
return "core"
-
def check_licenses(license_yaml, dependency_reports_root):
# Build a dictionary to facilitate comparing reported licenses and
registered ones.
# These dictionaries are the mapping of (group_id, artifact_id, version)
to license_name.
-
+
# Build reported license dictionary.
reported_dep_to_licenses = {}
compatible_license_names = build_compatible_license_names()
@@ -411,7 +304,7 @@ def check_licenses(license_yaml, dependency_reports_root):
full_path = os.path.join(dirpath, filename)
# Determine if it's druid core or an extension
druid_module_name = find_druid_module_name(dirpath)
- print_error("Parsing {}".format(full_path))
+ print_log_to_stderr("Parsing {}".format(full_path))
with open(full_path) as report_file:
parser = DependencyReportParser(druid_module_name,
compatible_license_names)
reported_dep_to_licenses.update(parser.parse(report_file))
@@ -419,7 +312,7 @@ def check_licenses(license_yaml, dependency_reports_root):
if len(reported_dep_to_licenses) == 0:
raise Exception("No dependency reports are found")
- print_error("Found {} reported
licenses\n".format(len(reported_dep_to_licenses)))
+ print_log_to_stderr("Found {} reported
licenses\n".format(len(reported_dep_to_licenses)))
# Build registered license dictionary.
registered_dep_to_licenses = {}
@@ -449,7 +342,7 @@ def check_licenses(license_yaml, dependency_reports_root):
if len(registered_dep_to_licenses) == 0:
raise Exception("No registered licenses are found")
-
+
# Compare licenses in registry and those in dependency reports.
mismatched_licenses = []
missing_licenses = []
@@ -465,25 +358,25 @@ def check_licenses(license_yaml, dependency_reports_root):
artifact_id = key[1]
version = key[2]
mismatched_licenses.append((druid_module, group_id,
artifact_id, version, reported_license, registered_license))
-
+
# If we find any mismatched license, stop immediately.
if len(mismatched_licenses) > 0:
- print_error("Error: found {} mismatches between reported licenses and
registered licenses".format(len(mismatched_licenses)))
+ print_log_to_stderr("Error: found {} mismatches between reported
licenses and registered licenses".format(len(mismatched_licenses)))
for mismatched_license in mismatched_licenses:
- print_error("druid_module: {}, groupId: {}, artifactId: {},
version: {}, reported_license: {}, registered_license:
{}".format(mismatched_license[0], mismatched_license[1], mismatched_license[2],
mismatched_license[3], mismatched_license[4], mismatched_license[5]))
- print_error("")
-
+ print_log_to_stderr("druid_module: {}, groupId: {}, artifactId:
{}, version: {}, reported_license: {}, registered_license:
{}".format(mismatched_license[0], mismatched_license[1], mismatched_license[2],
mismatched_license[3], mismatched_license[4], mismatched_license[5]))
+ print_log_to_stderr("")
+
# Let's find missing licenses, which are reported but missing in the
registry.
for key, reported_license_druid_module in reported_dep_to_licenses.items():
if reported_license_druid_module[0] != "-" and key not in
registered_dep_to_licenses and key not in skipping_licenses:
missing_licenses.append((reported_license_druid_module[1], key[0],
key[1], key[2], reported_license_druid_module[0]))
if len(missing_licenses) > 0:
- print_error("Error: found {} missing licenses. These licenses are
reported, but missing in the registry".format(len(missing_licenses)))
+ print_log_to_stderr("Error: found {} missing licenses. These licenses
are reported, but missing in the registry".format(len(missing_licenses)))
for missing_license in missing_licenses:
- print_error("druid_module: {}, groupId: {}, artifactId: {},
version: {}, license: {}".format(missing_license[0], missing_license[1],
missing_license[2], missing_license[3], missing_license[4]))
- print_error("")
-
+ print_log_to_stderr("druid_module: {}, groupId: {}, artifactId:
{}, version: {}, license: {}".format(missing_license[0], missing_license[1],
missing_license[2], missing_license[3], missing_license[4]))
+ print_log_to_stderr("")
+
# Let's find unchecked licenses, which are registered but missing in the
report.
# These licenses should be checked manually.
for key, registered_license in registered_dep_to_licenses.items():
@@ -491,78 +384,29 @@ def check_licenses(license_yaml, dependency_reports_root):
unchecked_licenses.append((key[0], key[1], key[2],
registered_license))
elif reported_dep_to_licenses[key][0] == "-":
unchecked_licenses.append((key[0], key[1], key[2],
registered_license))
-
+
if len(unchecked_licenses) > 0:
- print_error("Warn: found {} unchecked licenses. These licenses are
registered, but not found in dependency
reports.".format(len(unchecked_licenses)))
- print_error("These licenses must be checked manually.")
+ print_log_to_stderr("Warn: found {} unchecked licenses. These licenses
are registered, but not found in dependency
reports.".format(len(unchecked_licenses)))
+ print_log_to_stderr("These licenses must be checked manually.")
for unchecked_license in unchecked_licenses:
- print_error("groupId: {}, artifactId: {}, version: {},
reported_license: {}".format(unchecked_license[0], unchecked_license[1],
unchecked_license[2], unchecked_license[3]))
- print_error("")
+ print_log_to_stderr("groupId: {}, artifactId: {}, version: {},
reported_license: {}".format(unchecked_license[0], unchecked_license[1],
unchecked_license[2], unchecked_license[3]))
+ print_log_to_stderr("")
if len(mismatched_licenses) > 0 or len(missing_licenses) > 0:
sys.exit(1)
-def print_license_name_underbar(license_name):
- underbar = ""
- for _ in range(len(license_name)):
- underbar += "="
- print_outfile("{}\n".format(underbar))
-
-
-def generate_license(apache_license_v2, license_yaml):
- # Generate LICENSE.BINARY file
- print_error("=== Generating the contents of LICENSE.BINARY file ===\n")
-
- # Print Apache license first.
- print_outfile(apache_license_v2)
- with open(license_yaml) as registry_file:
- licenses_list = list(yaml.load_all(registry_file))
-
- # Group licenses by license_name, license_category, and then module.
- licenses_map = {}
- for license in licenses_list:
- if license['license_name'] not in licenses_map:
- licenses_map[license['license_name']] = {}
- licenses_of_name = licenses_map[license['license_name']]
- if license['license_category'] not in licenses_of_name:
- licenses_of_name[license['license_category']] = {}
- licenses_of_category = licenses_of_name[license['license_category']]
- if license['module'] not in licenses_of_category:
- licenses_of_category[license['module']] = []
- licenses_of_module = licenses_of_category[license['module']]
- licenses_of_module.append(license)
-
- for license_name, licenses_of_name in sorted(licenses_map.items()):
- print_outfile(license_name)
- print_license_name_underbar(license_name)
- for license_category, licenses_of_category in licenses_of_name.items():
- for module, licenses in licenses_of_category.items():
- print_outfile("{}/{}".format(license_category.upper(),
module_to_upper(module)))
- for license in licenses:
- print_license(license)
- print_outfile("")
- print_outfile("")
-
-
if __name__ == "__main__":
try:
parser = argparse.ArgumentParser(description='Check and generate
license file.')
- parser.add_argument('apache_license', metavar='<path to apache license
file>', type=str)
parser.add_argument('license_yaml', metavar='<path to license.yaml>',
type=str)
- parser.add_argument('out_path', metavar='<path to output file>',
type=str)
- parser.add_argument('--dependency-reports',
dest='dependency_reports_root', type=str, default=None, metavar='<root to maven
dependency reports>')
+ parser.add_argument('dependency_reports_root', metavar='<root to maven
dependency reports>', type=str)
args = parser.parse_args()
-
- with open(args.apache_license) as apache_license_file:
- apache_license_v2 = apache_license_file.read()
+
license_yaml = args.license_yaml
dependency_reports_root = args.dependency_reports_root
- with open(args.out_path, "w") as outfile:
- if dependency_reports_root is not None:
- check_licenses(license_yaml, dependency_reports_root)
- generate_license(apache_license_v2, license_yaml)
+ check_licenses(license_yaml, dependency_reports_root)
except KeyboardInterrupt:
- print('Interrupted, closing.')
+ print('Interrupted, closing.')
\ No newline at end of file
diff --git a/distribution/bin/find-missing-backports.py
b/distribution/bin/find-missing-backports.py
old mode 100644
new mode 100755
diff --git a/distribution/bin/generate-binary-license.py
b/distribution/bin/generate-binary-license.py
new file mode 100755
index 0000000..e69bc70
--- /dev/null
+++ b/distribution/bin/generate-binary-license.py
@@ -0,0 +1,184 @@
+#!/usr/bin/env python3
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import yaml
+import sys
+import argparse
+
+outfile = None
+
+def print_outfile(string):
+ print(string, file=outfile)
+
+def print_log_to_stderr(string):
+ print(string, file=sys.stderr)
+
+def get_dep_key(group_id, artifact_id, version):
+ return (group_id, artifact_id, version)
+
+def get_version_string(version):
+ if type(version) == str:
+ return version
+ else:
+ return str(version)
+
+def module_to_upper(module):
+ extensions_offset = module.lower().find("extensions")
+ if extensions_offset < 0:
+ return module.upper()
+ elif extensions_offset == 0:
+ return module[0:len("extensions")].upper() +
module[len("extensions"):len(module)]
+ else:
+ raise Exception("Expected extensions at 0, but
{}".format(extensions_offset))
+
+def is_non_empty(dic, key):
+ if key in dic and dic[key] is not None:
+ if type(dic[key]) == str:
+ return len(dic[key]) > 0
+ else:
+ return True
+ else:
+ return False
+
+def print_license_phrase(license_phrase):
+ remaining = license_phrase
+ while len(remaining) > 0:
+ if len(remaining) > 120:
+ chars_of_200 = remaining[0:120]
+ phrase_len = chars_of_200.rfind(" ")
+ if phrase_len < 0:
+ raise Exception("Can't find whitespace in
{}".format(chars_of_200))
+ print_outfile(" {}".format(remaining[0:phrase_len]))
+ remaining = remaining[phrase_len:]
+ else:
+ print_outfile(" {}".format(remaining))
+ remaining = ""
+
+def print_license(license):
+ license_phrase = "This product"
+ if license['license_category'] == "source":
+ license_phrase += " contains"
+ elif license['license_category'] == "binary":
+ license_phrase += " bundles"
+ license_phrase += " {}".format(license['name'])
+ if is_non_empty(license, 'version'):
+ license_phrase += " version {}".format(license['version'])
+ if is_non_empty(license, 'copyright'):
+ license_phrase += ", copyright {}".format(license['copyright'])
+ if is_non_empty(license, 'additional_copyright_statement'):
+ license_phrase += ",
{}".format(license['additional_copyright_statement'])
+ if license['license_name'] != 'Apache License version 2.0':
+ license_phrase += " which is available under
{}".format(license['license_name'])
+ if is_non_empty(license, 'additional_license_statement'):
+ license_phrase += ",
{}".format(license['additional_license_statement'])
+ if is_non_empty(license, 'license_file_path'):
+ license_file_list = []
+ if type(license['license_file_path']) == list:
+ license_file_list.extend(license['license_file_path'])
+ else:
+ license_file_list.append(license['license_file_path'])
+ if len(license_file_list) == 1:
+ license_phrase += ". For details, see
{}".format(license_file_list[0])
+ else:
+ license_phrase += ". For details, "
+ for each_file in license_file_list:
+ if each_file == license_file_list[-1]:
+ license_phrase += ", and {}".format(each_file)
+ elif each_file == license_file_list[0]:
+ license_phrase += "see {}".format(each_file)
+ else:
+ license_phrase += ", {}".format(each_file)
+
+ license_phrase += "."
+
+ print_license_phrase(license_phrase)
+
+ if 'source_paths' in license:
+ for source_path in license['source_paths']:
+ if type(source_path) is dict:
+ for class_name, path in source_path.items():
+ print_outfile(" {}:".format(class_name))
+ print_outfile(" * {}".format(path))
+ else:
+ print_outfile(" * {}".format(source_path))
+
+ if 'libraries' in license:
+ for library in license['libraries']:
+ if type(library) is not dict:
+ raise Exception("Expected dict but got
{}[{}]".format(type(library), library))
+ if len(library) > 1:
+ raise Exception("Expected 1 groupId and artifactId, but got
[{}]".format(library))
+ for group_id, artifact_id in library.items():
+ print_outfile(" * {}:{}".format(group_id, artifact_id))
+
+def print_license_name_underbar(license_name):
+ underbar = ""
+ for _ in range(len(license_name)):
+ underbar += "="
+ print_outfile("{}\n".format(underbar))
+
+def generate_license(apache_license_v2, license_yaml):
+ print_log_to_stderr("=== Generating the contents of LICENSE.BINARY file
===\n")
+
+ # Print Apache license first.
+ print_outfile(apache_license_v2)
+ with open(license_yaml, encoding='utf-8') as registry_file:
+ licenses_list = list(yaml.load_all(registry_file))
+
+ # Group licenses by license_name, license_category, and then module.
+ licenses_map = {}
+ for license in licenses_list:
+ if license['license_name'] not in licenses_map:
+ licenses_map[license['license_name']] = {}
+ licenses_of_name = licenses_map[license['license_name']]
+ if license['license_category'] not in licenses_of_name:
+ licenses_of_name[license['license_category']] = {}
+ licenses_of_category = licenses_of_name[license['license_category']]
+ if license['module'] not in licenses_of_category:
+ licenses_of_category[license['module']] = []
+ licenses_of_module = licenses_of_category[license['module']]
+ licenses_of_module.append(license)
+
+ for license_name, licenses_of_name in sorted(licenses_map.items()):
+ print_outfile(license_name)
+ print_license_name_underbar(license_name)
+ for license_category, licenses_of_category in licenses_of_name.items():
+ for module, licenses in licenses_of_category.items():
+ print_outfile("{}/{}".format(license_category.upper(),
module_to_upper(module)))
+ for license in licenses:
+ print_license(license)
+ print_outfile("")
+ print_outfile("")
+
+
+if __name__ == "__main__":
+ try:
+ parser = argparse.ArgumentParser(description='Check and generate
license file.')
+ parser.add_argument('apache_license', metavar='<path to apache license
file>', type=str)
+ parser.add_argument('license_yaml', metavar='<path to license.yaml>',
type=str)
+ parser.add_argument('out_path', metavar='<path to output file>',
type=str)
+ args = parser.parse_args()
+
+ with open(args.apache_license, encoding="ascii") as
apache_license_file:
+ apache_license_v2 = apache_license_file.read()
+ license_yaml = args.license_yaml
+
+ with open(args.out_path, "w", encoding="utf-8") as outfile:
+ generate_license(apache_license_v2, license_yaml)
+
+ except KeyboardInterrupt:
+ print('Interrupted, closing.')
diff --git a/distribution/bin/generate-notice-binary.py
b/distribution/bin/generate-binary-notice.py
similarity index 94%
rename from distribution/bin/generate-notice-binary.py
rename to distribution/bin/generate-binary-notice.py
index 2b11902..a777881 100755
--- a/distribution/bin/generate-notice-binary.py
+++ b/distribution/bin/generate-binary-notice.py
@@ -51,13 +51,12 @@ def print_notice(dependency):
print_outfile("{} {}-{}.jar {}".format(dependencyHeaderLine,
jar, dependency['version'], dependencyHeaderLine))
print_outfile("{}\n\n\n\n".format(notice))
-
def generate_notice(source_notice, dependences_yaml):
print_log_to_stderr("=== Generating the contents of NOTICE.BINARY file
===\n")
# Print Apache license first.
print_outfile(source_notice)
- with open(dependences_yaml) as registry_file:
+ with open(dependences_yaml, encoding='utf-8') as registry_file:
dependencies = list(yaml.load_all(registry_file))
# Group dependencies by module
@@ -81,11 +80,11 @@ if __name__ == "__main__":
parser.add_argument('out_path', metavar='<path to output file>',
type=str)
args = parser.parse_args()
- with open(args.notice) as apache_notice_file:
+ with open(args.notice, encoding="ascii") as apache_notice_file:
source_notice = apache_notice_file.read()
dependencies_yaml = args.license_yaml
- with open(args.out_path, "w") as outfile:
+ with open(args.out_path, "w", encoding="utf-8") as outfile:
generate_notice(source_notice, dependencies_yaml)
except KeyboardInterrupt:
diff --git a/distribution/bin/tag-missing-milestones.py
b/distribution/bin/tag-missing-milestones.py
old mode 100644
new mode 100755
diff --git a/distribution/pom.xml b/distribution/pom.xml
index 0f9c2d4..46e9c2a 100644
--- a/distribution/pom.xml
+++ b/distribution/pom.xml
@@ -117,13 +117,13 @@
<artifactId>exec-maven-plugin</artifactId>
<executions>
<execution>
- <id>versionify-readme</id>
+ <id>generate-readme</id>
<phase>initialize</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
-
<executable>${project.parent.basedir}/distribution/bin/build-textfile-readme.sh</executable>
+
<executable>${project.basedir}/bin/build-textfile-readme.sh</executable>
<arguments>
<argument>${project.basedir}/../</argument>
<argument>${project.parent.version}</argument>
@@ -131,17 +131,17 @@
</configuration>
</execution>
<execution>
- <id>generate-license</id>
+ <id>generate-binary-license</id>
<phase>initialize</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
-
<executable>${project.parent.basedir}/distribution/bin/generate-license.py</executable>
+
<executable>${project.basedir}/bin/generate-binary-license.py</executable>
<arguments>
-
<argument>${project.basedir}/../licenses/APACHE2</argument>
-
<argument>${project.basedir}/../licenses.yaml</argument>
-
<argument>${project.basedir}/../LICENSE.BINARY</argument>
+
<argument>${project.parent.basedir}/licenses/APACHE2</argument>
+
<argument>${project.parent.basedir}/licenses.yaml</argument>
+
<argument>${project.parent.basedir}/LICENSE.BINARY</argument>
</arguments>
</configuration>
</execution>
@@ -152,11 +152,11 @@
<goal>exec</goal>
</goals>
<configuration>
-
<executable>${project.parent.basedir}/distribution/bin/generate-notice-binary.py</executable>
+
<executable>${project.basedir}/bin/generate-binary-notice.py</executable>
<arguments>
-
<argument>${project.basedir}/../NOTICE</argument>
-
<argument>${project.basedir}/../licenses.yaml</argument>
-
<argument>${project.basedir}/../NOTICE.BINARY</argument>
+
<argument>${project.parent.basedir}/NOTICE</argument>
+
<argument>${project.parent.basedir}/licenses.yaml</argument>
+
<argument>${project.parent.basedir}/NOTICE.BINARY</argument>
</arguments>
</configuration>
</execution>
@@ -277,6 +277,43 @@
<build>
<plugins>
<plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>exec-maven-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>generate-licenses-report</id>
+ <phase>initialize</phase>
+ <goals>
+ <goal>exec</goal>
+ </goals>
+ <configuration>
+
<executable>${project.basedir}/bin/generate-license-dependency-reports.py</executable>
+ <arguments>
+
<argument>${project.basedir}/../</argument>
+
<argument>${project.basedir}/target</argument>
+
<argument>--clean-maven-artifact-transfer</argument>
+ <argument>--parallel</argument>
+ <argument>2</argument>
+ </arguments>
+ </configuration>
+ </execution>
+ <execution>
+ <id>check-licenses</id>
+ <phase>test</phase>
+ <goals>
+ <goal>exec</goal>
+ </goals>
+ <configuration>
+
<executable>${project.basedir}/bin/check-licenses.py</executable>
+ <arguments>
+
<argument>${project.parent.basedir}/licenses.yaml</argument>
+
<argument>${project.basedir}/target/license-reports</argument>
+ </arguments>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
