This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new e868c8da79a Update mass_comment.py when we cut release branch (#26506)
e868c8da79a is described below
commit e868c8da79a8cf727044cf805b2041fbdb8ff181
Author: Danny McCormick <dannymccorm...@google.com>
AuthorDate: Wed May 3 15:47:46 2023 -0400
Update mass_comment.py when we cut release branch (#26506)
* Update mass_comment when we cut release branch
* Update release guide
---
.github/workflows/cut_release_branch.yml | 43 +++++-
build.gradle.kts | 3 +
release/src/main/scripts/jenkins_jobs.txt | 167 +++++++++++++++++++++
release/src/main/scripts/mass_comment.py | 165 ++------------------
.../site/content/en/contribute/release-guide.md | 14 +-
5 files changed, 231 insertions(+), 161 deletions(-)
diff --git a/.github/workflows/cut_release_branch.yml
b/.github/workflows/cut_release_branch.yml
index 77a510b945f..9d34e0e2478 100644
--- a/.github/workflows/cut_release_branch.yml
+++ b/.github/workflows/cut_release_branch.yml
@@ -37,6 +37,12 @@ on:
description: Next release version
required: true
default: '2.XX.0'
+ JENKINS_USERNAME:
+ description: Username of the current Jenkins user. Used to update
mass_comment.py with all Jenkins jobs.
+ required: true
+ JENKINS_TOKEN:
+ description: API Token for the current Jenkins user. Can be generated
at https://ci-beam.apache.org/user/<user>/configure
+ required: true
jobs:
update_master:
@@ -46,6 +52,11 @@ jobs:
NEXT_RELEASE: ${{ github.event.inputs.NEXT_VERSION }}
SCRIPT_DIR: ./release/src/main/scripts
steps:
+ - name: Mask Jenkins token
+ run: |
+ echo "::add-mask::$JENKINS_TOKEN"
+ env:
+ JENKINS_TOKEN: ${{ github.event.inputs.JENKINS_TOKEN }}
- name: Validate Next Version
run: |
if [[ $NEXT_RELEASE =~ ([0-9]+\.[0-9]+\.[0-9]+) ]]; then
@@ -60,6 +71,36 @@ jobs:
run: |
git config user.name $GITHUB_ACTOR
git config user.email actions@"$RUNNER_NAME".local
+ - name: Install xmllint
+ run: sudo apt-get install -y libxml2-utils
+ - name: Update mass_comment.py with new Jenkins jobs
+ run: |
+ rm release/src/main/scripts/jenkins_jobs.txt
+ for obj in $(curl https://ci-beam.apache.org/api/json | jq '.jobs[]'
-c); do
+ echo $obj
+ url=$(echo $obj | jq '.url' -r || echo "")
+ name=$(echo $obj | jq '.name' -r || echo "")
+ if [ "$url" != "" ] && [ "$name" != "" ]; then
+ curl --user $JENKINS_USERNAME:$JENKINS_TOKEN $url/config.xml >
/tmp/config.xml
+
+ if [ "$(xmllint --xpath 'string(//disabled)' /tmp/config.xml)" =
"true" ]; then
+ echo "SKIPPING $url - disabled"
+ else
+ phrase=$(xmllint --xpath
'string(//triggers/org.jenkinsci.plugins.ghprb.GhprbTrigger/triggerPhrase)'
/tmp/config.xml)
+ if [ "$phrase" != "" ]; then
+ echo "Appending ${phrase},${name}"
+ echo "${phrase},${name}" >> /tmp/result
+ fi
+ fi
+
+ rm /tmp/config.xml
+ fi
+ done
+
+ cat /tmp/result | sort | uniq | grep -i -E
'precommit|postcommit|validates|vr|example|test|gradle build' | grep -v -i -E
'load|perf|website' >> release/src/main/scripts/jenkins_jobs.txt
+ env:
+ JENKINS_USERNAME: ${{ github.event.inputs.JENKINS_USERNAME }}
+ JENKINS_TOKEN: ${{ github.event.inputs.JENKINS_TOKEN }}
- name: Update master branch
run: |
bash "${SCRIPT_DIR}/set_version.sh" "${NEXT_VERSION_IN_BASE_BRANCH}"
@@ -152,4 +193,4 @@ jobs:
echo "NOTE: If there is no jenkins job started, please comment
generated PR with: Run Gradle Publish"
echo "Things remained you need to do manually after build
successful:"
echo "1. Close this generated PR in github website."
- echo "2. Delete your remote branch ${BRANCH_NAME} form your beam
repo in github website."
+ echo "2. Delete your remote branch ${BRANCH_NAME} form your beam
repo in github website."
\ No newline at end of file
diff --git a/build.gradle.kts b/build.gradle.kts
index d3b4dcadb9c..64da616ed2a 100644
--- a/build.gradle.kts
+++ b/build.gradle.kts
@@ -89,6 +89,9 @@ tasks.rat {
"website/www/site/static/js/keen-slider.min.js",
"website/www/site/assets/scss/_keen-slider.scss",
+ // Release automation files
+ "release/src/main/scripts/*.txt",
+
// Ignore ownership files
"ownership/**/*",
"**/OWNERS",
diff --git a/release/src/main/scripts/jenkins_jobs.txt
b/release/src/main/scripts/jenkins_jobs.txt
new file mode 100644
index 00000000000..0c77b1005eb
--- /dev/null
+++ b/release/src/main/scripts/jenkins_jobs.txt
@@ -0,0 +1,167 @@
+Run Beam Metrics deployment,beam_PostCommit_BeamMetrics_Publish_PR
+Run Chicago Taxi on Dataflow,beam_PostCommit_Python_Chicago_Taxi_Dataflow_PR
+Run Chicago Taxi on Flink,beam_PostCommit_Python_Chicago_Taxi_Flink_PR
+Run CommunityMetrics PreCommit,beam_PreCommit_CommunityMetrics_Phrase
+Run Dataflow Runner Nexmark Tests,beam_PostCommit_Java_Nexmark_Dataflow_PR
+Run Dataflow Runner Tpcds Tests,beam_PostCommit_Java_Tpcds_Dataflow_PR
+Run Dataflow Runner V2 Java 11 Nexmark
Tests,beam_PostCommit_Java_Nexmark_DataflowV2_Java11_PR
+Run Dataflow Runner V2 Java 17 Nexmark
Tests,beam_PostCommit_Java_Nexmark_DataflowV2_Java17_PR
+Run Dataflow Runner V2 Nexmark Tests,beam_PostCommit_Java_Nexmark_DataflowV2_PR
+Run Dataflow Streaming
ValidatesRunner,beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming_PR
+Run Dataflow ValidatesRunner Java
11,beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11_PR
+Run Dataflow ValidatesRunner Java
17,beam_PostCommit_Java_ValidatesRunner_Dataflow_Java17_PR
+Run Dataflow ValidatesRunner,beam_PostCommit_Java_ValidatesRunner_Dataflow_PR
+Run Direct Runner Nexmark Tests,beam_PostCommit_Java_Nexmark_Direct_PR
+Run Direct ValidatesRunner Java
11,beam_PostCommit_Java_ValidatesRunner_Direct_Java11_PR
+Run Direct ValidatesRunner Java
17,beam_PostCommit_Java_ValidatesRunner_Direct_Java17_PR
+Run Direct ValidatesRunner in Java
11,beam_PostCommit_Java11_ValidatesRunner_Direct_PR
+Run Direct ValidatesRunner,beam_PostCommit_Java_ValidatesRunner_Direct_PR
+Run Flink Runner Nexmark Tests,beam_PostCommit_Java_Nexmark_Flink_PR
+Run Flink Runner Tpcds Tests,beam_PostCommit_Java_Tpcds_Flink_PR
+Run Flink ValidatesRunner Java
11,beam_PostCommit_Java_ValidatesRunner_Flink_Java11_PR
+Run Flink ValidatesRunner,beam_PostCommit_Java_ValidatesRunner_Flink_PR
+Run Go Flink ValidatesRunner,beam_PostCommit_Go_VR_Flink_PR
+Run Go PostCommit,beam_PostCommit_Go_PR
+Run Go PreCommit,beam_PreCommit_Go_Phrase
+Run Go Samza ValidatesRunner,beam_PostCommit_Go_VR_Samza_PR
+Run Go Spark ValidatesRunner,beam_PostCommit_Go_VR_Spark_PR
+Run GoPortable PreCommit,beam_PreCommit_GoPortable_Phrase
+Run Java 11 Examples on Dataflow Runner
V2,beam_PostCommit_Java_Examples_Dataflow_V2_java11_PR
+Run Java 17 Examples on Dataflow Runner
V2,beam_PostCommit_Java_Examples_Dataflow_V2_java17_PR
+Run Java Dataflow V2 ValidatesRunner
Streaming,beam_PostCommit_Java_VR_Dataflow_V2_Streaming_PR
+Run Java Dataflow V2 ValidatesRunner,beam_PostCommit_Java_VR_Dataflow_V2_PR
+Run Java Examples on Dataflow Runner
V2,beam_PostCommit_Java_Examples_Dataflow_V2_PR
+Run Java Examples_Direct,beam_PostCommit_Java_Examples_Direct_PR
+Run Java Examples_Flink,beam_PostCommit_Java_Examples_Flink_PR
+Run Java Examples_Spark,beam_PostCommit_Java_Examples_Spark_PR
+Run Java Flink PortableValidatesRunner
Streaming,beam_PostCommit_Java_PVR_Flink_Streaming_PR
+Run Java InfluxDbIO_IT,beam_PostCommit_Java_InfluxDbIO_IT_PR
+Run Java Portability examples on Dataflow with Java
11,beam_PostCommit_Java11_Examples_Dataflow_Portability_PR
+Run Java PostCommit,beam_PostCommit_Java_PR
+Run Java PreCommit,beam_PreCommit_Java_Phrase
+Run Java Samza PortableValidatesRunner,beam_PostCommit_Java_PVR_Samza_PR
+Run Java Sickbay,beam_PostCommit_Java_Sickbay_PR
+Run Java SingleStoreIO_IT,beam_PostCommit_Java_SingleStoreIO_IT_PR
+Run Java Spark PortableValidatesRunner
Batch,beam_PostCommit_Java_PVR_Spark_Batch_PR
+Run Java Spark v3 PortableValidatesRunner
Streaming,beam_PostCommit_Java_PVR_Spark3_Streaming_PR
+Run Java examples on Dataflow Java
11,beam_PostCommit_Java_Examples_Dataflow_Java11_PR
+Run Java examples on Dataflow Java
17,beam_PostCommit_Java_Examples_Dataflow_Java17_PR
+Run Java examples on Dataflow with Java
11,beam_PostCommit_Java11_Examples_Dataflow_PR
+Run Java_Amazon-Web-Services2_IO_Direct
PreCommit,beam_PreCommit_Java_Amazon-Web-Services2_IO_Direct_Phrase
+Run Java_Amazon-Web-Services_IO_Direct
PreCommit,beam_PreCommit_Java_Amazon-Web-Services_IO_Direct_Phrase
+Run Java_Amqp_IO_Direct PreCommit,beam_PreCommit_Java_Amqp_IO_Direct_Phrase
+Run Java_Azure_IO_Direct PreCommit,beam_PreCommit_Java_Azure_IO_Direct_Phrase
+Run Java_Cassandra_IO_Direct
PreCommit,beam_PreCommit_Java_Cassandra_IO_Direct_Phrase
+Run Java_Cdap_IO_Direct PreCommit,beam_PreCommit_Java_Cdap_IO_Direct_Phrase
+Run Java_Clickhouse_IO_Direct
PreCommit,beam_PreCommit_Java_Clickhouse_IO_Direct_Phrase
+Run Java_Csv_IO_Direct PreCommit,beam_PreCommit_Java_Csv_IO_Direct_Phrase
+Run Java_Debezium_IO_Direct
PreCommit,beam_PreCommit_Java_Debezium_IO_Direct_Phrase
+Run Java_Elasticsearch_IO_Direct
PreCommit,beam_PreCommit_Java_ElasticSearch_IO_Direct_Phrase
+Run Java_Examples_Dataflow
PreCommit,beam_PreCommit_Java_Examples_Dataflow_Phrase
+Run Java_Examples_Dataflow_Java11
PreCommit,beam_PreCommit_Java_Examples_Dataflow_Java11_Phrase
+Run Java_Examples_Dataflow_Java17
PreCommit,beam_PreCommit_Java_Examples_Dataflow_Java17_Phrase
+Run Java_File-schema-transform_IO_Direct
PreCommit,beam_PreCommit_Java_File-schema-transform_IO_Direct_Phrase
+Run Java_GCP_IO_Direct PreCommit,beam_PreCommit_Java_GCP_IO_Direct_Phrase
+Run Java_Hbase_IO_Direct PreCommit,beam_PreCommit_Java_HBase_IO_Direct_Phrase
+Run Java_Hcatalog_IO_Direct
PreCommit,beam_PreCommit_Java_HCatalog_IO_Direct_Phrase
+Run Java_IOs_Direct PreCommit,beam_PreCommit_Java_IOs_Direct_Phrase
+Run Java_Influxdb_IO_Direct
PreCommit,beam_PreCommit_Java_InfluxDb_IO_Direct_Phrase
+Run Java_JDBC_IO_Direct PreCommit,beam_PreCommit_Java_JDBC_IO_Direct_Phrase
+Run Java_Jms_IO_Direct PreCommit,beam_PreCommit_Java_Jms_IO_Direct_Phrase
+Run Java_Kafka_IO_Direct PreCommit,beam_PreCommit_Java_Kafka_IO_Direct_Phrase
+Run Java_Kinesis_IO_Direct
PreCommit,beam_PreCommit_Java_Kinesis_IO_Direct_Phrase
+Run Java_Kudu_IO_Direct PreCommit,beam_PreCommit_Java_Kudu_IO_Direct_Phrase
+Run Java_MongoDb_IO_Direct
PreCommit,beam_PreCommit_Java_MongoDb_IO_Direct_Phrase
+Run Java_Mqtt_IO_Direct PreCommit,beam_PreCommit_Java_Mqtt_IO_Direct_Phrase
+Run Java_Neo4j_IO_Direct PreCommit,beam_PreCommit_Java_Neo4j_IO_Direct_Phrase
+Run Java_PVR_Flink_Batch PreCommit,beam_PreCommit_Java_PVR_Flink_Batch_Phrase
+Run Java_PVR_Flink_Docker PreCommit,beam_PreCommit_Java_PVR_Flink_Docker_Phrase
+Run Java_Parquet_IO_Direct
PreCommit,beam_PreCommit_Java_Parquet_IO_Direct_Phrase
+Run Java_Pulsar_IO_Direct PreCommit,beam_PreCommit_Java_Pulsar_IO_Direct_Phrase
+Run Java_Rabbitmq_IO_Direct
PreCommit,beam_PreCommit_Java_RabbitMq_IO_Direct_Phrase
+Run Java_Redis_IO_Direct PreCommit,beam_PreCommit_Java_Redis_IO_Direct_Phrase
+Run Java_Singlestore_IO_Direct
PreCommit,beam_PreCommit_Java_SingleStore_IO_Direct_Phrase
+Run Java_Snowflake_IO_Direct
PreCommit,beam_PreCommit_Java_Snowflake_IO_Direct_Phrase
+Run Java_Solr_IO_Direct PreCommit,beam_PreCommit_Java_Solr_IO_Direct_Phrase
+Run Java_Spark3_Versions PreCommit,beam_PreCommit_Java_Spark3_Versions_Phrase
+Run Java_Splunk_IO_Direct PreCommit,beam_PreCommit_Java_Splunk_IO_Direct_Phrase
+Run Java_Thrift_IO_Direct PreCommit,beam_PreCommit_Java_Thrift_IO_Direct_Phrase
+Run Java_Tika_IO_Direct PreCommit,beam_PreCommit_Java_Tika_IO_Direct_Phrase
+Run Java_hadoop_IO_Direct PreCommit,beam_PreCommit_Java_Hadoop_IO_Direct_Phrase
+Run Javadoc PostCommit,beam_PostCommit_Javadoc_PR
+Run Jpms Dataflow Java 11
PostCommit,beam_PostCommit_Java_Jpms_Dataflow_Java11_PR
+Run Jpms Dataflow Java 17
PostCommit,beam_PostCommit_Java_Jpms_Dataflow_Java17_PR
+Run Jpms Direct Java 11 PostCommit,beam_PostCommit_Java_Jpms_Direct_Java11_PR
+Run Jpms Direct Java 17 PostCommit,beam_PostCommit_Java_Jpms_Direct_Java17_PR
+Run Jpms Flink Java 11 PostCommit,beam_PostCommit_Java_Jpms_Flink_Java11_PR
+Run Jpms Spark Java 11 PostCommit,beam_PostCommit_Java_Jpms_Spark_Java11_PR
+Run Kotlin_Examples PreCommit,beam_PreCommit_Kotlin_Examples_Phrase
+Run PortableJar_Flink PostCommit,beam_PostCommit_PortableJar_Flink_PR
+Run PortableJar_Spark PostCommit,beam_PostCommit_PortableJar_Spark_PR
+Run Portable_Python PreCommit,beam_PreCommit_Portable_Python_Phrase
+Run PostCommit_Java_Avro_Versions,beam_PostCommit_Java_Avro_Versions_PR
+Run PostCommit_Java_Dataflow,beam_PostCommit_Java_DataflowV1_PR
+Run PostCommit_Java_DataflowV2,beam_PostCommit_Java_DataflowV2_PR
+Run PostCommit_Java_Hadoop_Versions,beam_PostCommit_Java_Hadoop_Versions_PR
+Run Python 3.10 PostCommit Sickbay,beam_PostCommit_Sickbay_Python310_PR
+Run Python 3.10 PostCommit,beam_PostCommit_Python310_PR
+Run Python 3.11 PostCommit Sickbay,beam_PostCommit_Sickbay_Python311_PR
+Run Python 3.11 PostCommit,beam_PostCommit_Python311_PR
+Run Python 3.7 PostCommit Sickbay,beam_PostCommit_Sickbay_Python37_PR
+Run Python 3.7 PostCommit,beam_PostCommit_Python37_PR
+Run Python 3.8 PostCommit Sickbay,beam_PostCommit_Sickbay_Python38_PR
+Run Python 3.8 PostCommit,beam_PostCommit_Python38_PR
+Run Python 3.9 PostCommit Sickbay,beam_PostCommit_Sickbay_Python39_PR
+Run Python 3.9 PostCommit,beam_PostCommit_Python39_PR
+Run Python Dataflow V2 ValidatesRunner,beam_PostCommit_Py_VR_Dataflow_V2_PR
+Run Python Dataflow ValidatesContainer,beam_PostCommit_Py_ValCont_PR
+Run Python Dataflow ValidatesRunner,beam_PostCommit_Py_VR_Dataflow_PR
+Run Python Direct Runner Nexmark Tests,beam_PostCommit_Python_Nexmark_Direct_PR
+Run Python Examples_Dataflow,beam_PostCommit_Python_Examples_Dataflow_PR
+Run Python Examples_Direct,beam_PostCommit_Python_Examples_Direct_PR
+Run Python Examples_Flink,beam_PostCommit_Python_Examples_Flink_PR
+Run Python Examples_Spark,beam_PostCommit_Python_Examples_Spark_PR
+Run Python Flink ValidatesRunner,beam_PostCommit_Python_VR_Flink_PR
+Run Python MongoDBIO_IT,beam_PostCommit_Python_MongoDBIO_IT_PR
+Run Python PreCommit,beam_PreCommit_Python_Phrase
+Run Python RC Dataflow ValidatesContainer,beam_PostCommit_Py_ValCont_with_RC_PR
+Run Python Samza ValidatesRunner,beam_PostCommit_Python_VR_Samza_PR
+Run Python Spark ValidatesRunner,beam_PostCommit_Python_VR_Spark_PR
+Run PythonDocker PreCommit,beam_PreCommit_PythonDocker_Phrase
+Run PythonDocs PreCommit,beam_PreCommit_PythonDocs_Phrase
+Run PythonFormatter PreCommit,beam_PreCommit_PythonFormatter_Phrase
+Run PythonLint PreCommit,beam_PreCommit_PythonLint_Phrase
+Run Python_Coverage PreCommit,beam_PreCommit_Python_Coverage_Phrase
+Run Python_Dataframes PreCommit,beam_PreCommit_Python_Dataframes_Phrase
+Run Python_Examples PreCommit,beam_PreCommit_Python_Examples_Phrase
+Run Python_Integration PreCommit,beam_PreCommit_Python_Integration_Phrase
+Run Python_PVR_Flink PreCommit,beam_PreCommit_Python_PVR_Flink_Phrase
+Run Python_Runners PreCommit,beam_PreCommit_Python_Runners_Phrase
+Run Python_Transforms PreCommit,beam_PreCommit_Python_Transforms_Phrase
+Run Python_Xlang_Gcp_Dataflow
PostCommit,beam_PostCommit_Python_Xlang_Gcp_Dataflow_PR
+Run Python_Xlang_Gcp_Direct
PostCommit,beam_PostCommit_Python_Xlang_Gcp_Direct_PR
+Run RAT PreCommit,beam_PreCommit_RAT_Phrase
+Run Release Gradle Build,beam_Release_Gradle_Build
+Run SQL PostCommit,beam_PostCommit_SQL_PR
+Run SQL PreCommit,beam_PreCommit_SQL_Phrase
+Run SQL_Java11 PreCommit,beam_PreCommit_SQL_Java11_Phrase
+Run SQL_Java17 PreCommit,beam_PreCommit_SQL_Java17_Phrase
+Run Samza ValidatesRunner,beam_PostCommit_Java_ValidatesRunner_Samza_PR
+Run Spark Runner Nexmark Tests,beam_PostCommit_Java_Nexmark_Spark_PR
+Run Spark Runner Tpcds Tests,beam_PostCommit_Java_Tpcds_Spark_PR
+Run Spark StructuredStreaming
ValidatesRunner,beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming_PR
+Run Spark ValidatesRunner Java
11,beam_PostCommit_Java_ValidatesRunner_Spark_Java11_PR
+Run Spark ValidatesRunner,beam_PostCommit_Java_ValidatesRunner_Spark_PR
+Run Spotless PreCommit,beam_PreCommit_Spotless_Phrase
+Run Twister2 ValidatesRunner,beam_PostCommit_Java_ValidatesRunner_Twister2_PR
+Run Typescript PreCommit,beam_PreCommit_Typescript_Phrase
+Run ULR Loopback ValidatesRunner,beam_PostCommit_Java_ValidatesRunner_ULR_PR
+Run Whitespace PreCommit,beam_PreCommit_Whitespace_Phrase
+Run XVR_Direct PostCommit,beam_PostCommit_XVR_Direct_PR
+Run XVR_Flink PostCommit,beam_PostCommit_XVR_Flink_PR
+Run XVR_GoUsingJava_Dataflow
PostCommit,beam_PostCommit_XVR_GoUsingJava_Dataflow_PR
+Run XVR_JavaUsingPython_Dataflow
PostCommit,beam_PostCommit_XVR_JavaUsingPython_Dataflow_PR
+Run XVR_PythonUsingJavaSQL_Dataflow
PostCommit,beam_PostCommit_XVR_PythonUsingJavaSQL_Dataflow_PR
+Run XVR_PythonUsingJava_Dataflow
PostCommit,beam_PostCommit_XVR_PythonUsingJava_Dataflow_PR
+Run XVR_Samza PostCommit,beam_PostCommit_XVR_Samza_PR
+Run XVR_Spark3 PostCommit,beam_PostCommit_XVR_Spark3_PR
\ No newline at end of file
diff --git a/release/src/main/scripts/mass_comment.py
b/release/src/main/scripts/mass_comment.py
index 48582edc5a6..e9a9e47b9e0 100644
--- a/release/src/main/scripts/mass_comment.py
+++ b/release/src/main/scripts/mass_comment.py
@@ -17,160 +17,11 @@
"""Script for mass-commenting Jenkins test triggers on a Beam PR."""
+import os
import requests
import socket
import time
-# This list can be found by querying the Jenkins API, see BEAM-13951
-COMMENTS_TO_ADD = [
- ("Run CommunityMetrics PreCommit", "beam_PreCommit_CommunityMetrics"),
- ("Run Dataflow Runner Nexmark Tests",
"beam_PostCommit_Java_Nexmark_Dataflow"),
- ("Run Dataflow Runner Tpcds Tests", "beam_PostCommit_Java_Tpcds_Dataflow"),
- ("Run Dataflow Runner V2 Java 11 Nexmark Tests",
"beam_PostCommit_Java_Nexmark_DataflowV2_Java11"),
- ("Run Dataflow Runner V2 Java 17 Nexmark Tests",
"beam_PostCommit_Java_Nexmark_DataflowV2_Java17"),
- ("Run Dataflow Runner V2 Nexmark Tests",
"beam_PostCommit_Java_Nexmark_DataflowV2"),
- ("Run Dataflow Streaming ValidatesRunner",
"beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming"),
- ("Run Dataflow ValidatesRunner Java 11",
"beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11"),
- ("Run Dataflow ValidatesRunner Java 17",
"beam_PostCommit_Java_ValidatesRunner_Dataflow_Java17"),
- ("Run Dataflow ValidatesRunner",
"beam_PostCommit_Java_ValidatesRunner_Dataflow"),
- ("Run Direct Runner Nexmark Tests", "beam_PostCommit_Java_Nexmark_Direct"),
- ("Run Direct ValidatesRunner Java 11",
"beam_PostCommit_Java_ValidatesRunner_Direct_Java11"),
- ("Run Direct ValidatesRunner Java 17",
"beam_PostCommit_Java_ValidatesRunner_Direct_Java17"),
- ("Run Direct ValidatesRunner Java 11",
"beam_PostCommit_Java_ValidatesRunner_Direct_Java11"),
- ("Run Direct ValidatesRunner",
"beam_PostCommit_Java_ValidatesRunner_Direct"),
- ("Run Flink Runner Nexmark Tests", "beam_PostCommit_Java_Nexmark_Flink"),
- ("Run Flink ValidatesRunner Java 11",
"beam_PostCommit_Java_ValidatesRunner_Flink_Java11"),
- ("Run Flink Runner Tpcds Tests", "beam_PostCommit_Java_Tpcds_Flink"),
- ("Run Flink ValidatesRunner",
"beam_PostCommit_Java_ValidatesRunner_Flink"),
- ("Run Go Flink ValidatesRunner", "beam_PostCommit_Go_VR_Flink"),
- ("Run Go PostCommit", "beam_PostCommit_Go"),
- ("Run Go PreCommit", "beam_PreCommit_Go"),
- ("Run Go Samza ValidatesRunner", "beam_PostCommit_Go_VR_Samza"),
- ("Run Go Spark ValidatesRunner", "beam_PostCommit_Go_VR_Spark"),
- ("Run GoPortable PreCommit", "beam_PreCommit_GoPortable"),
- ("Run Java 11 Examples on Dataflow Runner V2",
"beam_PostCommit_Java_Examples_Dataflow_V2_java11"),
- ("Run Java 17 Examples on Dataflow Runner V2",
"beam_PostCommit_Java_Examples_Dataflow_V2_java17"),
- ("Run Java Dataflow V2 ValidatesRunner Streaming",
"beam_PostCommit_Java_VR_Dataflow_V2_Streaming"),
- ("Run Java Dataflow V2 ValidatesRunner",
"beam_PostCommit_Java_VR_Dataflow_V2"),
- ("Run Java Examples on Dataflow Runner V2",
"beam_PostCommit_Java_Examples_Dataflow_V2"),
- ("Run Java Examples_Direct", "beam_PostCommit_Java_Examples_Direct"),
- ("Run Java Examples_Flink", "beam_PostCommit_Java_Examples_Flink"),
- ("Run Java Examples_Spark", "beam_PostCommit_Java_Examples_Spark"),
- ("Run Java Flink PortableValidatesRunner Streaming",
"beam_PostCommit_Java_PVR_Flink_Streaming"),
- ("Run Java PostCommit", "beam_PostCommit_Java"),
- ("Run Java PreCommit", "beam_PreCommit_Java"),
- ("Run Java Samza PortableValidatesRunner",
"beam_PostCommit_Java_PVR_Samza"),
- ("Run Java Spark PortableValidatesRunner Batch",
"beam_PostCommit_Java_PVR_Spark_Batch"),
- ("Run Java Spark v3 PortableValidatesRunner Streaming",
"beam_PostCommit_Java_PVR_Spark3_Streaming"),
- ("Run Java examples on Dataflow Java 11",
"beam_PostCommit_Java_Examples_Dataflow_Java11"),
- ("Run Java examples on Dataflow Java 17",
"beam_PostCommit_Java_Examples_Dataflow_Java17"),
- ("Run Java_Amazon-Web-Services2_IO_Direct PreCommit",
"beam_PreCommit_Java_Amazon-Web-Services2_IO_Direct"),
- ("Run Java_Amazon-Web-Services_IO_Direct PreCommit",
"beam_PreCommit_Java_Amazon-Web-Services_IO_Direct"),
- ("Run Java_Amqp_IO_Direct PreCommit",
"beam_PreCommit_Java_Amqp_IO_Direct"),
- ("Run Java_Azure_IO_Direct PreCommit",
"beam_PreCommit_Java_Azure_IO_Direct"),
- ("Run Java_Cassandra_IO_Direct PreCommit",
"beam_PreCommit_Java_Cassandra_IO_Direct"),
- ("Run Java_Cdap_IO_Direct PreCommit",
"beam_PreCommit_Java_Cdap_IO_Direct"),
- ("Run Java_Clickhouse_IO_Direct PreCommit",
"beam_PreCommit_Java_Clickhouse_IO_Direct"),
- ("Run Java_Debezium_IO_Direct PreCommit",
"beam_PreCommit_Java_Debezium_IO_Direct"),
- ("Run Java_ElasticSearch_IO_Direct PreCommit",
"beam_PreCommit_Java_ElasticSearch_IO_Direct"),
- ("Run Java_Examples_Dataflow PreCommit",
"beam_PreCommit_Java_Examples_Dataflow"),
- ("Run Java_Examples_Dataflow_Java11 PreCommit",
"beam_PreCommit_Java_Examples_Dataflow_Java11"),
- ("Run Java_Examples_Dataflow_Java17 PreCommit",
"beam_PreCommit_Java_Examples_Dataflow_Java17"),
- ("Run Java_GCP_IO_Direct PreCommit", "beam_PreCommit_Java_GCP_IO_Direct"),
- ("Run Java_HBase_IO_Direct PreCommit",
"beam_PreCommit_Java_HBase_IO_Direct"),
- ("Run Java_HCatalog_IO_Direct PreCommit",
"beam_PreCommit_Java_HCatalog_IO_Direct"),
- ("Run Java_Hadoop_IO_Direct PreCommit",
"beam_PreCommit_Java_Hadoop_IO_Direct"),
- ("Run Java_InfluxDb_IO_Direct PreCommit",
"beam_PreCommit_Java_InfluxDb_IO_Direct"),
- ("Run Java_JDBC_IO_Direct PreCommit",
"beam_PreCommit_Java_JDBC_IO_Direct"),
- ("Run Java_Jms_IO_Direct PreCommit", "beam_PreCommit_Java_Jms_IO_Direct"),
- ("Run Java_Kafka_IO_Direct PreCommit",
"beam_PreCommit_Java_Kafka_IO_Direct"),
- ("Run Java_Kinesis_IO_Direct PreCommit",
"beam_PreCommit_Java_Kinesis_IO_Direct"),
- ("Run Java_Kudu_IO_Direct PreCommit",
"beam_PreCommit_Java_Kudu_IO_Direct"),
- ("Run Java_MongoDb_IO_Direct PreCommit",
"beam_PreCommit_Java_MongoDb_IO_Direct"),
- ("Run Java_Mqtt_IO_Direct PreCommit",
"beam_PreCommit_Java_Mqtt_IO_Direct"),
- ("Run Java_Neo4j_IO_Direct PreCommit",
"beam_PreCommit_Java_Neo4j_IO_Direct"),
- ("Run Java_PVR_Flink_Batch PreCommit",
"beam_PreCommit_Java_PVR_Flink_Batch"),
- ("Run Java_PVR_Flink_Docker PreCommit",
"beam_PreCommit_Java_PVR_Flink_Docker"),
- ("Run Java_Parquet_IO_Direct PreCommit",
"beam_PreCommit_Java_Parquet_IO_Direct"),
- ("Run Java_Pulsar_IO_Direct PreCommit",
"beam_PreCommit_Java_Pulsar_IO_Direct"),
- ("Run Java_RabbitMq_IO_Direct PreCommit",
"beam_PreCommit_Java_RabbitMq_IO_Direct"),
- ("Run Java_Redis_IO_Direct PreCommit",
"beam_PreCommit_Java_Redis_IO_Direct"),
- ("Run Java_SingleStore_IO_Direct PreCommit",
"beam_PreCommit_Java_SingleStore_IO_Direct"),
- ("Run Java_Snowflake_IO_Direct PreCommit",
"beam_PreCommit_Java_Snowflake_IO_Direct"),
- ("Run Java_Solr_IO_Direct PreCommit",
"beam_PreCommit_Java_Solr_IO_Direct"),
- ("Run Java_Spark3_Versions PreCommit",
"beam_PreCommit_Java_Spark3_Versions"),
- ("Run Java_Splunk_IO_Direct PreCommit",
"beam_PreCommit_Java_Splunk_IO_Direct"),
- ("Run Java_Thrift_IO_Direct PreCommit",
"beam_PreCommit_Java_Thrift_IO_Direct"),
- ("Run Java_Tika_IO_Direct PreCommit",
"beam_PreCommit_Java_Tika_IO_Direct"),
- ("Run Javadoc PostCommit", "beam_PostCommit_Javadoc"),
- ("Run Jpms Dataflow Java 11 PostCommit",
"beam_PostCommit_Java_Jpms_Dataflow_Java11"),
- ("Run Jpms Dataflow Java 17 PostCommit",
"beam_PostCommit_Java_Jpms_Dataflow_Java17"),
- ("Run Jpms Direct Java 11 PostCommit",
"beam_PostCommit_Java_Jpms_Direct_Java11"),
- ("Run Jpms Direct Java 17 PostCommit",
"beam_PostCommit_Java_Jpms_Direct_Java17"),
- ("Run Jpms Flink Java 11 PostCommit",
"beam_PostCommit_Java_Jpms_Flink_Java11"),
- ("Run Jpms Spark Java 11 PostCommit",
"beam_PostCommit_Java_Jpms_Spark_Java11"),
- ("Run Kotlin_Examples PreCommit", "beam_PreCommit_Kotlin_Examples"),
- ("Run PortableJar_Flink PostCommit", "beam_PostCommit_PortableJar_Flink"),
- ("Run PortableJar_Spark PostCommit", "beam_PostCommit_PortableJar_Spark"),
- ("Run Portable_Python PreCommit", "beam_PreCommit_Portable_Python"),
- ("Run PostCommit_Java_Dataflow", "beam_PostCommit_Java_DataflowV1"),
- ("Run PostCommit_Java_DataflowV2", "beam_PostCommit_Java_DataflowV2"),
- ("Run PostCommit_Java_Hadoop_Versions",
"beam_PostCommit_Java_Hadoop_Versions"),
- ("Run Python 3.7 PostCommit", "beam_PostCommit_Python37"),
- ("Run Python 3.8 PostCommit", "beam_PostCommit_Python38"),
- ("Run Python 3.9 PostCommit", "beam_PostCommit_Python39"),
- ("Run Python 3.10 PostCommit", "beam_PostCommit_Python310"),
- ("Run Python 3.11 PostCommit", "beam_PostCommit_Python311"),
- ("Run Python Dataflow V2 ValidatesRunner",
"beam_PostCommit_Py_VR_Dataflow_V2"),
- ("Run Python Dataflow ValidatesContainer", "beam_PostCommit_Py_ValCont"),
- ("Run Python Dataflow ValidatesRunner", "beam_PostCommit_Py_VR_Dataflow"),
- ("Run Python Direct Runner Nexmark Tests",
"beam_PostCommit_Python_Nexmark_Direct"),
- ("Run Python Examples_Dataflow",
"beam_PostCommit_Python_Examples_Dataflow"),
- ("Run Python Examples_Direct", "beam_PostCommit_Python_Examples_Direct"),
- ("Run Python Examples_Flink", "beam_PostCommit_Python_Examples_Flink"),
- ("Run Python Examples_Spark", "beam_PostCommit_Python_Examples_Spark"),
- ("Run Python Flink ValidatesRunner", "beam_PostCommit_Python_VR_Flink"),
- ("Run Python PreCommit", "beam_PreCommit_Python"),
- ("Run Python Samza ValidatesRunner", "beam_PostCommit_Python_VR_Samza"),
- ("Run Python Spark ValidatesRunner", "beam_PostCommit_Python_VR_Spark"),
- ("Run PythonDocker PreCommit", "beam_PreCommit_PythonDocker"),
- ("Run PythonDocs PreCommit", "beam_PreCommit_PythonDocs"),
- ("Run PythonFormatter PreCommit", "beam_PreCommit_PythonFormatter"),
- ("Run PythonLint PreCommit", "beam_PreCommit_PythonLint"),
- ("Run Python_Coverage PreCommit", "beam_PreCommit_Python_Coverage"),
- ("Run Python_Dataframes PreCommit", "beam_PreCommit_Python_Dataframes"),
- ("Run Python_Examples PreCommit", "beam_PreCommit_Python_Examples"),
- ("Run Python_Integration PreCommit", "beam_PreCommit_Python_Integration"),
- ("Run Python_PVR_Flink PreCommit", "beam_PreCommit_Python_PVR_Flink"),
- ("Run Python_Runners PreCommit", "beam_PreCommit_Python_Runners"),
- ("Run Python_Transforms PreCommit", "beam_PreCommit_Python_Transforms"),
- ("Run RAT PreCommit", "beam_PreCommit_RAT"),
- ("Run Release Gradle Build", "beam_Release_Gradle_Build"),
- ("Run SQL PostCommit", "beam_PostCommit_SQL"),
- ("Run SQL PreCommit", "beam_PreCommit_SQL"),
- ("Run SQL_Java11 PreCommit", "beam_PreCommit_SQL_Java11"),
- ("Run SQL_Java17 PreCommit", "beam_PreCommit_SQL_Java17"),
- ("Run Samza ValidatesRunner",
"beam_PostCommit_Java_ValidatesRunner_Samza"),
- ("Run Spark Runner Nexmark Tests", "beam_PostCommit_Java_Nexmark_Spark"),
- ("Run Spark Runner Tpcds Tests", "beam_PostCommit_Java_Tpcds_Spark"),
- ("Run Spark StructuredStreaming ValidatesRunner",
"beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming"),
- ("Run Spark ValidatesRunner Java 11",
"beam_PostCommit_Java_ValidatesRunner_Spark_Java11"),
- ("Run Spark ValidatesRunner",
"beam_PostCommit_Java_ValidatesRunner_Spark"),
- ("Run Spotless PreCommit", "beam_PreCommit_Spotless"),
- ("Run Twister2 ValidatesRunner",
"beam_PostCommit_Java_ValidatesRunner_Twister2"),
- ("Run Typescript PreCommit", "beam_PreCommit_Typescript"),
- ("Run ULR Loopback ValidatesRunner",
"beam_PostCommit_Java_ValidatesRunner_ULR"),
- ("Run Whitespace PreCommit", "beam_PreCommit_Whitespace"),
- ("Run XVR_Direct PostCommit", "beam_PostCommit_XVR_Direct"),
- ("Run XVR_Flink PostCommit", "beam_PostCommit_XVR_Flink"),
- ("Run XVR_GoUsingJava_Dataflow PostCommit",
"beam_PostCommit_XVR_GoUsingJava_Dataflow"),
- ("Run XVR_JavaUsingPython_Dataflow PostCommit",
"beam_PostCommit_XVR_JavaUsingPython_Dataflow"),
- ("Run XVR_PythonUsingJavaSQL_Dataflow PostCommit",
"beam_PostCommit_XVR_PythonUsingJavaSQL_Dataflow"),
- ("Run XVR_PythonUsingJava_Dataflow PostCommit",
"beam_PostCommit_XVR_PythonUsingJava_Dataflow"),
- ("Run XVR_Samza PostCommit", "beam_PostCommit_XVR_Samza"),
- ("Run XVR_Spark3 PostCommit", "beam_PostCommit_XVR_Spark3"),
-]
-
def executeGHGraphqlQuery(accessToken, query):
'''Runs graphql query on GitHub.'''
@@ -281,7 +132,15 @@ if __name__ == '__main__':
wrap work code in module check.
'''
print("Started.")
-
+ comments = []
+ dirname = os.path.dirname(__file__)
+ with open(os.path.join(dirname, 'jenkins_jobs.txt')) as file:
+ comments = [line.strip() for line in file if len(line.strip()) > 0]
+
+ for i in range(len(comments)):
+ parts = comments[i].split(',')
+ comments[i] = (parts[0], parts[1])
+
if not probeGitHubIsUp():
print("GitHub is unavailable, skipping fetching data.")
exit()
@@ -293,7 +152,7 @@ if __name__ == '__main__':
pr = input("Enter the Beam PR number to test (e.g. 11403): ")
subjectId = getSubjectId(accessToken, pr)
- remainingComments = getRemainingComments(accessToken, pr, COMMENTS_TO_ADD)
+ remainingComments = getRemainingComments(accessToken, pr, comments)
if len(remainingComments) == 0:
print('Jobs have been started for all comments. If you would like to retry
all jobs, create a new commit before running this script.')
while len(remainingComments) > 0:
@@ -312,4 +171,4 @@ if __name__ == '__main__':
print(f'{len(remainingComments)} comments still must be reposted:
{str(remainingComments)}')
print('Trying to repost comments.')
- print('Done.')
+ print('Done.')
\ No newline at end of file
diff --git a/website/www/site/content/en/contribute/release-guide.md
b/website/www/site/content/en/contribute/release-guide.md
index 1c841b4f6a2..9be2798f5e7 100644
--- a/website/www/site/content/en/contribute/release-guide.md
+++ b/website/www/site/content/en/contribute/release-guide.md
@@ -275,7 +275,8 @@ The key points to know:
- The Dataflow container image should be modified to the version to be
released.
This will all be accomplished by the
[cut_release_branch](https://github.com/apache/beam/actions/workflows/cut_release_branch.yml)
-workflow.
+workflow. This workflow will also update
[mass_comment.py](https://github.com/apache/beam/blob/master/release/src/main/scripts/mass_comment.py)
+to contain all of the active Jenkins jobs.
After updating the master branch, the workflow will also start a build of
[the nightly
snapshot](https://ci-beam.apache.org/job/beam_Release_NightlySnapshot/) against
master branch.
@@ -286,6 +287,9 @@ The workflow starts the nightly snapshot by creating an
empty PR against apache:
#### Use cut_release_branch.sh to cut a release branch
* **Action:**
[cut_release_branch](https://github.com/apache/beam/actions/workflows/cut_release_branch.yml)
(click `run workflow`)
+In order to run this workflow, you will need to provide a Jenkins username and
API token. Your Jenkins username should be your Apache ID.
+Your Jenkins API token can be generated by visiting
https://ci-beam.apache.org/user/<your Jenkins username>/configure and clicking
+`Add new token` in the API token section.
* Tasks you need to do manually to __verify the SNAPSHOT build__
1. Check whether the Jenkins job gets triggered. If not, please comment
```Run Gradle Publish``` into the generated PR.
@@ -319,12 +323,8 @@ There are 2 ways to perform this verification, either
running automation script(
1. Trigger `beam_Release_Gradle_Build` and all Jenkins PostCommit jobs from
the PR created by the previous step.
You can run
[mass_comment.py](https://github.com/apache/beam/blob/master/release/src/main/scripts/mass_comment.py)
to do that.
Or manually add one trigger phrase per PR comment.
- See `COMMENTS_TO_ADD` in
[mass_comment.py](https://github.com/apache/beam/blob/master/release/src/main/scripts/mass_comment.py)
- for full list of phrases. Please note that this list of phrases can get
- out of date, it's your responsibility to run _all_ PostCommits, not just
- the ones listed there.
- [BEAM-13951](https://issues.apache.org/jira/browse/BEAM-13951) has
- directions for updating this list using the Jenkins API.
+ See
[jenkins_jobs.txt](https://github.com/apache/beam/blob/master/release/src/main/scripts/jenkins_jobs.txt)
+ for a full list of phrases.
* **Tasks included in the script**
1. Installs ```hub``` with your agreement and setup local git repo;
