If you are the person responsible for doing these steps, who double checks your work? Or reviews? Is there such a person? Should that be called out?
On January 16, 2017 at 15:04:36, James Sirota (jsir...@apache.org) wrote: If no one has additional comments on this document i'll go ahead and put it up for a vote... https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66854770 10.01.2017, 12:50, "James Sirota" <jsir...@apache.org>: > Hi Larry, > > Thanks for the comments. I beefed up the technical section. How does this look? > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66854770 > > 0.[FR++].0Metron Release Types > There are two types of Metron releases: > Feature Release (FR) - this is a release that has a significant step forward in feature capability and is denoted by an upgrade of the second digit > Maintenance Release (MR) - this is a set of patches and fixes that are issued following the FR and is denoted by an upgrade of the third digit > Release Naming Convention > Metron build naming convention is as follows: 0.[FR].[MR]. We keep the 0. notation to signify that the project is still under active development and we will hold a community vote to go to 1.x at a future time > Initiating a New Metron Release > Immediately upon the release of the previous Metron release create two branches: FR ++ and MR. Create the FR++ branch by incrementing the second digit like so 0.[FR++].0. Create the MR branch for the previous Metron release by incrementing the second digit of the previous release like so 0.[FR].[MR]. All patches to the previous Metron release will be checked in under the MR branch and where it makes sense also under the FR branch. All new features will be checked in under the FR branch. > Creating a Feature Release > Step 1 - Initiate a discuss thread > Prior to the release The Release manager should do the following (preferably a month before the release): > Make sure that the list of JIRAs slated for the release accurately reflects to reflects the pull requests that are currently in master > Construct an email to the Metron dev board ( dev@metron.incubator.apache.org) which discusses with the community the desire to do a release. This email should contain the following: > The list of JIRAs slated for the release with descriptions (use the output of git log and remove all the JIRAs from the last release’s changelog) > A solicitation of JIRAs that should be included with the next release. Users should rate them as must/need/good to have as well as volunteering. > A release email template is provided here. > Step 2 - Monitor and Verify JIRAs > Once the community votes for additional JIRAs they want included in the release verify that the pull requests are in before the release, close these JIRAs and tag them with the release name. All pull requests and JIRAs that were not slated for this release will go into the next releases. The release manager should continue to monitor the JIRA to ensure that the timetable is on track until the release date. On the release date the release manager should message the Metron dev board ( dev@metron.incubator.apache.org) announcing the code freeze for the release. > Step 3 - Create the Release Branch and Increment Metron version > Create an branch for the release (from a repo cloned from https://git-wip-us.apache.org/repos/asf/incubator-metron.git). (assuming the release is 0.[FR++].0 and working from master): > git checkout -b Metron_0.[FR++].0 > git push --set-upstream origin Metron_0.[FR++].0 > File a JIRA to increment the Metron version to 0.[FR++].0. Either do it yourself or have a community member increment the build version for you. You can look at a pull request for a previous build to see how this is done. METRON-533 - Up the version for release DONE > Also, the release manager should have a couple of things set up: > A SVN clone of the repo at https://dist.apache.org/repos/dist/dev/incubator/metron, We will refer to this as the dev repo. It will hold the release candidate artifacts > A SVN clone of the repo at https://dist.apache.org/repos/dist/release/incubator/metron, We will refer to this as the release repo. It will hold the release artifacts. > Step 4 - Create the Release Candidate > > Now, for each release candidate, we will tag from that branch. Assuming that this is RC1: > git checkout Metron_0.[FR++].0 && git pull > git tag apache-metron-0.[FR++].0-rc1-incubating > git push origin —tags > > Now, we must grab the release candidate binary from the github releases page (https://github.com/apache/incubator-metron/releases). In our case, for RC1, that would be https://github.com/apache/incubator-metron/archive/apache-metron-0.[FR++].0-rc1-incubating.tar.gz We will refer to this as the release candidate tarball. > The artifacts for a release (or a release candidate, for that matter) are as follows: > Release (candidate) Tarball > MD5 hash of the release tarball (md5 apache-metron-Now, we must grab the release candidate binary from the github releases page ( https://github.com/apache/incubator-metron/releases). In our case, for RC1, that would be https://github.com/apache/incubator-metron/archive/apache-metron-0.[FR++].0-rc1-incubating.tar.gz We will refer to this as the release candidate tarball.-rc1-incubating.tar.gz > apache-metron-0.[FR++].0-rc1-incubating.tar.gz.md5) > SHA1 hash of the release tarball (gpg --print-md SHA1 apache-metron-0.[FR++].0-rc1-incubating.tar.gz > apache-metron-0.[FR++].0-rc1-incubating.tar.gz.sha) > GPG signature of release tarball by the release manager > Assuming your public code signing key is 0xDEADBEEF, so signing for me would be: gpg -u 0xDEADBEEF --armor --output apache-metron-0.[FR++].0-rc1-incubating.tar.gz.asc --detach-sig apache-metron-0.[FR++].0-rc1-incubating.tar.gz > If you do not know your code signing key as release manager, you must follow the instructions at https://www.apache.org/dev/release-signing.html#generate > Note: You only need the -u arg if you have more than one public/private key pair generated. If you have forgotten it, you can find it from the output of gpg —fingerprint. It’s the last 4 bytes from the key fingerprint. > The LICENSE file from the release tarball > The KEYS file from the release tarball > The DISCLAIMER file from the release tarball > A CHANGES file denoting the changes > We usually construct this by taking the output of git log | grep METRON | sed 's/\[//g' | sed 's/\]//g' | grep -v “http” and removing the JIRAs from the previous releases (it’s in time sorted order so this is easy). > > Create a directory named ${VERSION}-RC${RC_NUM}-incubating (in our case, it’s 0.[FR++].0-RC1-incubating) in the dev repo. Place the artifacts from above into this directory, add the directory and commit via the subversion client: > svn add 0.[FR++].0-RC1-incubating > svn commit -m "Adding artifacts for Metron 0.[FR++].0-RC1 (incubating)” > Step 5 - Verify the build > Go through the build verification checklist to verify that everything works. These instructions can be found here: Verifying Builds > Step 6 - Verify licensing > Make sure the release compiles with the following Apache licensing guidelines: http://www.apache.org/foundation/license-faq.html > Step 7 - Call for a community release vote > Next initiate a [VOTE] threat on the dev list to announce the build vote. The vote email template can be found here: Build Vote Template. Allow at least 72 hours for the community to vote on the release. When you get enough votes close the vote by replying [RESULT][VOTE] to the email thread with the tally of all the votes > Step 8 - Call for a incubator release vote > Once the community has successfully voted on a release, we must escalate the vote to the incubator general. The same VOTE thread original email is sent to gene...@incubator.apache.org > > If issues are found with the release and the vote fails, then the vote thread is closed with a synopsis of the voting results and a new RC is worked on in the community > If issues are found with the release and the vote succeeds, then we proceed to cut the release, but should notify the community of the issues via an email on the dev list with the accompanying JIRA(s) required to correct the issue(s). > > If no issues are found, then we can cut a release > Again, wait for at least 72 hours and then close the vote. > Step 9 - Stage the finished release > A directory with the name of the version (i.e. 0.3.0) should be made in the release svn repository > > Collateral from the release candidate in the dev repo should be moved to the above directory and renamed to remove the rc (e.g. mv apache-metron-0.3.0-rc1-incubating.tar.gz.sha apache-metron-0.3.0-incubating.tar.gz.sha) > > Add the directory and commit via the subversion client: > > svn add 0.3.0-RC1-incubating > svn commit -m "Adding artifacts for Metron 0.3.0 (incubating)” > > Remove the old releases from the release repo (only the current version and the KEYS file should exist there). > Step 14 - Announce build > Send an email out to user@ and dev@ to announce the release along with the changelog and a word of thanks/praise. > Creating a Maintenance Release > Creation of the Maintenance Release should follow exactly the same set of steps as creating the Feature Release as outlined above, but with two exception. First, the version incremented on the maintenance release should be the MR++ so that the release is named 0.[FR].[MR++]. Second, if a critical JIRA comes in that requires an immediate patch we may forego steps 2-5 and immediately cut the MR release. A critical JIRA is something that is either a security vulnerability or a functional show stopper . > Ensuring Consistency between Feature and Maintenance releases > Being able to maintain the previous release train, with only critical or important bug fixes and security fixes (generally not new features) for users who are averse to frequent large changes is very important for production use. They get stability, while the feature code proceeds as fast as the community wishes. It is important to assure that all commits to the maintenance release also get made in the feature branch (if relevant), to avoid the appearance of regressions in the maintenance branch. The formal process for assuring this is as follows: > Every maintenance release JIRA should have a corresponding feature JIRA to make sure that the patch is applied consistently to both branches. The maintenance JIRA should be cloned and appropriate fix version for the feature release should be applied. If the fix is not relevant to the feature or maintenance branch then the submitter must explicitly state this. In general reviewers should refuse a patch PR unless both feature and maintenance JIRAs have been created. > The release manager has a responsibility to review all commits to the maintenance line since last release, and make sure they were duplicated to the feature branch (unless not relevant, which must also be determined). > > 05.01.2017, 06:32, "larry mccay" <lmc...@apache.org>: >> Hi James - >> >> This looks pretty good! >> >> A couple quick comments: >> >> * for step 10 - the KEYS file appears to be provided for each release as >> part of the release candidate itself. While I do see some projects do this, >> I think it is actually best practice to have a single KEYS file in a well >> known place outside of the rc. This decoupling is supposed to make it more >> difficult for an artifact to be tampered with and another KEYS file >> provided. I think most projects that keep the KEYS separate just put them at >> the top level of the ASF mirror area for the project such as at >> https://dist.apache.org/repos/dist/*release*/incubator/metron/ [1]. >> * Related to the above, it seems that in the KEYS file is duplicated at the >> top level of the ASF mirror area for the project as well as in the release >> directory. The one inside the release directory would probably go away by >> addressing the previous comment but it should be noted that there is a >> chance for those two files to be out of sync otherwise. >> * I notice that the DISCLAIMER, LICENSE and CHANGES files are kept outside >> of the archives along with the KEYS file. As long as they are also inside >> the archive it is probably fine but I don't think there is a need for >> LICENSE and DISCLAIMER to be outside. In Knox we do keep the CHANGES >> outside as well so that it can be easily reviewed to determine interest or >> need for upgrade etc. >> * I do also notice that there is no zip archive - you may want to consider >> adding a zip as well. >> * steps 10 and 13 instruct the release manager to stage the rc and the >> final release but there aren't any instructions as to how to do so. Is that >> documented elsewhere? We have specific ant targets to run for >> stage-candidate and promote-release [2]. >> >> Hope this is helpful. >> >> 1. https://www.apache.org/dev/release-signing.html#keys-policy >> 2. >> https://cwiki.apache.org/confluence/display/KNOX/Release+Process#ReleaseProcess-Stage >> >> thanks, >> >> --larry >> >> On Wed, Jan 4, 2017 at 7:25 PM, Matt Foley <ma...@apache.org> wrote: >> >>> Hi James, is there a formatted version of this somewhere we can look at? >>> Thanks, >>> --Matt >>> >>> On 1/4/17, 1:53 PM, "James Sirota" <jsir...@apache.org> wrote: >>> >>> Revised as per additional comments. Are there more comments? Or can >>> we put this up for a vote? >>> >>> Release Process [DRAFT] >>> Skip to end of metadata >>> Created by James Sirota, last modified just a moment ago Go to start >>> of metadata >>> Metron Release Types >>> There are two types of Metron releases: >>> Feature Release (FR) - this is a release that has a significant step >>> forward in feature capability and is denoted by an upgrade of the second >>> digit >>> Maintenance Release (MR) - this is a set of patches and fixes that are >>> issued following the FR and is denoted by an upgrade of the third digit >>> Release Naming Convention >>> Metron build naming convention is as follows: 0.[FR].[MR]. We keep >>> the 0. notation to signify that the project is still under active >>> development and we will hold a community vote to go to 1.x at a future time >>> Initiating a New Metron Release >>> Immediately upon the release of the previous Metron release create two >>> branches: FR ++ and MR. Create the FR++ branch by incrementing the second >>> digit like so 0.[FR++].0. Create the MR branch for the previous Metron >>> release by incrementing the second digit of the previous release like so >>> 0.[FR].[MR]. All patches to the previous Metron release will be checked in >>> under the MR branch and where it makes sense also under the FR branch. All >>> new features will be checked in under the FR branch. >>> Creating a Feature Release >>> Step 1 - Initiate a discuss thread >>> A week before a new feature release initiate a discuss thread on the >>> Metron dev board announcing the upcoming release and asking the community >>> which still outstanding pull requests people want to include in the next >>> build. >>> Step 2 - Verify JIRA >>> Go through the JIRA and verify that all pull requests that were merged >>> for the upcoming build have JIRAs that are in a closed state and are >>> appropriately labelled with the next build version. >>> Step 3 - Announce a code freeze >>> A day before the release date comment on the discuss thread and let >>> people know that the release is ready. Go through the JIRAs for pull >>> requests that came in during the last week and make sure they are labelled >>> with the next build version. >>> Step 4 - Increment Metron version >>> File a JIRA to increment the Metron version to 0.[FR++].0. Either do >>> it yourself or have a community member increment the build version for >>> you. You can look at a pull request for a previous build to see how this >>> is done >>> Step 5 - Increment build version >>> File a JIRA to increment the Metron version to 0.[FR++].0-RC(n), where >>> RC(n) is the number of the release candidate. Sometimes mistakes occur >>> (builds may get voted down) so it will take multiple RCs to get a build >>> through the vote. The RC(n) will be removed after the successful vote. >>> Step 6 - Verify the build >>> Go through the build verification checklist to verify that everything >>> works. These instructions can be found here: Verifying Builds >>> Step 7 - Verify licensing >>> Make sure the release compiles with the following Apache licensing >>> guidelines: http://www.apache.org/foundation/license-faq.html >>> Step 8 - Generate the changes file >>> Go through the JIRA to generate the changes file, which contains a >>> list of all JIRAs included in the upcoming release. An example of a >>> changes file can be found here: https://dist.apache.org/repos/ >>> dist/dev/incubator/metron/0.3.0-RC1-incubating/CHANGES >>> Step 9 - Tag the RC release >>> Tag the release for the RC in case we need to roll back at some >>> point. An example of a valid tag can be seen here: >>> https://git-wip-us.apache.org/repos/asf?p=incubator-metron. >>> git;a=shortlog;h=refs/tags/apache-metron-0.3.0-rc1-incubating >>> Step 10 - Stage the release >>> The next thing to do is to sign and stage the release including the >>> DISCLAIMER, KEYS, and LICENSE files. A properly signed and staged release >>> can be found here: >>> https://dist.apache.org/repos/dist/dev/incubator/metron/0.3. >>> 0-RC1-incubating/ >>> * Make sure you have your correct profile and keys uploaded to >>> https://id.apache.org/ to properly sign the release and to get access to >>> dist.apache.org >>> Step 11 - Call for a community release vote >>> Next initiate a [VOTE] threat on the dev list to announce the build >>> vote. The vote email template can be found here: Build Vote Template. >>> Allow at least 72 hours for the community to vote on the release. When you >>> get enough votes close the vote by replying [RESULT][VOTE] to the email >>> thread with the tally of all the votes >>> Step 12 - Call for a incubator release vote >>> Upon successful completion of step 11, repeat, but now send the email >>> to the incubator general boards. The email should be identical. Again, >>> wait for at least 72 hours and then close the vote. >>> Step 13 - Stage the finished release >>> If the vote fails at any stage then incorporate feedback, create >>> another RC, and repeat. If both votes pass then stage the resulting >>> artifacts here: https://dist.apache.org/repos/ >>> dist/release/incubator/metron/ >>> Step 14 - Announce build >>> Send a discuss thread to the Metron dev boards announcing the new >>> Metron build >>> Creating a Maintenance Release >>> Creation of the Maintenance Release should follow exactly the same set >>> of steps as creating the Feature Release as outlined above, but with two >>> exception. First, the version incremented on the maintenance release >>> should be the MR++ so that the release is named 0.[FR].[MR++]. Second, if >>> a critical JIRA comes in that requires an immediate patch we may forego >>> steps 2-5 and immediately cut the MR release. A critical JIRA is something >>> that is either a security vulnerability or a functional show stopper . >>> Ensuring Consistency between Feature and Maintenance releases >>> Being able to maintain the previous release train, with only critical >>> or important bug fixes and security fixes (generally not new features) for >>> users who are averse to frequent large changes is very important for >>> production use. They get stability, while the feature code proceeds as >>> fast as the community wishes. It is important to assure that all commits >>> to the maintenance release also get made in the feature branch (if >>> relevant), to avoid the appearance of regressions in the maintenance >>> branch. The formal process for assuring this is as follows: >>> Every maintenance release JIRA should have a corresponding feature >>> JIRA to make sure that the patch is applied consistently to both branches. >>> The maintenance JIRA should be cloned and appropriate fix version for the >>> feature release should be applied. If the fix is not relevant to the >>> feature or maintenance branch then the submitter must explicitly state >>> this. In general reviewers should refuse a patch PR unless both feature >>> and maintenance JIRAs have been created. >>> The release manager has a responsibility to review all commits to the >>> maintenance line since last release, and make sure they were duplicated to >>> the feature branch (unless not relevant, which must also be determined). >>> >>> 20.12.2016, 11:45, "Matt Foley" <ma...@apache.org>: >>> > 1. Agree. Being able to maintain the previous release train, with >>> only critical or important bug fixes and security fixes (generally not new >>> features) for users who are averse to frequent large changes, is very >>> important for production use. They get stability, while the mainline code >>> proceeds as fast as the community wishes. >>> > a. As Kyle points out, it is important to assure that all commits to >>> the maintenance line also get made in the mainline (if relevant), to avoid >>> the appearance of regressions in the mainline. There should be a formal >>> process for assuring this. Possibilities are: >>> > i. The release manager has a responsibility to review all commits to >>> the maint line since last release, and make sure they were duplicated to >>> the mainline (unless not relevant, which must also be determined). >>> > ii. Reviewers refuse to accept PRs for the maint line unless they >>> are twinned with PRs for corresponding changes in the mainline (unless not >>> relevant, which must be stated by the submitter). This should be reflected >>> in Jira practices as well as PR practices. Note Jira is poor at tracking >>> multiple “Fix Version/s” values (due to the ambiguous use of “Fix version” >>> to mean both “target version” and “done version”). Most teams just clone >>> jira tickets for multiple target releases. >>> > 2. Agree. Being a release manager is a significant commitment of >>> both time and care, and should be rotated around; both for the benefit of >>> the individuals involved and so that at least 2 or 3 people are deeply >>> familiar with the process at any given time. >>> > --Matt >>> > >>> > On 12/20/16, 8:15 AM, "James Sirota" <jsir...@apache.org> wrote: >>> > >>> > You are correct. This thread is about the release process: >>> > https://cwiki.apache.org/confluence/pages/viewpage. >>> action?pageId=66854770 >>> > >>> > Does anyone have additional opinions on this? >>> > >>> > 1. Maintenance release would just contain patches to the >>> existing release. Feature release would contain everything, including >>> patches and new features. >>> > 2. The intention is to rotate the build manager. I did it for >>> the first few releases, then Casey did it for the next few releasees, >>> someone else will probably do it for the next few releases, etc... >>> > >>> > Does this seem reasonable to everyone? >>> > >>> > Thanks, >>> > James >>> > >>> > 18.12.2016, 18:15, "Kyle Richardson" <kylerichards...@gmail.com >>> >: >>> > > I think this thread got commingled with the discussion on >>> Coding >>> > > Guidelines. The wiki page on the Release Process is at >>> > > https://cwiki.apache.org/confluence/pages/viewpage. >>> action?pageId=66854770. >>> > > >>> > > Overall, a really informative document. Thanks for pulling >>> this together. >>> > > Two questions: >>> > > >>> > > 1) I'm a little confused about how the feature release and >>> maintenance >>> > > release branches are going to work. Is the idea that all PRs >>> will be merged >>> > > into master and then also be committed to a FR++ or a MR++ >>> branch (or maybe >>> > > even both)? >>> > > >>> > > 2) Are these steps to be taken by a release manager only or is >>> the >>> > > intention that other committers or PMC members rotate through >>> this >>> > > responsibly? Just curious. I actually kind of like the idea of >>> shuffling >>> > > the duty every now and then to avoid burnout by one person. >>> > > >>> > > -Kyle >>> > > >>> > > On Fri, Dec 16, 2016 at 1:31 PM, James Sirota < >>> jsir...@apache.org> wrote: >>> > > >>> > >> fixed the link and made one addition that a qualified >>> reviewer is a >>> > >> committer or PPMC member >>> > >> >>> > >> 16.12.2016, 11:07, "zeo...@gmail.com" <zeo...@gmail.com>: >>> > >> > Right, I agree. That change looks good to me. >>> > >> > >>> > >> > Looks like the Log4j levels links is broken too. >>> > >> > >>> > >> > For a broken travis - how about "If somehow the tests get >>> into a failing >>> > >> > state on master (such as by a backwards incompatible >>> release of a >>> > >> > dependency) only pull requests intended to rectify master >>> may be merged, >>> > >> > and the removal or disabling of any tests must be +1'd by >>> two reviewers." >>> > >> > >>> > >> > Also, reading through this, should there should be a >>> delineation between >>> > >> a >>> > >> > "reviewer" and somebody who has the ability to vote/+1 a >>> PR? Unless I'm >>> > >> > missing something, right now it looks open to anybody. >>> > >> > >>> > >> > Jon >>> > >> > >>> > >> > On Fri, Dec 16, 2016 at 12:48 PM Nick Allen < >>> n...@nickallen.org> wrote: >>> > >> > >>> > >> > Personally, I don't think it matters who merges the pull >>> request. As long >>> > >> > as you meet the requirements for code review, then anyone >>> should be able >>> > >> to >>> > >> > merge it. In fact, I'd rather have the person who knows >>> most about the >>> > >> > change actually merge it into master to ensure that it goes >>> smoothly. >>> > >> > >>> > >> > On Fri, Dec 16, 2016 at 12:15 PM, James Sirota < >>> jsir...@apache.org> >>> > >> wrote: >>> > >> > >>> > >> >> Jon, for #2 I changed it to: A committer may merge their >>> own pull >>> > >> request, >>> > >> >> but only after a second reviewer has given it a +1. >>> > >> >> >>> > >> >> 16.12.2016, 10:07, "zeo...@gmail.com" <zeo...@gmail.com>: >>> > >> >> > I made some minor changes to the doc - check out the >>> history >>> > >> >> > <https://cwiki.apache.org/confluence/pages/ >>> > >> viewpreviousversions.action? >>> > >> >> pageId=61332235> >>> > >> >> > if you have any concerns. >>> > >> >> > >>> > >> >> > Regarding the larger doc - >>> > >> >> > 1. Not everybody can assign JIRAs to themselves. I >>> recall I had to >>> > >> >> request >>> > >> >> > this access, so that should probably be mentioned. >>> > >> >> > 2. "A committer may never merge their own pull request, >>> a second >>> > >> party >>> > >> >> must >>> > >> >> > merge their changes after it has be properly reviewed." >>> > >> >> > - Is this still true/accurate? I heard both ways. >>> > >> >> > 3. "If somehow the tests get into a failing state on >>> master (such as >>> > >> by >>> > >> > >>> > >> > a >>> > >> >> > backwards incompatible release of a dependency) no pull >>> requests may >>> > >> be >>> > >> >> > merged until this is rectified." >>> > >> >> > - Maybe this should get reassessed using the >>> > >> >> > <https://github.com/apache/incubator-metron/pull/383> >>> most >>> > >> >> > <https://github.com/apache/incubator-metron/pull/381> >>> recent >>> > >> >> > <https://issues.apache.org/jira/browse/METRON-601> build >>> > >> >> > <https://issues.apache.org/jira/browse/METRON-597> >>> failures >>> > >> >> > <https://github.com/apache/incubator-metron/pull/380> >>> as a valuable >>> > >> case >>> > >> >> > study. >>> > >> >> > >>> > >> >> > Jon >>> > >> >> > >>> > >> >> > On Fri, Dec 16, 2016 at 11:38 AM James Sirota < >>> jsir...@apache.org> >>> > >> >> wrote: >>> > >> >> > >>> > >> >> >> I threw together a draft document for our release >>> process. Would you >>> > >> >> want >>> > >> >> >> to add/change/delete anything? >>> > >> >> >> >>> > >> >> >> ------------------- >>> > >> >> >> Thank you, >>> > >> >> >> >>> > >> >> >> James Sirota >>> > >> >> >> PPMC- Apache Metron (Incubating) >>> > >> >> >> jsirota AT apache DOT org >>> > >> >> > -- >>> > >> >> > >>> > >> >> > Jon >>> > >> >> > >>> > >> >> > Sent from my mobile device >>> > >> >> >>> > >> >> ------------------- >>> > >> >> Thank you, >>> > >> >> >>> > >> >> James Sirota >>> > >> >> PPMC- Apache Metron (Incubating) >>> > >> >> jsirota AT apache DOT org >>> > >> > >>> > >> > -- >>> > >> > Nick Allen <n...@nickallen.org> >>> > >> > >>> > >> > -- >>> > >> > >>> > >> > Jon >>> > >> > >>> > >> > Sent from my mobile device >>> > >> >>> > >> ------------------- >>> > >> Thank you, >>> > >> >>> > >> James Sirota >>> > >> PPMC- Apache Metron (Incubating) >>> > >> jsirota AT apache DOT org >>> > >>> > ------------------- >>> > Thank you, >>> > >>> > James Sirota >>> > PPMC- Apache Metron (Incubating) >>> > jsirota AT apache DOT org >>> >>> ------------------- >>> Thank you, >>> >>> James Sirota >>> PPMC- Apache Metron (Incubating) >>> jsirota AT apache DOT org > > ------------------- > Thank you, > > James Sirota > PPMC- Apache Metron (Incubating) > jsirota AT apache DOT org ------------------- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
