Should we consider drafting some bare metal install instructions for 0.4.2 that can be a part of the release? Similar to https://github.com/apache/metron/tree/master/metron-deployment/other-examples/manual-install
Or do we want to continue to have those instructions outside of the releases themselves (wiki, etc.)? Just a thought, maybe something to do next release and not this time. I have a documentation PR that fixes some stellar documentation issues that I would like to get in (broken links, nonexistent functions, reordering, etc.), and I am about done with the bro 2.5.2 upgrade in full dev (PR soon, probably Monday). If we can get that in we will have some cleaner documentation and a nice clean bro plugin scenario without needing to do a one off. Jon On Fri, Nov 17, 2017, 21:28 Ryan Merriman <[email protected]> wrote: > Makes sense now. Thanks Matt. > > > On Nov 17, 2017, at 4:25 PM, Matt Foley <[email protected]> wrote: > > > > Hi Ryan, > > Yes and no. The last release (see > https://dist.apache.org/repos/dist/release/metron/ ) was 0.4.1, announced > on 9/19. > > Immediately after that we bumped the <version> of builds from master > branch, per https://issues.apache.org/jira/browse/METRON-1196 . This is > consistent with the Release Process “Clean up” phase: “It is good practice > to increment the build version in master immediately after a Feature > Release, so that dev builds with new stuff from master cannot be mistaken > for builds of the release version. So, immediately after a release, > increment the MINOR version number (eg, with the 0.4.0 just released, set > the new version number to 0.4.1)” ( > https://cwiki.apache.org/confluence/display/METRON/Release+Process#ReleaseProcess-Step15-Cleanup > ) > > > > So you’re correct that the version of development builds is currently > set to 0.4.2. After we actually make a release of 0.4.2, we’ll change dev > build <version> to 0.4.3 (regardless of whether we expect the following > release to be 0.4.3 or 0.5.0). > > > > Hope this clarifies, > > --Matt > > > > On 11/17/17, 1:59 PM, "Ryan Merriman" <[email protected]> wrote: > > > > Matt, > > > > I think we are currently on version 0.4.2. If that is the case would > the > > next version be 0.4.3? > > > > Ryan > > > >> On Fri, Nov 17, 2017 at 3:31 PM, Matt Foley <[email protected]> > wrote: > >> > >> (With release manager hat on) > >> > >> The community has proposed a release of Metron in the near future, > >> focusing on Meta-alerts running in Elasticsearch. > >> Congrats on getting so many of the below already done. At this point, > >> only METRON-1252, and the discussion of how to handle joint release of > the > >> Metron bro plugin, remain as gating items for the release. I project > these > >> will be resolved next week, so let’s propose the following: > >> > >> Sometime next week, after the last bits are done, I’ll start the release > >> process and create the release branch. > >> > >> The proposed new version will be 0.4.2, unless there are backward > >> incompatible changes that support making it 0.5.0. > >> Currently there are NO included Jiras labeled ‘backward-incompatible’, > nor > >> having Docs Text indicating so. > >> ***If anyone knows that some of the commits included since 0.4.1 > introduce > >> backward incompatibility, please say so now on this thread, and mark the > >> Jira as such.*** > >> > >> The 90 or so jiras/commits already in master branch since 0.4.1 are > listed > >> below. > >> Thanks, > >> --Matt > >> > >> METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly Filters > >> Some Records (nickwallen) closes apache/metron#832 > >> METRON-1294 IP addresses are not formatted correctly in facet and > >> group results (merrimanr) closes apache/metron#827 > >> METRON-1291 Kafka produce REST endpoint does not work in a Kerberized > >> cluster (merrimanr) closes apache/metron#826 > >> METRON-1290 Only first 10 alerts are update when a MetaAlert status > is > >> changed to inactive (justinleet) closes apache/metron#842 > >> METRON-1311 Service Check Should Check Elasticsearch Index Templates > >> (nickwallen) closes apache/metron#839 > >> METRON-1289 Alert fields are lost when a MetaAlert is created > >> (merrimanr) closes apache/metron#824 > >> METRON-1309 Change metron-deployment to pull the plugin from > >> apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837 > >> METRON-1310 Template Delete Action Deletes Search Indices > (nickwallen) > >> closes apache/metron#838 > >> METRON-1275: Fix Metron Documentation closes > >> apache/incubator-metron#833 > >> METRON-1295 Unable to Configure Logging for REST API (nickwallen) > >> closes apache/metron#828 > >> METRON-1307 Force install of java8 since java9 does not appear to > work > >> with the scripts (brianhurley via ottobackwards) closes > apache/metron#835 > >> METRON-1296 Full Dev Fails to Deploy Index Templates (nickwallen via > >> cestella) closes apache/incubator-metron#829 > >> METRON-1281 Remove hard-coded indices from the Alerts UI (merrimanr) > >> closes apache/metron#821 > >> METRON-1287 Full Dev Fails When Installing EPEL Repository > >> (nickwallen) closes apache/metron#820 > >> METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list > >> page (iraghumitra via merrimanr) closes apache/metron#819 > >> METRON-1283 Install Elasticsearch template as a part of the mpack > >> startup scripts (anandsubbu via nickwallen) closes apache/metron#817 > >> METRON-1254: Conditionals as map keys do not function in Stellar > >> closes apache/incubator-metron#801 > >> METRON-1261 Apply bro security patch (JonZeolla via ottobackwards) > >> closes apache/metron#805 > >> METRON-1284 Remove extraneous dead query in ElasticsearchDao > >> (justinleet) closes apache/metron#818 > >> METRON-1270: fix for warnings missing @return tag argument in > >> metron-analytics/metron-profiler-common and metron-profiler-client > closes > >> apache/incubator-metron#810 > >> METRON-1272 Hide child alerts from searches and grouping if they > >> belong to meta alerts (justinleet) closes apache/metron#811 > >> METRON-1224 Add time range selection to search control (iraghumitra > >> via james-sirota) closes apache/metron#796 > >> METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects (cestella via > >> justinleet) closes apache/metron#816 > >> METRON-1243: Add a REST endpoint which allows us to get a list of all > >> indice closes apache/incubator-metron#797 > >> METRON-1196 Increment master version number to 0.4.2 for on-going > >> development (mattf-horton) closes apache/metron#767 > >> METRON-1278 Strip "Build Status" widget from root README.md > >> in site-book build (mattf-horton) closes apache/metron#815 > >> METRON-1274 Master has failure in StormControllerIntegrationTest > >> (merrimanr) closes apache/metron#813 > >> METRON-1266 Profiler - SASL Authentication Failed (nickwallen) closes > >> apache/metron#809 > >> METRON-1260 Include Alerts UI in Ambari Service Check (nickwallen) > >> closes apache/metron#804 > >> METRON-1251: Typo and formatting fixes for metron-rest README closes > >> apache/incubator-metron#800 > >> METRON-1241: Enable the REST API to use a cache for the zookeeper > >> config similar to the Bolts closes apache/incubator-metron#795 > >> METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list > >> page (merrimanr) closes apache/metron#808 > >> METRON-1262 Unable to add comment for a alert in a meta-alert > >> (merrimanr) closes apache/metron#806 > >> METRON-1263 Start Alerts UI service after Metron REST (anandsubbu via > >> nickwallen) closes apache/metron#807 > >> METRON-1255 MetaAlert search is not filtering on status (merrimanr) > >> closes apache/metron#802 > >> METRON-1249 Improve Metron MPack Service Checks (nickwallen) closes > >> apache/metron#799 > >> METRON-1237 address javadoc warnings in metron-maas-common (dbist via > >> james-sirota) closes apache/metron#792 > >> METRON-1240 address javadoc warnings in metron-platform and > >> metron-analytics (dbist via james-sirota) closes apache/metron#794 > >> METRON-1226 Searching Can Errantly Query the Wrong Indices > >> (nickwallen) closes apache/metron#793 > >> METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota) closes > >> apache/metron#682 > >> METRON-1123 Add group by option using faceted search capabilities of > >> metron-rest-api (iraghumitra via james-sirota) closes apache/metron#768 > >> METRON-1223 Add support to add comments for alerts (iraghumitra via > >> james-sirota) closes apache/metron#788 > >> METRON-1083 Add filters using faceted search capabilities of > >> metron-rest-api (iraghumitra via james-sirota) closes apache/metron#710 > >> METRON-1232 Alert status changes are not reflected in list view > >> (iraghumitra via merrimanr) closes apache/metron#787 > >> METRON-1247 REST search and findOne endpoints return unexpected or > >> incorrect results for guids (justinleet) closes apache/metron#798 > >> METRON-1235: Document the properties pulled from the global > >> configuration closes apache/incubator-metron#791 > >> METRON-1234: fix for WARNING 'dependencies.dependency.( > >> groupId:artifactId:type:classifier)' must be unique: > >> org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes > >> apache/metron#790 > >> METRON-1222: fix warning for The expression ${parent.version} is > >> deprecated. Please use ${project.parent.version} instead. (dbist via > >> mmiklavc) closes apache/metron#782 > >> METRON-1220 Create documentation around alert nested field > >> (justinleet) closes apache/metron#780 > >> METRON-1229 Management UI type is part of the declarations of 2 > >> modules (merrimanr) closes apache/metron#784 > >> METRON-1228: Configuration Management PUSH immediately does DUMP > after > >> (mmiklavc via mmiklavc) closes apache/metron#783 > >> METRON-1218 Metron REST should return better error messages > >> (merrimanr) closes apache/metron#779 > >> METRON-1161 Add ability to edit parser command line options in the > >> management UI (merrimanr) closes apache/metron#737 > >> METRON-1209: Make stellar repl take logging properties, like other > CLI > >> apps in metron closes apache/incubator-metron#772 > >> METRON-1059 address checkstyle warning AvoidStarImport in > >> metron-stellar (dbist via ottobackwards) closes apache/metron#664 > >> METRON-1204 UI does not time out after being idle, but stops > >> functioning (merrimanr) closes apache/metron#771 > >> METRON-1052: Add forensic similarity hash functions to Stellar closes > >> apache/incubator-metron#781 > >> METRON-632: Added validation of "shew.enrichmentType" and > >> "shew.keyColumns" closes apache/incubator-metron#732 > >> METRON-1194 Add Profiler Debug Functions to Profiler README > >> (nickwallen via ottobackwards) closes apache/metron#765 > >> METRON-1055 Metron 0.4.0 manual installation guide for CentOS 6 > >> updates (lvets via ottobackwards) closes apache/metron#661 > >> METRON-1079 STELLAR NaN should be a keyword (ottobackwards) closes > >> apache/metron#681 > >> METRON-1085 Add REST endpoint to save a user profile for the Alerts > UI > >> (merrimanr) closes apache/metron#694 > >> METRON-1208 MPack for Alerts UI (merrimanr) closes apache/metron#778 > >> METRON-1207 Make RPMs for Alerts UI (merrimanr) closes > >> apache/metron#777 > >> METRON-1215 Fix link to RPMs chapter (DimDroll via justinleet) closes > >> apache/metron#776 > >> METRON-1206 Make alerts UI conform to ops UI for install (merrimanr) > >> closes apache/metron#773 > >> METRON-1195 Meta alerts improperly handle updates to non-alert fields > >> (justinleet) closes apache/metron#766 > >> METRON-1189 Add alert escalation to the Alerts UI (merrimanr) closes > >> apache/metron#762 > >> METRON-1156 Simulate Triage Rules in the Stellar REPL (nickwallen) > >> closes apache/metron#733 > >> METRON-1198 Pycapa - No such configuration property > >> 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769 > >> METRON-1202 ElasticsearchDao Has extraneous sleep call (justinleet) > >> closes apache/metron#770 > >> METRON-938 "service metron-rest start <password>" does not work on > >> CentOS 7. (justinleet) closes apache/metron#757 > >> METRON-1182 Refactor Code in alert list to accommodate new view types > >> (iraghumitra via merrimanr) closes apache/metron#756 > >> METRON-1188: Ambari global configuration management (mmiklavc) closes > >> apache/metron#760 > >> METRON-1191 update public web site to point at 0.4.1 new release > >> (mattf-horton) closes apache/metron#764 > >> METRON-1063 address javadoc warnings in metron-stellar (dbist via > >> ottobackwards) closes apache/metron#668 > >> METRON-1190 Fix Meta Alert Type handling in calculation of scores > >> (justinleet) closes apache/metron#763 > >> METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup Correctly > >> (nickwallen) closes apache/metron#759 > >> METRON-1185: Stellar REPL does not work on a kerberized cluster when > >> calling functions interacting with HBase closes > apache/incubator-metron#755 > >> METRON-1186: Profiler Functions use classutils from shaded storm > >> closes apache/incubator-metron#758 > >> METRON-1173: Fix pointers to old stellar docs closes > >> apache/incubator-metron#746 > >> METRON-1179: Make STATS_ADD to take a list closes > >> apache/incubator-metron#750 > >> METRON-1180: Make Stellar Shell accept zookeeper quorum as a CSV list > >> and not require a port closes apache/incubator-metron#751 > >> METRON-1183 Improve KDC Setup Instructions (nickwallen) closes > >> apache/metron#753 > >> METRON-1177 Stale running topologies seen post-kerberization and > cause > >> exceptions (nickwallen) closes apache/metron#748 > >> METRON-1158 Build backend for grouping alerts into meta alerts > >> (justinleet) closes apache/metron#734 > >> METRON-1146: Add ability to parse JSON string into JSONObject for > >> stellar closes apache/incubator-metron#727 > >> METRON-1176 REST: HDFS Service should support setting permissions on > >> files when writing (ottobackwards) closes apache/metron#749 > >> METRON-1114 Add group by capabilities to search REST endpoint > >> (merrimanr) closes apache/metron#702 > >> METRON-1167 Define Session Specific Global Configuration Values in > the > >> REPL (nickwallen) closes apache/metron#740 > >> METRON-1171: Better validation for the SUBSTRING stellar function > >> closes apache/incubator-metron#745 > >> > >> > >> > >> On 11/17/17, 11:59 AM, "Nick Allen" <[email protected]> wrote: > >> > >> I just wanted to send an update on where we are at. We've gotten a > lot > >> done here recently as you can see below. > >> > >> ✓ DONE (1) First, METRON-1289 needs to go in. This one was a > fairly > >> big > >> effort and I am hearing that we are pretty close. > >> > >> ✓ DONE (2) METRON-1294 fixes an issue in how field types are > >> looked-up. > >> > >> ✓ DONE (3) METRON-1290 is next. While this may have been fixed in > >> M-1289, there may be some test cases we want from this PR. > >> > >> ✓ DONE (4) METRON-1301 addresses a problem with the sorting logic. > >> > >> ✓ DONE (5) METRON-1291 fixes an issue with escalation of > metaalerts. > >> > >> (6) That leads us to Raghu's UI work in METRON-1252. This > >> introduces the > >> UI bits that depend on all the previous backend work. > >> > >> (7) At this point, we should have our best effort at running > >> Metaalerts > >> on Elasticsearch 2.x. I propose that we cut a release here. > >> > >> (8) After we cut the release, we can introduce the work for ES 5.x > in > >> METRON-939. I know we will need lots of help testing and reviewing > >> this > >> one. > >> > >> > >> > >> We also have an outstanding question that needs resolved BEFORE we > >> release. We need to come to a consensus on how to release having > >> moved our > >> Bro Plugin to a separate repo. I don't think we've heard from > >> everyone on > >> this. I'd urge everyone to chime in so we can choose a path forward. > >> > >> If anyone is totally confused in regards to that discussion, I can > try > >> and > >> send an options summary again as a separate discuss thread. The > >> original > >> chain was somewhere around here [1]. > >> > >> [1] > >> https://lists.apache.org/thread.html/54a4474881b97e559df24728b3a0e9 > >> 23a58345a282451085eef832ef@%3Cdev.metron.apache.org%3E > >> > >> > >> > >> On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <[email protected]> > >> wrote: > >> > >>> Hi Guys - > >>> > >>> I want to follow-up on this discussion. It sounds like most people > >> are in > >>> agreement with the general approach. > >>> > >>> A lot of people have been working hard on Metaalerts and > >> Elasticsearch. I > >>> have checked-in with those doing the heavy lifting and have compiled > >> a more > >>> detailed plan based on where we are at now. To the best of my > >> knowledge > >>> here is the plan of attack for finishing out this effort. > >>> > >>> (1) First, METRON-1289 needs to go in. This one was a fairly big > >> effort > >>> and I am hearing that we are pretty close. > >>> > >>> (2) METRON-1294 fixes an issue in how field types are looked-up. > >>> > >>> (3) METRON-1290 is next. While this may have been fixed in M-1289, > >>> there may be some test cases we want from this PR. > >>> > >>> (4) METRON-1301 addresses a problem with the sorting logic. > >>> > >>> (5) METRON-1291 fixes an issue with escalation of metaalerts. > >>> > >>> (6) That leads us to Raghu's UI work in METRON-1252. This > >> introduces > >>> the UI bits that depend on all the previous backend work. > >>> > >>> (7) At this point, we should have our best effort at running > >> Metaalerts > >>> on Elasticsearch 2.x. I propose that we cut a release here. > >>> > >>> (8) After we cut the release, we can introduce the work for ES 5.x > >> in > >>> METRON-939. I know we will need lots of help testing and reviewing > >> this > >>> one. > >>> > >>> Please correct me if I am wrong. I will try and send out updates as > >> we > >>> make progress. > >>> > >>> > >>> > >>> > >>> > >>> On Mon, Nov 6, 2017 at 1:03 PM, [email protected] <[email protected]> > >> wrote: > >>> > >>>> I agree, I think it's very reasonable to move in line with Nick's > >>>> proposal. I would also suggest that we outline what the target > >> versions > >>>> would be to add in the METRON-777 components, since it has been > >> functional > >>>> for a very long time but not reviewed and has some really rockstar > >>>> improvements. > >>>> > >>>> Jon > >>>> > >>>> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler < > >> [email protected]> > >>>> wrote: > >>>> > >>>>> I think the ES cutover should be the start of the 0.5.x series, > >> and we > >>>>> continue on with 0.4.x for the > >>>>> metadata improvements etc. We could chose to focus 0.5.x’s first > >>>> releases > >>>>> on not only ES but > >>>>> getting a handle on kibana and the mpack situation as well. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> On November 6, 2017 at 12:48:45, Michael Miklavcic ( > >>>>> [email protected]) wrote: > >>>>> > >>>>> I agree with your proposal, Nick. I think having a stabilizing > >> release > >>>>> prior to upgrading ES/Kibana makes sense. > >>>>> > >>>>> On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <[email protected]> > >> wrote: > >>>>> > >>>>>> I would like to start a discussion around upcoming releases. We > >> have a > >>>>>> couple separate significant tracks of work that we need to > >> reconcile > >>>> in > >>>>> our > >>>>>> release schedule. > >>>>>> > >>>>>> (1) We have had (and have in review) a good number of bug fixes > >>>> required > >>>>> to > >>>>>> support Metaalerts on the existing Elasticsearch 2.x > >> infrastructure. > >>>>>> > >>>>>> > >>>>>> (2) We also have ongoing work to upgrade our infrastructure to > >>>>>> Elasticsearch 5.x, which will not be backwards compatible. > >>>>>> > >>>>>> > >>>>>> I would like to see a release that has our best work on ES 2.x > >> before > >>>> we > >>>>>> migrate to 5.x. I would propose the following. > >>>>>> > >>>>>> Release N+1: Introduce Metaalerts running on ES 2.x > >>>>>> > >>>>>> Release N+2: Cut-over to ES 5.x > >>>>>> > >>>>>> > >>>>>> (Q) Is it worth cutting a separate release for ES 2.x? Is there > >> a > >>>> better > >>>>>> way to handle the cut-over to 5.x? > >>>>>> > >>>>> > >>>> -- > >>>> > >>>> Jon > >>>> > >>> > >>> > >> > >> > >> > >> > > > > > -- Jon
