Hi Jason,
I agree with Gabe, it seems like gem5 is taking a way to get better but it
will very hard for PhD students to suddenly adapt and review thousand lines
of code with months of yet unseen history and iteration and the reason
behind every change.
And in the end it may end up to be used as a course project rather than
thesis project or project for architecture conferences as people will use
the commit or the branch which they are most familiar with.
With so less releases every year, personally it will take a lot of time to
relearn and understand the implementation in detail.
But as this fully controlled by you, we can only suggest you. It should not
be like people are creating another public gem5 as we have now and use that.
This issue was raised by Gabe on Nov 27 email and today also, so I hope we
take that into account.
It feels like gem5 is getting towards privatization...
It’s just a opinion, it may be wrong.
On Wed, Dec 11, 2019 at 10:12 PM Gabe Black <gabebl...@google.com> wrote:
> Could you please clarify the policy on breaking APIs? It makes sense for
> releases to maintain stable APIs, but how does that apply to the
> development branch? I'm worried that it will be very hard to make changes
> that don't change any interfaces, and we definitely don't want to encourage
> a style of development where we just add and add and add without ever
> refactoring or consolidating things. If APIs can continue to change as
> needed in the development branch and we just need to warn people before
> they're released, that seems reasonable.
>
> In general (and not necessarily tied to this proposal), we should try very
> hard not to have silos where people develop incompatible features and don't
> add to gem5 in a holistic way. It's easiest to wall off functionality
> behind ifdefs or config options or in separate repositories and then just
> ignore it when implementing new features, but that creates a lot of
> technical debt and complexity which really cripples both gem5 and future
> development and compounds exponentially over time.
>
> Back to the branches, I'm still not a fan. I think it gives people an
> excuse not to make changes ("you want me to refactor 18 months of code for
> that?" (not an actual quote)), on top of other issues. Also there are more
> than two options. You can do your work on top of tree like I do, even when
> adding big new features. It ensures that you don't get too invested in bad
> paths and don't have a mountain of integration work to do when you finally
> decide to bridge back in. It also gives the community an appropriate
> opportunity to provide feedback through review which is not practical when
> dealing with dozens of files and thousands of lines of code with months of
> yet unseen history and iteration.
>
> Gabe
>
> On Wed, Dec 11, 2019 at 5:30 PM Jason Lowe-Power <ja...@lowepower.com>
> wrote:
>
> > Hi all,
> >
> > Here are the specific changes we're proposing. Please give us your
> > feedback! If you disagree with some part of this proposal, let us know
> > why and what you suggest instead.
> >
> > # What's changing now
> >
> > 1) We are going to release gem5 19 "soon" (two blocking issues: 1)
> > this proposal and 2) the website)
> > 2) When we release gem5 19 we are going to create a new branch called
> > "develop" which branches at the point we make the release.
> > 3) We are going to tag master with "19.0.0-beta"
> >
> > After this, all development will move to the develop branch instead of
> > master. This will allow the common case of people using gem5 for
> > research to be the default. When cloning gem5 you will get a "stable"
> > release.
> >
> > ## Changes for developers
> >
> > There will be two changes for developers:
> > 1) You'll have to checkout develop after cloning
> > 2) You'll have to push to "refs/for/develop" instead of "refs/for/master"
> >
> > # Next release
> >
> > When we get to our next release (gem5 20, yay!), we will merge the
> > develop branch into master. Assuming no hotfixes, this will be a
> > simple fast forward.
> >
> > We will have a major release once per year usually in April (after the
> > MICRO deadline but with enough lead time to put together an ISCA
> > tutorial). We will have two minor releases later in the year, one in
> > August and one in December.
> >
> > # Why are we going to releases?
> >
> > It's important for our *users* (remember, we're developing gem5 to be
> > used by other people, not just core developers!) to be able to build
> > off of stable APIs. Therefore, between major releases *we will not
> > change any APIs*. Code that we expect other people will build off of
> > are APIs. This includes, but is not limited to:
> > - Ports
> > - ExecContext
> > - Packet
> > - SimObject, ClockedObject, etc.
> > - Event, EventQueue, etc.
> > - Command line options
> > - And probably many more. We will work on finalizing this over the
> > next couple of months before the gem5-20 release.
> >
> > ## What does going to releases mean for developers?
> >
> > Thus, if any developers want to make changes to the above code, we must
> > Mark the API as "deprecated" at least by the *.2 December release to
> > give users at least one quarter to prepare.
> > AND either
> > 1) Make backwards compatible changes (e.g., just add a new function to
> > the API that we will switch to when the API merge window opens (after
> > the December minor release))
> > or 2) Create a new branch from develop that we wait to merge until the
> > merge window.
> >
> > # Feature branches
> >
> > Despite the disagreement in this thread so far, we're still proposing
> > feature branches. However, they should be *rare*, and only maintainers
> > will have the ability to add new branches. The exact gerrit mechanisms
> > for this need to be figured out.
> >
> > We *require* something like feature branches given this new release
> > model for two reasons:
> > 1) There will be API breaking changes that will be long-lived (at
> > least between a major release and the next API change merge window).
> > 2) Many people develop new features to incorporate into gem5 and
> > frequently they take months of *iteration* before they are ready to be
> > merged. AMD's GPU model in the gcn-staging branch is one example
> > (
> >
> https://gem5.googlesource.com/amd/gem5/+/refs/heads/agutierr/master-gcn3-staging
> > ),
> > the FlexCPU model my group has been working on is another
> > (https://github.com/darchr/gem5/tree/flexcpu).
> > We need an improved method of merging these kinds of changes.
> >
> > The way I see it, we have two options: continue development locally in
> > our own repos as we are doing right now. OR we could move the
> > development *into the open* by creating feature branches on the main
> > gem5 repo. I prefer the latter. There's little difference to Gerrit.
> > The main difference is what you get when you do git pull.
> >
> > It will be up to the maintainers to make sure branches are merged in a
> > timely fashion and we don't create too many branches. They should be
> > kept up to date by their maintainer by frequently merging the develop
> > branch.
> >
> > # Hotfixes to the stable/master branch
> >
> > It's likely we'll find bugs while developing gem5. If the maintainer
> > for the component thinks the bug is severe enough (I don't want to
> > make any specific guidelines right now), we will backport that bug to
> > the master branch. We'll have to work on our process for this to
> > ensure that develop will cleanly merge into master later, but this
> > should be straightforward.
> >
> > # Feedback
> >
> > Please let us know what you think! We're going to go one more round of
> > feedback on the gem5-dev list, then we'll post the finalized version
> > on gem5-users for a last round of feedback. I'm hoping to begin to
> > implement this early next year!
> >
> > Thanks for your patience on this. I know it's a lot to read through
> > and think about, but it's important that we get this process right the
> > first time. We don't want to have to change this every year :).
> >
> > Cheers,
> > Jason
> >
> > On Tue, Dec 10, 2019 at 4:49 PM Bobby Bruce <bbr...@ucdavis.edu> wrote:
> > >
> > > Thank you all for your comments on this.
> > >
> > > In regards to the master/develop branch idea, I'm a big advocate of
> > Jason's
> > > "make the most common use the easiest" philosophy. I think he did a
> good
> > > job of describing why this is important. We use the git repo as our
> > > primarily way of distributing gem5, and, as such, I believe the default
> > > behaviour should be that when a user clones the repo, they are getting
> > the
> > > latest stable release to build on top of. I strongly suspect users will
> > > disproportionately use whatever `git clone` gives them rather than move
> > > between branches or tags in order to get specific, stable releases. If
> > most
> > > users end up working on the very latest "unstable" development version
> > then
> > > our release system becomes pointless. I realize as everyone on this
> list
> > is
> > > a gem5 contributor, there is a preference for the development branch to
> > be
> > > default, but I'd hope we are in the minority of people who actually use
> > > gem5. My vote therefore still goes to having two branches: the
> > > master/"release" branch and the "develop" branch for day-to-day
> > > development. Hotfixes would be created on a separate branch and merged
> > into
> > > both upon completion.
> > >
> > > Regarding feature branching --- My intention with this part of the
> > proposal
> > > was not for feature branches to be the default manner in which people
> > > commit to gem5 (as they would be if we were to implement the GitFlow
> > model
> > > in full), I consider them something we should allow, in some form, to
> > > improve the development process in certain circumstances, but that they
> > > should be a rather rare thing. For example, a new component may need
> > > several months of development work, consisting of many commits.
> > Developers,
> > > at present, have two options: either A) gradually incorporate commits
> > over
> > > the months on the master branch, or B) push all the commits to the
> master
> > > branch in one go. I don't think either are ideal solutions. A
> dedicated
> > > feature branch would quarantine these changes while in active
> > development,
> > > to be merged at a later date. A second circumstance that could arise is
> > > when we wish to postpone the incorporation of a feature. A change that
> > > requires an API alteration, for example, should be held-off until we
> are
> > > ready to deploy a major release. Isolating such a change in a feature
> > > branch, to be merged later, may be necessary. This all being said, I
> > > acknowledge there is a risk in code getting lost indefinitely, or for
> > long
> > > periods of time, in feature branches, and our repo getting messy as a
> > > result. After some discussion with Jason, I believe the best of both
> > worlds
> > > can be achieved if we make feature branches "by request" --- anyone
> > wanting
> > > to create a feature branch would need to ask a maintainer (or a PCM) to
> > > create one for them. Whether or not a feature branch is to be created
> > would
> > > be decided upon on a case-by-case basis.
> > >
> > > I hope no one feels like I'm dictating anything here. I'm writing this
> > > primarily to prompt more discussion, so please feel free to counter any
> > of
> > > my points or suggest alternatives. Simple messages of support, as well
> as
> > > criticism, would also be appreciated to help us get a feel for what is,
> > or
> > > what is not, proving popular with the community.
> > >
> > > Kind regards,
> > > Bobby
> > > --
> > > Dr. Bobby R. Bruce
> > > Room 2235,
> > > Kemper Hall, UC Davis
> > > Davis,
> > > CA, 95616
> > >
> > >
> > > On Wed, Nov 27, 2019 at 4:49 PM Gabe Black <gabebl...@google.com>
> wrote:
> > >
> > > > The idea of feature branches makes me very nervous. It sounds like a
> > dark
> > > > corner for a large body of code to grow in, and then once it pops
> back
> > into
> > > > the light there may be very ingrained and difficult to fix problems
> > which
> > > > either have to be ignored or fixed. A long standing branch may also
> be
> > very
> > > > hard to reconcile with ToT, and it may be very tempting to just let
> it
> > die
> > > > rather than try to jump that energy barrier and get it back into
> > mainline.
> > > >
> > > > This would basically be sanctifying the practice of perma-forking
> gem5,
> > > > which unfortunately happens anyway now and then. There have been a
> > number
> > > > of times on the mailing list someone says something along the lines
> of
> > "I
> > > > downloaded special version of gem5 foo from some github" and it turns
> > out
> > > > it's X years old with a bunch of old and new bugs, nobody knows how
> it
> > > > works, etc. Let's not encourage that :-).
> > > >
> > > > Gabe
> > > >
> > > > On Wed, Nov 27, 2019 at 9:17 AM Vince Harron <vhar...@google.com>
> > wrote:
> > > >
> > > > > I agree with Ciro that the de facto standard used by most projects
> is
> > > > that
> > > > > mainline development should happen in master and releases should be
> > new
> > > > > branches created off of master.
> > > > >
> > > > > You could also create a branch “stable” or “latest” that gets
> > updated to
> > > > > the latest release.
> > > > >
> > > > > The default download instructions could say something like
> > > > >
> > > > > git clone <url> —branch stable
> > > > >
> > > > >
> > > > > On Wed, Nov 27, 2019 at 8:54 AM Jason Lowe-Power <
> > ja...@lowepower.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks for the input Ciro!
> > > > > >
> > > > > > My pushback here is that most people who use gem5 will be making
> > > > > > modifications to it (at least for the foreseeable future). I
> don't
> > > > think
> > > > > we
> > > > > > want to common use case to be 1) Download a tarball, 2) make
> local
> > > > > > modifications to the source. I think we want people to 1) clone a
> > > > > "stable"
> > > > > > gem5, 2) make modifications to the stable branch for their
> > research *in
> > > > > > git*.
> > > > > >
> > > > > > My experience with architecture grad students and researchers is
> > that
> > > > if
> > > > > > you don't force them to use things like git most of them won't.
> > One of
> > > > my
> > > > > > (personal) goals is to make it easy/default to use gem5 "the
> right
> > > > way".
> > > > > >
> > > > > > In fact, I almost don't want to provide tarballs at all. Unless
> we
> > can
> > > > > > provide pre-build gem5 binaries, I don't think it makes sense to
> > > > provide
> > > > > > tarballs. I don't think prebuilt binaries make sense for the
> > research
> > > > use
> > > > > > case of gem5 since almost all research will require extending
> gem5
> > and
> > > > > > there's no way to dynamically load in new modules (though we
> should
> > > > > support
> > > > > > this in the future!). For teaching with gem5 prebuilt binaries
> > could
> > > > make
> > > > > > sense, but I think we'll hold off on that for now, TBH.
> > > > > >
> > > > > > I think this comes down to "make the common case fast". I believe
> > the
> > > > > > common case is researchers who want to make minor changes to gem5
> > > > (e.g.,
> > > > > > add a new SimObject or modify the cache protocol) and run
> > experiments
> > > > for
> > > > > > their research. Developing gem5 for the community (using the
> > > > development
> > > > > > branch) is actually the uncommon case.
> > > > > >
> > > > > > BTW, I agree with you that we should try to follow whatever
> > practices
> > > > are
> > > > > > most popular when it makes sense. However, in many ways gem5 is a
> > > > unique
> > > > > > project, so following others doesn't always make sense :).
> > > > > >
> > > > > > I'm open to discussion on this. Thoughts?
> > > > > >
> > > > > > Cheers,
> > > > > > Jason
> > > > > >
> > > > > > On Wed, Nov 27, 2019 at 12:01 AM Ciro Santilli <
> > ciro.santi...@arm.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks Bobby for pushing this forward, having releases would
> be a
> > > > good
> > > > > > > thing for gem5.
> > > > > > >
> > > > > > > I would recommend against having both master and develop branch
> > > > though,
> > > > > > > because in the large majority of projects out there, master ==
> > the
> > > > > latest
> > > > > > > version, so I'm always confused when I have to switch to
> develop,
> > > > > > specially
> > > > > > > if it isn't the default HEAD (is is however possible to set the
> > > > default
> > > > > > > branch to develop as shown as:
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://gerrit-review.googlesource.com/Documentation/project-configuration.html#default-branch
> > > > > > > )
> > > > > > >
> > > > > > > This is even more important if we start having prebuilt
> releases
> > or
> > > > > tgzs
> > > > > > > with source (which are smaller than full repo) for the tags, at
> > which
> > > > > > point
> > > > > > > basically everyone who clones wants the latest.
> > > > > > >
> > > > > > > I would instead recommend:
> > > > > > >
> > > > > > > - master: latest version
> > > > > > > - v2019: a tag. The latest one can be easily found with:
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://stackoverflow.com/questions/1404796/how-to-get-the-latest-tag-name-in-current-branch-in-git
> > > > > > > - 2019 or b2019: branch created at the same time as v2019 to
> > which
> > > > > > > backports are applied, and to which v2019.0.1 tags can also be
> > > > > optionally
> > > > > > > applied
> > > > > > >
> > > > > > >
> > > > > > > ________________________________
> > > > > > > From: gem5-dev <gem5-dev-boun...@gem5.org> on behalf of Bobby
> > Bruce
> > > > <
> > > > > > > bbr...@ucdavis.edu>
> > > > > > > Sent: Wednesday, November 27, 2019 12:10 AM
> > > > > > > To: gem5-dev@gem5.org <gem5-dev@gem5.org>
> > > > > > > Subject: [gem5-dev] gem5 19.0.0 : New git branching proposal
> > > > > > >
> > > > > > > Dear all,
> > > > > > >
> > > > > > > As you know, at the end of this quarter we will be releasing
> > gem5-19,
> > > > > the
> > > > > > > first official release of gem5. As part of this release we'd
> > like to
> > > > > > change
> > > > > > > our git branching structure. Therefore, I'm writing to ask for
> > > > feedback
> > > > > > on
> > > > > > > what we have planned and whether it can be improved upon.
> > > > > > >
> > > > > > > We'd like to have a git repo structure similar to that used in
> > > > gitflow
> > > > > > > development:
> > https://nvie.com/posts/a-successful-git-branching-model
> > > > .
> > > > > > > I've
> > > > > > > seen this model work well before, as it has some worthwhile
> > > > abstractors
> > > > > > for
> > > > > > > public, open source git projects with regular releases. What
> I'd
> > like
> > > > > to
> > > > > > > incorporate from this model is the following:
> > > > > > >
> > > > > > > - Two permanent branches: master and develop.
> > > > > > > - "develop" would function as master does now. This would be
> the
> > main
> > > > > > point
> > > > > > > in which changes are applied between gem5 releases.
> > > > > > > - Upon a new release of gem5, the develop branch would be
> merged
> > into
> > > > > > > master and a new git tag added to master indicating the release
> > > > > version.
> > > > > > > Ergo, the master branch would always contain the latest release
> > of
> > > > > gem5.
> > > > > > > - If a quick hotfix is needed, a new "hotfix" branch would be
> > created
> > > > > and
> > > > > > > merged into both the develop and master branches upon
> completion.
> > > > This
> > > > > > > would also require a new tag on the master branch. (I suggest
> > using
> > > > the
> > > > > > > standard "Version [Major].[Minor].[Hotfix]" version numbering
> > system.
> > > > > > I.e.,
> > > > > > > the first version would be V19.0.0, a hotfix to this would make
> > it
> > > > > > V19.0.1,
> > > > > > > and a minor release would make it V19.1.0).
> > > > > > > - The creation of feature branches would be permitted. These
> > branches
> > > > > > would
> > > > > > > encapsulate the gradual development of large features (i.e.,
> ones
> > > > > carried
> > > > > > > out over many commits). When complete a feature branch would be
> > > > merged
> > > > > > into
> > > > > > > the develop branch. They'd be no obligation to use feature
> > branches
> > > > > > though
> > > > > > > we believe they could be of value in certain cases. For
> example,
> > if a
> > > > > > > developer wishes to postpone a developed feature for a given
> gem5
> > > > > release
> > > > > > > (e.g., something more suited for a major release rather than a
> > minor
> > > > > > one),
> > > > > > > then they could submit their changes as a feature branch and
> > wait to
> > > > > > merge
> > > > > > > to the develop branch at a later date.
> > > > > > >
> > > > > > > I believe this setup would make our development process run
> > smoother
> > > > > and
> > > > > > > give gem5 users more stability. Day-to-day development wouldn't
> > > > change
> > > > > > much
> > > > > > > as committing to the develop branch would work in the same way
> as
> > > > > > > submitting to master does now.
> > > > > > >
> > > > > > > If anyone has any thoughts about this, I'd be happy to hear
> from
> > you.
> > > > > > >
> > > > > > > Kind regards,
> > > > > > > Bobby
> > > > > > > --
> > > > > > > Dr. Bobby R. Bruce
> > > > > > > Room 2235,
> > > > > > > Kemper Hall, UC Davis
> > > > > > > Davis,
> > > > > > > CA, 95616
> > > > > > > _______________________________________________
> > > > > > > gem5-dev mailing list
> > > > > > > gem5-dev@gem5.org
> > > > > > > http://m5sim.org/mailman/listinfo/gem5-dev
> > > > > > > _______________________________________________
> > > > > > > gem5-dev mailing list
> > > > > > > gem5-dev@gem5.org
> > > > > > > http://m5sim.org/mailman/listinfo/gem5-dev
> > > > > > _______________________________________________
> > > > > > gem5-dev mailing list
> > > > > > gem5-dev@gem5.org
> > > > > > http://m5sim.org/mailman/listinfo/gem5-dev
> > > > >
> > > > > --
> > > > >
> > > > > Vince Harron | Engineering Manager | vhar...@google.com |
> > > > 858-442-0868 <(858)%20442-0868>
> > > > > <(858)%20442-0868>
> > > > > _______________________________________________
> > > > > gem5-dev mailing list
> > > > > gem5-dev@gem5.org
> > > > > http://m5sim.org/mailman/listinfo/gem5-dev
> > > > _______________________________________________
> > > > gem5-dev mailing list
> > > > gem5-dev@gem5.org
> > > > http://m5sim.org/mailman/listinfo/gem5-dev
> > > _______________________________________________
> > > gem5-dev mailing list
> > > gem5-dev@gem5.org
> > > http://m5sim.org/mailman/listinfo/gem5-dev
> > _______________________________________________
> > gem5-dev mailing list
> > gem5-dev@gem5.org
> > http://m5sim.org/mailman/listinfo/gem5-dev
> _______________________________________________
> gem5-dev mailing list
> gem5-dev@gem5.org
> http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
gem5-dev@gem5.org
http://m5sim.org/mailman/listinfo/gem5-dev
