This is a good list of things that we could and should improve. Getting
all minded to perfection would be difficult, but definitely worth living
into.
I appreciate your enthusiasm for and involvement in D. At the top level
a few simple realities need to be understood.
First, there is no OSS community that doesn't have its annoyances. I've
been a direct participant to one other and am lurking on the forums of
three more. Some are led by juntas of mini-tyrants; some are
unnecessarily snooty; some are consumed by intestine fighting, politics,
and horse-trading; and so on.
Second, some of your points revolve around the fact that Walter and
myself are not as good as we should at certain tasks and roles, such as
build master, PR person, manager, and operational officer. The general
argument pattern revolves around "I/we told Walter/Andrei several times
they need to do X, and they forget/ignore/do it badly." Clearly Walter
is a self-confessed mediocre build czar at best. But he's doing it
because, although there is no shortage of suggestions on how he should
be doing it, nobody has the time and inclination to actually _be_ the
build czar (which is entirely understandable).
Within a traditional organization where people are paid to work on a
certain project, the solution would be simple and obvious - Walter would
be relieved of the roles he doesn't excel in, and left to focus on those
he's really good at. Other people would fill in duties and
responsibilities such as preparing betas, sending them out for testing,
announcing them, etc.
Within our current organizational landscape, where nobody is paid to
work on D (not _with_ D) except for myself, addressing issues such as
"we should have a better build master" is much more difficult to address.
Third, for what it's worth, here's what I believe are the top issues
with D today:
1. Nobody is being paid to work on D (aside from me since recently, and
on work that would benefit my employer).
2. D is not backed up solidly by a large engineering organization.
3. We are unable to review and accept contributions at the rate they are
arriving.
I tend to ask myself how various proposals for improving our process
address these three key points. I believe your list effects mostly (3),
albeit not directly.
More answers inline.
> - We do not have any vision or major plans ahead for the language.
Currently we're stuck in a bug-driven development environment, where
bugs are arbitrarily picked off of bugzilla and fixed. But there's no
major plans ahead, e.g. "let's plan to fix these X major bugs for some
upcoming release". We can't force people to work on X or Y, but if
they're in a visual place and marked important and scheduled to be
fixed, this will give an incentive for contributors to work on these
problems.
Walter and I frequently think of ways to gently steer people toward
working on specific items. Votes, categorization, discussions are of
limited impact. On occasion we've emailed major contributors directly
and asked politely but point blank if they could look into an issue
(sometimes something they had an expertise in, or even an issue they had
caused). The rate of success has been very low. People still love
working on things, just not on the things they're told to.
We've added the "preapproved" tag - take a look:
http://d.puremagic.com/issues/buglist.cgi?quicksearch=preapproved. A
couple have opened pull requests, which have not yet received reviews
yet (which is not my or Walter's fault as much as a larger community
issue that we need to fix, see (3) above). Most don't. You yourself
didn't find the time to update a task you volunteered on.
That said, it's entirely possible a formula for success does exist.
Looking forward to proposals, like improving the visuals of "bugs to be
fixed". What I think is obvious is that solution involving more work for
Walter and myself are unlikely to work well, for the simple reason we
are both impatiently waiting for the sun to rise every day to do more
work on D.
- We do not have any defined release timeline. When is it time to
start prepping for a release? It's up to Walter's arbitrary decision
for when this happens, he doesn't even talk to the community or
contributors on whether it's a good time for a beta phase (maybe
there's a huge or disruptive new pull request that's planning to be
merged and a beta should be delayed).
I understand how that can be a bother. Walter figures the time is ripe
for a new release when we have enough compelling features and fixes.
I'll try to make the appropriate announcements in the future prior to
deciding on starting a beta.
- We do not have a defined timeline for the beta testing period. How
long before we decide that the beta has been tested for long enough
before a release is made? Again, it's up to Walter's decision.
We are generally aiming for two weeks, but it sometimes gets longer
because of the impending regressions. One good phenomenon has been that
we've got an increasing number of people who are testing the beta.
Having a defined release cycle and a beta testing period will
ultimately be beneficial for everyone. People who are busy can
rearrange their schedule to make free time and ensure they have enough
time to test a beta compiler on their projects, and contributors to D
can prepare for a cycle of days or weeks where they can rapidly work
on reducing regressions and polish everything up for the release (e.g.
writing an elaborate changelog).
I also believe a better cycle would be beneficial, but I don't think it
would address our core issues.
- We do not have a proper release system. Nick Sabalausky has been
working hard on one[1], but Walter seems to have completely ignored
it. It would have been a terrific opportunity to try and make it work
for this release. What better way to test it than to test it on a
release?
We'll look into it, but this harkens back to the simple dynamics
mentioned above. That is essentially a request for Walter to change the
way he does releases, and people tend to get set in their ways and have
difficulty finding the time to stop and change things when there are so
many other things vying for their attention. The best solution here is
if Nick (or someone else) would _be_ the build manager using those great
tools that Nick wrote.
Anyhow, I'll look into that.
- The betas are still not visible. The homepage makes no notes on a
new beta being available, the download page does not list the betas
either, and AFAICT there's no RSS feed either. The social media groups
are not notified either (neither Andrei's nor Walter's twitter feed
make a mention of a new beta being out, the same applies to
https://twitter.com/D_Programming or the #dlang hash tags). To top it
all off, you cannot post to the dmd-beta newsgroups from the D Forums,
you have to separately register to this mailing list.
As a simple start, did you consider announcing the beta yourself? Anyone
can tweet #dlang @D_programming, and in all likelihood we and many
others would retweet. Also, did you consider submitting a pull request
for placing the beta announcement on the homepage?
If we want user feedback on betas we absolutely must make the betas
visible and give an opportunity for people to post feedback.
I agree. "We" is the right word.
- Walter is still not tagging the beta releases by the file name (it's
always dmd2beta.zip). I've complained about this several times and
IIRC someone else did as well at dconf (maybe I'm remembering wrong
though). They should at least be named as "dmd2_064_beta1.zip",
"dmd2_064_beta2.zip". And all of them should always be available for
download (including visibility on the download page), so people who do
not use Git or build manually from master can quicky check whether a
regression was introduced in a specific beta version.
Probably that's a good thing to do, and easy enough. I don't think it
would push the needle significantly.
- I still sigh when I hear about Walter and Andrei having private
phone conversations, or any kind of decision is made behind-the-scenes
rather than publicly where the community has a say in it. Walter's
implementation of UDAs directly in master which lead to having a
deprecated syntax even though nobody used this syntax is what comes to
mind.
I understand this is well intended but it's the kind of remark that
bends me out of shape. First, there's no substitute for real
communication than direct conversation. We can't have a phone
conversation with the community. Then, your first three points complain
about a lack of leadership, and in the same breath you complain about
there being too much of it.
We believe we've done well in making this community a meritocracy where
competent contributors can make a difference and make their word heard.
To the extent we're doing it insufficiently, it's because too few people
assume the responsibility to review and accept contributions.
Walter scrambled to implement UDAs in a rush and breaking protocol in
order to win a corporate D user, Remedy Games. It was a major,
exceptional event. Would you have preferred the protocol to have been
followed at the cost of Remedy?
- Both Walter and Andrei not following the rules and making novice
mistakes w.r.t Git and Github. Walter still seems to struggle with
basic usage of git, where people continually have to re-explain what's
wrong and how to fix an issue. I'm sorry, but if someone bought the
Git book years ago and is still struggling with *concepts* then no
amount of hand-guiding is going to help.
I agree that's a problem. Probably what would help would be more quality
reviews and pulls by our members with commit rights, of whom you are one.
And Andrei doing things like
merging a dozen pull requests at once, with complete disregard to the
fact that merging to master means other pulls could easily break (and
so master can be broken). You cannot make so many merges unless you're
absolutely sure each pull request does not interfere with another.
If a pull request is sensible and meaningful it shall be pulled. That's
the way we do things at Facebook, too, and it scales. There may be pulls
that are so closely related they break the build when put in
conjunction, but I believe that case is rare.
I also want to convey a bit of perspective here. There are 221 open pull
requests right now. These are 221 instances of good work put in by
talented people, who associated hopes and wishes to improve things with
that work. This bothers me. I lose sleep at night because of it. And the
fact that those contributions don't get looked at is our entire
community's fault as it is mine.
So when I have a few free minutes, I try to take a look at the extant
pull requests - really, any would do. I try to pull in those that are
meaningful and uncontroversial. I agree I could probably spend more time
on carefully analyzing interactions between pulls, but that's time I
can't afford.
- Back to Walter, a few weeks ago he merged a pull request over night,
without regard to the pull request not being fully tested by the test
machines. The result? Master was broken **for the entire next day**.
Nobody knew which commit broke master, so nothing was done until
Walter came back to Github the next day and started reverting his
pulls. In the meantime the entire day was wasted because nobody's pull
request could get green. Luckily it was a sunday, so there weren't too
many complaints. But I could have easily merged a few pulls that day
(as it happens I like to do things on a sunday), and as a result we
would have a smaller pull queue.
I guess such accidents are bound to happen to the best of us. A build
czar with the appropriate skills would have swiftly undone the damage.
But there is no build czar.
There's just many things that are going plain wrong here, and a lot of
promises are always made but ultimately never delivered (whether it's
about breaking changes or an improved development process -- again
think about scheduling bug fixes for the future, we could help people
prepare for the breaking changes).
Whatever we don't get delivered, it's not for the lack of trying.
Whatever we do, it doesn't come easily.
Personally I find D to be a huge part of me right now (probably not
the other way around, I'm a small part of this compared to the huge
contributors), and I feel really bummed out when both Andrei and
Walter take a casual stance when things are broken (whether it's the
system itself or something specific like master being broken). I
really think we have a great thing going here with the language, but
everything else has to improve, and particularly the development and
release process.
So that's what I'm protesting about.
The individual points are salient, but I fail to grab the most
significant bit. The logic doesn't follow.
We're in this together; to wit, for many of the things you're asking why
they don't get done, you could be asked the same thing.
You did a great job on formatting the changelog. It has been an
unqualified success, it was amazing marketing for D, and it has been
thoroughly appreciated by everyone. It is great work that we need more
of. Now you're not doing that work in protest against... not enough work
getting done.
Andrei
