The meeting notes are now appended to the meeting agenda page:
https://cwiki.apache.org/confluence/display/SOLR/2019-11+Meeting+on+SolrCloud+and+project+health
I'll paste that part here for convenience but it's formatted nicer online:
Meeting Notes
*Attended by*: Andrzej Bialecki, Anshum Gupta, Cassandra Targett, Chris
Hostetter, David Smiley, Erick Erickson, Gus Heck, Ishan Chattopadhyaya,
Jan Hoydahl, Jason Gerlowski, Mike D. (Apple), Noble Paul, Shawn Heisey,
Scott Blum, Tomas F. Lobbe, Yonik Seeley
*Duration*: 95 minutes.
Mark’s WIP Code
We heard from Thomas, Anshum, and Mike D. (Mark Miller’s colleagues.) who
spoke with Mark at length and have his work in progress.
- Mark shared a large code dump with his colleagues.
- He’s not comfortable sharing it in this state as it’s too
work-in-progress. His colleagues are going to respect this and not simply
share it as-is.
- His colleagues are actively working together on teasing out separate
improvements that will each get their own JIRA issue and code. This is
hard work and it will occur rather slowly over time (probably more than a
month). And when each issue is filed with code, it will usually be WIP.
Rarely will it be something immediately committable.
Apache Curator
Migrating to use Curator is a great thing for many reasons (see agenda)
including performance, though it’s not a singular solution for any/most
SolrCloud problems. Probably no draw-backs but it’s work. Changing this
(or many SolrCloud internals for that matter) cause tests to break (Mark
said), and it’ll take time to fix such tests.
Overseer
- It “clings to leadership” much more than it should.
- SolrCloud over-uses the Overseer for too many functions that could be
done without it. We’ll probably always want an Overseer though.
- Noble’s work on Smart State Caching (SOLR-13951
<https://issues.apache.org/jira/browse/SOLR-13951>)will help.
- Sometimes writing directly to ZooKeeper (helped with Curator
recipes) is sufficient.
Service Protection
Solr doesn’t have much service protection. If you create thousands of
collections, it’ll lock up and become inoperable. Scott reported that If
you boot up a 100+ node cluster, SolrCloud won’t get to a happy state;
currently you need to start them gradually. A well-written service won’t
lock-up; it will make the client wait and/or give the client an error. The
autoscaling framework is supposed to help; it’s a start and AB is working
on that somewhat. It’s probably not the only answer here.
Benchmarks
Ishan is working on addressing the need for continuously running benchmarks
[SOLR-13933]. Having such benchmarks is rather foundational for the theme
of performance improvements. And that, perhaps surprisingly, helps us
achieve stability.
Tests
Mark believes in tightly limiting test times that shouldn’t take long. He
used this while working on his improvements. Smiley suspects this approach
may only be useful in local dev but not in C.I. where virtual overloaded
machines cold be quite slow. Furthermore, he believes the objective there
can be addressed better via benchmarks.
Logging
Much was said but unclear what action to take here; it’s a bike-shed
topic. Separate concerns depending on the audience -- production users or
us developers? Hoss reminded us of the LogLevel annotation and suggested
it’d be neat if the level could be automatically set to debug based on the
package of the test.
Tech Debt
The overarching theme of what Mark raised is perhaps tech debt. Some
miscellaneous things to add here: We should spend more effort removing old
things (Smiley cares about this). And for lots of functionality to
continue to maintain, we hope the plugin system would lead to a future
where Solr needn’t absorb everything or needn’t be official contribs.
Code Reviews
We want to get reviews, even extremely superficial reviews that might not
look at the code but do look at the description and comments said about the
state of the code. Apparently ASF “RTC” policy suggests 3 binding votes
are required, which is of course an extremely high bar and not paletable.
Even without formally changing, we’d like to try out a 6 month period of
behaving this way for all but the most trivial of changes. Smiley takes an
action item to make a specific proposal soon.
Major Change Proposals
See Kafka “KIP” as an example. This interests us but it’s very unfamiliar
to us. We want to try it out. Perhaps SOLR-13951 might be worth
experimenting with this. Perhaps a Confluence page is the right place to
put the text? Issue argued Google Docs is more collaborative, e.g. inline
commenting. Hoss argued Confluence has this to, but might need to be
enabled. Today, without a major change proposal mechanism, some JIRA
issues are onerous to decipher. Irrespective of this, Hoss advocated we
continously update our JIRA issue descriptions to be useful during the
course of the issue, especially at conclusion.
Documentation
We agreed we need several layers of docs: Javadocs, Developer Guide, User
Guide. Javadocs is clearly in the code and we want more of them! Developer
Guide is currently unknown wether we prefer Confluence or use
asciidoc/markdown in a dedicated directory in our code repo.
Closing Remarks
We really liked meeting to discuss these matters. Gus and Jan proposed
doing this quarterly timed to occur near when the ASF board reports are due
so that we can discuss anything to add.
Action items
- Mark's colleagues to introduce Mark's code piece by piece into new
JIRA issues over time
- Ishan to introduce a periodic benchmark system
- Noble to try out a "Solr Improvement Proposal" or some-such in a new
initiative pertaining to ZK / clusterstate matters.
- David to propose a code review proposal to discuss on the dev list
- David to organize the next meeting near March 1st (before ASF board
report being due that month)
On Mon, Nov 18, 2019 at 11:41 AM Erick Erickson <erickerick...@gmail.com>
wrote:
> No problem in deferring the Gradle discussion. I’m getting into it a
> little deeper so I’ll know more about how close it is to prime-time as time
> passes..
>
> > On Nov 18, 2019, at 11:04 AM, Mike Drob <md...@apache.org> wrote:
> >
> > What's the medium for this? Is somebody going to make a
> webex/zoom/meet/whatever link? I'd like to make sure I have the right tools
> installed and will do my best to join from airport wifi.
> >
> > On Mon, Nov 18, 2019 at 2:49 AM Jan Høydahl <jan....@cominvent.com>
> wrote:
> > I view Jörn’s SIP proposal as a way for tighter gate keeping of future
> major changes, helping maintain hard fought stability and speed.
> >
> > Jan Høydahl
> >
> >> 18. nov. 2019 kl. 06:26 skrev David Smiley <david.w.smi...@gmail.com>:
> >>
> >>
> >> This particular committer's meeting has a particular subject/theme. So
> as Erick indicated, while all committers are invited, I think if you're not
> interested in the subject then I'm sure you can find a better use of your
> time. The "Solr Implementation Proposal" could fit in I suppose, but not
> Gradle (sorry Erick).
> >>
> >> Lets have another committer meeting in the near future too to discuss
> whatever Gradle and whatever. I look forward to having more opportunities
> to collaborate with more direct synchronous conversation.
> >>
> >> ~ David Smiley
> >> Apache Lucene/Solr Search Developer
> >> http://www.linkedin.com/in/davidwsmiley
> >>
> >>
> >> On Sun, Nov 17, 2019 at 8:29 PM Ishan Chattopadhyaya <
> ichattopadhy...@gmail.com> wrote:
> >> Jan, I think that is something that merits its own mail thread or jira
> for broader community participation.
> >>
> >> On Mon, 18 Nov, 2019, 5:19 AM Jan Høydahl, <jan....@cominvent.com>
> wrote:
> >> I added this bullet point under "Community":
> >>
> >> • Should we adopt a formal decision process for proposing major
> changes/APIs to Solr, aka "Solr Implementation Proposal (SIP)"?
> >> • See proposal from Jörn Franke in dev@ referring to KIP
> and FLIP
> >>
> >> Example: With SIP we would probably not have ended up with three
> overlapping facet/stats implementations without a migration plan :)
> >>
> >> --
> >> Jan Høydahl, search solution architect
> >> Cominvent AS - www.cominvent.com
> >>
> >>> 17. nov. 2019 kl. 15:13 skrev David Smiley <david.w.smi...@gmail.com>:
> >>>
> >>> The meeting will be held on Wednesday at 11am PST (2pm EST). I'll
> send a calendar invite tomorrow to those that used the Doodle poll.
> >>> The agenda is here:
> https://cwiki.apache.org/confluence/display/SOLR/2019-11+Meeting+on+SolrCloud+and+project+health
> Edit it if you wish. If you can't (e.g. you are not a committer), then
> maybe reply here to offer a suggestion.
> >>>
> >>> ~ David Smiley
> >>> Apache Lucene/Solr Search Developer
> >>> http://www.linkedin.com/in/davidwsmiley
> >>>
> >>>
> >>> On Wed, Nov 13, 2019 at 1:56 PM David Smiley <david.w.smi...@gmail.com>
> wrote:
> >>> I am proposing a virtual "Solr committer meeting" next week to discuss
> the concerns and ideas Mark Miller has raised recently. The scope of
> conversation of this one is not the other wonderful things people are
> working on (e.g. new plugins architecture), though I would *love* to have
> future virtual committer meetings that are open-ended to discuss all manner
> of things on our minds.
> >>>
> >>> Why? (A) I felt the committer meeting at Activate this year was
> fantastically productive (B) I love opportunities to see/hear from my
> awesome virtual colleagues (C) Mark raised issues of grave concern to our
> community
> >>>
> >>> When exactly is this?: I'm using a "Doodle poll" to determine an
> optimal time slot. For the link to the poll, go to the ASF Slack,
> #solr-dev channel, and you will see it. You could also email me directly
> for it.
> >>>
> >>> For this virtual committer meeting and future ones:
> >>> * This is in the spirit of committer meetings co-located with
> conferences. I recall that ASF policy says that no "decisions" can be made
> in such a venue. We make decisions on this dev list and indirectly via
> JIRA out in the open and with the opportunity for anyone to comment.
> >>> * Who: Committer-only or by invitation
> >>> * Video chat with option of audio dial-in. This time I will use
> Google Hangout.
> >>> * Recorded for those invited only. I'll dispose of the recording a
> week after. The intention is for those who cannot be there due to a
> scheduling conflict to see/hear what was said. I have the ability to do
> this recording via Salesforce's G-Suite subscription.
> >>> * Published notes: I (or someone) will take written meeting notes
> that are ultimately published for anyone to see (not restricted to those
> invited). They will be transmitted to the dev list.
> >>>
> >>> Thanks everyone. I hope we do more of these! And I hope
> non-committers don't feel slighted; the published notes is a great time to
> make your voice heard here! I'm open to suggestions.
> >>>
> >>> ~ David Smiley
> >>> Apache Lucene/Solr Search Developer
> >>> http://www.linkedin.com/in/davidwsmiley
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>
