Re: Welcome Tomoko Uchida to the PMC

[Tomoko Uchida]

 > How would you prefer to be called?

Call me Tomoko, or Uchida-san (Japanese style), either way you like!

> Omedetou Uchida-san! :-)

Arigatou!

Tomoko


2020年7月27日(月) 10:38 Ishan Chattopadhyaya :

> Omedetou Uchida-san! :-)
>
> On Mon, Jul 27, 2020 at 7:02 AM Christian Moen  wrote:
>
>> Congrats, Uchida-san.
>>
>> How would you prefer to be called?
>>
>> Thanks.
>>
>>
>> On Mon, Jul 27, 2020 at 9:41 AM Tomoko Uchida <
>> tomoko.uchida.1...@gmail.com> wrote:
>>
>>> Thank you all,
>>>
>>> Martin, let me just correct this...
>>>
>>> > ウェルコム  ウーキーダー
>>>
>>> ウェルカム ウチダ // there's no PROLONGED SOUND MARK! :)
>>>
>>> And if you're interested, here is my name written in four notations:
>>> Uchida Tomoko (in Roma-ji)
>>> ウチダ トモコ (in Katakana)
>>> うちだ ともこ (in Hiragana)
>>> 打田 智子 (in Kanji)
>>>
>>>
>>>
>>> 2020年7月27日(月) 4:28 Dawid Weiss :
>>>
 Congratulations and welcome, Tomoko!


 Dawid

 On Fri, Jul 24, 2020 at 7:03 PM Christine Poerschke (BLOOMBERG/
 LONDON)  wrote:
 >
 > Welcome Tomoko!
 >
 > From: dev@lucene.apache.org At: 07/04/20 08:27:28
 > To: dev@lucene.apache.org
 > Subject: Re: Welcome Tomoko Uchida to the PMC
 >
 > Thank you Adrien and the whole PMCs, it's a great honour for me!
 >
 > Tomoko
 >
 >
 > 2020年7月4日(土) 15:28 Christian Moen :
 >>
 >> Congrats, Tomoko-san!
 >>
 >> On Sat, Jul 4, 2020 at 15:26 Adrien Grand  wrote:
 >>>
 >>> I am pleased to announce that Tomoko Uchida has accepted the PMC's
 invitation to join.
 >>>
 >>> Welcome Tomoko!
 >>>
 >>>
 >>> --
 >>> Adrien
 >
 >

 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Tomoko Uchida to the PMC

[Ishan Chattopadhyaya]

Omedetou Uchida-san! :-)

On Mon, Jul 27, 2020 at 7:02 AM Christian Moen  wrote:

> Congrats, Uchida-san.
>
> How would you prefer to be called?
>
> Thanks.
>
>
> On Mon, Jul 27, 2020 at 9:41 AM Tomoko Uchida <
> tomoko.uchida.1...@gmail.com> wrote:
>
>> Thank you all,
>>
>> Martin, let me just correct this...
>>
>> > ウェルコム  ウーキーダー
>>
>> ウェルカム ウチダ // there's no PROLONGED SOUND MARK! :)
>>
>> And if you're interested, here is my name written in four notations:
>> Uchida Tomoko (in Roma-ji)
>> ウチダ トモコ (in Katakana)
>> うちだ ともこ (in Hiragana)
>> 打田 智子 (in Kanji)
>>
>>
>>
>> 2020年7月27日(月) 4:28 Dawid Weiss :
>>
>>> Congratulations and welcome, Tomoko!
>>>
>>>
>>> Dawid
>>>
>>> On Fri, Jul 24, 2020 at 7:03 PM Christine Poerschke (BLOOMBERG/
>>> LONDON)  wrote:
>>> >
>>> > Welcome Tomoko!
>>> >
>>> > From: dev@lucene.apache.org At: 07/04/20 08:27:28
>>> > To: dev@lucene.apache.org
>>> > Subject: Re: Welcome Tomoko Uchida to the PMC
>>> >
>>> > Thank you Adrien and the whole PMCs, it's a great honour for me!
>>> >
>>> > Tomoko
>>> >
>>> >
>>> > 2020年7月4日(土) 15:28 Christian Moen :
>>> >>
>>> >> Congrats, Tomoko-san!
>>> >>
>>> >> On Sat, Jul 4, 2020 at 15:26 Adrien Grand  wrote:
>>> >>>
>>> >>> I am pleased to announce that Tomoko Uchida has accepted the PMC's
>>> invitation to join.
>>> >>>
>>> >>> Welcome Tomoko!
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Adrien
>>> >
>>> >
>>>
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>>
>>>

Re: Welcome Tomoko Uchida to the PMC

[Christian Moen]

Congrats, Uchida-san.

How would you prefer to be called?

Thanks.


On Mon, Jul 27, 2020 at 9:41 AM Tomoko Uchida 
wrote:

> Thank you all,
>
> Martin, let me just correct this...
>
> > ウェルコム  ウーキーダー
>
> ウェルカム ウチダ // there's no PROLONGED SOUND MARK! :)
>
> And if you're interested, here is my name written in four notations:
> Uchida Tomoko (in Roma-ji)
> ウチダ トモコ (in Katakana)
> うちだ ともこ (in Hiragana)
> 打田 智子 (in Kanji)
>
>
>
> 2020年7月27日(月) 4:28 Dawid Weiss :
>
>> Congratulations and welcome, Tomoko!
>>
>>
>> Dawid
>>
>> On Fri, Jul 24, 2020 at 7:03 PM Christine Poerschke (BLOOMBERG/
>> LONDON)  wrote:
>> >
>> > Welcome Tomoko!
>> >
>> > From: dev@lucene.apache.org At: 07/04/20 08:27:28
>> > To: dev@lucene.apache.org
>> > Subject: Re: Welcome Tomoko Uchida to the PMC
>> >
>> > Thank you Adrien and the whole PMCs, it's a great honour for me!
>> >
>> > Tomoko
>> >
>> >
>> > 2020年7月4日(土) 15:28 Christian Moen :
>> >>
>> >> Congrats, Tomoko-san!
>> >>
>> >> On Sat, Jul 4, 2020 at 15:26 Adrien Grand  wrote:
>> >>>
>> >>> I am pleased to announce that Tomoko Uchida has accepted the PMC's
>> invitation to join.
>> >>>
>> >>> Welcome Tomoko!
>> >>>
>> >>>
>> >>> --
>> >>> Adrien
>> >
>> >
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>

Re: Welcome Tomoko Uchida to the PMC

[Tomoko Uchida]

Thank you all,

Martin, let me just correct this...

> ウェルコム  ウーキーダー

ウェルカム ウチダ // there's no PROLONGED SOUND MARK! :)

And if you're interested, here is my name written in four notations:
Uchida Tomoko (in Roma-ji)
ウチダ トモコ (in Katakana)
うちだ ともこ (in Hiragana)
打田 智子 (in Kanji)



2020年7月27日(月) 4:28 Dawid Weiss :

> Congratulations and welcome, Tomoko!
>
>
> Dawid
>
> On Fri, Jul 24, 2020 at 7:03 PM Christine Poerschke (BLOOMBERG/
> LONDON)  wrote:
> >
> > Welcome Tomoko!
> >
> > From: dev@lucene.apache.org At: 07/04/20 08:27:28
> > To: dev@lucene.apache.org
> > Subject: Re: Welcome Tomoko Uchida to the PMC
> >
> > Thank you Adrien and the whole PMCs, it's a great honour for me!
> >
> > Tomoko
> >
> >
> > 2020年7月4日(土) 15:28 Christian Moen :
> >>
> >> Congrats, Tomoko-san!
> >>
> >> On Sat, Jul 4, 2020 at 15:26 Adrien Grand  wrote:
> >>>
> >>> I am pleased to announce that Tomoko Uchida has accepted the PMC's
> invitation to join.
> >>>
> >>> Welcome Tomoko!
> >>>
> >>>
> >>> --
> >>> Adrien
> >
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Approach for a new Autoscaling framework

[Varun Thacker]

On Sun, Jul 26, 2020 at 1:05 AM Ilan Ginzburg  wrote:

> Varun, you're correct.
> This PR was built based on what's needed for creation (easiest starting
> point for me and likely most urgent need). It's still totally WIP and
> following steps include building the API required for move and other
> placement based needs, then also everything related to triggers (see the
> Jira).
>
> Collection API commands (Solr provided implementation, not a plug-in) will
> build the requests they need, then call the plug-in (custom one or a defaut
> one), and use the returned "work items" (more types of work items will be
> introduced of course) to do the job (know where to place or where to move
> or what to remove or add etc.)
>

This sounds perfect!

I'd be interested to see how can we use SamplePluginMinimizeCores for say
create collection but use FooPluginMinimizeLoad for add-replica

>
> Ilan
>
> Le dim. 26 juil. 2020 à 04:13, Varun Thacker  a écrit :
>
>> Hi Ilan,
>>
>> I like where we're going with
>> https://github.com/apache/lucene-solr/pull/1684 . Correct me if I am
>> wrong, but my understanding of this PR is we're defining the interfaces for
>> creating policies
>>
>> What's not clear to me is how will existing collection APIs like
>> create-collections/add-replica etc make use of it? Is that something that
>> has been discussed somewhere that I could read up on?
>>
>>
>>
>> On Sat, Jul 25, 2020 at 2:03 PM Ilan Ginzburg  wrote:
>>
>>> Thanks Gus!
>>> This makes a lot of sense but significantly increases IMO the scope and
>>> effort to define an "Autoscaling" framework interface.
>>>
>>> I'd be happy to try to see what concepts could be shared and how a
>>> generic plugin facade could be defined.
>>>
>>> What are the other types of plugins that would share such a unified
>>> approach? Do they already exist under another form or are just projects at
>>> this stage, like Autoscaling plugins?
>>>
>>> But... Assuming this is the first "facade" layer to be defined between
>>> Solr and external code, it might be hard to make it generic and get it
>>> right. There's value in starting simple, understanding the tradeoffs and
>>> generalizing later.
>>>
>>> Also I'd like to make sure we're not paying a performance "genericity
>>> tax" in Autoscaling for unneeded features.
>>>
>>> Ilan
>>>
>>> Le sam. 25 juil. 2020 à 16:02, Gus Heck  a écrit :
>>>
 Scanned through the PR and read some of this thread. I likely have
 missed much other discussion, so forgive me if I'm dredging up somethings
 that are already discussed elsewhere.

 The idea of designing the interfaces defining what information is
 available seems good here, but I worry that it's too auto-scaling focused.
 In my imagination, I would see solr having a standard informational
 interface that is useful to any plugin of any sort. Autoscaling should be
 leveraging that and we should be enhancing that to enable autoscaling. The
 current state of  the system is one key type of information, but another
 type of information that should exist within solr and be exposed to plugins
 (including autoscaling) is events. When a new node joins there should be an
 event for example so that plugins can listen for that rather than
 incessantly polling and comparing the list of 100 nodes to a cached list of
 100 nodes.

 In the PR I see a bunch of classes all off in a separate package, which
 looks like an autoscaling fiefdom which will be tempted if not forced to
 duplicate lots of stuff relative to other plugins and/or core.

 As a side note I would think the metrics system could be a plugin that
 leverages the same set of informational interfaces

 So there should be 3 parts to this as I imagine it.

 1) Enhancements to the **plugin system** that make information about
 the cluster available solr to ALL plugins
 2) Enhancements to the **plugin system** API's provided to ALL plugins
 that allow them to mutate solr safely.
 3) A plugin that we intend to support for our users currently using
 auto scaling utilizes the enhanced information to provide a similar level
 of functionality as is *promised* by our current documentation of
 autoscaling, there might be some gaps or differences but we should be
 discussing what they are and providing recommended workarounds for users
 that relied on those promises to the users. Even if there were cases where
 we failed to deliver, if there were at least some conditions under which we
 could deliver the promised functionality those should be supported. Only if
 we never were able to deliver and it never worked under any circumstance
 should we rip stuff out entirely.

 Implicit in the above is the concept that there should be a facade
 between plugins and the core of solr.

 WRT #1 which will necessarily involve information collected from remote
 nodes, we need 

Re: Welcome Mike Drob to the PMC

[Dawid Weiss]

Congratulations and welcome, Mike.

Dawid

On Fri, Jul 24, 2020 at 9:57 PM Anshum Gupta  wrote:
>
> I am pleased to announce that Mike Drob has accepted the PMC's invitation to 
> join.
>
> Congratulations and welcome, Mike!
>
> --
> Anshum Gupta

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Tomoko Uchida to the PMC

[Dawid Weiss]

Congratulations and welcome, Tomoko!


Dawid

On Fri, Jul 24, 2020 at 7:03 PM Christine Poerschke (BLOOMBERG/
LONDON)  wrote:
>
> Welcome Tomoko!
>
> From: dev@lucene.apache.org At: 07/04/20 08:27:28
> To: dev@lucene.apache.org
> Subject: Re: Welcome Tomoko Uchida to the PMC
>
> Thank you Adrien and the whole PMCs, it's a great honour for me!
>
> Tomoko
>
>
> 2020年7月4日(土) 15:28 Christian Moen :
>>
>> Congrats, Tomoko-san!
>>
>> On Sat, Jul 4, 2020 at 15:26 Adrien Grand  wrote:
>>>
>>> I am pleased to announce that Tomoko Uchida has accepted the PMC's 
>>> invitation to join.
>>>
>>> Welcome Tomoko!
>>>
>>>
>>> --
>>> Adrien
>
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Approach for a new Autoscaling framework

[Gus Heck]

"There's value in starting simple, understanding the tradeoffs and
generalizing later"

Yes, this is what I am alluding to when I said the facade should "grow and
expand such that more plugins can rely on it". One plugin that might
re-use some of the same informational API's is the HealthCheckHandler. One
of the difficult things about the current situation is that it is quite
difficult to identify what is and is not a plugin in the code base.
PluginBag means that any interface could indicate a plugin if someone
has added a plugin bag for that type somewhere... I think we really ought
to have a base interface for all plugins. Another issue is that the
"plug-in" designation doesn't really have a lot of meaning in solr other
than it can be loaded by a config somewhere. Metrics currently have
reporters that "plug in" but  are held in a plain hashmap not a PluginBag.
The metrics system is another place where the info API's used by auto
scaling might be reused.

As I understand it, the state of the plugin system is that we've plumbed in
tools for fetching jars, replicating them to the nodes, installing them at
the cluster level and deploying them which is all great (I learned much of
this from Dave Smiley's talk at a local metup

-
starts at 0:30:29) . I'm unaware of what's been done relative to the
environment in which plugins live and may not be up to date on developments
since then.  As painful as it is, I think we will regret it if we don't
cordon off internal (behind the facade) and external (the facade) api's for
plugins. Other issues that come up include class-loading especially as
concerns different versions of dependencies... differing from something in
solr and differing from something loaded by some other plugin. I
recently did some work in JesterJ to enable such, and I would like to look
at what we are doing there (in some other disucssion, perhaps after I
refresh my memory on SolrResourceLoader's details etc) . I suspect that has
been thought about. Also, there should be an easy way for a plugin to load
a resource and have it be shared across cores. Many moons ago I contributed
a means by which to share large resources across cores and for
multi-tenant systems with lots of collections all created from the same
config, this can be important (the motivating case involved a gigabyte
sized map of geo locations to be applied at query time that was being
duplicated across 40 cores and crushing the servers).

So something has to be the starting point for thinking about the
"environment" in which plugins reside once they are deployed... Autoscaling
poked its head up ;)... I think it's going to be big enough we really
should build it on a foundation we want to keep.

The alternative of course is that everything in solr is available for
plugins to touch and we just break any number of plugins any time we change
anything... That's the current state really, but the majority of useful
plugins are in the same code base and have unit tests that force us to
maintain them. As that goes away, and as we (hopefully) encourage an
ecosystem of 3rd party plugins this becomes a bigger issue because we *won't
know* what we are breaking.

-Gus

On Sat, Jul 25, 2020 at 5:03 PM Ilan Ginzburg  wrote:

> Thanks Gus!
> This makes a lot of sense but significantly increases IMO the scope and
> effort to define an "Autoscaling" framework interface.
>
> I'd be happy to try to see what concepts could be shared and how a generic
> plugin facade could be defined.
>
> What are the other types of plugins that would share such a unified
> approach? Do they already exist under another form or are just projects at
> this stage, like Autoscaling plugins?
>
> But... Assuming this is the first "facade" layer to be defined between
> Solr and external code, it might be hard to make it generic and get it
> right. There's value in starting simple, understanding the tradeoffs and
> generalizing later.
>
> Also I'd like to make sure we're not paying a performance "genericity tax"
> in Autoscaling for unneeded features.
>
> Ilan
>
> Le sam. 25 juil. 2020 à 16:02, Gus Heck  a écrit :
>
>> Scanned through the PR and read some of this thread. I likely have missed
>> much other discussion, so forgive me if I'm dredging up somethings that are
>> already discussed elsewhere.
>>
>> The idea of designing the interfaces defining what information is
>> available seems good here, but I worry that it's too auto-scaling focused.
>> In my imagination, I would see solr having a standard informational
>> interface that is useful to any plugin of any sort. Autoscaling should be
>> leveraging that and we should be enhancing that to enable autoscaling. The
>> current state of  the system is one key type of information, but another
>> type of information that should exist within solr and be exposed to plugins
>> (including autoscaling) is events. When a 

Re: Approach for a new Autoscaling framework

[Ilan Ginzburg]

Varun, you're correct.
This PR was built based on what's needed for creation (easiest starting
point for me and likely most urgent need). It's still totally WIP and
following steps include building the API required for move and other
placement based needs, then also everything related to triggers (see the
Jira).

Collection API commands (Solr provided implementation, not a plug-in) will
build the requests they need, then call the plug-in (custom one or a defaut
one), and use the returned "work items" (more types of work items will be
introduced of course) to do the job (know where to place or where to move
or what to remove or add etc.)

Ilan

Le dim. 26 juil. 2020 à 04:13, Varun Thacker  a écrit :

> Hi Ilan,
>
> I like where we're going with
> https://github.com/apache/lucene-solr/pull/1684 . Correct me if I am
> wrong, but my understanding of this PR is we're defining the interfaces for
> creating policies
>
> What's not clear to me is how will existing collection APIs like
> create-collections/add-replica etc make use of it? Is that something that
> has been discussed somewhere that I could read up on?
>
>
>
> On Sat, Jul 25, 2020 at 2:03 PM Ilan Ginzburg  wrote:
>
>> Thanks Gus!
>> This makes a lot of sense but significantly increases IMO the scope and
>> effort to define an "Autoscaling" framework interface.
>>
>> I'd be happy to try to see what concepts could be shared and how a
>> generic plugin facade could be defined.
>>
>> What are the other types of plugins that would share such a unified
>> approach? Do they already exist under another form or are just projects at
>> this stage, like Autoscaling plugins?
>>
>> But... Assuming this is the first "facade" layer to be defined between
>> Solr and external code, it might be hard to make it generic and get it
>> right. There's value in starting simple, understanding the tradeoffs and
>> generalizing later.
>>
>> Also I'd like to make sure we're not paying a performance "genericity
>> tax" in Autoscaling for unneeded features.
>>
>> Ilan
>>
>> Le sam. 25 juil. 2020 à 16:02, Gus Heck  a écrit :
>>
>>> Scanned through the PR and read some of this thread. I likely have
>>> missed much other discussion, so forgive me if I'm dredging up somethings
>>> that are already discussed elsewhere.
>>>
>>> The idea of designing the interfaces defining what information is
>>> available seems good here, but I worry that it's too auto-scaling focused.
>>> In my imagination, I would see solr having a standard informational
>>> interface that is useful to any plugin of any sort. Autoscaling should be
>>> leveraging that and we should be enhancing that to enable autoscaling. The
>>> current state of  the system is one key type of information, but another
>>> type of information that should exist within solr and be exposed to plugins
>>> (including autoscaling) is events. When a new node joins there should be an
>>> event for example so that plugins can listen for that rather than
>>> incessantly polling and comparing the list of 100 nodes to a cached list of
>>> 100 nodes.
>>>
>>> In the PR I see a bunch of classes all off in a separate package, which
>>> looks like an autoscaling fiefdom which will be tempted if not forced to
>>> duplicate lots of stuff relative to other plugins and/or core.
>>>
>>> As a side note I would think the metrics system could be a plugin that
>>> leverages the same set of informational interfaces
>>>
>>> So there should be 3 parts to this as I imagine it.
>>>
>>> 1) Enhancements to the **plugin system** that make information about the
>>> cluster available solr to ALL plugins
>>> 2) Enhancements to the **plugin system** API's provided to ALL plugins
>>> that allow them to mutate solr safely.
>>> 3) A plugin that we intend to support for our users currently using auto
>>> scaling utilizes the enhanced information to provide a similar level of
>>> functionality as is *promised* by our current documentation of autoscaling,
>>> there might be some gaps or differences but we should be discussing what
>>> they are and providing recommended workarounds for users that relied on
>>> those promises to the users. Even if there were cases where we failed to
>>> deliver, if there were at least some conditions under which we could
>>> deliver the promised functionality those should be supported. Only if we
>>> never were able to deliver and it never worked under any circumstance
>>> should we rip stuff out entirely.
>>>
>>> Implicit in the above is the concept that there should be a facade
>>> between plugins and the core of solr.
>>>
>>> WRT #1 which will necessarily involve information collected from remote
>>> nodes, we need to be designing that thinking about what informational
>>> guarantees it provides. Latency, consistency, delivery, etc. We also need
>>> to think about what is exposed in a read-only fashion vs what plugins might
>>> write back to solr. Certainly there will be a lot of information that most
>>> plugins ignore, and we might