[Wikitech-l] Re: ORES To Lift Wing Migration
Luca writes: > Managing several hundreds models for goodfaith and damaging is not very scalable in a modern micro-service architecture like Lift Wing > (since we have a model for each supported wiki). We (both Research and ML) are oriented on having fewer models that manage more languages at the same time, Is there a reason to think that separate models for each wiki are more effective than one general model that sees the name of the wiki as part of its context? I'd love to read more about the cost of training and updating current models, how much material they are trained on, and how others w/ their own GPUs can contribute to updates. Personally I wouldn't mind a single model that can suggest multiple properties of an edit, including goodfaith, damaging, and likelihood of reversion. They are different if related concepts -- the first deals with the intent and predicted further editing history of the editor, the second with article accuracy and quality, and the latter with the size + activity + norms of the other editors... SJ On Fri, Sep 22, 2023 at 5:34 PM Aaron Halfaker wrote: > All fine points. As you can see, I've filed some phab tasks where I saw a > clear opportunity to do so. > > > as mentioned before all the models that currently run on ORES are > available in both ores-legacy and Lift Wing. > > I thought I read that damaging and goodfaith models are going to be > replaced. Should I instead read that they are likely to remain available > for the foreseeable future? When I asked about a community discussion > about the transition from damaging/goodfaith to revertrisk, I was imagining > that many people who use those predictions might have an opinion about them > going away. E.g. people who use the relevant filters in RecentChanges. > Maybe I missed the discussions about that. > > I haven't seen a mention of the article quality or article topic models in > the docs. Are those also going to remain available? I have some user > scripts that use these models and are relatively widely used. I didn't > notice anyone reaching out. ... So I checked and setting a User-Agent on my > user scripts doesn't actually change the User-Agent. I've read that you > need to set "Api-User-Agent" instead, but that causes a CORS error when > querying ORES. I'll file a bug. > > On Fri, Sep 22, 2023 at 1:22 PM Luca Toscano > wrote: > >> >> >> On Fri, Sep 22, 2023 at 8:59 PM Aaron Halfaker >> wrote: >> >>> We could definitely file a task. However, it does seem like >>> highlighting the features that will no longer be available is an >>> appropriate topic for a discussion about migration in a technical mailing >>> list. >>> >> >> A specific question related to a functionality is the topic for a task, I >> don't think that we should discuss every detail that differs from the ORES >> API (Wikitech-l doesn't seem a good medium for it). We are already >> following up on Phabricator, let's use tasks if possible to keep the >> conversation as light and targeted as possible. >> >> Is there a good reference for which features have been excluded from >>> ores-legacy? It looks like https://wikitech.wikimedia.org/wiki/ORES covers >>> some of the excluded features/models, but not all of them. >>> >> >> We spent the last months helping the community to migrate away from the >> ORES API (to use Lift Wing instead), the remaining traffic is only related >> to few low traffic IPs that we are not able to contact. We didn't add >> feature injection or threshold optimization to ores-legacy, for example, >> since there was no indication on our logs that users were relying on it. We >> have always stated everywhere (including all emails sent in this mailing >> list) that we are 100% open to add a functionality if it is backed up by a >> valid use case. >> >> >>> I see now that it looks like the RevertRisk model will be replacing the >>> *damaging >>> *and *goodfaith *models that differentiate intentional damage from >>> unintentional damage. There's a large body of research on why this is >>> valuable and important to the social functioning of the wikis. This >>> literature also discusses why being reverted is not a very good signal for >>> damage/vandalism and can lead to problems when used as a signal for >>> patrolling. Was there a community discussion about this deprecation that I >>> missed? I have some preliminary results (in press) that demonstrate that >>> the RevertRisk model performs significantly worse than the damaging and >>> goodfaith models in English Wikipedia for patrolling work. Do you have >>> documentation for how you evaluated this model and compared it to >>> damaging/goodfaith? >>> >> >> We have model cards related to both Revert Risk models, all of them >> linked in the API portal docs (more info: >> https://api.wikimedia.org/wiki/Lift_Wing_API). All the community folks >> that migrated their bots/tools/etc.. to Revert Risk were very happy about >> the change, and we haven't had any
[Wikitech-l] Re: ORES To Lift Wing Migration
All fine points. As you can see, I've filed some phab tasks where I saw a clear opportunity to do so. > as mentioned before all the models that currently run on ORES are available in both ores-legacy and Lift Wing. I thought I read that damaging and goodfaith models are going to be replaced. Should I instead read that they are likely to remain available for the foreseeable future? When I asked about a community discussion about the transition from damaging/goodfaith to revertrisk, I was imagining that many people who use those predictions might have an opinion about them going away. E.g. people who use the relevant filters in RecentChanges. Maybe I missed the discussions about that. I haven't seen a mention of the article quality or article topic models in the docs. Are those also going to remain available? I have some user scripts that use these models and are relatively widely used. I didn't notice anyone reaching out. ... So I checked and setting a User-Agent on my user scripts doesn't actually change the User-Agent. I've read that you need to set "Api-User-Agent" instead, but that causes a CORS error when querying ORES. I'll file a bug. On Fri, Sep 22, 2023 at 1:22 PM Luca Toscano wrote: > > > On Fri, Sep 22, 2023 at 8:59 PM Aaron Halfaker > wrote: > >> We could definitely file a task. However, it does seem like highlighting >> the features that will no longer be available is an appropriate topic for a >> discussion about migration in a technical mailing list. >> > > A specific question related to a functionality is the topic for a task, I > don't think that we should discuss every detail that differs from the ORES > API (Wikitech-l doesn't seem a good medium for it). We are already > following up on Phabricator, let's use tasks if possible to keep the > conversation as light and targeted as possible. > > Is there a good reference for which features have been excluded from >> ores-legacy? It looks like https://wikitech.wikimedia.org/wiki/ORES covers >> some of the excluded features/models, but not all of them. >> > > We spent the last months helping the community to migrate away from the > ORES API (to use Lift Wing instead), the remaining traffic is only related > to few low traffic IPs that we are not able to contact. We didn't add > feature injection or threshold optimization to ores-legacy, for example, > since there was no indication on our logs that users were relying on it. We > have always stated everywhere (including all emails sent in this mailing > list) that we are 100% open to add a functionality if it is backed up by a > valid use case. > > >> I see now that it looks like the RevertRisk model will be replacing the >> *damaging >> *and *goodfaith *models that differentiate intentional damage from >> unintentional damage. There's a large body of research on why this is >> valuable and important to the social functioning of the wikis. This >> literature also discusses why being reverted is not a very good signal for >> damage/vandalism and can lead to problems when used as a signal for >> patrolling. Was there a community discussion about this deprecation that I >> missed? I have some preliminary results (in press) that demonstrate that >> the RevertRisk model performs significantly worse than the damaging and >> goodfaith models in English Wikipedia for patrolling work. Do you have >> documentation for how you evaluated this model and compared it to >> damaging/goodfaith? >> > > We have model cards related to both Revert Risk models, all of them linked > in the API portal docs (more info: > https://api.wikimedia.org/wiki/Lift_Wing_API). All the community folks > that migrated their bots/tools/etc.. to Revert Risk were very happy about > the change, and we haven't had any request to switch back since then. > > The ML team provides all the models deployed on ORES on Lift Wing, so any > damaging and goodfaith variant is available in the new API. We chose to not > pursue the development of those models for several reasons: > - We haven't had any indication/request from the community about those > models in almost two years, except few Phabricator updates that we followed > up on. > - Managing several hundreds models for goodfaith and damaging is not very > scalable in a modern micro-service architecture like Lift Wing (since we > have a model for each supported wiki). We (both Research and ML) are > oriented on having fewer models that manage more languages at the same > time, and this is the direction that we are following at the moment. It may > not be the perfect one but so far it seems a good choice. If you want to > chime in and provide your inputs we are 100% available in hearing > suggestions/concerns/doubts/recommendations/etc.., please follow up in any > of our channels (IRC, mailing lists, Phabricator for example). > - Last but not the least, most of the damaging/goodfaith models have been > trained with data coming from years ago, and never re-trained. The
[Wikitech-l] Re: ORES To Lift Wing Migration
On Fri, Sep 22, 2023 at 8:59 PM Aaron Halfaker wrote: > We could definitely file a task. However, it does seem like highlighting > the features that will no longer be available is an appropriate topic for a > discussion about migration in a technical mailing list. > A specific question related to a functionality is the topic for a task, I don't think that we should discuss every detail that differs from the ORES API (Wikitech-l doesn't seem a good medium for it). We are already following up on Phabricator, let's use tasks if possible to keep the conversation as light and targeted as possible. Is there a good reference for which features have been excluded from > ores-legacy? It looks like https://wikitech.wikimedia.org/wiki/ORES covers > some of the excluded features/models, but not all of them. > We spent the last months helping the community to migrate away from the ORES API (to use Lift Wing instead), the remaining traffic is only related to few low traffic IPs that we are not able to contact. We didn't add feature injection or threshold optimization to ores-legacy, for example, since there was no indication on our logs that users were relying on it. We have always stated everywhere (including all emails sent in this mailing list) that we are 100% open to add a functionality if it is backed up by a valid use case. > I see now that it looks like the RevertRisk model will be replacing the > *damaging > *and *goodfaith *models that differentiate intentional damage from > unintentional damage. There's a large body of research on why this is > valuable and important to the social functioning of the wikis. This > literature also discusses why being reverted is not a very good signal for > damage/vandalism and can lead to problems when used as a signal for > patrolling. Was there a community discussion about this deprecation that I > missed? I have some preliminary results (in press) that demonstrate that > the RevertRisk model performs significantly worse than the damaging and > goodfaith models in English Wikipedia for patrolling work. Do you have > documentation for how you evaluated this model and compared it to > damaging/goodfaith? > We have model cards related to both Revert Risk models, all of them linked in the API portal docs (more info: https://api.wikimedia.org/wiki/Lift_Wing_API). All the community folks that migrated their bots/tools/etc.. to Revert Risk were very happy about the change, and we haven't had any request to switch back since then. The ML team provides all the models deployed on ORES on Lift Wing, so any damaging and goodfaith variant is available in the new API. We chose to not pursue the development of those models for several reasons: - We haven't had any indication/request from the community about those models in almost two years, except few Phabricator updates that we followed up on. - Managing several hundreds models for goodfaith and damaging is not very scalable in a modern micro-service architecture like Lift Wing (since we have a model for each supported wiki). We (both Research and ML) are oriented on having fewer models that manage more languages at the same time, and this is the direction that we are following at the moment. It may not be the perfect one but so far it seems a good choice. If you want to chime in and provide your inputs we are 100% available in hearing suggestions/concerns/doubts/recommendations/etc.., please follow up in any of our channels (IRC, mailing lists, Phabricator for example). - Last but not the least, most of the damaging/goodfaith models have been trained with data coming from years ago, and never re-trained. The efforts to keep several hundreds models up-to-date with recent data versus doing the same of few models (like revert risk) weights in favor of the latter for a relatively small team of engineers like us. > FWIW, from my reading of these announcement threads, I believed that > generally functionality and models would be preserved in > ores-legacy/LiftWing. This is the first time I've realized the scale of > what will become unavailable. > This is the part that I don't get, since as mentioned before all the models that currently run on ORES are available in both ores-legacy and Lift Wing. What changes is that we don't expose anymore functionality that logs clearly show are not used, and that would need to be maintained and improved over time. We are open to improve and add any requirement that the community needs, the only thing that we ask is to provide a valid use case to support it. I do think that Lift Wing is a great improvement for the community, we have been working with all the folks that reached out to us, without hiding anything (including deprecation plans and path forwards). Thanks for following up! Luca ___ Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
[Wikitech-l] Re: ORES To Lift Wing Migration
We could definitely file a task. However, it does seem like highlighting
the features that will no longer be available is an appropriate topic for a
discussion about migration in a technical mailing list.
Is there a good reference for which features have been excluded from
ores-legacy? It looks like https://wikitech.wikimedia.org/wiki/ORES covers
some of the excluded features/models, but not all of them.
I see now that it looks like the RevertRisk model will be replacing
the *damaging
*and *goodfaith *models that differentiate intentional damage from
unintentional damage. There's a large body of research on why this is
valuable and important to the social functioning of the wikis. This
literature also discusses why being reverted is not a very good signal for
damage/vandalism and can lead to problems when used as a signal for
patrolling. Was there a community discussion about this deprecation that I
missed? I have some preliminary results (in press) that demonstrate that
the RevertRisk model performs significantly worse than the damaging and
goodfaith models in English Wikipedia for patrolling work. Do you have
documentation for how you evaluated this model and compared it to
damaging/goodfaith?
FWIW, from my reading of these announcement threads, I believed that
generally functionality and models would be preserved in
ores-legacy/LiftWing. This is the first time I've realized the scale of
what will become unavailable.
On Fri, Sep 22, 2023 at 9:07 AM Luca Toscano wrote:
> Let's discuss the issue in a Phabricator task, it seems more appropriate
> than here (so other folks can chime in etc.. more easily).
>
> From our traffic analysis there is no current client using model_info, so
> we didn't add it to the feature set. We are working on an equivalent
> solution in Lift Wing for all hosted models, not only revscoring ones, but
> we don't have anything available now (a sort of "explainer" for the model's
> metadata basically).
>
> Luca
>
> On Fri, Sep 22, 2023 at 6:01 PM Aaron Halfaker
> wrote:
>
>> It looks like model_info is not implemented at all. E.g.
>> https://ores-legacy.wikimedia.org/v3/scores/enwiki?model_info=statistics.thresholds.true.%22maximum+recall+@+precision+%3E=+0.9%22=damaging
>>
>> I get {"detail":{"error":{"code":"bad request","message":"model_info
>> query parameter is not supported by this endpoint anymore. For more
>> information please visit https://wikitech.wikimedia.org/wiki/ORES"}}}
>>
>> But when I go to that page, nothing discusses model_info. Is there a way
>> to get this from LiftWing?
>>
>> On Fri, Sep 22, 2023 at 8:53 AM Aaron Halfaker
>> wrote:
>>
>>> Do you have a tag for filing bugs against ORES-legacy? I can't seem to
>>> find a relevant one in phab.
>>>
>>> On Fri, Sep 22, 2023 at 8:39 AM Luca Toscano
>>> wrote:
>>>
Hi Aaron!
Thanks for following up. The API is almost compatible with what ORES
currently does, but there are limitations (like the max number of revisions
in a batch etc..). The API clearly states when something is not supported,
so you can check its compatibility now making some requests to:
https://ores-legacy.wikimedia.org
If you open a task with a list of systems that you need to migrate we
can definitely take a look and help. So far the traffic being served by
ORES has been reduced to few clients, and all of them don't run with
recognizable UAs (see https://meta.wikimedia.org/wiki/User-Agent_policy)
so we'll try our best to support them. The migration to Lift Wing has been
widely publicized, a lot of documentation is available to migrate. We'd
suggest trying Lift Wing for your systems instead (see
https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Usage).
The Machine Learning plan is to eventually deprecate ores-legacy too,
to maintain only one system (namely Lift Wing). There is no final date yet,
we'll try to reach out to all remaining users first, so if you plan to keep
using ores-legacy please follow up with us first :)
Thanks!
Luca (on behalf of the ML Team)
On Fri, Sep 22, 2023 at 5:10 PM Aaron Halfaker <
aaron.halfa...@gmail.com> wrote:
> Does the new ores-legacy support the same feature set. E.g. features
> output, injection, and threshold optimizations. Or is it just prediction?
> This will affect some of the systems I need to migrate.
>
> On Fri, Sep 22, 2023, 06:21 Ilias Sarantopoulos <
> isarantopou...@wikimedia.org> wrote:
>
>> Hello!
>>
>>
>> As a next step in the deprecation process of ORES
>> https://wikitech.wikimedia.org/wiki/ORES the Machine Learning team will
>> switch the backend of ores.wikimedia.org to ores-legacy, a k8s
>> application meant to provide a compatibility layer between ORES and Lift
>> Wing so users that have not yet migrated to Lift Wing will be
>> transparently migrated.
[Wikitech-l] Re: ORES To Lift Wing Migration
Let's discuss the issue in a Phabricator task, it seems more appropriate
than here (so other folks can chime in etc.. more easily).
>From our traffic analysis there is no current client using model_info, so
we didn't add it to the feature set. We are working on an equivalent
solution in Lift Wing for all hosted models, not only revscoring ones, but
we don't have anything available now (a sort of "explainer" for the model's
metadata basically).
Luca
On Fri, Sep 22, 2023 at 6:01 PM Aaron Halfaker
wrote:
> It looks like model_info is not implemented at all. E.g.
> https://ores-legacy.wikimedia.org/v3/scores/enwiki?model_info=statistics.thresholds.true.%22maximum+recall+@+precision+%3E=+0.9%22=damaging
>
> I get {"detail":{"error":{"code":"bad request","message":"model_info
> query parameter is not supported by this endpoint anymore. For more
> information please visit https://wikitech.wikimedia.org/wiki/ORES"}}}
>
> But when I go to that page, nothing discusses model_info. Is there a way
> to get this from LiftWing?
>
> On Fri, Sep 22, 2023 at 8:53 AM Aaron Halfaker
> wrote:
>
>> Do you have a tag for filing bugs against ORES-legacy? I can't seem to
>> find a relevant one in phab.
>>
>> On Fri, Sep 22, 2023 at 8:39 AM Luca Toscano
>> wrote:
>>
>>> Hi Aaron!
>>>
>>> Thanks for following up. The API is almost compatible with what ORES
>>> currently does, but there are limitations (like the max number of revisions
>>> in a batch etc..). The API clearly states when something is not supported,
>>> so you can check its compatibility now making some requests to:
>>>
>>> https://ores-legacy.wikimedia.org
>>>
>>> If you open a task with a list of systems that you need to migrate we
>>> can definitely take a look and help. So far the traffic being served by
>>> ORES has been reduced to few clients, and all of them don't run with
>>> recognizable UAs (see https://meta.wikimedia.org/wiki/User-Agent_policy)
>>> so we'll try our best to support them. The migration to Lift Wing has been
>>> widely publicized, a lot of documentation is available to migrate. We'd
>>> suggest trying Lift Wing for your systems instead (see
>>> https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Usage).
>>>
>>> The Machine Learning plan is to eventually deprecate ores-legacy too, to
>>> maintain only one system (namely Lift Wing). There is no final date yet,
>>> we'll try to reach out to all remaining users first, so if you plan to keep
>>> using ores-legacy please follow up with us first :)
>>>
>>> Thanks!
>>>
>>> Luca (on behalf of the ML Team)
>>>
>>> On Fri, Sep 22, 2023 at 5:10 PM Aaron Halfaker
>>> wrote:
>>>
Does the new ores-legacy support the same feature set. E.g. features
output, injection, and threshold optimizations. Or is it just prediction?
This will affect some of the systems I need to migrate.
On Fri, Sep 22, 2023, 06:21 Ilias Sarantopoulos <
isarantopou...@wikimedia.org> wrote:
> Hello!
>
>
> As a next step in the deprecation process of ORES
> https://wikitech.wikimedia.org/wiki/ORES the Machine Learning team will
> switch the backend of ores.wikimedia.org to ores-legacy, a k8s
> application meant to provide a compatibility layer between ORES and Lift
> Wing so users that have not yet migrated to Lift Wing will be
> transparently migrated. Ores-legacy is an application that has the same
> API
> as ORES but in the background makes requests to Lift Wing, allowing us to
> decommission the ORES servers until all clients have moved.
>
> This change is planned to take place on Monday 25th of September. If
> you have a client/application that is still using ORES we expect that this
> switch is going to be transparent for you.
>
> However keep in mind that ores-legacy is not a 100% replacement for
> ORES as some old and unused features are no longer supported.
>
> If you see anything out of the ordinary, feel free to contact the
> Machine Learning team:
>
> IRC libera: #wikimedia-ml
>
> Phabricator: Machine-Learning-team tag
>
> Thank you!
>
>
> On Wed, Aug 9, 2023 at 1:22 PM Chaloemphon Praphuchakang <
> yoshrakpra...@gmail.com> wrote:
>
>>
>> On Tue, 8 Aug 2023, 10:45 Tilman Bayer, wrote:
>>
>>>
>>> Hi Chris,
>>>
>>> On Mon, Aug 7, 2023 at 11:51 AM Chris Albon
>>> wrote:
>>>
Hi Tilman,
Most of the work is still very experimental. We have hosted a few
LLMs on Lift Wing already (StarCoder for example) but they were just
running on CPU, far too slow for real use cases. But it proves that we
can
easily host LLMs on Lift Wing. We have been pretty quiet about it
while we
focus on the ORES migration, but it is our next big project. More soon
hopefully!
>>> Understood. Looking forward to learning more later!
[Wikitech-l] Re: ORES To Lift Wing Migration
Hi! The tag 'Machine-Learning-Team' is the one that we pay more attention to :) Luca On Fri, Sep 22, 2023 at 5:54 PM Aaron Halfaker wrote: > Do you have a tag for filing bugs against ORES-legacy? I can't seem to > find a relevant one in phab. > > On Fri, Sep 22, 2023 at 8:39 AM Luca Toscano > wrote: > >> Hi Aaron! >> >> Thanks for following up. The API is almost compatible with what ORES >> currently does, but there are limitations (like the max number of revisions >> in a batch etc..). The API clearly states when something is not supported, >> so you can check its compatibility now making some requests to: >> >> https://ores-legacy.wikimedia.org >> >> If you open a task with a list of systems that you need to migrate we >> can definitely take a look and help. So far the traffic being served by >> ORES has been reduced to few clients, and all of them don't run with >> recognizable UAs (see https://meta.wikimedia.org/wiki/User-Agent_policy) >> so we'll try our best to support them. The migration to Lift Wing has been >> widely publicized, a lot of documentation is available to migrate. We'd >> suggest trying Lift Wing for your systems instead (see >> https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Usage). >> >> The Machine Learning plan is to eventually deprecate ores-legacy too, to >> maintain only one system (namely Lift Wing). There is no final date yet, >> we'll try to reach out to all remaining users first, so if you plan to keep >> using ores-legacy please follow up with us first :) >> >> Thanks! >> >> Luca (on behalf of the ML Team) >> >> On Fri, Sep 22, 2023 at 5:10 PM Aaron Halfaker >> wrote: >> >>> Does the new ores-legacy support the same feature set. E.g. features >>> output, injection, and threshold optimizations. Or is it just prediction? >>> This will affect some of the systems I need to migrate. >>> >>> On Fri, Sep 22, 2023, 06:21 Ilias Sarantopoulos < >>> isarantopou...@wikimedia.org> wrote: >>> Hello! As a next step in the deprecation process of ORES https://wikitech.wikimedia.org/wiki/ORES the Machine Learning team will switch the backend of ores.wikimedia.org to ores-legacy, a k8s application meant to provide a compatibility layer between ORES and Lift Wing so users that have not yet migrated to Lift Wing will be transparently migrated. Ores-legacy is an application that has the same API as ORES but in the background makes requests to Lift Wing, allowing us to decommission the ORES servers until all clients have moved. This change is planned to take place on Monday 25th of September. If you have a client/application that is still using ORES we expect that this switch is going to be transparent for you. However keep in mind that ores-legacy is not a 100% replacement for ORES as some old and unused features are no longer supported. If you see anything out of the ordinary, feel free to contact the Machine Learning team: IRC libera: #wikimedia-ml Phabricator: Machine-Learning-team tag Thank you! On Wed, Aug 9, 2023 at 1:22 PM Chaloemphon Praphuchakang < yoshrakpra...@gmail.com> wrote: > > On Tue, 8 Aug 2023, 10:45 Tilman Bayer, wrote: > >> >> Hi Chris, >> >> On Mon, Aug 7, 2023 at 11:51 AM Chris Albon >> wrote: >> >>> Hi Tilman, >>> >>> Most of the work is still very experimental. We have hosted a few >>> LLMs on Lift Wing already (StarCoder for example) but they were just >>> running on CPU, far too slow for real use cases. But it proves that we >>> can >>> easily host LLMs on Lift Wing. We have been pretty quiet about it while >>> we >>> focus on the ORES migration, but it is our next big project. More soon >>> hopefully! >>> >> Understood. Looking forward to learning more later! >> >> >>> Where we are now is that we have budget for a big GPU purchase >>> (~10-20 GPUs depending on cost), the question we will try to answer >>> after >>> the ORES migration is complete is: what GPUs should we purchase? We are >>> trying to balance our strong preference to stay open source (i.e. AMD >>> mROC) >>> in a world dominated by a single closed source vendor (i.e. Nvidia). In >>> addition, do we go for a few expensive GPUs better suited to LLMs >>> (A1000, >>> H100, etc) or a mix of big and small? We will need to figure out all >>> this. >>> >> I see. On that matter, what do you folks make of the recent >> announcements of AMD's partnerships with Hugging Face and Pytorch[5]? >> (which, I understand, came after the ML team had already launched the >> aforementioned new AMD explorations) >> >> "Open-source AI: AMD looks to Hugging Face and Meta spinoff PyTorch >> to take on Nvidia [...] >> Both partnerships involve AMD’s ROCm AI software stack,
[Wikitech-l] Re: ORES To Lift Wing Migration
It looks like model_info is not implemented at all. E.g.
https://ores-legacy.wikimedia.org/v3/scores/enwiki?model_info=statistics.thresholds.true.%22maximum+recall+@+precision+%3E=+0.9%22=damaging
I get {"detail":{"error":{"code":"bad request","message":"model_info query
parameter is not supported by this endpoint anymore. For more information
please visit https://wikitech.wikimedia.org/wiki/ORES"}}}
But when I go to that page, nothing discusses model_info. Is there a way
to get this from LiftWing?
On Fri, Sep 22, 2023 at 8:53 AM Aaron Halfaker
wrote:
> Do you have a tag for filing bugs against ORES-legacy? I can't seem to
> find a relevant one in phab.
>
> On Fri, Sep 22, 2023 at 8:39 AM Luca Toscano
> wrote:
>
>> Hi Aaron!
>>
>> Thanks for following up. The API is almost compatible with what ORES
>> currently does, but there are limitations (like the max number of revisions
>> in a batch etc..). The API clearly states when something is not supported,
>> so you can check its compatibility now making some requests to:
>>
>> https://ores-legacy.wikimedia.org
>>
>> If you open a task with a list of systems that you need to migrate we
>> can definitely take a look and help. So far the traffic being served by
>> ORES has been reduced to few clients, and all of them don't run with
>> recognizable UAs (see https://meta.wikimedia.org/wiki/User-Agent_policy)
>> so we'll try our best to support them. The migration to Lift Wing has been
>> widely publicized, a lot of documentation is available to migrate. We'd
>> suggest trying Lift Wing for your systems instead (see
>> https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Usage).
>>
>> The Machine Learning plan is to eventually deprecate ores-legacy too, to
>> maintain only one system (namely Lift Wing). There is no final date yet,
>> we'll try to reach out to all remaining users first, so if you plan to keep
>> using ores-legacy please follow up with us first :)
>>
>> Thanks!
>>
>> Luca (on behalf of the ML Team)
>>
>> On Fri, Sep 22, 2023 at 5:10 PM Aaron Halfaker
>> wrote:
>>
>>> Does the new ores-legacy support the same feature set. E.g. features
>>> output, injection, and threshold optimizations. Or is it just prediction?
>>> This will affect some of the systems I need to migrate.
>>>
>>> On Fri, Sep 22, 2023, 06:21 Ilias Sarantopoulos <
>>> isarantopou...@wikimedia.org> wrote:
>>>
Hello!
As a next step in the deprecation process of ORES
https://wikitech.wikimedia.org/wiki/ORES the Machine Learning team will
switch the backend of ores.wikimedia.org to ores-legacy, a k8s
application meant to provide a compatibility layer between ORES and Lift
Wing so users that have not yet migrated to Lift Wing will be
transparently migrated. Ores-legacy is an application that has the same API
as ORES but in the background makes requests to Lift Wing, allowing us to
decommission the ORES servers until all clients have moved.
This change is planned to take place on Monday 25th of September. If
you have a client/application that is still using ORES we expect that this
switch is going to be transparent for you.
However keep in mind that ores-legacy is not a 100% replacement for
ORES as some old and unused features are no longer supported.
If you see anything out of the ordinary, feel free to contact the
Machine Learning team:
IRC libera: #wikimedia-ml
Phabricator: Machine-Learning-team tag
Thank you!
On Wed, Aug 9, 2023 at 1:22 PM Chaloemphon Praphuchakang <
yoshrakpra...@gmail.com> wrote:
>
> On Tue, 8 Aug 2023, 10:45 Tilman Bayer, wrote:
>
>>
>> Hi Chris,
>>
>> On Mon, Aug 7, 2023 at 11:51 AM Chris Albon
>> wrote:
>>
>>> Hi Tilman,
>>>
>>> Most of the work is still very experimental. We have hosted a few
>>> LLMs on Lift Wing already (StarCoder for example) but they were just
>>> running on CPU, far too slow for real use cases. But it proves that we
>>> can
>>> easily host LLMs on Lift Wing. We have been pretty quiet about it while
>>> we
>>> focus on the ORES migration, but it is our next big project. More soon
>>> hopefully!
>>>
>> Understood. Looking forward to learning more later!
>>
>>
>>> Where we are now is that we have budget for a big GPU purchase
>>> (~10-20 GPUs depending on cost), the question we will try to answer
>>> after
>>> the ORES migration is complete is: what GPUs should we purchase? We are
>>> trying to balance our strong preference to stay open source (i.e. AMD
>>> mROC)
>>> in a world dominated by a single closed source vendor (i.e. Nvidia). In
>>> addition, do we go for a few expensive GPUs better suited to LLMs
>>> (A1000,
>>> H100, etc) or a mix of big and small? We will need to figure out all
>>> this.
>>>
>> I see. On
[Wikitech-l] Re: ORES To Lift Wing Migration
Do you have a tag for filing bugs against ORES-legacy? I can't seem to find a relevant one in phab. On Fri, Sep 22, 2023 at 8:39 AM Luca Toscano wrote: > Hi Aaron! > > Thanks for following up. The API is almost compatible with what ORES > currently does, but there are limitations (like the max number of revisions > in a batch etc..). The API clearly states when something is not supported, > so you can check its compatibility now making some requests to: > > https://ores-legacy.wikimedia.org > > If you open a task with a list of systems that you need to migrate we can > definitely take a look and help. So far the traffic being served by ORES > has been reduced to few clients, and all of them don't run with > recognizable UAs (see https://meta.wikimedia.org/wiki/User-Agent_policy) > so we'll try our best to support them. The migration to Lift Wing has been > widely publicized, a lot of documentation is available to migrate. We'd > suggest trying Lift Wing for your systems instead (see > https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Usage). > > The Machine Learning plan is to eventually deprecate ores-legacy too, to > maintain only one system (namely Lift Wing). There is no final date yet, > we'll try to reach out to all remaining users first, so if you plan to keep > using ores-legacy please follow up with us first :) > > Thanks! > > Luca (on behalf of the ML Team) > > On Fri, Sep 22, 2023 at 5:10 PM Aaron Halfaker > wrote: > >> Does the new ores-legacy support the same feature set. E.g. features >> output, injection, and threshold optimizations. Or is it just prediction? >> This will affect some of the systems I need to migrate. >> >> On Fri, Sep 22, 2023, 06:21 Ilias Sarantopoulos < >> isarantopou...@wikimedia.org> wrote: >> >>> Hello! >>> >>> >>> As a next step in the deprecation process of ORES >>> https://wikitech.wikimedia.org/wiki/ORES the Machine Learning team will >>> switch the backend of ores.wikimedia.org to ores-legacy, a k8s >>> application meant to provide a compatibility layer between ORES and Lift >>> Wing so users that have not yet migrated to Lift Wing will be >>> transparently migrated. Ores-legacy is an application that has the same API >>> as ORES but in the background makes requests to Lift Wing, allowing us to >>> decommission the ORES servers until all clients have moved. >>> >>> This change is planned to take place on Monday 25th of September. If >>> you have a client/application that is still using ORES we expect that this >>> switch is going to be transparent for you. >>> >>> However keep in mind that ores-legacy is not a 100% replacement for ORES >>> as some old and unused features are no longer supported. >>> >>> If you see anything out of the ordinary, feel free to contact the >>> Machine Learning team: >>> >>> IRC libera: #wikimedia-ml >>> >>> Phabricator: Machine-Learning-team tag >>> >>> Thank you! >>> >>> >>> On Wed, Aug 9, 2023 at 1:22 PM Chaloemphon Praphuchakang < >>> yoshrakpra...@gmail.com> wrote: >>> On Tue, 8 Aug 2023, 10:45 Tilman Bayer, wrote: > > Hi Chris, > > On Mon, Aug 7, 2023 at 11:51 AM Chris Albon > wrote: > >> Hi Tilman, >> >> Most of the work is still very experimental. We have hosted a few >> LLMs on Lift Wing already (StarCoder for example) but they were just >> running on CPU, far too slow for real use cases. But it proves that we >> can >> easily host LLMs on Lift Wing. We have been pretty quiet about it while >> we >> focus on the ORES migration, but it is our next big project. More soon >> hopefully! >> > Understood. Looking forward to learning more later! > > >> Where we are now is that we have budget for a big GPU purchase >> (~10-20 GPUs depending on cost), the question we will try to answer after >> the ORES migration is complete is: what GPUs should we purchase? We are >> trying to balance our strong preference to stay open source (i.e. AMD >> mROC) >> in a world dominated by a single closed source vendor (i.e. Nvidia). In >> addition, do we go for a few expensive GPUs better suited to LLMs (A1000, >> H100, etc) or a mix of big and small? We will need to figure out all >> this. >> > I see. On that matter, what do you folks make of the recent > announcements of AMD's partnerships with Hugging Face and Pytorch[5]? > (which, I understand, came after the ML team had already launched the > aforementioned new AMD explorations) > > "Open-source AI: AMD looks to Hugging Face and Meta spinoff PyTorch to > take on Nvidia [...] > Both partnerships involve AMD’s ROCm AI software stack, the company’s > answer to Nvidia’s proprietary CUDA platform and application-programming > interface. AMD called ROCm an open and portable AI system with > out-of-the-box support that can port to existing AI models. [...B]oth AMD > and Hugging Face are dedicating
[Wikitech-l] Re: ORES To Lift Wing Migration
Hi Aaron! Thanks for following up. The API is almost compatible with what ORES currently does, but there are limitations (like the max number of revisions in a batch etc..). The API clearly states when something is not supported, so you can check its compatibility now making some requests to: https://ores-legacy.wikimedia.org If you open a task with a list of systems that you need to migrate we can definitely take a look and help. So far the traffic being served by ORES has been reduced to few clients, and all of them don't run with recognizable UAs (see https://meta.wikimedia.org/wiki/User-Agent_policy) so we'll try our best to support them. The migration to Lift Wing has been widely publicized, a lot of documentation is available to migrate. We'd suggest trying Lift Wing for your systems instead (see https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Usage). The Machine Learning plan is to eventually deprecate ores-legacy too, to maintain only one system (namely Lift Wing). There is no final date yet, we'll try to reach out to all remaining users first, so if you plan to keep using ores-legacy please follow up with us first :) Thanks! Luca (on behalf of the ML Team) On Fri, Sep 22, 2023 at 5:10 PM Aaron Halfaker wrote: > Does the new ores-legacy support the same feature set. E.g. features > output, injection, and threshold optimizations. Or is it just prediction? > This will affect some of the systems I need to migrate. > > On Fri, Sep 22, 2023, 06:21 Ilias Sarantopoulos < > isarantopou...@wikimedia.org> wrote: > >> Hello! >> >> >> As a next step in the deprecation process of ORES >> https://wikitech.wikimedia.org/wiki/ORES the Machine Learning team will >> switch the backend of ores.wikimedia.org to ores-legacy, a k8s >> application meant to provide a compatibility layer between ORES and Lift >> Wing so users that have not yet migrated to Lift Wing will be >> transparently migrated. Ores-legacy is an application that has the same API >> as ORES but in the background makes requests to Lift Wing, allowing us to >> decommission the ORES servers until all clients have moved. >> >> This change is planned to take place on Monday 25th of September. If you >> have a client/application that is still using ORES we expect that this >> switch is going to be transparent for you. >> >> However keep in mind that ores-legacy is not a 100% replacement for ORES >> as some old and unused features are no longer supported. >> >> If you see anything out of the ordinary, feel free to contact the Machine >> Learning team: >> >> IRC libera: #wikimedia-ml >> >> Phabricator: Machine-Learning-team tag >> >> Thank you! >> >> >> On Wed, Aug 9, 2023 at 1:22 PM Chaloemphon Praphuchakang < >> yoshrakpra...@gmail.com> wrote: >> >>> >>> On Tue, 8 Aug 2023, 10:45 Tilman Bayer, wrote: >>> Hi Chris, On Mon, Aug 7, 2023 at 11:51 AM Chris Albon wrote: > Hi Tilman, > > Most of the work is still very experimental. We have hosted a few LLMs > on Lift Wing already (StarCoder for example) but they were just running on > CPU, far too slow for real use cases. But it proves that we can easily > host > LLMs on Lift Wing. We have been pretty quiet about it while we focus on > the > ORES migration, but it is our next big project. More soon hopefully! > Understood. Looking forward to learning more later! > Where we are now is that we have budget for a big GPU purchase (~10-20 > GPUs depending on cost), the question we will try to answer after the ORES > migration is complete is: what GPUs should we purchase? We are trying to > balance our strong preference to stay open source (i.e. AMD mROC) in a > world dominated by a single closed source vendor (i.e. Nvidia). In > addition, do we go for a few expensive GPUs better suited to LLMs (A1000, > H100, etc) or a mix of big and small? We will need to figure out all this. > I see. On that matter, what do you folks make of the recent announcements of AMD's partnerships with Hugging Face and Pytorch[5]? (which, I understand, came after the ML team had already launched the aforementioned new AMD explorations) "Open-source AI: AMD looks to Hugging Face and Meta spinoff PyTorch to take on Nvidia [...] Both partnerships involve AMD’s ROCm AI software stack, the company’s answer to Nvidia’s proprietary CUDA platform and application-programming interface. AMD called ROCm an open and portable AI system with out-of-the-box support that can port to existing AI models. [...B]oth AMD and Hugging Face are dedicating engineering resources to each other and sharing data to ensure that the constantly updated AI models from Hugging Face, which might not otherwise run well on AMD hardware, would be “guaranteed” to work on hardware like the MI300X. [...] AMD said PyTorch will fully upstream the ROCm software
[Wikitech-l] Re: ORES To Lift Wing Migration
Does the new ores-legacy support the same feature set. E.g. features output, injection, and threshold optimizations. Or is it just prediction? This will affect some of the systems I need to migrate. On Fri, Sep 22, 2023, 06:21 Ilias Sarantopoulos < isarantopou...@wikimedia.org> wrote: > Hello! > > > As a next step in the deprecation process of ORES > https://wikitech.wikimedia.org/wiki/ORES the Machine Learning team will > switch the backend of ores.wikimedia.org to ores-legacy, a k8s > application meant to provide a compatibility layer between ORES and Lift > Wing so users that have not yet migrated to Lift Wing will be > transparently migrated. Ores-legacy is an application that has the same API > as ORES but in the background makes requests to Lift Wing, allowing us to > decommission the ORES servers until all clients have moved. > > This change is planned to take place on Monday 25th of September. If you > have a client/application that is still using ORES we expect that this > switch is going to be transparent for you. > > However keep in mind that ores-legacy is not a 100% replacement for ORES > as some old and unused features are no longer supported. > > If you see anything out of the ordinary, feel free to contact the Machine > Learning team: > > IRC libera: #wikimedia-ml > > Phabricator: Machine-Learning-team tag > > Thank you! > > > On Wed, Aug 9, 2023 at 1:22 PM Chaloemphon Praphuchakang < > yoshrakpra...@gmail.com> wrote: > >> >> On Tue, 8 Aug 2023, 10:45 Tilman Bayer, wrote: >> >>> >>> Hi Chris, >>> >>> On Mon, Aug 7, 2023 at 11:51 AM Chris Albon >>> wrote: >>> Hi Tilman, Most of the work is still very experimental. We have hosted a few LLMs on Lift Wing already (StarCoder for example) but they were just running on CPU, far too slow for real use cases. But it proves that we can easily host LLMs on Lift Wing. We have been pretty quiet about it while we focus on the ORES migration, but it is our next big project. More soon hopefully! >>> Understood. Looking forward to learning more later! >>> >>> Where we are now is that we have budget for a big GPU purchase (~10-20 GPUs depending on cost), the question we will try to answer after the ORES migration is complete is: what GPUs should we purchase? We are trying to balance our strong preference to stay open source (i.e. AMD mROC) in a world dominated by a single closed source vendor (i.e. Nvidia). In addition, do we go for a few expensive GPUs better suited to LLMs (A1000, H100, etc) or a mix of big and small? We will need to figure out all this. >>> I see. On that matter, what do you folks make of the recent >>> announcements of AMD's partnerships with Hugging Face and Pytorch[5]? >>> (which, I understand, came after the ML team had already launched the >>> aforementioned new AMD explorations) >>> >>> "Open-source AI: AMD looks to Hugging Face and Meta spinoff PyTorch to >>> take on Nvidia [...] >>> Both partnerships involve AMD’s ROCm AI software stack, the company’s >>> answer to Nvidia’s proprietary CUDA platform and application-programming >>> interface. AMD called ROCm an open and portable AI system with >>> out-of-the-box support that can port to existing AI models. [...B]oth AMD >>> and Hugging Face are dedicating engineering resources to each other and >>> sharing data to ensure that the constantly updated AI models from Hugging >>> Face, which might not otherwise run well on AMD hardware, would be >>> “guaranteed” to work on hardware like the MI300X. [...] AMD said PyTorch >>> will fully upstream the ROCm software stack and “provide immediate ‘day >>> zero’ support for PyTorch 2.0 with ROCm release 5.4.2 on all AMD Instinct >>> accelerators,” which is meant to appeal to those customers looking to >>> switch from Nvidia’s software ecosystem." >>> >>> >>> In their own announcement, Hugging Face offered further details, >>> including a pretty impressive list of models to be supported:[6] >>> >>> >>> "We intend to support state-of-the-art transformer architectures for >>> natural language processing, computer vision, and speech, such as BERT, >>> DistilBERT, ROBERTA, Vision Transformer, CLIP, and Wav2Vec2. Of course, >>> generative AI models will be available too (e.g., GPT2, GPT-NeoX, T5, OPT, >>> LLaMA), including our own BLOOM and StarCoder models. Lastly, we will also >>> support more traditional computer vision models, like ResNet and ResNext, >>> and deep learning recommendation models, a first for us. [..] We'll do our >>> best to test and validate these models for PyTorch, TensorFlow, and ONNX >>> Runtime for the above platforms. [...] We will integrate the AMD ROCm SDK >>> seamlessly in our open-source libraries, starting with the transformers >>> library." >>> >>> >>> Do you think this may promise too much, or could it point to a possible >>> solution of the Foundation's conundrum? >>> In any case, this seems to be an interesting moment where many
[Wikitech-l] Re: ORES To Lift Wing Migration
Hello! As a next step in the deprecation process of ORES https://wikitech.wikimedia.org/wiki/ORES the Machine Learning team will switch the backend of ores.wikimedia.org to ores-legacy, a k8s application meant to provide a compatibility layer between ORES and Lift Wing so users that have not yet migrated to Lift Wing will be transparently migrated. Ores-legacy is an application that has the same API as ORES but in the background makes requests to Lift Wing, allowing us to decommission the ORES servers until all clients have moved. This change is planned to take place on Monday 25th of September. If you have a client/application that is still using ORES we expect that this switch is going to be transparent for you. However keep in mind that ores-legacy is not a 100% replacement for ORES as some old and unused features are no longer supported. If you see anything out of the ordinary, feel free to contact the Machine Learning team: IRC libera: #wikimedia-ml Phabricator: Machine-Learning-team tag Thank you! On Wed, Aug 9, 2023 at 1:22 PM Chaloemphon Praphuchakang < yoshrakpra...@gmail.com> wrote: > > On Tue, 8 Aug 2023, 10:45 Tilman Bayer, wrote: > >> >> Hi Chris, >> >> On Mon, Aug 7, 2023 at 11:51 AM Chris Albon wrote: >> >>> Hi Tilman, >>> >>> Most of the work is still very experimental. We have hosted a few LLMs >>> on Lift Wing already (StarCoder for example) but they were just running on >>> CPU, far too slow for real use cases. But it proves that we can easily host >>> LLMs on Lift Wing. We have been pretty quiet about it while we focus on the >>> ORES migration, but it is our next big project. More soon hopefully! >>> >> Understood. Looking forward to learning more later! >> >> >>> Where we are now is that we have budget for a big GPU purchase (~10-20 >>> GPUs depending on cost), the question we will try to answer after the ORES >>> migration is complete is: what GPUs should we purchase? We are trying to >>> balance our strong preference to stay open source (i.e. AMD mROC) in a >>> world dominated by a single closed source vendor (i.e. Nvidia). In >>> addition, do we go for a few expensive GPUs better suited to LLMs (A1000, >>> H100, etc) or a mix of big and small? We will need to figure out all this. >>> >> I see. On that matter, what do you folks make of the recent announcements >> of AMD's partnerships with Hugging Face and Pytorch[5]? (which, I >> understand, came after the ML team had already launched the aforementioned >> new AMD explorations) >> >> "Open-source AI: AMD looks to Hugging Face and Meta spinoff PyTorch to >> take on Nvidia [...] >> Both partnerships involve AMD’s ROCm AI software stack, the company’s >> answer to Nvidia’s proprietary CUDA platform and application-programming >> interface. AMD called ROCm an open and portable AI system with >> out-of-the-box support that can port to existing AI models. [...B]oth AMD >> and Hugging Face are dedicating engineering resources to each other and >> sharing data to ensure that the constantly updated AI models from Hugging >> Face, which might not otherwise run well on AMD hardware, would be >> “guaranteed” to work on hardware like the MI300X. [...] AMD said PyTorch >> will fully upstream the ROCm software stack and “provide immediate ‘day >> zero’ support for PyTorch 2.0 with ROCm release 5.4.2 on all AMD Instinct >> accelerators,” which is meant to appeal to those customers looking to >> switch from Nvidia’s software ecosystem." >> >> >> In their own announcement, Hugging Face offered further details, >> including a pretty impressive list of models to be supported:[6] >> >> >> "We intend to support state-of-the-art transformer architectures for >> natural language processing, computer vision, and speech, such as BERT, >> DistilBERT, ROBERTA, Vision Transformer, CLIP, and Wav2Vec2. Of course, >> generative AI models will be available too (e.g., GPT2, GPT-NeoX, T5, OPT, >> LLaMA), including our own BLOOM and StarCoder models. Lastly, we will also >> support more traditional computer vision models, like ResNet and ResNext, >> and deep learning recommendation models, a first for us. [..] We'll do our >> best to test and validate these models for PyTorch, TensorFlow, and ONNX >> Runtime for the above platforms. [...] We will integrate the AMD ROCm SDK >> seamlessly in our open-source libraries, starting with the transformers >> library." >> >> >> Do you think this may promise too much, or could it point to a possible >> solution of the Foundation's conundrum? >> In any case, this seems to be an interesting moment where many in AI are >> trying to move away from Nvidia's proprietary CUDA platform. Most of them >> probably more for financial and availability reasons though, given the >> current GPU shortages[7] (which the ML team is undoubtedly aware of >> already; mentioning this as context for others on this list. See also >> Marketwatch's remarks about current margins[5]). >> >> Regards, Tilman >> >> >> [5] >>
