[Wikidata-bugs] [Maniphest] [Updated] T84923: Reliable publish / subscribe event bus

2015-10-21 Thread GWicke 
GWicke added a blocked task: T102476: RFC: Requirements for change propagation.

TASK DETAIL
  https://phabricator.wikimedia.org/T84923

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: GWicke
Cc: Aklapper, Matanya, Mattflaschen, Ottomata, mmodell, Eevans, chasemp, brion, 
Krenair, Halfak, JanZerebecki, bd808, MZMcBride, mobrovac, GWicke, aaron, 
daniel, Hardikj, yuvipanda, JAllemandou, jkroll, Smalyshev, Wikidata-bugs, 
Jdouglas, aude, Deskana, Manybubbles, RobLa-WMF, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T84923: Reliable publish / subscribe event bus

2015-08-31 Thread Ottomata 
Ottomata added a project: EventBus.

TASK DETAIL
  https://phabricator.wikimedia.org/T84923

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Aklapper, Matanya, Mattflaschen, Ottomata, mmodell, Eevans, chasemp, brion, 
Krenair, Halfak, JanZerebecki, bd808, MZMcBride, mobrovac, GWicke, aaron, 
daniel, Hardikj, yuvipanda, JAllemandou, jkroll, Smalyshev, Wikidata-bugs, 
Jdouglas, RobH, aude, Deskana, Manybubbles, mark, RobLa-WMF, faidon, 
fgiunchedi, Dzahn, jeremyb, Malyacko



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T84923: Reliable publish / subscribe event bus

2015-08-28 Thread Ottomata 
Ottomata added a blocking task: T110748: Event Bus.

TASK DETAIL
  https://phabricator.wikimedia.org/T84923

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Aklapper, Matanya, Mattflaschen, Ottomata, mmodell, Eevans, chasemp, brion, 
Krenair, Halfak, JanZerebecki, bd808, MZMcBride, mobrovac, GWicke, aaron, 
daniel, Hardikj, yuvipanda, JAllemandou, jkroll, Smalyshev, Wikidata-bugs, 
Jdouglas, RobH, aude, Deskana, Manybubbles, mark, RobLa-WMF, faidon, 
fgiunchedi, Dzahn, jeremyb, Malyacko



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T84923: Reliable publish / subscribe event bus

2015-06-22 Thread Ottomata 
Ottomata added a comment.

BTW, https://phabricator.wikimedia.org/T102082 is mainly about analytics 
eventlogging, but the confluent stuff would be good for an event bus used for 
application stuff too.


TASK DETAIL
  https://phabricator.wikimedia.org/T84923

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Mattflaschen, Ottomata, mmodell, Eevans, chasemp, brion, Krenair, Halfak, 
JanZerebecki, bd808, MZMcBride, mobrovac, GWicke, aaron, daniel, Hardikj, 
yuvipanda, JAllemandou, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, RobH, aude, 
Manybubbles, mark, RobLa-WMF, faidon, fgiunchedi, Dzahn, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T84923: Reliable publish / subscribe event bus

2015-06-21 Thread Mattflaschen 
Mattflaschen added a subscriber: Mattflaschen.
Mattflaschen added a comment.
Herald added a project: Discovery.

This could also be seen as a replacement for some use cases of MW hooks and 
hook listeners.  Right now, if the hook listener throws an exception, that 
breaks the code that triggered the hook (e.g. T102874: Using Special:EnableFlow 
on a French Wikiproject page has broken the page completely 
https://phabricator.wikimedia.org/T102874).

If the hook doesn't have to alter or return any values (e.g. to tell the 
hook-running code to do something), it might be able to use an event bus 
instead.  If the event bus listener has a bug, my understanding is that 
wouldn't cause the code that emits the event to break.


TASK DETAIL
  https://phabricator.wikimedia.org/T84923

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Mattflaschen
Cc: Mattflaschen, Ottomata, mmodell, Eevans, chasemp, brion, Krenair, Halfak, 
JanZerebecki, bd808, MZMcBride, mobrovac, GWicke, aaron, daniel, Hardikj, 
yuvipanda, JAllemandou, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, RobH, aude, 
Manybubbles, mark, RobLa-WMF, faidon, fgiunchedi, Dzahn, jeremyb, Malyacko, 
P.Copp



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T84923: Reliable publish / subscribe event bus

2015-02-20 Thread bd808 
bd808 added a project: Epic.

TASK DETAIL
  https://phabricator.wikimedia.org/T84923

REPLY HANDLER ACTIONS
  Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign 
username.

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: bd808
Cc: chasemp, brion, Krenair, Halfak, JanZerebecki, bd808, MZMcBride, mobrovac, 
GWicke, aaron, daniel, Hardikj, yuvipanda, JAllemandou, jkroll, Smalyshev, 
Wikidata-bugs, Jdouglas, RobH, aude, marcoil, Manybubbles, mark, RobLa-WMF, 
faidon, fgiunchedi, Joe, Dzahn



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T84923: Reliable publish / subscribe event bus

2015-01-07 Thread GWicke 
GWicke added a project: Analytics.
GWicke added a comment.

In https://phabricator.wikimedia.org/T84923#961622, @bd808 wrote:

  can support large delays (order of days) for individual consumers


 Do you have a strong use case to support this need?


Yes. Hosts can go down for multiple days, and if the even stream is used to do 
something like reliable purges then it'll be necessary to replay those or throw 
away the entire cache.

There can also be bugs in consumers, which need to be fixed by re-starting the 
processing from a clean snapshot.

From what I hear, https://phabricator.wikimedia.org/tag/analytics/ would love 
to get even longer event traces. @halfak mentioned a back-of-the-envelope 
calculation that basically all the primary events he lists on his proposal 
https://meta.wikimedia.org/wiki/Research:MediaWiki_events:_a_generalized_public_event_datasource
 since the beginning of Wikipedia might fit into 200G.

For comparison, I think we currently have several day's worth of buffer for our 
traffic logs in kafka, which helps to avoid loss if the consumer has issues. 
That's much higher volume at up to 150k messages/s, while we are looking at low 
hundreds for edit-related events.


TASK DETAIL
  https://phabricator.wikimedia.org/T84923

REPLY HANDLER ACTIONS
  Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign 
username.

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: GWicke
Cc: GWicke, aaron, JanZerebecki, mobrovac, Halfak, yuvipanda, Hardikj, daniel, 
Krenair, bd808, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, RobH, aude, 
Manybubbles, mark, RobLa-WMF, Joe, QChris, chasemp



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T84923: Reliable publish / subscribe event bus

2014-12-24 Thread JanZerebecki 
JanZerebecki added a comment.

In https://phabricator.wikimedia.org/T84923#942248, @GWicke wrote:

 In https://phabricator.wikimedia.org/T84923#940155, @JanZerebecki wrote:

  The actual data can be pulled by the consumer from the provider;


 Is there an API for this already?


https://www.wikidata.org/w/api.php?action=helpmodules=wbgetentities and recent 
changes, see also https://phabricator.wikimedia.org/T85103 and 
https://phabricator.wikimedia.org/T85100.


TASK DETAIL
  https://phabricator.wikimedia.org/T84923

REPLY HANDLER ACTIONS
  Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign 
username.

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: JanZerebecki
Cc: GWicke, aaron, JanZerebecki, mobrovac, jkroll, Smalyshev, Wikidata-bugs, 
Jdouglas, RobH, aude, Manybubbles, daniel, mark, Joe



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T84923: Reliable publish / subscribe event bus

2014-12-23 Thread Lydia_Pintscher 
Lydia_Pintscher added a project: Wikidata.

TASK DETAIL
  https://phabricator.wikimedia.org/T84923

REPLY HANDLER ACTIONS
  Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign 
username.

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lydia_Pintscher
Cc: GWicke, aaron, JanZerebecki, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, 
RobH, aude, Manybubbles, daniel, mark, Joe



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs