from:"Ottomata"

Ottomata added a comment.

> topics named something like mw-edit and mw-edit-private perhaps (where the 
> latter contains this extra info).


I'd prefer if we did this the other way around.  The 'private' topic will have 
more data and be the main source of truth.  The public one will contain a 
subset of this data, and thus is subordinate to the main one.


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Smalyshev, yuvipanda, Hardikj, daniel, aaron, GWicke, mobrovac, MZMcBride, 
bd808, JanZerebecki, Halfak, Krenair, brion, chasemp, Eevans, mmodell, 
Ottomata, Mattflaschen, Matanya, Aklapper, JAllemandou, jkroll, Wikidata-bugs, 
Jdouglas, aude, Deskana, Manybubbles, RobLa-WMF, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

Ottomata added a comment.

I'd like an actual timestamp to be part of the framing for all events too.  I'm 
all for a reqid, (although I'd bikeshed about the name a bit), but Having a 
standardized canonical timestamp in all events is very useful.  Can we add:

- **ts**: iso 8601 timestamp.  This may be the time of the event creation, or 
it might be something else.  It can be set by the producer.

> Generally, no overly sensitive information (like client IPs for authenticated 
> edits) in primary events.

>  Can be included in expanded message in separate topic, or stored separately 
> based on reqid.


In the meeting we said that MW would generate two event streams directly, one 
that had more information, and another that had less, minus fields with privacy 
concerns.


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Smalyshev, yuvipanda, Hardikj, daniel, aaron, GWicke, mobrovac, MZMcBride, 
bd808, JanZerebecki, Halfak, Krenair, brion, chasemp, Eevans, mmodell, 
Ottomata, Mattflaschen, Matanya, Aklapper, JAllemandou, jkroll, Wikidata-bugs, 
Jdouglas, aude, Deskana, Manybubbles, RobLa-WMF, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2015-10-22 Thread Ottomata

Ottomata added a comment.

etherpad from today's meeting:

https://etherpad.wikimedia.org/p/eventbus-events


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Smalyshev, yuvipanda, Hardikj, daniel, aaron, GWicke, mobrovac, MZMcBride, 
bd808, JanZerebecki, Halfak, Krenair, brion, chasemp, Eevans, mmodell, 
Ottomata, Mattflaschen, Matanya, Aklapper, JAllemandou, jkroll, Wikidata-bugs, 
Jdouglas, aude, Deskana, Manybubbles, RobLa-WMF, 01tonythomas, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T114443: EventBus MVP

2015-10-22 Thread Ottomata

Ottomata added a blocking task: T114191: Setup a 3 server Kafka instance in 
both eqiad and codfw for reliable purge streams.

TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Smalyshev, mark, MZMcBride, Krinkle, EBernhardson, bd808, Joe, dr0ptp4kt, 
madhuvishy, Nuria, ori, faidon, aaron, GWicke, mobrovac, Eevans, Ottomata, 
Matanya, Aklapper, JAllemandou, jkroll, Hardikj, Wikidata-bugs, Jdouglas, aude, 
Deskana, Manybubbles, daniel, RobLa-WMF, 01tonythomas, Jay8g, Ltrlg, jeremyb, 
Legoktm



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2015-10-22 Thread Ottomata

Ottomata added a comment.

COOL.  As part of this discussion, I'd like us to think about not only fields 
that are relevant to edit events, but also those fields that might be useful 
for most, if not all, standardized WMF events to have.   These might be 
required for all events to share.  Things like timestamp (what format, what 
name?), hostname (from where the event originated), etc.  See 
https://meta.wikimedia.org/wiki/Schema:EventCapsule for how EventLogging does 
it.

There might be 2 levels here:  fields that all events have, and fields that 
most events coming from Mediawiki have.  Things like project, username, and I 
don't know what else.


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Smalyshev, yuvipanda, Hardikj, daniel, aaron, GWicke, mobrovac, MZMcBride, 
bd808, JanZerebecki, Halfak, Krenair, brion, chasemp, Eevans, mmodell, 
Ottomata, Mattflaschen, Matanya, Aklapper, JAllemandou, jkroll, Wikidata-bugs, 
Jdouglas, aude, Deskana, Manybubbles, RobLa-WMF, 01tonythomas, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

Ottomata added a comment.

> If we have a use case for emitting two secondary events *to the same topic* 
> that were both triggered by the same primary event (user click / request id), 
> then we can generate a new ID for at least one of those events, and record 
> the parent event id in a separate field (ex: par_id). This way, we can get 
> the right deduplication semantics for each of those events.


?  What's the point of the request_id then?  I thought we wanted X-Request-Id 
so that we can easily tie together events generated by the same http request.

Why not just have `request_id` and `uuid` as separate fields that always exist?

> IMHO, the timestamps of the event ID and explicit timestamp (ts or dt) should 
> always match. This makes it a lot simpler to automatically derive dt from id 
> in the producer REST proxy. Other event-specific times (like the save time as 
> recorded by MediaWiki) should imho go into the event body.


Why?  I agree, that specific schemas can define additional timestamps, but what 
is the harm in having a standard one that is set and used semantically by the 
producer?  What if I wanted to explicitly feed a topic with events dated in the 
past, perhaps for backfilling or recovery reasons?


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: mobrovac, Ottomata
Cc: EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron, GWicke, 
mobrovac, MZMcBride, bd808, JanZerebecki, Halfak, Krenair, brion, chasemp, 
Eevans, mmodell, Ottomata, Mattflaschen, Matanya, Aklapper, JAllemandou, 
jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, RobLa-WMF, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

Ottomata added a comment.

What do y'all think about keeping these 'framing' fields in a nested object?  
I'm not sure if this is a good or bad idea.  If later we decide we do want to 
use $ref to share common schema fields between different schemas, it'll be 
easier to do so if these are in a separate object.


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: mobrovac, Ottomata
Cc: EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron, GWicke, 
mobrovac, MZMcBride, bd808, JanZerebecki, Halfak, Krenair, brion, chasemp, 
Eevans, mmodell, Ottomata, Mattflaschen, Matanya, Aklapper, JAllemandou, 
jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, RobLa-WMF, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

Ottomata added a comment.

> @Ottomata, I think understanding the semantics of an event primarily requires 
> knowledge of the topic.


Hm, I don't think this is true.  You will need some understanding of what a 
historical dataset is, but that's all.  The historical datasets are going to be 
made up of many revisions of the same schema.  January may have data validated 
with schema revision 1, and February with schema revision 2.  If you don't know 
which exact schema-revision an event was validated with, you will have a hard 
time reasoning about what the data looks like.  This will be especially true if 
we are attempting to do Avro conversion.


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: mobrovac, Ottomata
Cc: intracer, EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron, 
GWicke, mobrovac, MZMcBride, bd808, JanZerebecki, Halfak, Krenair, brion, 
chasemp, Eevans, mmodell, Ottomata, Mattflaschen, Matanya, Aklapper, 
JAllemandou, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, 
RobLa-WMF, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

Ottomata added a comment.

Producer A has schema version 1. 
Producer B has schema version 2, which has added field "name" with default 
"nonya".

All of these events are being imported into Hadoop.  An analyst looks at the 
latest schema and wants to do some analysis on "name".  They write a job to 
query the data in Hadoop selecting the "name" field.  The data produced by 
Producer A does not have a "name" field.  The analyst gets key errors and their 
job fails.


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: mobrovac, Ottomata
Cc: intracer, EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron, 
GWicke, mobrovac, MZMcBride, bd808, JanZerebecki, Halfak, Krenair, brion, 
chasemp, Eevans, mmodell, Ottomata, Mattflaschen, Matanya, Aklapper, 
JAllemandou, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, 
RobLa-WMF, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

Ottomata added a comment.

Have we decided that defaults will be filled in for missing fields?


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: mobrovac, Ottomata
Cc: intracer, EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron, 
GWicke, mobrovac, MZMcBride, bd808, JanZerebecki, Halfak, Krenair, brion, 
chasemp, Eevans, mmodell, Ottomata, Mattflaschen, Matanya, Aklapper, 
JAllemandou, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, 
RobLa-WMF, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

Ottomata added a comment.

Or produce time.

But really, even if we fill in defaults during production or consumption, this 
will still be a problem for historical data.  Data is only consumed into Hadoop 
once, and schema changes can happen after consumption time.   If you have no 
way of associating a particular historical event with a schema, you won't be 
able to fill in defaults properly for missing keys.  Avro solves this during 
binary deserialization, by filling in the defaults from the reader's schema.

We can't avoid this problem with JSON Schema, but we can make it easier to deal 
with with by having a built in mapping from an event to a schema.


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: mobrovac, Ottomata
Cc: intracer, EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron, 
GWicke, mobrovac, MZMcBride, bd808, JanZerebecki, Halfak, Krenair, brion, 
chasemp, Eevans, mmodell, Ottomata, Mattflaschen, Matanya, Aklapper, 
JAllemandou, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, 
RobLa-WMF, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

Ottomata added a comment.

Events will be consumed into Hadoop close to production time (within an hour 
usually).  Schema changes made years after the fact cannot be reflected in 
years old historical data unless it is reprocessed and rewritten.


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: mobrovac, Ottomata
Cc: intracer, EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron, 
GWicke, mobrovac, MZMcBride, bd808, JanZerebecki, Halfak, Krenair, brion, 
chasemp, Eevans, mmodell, Ottomata, Mattflaschen, Matanya, Aklapper, 
JAllemandou, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, 
RobLa-WMF, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2015-10-28 Thread Ottomata

Ottomata added a comment.

> > If we adopt a convention of always storing schema name and/or revision in 
> > the schemas themselves, then we can do like EventLogging does and infer and 
> > validate the schema based on this value. This would especially be helpful 
> > in associating a message with an Avro Schema when serializing into binary.

> 

> 

> The topic configuration will take precedence, so we wouldn't use 
> client-supplied values for these fields, and would basically just write a 
> part of the topic configuration into each event. We also decided that we will 
> only evolve schemas in backwards-compatible ways. In practice, this means 
> that we'll only add fields, and the latest schema will be able to validate 
> both new and old data in each topic.

> 

> @Ottomata, which value do you see in recording the schema configured for a 
> topic at enqueue time in each event?


Mainly for analytics purposes.  For historical data and other analytics 
contexts, the data may be analyzed much farther down the line than from Kafka.  
In those contexts, the information about which topic the event came from will 
be lost.  If we don't have to topic, we won't be able to know which schema the 
event was validated with.

Also, it will be cumbersome to always need to load the topic/schema config from 
the schema repository for analytics purposes.


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: mobrovac, Ottomata
Cc: intracer, EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron, 
GWicke, mobrovac, MZMcBride, bd808, JanZerebecki, Halfak, Krenair, brion, 
chasemp, Eevans, mmodell, Ottomata, Mattflaschen, Matanya, Aklapper, 
JAllemandou, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, 
RobLa-WMF, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

Ottomata added a comment.

To avoid possible conflicts, I'd suggest we call this not just `id`.  How about 
`uuid`?  That's what EventLogging capsule does: 
https://meta.wikimedia.org/wiki/Schema:EventCapsule


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron, GWicke, 
mobrovac, MZMcBride, bd808, JanZerebecki, Halfak, Krenair, brion, chasemp, 
Eevans, mmodell, Ottomata, Mattflaschen, Matanya, Aklapper, JAllemandou, 
jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, RobLa-WMF, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

Ottomata added a comment.

I'm still a little confused about how this reqid/id will work?  You are 
suggesting that it comes from the x-request-id that we want varnish to set, 
right?  Won't this mean that multiple events (those produced during the same 
http request at varnish level) will have the same reqid?


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron, GWicke, 
mobrovac, MZMcBride, bd808, JanZerebecki, Halfak, Krenair, brion, chasemp, 
Eevans, mmodell, Ottomata, Mattflaschen, Matanya, Aklapper, JAllemandou, 
jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, RobLa-WMF, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

Ottomata added a comment.

Also, this is just a personal preference, but I'd prefer if we had a convention 
differentiating integer/second based 'timestamps' and string/date based 
'datetimes'.  For webrequest data, the ISO8601 is called `dt`.

https://wikitech.wikimedia.org/wiki/Analytics/Data/Webrequest


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron, GWicke, 
mobrovac, MZMcBride, bd808, JanZerebecki, Halfak, Krenair, brion, chasemp, 
Eevans, mmodell, Ottomata, Mattflaschen, Matanya, Aklapper, JAllemandou, 
jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, RobLa-WMF, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

Ottomata added a comment.

> Hm, I think duplicates should be detected based on the content of the message 
> itself and the time stamp.


EventLogging explicitly uses the uuid in MySQL as a unique key for all tables.  
Having it standardized on a single field means that the unique index creation 
is standard for all events.  This keeps duplicate events from ever being 
inserted into a single table.

MySQL unique indexes aren't really a consideration here, and I'd be fine if we 
could define a unique index across multiple keys, but I'd want those keys to be 
standardized in the framing of all events.  A `uuid` like EventLogging does 
would be an easy way to do this.


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron, GWicke, 
mobrovac, MZMcBride, bd808, JanZerebecki, Halfak, Krenair, brion, chasemp, 
Eevans, mmodell, Ottomata, Mattflaschen, Matanya, Aklapper, JAllemandou, 
jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, RobLa-WMF, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

Ottomata added a comment.

> Manual schema versions. We could increase the schema version every time we 
> change something in the schema. Easy to achieve but it's also easy to forget 
> to bump the version when something has been changed.


FWIW, this is how EventLogging does it, although the revisions are managed by 
Mediawiki revision IDs.  I'm still not sure, but I think it would make sense to 
keep each schema iteration present in the HEAD of the repo, so it would be 
fairly easy to manually bump revision IDs.


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron, GWicke, 
mobrovac, MZMcBride, bd808, JanZerebecki, Halfak, Krenair, brion, chasemp, 
Eevans, mmodell, Ottomata, Mattflaschen, Matanya, Aklapper, JAllemandou, 
jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, RobLa-WMF, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

Ottomata added a comment.

> I don't see a conflicting problem with id (even though id is a JSONSchema 
> keyword, but it relates to the schema, not its properties, so we're good 
> there). uuid is not a good choice, IMHO, it's like naming a field string 
> because its value is a string. The most accurate name would be reqid, since 
> that's what it is.


Ok cool, if that's the case, then `reqid` or even `request_id` (I like long 
names...what can I say?) sounds good.  EventLogging gives every event a really 
unique uuid, based on the message itself, so that you can always uniquely ID 
any event.  It mainly uses this for avoiding duplicates.  Can we add this to 
the description too?   See: 
https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/master/server/eventlogging/parse.py#L69


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron, GWicke, 
mobrovac, MZMcBride, bd808, JanZerebecki, Halfak, Krenair, brion, chasemp, 
Eevans, mmodell, Ottomata, Mattflaschen, Matanya, Aklapper, JAllemandou, 
jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, RobLa-WMF, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

Ottomata added a comment.

> So the producer would store the same time stamp twice? UUID v1 already 
> contains it.


Could you provide an example of what this UUID would look like?

A reason for having a timestamp only field is so that applications can use it 
for time based logic without having to also know how to extract the timestamp 
out of an overloaded uuid.

Also, who is responsible for setting this reqid?  In many cases, varnish, 
right?  A producer may emit several events during a given request, and it 
should have to ability to set what it considers to be the real timestamp of 
each event.


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Smalyshev, yuvipanda, Hardikj, daniel, aaron, GWicke, mobrovac, MZMcBride, 
bd808, JanZerebecki, Halfak, Krenair, brion, chasemp, Eevans, mmodell, 
Ottomata, Mattflaschen, Matanya, Aklapper, JAllemandou, jkroll, Wikidata-bugs, 
Jdouglas, aude, Deskana, Manybubbles, RobLa-WMF, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

Ottomata added a comment.

Right, but how would you do this in say, Hive?  Or in bash?

Timestamp logic should be easy and immediate.

> Regarding a separate timestamp in the framing information: Which time would 
> this correspond to?


This is up to the producer, I think.  If there are more timestamps needed for 
specific schema, that is fine, but I see a lot of value in having a canonical 
and easily readable timestamp.  Camus uses this timestamp to auto partition 
files by hour when they are imported from Kafka into HDFS.  ISO 8601 works, 
unix epoch seconds and milliseconds work.  We'd have to add more code to make 
UUID timestamp work.

Maybe this is ok, but I'd much rather be able to use my eyes and easy tools to 
do time logic.


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron, GWicke, 
mobrovac, MZMcBride, bd808, JanZerebecki, Halfak, Krenair, brion, chasemp, 
Eevans, mmodell, Ottomata, Mattflaschen, Matanya, Aklapper, JAllemandou, 
jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, RobLa-WMF, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2015-10-27 Thread Ottomata

Ottomata added a comment.

Ok, cool, I'm cool with that, so:

`request_id` - UUID1 from Varnish, not necessarily unique for an individual 
event
`event_id` (or maybe just `uuid`? since that is what EL uses?) - Actual UUID 
for an event.
`dt` - IS08601 timestamp, usually derived from the timestamp in `event_id`


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: mobrovac, Ottomata
Cc: EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron, GWicke, 
mobrovac, MZMcBride, bd808, JanZerebecki, Halfak, Krenair, brion, chasemp, 
Eevans, mmodell, Ottomata, Mattflaschen, Matanya, Aklapper, JAllemandou, 
jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, RobLa-WMF, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T117732: Create a Graphite instance in the Analytics cluster

2015-11-09 Thread Ottomata

Ottomata added a comment.

I'm not very opinionated here, but I think an analytics graphite instance will 
be very useful for other things than just this ticket, especially if these 
metrics are backed up by data in HDFS, and graphite is used as a visualization 
and rollup tool.


TASK DETAIL
  https://phabricator.wikimedia.org/T117732

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Lydia_Pintscher, fgiunchedi, Christopher, JanZerebecki, Nuria, Ottomata, 
Aklapper, Addshore, StudiesWorld, Wikidata-bugs, aude, Mbch331



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-11-10 Thread Ottomata

Ottomata added a comment.

Sounds good.  Shall I just find a time and set one up?


TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Milimetric, RobLa-WMF, brion, intracer, Smalyshev, mark, MZMcBride, 
Krinkle, EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, 
aaron, GWicke, mobrovac, Eevans, Ottomata, Matanya, Aklapper, JAllemandou, 
jkroll, Hardikj, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, daniel, 
Mbch331, Jay8g, Ltrlg, jeremyb, Legoktm



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-11-10 Thread Ottomata

Ottomata added a comment.

Just made a calendar event for Tuesday at 10:30 PST.  Happy to move it if some 
other time is better.


TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Milimetric, RobLa-WMF, brion, intracer, Smalyshev, mark, MZMcBride, 
Krinkle, EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, 
aaron, GWicke, mobrovac, Eevans, Ottomata, Matanya, Aklapper, JAllemandou, 
jkroll, Hardikj, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, daniel, 
Mbch331, Jay8g, Ltrlg, jeremyb, Legoktm



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T117732: Create a Graphite instance in the Analytics cluster

2015-11-09 Thread Ottomata

Ottomata added a comment.

Cassandra is a complicated clustered solution, graphite will just require a 
simple single instance running and intaking events somewhere.


TASK DETAIL
  https://phabricator.wikimedia.org/T117732

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Joe, Lydia_Pintscher, fgiunchedi, Christopher, JanZerebecki, Nuria, 
Ottomata, Aklapper, Addshore, StudiesWorld, Wikidata-bugs, aude, Mbch331



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Unblock] T114443: EventBus MVP

2015-11-16 Thread Ottomata

Ottomata closed blocking task T118578: Package EventLogging and dependencies 
for Jessie as "Resolved".

TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Milimetric, RobLa-WMF, brion, intracer, Smalyshev, mark, MZMcBride, 
Krinkle, EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, 
aaron, GWicke, mobrovac, Eevans, Ottomata, Matanya, Aklapper, JAllemandou, 
jkroll, Hardikj, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, daniel, 
Mbch331, Jay8g, Ltrlg, jeremyb, Legoktm



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2015-11-12 Thread Ottomata

Ottomata added a comment.

@daniel

This schema repo will be used by many codebases.  EventLogging, Mediawiki, 
analytics refinery, etc. etc.  Anyone creating events will need this code.

There are various ways to share these schemas, but one idea is to use git 
submodules.


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: mobrovac, Ottomata
Cc: intracer, EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron, 
GWicke, mobrovac, MZMcBride, bd808, JanZerebecki, Halfak, Krenair, brion, 
chasemp, Eevans, mmodell, Ottomata, Matanya, Aklapper, JAllemandou, jkroll, 
Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, RobLa-WMF, Mbch331, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2015-11-11 Thread Ottomata

Ottomata added a comment.

Is it time to consider creating a standalone repo for these schemas?  If so, 
then that means it is time for repo name bikeshed, woohoo!

No idea what to call this or where to put it.  `mediawiki/schemas`?


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: mobrovac, Ottomata
Cc: intracer, EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron, 
GWicke, mobrovac, MZMcBride, bd808, JanZerebecki, Halfak, Krenair, brion, 
chasemp, Eevans, mmodell, Ottomata, Matanya, Aklapper, JAllemandou, jkroll, 
Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, RobLa-WMF, Mbch331, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Unblock] T114443: EventBus MVP

2015-11-17 Thread Ottomata

Ottomata closed blocking task T118761: Move EventLogging/server to its own repo 
and set up CI as "Resolved".

TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Milimetric, RobLa-WMF, brion, intracer, Smalyshev, mark, MZMcBride, 
Krinkle, EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, 
aaron, GWicke, mobrovac, Eevans, Ottomata, Matanya, Aklapper, JAllemandou, 
jkroll, Hardikj, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, daniel, 
Mbch331, Jay8g, Ltrlg, jeremyb, Legoktm



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Created] T118869: Send HTTP stats about eventlogging-service to statsd

2015-11-17 Thread Ottomata

Ottomata created this task.
Ottomata claimed this task.
Ottomata added subscribers: Milimetric, Nuria, madhuvishy.
Ottomata added projects: Analytics-Kanban, operations, MediaWiki-RfCs, 
Wikidata-Query-Service, Service-Architecture, Services, 
MediaWiki-General-or-Unknown, Wikidata, Analytics, Epic, Discovery, EventBus.
Herald added a subscriber: Aklapper.

TASK DESCRIPTION
  Request response rates, timing (maybe?) errors, etc.
  
  Not sure of the best way to do this.  Maybe there is some plug in for tornado 
to just do this?  Or...we can change web frameworks pretty easily I'm sure, if 
we should.

TASK DETAIL
  https://phabricator.wikimedia.org/T118869

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: madhuvishy, Nuria, Milimetric, Aklapper, Eevans, JAllemandou, mobrovac, 
jkroll, Smalyshev, Hardikj, Wikidata-bugs, Jdouglas, aude, GWicke, Deskana, 
Manybubbles, daniel, RobLa-WMF, Mbch331, Jay8g, Ltrlg, bd808, jeremyb, Legoktm



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-11-03 Thread Ottomata

Ottomata added a comment.

> In my recollection of the discussion & the log you linked to, the question of 
> which REST producer proxy to use was left open.


I think you may be referring to the first link of the meetbot notes, which was 
ended before we stopped discussing.  Starting at [19:17:26]  in the chat 
logs <http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/20151030.txt>, 
it seems clear to me that the consensus is that unless there are good reasons 
to ditch something that already does most of what this project is about, then 
we should adapt what we are already using.  If I'm mistaken, please correct me.

> if you have something based on EventLogging soon then we can consider using 
> that too. Lets just make sure that the APIs are compatible & make sense in 
> the longer term.


Getting closer here <https://gerrit.wikimedia.org/r/#/c/235671>, need some help 
on the API big time.  Will also need to revisit some meta schema design things 
over on https://phabricator.wikimedia.org/T116247 to make things easier for 
EventLogging.

> Our priority is to get basic events into Kafka before the end of this month, 
> so that we can start building on top of this for change propagation


@gwicke, I think this may be a problem.  From my perspective, the goal of this 
project is a generalized event service with well designed and standardized 
schemas for all of WMF.  For this MVP, we have chosen to model change events 
because that is what you are interested in.  This is an 'MVP', and will likely 
require iteration after the first deployment.  I don't think having a live 
services production goal based on this is realistic.

> FWIW, one does not exclude the other: the EL-based service can be used in 
> production, while the node-based REST proxy may be used for development 
> and/or small installs.


That is one of the reasons for sticking with EventLogging.  It is already 
useable by small installs without Kafka.


TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: RobLa-WMF, brion, intracer, Smalyshev, mark, MZMcBride, Krinkle, 
EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron, 
GWicke, mobrovac, Eevans, Ottomata, Matanya, Aklapper, JAllemandou, jkroll, 
Hardikj, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, daniel, Mbch331, 
Jay8g, Ltrlg, jeremyb, Legoktm



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-10-30 Thread Ottomata

Ottomata added a comment.

Today's EventBus RFC 
<https://tools.wmflabs.org/meetbot/wikimedia-office/2015/wikimedia-office.2015-10-30-18.17.html>
  discussion 
<http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/20151030.txt>  ended 
with the general consensus that we will implement this project in EventLogging.

This is a large project that is meant to be useful to many teams.  Whatever the 
final implementation, there will need to be a path forward for deprecating 
existing use cases (EventLogging, RCStream, etc.) in favor of this system.

Given this project's cross-team generality, and the need to port existing use 
cases , and barring any practical or technical reasons not to, we will adapt 
EventLogging to include an HTTP REST Service.

We may need to conform some of our schema designs to EventLogging, and there 
may be other unknowns that we will discover as work on the implementation.


TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: RobLa-WMF, brion, intracer, Smalyshev, mark, MZMcBride, Krinkle, 
EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron, 
GWicke, mobrovac, Eevans, Ottomata, Matanya, Aklapper, JAllemandou, jkroll, 
Hardikj, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, daniel, Jay8g, 
Ltrlg, jeremyb, Legoktm



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2015-11-03 Thread Ottomata

Ottomata added a comment.

Cool, added some comments.


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: mobrovac, Ottomata
Cc: intracer, EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron, 
GWicke, mobrovac, MZMcBride, bd808, JanZerebecki, Halfak, Krenair, brion, 
chasemp, Eevans, mmodell, Ottomata, Mattflaschen, Matanya, Aklapper, 
JAllemandou, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, 
RobLa-WMF, Mbch331, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Changed Subscribers] T114443: EventBus MVP

2015-11-03 Thread Ottomata

Ottomata added a subscriber: Milimetric.
Ottomata added a comment.

Ok, still various TODOs around the code, but this is ready for review.
https://gerrit.wikimedia.org/r/#/c/235671

There are concepts that it'll be good to do close review with folks familiar 
with EventLogging (probably @nuria or @milimetric), and other that I'd really 
like services folks to look at @mobrovac and/or @eevans?


TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Milimetric, RobLa-WMF, brion, intracer, Smalyshev, mark, MZMcBride, 
Krinkle, EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, 
aaron, GWicke, mobrovac, Eevans, Ottomata, Matanya, Aklapper, JAllemandou, 
jkroll, Hardikj, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, daniel, 
Mbch331, Jay8g, Ltrlg, jeremyb, Legoktm



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Changed Project Column] T114443: EventBus MVP

2015-11-02 Thread Ottomata

Ottomata moved this task to In Progress on the Analytics-Kanban workboard.

TASK DETAIL
  https://phabricator.wikimedia.org/T114443

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1030/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: RobLa-WMF, brion, intracer, Smalyshev, mark, MZMcBride, Krinkle, 
EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron, 
GWicke, mobrovac, Eevans, Ottomata, Matanya, Aklapper, JAllemandou, jkroll, 
Hardikj, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, daniel, Jay8g, 
Ltrlg, jeremyb, Legoktm



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T114443: EventBus MVP

2015-11-02 Thread Ottomata

Ottomata added a project: Analytics-Kanban.

TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: RobLa-WMF, brion, intracer, Smalyshev, mark, MZMcBride, Krinkle, 
EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron, 
GWicke, mobrovac, Eevans, Ottomata, Matanya, Aklapper, JAllemandou, jkroll, 
Hardikj, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, daniel, Jay8g, 
Ltrlg, jeremyb, Legoktm



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-11-05 Thread Ottomata

Ottomata added a comment.

> Until very recently (last days), there wasn't actually an EventBus-like REST 
> proxy with schema validation in the EventLogging repository.


Not quite true, this was started Sept 3.
https://phabricator.wikimedia.org/T88459#1601022


TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Milimetric, RobLa-WMF, brion, intracer, Smalyshev, mark, MZMcBride, 
Krinkle, EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, 
aaron, GWicke, mobrovac, Eevans, Ottomata, Matanya, Aklapper, JAllemandou, 
jkroll, Hardikj, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, daniel, 
Mbch331, Jay8g, Ltrlg, jeremyb, Legoktm



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-10-19 Thread Ottomata

Ottomata added a comment.

So, we need to be really careful here.  This MVP as of yet has zero buy in from 
anyone in ops.  In addition, both @ori and @eevans point out that EventLogging 
already does everything that this MVP encompasses, minus the HTTP service part. 
 Now it is time for me to chime in too, woowee!

> Could you explain how you arrived at the figure of 50k requests per second, 
> which you project for this service?


This is just an arbitrary goal, some number we came up with.  I'd like to be 
able to encourage developers to use EventBus for everything they can think of.

We've scaled EventLogging to about 10k / second by using Kafka, but that is 
only on a single node.  EventLogging is horizontally scalable.  Need more 
throughput?  Add more partitions and processors.

In addition, the EventLogging processors are doing more than just validating 
JSON messages.  They are parsing the JSON data out of encoded query strings via 
regexes, wrapping the incoming event data with generic metadata, anonymizing IP 
addresses using a shared rotating salt key from etcd, sending invalid events 
off to Kafka as EventErrors, etc. etc.

> We also need proper code review and versioning for core schemas, and wikis 
> don't really support code review. We could consider storing pointers to 
> schemas (URLs) instead of the actual schemas in git, but this adds complexity 
> without much apparent benefit:


I think this is true, especially for the 'production' use case of EventBus.  
EventLogging was originally designed for analytics use cases, some of which are 
short lived one-off's (A/B testing, whatever).  Making quick changes via a wiki 
is awesome for this.  Having more control over changes to production schemas 
sounds like a good idea.

> However, if we assume that we need an additional class of in-tree schemas, 
> then the inverse is also true; It would be just as trivial to implement 
> reading from the filesystem.


Agreed, it would trivial to add filesystem based schemas to EventLogging.  In 
fact, this is sort of already done, via the cached schema system.  Schemas 
needed for unit testing are hardcoded into the source and manually inserted 
into the in memory schema cache.  We could do the same thing with a filesystem 
tree of schemas: preload them into the in memory cache.  When asked to validate 
a schema from the filesystem, EventLogging wouldn't even bother trying to reach 
out to meta, since it would already be in the in memory cache.  See: 
https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/master/server/eventlogging/schema.py#L64

> > Provides a pluggable, composable, architecture with support for a wide 
> > range of readers/writers

> 

> 

> How would this be an advantage for the EventBus portion? Many third-party 
> users will actually only want a minimal event bus, and EL doesn't seem to 
> help with this from what I have seen.


It does, no? EventLogging is already a useable extension with third party 
Mediawiki installations.  Kafka isn't needed to use EventLogging at all.

> See https://phabricator.wikimedia.org/T88459#1604768. tl;dr: It's not 
> necessarily clear that saving very little code (see above) for EL schema 
> fetching outweights the cost of additional hardware.


As mentioned here <https://phabricator.wikimedia.org/T88459#1597109>, here 
<https://phabricator.wikimedia.org/T88459#1614048>,  here 
<https://phabricator.wikimedia.org/T88459#1617315>,  here 
<https://phabricator.wikimedia.org/T88459#1597109>, and here 
<https://phabricator.wikimedia.org/T114443#1703643>, comparing the performance 
now is interesting, but provides little insight as to how this system will 
perform in the real world with more features.  EventLogging is doing more than 
just accepting a JSON message and validating it against a schema.  In any case, 
this system will need to be horizontally scalable.  As noted, the production 
use case will be much lower volume than the analytics one.  The performance of 
all the solutions we evaluated in https://phabricator.wikimedia.org/T88459 is 
suitable for the production use case, especially since they are all 
horizontally scalable.


TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: mark, MZMcBride, Krinkle, EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, 
Nuria, ori, faidon, aaron, GWicke, mobrovac, Eevans, Ottomata, Matanya, 
Aklapper, JAllemandou, jkroll, Smalyshev, Hardikj, Wikidata-bugs, Jdouglas, 
RobH, aude, Deskana, Manybubbles, daniel, JanZerebecki, RobLa-WMF, Jay8g, 
Ltrlg, fgiunchedi, Dzahn, jeremyb, Legoktm, chasemp, Krenair



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-10-19 Thread Ottomata

Ottomata added a comment.

Hey yalls,

I've had requests that we postpone the RFC for this one more week, until Oct 
28th.  I'd like for one opsen and @ori to be able to attend, and the relevant 
opsens are all traveling, and Ori can't make this one either.


TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: mark, MZMcBride, Krinkle, EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, 
Nuria, ori, faidon, aaron, GWicke, mobrovac, Eevans, Ottomata, Matanya, 
Aklapper, JAllemandou, jkroll, Smalyshev, Hardikj, Wikidata-bugs, Jdouglas, 
RobH, aude, Deskana, Manybubbles, daniel, JanZerebecki, RobLa-WMF, Jay8g, 
Ltrlg, fgiunchedi, Dzahn, jeremyb, Legoktm, chasemp, Krenair



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-10-06 Thread Ottomata

Ottomata added a comment.

@Joe,  there are two parts to this MVP:

- Centralized (and CI controlled) schema sharing
- An easy way to get valid data into Kafka.

With eventlogging right now, we are spending a lot of resources just processing 
incoming data and making sure it is valid.  When data is produced by N clients, 
it is difficult to guarantee that all of them are producing valid data.  
Invalid data in a stream can make processing and sanity checking difficult, 
especially in a distributed environment like Hadoop.

The goal of this system is to standardize the way we pass messages between 
different infrastructures.  For that we need strict schemas and schema 
validation.  For lowish volume production data (job queue, change propagation, 
etc.), using a proxy that can tell a client that it has sent invalid data is 
valuable.  If a client tries to produce invalid data, they will get a 500 
response of some kind, and can react appropriately.

You are right though, we could achieve a similar thing if we built Kafka 
wrappers that new how to use our centralized schema system in all languages.  
Then each wrapper could validate the message a client is trying to send before 
it actually sends.  I think this could potentially introduce more bugs, as 
there is more actual code to maintain, as well as more places that the code is 
deployed.  E.g. we'd have to make sure that all clients updated their wrapper 
library if we fix a bug.


TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron, 
GWicke, mobrovac, Eevans, Ottomata, Matanya, Aklapper, JAllemandou, jkroll, 
Smalyshev, Hardikj, Wikidata-bugs, Jdouglas, RobH, aude, Deskana, Manybubbles, 
mark, JanZerebecki, RobLa-WMF, fgiunchedi, Dzahn, jeremyb, chasemp, Krenair



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-10-06 Thread Ottomata

Ottomata added a comment.

@joe, yes, please see this ticket:

https://phabricator.wikimedia.org/T88459


TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron, 
GWicke, mobrovac, Eevans, Ottomata, Matanya, Aklapper, JAllemandou, jkroll, 
Smalyshev, Hardikj, Wikidata-bugs, Jdouglas, RobH, aude, Deskana, Manybubbles, 
mark, JanZerebecki, RobLa-WMF, fgiunchedi, Dzahn, jeremyb, chasemp, Krenair



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-10-06 Thread Ottomata

Ottomata added a comment.

@ori I just edited the EventBus project description 
<https://phabricator.wikimedia.org/project/sprint/profile/1474/> to include a 
version of the problem statement I sent out to engineering@ back in July.


TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron, 
GWicke, mobrovac, Eevans, Ottomata, Matanya, Aklapper, JAllemandou, jkroll, 
Smalyshev, Hardikj, Wikidata-bugs, Jdouglas, RobH, aude, Deskana, Manybubbles, 
mark, JanZerebecki, RobLa-WMF, fgiunchedi, Dzahn, jeremyb, chasemp, Krenair



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T84923: Reliable publish / subscribe event bus

2015-08-31 Thread Ottomata

Ottomata added a project: EventBus.

TASK DETAIL
  https://phabricator.wikimedia.org/T84923

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Aklapper, Matanya, Mattflaschen, Ottomata, mmodell, Eevans, chasemp, brion, 
Krenair, Halfak, JanZerebecki, bd808, MZMcBride, mobrovac, GWicke, aaron, 
daniel, Hardikj, yuvipanda, JAllemandou, jkroll, Smalyshev, Wikidata-bugs, 
Jdouglas, RobH, aude, Deskana, Manybubbles, mark, RobLa-WMF, faidon, 
fgiunchedi, Dzahn, jeremyb, Malyacko



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Unblock] T84923: Reliable publish / subscribe event bus

2015-08-31 Thread Ottomata

Ottomata closed blocking task T110748: Event Bus as "Invalid".

TASK DETAIL
  https://phabricator.wikimedia.org/T84923

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Aklapper, Matanya, Mattflaschen, Ottomata, mmodell, Eevans, chasemp, brion, 
Krenair, Halfak, JanZerebecki, bd808, MZMcBride, mobrovac, GWicke, aaron, 
daniel, Hardikj, yuvipanda, JAllemandou, jkroll, Smalyshev, Wikidata-bugs, 
Jdouglas, RobH, aude, Deskana, Manybubbles, mark, RobLa-WMF, faidon, 
fgiunchedi, Dzahn, jeremyb, Malyacko



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

Ottomata added a comment.

Does the description look ok?  Feel free to edit and discuss here.


TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: madhuvishy, Nuria, ori, faidon, aaron, GWicke, mobrovac, Halfak, Eevans, 
Ottomata, Matanya, Aklapper, JAllemandou, jkroll, Smalyshev, Hardikj, 
Wikidata-bugs, Jdouglas, RobH, aude, Deskana, Manybubbles, mark, JanZerebecki, 
RobLa-WMF, bd808, fgiunchedi, Dzahn, jeremyb, chasemp, Krenair



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

Ottomata added a comment.

Nuria pointed out to me that (as engineers love to do) we are focusing a lot 
here on technical architecture, but haven't thought a lot about what use case 
this MVP will actual serve.  I agree, and it would be good for us to pick a 
simple use case for this system and actually implement it using this MVP.  I'm 
not sure which use case that would be, but it would be nice to have something 
functional and useful as the real goal of the MVP.

Thoughts?  Use cases were collected at the bottom of this etherpad:
https://etherpad.wikimedia.org/p/scalable_events_system


TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: madhuvishy, Nuria, ori, faidon, aaron, GWicke, mobrovac, Halfak, Eevans, 
Ottomata, Matanya, Aklapper, JAllemandou, jkroll, Smalyshev, Hardikj, 
Wikidata-bugs, Jdouglas, RobH, aude, Deskana, Manybubbles, mark, JanZerebecki, 
RobLa-WMF, bd808, fgiunchedi, Dzahn, jeremyb, chasemp, Krenair



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Changed Subscribers] T114443: EventBus MVP

Ottomata added a subscriber: dr0ptp4kt.
Ottomata set Security to None.

TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron, GWicke, mobrovac, Halfak, 
Eevans, Ottomata, Matanya, Aklapper, JAllemandou, jkroll, Smalyshev, Hardikj, 
Wikidata-bugs, Jdouglas, RobH, aude, Deskana, Manybubbles, mark, JanZerebecki, 
RobLa-WMF, bd808, fgiunchedi, Dzahn, jeremyb, chasemp, Krenair



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Created] T114443: EventBus MVP

Ottomata created this task.
Ottomata added subscribers: Aklapper, Ottomata, Eevans, Halfak, mobrovac, 
GWicke, aaron, faidon, ori, Nuria, madhuvishy.
Ottomata added projects: EventBus, Discovery, Epic, Analytics, Wikidata, 
operations, MediaWiki-General-or-Unknown, Services, Service-Architecture, 
Wikidata-Query-Service.
Herald added a subscriber: Matanya.

TASK DESCRIPTION
  Over in T88459, and in a few recent meetings, we've flushed out a sketch for 
to get standardized messages into Kafka for later consumption.  We've coalesced 
on a way to move forward, and a MVP.  This task will track the creation of the 
EventBus MVP.
  
  == Architecture Decisions
  - We will standardize on JSON Schema as our canonical schema spec, but do so 
in such a way that Avro can be used in Analytics type systems.  Equivalent Avro 
Schemas may be generated as part of CI.
  - For MVP, JSON data will be produced to Kafka.  We consider Avro Binary 
later.
  - There will be a Kafka Topic -> Schema mapping, and only that schema can be 
produced to a topic.
  
  == MVP Description
  
  The MVP will consist of:
  - REST Service that validates JSON data against a schema and produces to 
Kafka.
  - Schema Repository Layout and Topic -> Schema mapping config that Service 
loads on startup.
  - A TBD implemented use case of this system.
  
  The MVP might also include:
  - Schema review and CI processes:
  -- schema evolution rules
  -- Auto Avro schema generation
  -- Auto Avro java class generation
  - Schema lookup service
  - Schema listing and discussion UI (Mediwiki Extension?)
  - Schema metadata conventions (fields common to all schemas?)

TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: madhuvishy, Nuria, ori, faidon, aaron, GWicke, mobrovac, Halfak, Eevans, 
Ottomata, Matanya, Aklapper, JAllemandou, jkroll, Smalyshev, Hardikj, 
Wikidata-bugs, Jdouglas, RobH, aude, Deskana, Manybubbles, mark, JanZerebecki, 
RobLa-WMF, bd808, fgiunchedi, Dzahn, jeremyb, chasemp, Krenair



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T114443: EventBus MVP

Ottomata added a comment.

Edit events would be awesome and totally doable with this MVP, but I'm a little 
worried about the amount of bike shedding that will go into designing that 
schema!

What about api.php logging?  See https://phabricator.wikimedia.org/T108618.  
Those logs are still being collected via udp2log, whereas the edit ones do have 
a functional subscribe-able  endpoint.  Also, the volume of API logs is much 
higher than edits, and maybe it'd be nice to have a use case that pushes the 
system a bit.


TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron, GWicke, mobrovac, 
Halfak, Eevans, Ottomata, Matanya, Aklapper, JAllemandou, jkroll, Smalyshev, 
Hardikj, Wikidata-bugs, Jdouglas, RobH, aude, Deskana, Manybubbles, mark, 
JanZerebecki, RobLa-WMF, bd808, fgiunchedi, Dzahn, jeremyb, chasemp, Krenair



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

Ottomata added a comment.

@bd808, what's your timeline on this?  Producing directly into Kafka is fine, 
but we are trying to do two things with this MVP:

- centralize and standardize schemas
- provide a REST API for producing to Kafka

We aren't going to block you on your work, but if you could, would you prefer 
to produce directly to Kafka, or to POST to a REST api?  Meh, probably you 
don't care. :)


TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron, GWicke, 
mobrovac, Halfak, Eevans, Ottomata, Matanya, Aklapper, JAllemandou, jkroll, 
Smalyshev, Hardikj, Wikidata-bugs, Jdouglas, RobH, aude, Deskana, Manybubbles, 
mark, JanZerebecki, RobLa-WMF, fgiunchedi, Dzahn, jeremyb, chasemp, Krenair



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

Ottomata added a comment.

Oh, so the service is planned to be kafka agnostic?  (I will come hang on your 
side after lunch :) )


TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron, 
GWicke, mobrovac, Halfak, Eevans, Ottomata, Matanya, Aklapper, JAllemandou, 
jkroll, Smalyshev, Hardikj, Wikidata-bugs, Jdouglas, RobH, aude, Deskana, 
Manybubbles, mark, JanZerebecki, RobLa-WMF, fgiunchedi, Dzahn, jeremyb, 
chasemp, Krenair



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

Ottomata added a comment.

Hm, @gwicke, if Mediawiki has access to schemas, and is relatively sure that it 
can produce valid messages for a given topic, why should a Mediawiki client use 
EventBus at all?  They can produce directly to Kafka, bypassing the HTTP and 
proxying overhead of EventBus.


TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron, 
GWicke, mobrovac, Halfak, Eevans, Ottomata, Matanya, Aklapper, JAllemandou, 
jkroll, Smalyshev, Hardikj, Wikidata-bugs, Jdouglas, RobH, aude, Deskana, 
Manybubbles, mark, JanZerebecki, RobLa-WMF, fgiunchedi, Dzahn, jeremyb, 
chasemp, Krenair



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T84923: Reliable publish / subscribe event bus

2015-08-28 Thread Ottomata

Ottomata added a blocking task: T110748: Event Bus.

TASK DETAIL
  https://phabricator.wikimedia.org/T84923

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Aklapper, Matanya, Mattflaschen, Ottomata, mmodell, Eevans, chasemp, brion, 
Krenair, Halfak, JanZerebecki, bd808, MZMcBride, mobrovac, GWicke, aaron, 
daniel, Hardikj, yuvipanda, JAllemandou, jkroll, Smalyshev, Wikidata-bugs, 
Jdouglas, RobH, aude, Deskana, Manybubbles, mark, RobLa-WMF, faidon, 
fgiunchedi, Dzahn, jeremyb, Malyacko



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2015-12-03 Thread Ottomata

Ottomata added a comment.

Hm, not sure I follow.  We are proposing that a schema be ID-able via a URI, 
and also remotely locatable if that URI happens to be a full URL with schema 
and domain information.  Is the opaqueness issue the fact that it is 
`/` that IDs the schema, instead of just a unique  
`/`?


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: mobrovac, Ottomata
Cc: RobLa-WMF, Nuria, gerritbot, intracer, EBernhardson, Smalyshev, yuvipanda, 
Hardikj, daniel, aaron, GWicke, mobrovac, MZMcBride, bd808, JanZerebecki, 
Halfak, Krenair, brion, chasemp, Eevans, mmodell, Ottomata, Matanya, Aklapper, 
JAllemandou, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, 
Mbch331, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2015-12-03 Thread Ottomata

Ottomata added a comment.

Hm, I think I see.  We are coupling the URI to the ID, which according to the 
W3C should not be relied upon.  Ok, noted.


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: mobrovac, Ottomata
Cc: RobLa-WMF, Nuria, gerritbot, intracer, EBernhardson, Smalyshev, yuvipanda, 
Hardikj, daniel, aaron, GWicke, mobrovac, MZMcBride, bd808, JanZerebecki, 
Halfak, Krenair, brion, chasemp, Eevans, mmodell, Ottomata, Matanya, Aklapper, 
JAllemandou, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, 
Mbch331, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-12-23 Thread Ottomata

Ottomata added a comment.

No, consumption is not part of the MVP.

There may be future work to make consumption from Kafka via websockets easy to 
set up, but we will not make any Events public by default.  We will have to set 
up special endpoints for approved event streams.


TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: yuvipanda, JanZerebecki, Glaisher, Milimetric, RobLa-WMF, brion, intracer, 
Smalyshev, mark, MZMcBride, Krinkle, EBernhardson, bd808, Joe, dr0ptp4kt, 
madhuvishy, Nuria, ori, faidon, aaron, GWicke, mobrovac, Eevans, Ottomata, 
Matanya, Aklapper, jkroll, Hardikj, Wikidata-bugs, Jdouglas, aude, Deskana, 
Manybubbles, jayvdb, daniel, Mbch331, Jay8g, Ltrlg, Legoktm



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Unblock] T114443: EventBus MVP

2015-12-18 Thread Ottomata

Ottomata closed blocking task T118869: Send HTTP stats about 
eventlogging-service to statsd [3 pts] as "Resolved".

TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Glaisher, Milimetric, RobLa-WMF, brion, intracer, Smalyshev, mark, 
MZMcBride, Krinkle, EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, 
ori, faidon, aaron, GWicke, mobrovac, Eevans, Ottomata, Matanya, Aklapper, 
jkroll, Hardikj, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, jayvdb, 
daniel, Mbch331, Jay8g, Ltrlg, Legoktm



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Unblock] T114443: EventBus MVP

2015-12-22 Thread Ottomata

Ottomata closed blocking task T118780: Puppetize eventlogging-service as 
"Resolved".

TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Glaisher, Milimetric, RobLa-WMF, brion, intracer, Smalyshev, mark, 
MZMcBride, Krinkle, EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, 
ori, faidon, aaron, GWicke, mobrovac, Eevans, Ottomata, Matanya, Aklapper, 
jkroll, Hardikj, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, jayvdb, 
daniel, Mbch331, Jay8g, Ltrlg, Legoktm



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Changed Subscribers] T116247: Define edit related events for change propagation

2015-11-30 Thread Ottomata

Ottomata added a subscriber: Nuria.
Ottomata added a comment.

@gwicke and I discussed the schema/revision in meta issue in IRC today. He had 
an idea that I quite like!

@gwicke suggested that instead of using (schema, revision) to uniquely ID a 
schema, that we just use a URI.  EventLogging does this already with schemas 
stored in meta.wikmedia.org, but the URI resolution is done behind the scenes.  
Explicitly setting meta.schema to a URI in each event allows us to easily look 
up a schema outside of any EventLogging/EventBus context.  I believe it would 
be easy to support this in EventLogging code as long as extracting the schema 
name and revision from the URI is standardized.  Whatever the URI is, its last 
two path elements should be name/revision, e.g. 
`.../schemas/jsonschema/{title}/{rev}`.

This would certainly solve the issues that @nuria and I had about not including 
schema ids in the events.

Thoughts?  I'll look into the implementation of this tomorrow to make sure 
there isn't something that would make this difficult.


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: mobrovac, Ottomata
Cc: Nuria, gerritbot, intracer, EBernhardson, Smalyshev, yuvipanda, Hardikj, 
daniel, aaron, GWicke, mobrovac, MZMcBride, bd808, JanZerebecki, Halfak, 
Krenair, brion, chasemp, Eevans, mmodell, Ottomata, Matanya, Aklapper, 
JAllemandou, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, 
RobLa-WMF, Mbch331, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2015-11-19 Thread Ottomata

Ottomata added a comment.

FYI, the repo is here, waiting for some schemas! :)

https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/event-schemas

For Avro, @ebernhardson, you can go ahead and submit a patch there in an avro/ 
directory.  We should should probably maintain the usual Java directory 
hierarchy there for them.


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: mobrovac, Ottomata
Cc: intracer, EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron, 
GWicke, mobrovac, MZMcBride, bd808, JanZerebecki, Halfak, Krenair, brion, 
chasemp, Eevans, mmodell, Ottomata, Matanya, Aklapper, JAllemandou, jkroll, 
Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, RobLa-WMF, Mbch331, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2016-01-12 Thread Ottomata

Ottomata added a comment.

I believe we can close this task, ja?  Got a few defined here: 
https://github.com/wikimedia/mediawiki-event-schemas/tree/master/jsonschema/mediawiki


TASK DETAIL
  https://phabricator.wikimedia.org/T116247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: mobrovac, Ottomata
Cc: RobLa-WMF, Nuria, gerritbot, intracer, EBernhardson, Smalyshev, yuvipanda, 
Hardikj, daniel, aaron, GWicke, mobrovac, MZMcBride, bd808, JanZerebecki, 
Halfak, Krenair, brion, chasemp, Eevans, mmodell, Ottomata, Matanya, Aklapper, 
JAllemandou, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, 
Mbch331, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T118739: Push dumps.wm.o logs files to stat1002

2016-02-11 Thread Ottomata

Ottomata added a subscriber: Ottomata.
Ottomata added a comment.

Hey sorry, I don't think I've seen this ticket before, hence the silence!  I 
just commented on ticket about pull vs. push.


TASK DETAIL
  https://phabricator.wikimedia.org/T118739

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ArielGlenn, Ottomata
Cc: Ottomata, gerritbot, ArielGlenn, hoo, Deskana, Aklapper, Addshore, Izno, 
Wikidata-bugs, aude, Svick, Mbch331, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Unblock] T84923: Reliable publish / subscribe event bus

2016-02-01 Thread Ottomata

Ottomata closed blocking task T88459: Implementing the reliable event bus using 
Kafka as "Resolved".

TASK DETAIL
  https://phabricator.wikimedia.org/T84923

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: GWicke, Ottomata
Cc: Aklapper, Matanya, Ottomata, mmodell, Eevans, chasemp, brion, Krenair, 
Halfak, JanZerebecki, bd808, MZMcBride, mobrovac, GWicke, aaron, daniel, 
Hardikj, yuvipanda, debt, Izno, JAllemandou, jkroll, Smalyshev, Wikidata-bugs, 
Jdouglas, aude, Deskana, Manybubbles, RobLa-WMF, Mbch331, jeremyb



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T114443: EventBus MVP

2016-02-29 Thread Ottomata

Ottomata added a comment.


  We will resolve this after https://phabricator.wikimedia.org/T120212 is 
closed, and after we have the first consumer (change propagation) in production.

TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: yuvipanda, JanZerebecki, Glaisher, Milimetric, RobLa-WMF, brion, intracer, 
Smalyshev, mark, MZMcBride, Krinkle, EBernhardson, bd808, Joe, dr0ptp4kt, 
madhuvishy, Nuria, ori, faidon, aaron, GWicke, mobrovac, Eevans, Ottomata, 
Matanya, Aklapper, debt, Gehel, D3r1ck01, FloNight, Izno, JAllemandou, jkroll, 
Hardikj, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, jayvdb, daniel, 
Mbch331, Jay8g, Ltrlg, jeremyb, Legoktm



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Changed Project Column] T114443: EventBus MVP

2016-02-01 Thread Ottomata

Ottomata moved this task to In Progress on the EventBus workboard.

TASK DETAIL
  https://phabricator.wikimedia.org/T114443

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1474/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: yuvipanda, JanZerebecki, Glaisher, Milimetric, RobLa-WMF, brion, intracer, 
Smalyshev, mark, MZMcBride, Krinkle, EBernhardson, bd808, Joe, dr0ptp4kt, 
madhuvishy, Nuria, ori, faidon, aaron, GWicke, mobrovac, Eevans, Ottomata, 
Matanya, Aklapper, debt, Izno, jkroll, Hardikj, Wikidata-bugs, Jdouglas, aude, 
Deskana, Manybubbles, jayvdb, daniel, Mbch331, Jay8g, Ltrlg, Legoktm



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2016-04-27 Thread Ottomata

Ottomata added a comment.


  YEEHAW

TASK DETAIL
  https://phabricator.wikimedia.org/T114443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: yuvipanda, JanZerebecki, Glaisher, Milimetric, RobLa-WMF, brion, intracer, 
Smalyshev, mark, MZMcBride, Krinkle, EBernhardson, bd808, Joe, dr0ptp4kt, 
madhuvishy, Nuria, ori, faidon, aaron, GWicke, mobrovac, Eevans, Ottomata, 
Matanya, Aklapper, Avner, debt, Gehel, D3r1ck01, FloNight, Izno, JAllemandou, 
jkroll, Hardikj, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, jayvdb, 
fbstj, Mbch331, Jay8g, Ltrlg, jeremyb, Legoktm



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T119070: Track the number of wikidata dumps that are downloaded by type

2016-05-19 Thread Ottomata

Ottomata added a comment.


  Done!

TASK DETAIL
  https://phabricator.wikimedia.org/T119070

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Ottomata, Aklapper, Addshore, StudiesWorld, D3r1ck01, Izno, Wikidata-bugs, 
aude, Mbch331



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T161731: Create reliable change stream for specific wiki

2017-04-06 Thread Ottomata

Ottomata added a comment.
@Smalyshev let's jump in a hangout sometime to discuss this more.

Just a few quick points:

Does not have data back more than 7 days

We could probably bump this up to 14 days for specific topics like recentchange.

Scalable - there's no hard limit on the number of clients connecting

EventStreams does not have a 'hard limit', but it definitely isn't intended to be used for wiki-reader scale. Scalable service updates should be fine though.

Not consumable per-wiki due to the lack of filtering (T152731)

We haven't done this because we are waiting for a strong use case. Implementing it will be a lot of bike shedding, and we've put that off due to a lack of solid use case. If you really need this, say so, and we will start the bike shedding!

Qs:

We have a more than recentchange in other schemas, and we can add more. If there is a desire, we can expose these in EventStreams. Do you have desire? :)

Does WDQS run in production? I think it does, right? If so, you may want to consider consuming from Kafka rather than EventStreams. Kafka consumers support parallelization and scale much better than the EventStreams HTTP Kafka consumer proxy.

TASK DETAILhttps://phabricator.wikimedia.org/T161731EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Anomie, Aklapper, Smalyshev, QZanden, EBjune, merbst, Salgo60, Avner, debt, Gehel, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, JAllemandou, mobrovac, jkroll, Wikidata-bugs, Jdouglas, aude, GWicke, Deskana, Manybubbles, Mbch331, Krenair, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T161731: Create reliable change stream for specific wiki

2017-04-06 Thread Ottomata

Ottomata added a comment.
If so, you may want to consider consuming from Kafka rather than EventStreams.

I am considering this too, but I assume it's more code for me to write (maybe wrongly, I didn't look at it closely).

It will be more, a lot more.  What language are you working in?

But what in fact I need is something like 10-20 parallel consumers

Load balanced parallel consumers, or all distinct consumers doing different stuff?

Nice thing about using a Kafka client directly, is you can subscribe to multiple topics/partitions in a single consumer group, and have the load automatically balanced between them (up to the total number of topic-partitions).  If any of those processes goes down, a different consumer process will be auto-assigned its work.TASK DETAILhttps://phabricator.wikimedia.org/T161731EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Anomie, Aklapper, Smalyshev, QZanden, EBjune, merbst, Salgo60, Avner, debt, Gehel, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, JAllemandou, mobrovac, jkroll, Wikidata-bugs, Jdouglas, aude, GWicke, Deskana, Manybubbles, Mbch331, Krenair, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T161731: Create reliable change stream for specific wiki

2017-04-06 Thread Ottomata

Ottomata added a comment.
I'd rather have some intermediary that cleans up, deduplicates, etc. the changes.

FYI, neither base Kafka Consumer clients nor EventStreams does this.TASK DETAILhttps://phabricator.wikimedia.org/T161731EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Anomie, Aklapper, Smalyshev, QZanden, EBjune, merbst, Salgo60, Avner, debt, Gehel, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, JAllemandou, mobrovac, jkroll, Wikidata-bugs, Jdouglas, aude, GWicke, Deskana, Manybubbles, Mbch331, Krenair, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Triaged] T154017: compile number of http uses for http://www.wikidata.org/entity

2017-03-06 Thread Ottomata

Ottomata triaged this task as "Low" priority.Ottomata removed a project: Operations.Herald added a project: Operations.
TASK DETAILhttps://phabricator.wikimedia.org/T154017EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Aklapper, Lydia_Pintscher, Esc3300, Th3d3v1ls, EBjune, merbst, Avner, Zppix, debt, Gehel, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Deskana, Dinoguy1000, Manybubbles, faidon, Seb35, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T154015: announce breaking change: http > https for entities in rdf

2017-03-06 Thread Ottomata

Ottomata removed a project: Operations.Ottomata added a comment.Herald added a project: Operations.
This has been placed on the Traffic board, removing operations tag.TASK DETAILhttps://phabricator.wikimedia.org/T154015EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Ottomata, Ricordisamoa, Boshomi, Lydia_Pintscher, Aklapper, Esc3300, Th3d3v1ls, EBjune, merbst, Avner, Zppix, debt, Gehel, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Deskana, Dinoguy1000, Manybubbles, faidon, Seb35, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Triaged] T154015: announce breaking change: http > https for entities in rdf

2017-03-06 Thread Ottomata

Ottomata triaged this task as "Normal" priority.
TASK DETAILhttps://phabricator.wikimedia.org/T154015EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Ottomata, Ricordisamoa, Boshomi, Lydia_Pintscher, Aklapper, Esc3300, Th3d3v1ls, EBjune, merbst, Avner, Zppix, debt, Gehel, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Deskana, Dinoguy1000, Manybubbles, faidon, Seb35, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T151717: Usage tracking: record which statement group is used

2017-08-10 Thread Ottomata

Ottomata added a comment.
I haven't fully grokked this ticket, but in general I am for events! :) 2 qs:


How many events per second?
What would be emitting this event?  Mediawiki?
TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Ottomata, PokestarFan, Ladsgroup, Stashbot, gerritbot, Halfak, jcrespo, TomT0m, Hall1467, hoo, zhuyifei1999, Eloquence, Lydia_Pintscher, Sannita, Ainali, Liuxinyu970226, MZMcBride, Ricordisamoa, Micru, jayvdb, Daniel_Mietchen, Tobi_WMDE_SW, Legoktm, Abraham, Wikidata-bugs, liangent, jeremyb, aude, Candalua, Bianjiang, Aklapper, DixonD, daniel, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Marostegui, Lewizho99, Minhnv-2809, Maathavan, Izno, Luke081515, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T173850: Possible WMF deployed extension PHP 7 issues

2017-08-24 Thread Ottomata

Ottomata edited projects, added Analytics-Kanban; removed Analytics.
TASK DETAILhttps://phabricator.wikimedia.org/T173850EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Liuxinyu970226, WMDE-leszek, Anomie, Aklapper, Jdforrester-WMF, Addshore, Krinkle, MaxSem, Krenair, Reedy, Minilexikon, Timoavax79, GoranSMilovanovic, TheDragonFire, Maosef, 45Jayjay1969, QZanden, EBjune, Orienteerix, EnricoCNC, Litanwei, Avner, Ddproxy, debt, TerraCodes, SundanceRaphael, Reasno, Gehel, Debenben, Jseddon, JGirault, gabriel-wmde, phabyogi, FloNight, DStrine, Trizek-WMF, GAllegre, Susannaanas, XenoRyet, Thibaut120094, Izno, SBisson, Matiia, ferdbold, lxbarth, Wong128hk, Luke081515, Nirmos, Planemad, Wikidata-bugs, Snowolf, Prod, Hydriz, aude, fredw, SPQRobin, AndyRussG, Ebe123, Pcoombe, Huji, Physikerwelt, Pkra, Dinoguy1000, Gryllida, jeblad, jayvdb, Ricordisamoa, MrStradivarius, Catrope, Lydia_Pintscher, Yurik, Darkdadaah, Arrbee, scfc, KartikMistry, Se4598, Jackmcbarn, Mbch331, Jay8g, Ltrlg, Joe, jeremyb, Legoktm, Tgr___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T173850: Possible WMF deployed extension PHP 7 issues

2017-08-30 Thread Ottomata

Ottomata added a comment.
I fixed the two warnings from the EventLogging extension, but I have not been able to test them.  If anyone knows how, I would be much obliged.

https://gerrit.wikimedia.org/r/#/c/374816/TASK DETAILhttps://phabricator.wikimedia.org/T173850EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Ottomata, gerritbot, Mattflaschen-WMF, Liuxinyu970226, WMDE-leszek, Anomie, Aklapper, Jdforrester-WMF, Addshore, Krinkle, MaxSem, Krenair, Reedy, Minilexikon, Timoavax79, Lordiis, GoranSMilovanovic, TheDragonFire, Maosef, Adik2382, 45Jayjay1969, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, Orienteerix, EnricoCNC, Litanwei, Avner, Lewizho99, Ddproxy, Maathavan, debt, TerraCodes, SundanceRaphael, Reasno, Gehel, Debenben, Jseddon, JGirault, gabriel-wmde, phabyogi, FloNight, DStrine, Dixtosa, Trizek-WMF, GAllegre, Susannaanas, XenoRyet, Thibaut120094, Izno, SBisson, Matiia, ferdbold, lxbarth, Wong128hk, Luke081515, Nirmos, Planemad, Wikidata-bugs, Snowolf, Prod, Hydriz, aude, fredw, SPQRobin, AndyRussG, Ebe123, Pcoombe, Huji, Physikerwelt, Pkra, Dinoguy1000, Gryllida, jeblad, jayvdb, Ricordisamoa, MrStradivarius, Catrope, Lydia_Pintscher, Yurik, Darkdadaah, Arrbee, scfc, KartikMistry, Se4598, Jackmcbarn, Mbch331, Jay8g, Ltrlg, Joe, jeremyb, Legoktm, Tgr___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T161731: Create reliable change stream for specific wiki

2017-12-04 Thread Ottomata

Ottomata added a comment.
Sure, I suppose!  You can connect to it with a Kafka client now.  The Kafka brokers are kafka-jumbo100[1-6].eqiad.wmnet:9092

I think you are most interested in the eqiad.mediawiki.revision-create topic.  I haven't tried yet at all, but these topics should have a broker received timestamp index on them.  I am not yet familiar with the Kafka client APIs that know how to consume from timestamps, but they should be out there.TASK DETAILhttps://phabricator.wikimedia.org/T161731EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Ladsgroup, Nuria, Anomie, Aklapper, Smalyshev, Lahi, Gq86, Vacio, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, Avner, Gehel, Jonas, FloNight, Xmlizer, Nirmos, JAllemandou, mobrovac, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, Krenair, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T161731: Create reliable change stream for specific wiki

2017-12-08 Thread Ottomata

Ottomata added a comment.
Seems like the mirroring is done by 0.9 MirrorMaker and timestamp handling was added only in 0.10 MirrorMaker.

Hm, ya but I had thought that if a timestamp was not set by the producer, it would be set to server receive time.  Maybe I was wrong!TASK DETAILhttps://phabricator.wikimedia.org/T161731EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: JAllemandou, Pchelolo, Ladsgroup, Nuria, Anomie, Aklapper, Smalyshev, Lahi, Gq86, Vacio, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, Avner, Gehel, Jonas, FloNight, Xmlizer, Nirmos, mobrovac, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, Krenair, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T161731: Create reliable change stream for specific wiki

2017-12-08 Thread Ottomata

Ottomata added a comment.
Woot, that did it ^.  We need topics to default to LogAppendTime.

[@stat1005:/home/otto] $ ./kafkacat -Q -b kafka-jumbo1001.eqiad.wmnet:9092 -t eqiad.mediawiki.revision-create:0:151275919
eqiad.mediawiki.revision-create [0] offset 3658631TASK DETAILhttps://phabricator.wikimedia.org/T161731EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: gerritbot, JAllemandou, Pchelolo, Ladsgroup, Nuria, Anomie, Aklapper, Smalyshev, Lahi, Gq86, Baloch007, Vacio, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, Avner, Lewizho99, Maathavan, Gehel, Jonas, FloNight, Xmlizer, Nirmos, mobrovac, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, Krenair, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T161731: Create reliable change stream for specific wiki

2017-12-08 Thread Ottomata

Ottomata added a comment.
So, FYI, the timestamps as they are now are the timestamp that the kafka jumbo-eqiad cluster received the messages.  These are replicated from the main-eqiad cluster, and might have a short (seconds usually, minutes max) delay.

Eventually (work not planned yet) we will upgrade the other Kafka clusters, and also configure producers to set timestamps based content time, e.g. revision create timestamp, instead.TASK DETAILhttps://phabricator.wikimedia.org/T161731EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: gerritbot, JAllemandou, Pchelolo, Ladsgroup, Nuria, Anomie, Aklapper, Smalyshev, Lahi, Gq86, Baloch007, Vacio, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, Avner, Lewizho99, Maathavan, Gehel, Jonas, FloNight, Xmlizer, Nirmos, mobrovac, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, Krenair, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T161731: Create reliable change stream for specific wiki

2017-12-04 Thread Ottomata

Ottomata added a subtask: T175461: Port Clients to new jumbo cluster.
TASK DETAILhttps://phabricator.wikimedia.org/T161731EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Ladsgroup, Nuria, Anomie, Aklapper, Smalyshev, Lahi, Gq86, Vacio, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, Avner, debt, Gehel, Jonas, FloNight, Xmlizer, Nirmos, JAllemandou, mobrovac, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, Krenair, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T161731: Create reliable change stream for specific wiki

2017-12-04 Thread Ottomata

Ottomata added a comment.
Unfortunately not yet!  We are very close...the cluster is up and running, but porting clients has been blocked on getting proper keys and certificates for SSL support for a long time now.  SSL is finally moving now, so we should be able to start porting clients over soon.  We have a goal of getting some of the misc varnish traffic ported to the new cluster this quarter, and will likely (not totally sure) make a goal of getting all remaining clients ported next quarter.

Sorry this is taking so long!  Lots of moving parts...TASK DETAILhttps://phabricator.wikimedia.org/T161731EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Ladsgroup, Nuria, Anomie, Aklapper, Smalyshev, Lahi, Gq86, Vacio, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, Avner, debt, Gehel, Jonas, FloNight, Xmlizer, Nirmos, JAllemandou, mobrovac, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, Krenair, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T161731: Create reliable change stream for specific wiki

2017-12-06 Thread Ottomata

Ottomata added a comment.
You can easily 'quickbuild' kafkacat with a statically linked librdkafka.  I've just done this on a stretch labs host, and copied the kafkacat binary to stat1005 at /home/otto/kafkacat.  Try it out!TASK DETAILhttps://phabricator.wikimedia.org/T161731EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Pchelolo, Ladsgroup, Nuria, Anomie, Aklapper, Smalyshev, Lahi, Gq86, Vacio, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, Avner, Gehel, Jonas, FloNight, Xmlizer, Nirmos, JAllemandou, mobrovac, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, Krenair, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T187296: Increase kafka event retention to 14 or 21 days

Ottomata added a comment.
mediawiki eventbus topics should now be retained for 31 days in main Kafka clusters.  If we add a new mediawiki topic, we need to remember to run this command for it.TASK DETAILhttps://phabricator.wikimedia.org/T187296EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: mforns, elukey, Ottomata, Aklapper, Nuria, Ladsgroup, Pchelolo, JAllemandou, Smalyshev, Lahi, Gq86, Vacio, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, Gehel, Jonas, FloNight, Xmlizer, Nirmos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Krenair, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T187296: Increase kafka event retention to 14 or 21 days

Ottomata added a project: Analytics-Kanban.
TASK DETAILhttps://phabricator.wikimedia.org/T187296EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: mforns, elukey, Ottomata, Aklapper, Nuria, Ladsgroup, Pchelolo, JAllemandou, Smalyshev, Lahi, Gq86, Vacio, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, Gehel, Jonas, FloNight, Xmlizer, Nirmos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Krenair, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T187296: Increase kafka event retention to 14 or 21 days

Ottomata added a comment.
Doing the following for all main-eqiad and main-codfw:

for t in \
mediawiki.page-create\
mediawiki.page-delete\
mediawiki.page-edit  \
mediawiki.page-move  \
mediawiki.page-properties-change \
mediawiki.page-restrictions-change   \
mediawiki.page-undelete  \
mediawiki.page_delete\
mediawiki.page_move  \
mediawiki.page_restore   \
mediawiki.recentchange   \
mediawiki.resource-change\
mediawiki.resource_change\
mediawiki.resourcechange \
mediawiki.revision-create\
mediawiki.revision-score \
mediawiki.revision-visibility-change \
mediawiki.revision_create\
mediawiki.revision_visibility_set\
mediawiki.user-blocks-change \
mediawiki.user_block ; do
for dc in eqiad codfw; do
topic="${dc}.${t}"
echo kafka configs --alter --entity-type topics --entity-name $topic --add-config retention.ms=267840
done
done



kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.page-create --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.page-create --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.page-delete --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.page-delete --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.page-edit --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.page-edit --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.page-move --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.page-move --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.page-properties-change --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.page-properties-change --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.page-restrictions-change --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.page-restrictions-change --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.page-undelete --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.page-undelete --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.page_delete --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.page_delete --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.page_move --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.page_move --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.page_restore --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.page_restore --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.recentchange --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.recentchange --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.resource-change --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.resource-change --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.resource_change --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.resource_change --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.resourcechange --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.resourcechange --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.revision-create --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.revision-create --add-config re

[Wikidata-bugs] [Maniphest] [Retitled] T187296: Increase kafka event retention to 31

Ottomata renamed this task from "Increase kafka event retention to 14 or 21 days" to "Increase kafka event retention to 31".Ottomata claimed this task.
TASK DETAILhttps://phabricator.wikimedia.org/T187296EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: mforns, elukey, Ottomata, Aklapper, Nuria, Ladsgroup, Pchelolo, JAllemandou, Smalyshev, Lahi, Gq86, Vacio, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, Gehel, Jonas, FloNight, Xmlizer, Nirmos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Krenair, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T187296: Increase kafka event retention to 14 or 21 days

Ottomata set the point value for this task to "2".
TASK DETAILhttps://phabricator.wikimedia.org/T187296EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: mforns, elukey, Ottomata, Aklapper, Nuria, Ladsgroup, Pchelolo, JAllemandou, Smalyshev, Lahi, Gq86, Vacio, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, Gehel, Jonas, FloNight, Xmlizer, Nirmos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Krenair, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T187296: Increase kafka event retention to 14 or 21 days

Ottomata merged a task: T196409: Consider increasing retention for mediawiki event topics.
TASK DETAILhttps://phabricator.wikimedia.org/T187296EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: mforns, elukey, Ottomata, Aklapper, Nuria, Ladsgroup, Pchelolo, JAllemandou, Smalyshev, Lahi, Gq86, Vacio, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, Gehel, Jonas, FloNight, Xmlizer, Nirmos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Krenair, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T187296: Increase kafka event retention to 14 or 21 days