Ottomata created this task. Ottomata added subscribers: Aklapper, Ottomata, Eevans, Halfak, mobrovac, GWicke, aaron, faidon, ori, Nuria, madhuvishy. Ottomata added projects: EventBus, Discovery, Epic, Analytics, Wikidata, operations, MediaWiki-General-or-Unknown, Services, Service-Architecture, Wikidata-Query-Service. Herald added a subscriber: Matanya.
TASK DESCRIPTION Over in T88459, and in a few recent meetings, we've flushed out a sketch for to get standardized messages into Kafka for later consumption. We've coalesced on a way to move forward, and a MVP. This task will track the creation of the EventBus MVP. == Architecture Decisions - We will standardize on JSON Schema as our canonical schema spec, but do so in such a way that Avro can be used in Analytics type systems. Equivalent Avro Schemas may be generated as part of CI. - For MVP, JSON data will be produced to Kafka. We consider Avro Binary later. - There will be a Kafka Topic -> Schema mapping, and only that schema can be produced to a topic. == MVP Description The MVP will consist of: - REST Service that validates JSON data against a schema and produces to Kafka. - Schema Repository Layout and Topic -> Schema mapping config that Service loads on startup. - A TBD implemented use case of this system. The MVP might also include: - Schema review and CI processes: -- schema evolution rules -- Auto Avro schema generation -- Auto Avro java class generation - Schema lookup service - Schema listing and discussion UI (Mediwiki Extension?) - Schema metadata conventions (fields common to all schemas?) TASK DETAIL https://phabricator.wikimedia.org/T114443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: madhuvishy, Nuria, ori, faidon, aaron, GWicke, mobrovac, Halfak, Eevans, Ottomata, Matanya, Aklapper, JAllemandou, jkroll, Smalyshev, Hardikj, Wikidata-bugs, Jdouglas, RobH, aude, Deskana, Manybubbles, mark, JanZerebecki, RobLa-WMF, bd808, fgiunchedi, Dzahn, jeremyb, chasemp, Krenair _______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
