On Wed, Feb 24, 2016 at 11:16 PM, Chad Horohoe <[email protected]> wrote:
> You may also find these diagrams useful: > > > https://commons.wikimedia.org/wiki/File:Wikipedia_webrequest_flow_2015-10.png > I don't know the source of this image, but there are a couple corrections that could be made. EventLogging comes from either the web or the MediaWiki instances. I believe there was a recent change so MediaWiki instances produce EventLogging data by hitting an http endpoint much like the web based EventLogging does. The EventLogging data then flows into Kafka. From there it goes somewhere (not sure what happens here) and ends up in the analytics DB servers, along with HDFS. I may be missing a few steps, and this may be partially incorrect. But it's my current understanding. Additionally CirrusSearch logs flow from MediaWiki into kafaka. Very soon API logs will also start flowing from MediaWiki into Kafka. Yet another use case is that EventBus will soon (or already does?) ship events about edit's from MediaWiki into Kafka. Initially those EventBus events will be read out of kafka by RestBase. > https://wikitech.wikimedia.org/wiki/File:Infrastructure_overview.png > > -Chad > On Feb 24, 2016 6:58 PM, "Mukunda Modell" <[email protected]> wrote: > >> >> On Feb 17, 2016 1:50 AM, "Guillaume Lederrey" <[email protected] >> > >> >> wrote: >> >>> * I still have not found a global architecture schema (something like >> >>> a high level component or deplyoment diagram). But I have never seen >> >>> any company having those... >> >> I made a diagram of the scap (mediawiki) deployment architecture a while >> back: https://commons.wikimedia.org/wiki/File:Scap-diagram.png .. >> >> That does not exactly apply to the new scap3 architecture but it's not >> too far off. >> >> .... >> >> On Thu, Feb 18, 2016 at 10:37 AM, Giuseppe Lavagetto >> <[email protected]> wrote: >> > >> > About cherry-picks in beta: the problem is not cherry-picking (I think >> > it's a reasonable way to test things) but persistent cherry-picking to >> > monkey patch problems is. I think if we follow the flow of: >> > >> > - writing a patch >> > - testing it on beta with a cherry-pick >> > - get it merged on ops/puppet and production >> >> There are a lot of patches on beta these days and there have been a lot >> of different people cherry-picking without much coordination. This has >> lead to breakage quite often. Patches also get lost regularly. I assume >> this usually happens because someone has rebased the HEAD and accidentally >> dropped a patch. >> >> It can be really difficult to get a patch merged in ops/puppet within a >> week (or even a month). I've seen a lot of patches sit around for weeks and >> even now with the Puppet SWAT windows, it's still sometimes unrealistic to >> expect patches get merged into production that quickly. (+CC Tyler) >> >> Without a system to manage things, and with very little coordination >> between everyone who is working on beta, I don't expect the situation to >> improve too much. >> >> I intend to propose a solution for beta & puppet patch cherry-picks very >> soon, however, I haven't fully formulated my proposal yet. I will write to >> the ops list when I have something written in a clear and presentable way. >> >> _______________________________________________ >> Ops mailing list >> [email protected] >> https://lists.wikimedia.org/mailman/listinfo/ops >> >> > _______________________________________________ > discovery mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/discovery > >
_______________________________________________ discovery mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/discovery
