Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Nuria Ruiz]

> In general I'm in favor of more ad-hoc project-specific teams rather than
completely siloing every service to the Services group, or every mobile UI
to the Mobile group.=
Agreed, as long as everyone deploying services communicates through
services team so there is no duplication of solutions.
We should have 1 routing layer, standard monitoring and standard caching as
it is been pointed out before. Specifics of services might be different but
it seems the core mission of service team to provide common infrastructure,
right?.
That means that the 1st service we develop will be more work for services
team than subsequent services. *Ideally* the mobile team should develop the
service without having to solve (for example) any caching problems that are
not mobile specific.





On Wed, Feb 4, 2015 at 3:04 PM, James Douglas 
wrote:

> > In general I'm in favor of more ad-hoc project-specific teams rather than
> completely siloing every service to the Services group, or every mobile UI
> to the Mobile group.
>
> I strongly agree.  Based on experience on both sides of this spectrum, I
> recommend (when feasible) favoring feature teams over functional teams.
>
> On Wed, Feb 4, 2015 at 3:00 PM, Brion Vibber 
> wrote:
>
> > I think the way we'd want to go is roughly to have a *partnership
> between*
> > the Services and Mobile teams produce and maintain the service.
> >
> > (Note that the state of the art is that Mobile Apps are using Mobile
> Web's
> > MobileFrontend extension as an intermediate API to aggregate & format
> page
> > data -- which basically means Max has done the server-side API work for
> > Mobile Apps so far.)
> >
> > I'd expect to see Max and/or someone else from the Mobile team
> > collaborating with the Services team to create what y'all need:
> > 1) something that does what Mobile Apps needs it to...
> > 2) and can be maintained like Services needs it to.
> >
> > In general I'm in favor of more ad-hoc project-specific teams rather than
> > completely siloing every service to the Services group, or every mobile
> UI
> > to the Mobile group.
> >
> > -- brion
> >
> > On Wed, Feb 4, 2015 at 2:29 PM, Corey Floyd 
> wrote:
> >
> > > On Wed, Feb 4, 2015 at 11:41 AM, Gabriel Wicke 
> > > wrote:
> > >
> > > > Regarding general-purpose APIs vs. mobile: I think mobile is in some
> > > ways a
> > > > special case as their content transformation needs are closely
> coupled
> > > with
> > > > the way the apps are presenting the content. Additionally, at least
> > until
> > > > SPDY is deployed there is a strong performance incentive to bundle
> > > > information in a single response tailored to the app's needs. One
> > > strategy
> > > > employed by Netflix is to introduce a second API layer
> > > > <
> > > >
> > >
> >
> http://techblog.netflix.com/2012/07/embracing-differences-inside-netflix.html
> > > > >
> > > > on
> > > > top of the general content API to handle device-specific needs. I
> think
> > > > this is a sound strategy, as it contains the volatility in a separate
> > > layer
> > > > while ensuring that everything is ultimately consuming the
> > > general-purpose
> > > > API. If the need for app-specific massaging disappears over time, we
> > can
> > > > simply shut down the custom service / API end point without affecting
> > the
> > > > general API.
> > > >
> > >
> > >
> > > I can definitely understand that motivation for providing mobile
> specific
> > > service layer - so if the services team wants to implement the API in
> > this
> > > way and support that architecture, I am totally on board.
> > >
> > > My remaining hesitation here is that from the reading of this proposal,
> > the
> > > mobile team is the owner of implementing this service, not the services
> > > team (Maybe I am misreading?).
> > >
> > > This leads me to ask questions like:
> > > Why is the mobile apps team investigating which is the best server side
> > > technology? That seems outside of our domain knowledge.
> > > Who will be responsible for maintaining this code?
> > > Who will be testing it to make sure that is performant?
> > >
> > > I'm new, so maybe these answers are obvious to others, but to me they
> > seem
> > > fuzzy when responsibilities are divided between two teams.
> > >
> > > I would propose that this be a project that the Services Team owns. And
> > > that the Mobile Apps Team defines specs on what they need the new
> service
> > > to provide.
> > > ___
> > > Wikitech-l mailing list
> > > Wikitech-l@lists.wikimedia.org
> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > >
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Giuseppe Lavagetto]

On Wed, Feb 4, 2015 at 6:59 AM, Marko Obrovac  wrote:
> On Tue, Feb 3, 2015 at 8:42 PM, Tim Starling 
> wrote:
>
>> I don't really understand why you want it to be integrated with
>> RESTBase. As far as I can tell (it is hard to pin these things down),
>> RESTBase is a revision storage backend and possibly a public API for
>> that backend.
>
>
> Actually, RESTBase's logic applies to the Mobile Apps case quite naturally.
> When a page is fetched and transformed, it can be stored so that consequent
> requests can simply retrieve the transformed document form storage.
>
>

Ok, so in this vision RESTBase is a caching layer for revisions

>> I thought the idea of SOA was to separate concerns.
>> Wouldn't monitoring, caching and authorization would be best done as a
>> node.js library which RESTBase and other services use?
>>
>
> Good point. Ideally, what we would need to do is provide the right tools to
> developers to create services, which can then be placed "strategically"
> around DCs (in cooperation with Ops, ofc). For v1, however, we plan to
> provide only logical separation (to a certain extent) via modules which can
> be dynamically loaded/unloaded from RESTBase. In return, RESTBase will
> provide them with routing, monitoring, caching and authorisation out of the
> box. The good point here is that this 'modularisation' eases the transition
> to a more-decomposed orchestration SOA model. Going in that direction,
> however, requires some prerequisites to be fulfilled, such as [1].
>

So, now RESTBase has become a router and an auth provider as well?
(Gabriel already clarified me that "monitoring" means that RESTbase
will expose its own metrics for that specific service, so this is not
monitoring of the service at all, rather accounting).

I need some clarification at this point - what is RESTBase really
going to do? I'm asking because when I read "RESTBase will provide
them with routing, [...] and authorisation" I immediately think of a
request router and a general on-wiki auth provider. And we already
have both, and re-doing them in RESTBase would be plainly wrong.

Maybe you intend very specific things when you say "routing" and not
request routing, which is what everybody here will think of.
And when you say "auth" you mean that RESTBase implements an auth
schema for its clients, so that no client can access data from another
one. If this is the case, I have some further questions: is this going
to be RBAC? Which permissions models are you implementing? Are we sure
it is what we will need? And foremost: will this be exposable to
external consumers? will it be able to hook up to our traditional wiki
auth scheme?

Can you please expand a bit on those concepts? Or a lot of confusion,
uncertainty and doubt will spread amongst your fellow engineers,
resulting in an hostile view of what may be a perfectly well designed
software.

Cheers,

Giuseppe

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Giuseppe Lavagetto]

On Wed, Feb 4, 2015 at 6:59 AM, Marko Obrovac  wrote:
> On Tue, Feb 3, 2015 at 8:42 PM, Tim Starling 
> wrote:
>
>> I don't really understand why you want it to be integrated with
>> RESTBase. As far as I can tell (it is hard to pin these things down),
>> RESTBase is a revision storage backend and possibly a public API for
>> that backend.
>
>
> Actually, RESTBase's logic applies to the Mobile Apps case quite naturally.
> When a page is fetched and transformed, it can be stored so that consequent
> requests can simply retrieve the transformed document form storage.
>
>

Ok, so in this vision RESTBase is a caching layer for revisions

>> I thought the idea of SOA was to separate concerns.
>> Wouldn't monitoring, caching and authorization would be best done as a
>> node.js library which RESTBase and other services use?
>>
>
> Good point. Ideally, what we would need to do is provide the right tools to
> developers to create services, which can then be placed "strategically"
> around DCs (in cooperation with Ops, ofc). For v1, however, we plan to
> provide only logical separation (to a certain extent) via modules which can
> be dynamically loaded/unloaded from RESTBase. In return, RESTBase will
> provide them with routing, monitoring, caching and authorisation out of the
> box. The good point here is that this 'modularisation' eases the transition
> to a more-decomposed orchestration SOA model. Going in that direction,
> however, requires some prerequisites to be fulfilled, such as [1].
>

So, now RESTBase has become a router and an auth provider as well?
(Gabriel already clarified me that "monitoring" means that RESTbase
will expose its own metrics for that specific service, so this is not
monitoring of the service at all, rather accounting).

I need some clarification at this point - what is RESTBase really
going to do? I'm asking because when I read "RESTBase will provide
them with routing, [...] and authorisation" I immediately think of a
request router and a general on-wiki auth provider. And we already
have both, and re-doing them in RESTBase would be plainly wrong.

Maybe you intend very specific things when you say "routing" and not
request routing, which is what everybody here will think of.
And when you say "auth" you mean that RESTBase implements an auth
schema for its clients, so that no client can access data from another
one. If this is the case, I have some further questions: is this going
to be RBAC? Which permissions models are you implementing? Are we sure
it is what we will need? And foremost: will this be exposable to
external consumers? will it be able to hook up to our traditional wiki
auth scheme?

Can you please expand a bit on those concepts? Or a lot of confusion,
uncertainty and doubt will spread amongst your fellow engineers,
resulting in an hostile view of what may be a perfectly well designed
software.

Cheers,

Giuseppe

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Bryan Davis]

On Thu, Feb 5, 2015 at 9:00 PM, Yuri Astrakhan  wrote:
> Lastly, semi related - my service will also need a node.js lib to access MW
> api. How should we manage common libs like that?

Libraries published on npm using semantic versioning and imported to
the individual services as needed.

Bryan
-- 
Bryan Davis  Wikimedia Foundation
[[m:User:BDavis_(WMF)]]  Sr Software EngineerBoise, ID USA
irc: bd808v:415.839.6885 x6855

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Yuri Astrakhan]

Tim, I like Varnish's vcl flexibility, but not the debugging aspects.
Still, +1, but could you elaborate on how you see:

* Services communicate with each other - via varnish as well or directly?
* Do you see routing varnish layer as non-caching, only to forward request
to the second tear service-specific varnish instance that will actually
perform caching?
* Will there be any problems separating services into separate, non
mutually trusted clusters?
* service management - varnish would only indicate overall health of the
service. What about service-specific real time monitored values (if
needed), logs, controlling service instances. Any thoughts on generalizing
that infrastructure?

Lastly, semi related - my service will also need a node.js lib to access MW
api. How should we manage common libs like that?

Thanks!
On Feb 6, 2015 4:15 AM, "Tim Starling"  wrote:

> On 04/02/15 16:59, Marko Obrovac wrote:
> > For v1, however, we plan to
> > provide only logical separation (to a certain extent) via modules which
> can
> > be dynamically loaded/unloaded from RESTBase. In return, RESTBase will
> > provide them with routing, monitoring, caching and authorisation out of
> the
> > box.
>
> We already have routing, monitoring and caching in Varnish. That seems
> like a good place to implement further service routing to me.
>
> <
> https://git.wikimedia.org/blob/operations%2Fpuppet/4a7f5ce62d9cdd1ace20ca6c489cbdb538503750/templates%2Fvarnish%2Fmisc.inc.vcl.erb
> >
>
> It's simple to configure, has excellent performance and scalability,
> and monitoring and a distributed logging system are already implemented.
>
> It doesn't have authorisation, but I thought that was going to be in a
> separate service from RESTBase anyway.
>
> -- Tim Starling
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Tim Starling]

On 04/02/15 16:59, Marko Obrovac wrote:
> For v1, however, we plan to
> provide only logical separation (to a certain extent) via modules which can
> be dynamically loaded/unloaded from RESTBase. In return, RESTBase will
> provide them with routing, monitoring, caching and authorisation out of the
> box.

We already have routing, monitoring and caching in Varnish. That seems
like a good place to implement further service routing to me.



It's simple to configure, has excellent performance and scalability,
and monitoring and a distributed logging system are already implemented.

It doesn't have authorisation, but I thought that was going to be in a
separate service from RESTBase anyway.

-- Tim Starling


___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Dan Garry]

On 4 February 2015 at 15:00, Brion Vibber  wrote:
>
> In general I'm in favor of more ad-hoc project-specific teams rather than
> completely siloing every service to the Services group, or every mobile UI
> to the Mobile group.
>

Agreed. This also ensures that the service exactly meets the functional
requirements, no more and no less.

Dan

-- 
Dan Garry
Associate Product Manager, Mobile Apps
Wikimedia Foundation
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Bryan Davis]

On Wed, Feb 4, 2015 at 4:00 PM, Brion Vibber  wrote:
> I think the way we'd want to go is roughly to have a *partnership between*
> the Services and Mobile teams produce and maintain the service.
>
> (Note that the state of the art is that Mobile Apps are using Mobile Web's
> MobileFrontend extension as an intermediate API to aggregate & format page
> data -- which basically means Max has done the server-side API work for
> Mobile Apps so far.)
>
> I'd expect to see Max and/or someone else from the Mobile team
> collaborating with the Services team to create what y'all need:
> 1) something that does what Mobile Apps needs it to...
> 2) and can be maintained like Services needs it to.
>
> In general I'm in favor of more ad-hoc project-specific teams rather than
> completely siloing every service to the Services group, or every mobile UI
> to the Mobile group.

+1. This is the only thing that will scale in my opinion. "Full stack"
teams involving design, front end, back end, ops, release, project
management, and testing resources should be formed to work on vertical
slices of functionality ("features" or "products") that are
prioritized by the entire organization. Thinking that some team can be
called on to fulfill all of the cross-feature/product needs is
madness. Services is 3 people, MediaWiki-Core is 9 people (minus
standing obligations like security and performance reviews). Teams of
this size cannot be expected to service all the "backend" needs of the
myriad product/feature verticals that are under the WMF umbrella. If
we don't have enough people to staff projects this way we are trying
to do too many things at once. (Which I'm pretty sure is actually the
case.)

Bryan
-- 
Bryan Davis  Wikimedia Foundation
[[m:User:BDavis_(WMF)]]  Sr Software EngineerBoise, ID USA
irc: bd808v:415.839.6885 x6855

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[James Douglas]

> In general I'm in favor of more ad-hoc project-specific teams rather than
completely siloing every service to the Services group, or every mobile UI
to the Mobile group.

I strongly agree.  Based on experience on both sides of this spectrum, I
recommend (when feasible) favoring feature teams over functional teams.

On Wed, Feb 4, 2015 at 3:00 PM, Brion Vibber  wrote:

> I think the way we'd want to go is roughly to have a *partnership between*
> the Services and Mobile teams produce and maintain the service.
>
> (Note that the state of the art is that Mobile Apps are using Mobile Web's
> MobileFrontend extension as an intermediate API to aggregate & format page
> data -- which basically means Max has done the server-side API work for
> Mobile Apps so far.)
>
> I'd expect to see Max and/or someone else from the Mobile team
> collaborating with the Services team to create what y'all need:
> 1) something that does what Mobile Apps needs it to...
> 2) and can be maintained like Services needs it to.
>
> In general I'm in favor of more ad-hoc project-specific teams rather than
> completely siloing every service to the Services group, or every mobile UI
> to the Mobile group.
>
> -- brion
>
> On Wed, Feb 4, 2015 at 2:29 PM, Corey Floyd  wrote:
>
> > On Wed, Feb 4, 2015 at 11:41 AM, Gabriel Wicke 
> > wrote:
> >
> > > Regarding general-purpose APIs vs. mobile: I think mobile is in some
> > ways a
> > > special case as their content transformation needs are closely coupled
> > with
> > > the way the apps are presenting the content. Additionally, at least
> until
> > > SPDY is deployed there is a strong performance incentive to bundle
> > > information in a single response tailored to the app's needs. One
> > strategy
> > > employed by Netflix is to introduce a second API layer
> > > <
> > >
> >
> http://techblog.netflix.com/2012/07/embracing-differences-inside-netflix.html
> > > >
> > > on
> > > top of the general content API to handle device-specific needs. I think
> > > this is a sound strategy, as it contains the volatility in a separate
> > layer
> > > while ensuring that everything is ultimately consuming the
> > general-purpose
> > > API. If the need for app-specific massaging disappears over time, we
> can
> > > simply shut down the custom service / API end point without affecting
> the
> > > general API.
> > >
> >
> >
> > I can definitely understand that motivation for providing mobile specific
> > service layer - so if the services team wants to implement the API in
> this
> > way and support that architecture, I am totally on board.
> >
> > My remaining hesitation here is that from the reading of this proposal,
> the
> > mobile team is the owner of implementing this service, not the services
> > team (Maybe I am misreading?).
> >
> > This leads me to ask questions like:
> > Why is the mobile apps team investigating which is the best server side
> > technology? That seems outside of our domain knowledge.
> > Who will be responsible for maintaining this code?
> > Who will be testing it to make sure that is performant?
> >
> > I'm new, so maybe these answers are obvious to others, but to me they
> seem
> > fuzzy when responsibilities are divided between two teams.
> >
> > I would propose that this be a project that the Services Team owns. And
> > that the Mobile Apps Team defines specs on what they need the new service
> > to provide.
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Brion Vibber]

I think the way we'd want to go is roughly to have a *partnership between*
the Services and Mobile teams produce and maintain the service.

(Note that the state of the art is that Mobile Apps are using Mobile Web's
MobileFrontend extension as an intermediate API to aggregate & format page
data -- which basically means Max has done the server-side API work for
Mobile Apps so far.)

I'd expect to see Max and/or someone else from the Mobile team
collaborating with the Services team to create what y'all need:
1) something that does what Mobile Apps needs it to...
2) and can be maintained like Services needs it to.

In general I'm in favor of more ad-hoc project-specific teams rather than
completely siloing every service to the Services group, or every mobile UI
to the Mobile group.

-- brion

On Wed, Feb 4, 2015 at 2:29 PM, Corey Floyd  wrote:

> On Wed, Feb 4, 2015 at 11:41 AM, Gabriel Wicke 
> wrote:
>
> > Regarding general-purpose APIs vs. mobile: I think mobile is in some
> ways a
> > special case as their content transformation needs are closely coupled
> with
> > the way the apps are presenting the content. Additionally, at least until
> > SPDY is deployed there is a strong performance incentive to bundle
> > information in a single response tailored to the app's needs. One
> strategy
> > employed by Netflix is to introduce a second API layer
> > <
> >
> http://techblog.netflix.com/2012/07/embracing-differences-inside-netflix.html
> > >
> > on
> > top of the general content API to handle device-specific needs. I think
> > this is a sound strategy, as it contains the volatility in a separate
> layer
> > while ensuring that everything is ultimately consuming the
> general-purpose
> > API. If the need for app-specific massaging disappears over time, we can
> > simply shut down the custom service / API end point without affecting the
> > general API.
> >
>
>
> I can definitely understand that motivation for providing mobile specific
> service layer - so if the services team wants to implement the API in this
> way and support that architecture, I am totally on board.
>
> My remaining hesitation here is that from the reading of this proposal, the
> mobile team is the owner of implementing this service, not the services
> team (Maybe I am misreading?).
>
> This leads me to ask questions like:
> Why is the mobile apps team investigating which is the best server side
> technology? That seems outside of our domain knowledge.
> Who will be responsible for maintaining this code?
> Who will be testing it to make sure that is performant?
>
> I'm new, so maybe these answers are obvious to others, but to me they seem
> fuzzy when responsibilities are divided between two teams.
>
> I would propose that this be a project that the Services Team owns. And
> that the Mobile Apps Team defines specs on what they need the new service
> to provide.
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Corey Floyd]

On Wed, Feb 4, 2015 at 11:41 AM, Gabriel Wicke  wrote:

> Regarding general-purpose APIs vs. mobile: I think mobile is in some ways a
> special case as their content transformation needs are closely coupled with
> the way the apps are presenting the content. Additionally, at least until
> SPDY is deployed there is a strong performance incentive to bundle
> information in a single response tailored to the app's needs. One strategy
> employed by Netflix is to introduce a second API layer
> <
> http://techblog.netflix.com/2012/07/embracing-differences-inside-netflix.html
> >
> on
> top of the general content API to handle device-specific needs. I think
> this is a sound strategy, as it contains the volatility in a separate layer
> while ensuring that everything is ultimately consuming the general-purpose
> API. If the need for app-specific massaging disappears over time, we can
> simply shut down the custom service / API end point without affecting the
> general API.
>


I can definitely understand that motivation for providing mobile specific
service layer - so if the services team wants to implement the API in this
way and support that architecture, I am totally on board.

My remaining hesitation here is that from the reading of this proposal, the
mobile team is the owner of implementing this service, not the services
team (Maybe I am misreading?).

This leads me to ask questions like:
Why is the mobile apps team investigating which is the best server side
technology? That seems outside of our domain knowledge.
Who will be responsible for maintaining this code?
Who will be testing it to make sure that is performant?

I'm new, so maybe these answers are obvious to others, but to me they seem
fuzzy when responsibilities are divided between two teams.

I would propose that this be a project that the Services Team owns. And
that the Mobile Apps Team defines specs on what they need the new service
to provide.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Erik Moeller]

On Wed, Feb 4, 2015 at 8:41 AM, Gabriel Wicke  wrote:

> Regarding general-purpose APIs vs. mobile: I think mobile is in some ways a
> special case as their content transformation needs are closely coupled with
> the way the apps are presenting the content. Additionally, at least until
> SPDY is deployed there is a strong performance incentive to bundle
> information in a single response tailored to the app's needs.

A notion of schemas that declare a specific set of transformations to
be applied/not applied might help avoid overcomplicating things early
on while addressing different transformation needs even within the
growing number of mobile use cases (Android app alpha/beta/stable, iOS
app alpha/beta/stable, mobile web alpha/beta/stable, third party
apps), and potentially making code re-usable for desktop needs down
the road. Since the number of schemas would be limited, and specifying
the correct schema would result in a single response, performance
could be optimized for each use case.

Erik
-- 
Erik Möller
VP of Product & Strategy, Wikimedia Foundation

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Dan Garry]

On 4 February 2015 at 08:40, Bryan Davis  wrote:

> +1 This sort of major design change is exactly the sort of thing that
> I think the RfC process is good at helping with. Start with a straw
> man proposal, get feedback from other engineers and iterate before
> investing in code changes. The sometime frustrating part is that
> feedback doesn't always come as fast as Product and/or the team would
> like but we can try to accelerate that by promoting the topic more
> often.
>

Our plan is to have a spike to experiment to determine whether there are
any early roadblocks in the proposed solution. We're not going to consider
commiting to the RESTBase/Node.js path until after that. It seems quite
reasonable to me to also have an RfC alongside our experimentation to try
to think up alternative solutions, and invest in experimenting with those
solutions too, because we're definitely open to anything that helps us move
forwards at this stage.

We'll start writing up an RfC and see where it takes us.

Dan

-- 
Dan Garry
Associate Product Manager, Mobile Apps
Wikimedia Foundation
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Chris McMahon]

> One strategy
> employed by Netflix is to introduce a second API layer
> <
> http://techblog.netflix.com/2012/07/embracing-differences-inside-netflix.html
> >
> on
> top of the general content API to handle device-specific needs. I think
> this is a sound strategy, as it contains the volatility in a separate layer
> while ensuring that everything is ultimately consuming the general-purpose
> API.


This design appears often enough that it can likely be called a "design
pattern". The Selenium/Webdriver project did exactly the same thing[1].
The API for  Selenium v2 has about 1/3 as many functions as Selenium v1.
People that use Selenium v2 build their own high-level APIs based on the
basic core set of functions available.

Defining the scope of the "general content API" can be challenging.

[1] http://w3c.github.io/webdriver/webdriver-spec.html
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Gabriel Wicke]

On Tue, Feb 3, 2015 at 11:33 PM, Erik Moeller  wrote:

> I think you will generally find agreement that moving client-side
> transformations that only live in the app to server-side code that
> enables access by multiple consumers and caching is a good idea. If
> there are reasons not do to this, now'd be a good time to speak up.
>
> If not, then I think one thing to keep in mind is how to organize the
> transformation code in a manner that it doesn't just become a
> server-side hodgepodge still only useful to one consumer, to avoid
> some of the pitfalls Brian mentions. Say you want to reformat
> infoboxes on the mobile web, but not do all the other stuff the mobile
> app does. Can you just get that specific transformation? Are some
> transformations dependent on others?  Or say we want to make a change
> only for the output that gets fed into the PDF generator, but not for
> any other outputs. Can we do that?
>


Right now the plan is to start from plain Parsoid HTML. The mobile app
service would be called for each new revision to prime the cache / storage.
Chaining transformations might be possible, but right now it's not clear
that it would be worth the complexity. Currently AFAIK only OCG and mobile
apps have strong transformation needs, and there seems to be little overlap
in the way they transform the content. Mobile web still wraps sections into
divs, but we are looking into eliminating that by possibly integrating the
section markup into the regular Parsoid output.

Regarding general-purpose APIs vs. mobile: I think mobile is in some ways a
special case as their content transformation needs are closely coupled with
the way the apps are presenting the content. Additionally, at least until
SPDY is deployed there is a strong performance incentive to bundle
information in a single response tailored to the app's needs. One strategy
employed by Netflix is to introduce a second API layer

on
top of the general content API to handle device-specific needs. I think
this is a sound strategy, as it contains the volatility in a separate layer
while ensuring that everything is ultimately consuming the general-purpose
API. If the need for app-specific massaging disappears over time, we can
simply shut down the custom service / API end point without affecting the
general API.

Gabriel
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Bryan Davis]

On Wed, Feb 4, 2015 at 8:51 AM, Brian Gerstle  wrote:
> TL;DR; this discussion is great, but I think moving to docs/wikis/etc.
> instead of continuing the thread could improve communication and give the
> people who end up working on this something to reference later. could just
> be my n00b-ness, but I thought others might share the sentiment.
>
> I'm still new here, so please excuse me for possibly going against
> convention, but does anyone else think it would be beneficial to move this
> problem & proposal into a living document (RFC, wiki, google doc,
> whatever)?  In doing so, I hope we can:
>
>1. Keep track of what the actual problems are along with the proposed
>solution(s)
>2. Group related concerns together, making them easier for those voicing
>them to be heard while also facilitating understanding and resolution
>3. Give us something concrete to go back to whenever we decide to
>dedicate resources to solving this problem, whether it's the next mobile
>apps sprint or something the mobile web team needs more urgently
>4. Prevent the points raised in the email (or the problem itself) from
>being forgotten or lost in the deluge of other emails we get every day
>
> I don't know about you, but I can't mentally juggle the multiple problems,
> implications, and the great points everyone is raising—which keeping it in
> an email forces me to do.
>
> Either way, looking forward to discussing this further and taking steps to
> solve it in the near term.

+1 This sort of major design change is exactly the sort of thing that
I think the RfC process is good at helping with. Start with a straw
man proposal, get feedback from other engineers and iterate before
investing in code changes. The sometime frustrating part is that
feedback doesn't always come as fast as Product and/or the team would
like but we can try to accelerate that by promoting the topic more
often.

Bryan
-- 
Bryan Davis  Wikimedia Foundation
[[m:User:BDavis_(WMF)]]  Sr Software EngineerBoise, ID USA
irc: bd808v:415.839.6885 x6855

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Brian Gerstle]

TL;DR; this discussion is great, but I think moving to docs/wikis/etc.
instead of continuing the thread could improve communication and give the
people who end up working on this something to reference later. could just
be my n00b-ness, but I thought others might share the sentiment.

I'm still new here, so please excuse me for possibly going against
convention, but does anyone else think it would be beneficial to move this
problem & proposal into a living document (RFC, wiki, google doc,
whatever)?  In doing so, I hope we can:

   1. Keep track of what the actual problems are along with the proposed
   solution(s)
   2. Group related concerns together, making them easier for those voicing
   them to be heard while also facilitating understanding and resolution
   3. Give us something concrete to go back to whenever we decide to
   dedicate resources to solving this problem, whether it's the next mobile
   apps sprint or something the mobile web team needs more urgently
   4. Prevent the points raised in the email (or the problem itself) from
   being forgotten or lost in the deluge of other emails we get every day

I don't know about you, but I can't mentally juggle the multiple problems,
implications, and the great points everyone is raising—which keeping it in
an email forces me to do.

Either way, looking forward to discussing this further and taking steps to
solve it in the near term.

- Brian


On Wed, Feb 4, 2015 at 10:22 AM, Brad Jorsch (Anomie)  wrote:

> On Wed, Feb 4, 2015 at 2:33 AM, Erik Moeller  wrote:
>
> > If not, then I think one thing to keep in mind is how to organize the
> > transformation code in a manner that it doesn't just become a
> > server-side hodgepodge still only useful to one consumer, to avoid
> > some of the pitfalls Brian mentions.
>
>
> I think the MobileFrontend extension has probably run into these pitfalls
> already.
>
>
> > Say you want to reformat infoboxes on the mobile web, but not do all the
> > other stuff the mobile app does. Can you just get that specific
> > transformation? Are some transformations dependent on others?  Or say we
> > want to make a change only for the output that gets fed into the PDF
> > generator, but not for any other outputs. Can we do that?
> >
>
> Maybe what we really need is a way to register transformation classes (e.g.
> something like $wgAPIModules). Then have ApiParse have a parameter to
> select transformations and apply them to wikitext before and to HTML after
> calling the parser. And we'd probably want to do the wikitext-before bit in
> ApiExpandTemplates too, and add a new action that takes HTML and applies
> only the HTML-after transforms to it.
>
> Or we could go as far as giving ParserOptions (or the ParserEnvironment I
> recently heard Tim propose) a list of transformations, to allow for
> transformations at some of the points where we have parser hooks. Although
> that would probably cause problems for Parsoid.
>
>
> --
> Brad Jorsch (Anomie)
> Software Engineer
> Wikimedia Foundation
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



-- 
EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle
IRC: bgerstle
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Brad Jorsch (Anomie)]

On Wed, Feb 4, 2015 at 2:33 AM, Erik Moeller  wrote:

> If not, then I think one thing to keep in mind is how to organize the
> transformation code in a manner that it doesn't just become a
> server-side hodgepodge still only useful to one consumer, to avoid
> some of the pitfalls Brian mentions.


I think the MobileFrontend extension has probably run into these pitfalls
already.


> Say you want to reformat infoboxes on the mobile web, but not do all the
> other stuff the mobile app does. Can you just get that specific
> transformation? Are some transformations dependent on others?  Or say we
> want to make a change only for the output that gets fed into the PDF
> generator, but not for any other outputs. Can we do that?
>

Maybe what we really need is a way to register transformation classes (e.g.
something like $wgAPIModules). Then have ApiParse have a parameter to
select transformations and apply them to wikitext before and to HTML after
calling the parser. And we'd probably want to do the wikitext-before bit in
ApiExpandTemplates too, and add a new action that takes HTML and applies
only the HTML-after transforms to it.

Or we could go as far as giving ParserOptions (or the ParserEnvironment I
recently heard Tim propose) a list of transformations, to allow for
transformations at some of the points where we have parser hooks. Although
that would probably cause problems for Parsoid.


-- 
Brad Jorsch (Anomie)
Software Engineer
Wikimedia Foundation
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Alexandros Kosiaris]

> Good point. Ideally, what we would need to do is provide the right tools to
> developers to create services, which can then be placed "strategically"
> around DCs (in cooperation with Ops, ofc).

Yes. As an organization we should provide good tools that allow
developers to create services. I do fail to understand the
"strategically" around DCs part though.

> For v1, however, we plan to
> provide only logical separation (to a certain extent) via modules which can
> be dynamically loaded/unloaded from RESTBase.

modules ? Care to explain a bit more ? AFAIK RESTBase is a revision
storage service and to be honest I am fighting to understand what
modules you are referring to and the architecture behind those
modules.

> In return, RESTBase will
> provide them with routing, monitoring, caching and authorisation out of the
> box. The good point here is that this 'modularisation' eases the transition
> to a more-decomposed orchestration SOA model. Going in that direction,
> however, requires some prerequisites to be fulfilled, such as [1].

While revision caching can very well be done by RESTBase (AFAIK, that
is one reason it is being created for), authorization (It's not
revision authorization, but generic authentication/authorization I am
referring to) and monitoring should not be provided by RESTBase to any
service. Especially monitoring. Services (whatever their nature)
should provide discoverable (REST if you like, as I suspect you do)
endpoints that allow monitoring via third party tools and not depend
on an another service for that. My take is that there should be a
swagger manifest that describes a basic monitoring framework and
services should each independently implement it (including RESTBase)

I am also a bit unclear on the routing aspect. Care to point out an up
to date architectural diagram ? I have been told in person that the
one https://www.npmjs.com/package/restbase is not up to date so I
can't comment on that.


-- 
Alexandros Kosiaris 

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Giuseppe Lavagetto]

On Wed, Feb 4, 2015 at 5:42 AM, Tim Starling  wrote:
> On 04/02/15 12:46, Dan Garry wrote:
>> To address these challenges, we are considering performing some or all of
>> these tasks in a service developed by the Mobile Apps Team with help from
>> Services. This service will hit the APIs we currently hit on the client,
>> aggregate the content we need on the server side, perform transforms we're
>> currently doing on the client on the server instead, and serve the full
>> response to the user via RESTBase. In addition to providing a public API
>> end point, RESTBase would help with common tasks like monitoring, caching
>> and authorisation.
>
> I don't really understand why you want it to be integrated with
> RESTBase. As far as I can tell (it is hard to pin these things down),
> RESTBase is a revision storage backend and possibly a public API for
> that backend. I thought the idea of SOA was to separate concerns.
> Wouldn't monitoring, caching and authorization would be best done as a
> node.js library which RESTBase and other services use?
>

I agree with Tim. Using RESTBase as an integration layer for
everything is SOA done wrong. If we need to have an authorization
system, which is different from our APIs, we need to build it
separately, not to add levels of indirection.

Doing 4 things from one single service is basically rebuilding the
mediawiki monolith, only in a different language :)

What you need, IMO, is a thin proxy layer in front of all the separate
APIs you have to call, including restbase for caching/revision
storage. It may be built into the app or, if it is consumed by
multiple apps, built as a thin proxy service itself.

(I also don't get what "monitoring" means here, but someone could
probably explain it to me)

Cheers,

Giuseppe

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Erik Moeller]

On Tue, Feb 3, 2015 at 5:46 PM, Dan Garry  wrote:
> To address these challenges, we are considering performing some or all of
> these tasks in a service developed by the Mobile Apps Team with help from
> Services. This service will hit the APIs we currently hit on the client,
> aggregate the content we need on the server side, perform transforms we're
> currently doing on the client on the server instead, and serve the full
> response to the user via RESTBase. In addition to providing a public API
> end point, RESTBase would help with common tasks like monitoring, caching
> and authorisation.

Using https://phabricator.wikimedia.org/T87824 as a reference point
for what you're talking about -

I think you will generally find agreement that moving client-side
transformations that only live in the app to server-side code that
enables access by multiple consumers and caching is a good idea. If
there are reasons not do to this, now'd be a good time to speak up.

If not, then I think one thing to keep in mind is how to organize the
transformation code in a manner that it doesn't just become a
server-side hodgepodge still only useful to one consumer, to avoid
some of the pitfalls Brian mentions. Say you want to reformat
infoboxes on the mobile web, but not do all the other stuff the mobile
app does. Can you just get that specific transformation? Are some
transformations dependent on others?  Or say we want to make a change
only for the output that gets fed into the PDF generator, but not for
any other outputs. Can we do that?

Or a more pressing concern for the app team itself, what about alpha,
beta, stable version of the apps -- how would those get more/less
experimental versions of the output? Or languages -- are there cases
where we apply a transformation only in one language, but not another?

Do we need a way to register schemas so we can easily get a certain
set of inter-dependent transformations, like "mobile app stable",
"desktop web", etc.? Or are these all just API/service parameters?

Just some early questions as we're thinking this through.

Erik

-- 
Erik Möller
VP of Product & Strategy, Wikimedia Foundation

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Corey Floyd]

tl;dr; Before the Mobile Apps Team embarks on its own path, I really think
we should work together as an organization to figure out a better strategy
for implementing a set of consistent services that address the needs of all
platforms.


So… I've been involved in similar conversations at other organizations.
They generally sound like this:

"These APIs were designed before mobile. Mobile needs mobile friendly
services to be great. This services currently don't exist and/or aren't
being implemented fast enough for us to build the features we want. Lets
implement our own services!" (Ok that was a bit contrived but you get the
point)

The problem I have with this solution is two fold:

1. The owners of this services project should really be the services team.
2. Building a separate service for mobile does not align the needs of the
mobile team with the rest of the MediaWiki API consumers.

To be clear - I totally agree that we should be working with services
engineers to collaboratively develop APIs. In fact, Mobile Apps, along with
other front end teams, should be proactively giving the services team input
and support so they can build the services that we need. This results in
making the API not only better for Mobile Apps, but the other platforms as
well.

Working around our services issues by providing the mobile apps with
different set of APIs than the rest of the platforms comes with its own set
of problems. In the long run it creates more work (maintaining multiple
APIs) and leads ambiguity ("Oh, that only works on the mobile API"). It
also creates a new point of failure that would only affect mobile apps.
This means at times consumers of the "Real API" may have needs that
conflict with mobile apps and compete for the time and attention of the
services engineers.

Being on iOS and Android already puts the mobile apps on an island because
of the language and framework differences. Using a different API won't help
that, but development of an API that serves the needs of all the platforms
aligns our needs with the rest of the organization while providing all
front end teams a consistent way to access data.

This doesn't even address the fact that the mobile team is staffed with
experts in tuning native apps - which is quite at different skill set than
writing, deploying, and maintaining server side java script. Write good
scalable server code is difficult, and this will be extremely important
here, since, for the foreseeable future, mobile traffic is expected to see
the most growth.

It should go without saying, but I'm obviously not against Mobile App
Engineers submitting patches to the API if they want to, maybe even hack
using days (though rumors have it that iOS is a bit busy this quarter). But
this is quite different than owning the service.

After reading this, you would probably assume all my experiences with this
strategy were bad. Ironically, I was recently on a mobile team that
developed its own services successfully. I even helped to come up with the
Acronym: MISL (Mobile Intermediary Services Layer). :)

However, on that mobile team we had 2 experienced AND dedicated
javascript/python engineers that were specifically tasked to write
services/administration panels and who interfaced directly with the main
services team.

(Now, If we are going to start talking about embedding web services
engineers on the Mobile Apps team, I'm definitely in!)


On Wed, Feb 4, 2015 at 12:59 AM, Marko Obrovac 
wrote:

> On Tue, Feb 3, 2015 at 8:42 PM, Tim Starling 
> wrote:
>
> > I don't really understand why you want it to be integrated with
> > RESTBase. As far as I can tell (it is hard to pin these things down),
> > RESTBase is a revision storage backend and possibly a public API for
> > that backend.
>
>
> Actually, RESTBase's logic applies to the Mobile Apps case quite naturally.
> When a page is fetched and transformed, it can be stored so that consequent
> requests can simply retrieve the transformed document form storage.
>
>
> > I thought the idea of SOA was to separate concerns.
> > Wouldn't monitoring, caching and authorization would be best done as a
> > node.js library which RESTBase and other services use?
> >
>
> Good point. Ideally, what we would need to do is provide the right tools to
> developers to create services, which can then be placed "strategically"
> around DCs (in cooperation with Ops, ofc). For v1, however, we plan to
> provide only logical separation (to a certain extent) via modules which can
> be dynamically loaded/unloaded from RESTBase. In return, RESTBase will
> provide them with routing, monitoring, caching and authorisation out of the
> box. The good point here is that this 'modularisation' eases the transition
> to a more-decomposed orchestration SOA model. Going in that direction,
> however, requires some prerequisites to be fulfilled, such as [1].
>
> Marko
>
> [1] https://phabricator.wikimedia.org/T84923
>
> Marko Obrovac
> Senior Services Engineer
> Wikimedia Foundation
>

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Marko Obrovac]

On Tue, Feb 3, 2015 at 8:42 PM, Tim Starling 
wrote:

> I don't really understand why you want it to be integrated with
> RESTBase. As far as I can tell (it is hard to pin these things down),
> RESTBase is a revision storage backend and possibly a public API for
> that backend.


Actually, RESTBase's logic applies to the Mobile Apps case quite naturally.
When a page is fetched and transformed, it can be stored so that consequent
requests can simply retrieve the transformed document form storage.


> I thought the idea of SOA was to separate concerns.
> Wouldn't monitoring, caching and authorization would be best done as a
> node.js library which RESTBase and other services use?
>

Good point. Ideally, what we would need to do is provide the right tools to
developers to create services, which can then be placed "strategically"
around DCs (in cooperation with Ops, ofc). For v1, however, we plan to
provide only logical separation (to a certain extent) via modules which can
be dynamically loaded/unloaded from RESTBase. In return, RESTBase will
provide them with routing, monitoring, caching and authorisation out of the
box. The good point here is that this 'modularisation' eases the transition
to a more-decomposed orchestration SOA model. Going in that direction,
however, requires some prerequisites to be fulfilled, such as [1].

Marko

[1] https://phabricator.wikimedia.org/T84923

Marko Obrovac
Senior Services Engineer
Wikimedia Foundation
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Dan Garry]

Hey Kunal,

Responses in-line.

On 3 February 2015 at 21:09, Legoktm  wrote:
>
> Is there a bug for this and the other issues? I'm subscribed to [1], but
> I don't see anything like the issues you've mentioned on it.
>

There's this, which documents some of them, but it's more descriptive of a
specific proposed solution: https://phabricator.wikimedia.org/T87824

I welcome any other ideas that anyone has for solving our problems. We're
not tied to a specific solution.


> What are "read more recommendations"?


When you scroll to the end of an article, we give three suggestions as to
what you could read next. The suggestions are generated quite naively by
running a full text search using the current page's title as a query. Our
metrics have shown a lot of positive engagement with the feature so far.

Here's a screenshot on [[Bern]]: http://i.imgur.com/qnHcsdT.png


> Is there an actual instance where an API change has broken an app or is
> this merely a hypothetical concern? Speaking as an API developer, we
> occasionally have to make breaking changes, and if we broke something
> too fast, it would be nice to have feedback if that was the case so we
> can improve in the future.


I'm referring to a possible upcoming breaking change in MobileFrontend,
which is the API we depend on for our content. A certain API parameter may
be removed. Since the apps were made under the assumption that that value
would always be in the API response, the app doesn't handle this issue
well. It leaves users unable to see file pages.

This issue is particularly prominent on apps since if the user never
updates the app then they'll never get the fix.

Dan

-- 
Dan Garry
Associate Product Manager, Mobile Apps
Wikimedia Foundation
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Legoktm]

On 02/03/2015 05:46 PM, Dan Garry wrote:
> *tl;dr: Mobile Apps will, in partnership with the Services, investigate
> building a content service for the Mobile Apps.*
> 
> The Mobile Apps Team currently has quite a few pain points with the way we
> fetch article content currently:
> 
>- We have to make a lot of API requests to load an article: article
>HTML, lead image, read more recommendations, and more

Is there a bug for this and the other issues? I'm subscribed to [1], but
I don't see anything like the issues you've mentioned on it.

What are "read more recommendations"?

>- We send the user HTML that we then discard, needlessly increasing data
>usage
>- We do transforms to the HTML in JavaScript on the client side, which
>causes code duplication across the apps and degrades user-perceived
>performance
>- Trivial changes to the API (e.g. renaming a parameter) can break the
>app which is problematic since apps can't be hotfixed easily

Is there an actual instance where an API change has broken an app or is
this merely a hypothetical concern? Speaking as an API developer, we
occasionally have to make breaking changes, and if we broke something
too fast, it would be nice to have feedback if that was the case so we
can improve in the future.

[1] https://phabricator.wikimedia.org/T75616

-- Legoktm

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Tim Starling]

On 04/02/15 12:46, Dan Garry wrote:
> To address these challenges, we are considering performing some or all of
> these tasks in a service developed by the Mobile Apps Team with help from
> Services. This service will hit the APIs we currently hit on the client,
> aggregate the content we need on the server side, perform transforms we're
> currently doing on the client on the server instead, and serve the full
> response to the user via RESTBase. In addition to providing a public API
> end point, RESTBase would help with common tasks like monitoring, caching
> and authorisation.

I don't really understand why you want it to be integrated with
RESTBase. As far as I can tell (it is hard to pin these things down),
RESTBase is a revision storage backend and possibly a public API for
that backend. I thought the idea of SOA was to separate concerns.
Wouldn't monitoring, caching and authorization would be best done as a
node.js library which RESTBase and other services use?

-- Tim Starling


___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Brian Gerstle]

Thanks for getting this ball rolling, Dan! Couldn't agree more with the
points you raised—having in fact raised a few of them myself. Put me down
as one of the mobile/full-stack engineers who wants to work on this service
:-).

On Tue, Feb 3, 2015 at 8:46 PM, Dan Garry  wrote:

> *tl;dr: Mobile Apps will, in partnership with the Services, investigate
> building a content service for the Mobile Apps.*
>
> The Mobile Apps Team currently has quite a few pain points with the way we
> fetch article content currently:
>
>- We have to make a lot of API requests to load an article: article
>HTML, lead image, read more recommendations, and more
>- We send the user HTML that we then discard, needlessly increasing data
>usage
>- We do transforms to the HTML in JavaScript on the client side, which
>causes code duplication across the apps and degrades user-perceived
>performance
>- Trivial changes to the API (e.g. renaming a parameter) can break the
>app which is problematic since apps can't be hotfixed easily
>
> To address these challenges, we are considering performing some or all of
> these tasks in a service developed by the Mobile Apps Team with help from
> Services. This service will hit the APIs we currently hit on the client,
> aggregate the content we need on the server side, perform transforms we're
> currently doing on the client on the server instead, and serve the full
> response to the user via RESTBase. In addition to providing a public API
> end point, RESTBase would help with common tasks like monitoring, caching
> and authorisation.


> So the Mobile Apps Team is going to spend a bit of time investigating
> whether using RESTBase with Node.js is an option for building a content
> service for the Wikipedia app to replace our current method of retrieving
> article content. Our initial scope for this is feature parity with our
> current content retrieval method.
>
> Our action items are as follows:
>
>- Wait for RESTBase to be deployed.
>- Timescale: Weeks
>   - Owner: All of us :-)

   - Figure out what information the service should serve for the first
>iteration (i.e. for feature parity) and what APIs it needs to hit to do
> that
>- Timescale: Wed 4th Feb
>   - Owner: Dan Garry
>- Start implementing the service and see whether it meets our needs
>- Timescale: Planning a spike for next apps sprint (16th Feb - 27th Feb)
>   to perform initial investigation
>   - Owner: Currently undecided engineer from Mobile Apps, with Services
>   engineers serving as consultants
>
> As always, feel free to ask if there are any questions.
>
> Dan
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l




-- 
EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle
IRC: bgerstle
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Dan Garry]

*tl;dr: Mobile Apps will, in partnership with the Services, investigate
building a content service for the Mobile Apps.*

The Mobile Apps Team currently has quite a few pain points with the way we
fetch article content currently:

   - We have to make a lot of API requests to load an article: article
   HTML, lead image, read more recommendations, and more
   - We send the user HTML that we then discard, needlessly increasing data
   usage
   - We do transforms to the HTML in JavaScript on the client side, which
   causes code duplication across the apps and degrades user-perceived
   performance
   - Trivial changes to the API (e.g. renaming a parameter) can break the
   app which is problematic since apps can't be hotfixed easily

To address these challenges, we are considering performing some or all of
these tasks in a service developed by the Mobile Apps Team with help from
Services. This service will hit the APIs we currently hit on the client,
aggregate the content we need on the server side, perform transforms we're
currently doing on the client on the server instead, and serve the full
response to the user via RESTBase. In addition to providing a public API
end point, RESTBase would help with common tasks like monitoring, caching
and authorisation.

So the Mobile Apps Team is going to spend a bit of time investigating
whether using RESTBase with Node.js is an option for building a content
service for the Wikipedia app to replace our current method of retrieving
article content. Our initial scope for this is feature parity with our
current content retrieval method.

Our action items are as follows:

   - Wait for RESTBase to be deployed.
   - Timescale: Weeks
  - Owner: All of us :-)
   - Figure out what information the service should serve for the first
   iteration (i.e. for feature parity) and what APIs it needs to hit to do that
   - Timescale: Wed 4th Feb
  - Owner: Dan Garry
   - Start implementing the service and see whether it meets our needs
   - Timescale: Planning a spike for next apps sprint (16th Feb - 27th Feb)
  to perform initial investigation
  - Owner: Currently undecided engineer from Mobile Apps, with Services
  engineers serving as consultants

As always, feel free to ask if there are any questions.

Dan
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l