Re: [WikimediaMobile] [Apps] Scope of the apps content service

[Bernd Sitzmann]

Max,

For regular page views we split it into two requests: one for lead section
and another for the remaining sections.
As a general rule of thumb we want to put only the information that is
needed to display above the fold (section 0, lead image URL, Wikidata
description, ...), plus maybe some minor metadata fields that help us with
housekeeping.

The gallery data for should go into the second request.

As far as housekeeping goes, I think it would be great to have the revision
ID and some hash for the things outside the pure pagedata (gallery items,
...)
so we could determine after the first request if we could skip the second
request because we already have that as a saved page or in a cache.
Ideally, those would go into an ETag, and the client uses an If-None-Match
header the next time we request it, so that in the case we already have the
data there wouldn't be a payload in the response, just the header.

Bernd

On Thu, Feb 5, 2015 at 9:59 PM, Monte Hurd  wrote:

> Saving pages is not the primary use case for the image meta data portion -
> users tapping on an image and not having to wait for another server round
> trip to determine the high res url and meta data is the primary use case.
>
> Having all this data as part of one request means image taps can be much
> lower latency from the user's perspective. The panel can pop up immediately
> with meta data overlay over the already downloaded low res image and can
> immediately begin lazy loading the high res version. It will seem much more
> responsive, which is really important.
>
> As a side benefit, saving page data for offline access also becomes
> easier, but users tap on images (I think) more than they save pages.
>
>
>
> On Feb 5, 2015, at 1:40 PM, Max Semenik  wrote:
>
> Saving a page for offline is not that speed sensitive, however, serving
> this information as part of normal page views would definitely impact
> overall latency.
>
> On Thu, Feb 5, 2015 at 1:24 PM, Dan Garry  wrote:
>
>> On 5 February 2015 at 13:20, Max Semenik  wrote:
>>>
>>> Wouldn't that bloat the response size?
>>>
>>
>> The rationale for including this is because that way the image viewer
>> will work even if you've not tapped on any of the images. This is pretty
>> important functionality that doesn't work offline right now, even if you
>> save the page for offline reading.
>>
>> That said, we may want to punt this until after the MVP as being possibly
>> outside of scope.
>>
>> Dan
>>
>> --
>> Dan Garry
>> Associate Product Manager, Mobile Apps
>> Wikimedia Foundation
>>
>
>
>
> --
> Best regards,
> Max Semenik ([[User:MaxSem]])
>
> ___
> Mobile-l mailing list
> Mobile-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/mobile-l
>
>
> ___
> Mobile-l mailing list
> Mobile-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/mobile-l
>
>
___
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l

Re: [WikimediaMobile] [Apps] Scope of the apps content service

[Monte Hurd]

Saving pages is not the primary use case for the image meta data portion - 
users tapping on an image and not having to wait for another server round trip 
to determine the high res url and meta data is the primary use case. 

Having all this data as part of one request means image taps can be much lower 
latency from the user's perspective. The panel can pop up immediately with meta 
data overlay over the already downloaded low res image and can immediately 
begin lazy loading the high res version. It will seem much more responsive, 
which is really important. 

As a side benefit, saving page data for offline access also becomes easier, but 
users tap on images (I think) more than they save pages.



> On Feb 5, 2015, at 1:40 PM, Max Semenik  wrote:
> 
> Saving a page for offline is not that speed sensitive, however, serving this 
> information as part of normal page views would definitely impact overall 
> latency.
> 
>> On Thu, Feb 5, 2015 at 1:24 PM, Dan Garry  wrote:
>> On 5 February 2015 at 13:20, Max Semenik  wrote:
>>> 
>>> 
>>> Wouldn't that bloat the response size?
>> 
>> 
>> The rationale for including this is because that way the image viewer will 
>> work even if you've not tapped on any of the images. This is pretty 
>> important functionality that doesn't work offline right now, even if you 
>> save the page for offline reading.
>> 
>> That said, we may want to punt this until after the MVP as being possibly 
>> outside of scope.
>> 
>> Dan
>> 
>> -- 
>> Dan Garry
>> Associate Product Manager, Mobile Apps
>> Wikimedia Foundation
> 
> 
> 
> -- 
> Best regards,
> Max Semenik ([[User:MaxSem]])
> ___
> Mobile-l mailing list
> Mobile-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/mobile-l
___
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l

Re: [WikimediaMobile] [Apps] Scope of the apps content service

[Monte Hurd]

I think it will be more than offset by the reduction in response size gained by 
the "Exclude unused HTML" bullet point. Also, gzip encoding is pretty good at 
text compression. 


> On Feb 5, 2015, at 1:20 PM, Max Semenik  wrote:
> 
>> On Thu, Feb 5, 2015 at 12:00 PM, Dan Garry  wrote:
>> Image meta data for all images in the page
>> License
>> Description
>> Includes the URL for the high resolution version used for the lead image
> Wouldn't that bloat the response size?
> 
> -- 
> Best regards,
> Max Semenik ([[User:MaxSem]])
> ___
> Mobile-l mailing list
> Mobile-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/mobile-l
___
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l

Re: [WikimediaMobile] [Apps] Scope of the apps content service

[Max Semenik]

Saving a page for offline is not that speed sensitive, however, serving
this information as part of normal page views would definitely impact
overall latency.

On Thu, Feb 5, 2015 at 1:24 PM, Dan Garry  wrote:

> On 5 February 2015 at 13:20, Max Semenik  wrote:
>>
>> Wouldn't that bloat the response size?
>>
>
> The rationale for including this is because that way the image viewer will
> work even if you've not tapped on any of the images. This is pretty
> important functionality that doesn't work offline right now, even if you
> save the page for offline reading.
>
> That said, we may want to punt this until after the MVP as being possibly
> outside of scope.
>
> Dan
>
> --
> Dan Garry
> Associate Product Manager, Mobile Apps
> Wikimedia Foundation
>



-- 
Best regards,
Max Semenik ([[User:MaxSem]])
___
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l

Re: [WikimediaMobile] [Apps] Scope of the apps content service

[Dan Garry]

On 5 February 2015 at 13:20, Max Semenik  wrote:
>
> Wouldn't that bloat the response size?
>

The rationale for including this is because that way the image viewer will
work even if you've not tapped on any of the images. This is pretty
important functionality that doesn't work offline right now, even if you
save the page for offline reading.

That said, we may want to punt this until after the MVP as being possibly
outside of scope.

Dan

-- 
Dan Garry
Associate Product Manager, Mobile Apps
Wikimedia Foundation
___
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l

Re: [WikimediaMobile] [Apps] Scope of the apps content service

[Max Semenik]

On Thu, Feb 5, 2015 at 12:00 PM, Dan Garry  wrote:

>
>- Image meta data for all images in the page
>- License
>   - Description
>   - Includes the URL for the high resolution version used for the
>   lead image
>
> Wouldn't that bloat the response size?

-- 
Best regards,
Max Semenik ([[User:MaxSem]])
___
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l

Re: [WikimediaMobile] [Apps] Scope of the apps content service

[Brian Gerstle]

I forgot to fill in my "*" next to "app services."  The name is not a great
one, and I'm open to suggestions that are more specific or intuitive.

On Thu, Feb 5, 2015 at 4:04 PM, Brian Gerstle 
wrote:

> In an effort to avoid yet another mega thread, I went ahead and created a
> Wiki page under the apps space which holds the problems & requirements for app
> services *.
> There's even a discussion page
>  where
> I've cherry-picked Brion & Gabriel's comments, organized by topic.  I hope
> this proves useful and will give us something to go on in the future.  I've
> put my comments so far on the talk page.
>
> On Thu, Feb 5, 2015 at 3:15 PM, Gabriel Wicke 
> wrote:
>
>> On Thu, Feb 5, 2015 at 12:04 PM, Brion Vibber 
>> wrote:
>>>
>>>
- Versioning? What if we add or remove a field from the response?
   We should version this. Can we do this easily?


>> In RESTBase we generally use mime type versioning like this:
>>
>> application/json; profile=mediawiki.org/specs/app_page_bundle/1.0.0
>>
>> This is specced in the Swagger spec & enforced by RESTBase. On version
>> mismatch from stored content, the backend service is called to re-generate
>> or upgrade the content, which is then saved back. It can be passed the
>> stored content to make re-generation more efficient.
>>
>> If needed, clients could also signal the content-type they expect with an
>> accept header. The details of how things would then be upgraded /
>> downgraded would need to be worked out in case you really need this. In
>> general, you can go a long way by only adding properties while remaining
>> backwards-compatible with the old properties.
>>
>> We also distinguish stability per end point:
>>
>>
>>- *experimental *end points can change at any time (effectively
>>private, use at your own risk)
>>- *unstable *end points can change, but we make an effort to avoid
>>breakage and notify users
>>- any change to *stable* end points will increment the major API
>>version (/v1/) following semver
>>
>> The mobile API should probably be marked as experimental, as it is
>> primarily intended as an API for apps we control.
>>
>> Gabriel
>>
>> ___
>> Mobile-l mailing list
>> Mobile-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/mobile-l
>>
>>
>
>
> --
> EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle
> IRC: bgerstle
>



-- 
EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle
IRC: bgerstle
___
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l

Re: [WikimediaMobile] [Apps] Scope of the apps content service

[Brian Gerstle]

In an effort to avoid yet another mega thread, I went ahead and created a
Wiki page under the apps space which holds the problems & requirements for app
services *.
There's even a discussion page
 where
I've cherry-picked Brion & Gabriel's comments, organized by topic.  I hope
this proves useful and will give us something to go on in the future.  I've
put my comments so far on the talk page.

On Thu, Feb 5, 2015 at 3:15 PM, Gabriel Wicke  wrote:

> On Thu, Feb 5, 2015 at 12:04 PM, Brion Vibber 
> wrote:
>>
>>
>>>- Versioning? What if we add or remove a field from the response? We
>>>   should version this. Can we do this easily?
>>>
>>>
> In RESTBase we generally use mime type versioning like this:
>
> application/json; profile=mediawiki.org/specs/app_page_bundle/1.0.0
>
> This is specced in the Swagger spec & enforced by RESTBase. On version
> mismatch from stored content, the backend service is called to re-generate
> or upgrade the content, which is then saved back. It can be passed the
> stored content to make re-generation more efficient.
>
> If needed, clients could also signal the content-type they expect with an
> accept header. The details of how things would then be upgraded /
> downgraded would need to be worked out in case you really need this. In
> general, you can go a long way by only adding properties while remaining
> backwards-compatible with the old properties.
>
> We also distinguish stability per end point:
>
>
>- *experimental *end points can change at any time (effectively
>private, use at your own risk)
>- *unstable *end points can change, but we make an effort to avoid
>breakage and notify users
>- any change to *stable* end points will increment the major API
>version (/v1/) following semver
>
> The mobile API should probably be marked as experimental, as it is
> primarily intended as an API for apps we control.
>
> Gabriel
>
> ___
> Mobile-l mailing list
> Mobile-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/mobile-l
>
>


-- 
EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle
IRC: bgerstle
___
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l

Re: [WikimediaMobile] [Apps] Scope of the apps content service

[Gabriel Wicke]

On Thu, Feb 5, 2015 at 12:04 PM, Brion Vibber  wrote:
>
>
>>- Versioning? What if we add or remove a field from the response? We
>>   should version this. Can we do this easily?
>>
>>
In RESTBase we generally use mime type versioning like this:

application/json; profile=mediawiki.org/specs/app_page_bundle/1.0.0

This is specced in the Swagger spec & enforced by RESTBase. On version
mismatch from stored content, the backend service is called to re-generate
or upgrade the content, which is then saved back. It can be passed the
stored content to make re-generation more efficient.

If needed, clients could also signal the content-type they expect with an
accept header. The details of how things would then be upgraded /
downgraded would need to be worked out in case you really need this. In
general, you can go a long way by only adding properties while remaining
backwards-compatible with the old properties.

We also distinguish stability per end point:


   - *experimental *end points can change at any time (effectively private,
   use at your own risk)
   - *unstable *end points can change, but we make an effort to avoid
   breakage and notify users
   - any change to *stable* end points will increment the major API version
   (/v1/) following semver

The mobile API should probably be marked as experimental, as it is
primarily intended as an API for apps we control.

Gabriel
___
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l

Re: [WikimediaMobile] [Apps] Scope of the apps content service

[Brion Vibber]

There are two main approaches to versioning y'all can take:

One is the method used by most MediaWiki API modules where if you want a
new bit of info that wasn't in the default before, you have to ask for
it... and you never remove anything from defaults for fear of breaking
things.

The other is to use a nice linear versioning scheme, where you go to
"/api/v1/" or "/api/v2/" etc and the version number controls the business
logic of which elements get fetched and inserted into the response.

I tend to favor the latter as it's conceptually simpler and fits with the
concept of versioning up the apps along with their associated service. (The
former is more flexible but also means your requests get longer and longer
as you add features.)

-- brion

On Thu, Feb 5, 2015 at 12:00 PM, Dan Garry  wrote:

> Some of the Mobile Apps Team engineers met to discuss the scope of the MVP
> for a mobile apps content service.
>
> Firstly, we enumerated our goals for this content service:
>
>- Minimise number of requests the app has to make to generate a page
>- Reduce payload size of requests
>- Abstract away from of the API logic contained in the client into the
>server instead
>- Do some of the transforms that are currently done on the client on
>the server instead, where appropriate
>
> We decided that the scope of the service should be feature parity; the
> service should be a drop-in replacement for our current API calls. More
> concretely, this means it should give us:
>
>- Data from mobileview
>   - Wikidata description
>   - Revision ID of page
>   - Wikibase ID
>- Image meta data for all images in the page
>- License
>   - Description
>   - Includes the URL for the high resolution version used for the
>   lead image
>- Exclude unused HTML
>- Navboxes, etc.
>- Exclude hidden elements
>   - CSS: display: none; unless if we display later
>- Related articles suggestions from full text search API
>
> Things which we want to add, but are out of scope for the first iteration:
>
>- Arbitrary Wikidata properties
>- Disambiguation (already "transcluded", following the chain of
>disambiguation pages and returning a JSON array of those links)
>- Page issues
>- Interwiki links
>- Table of contents (as a JSON array)
>
> Open questions:
>
>- Versioning? What if we add or remove a field from the response? We
>should version this. Can we do this easily?
>
> Thanks,
> Dan
>
> --
> Dan Garry
> Associate Product Manager, Mobile Apps
> Wikimedia Foundation
>
> ___
> Mobile-l mailing list
> Mobile-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/mobile-l
>
>
___
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l

[WikimediaMobile] [Apps] Scope of the apps content service

[Dan Garry]

Some of the Mobile Apps Team engineers met to discuss the scope of the MVP
for a mobile apps content service.

Firstly, we enumerated our goals for this content service:

   - Minimise number of requests the app has to make to generate a page
   - Reduce payload size of requests
   - Abstract away from of the API logic contained in the client into the
   server instead
   - Do some of the transforms that are currently done on the client on the
   server instead, where appropriate

We decided that the scope of the service should be feature parity; the
service should be a drop-in replacement for our current API calls. More
concretely, this means it should give us:

   - Data from mobileview
  - Wikidata description
  - Revision ID of page
  - Wikibase ID
   - Image meta data for all images in the page
   - License
  - Description
  - Includes the URL for the high resolution version used for the lead
  image
   - Exclude unused HTML
   - Navboxes, etc.
   - Exclude hidden elements
  - CSS: display: none; unless if we display later
   - Related articles suggestions from full text search API

Things which we want to add, but are out of scope for the first iteration:

   - Arbitrary Wikidata properties
   - Disambiguation (already "transcluded", following the chain of
   disambiguation pages and returning a JSON array of those links)
   - Page issues
   - Interwiki links
   - Table of contents (as a JSON array)

Open questions:

   - Versioning? What if we add or remove a field from the response? We
   should version this. Can we do this easily?

Thanks,
Dan

-- 
Dan Garry
Associate Product Manager, Mobile Apps
Wikimedia Foundation
___
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l