[Wikidata-bugs] [Maniphest] [Commented On] T92961: [Story] Versioning in JSON output

2019-04-09 Thread Addshore 
Addshore added a comment.


  This will likely get worked on, at least for Api output with the work on 
federation that should happen toward the end of this year.

TASK DETAIL
  https://phabricator.wikimedia.org/T92961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: Pintoch, Pfps, aude, Smalyshev, Ricordisamoa, Addshore, Lydia_Pintscher, 
adrianheine, JanZerebecki, daniel, Tobi_WMDE_SW, Aklapper, alaa_wmde, Nandana, 
Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, 
Wikidata-bugs, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T92961: [Story] Versioning in JSON output

2016-08-24 Thread daniel 
daniel added a comment.

In T92961#2580231, @Smalyshev wrote:
Metadata file maybe would be fine too, then it should look the same as the API format without entities, I think:


agreedTASK DETAILhttps://phabricator.wikimedia.org/T92961EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: danielCc: Pfps, aude, Smalyshev, Ricordisamoa, Addshore, Lydia_Pintscher, adrianheine, JanZerebecki, daniel, Tobi_WMDE_SW, Aklapper, D3r1ck01, Izno, Wikidata-bugs, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T92961: [Story] Versioning in JSON output

2016-08-24 Thread daniel 
daniel added a comment.

In T92961#2579283, @Pfps wrote:
Well I think that it should be a breaking change, but I read the stable interface policy as saying that it isn't.  Well-behaved consumers are supposed to be tolerant of extra information.  Adding a new item to the dump array is adding extra information.  This extra information does not change the meaning of the existing information in any way.


I would definitely treat it as a breaking change, but you are right that the policy isn't very clear about that. The reason I think it is breaking is that if we have a concept of "list of X", you can't add a Y to it (if Y is not an X) without breaking the format. Adding an incompatible Y to the list means that the list is no longer homogenous. In my mind, that changes the interpretation. But I'll add a note-to-self to clarify this point. Thanks!TASK DETAILhttps://phabricator.wikimedia.org/T92961EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: danielCc: Pfps, aude, Smalyshev, Ricordisamoa, Addshore, Lydia_Pintscher, adrianheine, JanZerebecki, daniel, Tobi_WMDE_SW, Aklapper, D3r1ck01, Izno, Wikidata-bugs, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T92961: [Story] Versioning in JSON output

2016-08-24 Thread Smalyshev 
Smalyshev added a comment.
Metadata file maybe would be fine too, then it should look the same as the API format without entities, I think, e.g. we'd have:

{
  "@metadata": {
 "version": "1.0",
 "phase-of-moon": "gibbous"
  }
}

That'd make parsing it easier, I think.TASK DETAILhttps://phabricator.wikimedia.org/T92961EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Pfps, aude, Smalyshev, Ricordisamoa, Addshore, Lydia_Pintscher, adrianheine, JanZerebecki, daniel, Tobi_WMDE_SW, Aklapper, D3r1ck01, Izno, Wikidata-bugs, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T92961: [Story] Versioning in JSON output

2016-08-24 Thread daniel 
daniel added a comment.
@Smalyshev Oh right, I forgot that we do already output an envelope from Special:EntityData! That makes things easier.

For the dump, yea, we have the choice to break the format really hard, or associate a secondary file containing the meta-data.TASK DETAILhttps://phabricator.wikimedia.org/T92961EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: danielCc: Pfps, aude, Smalyshev, Ricordisamoa, Addshore, Lydia_Pintscher, adrianheine, JanZerebecki, daniel, Tobi_WMDE_SW, Aklapper, D3r1ck01, Izno, Wikidata-bugs, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T92961: [Story] Versioning in JSON output

2016-08-24 Thread Smalyshev 
Smalyshev added a comment.
I'm looking at https://www.wikidata.org/wiki/Special:EntityData/Q42.json structure, and I see it is:

{
  "entities": {
"Q42": { ...stuff... }
  }
}

So, why can't we have:

{
  "entities": {
"Q42": { ...stuff... }
  },
  "@metadata": {
 "version": "1.0",
 "phase-of-moon": "gibbous"
  }
}

etc.? I don't think well-behaved clients should mind? But even if they do, it shouldn't be hard for them to make one-time change to ignore @metadata if they don't want it? After all, we already have other items besides entities - e.g. in API response there's success.

The same can be done with wbgetentities, etc. APIs. For the dump it's a bit harder as we don't have any kind of encompassing structure beyond entities... Maybe we should make it use "entities" too? It will be a breaking change but at least it breaks the stalemate and allows us to easily add more info later. It won't be also hard to distinguish between the formats, including by streaming readers - you actually need only the first character to know which one you're dealing with, and converting new one into old one or back should be rather easy.TASK DETAILhttps://phabricator.wikimedia.org/T92961EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Pfps, aude, Smalyshev, Ricordisamoa, Addshore, Lydia_Pintscher, adrianheine, JanZerebecki, daniel, Tobi_WMDE_SW, Aklapper, D3r1ck01, Izno, Wikidata-bugs, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T92961: [Story] Versioning in JSON output

2016-08-24 Thread Pfps 
Pfps added a comment.
Of course it would by a breaking change. There is no formal spec of the JSON dump beyond the spec for the individual entities, but we have always said that the dump is a set (an array) of entities. Putting something in there that is not an entity will break consumers.

Well I think that it should be a breaking change, but I read the stable interface policy as saying that it isn't.  Well-behaved consumers are supposed to be tolerant of extra information.  Adding a new item to the dump array is adding extra information.  This extra information does not change the meaning of the existing information in any way.TASK DETAILhttps://phabricator.wikimedia.org/T92961EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: PfpsCc: Pfps, aude, Smalyshev, Ricordisamoa, Addshore, Lydia_Pintscher, adrianheine, JanZerebecki, daniel, Tobi_WMDE_SW, Aklapper, D3r1ck01, Izno, Wikidata-bugs, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T92961: [Story] Versioning in JSON output

2016-08-24 Thread daniel 
daniel added a comment.

In T92961#2577993, @Pfps wrote:
There is nothing preventing the dump format from having the first JSON object be information about the dump, including version of the dump format, version of wikidata format, time of dump, etc.

As long as this JSON object did not conform to the form of JSON objects that encode Wikidata entities this change would not be a breaking change!


Of course it would by a breaking change. There is no formal spec of the JSON dump beyond the spec for the individual entities, but we have always said that the dump is a set (an array) of entities. Putting something in there that is not an entity will break consumers.

If we are going to break the format, I prefer to introduce a proper envelope with a clear place for meta-data.

Note however that for the Special:EntityData interface, we have a similar but different problem: There we have only a single entity object, with no array or other structure around it. We can easily put the meta-info into the object itself, but that is semantically ugly. We are already mixing info about the page (revision, timestamp, etc) with the item data. Adding meta-info about the file would be possible, but would increase the mess.

So we might want to introduce a similar envelop structure there - which would be a pretty huge breaking change to the interface we use to resolve URIs. That's not to be taken lightly. Even if we introduce version info into the URL,  we can't change the URIs, so clients would still get an unexpected data structure.

I'm afraid we painted ourselves into a corner here...TASK DETAILhttps://phabricator.wikimedia.org/T92961EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: danielCc: Pfps, aude, Smalyshev, Ricordisamoa, Addshore, Lydia_Pintscher, adrianheine, JanZerebecki, daniel, Tobi_WMDE_SW, Aklapper, D3r1ck01, Izno, Wikidata-bugs, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T92961: [Story] Versioning in JSON output

2016-08-23 Thread Pfps 
Pfps added a comment.
Right now, the JSON dump format is a sequence of JSON objects.  Each of these JSON objects is a Wikidata entity.  There is nothing preventing the dump format from having the first JSON object be information about the dump, including version of the dump format, version of wikidata format, time of dump, etc.

As long as this JSON object did not conform to the form of JSON objects that encode Wikidata entities this change would not be a breaking change!  (I do think that it would be better for it to not have any of the names that are currently being used in JSON objects that encode Wikidata entities.)TASK DETAILhttps://phabricator.wikimedia.org/T92961EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: PfpsCc: Pfps, aude, Smalyshev, Ricordisamoa, Addshore, Lydia_Pintscher, adrianheine, JanZerebecki, daniel, Tobi_WMDE_SW, Aklapper, D3r1ck01, Izno, Wikidata-bugs, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs