matthiasmullie added a comment.
I'm not really sure what number we want to go with, but I can probably help clarify what kind of data is in which db tables (and what numbers derived from those actually mean) So, it seems we have 2 completely separate definitions of "structured data": 1. MediaInfo entities created for file pages, with captions and/or statements 2. Existing "entities" (Wikidata or MediaInfo) pulled in via Lua to enrich (file) pages (e.g. given that we know the artwork, we can pull in author information etc.) And possibly, since IMO both are a valid definition of "structured data": 3. A combination of both: files with either a MediaInfo entity and/or other (Wikidata/MediaInfo) entities' information included via Lua **1. MediaInfo entities created for file pages, with captions and/or statements: 3 082 976** We can query for it by counting the amount of 'mediainfo' slots & excluding deleted pages & empty content: SELECT COUNT(DISTINCT page_id) # page excludes deleted pages (which are in archive) FROM page # joining on page_latest - we only care about most recent INNER JOIN slots ON slot_revision_id = page_latest # mediainfo slot must contain actual content INNER JOIN content ON slot_content_id = content_id AND content_size > 122 INNER JOIN slot_roles ON role_id = slot_role_id AND role_name = 'mediainfo'; //AFAICT, there is no easy way to break up stats about MediaInfo entities more (e.g. "how many have only captions", "how many have X number of statements", ...) - not with a simple query on the raw data anyway. The entity's actual data is in external store (just like any other wikitext page's content) in the form of a JSON blob (so would have to be deserialized)// **2. Existing "entities" (Wikidata or MediaInfo) pulled in via Lua to enrich file pages: 7 936 829** That data lives in `wbc_entity_usage`, which is a table much like `categorylinks` and `templatelinks`. It does not contain all existing entities (and their labels etc), it's just a place to store the relationship of entities and other pages these entities are used on (via Lua) See T231952#5717638 <https://phabricator.wikimedia.org/T231952#5717638> & Wikibase/Schema/wbc_entity_usage on mw.org <https://www.mediawiki.org/wiki/Wikibase/Schema/wbc_entity_usage> for more details on this table and what data is holds. Something like this should give us the total amount of file pages that are including Wikidata/MediaInfo data via Lua (overwhelming majority is Wikidata entities - it's not been possible to fetch MediaInfo entities via Lua until a month ago) SELECT COUNT(DISTINCT page_id) # page excludes deleted pages (which are in archive) FROM page INNER JOIN wbc_entity_usage ON eu_page_id = page_id # only include file pages and non-sitelink usage WHERE page_namespace = 6 AND eu_aspect != 'S'; Note that we might want to exclude Wikibase sitelinks usage <https://www.wikidata.org/wiki/Help:Sitelinks> (recorded in this table with eu_aspect = 'S'). While it's useful linking of data, I'm not these should be considered "structured data" usage. Anyway, their usage is negligible anyway: excluding those (`eu_aspect != 'S'`) returns a result of **7 935 849** (or only 980 files that *only* have a wikidata sitelink) //It is somewhat possible to break down stats further (e.g. how many are including a label vs how many are including statements), but not in too much detail (e.g. `L<lang id>` would be unreliable to figure out how much it's used in a particular language, because that language could also be covered by `L`// //FYI: there are about ~3M more (non-file) pages with structured data via Lua, mostly category pages.// **3. A combination of both: 10 440 129** This is pretty much just a combination of the above numbers, except that there is some overlap: entities could have MediaInfo entities as well as other data from Wikidata (and having MediaInfo entities is likely going to make it easier to fetch other related info, so I'd expect overlap to grow over time) Currently, there’s an overlap of (only) ~0.58M files (that have both an own MediaInfo entity, and existing Wikidata/MediaInfo usage via Lua) A simple union for both of the above queries should get us that data: SELECT COUNT(*) FROM ( SELECT DISTINCT page_id FROM page INNER JOIN slots ON slot_revision_id = page_latest INNER JOIN content ON slot_content_id = content_id AND content_size > 122 INNER JOIN slot_roles ON role_id = slot_role_id AND role_name = 'mediainfo' UNION SELECT DISTINCT page_id FROM page INNER JOIN wbc_entity_usage ON eu_page_id = page_id WHERE page_namespace = 6 ) AS t; TASK DETAIL https://phabricator.wikimedia.org/T239565 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Milimetric, matthiasmullie Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs