[Wikidata-bugs] [Maniphest] [Commented On] T68108: [Epic] Store media information for files on Wikimedia Commons as structured data

2019-06-19 Thread jcrespo 
jcrespo added a comment.


  So just to be clear- based on growth projections I recently got from SDC team 
of wikibase on commons, the separation is not only convenient or needed for 
performance, literally s4 would not be able to fit except the initial 
deployment of data, or do it for very short term. Disk usage is close to 2 TB 
right now, with and additional 2TB of structured data (metadata only). IOPS 
would be close to wikidata, which requires a dedicated cluster. This is based 
on their statistics and projected growth, on top of the current growth and 
usage. We can scale those out with already budgeted hw, but we need software 
support. Extra content data would not be a concern as we already planned ES 
expansion for next fiscal.
  
  I think we should plan a bit the different high level tasks related to the 
database, to schedule them properly, both the ones that directly affect SDC 
(implementation, wikibase dedicated db, wb_terms migration, MCR) and the ones 
that affect it indirectly by increase or release of used resources (actor, 
comment, *links migrations).

TASK DETAIL
  https://phabricator.wikimedia.org/T68108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: jcrespo
Cc: Ladsgroup, MarkTraceur, WMDE-leszek, jcrespo, Marostegui, 
AfroThundr3007730, Stashbot, _jensen, SandraF_WMF, Ramsey-WMF, CCicalese_WMF, 
PokestarFan, Saerdnaer, Juandev, Elitre, Wesalius, Zppix, NMaia, 
Mattias_Ostmar-WMSE, Sadads, Poyekhali, -jem-, Deskana, Tfinc, Smalyshev, 
Jheald, LikeLifer, Yann, intracer, Spinster, Orofarne, Filceolaire, MZMcBride, 
bzimport, TheDJ, zhuyifei1999, DixonD, Bugreporter, RP88, Aklapper, Matanya, 
waldyrious, El_Grafo, Daniel_Mietchen, Jdforrester-WMF, GPHemsley, Bene, 
Legoktm, Nemo_bis, Lokal_Profil, Tobi_WMDE_SW, He7d3r, Petrb, Gilles, jayvdb, 
Kelson, Steinsplitter, JeroenDeDauw, iecetcwcpggwqpgciazwvzpfjpwomjxn, revi, 
JanZerebecki, JeanFred, Ricordisamoa, Snowolf, Keegan, Rillke, Bawolff, 
Fabrice_Florin, Multichill, Liuxinyu970226, Ainali, Tgr, Lydia_Pintscher, 
jeremyb, Stryn, Ltrlg, daniel, Dereckson, JohnLewis, darthmon_wmde, Nandana, 
JKSTNK, Lahi, PDrouin-WMF, Gq86, E1presidente, Cparle, Anooprao, 
GoranSMilovanovic, QZanden, Tramullas, Acer, V4switch, LawExplorer, Salgo60, 
Silverfish, rosalieper, Taiwania_Justo, Susannaanas, Wong128hk, Jane023, 
Wikidata-bugs, Base, matthiasmullie, aude, Dinoguy1000, Raymond, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T68108: [Epic] Store media information for files on Wikimedia Commons as structured data

2019-06-19 Thread WMDE-leszek 
WMDE-leszek added a comment.


  I believe the only code change that Wikibase might be facing might then be a 
new to have a DB connection for wb_terms table server, and the DB connection 
for the regular mw tables server. This is not going to be rocket science, but 
also not something that would just work out of the box.

TASK DETAIL
  https://phabricator.wikimedia.org/T68108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: WMDE-leszek
Cc: Ladsgroup, MarkTraceur, WMDE-leszek, jcrespo, Marostegui, 
AfroThundr3007730, Stashbot, _jensen, SandraF_WMF, Ramsey-WMF, CCicalese_WMF, 
PokestarFan, Saerdnaer, Juandev, Elitre, Wesalius, Zppix, NMaia, 
Mattias_Ostmar-WMSE, Sadads, Poyekhali, -jem-, Deskana, Tfinc, Smalyshev, 
Jheald, LikeLifer, Yann, intracer, Spinster, Orofarne, Filceolaire, MZMcBride, 
bzimport, TheDJ, zhuyifei1999, DixonD, Bugreporter, RP88, Aklapper, Matanya, 
waldyrious, El_Grafo, Daniel_Mietchen, Jdforrester-WMF, GPHemsley, Bene, 
Legoktm, Nemo_bis, Lokal_Profil, Tobi_WMDE_SW, He7d3r, Petrb, Gilles, jayvdb, 
Kelson, Steinsplitter, JeroenDeDauw, iecetcwcpggwqpgciazwvzpfjpwomjxn, revi, 
JanZerebecki, JeanFred, Ricordisamoa, Snowolf, Keegan, Rillke, Bawolff, 
Fabrice_Florin, Multichill, Liuxinyu970226, Ainali, Tgr, Lydia_Pintscher, 
jeremyb, Stryn, Ltrlg, daniel, Dereckson, JohnLewis, darthmon_wmde, Nandana, 
JKSTNK, Lahi, PDrouin-WMF, Gq86, E1presidente, Cparle, Anooprao, 
GoranSMilovanovic, QZanden, Tramullas, Acer, V4switch, LawExplorer, Salgo60, 
Silverfish, rosalieper, Taiwania_Justo, Susannaanas, Wong128hk, Jane023, 
Wikidata-bugs, Base, matthiasmullie, aude, Dinoguy1000, Raymond, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T68108: [Epic] Store media information for files on Wikimedia Commons as structured data

2019-06-19 Thread WMDE-leszek 
WMDE-leszek added a comment.


  In T68108#5267982 , 
@WMDE-leszek wrote:
  
  > In T68108#5262336 , 
@jcrespo wrote:
  >
  > > > The move would likely require some changes to the Wikibase code
  > >
  > > Could you clarify why? As all other hosts seem to be ok with wikibase 
server for wikidata being on a separate database? Does MCR use wikibase 
differently or is it something else? Maybe having 2 wikibase services to use? 
Note again we notified of this need months in advance during planning phase, as 
new features require usually extra resources.
  >
  >
  > What I had in mind is wb_terms (and its successor being introduced 
currently) are also used in queries that do joins with Mediawiki's "standard" 
tables, e.g. page or revision. This code/queries will not continue to work if 
the wb_terms table of commons gets moved to another server than the one mw 
tables are.
  
  
  I was just corrected by the WMDE colleague the above point about joins is not 
correct, there are no such joins

TASK DETAIL
  https://phabricator.wikimedia.org/T68108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: WMDE-leszek
Cc: Ladsgroup, MarkTraceur, WMDE-leszek, jcrespo, Marostegui, 
AfroThundr3007730, Stashbot, _jensen, SandraF_WMF, Ramsey-WMF, CCicalese_WMF, 
PokestarFan, Saerdnaer, Juandev, Elitre, Wesalius, Zppix, NMaia, 
Mattias_Ostmar-WMSE, Sadads, Poyekhali, -jem-, Deskana, Tfinc, Smalyshev, 
Jheald, LikeLifer, Yann, intracer, Spinster, Orofarne, Filceolaire, MZMcBride, 
bzimport, TheDJ, zhuyifei1999, DixonD, Bugreporter, RP88, Aklapper, Matanya, 
waldyrious, El_Grafo, Daniel_Mietchen, Jdforrester-WMF, GPHemsley, Bene, 
Legoktm, Nemo_bis, Lokal_Profil, Tobi_WMDE_SW, He7d3r, Petrb, Gilles, jayvdb, 
Kelson, Steinsplitter, JeroenDeDauw, iecetcwcpggwqpgciazwvzpfjpwomjxn, revi, 
JanZerebecki, JeanFred, Ricordisamoa, Snowolf, Keegan, Rillke, Bawolff, 
Fabrice_Florin, Multichill, Liuxinyu970226, Ainali, Tgr, Lydia_Pintscher, 
jeremyb, Stryn, Ltrlg, daniel, Dereckson, JohnLewis, darthmon_wmde, Nandana, 
JKSTNK, Lahi, PDrouin-WMF, Gq86, E1presidente, Cparle, Anooprao, 
GoranSMilovanovic, QZanden, Tramullas, Acer, V4switch, LawExplorer, Salgo60, 
Silverfish, rosalieper, Taiwania_Justo, Susannaanas, Wong128hk, Jane023, 
Wikidata-bugs, Base, matthiasmullie, aude, Dinoguy1000, Raymond, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T68108: [Epic] Store media information for files on Wikimedia Commons as structured data

2019-06-19 Thread WMDE-leszek 
WMDE-leszek added a comment.


  In T68108#5262336 , 
@jcrespo wrote:
  
  > > The move would likely require some changes to the Wikibase code
  >
  > Could you clarify why? As all other hosts seem to be ok with wikibase 
server for wikidata being on a separate database? Does MCR use wikibase 
differently or is it something else? Maybe having 2 wikibase services to use? 
Note again we notified of this need months in advance during planning phase, as 
new features require usually extra resources.
  
  
  What I had in mind is wb_terms (and its successor being introduced currently) 
are also used in queries that do joins with Mediawiki's "standard" tables, e.g. 
page or revision. This code/queries will not continue to work if the wb_terms 
table of commons gets moved to another server than the one mw tables are.

TASK DETAIL
  https://phabricator.wikimedia.org/T68108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: WMDE-leszek
Cc: Ladsgroup, MarkTraceur, WMDE-leszek, jcrespo, Marostegui, 
AfroThundr3007730, Stashbot, _jensen, SandraF_WMF, Ramsey-WMF, CCicalese_WMF, 
PokestarFan, Saerdnaer, Juandev, Elitre, Wesalius, Zppix, NMaia, 
Mattias_Ostmar-WMSE, Sadads, Poyekhali, -jem-, Deskana, Tfinc, Smalyshev, 
Jheald, LikeLifer, Yann, intracer, Spinster, Orofarne, Filceolaire, MZMcBride, 
bzimport, TheDJ, zhuyifei1999, DixonD, Bugreporter, RP88, Aklapper, Matanya, 
waldyrious, El_Grafo, Daniel_Mietchen, Jdforrester-WMF, GPHemsley, Bene, 
Legoktm, Nemo_bis, Lokal_Profil, Tobi_WMDE_SW, He7d3r, Petrb, Gilles, jayvdb, 
Kelson, Steinsplitter, JeroenDeDauw, iecetcwcpggwqpgciazwvzpfjpwomjxn, revi, 
JanZerebecki, JeanFred, Ricordisamoa, Snowolf, Keegan, Rillke, Bawolff, 
Fabrice_Florin, Multichill, Liuxinyu970226, Ainali, Tgr, Lydia_Pintscher, 
jeremyb, Stryn, Ltrlg, daniel, Dereckson, JohnLewis, darthmon_wmde, Nandana, 
JKSTNK, Lahi, PDrouin-WMF, Gq86, E1presidente, Cparle, Anooprao, 
GoranSMilovanovic, QZanden, Tramullas, Acer, V4switch, LawExplorer, Salgo60, 
Silverfish, rosalieper, Taiwania_Justo, Susannaanas, Wong128hk, Jane023, 
Wikidata-bugs, Base, matthiasmullie, aude, Dinoguy1000, Raymond, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T68108: [Epic] Store media information for files on Wikimedia Commons as structured data

2019-06-17 Thread jcrespo 
jcrespo added a comment.


  > The move would likely require some changes to the Wikibase code
  
  Could you clarify why? As all other hosts seem to be ok with wikibase server 
for wikidata being on a separate database? Does MCR use wikibase differently or 
is it something else? Maybe having 2 wikibase services to use? Note again we 
notified of this need months in advance during planning phase, as new features 
require usually extra resources.
  
  > Wikibase-specific tables
  
  there is wbc_changes, and maybe other wikibase client tables- we don't have a 
problem with those, as those exist locally on all (wikidata-enabled) wikis. The 
ones we are worried about are the wikibase server ones (aka the equivalent of 
s8 on s4). I am not sure we should wait for the refactoring, but as long as the 
tables on s4 are empty or almost empty (with I think a single row on 
wb_id_counters), we are flexible. What we don't want is lots of data there that 
later is more complex to migrate away (we are assuming there will be a large 
amount of updates there when SDC is at full steam).
  
  A few hosts were budgeted for the split for FY2019-2020, we are not in a 
rush, but it would be nice to have some planning in place for next fiscal year 
so there are no unexpected delays, specially given persistence team is at 50% 
capacity ATM.

TASK DETAIL
  https://phabricator.wikimedia.org/T68108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: jcrespo
Cc: Ladsgroup, MarkTraceur, WMDE-leszek, jcrespo, Marostegui, 
AfroThundr3007730, Stashbot, _jensen, SandraF_WMF, Ramsey-WMF, CCicalese_WMF, 
PokestarFan, Saerdnaer, Juandev, Elitre, Wesalius, Zppix, NMaia, 
Mattias_Ostmar-WMSE, Sadads, Poyekhali, -jem-, Deskana, Tfinc, Smalyshev, 
Jheald, LikeLifer, Yann, intracer, Spinster, Orofarne, Filceolaire, MZMcBride, 
bzimport, TheDJ, zhuyifei1999, DixonD, Bugreporter, RP88, Aklapper, Matanya, 
waldyrious, El_Grafo, Daniel_Mietchen, Jdforrester-WMF, GPHemsley, Bene, 
Legoktm, Nemo_bis, Lokal_Profil, Tobi_WMDE_SW, He7d3r, Petrb, Gilles, jayvdb, 
Kelson, Steinsplitter, JeroenDeDauw, iecetcwcpggwqpgciazwvzpfjpwomjxn, revi, 
JanZerebecki, JeanFred, Ricordisamoa, Snowolf, Keegan, Rillke, Bawolff, 
Fabrice_Florin, Multichill, Liuxinyu970226, Ainali, Tgr, Lydia_Pintscher, 
jeremyb, Stryn, Ltrlg, daniel, Dereckson, JohnLewis, darthmon_wmde, Nandana, 
JKSTNK, Lahi, PDrouin-WMF, Gq86, E1presidente, Cparle, Anooprao, 
GoranSMilovanovic, QZanden, Tramullas, Acer, V4switch, LawExplorer, Salgo60, 
Silverfish, rosalieper, Taiwania_Justo, Susannaanas, Wong128hk, Jane023, 
Wikidata-bugs, Base, matthiasmullie, aude, Dinoguy1000, Raymond, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T68108: [Epic] Store media information for files on Wikimedia Commons as structured data

2019-01-09 Thread Stashbot 
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2019-01-09T21:04:30Z]  Creating Wikibase repo tables on Commons for T68108TASK DETAILhttps://phabricator.wikimedia.org/T68108EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: StashbotCc: Stashbot, _jensen, SandraF_WMF, Ramsey-WMF, CCicalese_WMF, PokestarFan, Saerdnaer, Juandev, Elitre, Wesalius, Zppix, NMaia, Mattias_Ostmar-WMSE, Sadads, Poyekhali, -jem-, Deskana, Tfinc, Smalyshev, Jheald, LikeLifer, Yann, intracer, Spinster, Orofarne, Filceolaire, MZMcBride, bzimport, TheDJ, zhuyifei1999, DixonD, Bugreporter, RP88, Aklapper, Matanya, waldyrious, El_Grafo, Daniel_Mietchen, Jdforrester-WMF, GPHemsley, Bene, Legoktm, Nemo_bis, Lokal_Profil, Tobi_WMDE_SW, He7d3r, Petrb, Gilles, jayvdb, Kelson, Steinsplitter, JeroenDeDauw, Micru, revi, JanZerebecki, JeanFred, Ricordisamoa, Snowolf, Keegan, Rillke, Bawolff, Fabrice_Florin, Multichill, Liuxinyu970226, Ainali, Tgr, Lydia_Pintscher, jeremyb, Stryn, Ltrlg, daniel, Dereckson, JohnLewis, Nandana, JKSTNK, Lahi, PDrouin-WMF, Gq86, E1presidente, Cparle, Anooprao, GoranSMilovanovic, QZanden, Tramullas, Acer, V4switch, LawExplorer, Silverfish, D3r1ck01, Susannaanas, Wong128hk, Jane023, Wikidata-bugs, Base, matthiasmullie, aude, Dinoguy1000, Raymond, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs