[Wikidata-bugs] [Maniphest] [Commented On] T249587: Wikidata API wbgetentities: Allowed memory size exceeded

2020-05-07 Thread Krinkle 
Krinkle added a comment.


--- a/includes/api/ApiResult.php
+++ b/includes/api/ApiResult.php
@@ -396,9 +396,15 @@ class ApiResult implements ApiSerializable {
if ( !( $flags & self::NO_SIZE_CHECK ) ) {
// self::size needs the validated value. Then flag
// to not re-validate later.
+   $start = microtime( true );
$value = self::validateValue( $value );
+   $duration = microtime( true ) - $start;
$flags |= self::NO_VALIDATE;
 
+   if ( @$_GET['krinkle'] ) {
+   wfDebugLog( 'AdHocDebug', 'Spent ' . round( $duration * 
1000, 3 ) . ' ms in ' . __METHOD__ );
+   }
  
  Two runs:
  
  - Run 1
- Total time in PHP: 97 ms
- Time in ApiResult::validateValue (from ApiResult::addValue): ~ 4.8ms (5 
top-level calls)
  
  - Run 1
- Total time in PHP: 94 ms
- Time in ApiResult::validateValue (from ApiResult::addValue): ~ 3.7ms (5 
top-level calls)
  
  name=Raw data
krinkle@mw1378:/srv/mediawiki/php-1.35.0-wmf.31/includes/api$ curl 
'http://mw1378.eqiad.wmnet/w/api.php?action=wbgetentities=Q12184|Q17076801=info|aliases|labels|descriptions|claims|sitelinks|datatype=json=1'
 -i -H 'Host: www.wikidata.org' | grep Backend-Timing
Backend-Timing: D=97086 t=1588895607974196

timestamp:  2020-05-07T23:53:28+00:00
reqId: XrSfdwpAMNwAA5U6DdIAAABU

- Spent 3.655 ms in ApiResult::addValue
- Spent 0.002 ms in ApiResult::addValue
- Spent 1.088 ms in ApiResult::addValue
- Spent 0.001 ms in ApiResult::addValue
- Spent 0.001 ms in ApiResult::addValue


krinkle@mw1378:/srv/mediawiki/php-1.35.0-wmf.31/includes/api$ curl 
'http://mw1378.eqiad.wmnet/w/api.php?action=wbgetentities=Q12184|Q17076801=info|aliases|labels|descriptions|claims|sitelinks|datatype=json=1'
 -i -H 'Host: www.wikidata.org' | grep Backend-Timing
Backend-Timing: D=93593 t=1588895609066951

timestamp:  2020-05-07T23:53:29+00:00
reqId: XrSfeQpAMNwAA5U6Df4AAABX

- Spent 3.598 ms in ApiResult::addValue
- Spent 0.002 ms in ApiResult::addValue
- Spent 1.028 ms in ApiResult::addValue
- Spent 0.001 ms in ApiResult::addValue
- Spent 0.001 ms in ApiResult::addValue

TASK DETAIL
  https://phabricator.wikimedia.org/T249587

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Krinkle
Cc: Krinkle, WMDE-leszek, Ladsgroup, Lea_Lacroix_WMDE, Addshore, Ixocactus, 
Tarrow, Magnus, Aklapper, darthmon_wmde, Nandana, Lahi, Gq86, 
GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T249587: Wikidata API wbgetentities: Allowed memory size exceeded

2020-05-07 Thread Addshore 
Addshore added a comment.


  I got the memory and time limits removed on mwdebug1002 so that we could get 
a full profile.
  
  https://performance.wikimedia.org/xhgui/run/view?id=5eb3c4dbbb8544687a18a8f1
  https://performance.wikimedia.org/xhgui/run/view?id=5eb3c579bb8544737a2869da
  https://performance.wikimedia.org/xhgui/run/view?id=5eb3c5fcbb8544767a7343ce
  
  Which shows the full results is over 2x as bad as the first set of profiles 
when actually complete!

TASK DETAIL
  https://phabricator.wikimedia.org/T249587

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: WMDE-leszek, Ladsgroup, Lea_Lacroix_WMDE, Addshore, Ixocactus, Tarrow, 
Magnus, Aklapper, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, 
aude, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T249587: Wikidata API wbgetentities: Allowed memory size exceeded

2020-04-27 Thread Ladsgroup 
Ladsgroup added a comment.


  Currently, EntityId stores everything twice, once as `$serialization` (e.g. 
`commons:M12`) and once as combination of both: (`$repositoryName` + 
`$localPart`).  Fixing this would drop the memory footprint of Wikidata.

TASK DETAIL
  https://phabricator.wikimedia.org/T249587

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup
Cc: Ladsgroup, Lea_Lacroix_WMDE, Addshore, Ixocactus, Tarrow, Magnus, Aklapper, 
darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, 
Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T249587: Wikidata API wbgetentities: Allowed memory size exceeded

2020-04-27 Thread Ladsgroup 
Ladsgroup added a comment.


  In T249587#6078864 , 
@Addshore wrote:
  
  > One thing to note in the profiling is that a significant portion of the 
time that the query is executing is spent in a section of code in 
ApiResult::addValue, specifically the validateValue and size methods.
  > I don't think we really need to do this, but we should consult with the 
core platform team and potentially remove this from the execution path for our 
apis?
  > An equal amount of time seems to be spent in 
Wikibase\Lib\Serialization\SerializationModifier::modifyUsingCallback which 
could likely do with some optimizations.
  > Noticeably in this call property id creation seems to take quite some time, 
it looks like the bulk of this overhead is due to joining and splitting around 
id prefixes, which we do not actually use.
  > @Ladsgroup recently filed a ticket about the performance of EntityId and 
ItemId etc.
  > I think we could make an improvement here probably by not doing so much 
logic in the constructor when we don't need to.
  > Perhap we can also just remove all of this id prefix stuff for now as we 
don't use it.
  > It'll be good to try and get a full profile of a completed execution.
  > To do that we will have to see if we can temporarily remove the timeline 
restriction on one of the mwdebug servers.
  
  Thank you for running the performance inspections and it's very useful but we 
should check memory instead of time, they actually have a trade-off. For 
example if we try to to cache property objects so it wouldn't spend time 
checking the regex, it would be faster but it would use more even memory and 
worsens the situation mentioned in this ticket. Checking the memory explosions:
  
Wikibase\Lib\Serialization\SerializationModifier::getPathParts
138,755,064 bytes
Wikibase\DataModel\Entity\EntityId::extractRepositoryNameAndLocalPart
73,650,136 bytes
Wikibase\DataModel\Entity\EntityId::extractSerializationParts
71,232,576 bytes
Wikibase\DataModel\Snak\PropertyValueSnak::unserialize
66,952,800 bytes

Wikibase\Lib\Serialization\CallbackFactory::Wikibase\Lib\Serialization\{closure}
64,796,608 bytes
Wikibase\DataModel\Snak\SnakList::unserialize
  
  I guess data model needs a little tuning in storing values.

TASK DETAIL
  https://phabricator.wikimedia.org/T249587

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup
Cc: Ladsgroup, Lea_Lacroix_WMDE, Addshore, Ixocactus, Tarrow, Magnus, Aklapper, 
darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, 
Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T249587: Wikidata API wbgetentities: Allowed memory size exceeded

2020-04-22 Thread Addshore 
Addshore added a comment.


  So I tried the query on the debug servers with profiling on and I hit 
timeouts instead of memory limit.
  But generally at the time the requests hit the timeout, the memory limit is 
quite high.
  
  https://performance.wikimedia.org/xhgui/run/view?id=5ea0c61e3f3dfa135999325b
  https://performance.wikimedia.org/xhgui/run/view?id=5ea0c5ab3f3dfa1a59c01fa7
  https://performance.wikimedia.org/xhgui/run/view?id=5ea0c7106318cbb47153e294
  
  When running on the debug servers without profiling I end up hitting into the 
memory limit.

TASK DETAIL
  https://phabricator.wikimedia.org/T249587

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: Lea_Lacroix_WMDE, Addshore, Ixocactus, Tarrow, Magnus, Aklapper, 
darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, 
Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T249587: Wikidata API wbgetentities: Allowed memory size exceeded

2020-04-14 Thread Magnus 
Magnus added a comment.


  Still happening.

TASK DETAIL
  https://phabricator.wikimedia.org/T249587

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Magnus
Cc: Lea_Lacroix_WMDE, Addshore, Ixocactus, Tarrow, Magnus, Aklapper, 
darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T249587: Wikidata API wbgetentities: Allowed memory size exceeded

2020-04-07 Thread Magnus 
Magnus added a comment.


  Maybe this helps:
  F31740598: Screenshot 2020-04-07 at 15.24.56.png 


TASK DETAIL
  https://phabricator.wikimedia.org/T249587

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Magnus
Cc: Lea_Lacroix_WMDE, Addshore, Ixocactus, Tarrow, Magnus, Aklapper, 
darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T249587: Wikidata API wbgetentities: Allowed memory size exceeded

2020-04-07 Thread Magnus 
Magnus added a comment.


  Yes, still seeing it. Runs for ~20sec so probably not a cache. Changed the 
JSONP callback(s) and still same error, so definitely no content-level cache.

TASK DETAIL
  https://phabricator.wikimedia.org/T249587

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Magnus
Cc: Lea_Lacroix_WMDE, Addshore, Ixocactus, Tarrow, Magnus, Aklapper, 
darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs