jenkins-bot has submitted this change and it was merged.
Change subject: Return more rows per query and use unbuffered query to manage
the memory
......................................................................
Return more rows per query and use unbuffered query to manage the memory
CRM-18128 The Limit was set in order to protect memory use. However, if means
running often slow queries many times for large exports .. trying another way
Bug: T120892
Change-Id: Ic30732a0bc38c92210f3953c7052bd7d2e597bd1
---
M CRM/Export/BAO/Export.php
1 file changed, 11 insertions(+), 6 deletions(-)
Approvals:
Awight: Looks good to me, approved
jenkins-bot: Verified
diff --git a/CRM/Export/BAO/Export.php b/CRM/Export/BAO/Export.php
index d74d1bf..fae116b 100644
--- a/CRM/Export/BAO/Export.php
+++ b/CRM/Export/BAO/Export.php
@@ -41,7 +41,7 @@
// increase this number a lot to avoid making too many queries
// LIMIT is not much faster than a no LIMIT query
// CRM-7675
- const EXPORT_ROW_COUNT = 10000;
+ const EXPORT_ROW_COUNT = 100000;
/**
* Get the list the export fields.
@@ -663,15 +663,17 @@
foreach ($returnProperties as $key => $value) {
$outputColumns[$key] = $value;
}
- while (1) {
+ $limitReached = FALSE;
+ while (!$limitReached) {
$limitQuery = "{$queryString} LIMIT {$offset}, {$rowCount}";
- $dao = CRM_Core_DAO::executeQuery($limitQuery);
- if ($dao->N <= 0) {
- break;
- }
+ $dao = CRM_Core_DAO::executeUnbufferedQuery($limitQuery);
+ // If this is less than our limit by the end of the iteration we do not
need to run the query again to
+ // check if some remain.
+ $rowsThisIteration = 0;
while ($dao->fetch()) {
$count++;
+ $rowsThisIteration ++;
$row = array();
//convert the pseudo constants
@@ -1124,6 +1126,9 @@
}
}
$dao->free();
+ if ($rowsThisIteration < self::EXPORT_ROW_COUNT) {
+ $limitReached = TRUE;
+ }
$offset += $rowCount;
}
--
To view, visit https://gerrit.wikimedia.org/r/274061
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic30732a0bc38c92210f3953c7052bd7d2e597bd1
Gerrit-PatchSet: 2
Gerrit-Project: wikimedia/fundraising/crm/civicrm
Gerrit-Branch: master
Gerrit-Owner: Eileen <emcnaugh...@wikimedia.org>
Gerrit-Reviewer: Awight <awi...@wikimedia.org>
Gerrit-Reviewer: Ejegg <eeggles...@wikimedia.org>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
