[
https://issues.apache.org/jira/browse/SVN-4796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16714633#comment-16714633
]
Julian Foad commented on SVN-4796:
----------------------------------
Thank you for the detailed report.
Here is your test function converted to a runnable test in
subversion/tests/libsvn_repos/repos-test.c:
[^issue-4796-test-1.patch]
I do not see the same memory growth. I see the memory (reported by 'top') for
this test process start at roughly 22 MB resident / 70 MB virtual and grow
steadily to 29 MB resident / 77 MB virtual after 5000 iterations. It does slow
down progressively.
This was svn 1.11.0 on Ubuntu 16.04, gcc 5.4.0, apr 1.5.2, tmpfs ramdisk.
Please could you write to the dev@ mailing list to discuss this.
> SVN leaks exponentially significant amount of memory on commit
> --------------------------------------------------------------
>
> Key: SVN-4796
> URL: https://issues.apache.org/jira/browse/SVN-4796
> Project: Subversion
> Issue Type: Bug
> Components: libsvn_delta, libsvn_fs, libsvn_fs_fs, libsvn_fs_x,
> libsvn_repos, libsvn_subr
> Affects Versions: 1.11.0, 1.10.2
> Environment: Test environment:
> * Linux (Ubuntu 18.04.1 LTS)
> * 8 & 16GB RAM dev environments
> * GCC/G++ 7.3.0
> * Apache Portable Runtime Library, version 1.6.3-2 amd64
> * Repositories are created on EXT4 or XFS
> * Repositories are created on SSD
> * SVN repositories` type is FSFS
> * SVN repositories` file system version is the latest for the respective API
> Reporter: BINETIX SUPPORT TEAM
> Priority: Critical
> Labels: features, ready-to-commit, usability
> Attachments: issue-4796-test-1.patch
>
>
> This is a clearly reproducible problem on SVN versions 1.10.2 and the latest
> 1.11.0.
> We found two similar issues SVN-1964, SVN-3411 closed already, but obviously
> beyond our case. So, these are just mentioned for reference.
> *Description*
> We're going to use the SVN API in order to create and then commit multiple
> transactions with multiple files. In order to simplify the scenario, within
> the code below, we will apply only 3 actions per transaction:
> # Create a transaction
> # Add only one directory or one empty file (file with content makes no
> difference)
> # Commit the transaction
> Now let`s loop many transactions like this one in sequence (no concurrent
> usage).
> After each commit SVN consumes exponentially amount of memory which is never
> released afterwords, as follows (Number of transaction - Application's memory
> usage):
> 0 - 380 kb (initial state when the application is started with 0 transactions)
> 1`000 - 20 Mb
> 2`000 - 66 Mb
> 3`000 - 120 Mb
> 4`000 - 210 Mb
> 5`000 - 320 Mb
> 6`000 - 455 Mb
> 7`000 - 607 Mb
> 8`000 - 808 Mb
> 9`000 - 1.1 Gb
> 10`000 - 1.4 Gb
> –
> *Observations*
> Few curious facts observed during the tests:
> * If we delete the added file before committing, the transaction remains
> empty, then there is no memory leakage;
> * If we add not only one file or dir but 40`000 - 48`000 entities in one
> transaction then the files are being created VERY slowly, and the memory
> consumption is doubled or even tippled compared to the table above;
> * If we create and commit 10`000 entities (in our example) in the root
> directory (it is important to do that in the root), and we stop the
> application, then on the next application start with the same repository
> there are three cases:
> ** if we add more entities in the root again, then the memory leakage is
> increased with the same proportions, as described above, BUT,
> ** if we add more entities in subdirectory, then the memory allocation is
> stopped by 20 Mb until a threshold of about 9`000 files is reached. Above
> this relative threshold, if we add more entities in the same subdirectory
> then the memory leakage is increased again, AND FINALLY,
> ** if we add more entities in any other subdirectory below the described
> relative threshold, then there is no leakage observed. In other words, we
> create (relatively speaking) 10`000 files in the root, then we can add up to
> 9`000 in each subdirectory without memory leakage;
> * On Transaction destroy there is no memory leakage;
> Despite these observations and strange "workarounds", obviously we CAN NOT
> use Subversion in productions conditions. So we hope you will find a way to
> fix these issues.
> –
> *Sample code for reproduction*
> _#define_ _MAX_LOOP 10000_
> _svn_error_t* test_func(const char* _szRepository_path_
> _, apr_pool_t* pool)_
> _{_
> _svn_repos_t* repos;_
> _svn_fs_t* fs;_
> _SVN_ERR(svn_repos_open3(&repos, _szRepository_path, NULL, pool, pool));_
> _fs = svn_repos_fs(repos);_
> _apr_pool_t* subpool = svn_pool_create(pool);_
> _for (int i = 0; i < MAX_LOOP; i++) {_
> _svn_fs_txn_t* txn;_
> _svn_fs_root_t* txn_root;_
> _svn_revnum_t youngest;_
> _const char* txn_id = NULL;_
>
> _svn_pool_clear(subpool);_
>
> _SVN_ERR(svn_fs_youngest_rev(&youngest, fs, subpool));_
> _SVN_ERR(svn_repos_fs_begin_txn_for_commit (&txn_
> _, repos_
> _, youngest_
> _, "user_name"_
> _, "comment default"_
> _, subpool));_
> _//---- Create unique file name and add to "/" in repository_
> _SVN_ERR(svn_fs_txn_root(&txn_root, txn, subpool));_
> _const char* pszFile_name = create_unique_file_name (subpool);_
> _SVN_ERR(svn_fs_make_file(txn_root, pszFile_name, subpool));_
> _if ((i % 100) == 0)_
> _printf ("added: %d\n", i+1);_
> _const char* conflict = NULL;_
> _svn_revnum_t new_rev = SVN_INVALID_REVNUM;_
>
> _SVN_ERR(svn_fs_commit_txn(&conflict, &new_rev, txn, subpool));_
> _if (!SVN_IS_VALID_REVNUM(new_rev))_
> _return svn_error_create(SVN_ERR_FS_GENERAL, NULL,_
> _"Commit failed...");_
> _}_
> _svn_pool_destroy(subpool);_
> _return SVN_NO_ERROR;_
> _}_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
