Re: [basex-talk] Gravierende Performance-Einbüße bei Persistierung von mehr als 5000, 160 KB große XML Datenstrukturen.
Hi Lucian, Thanks for taking your time, rerunning the tests and do some profiling. > When inserting a 160 KB xml structure 100.000 times, the persist operation > duration starts by ~45 ms and reach ~2000 ms after 68.000 persist > invocations and 16 hours of run time (!) Indeed this differs quite a lot from the tests I have made so far, and from the patterns I am used to. It was helpful to have a look into the Java profiling files: A plain FileOutputStream.open call takes most of the time, while it’s hardly measurable in my own tests. Do you work with a local file system? Maybe the file listing of your database directory could shed some more light here. I would additionally assume that this that you were closing and opening your database after each addition, right? Obviously this makes sense if no bulk operations take place; it’s just different from what I did in my tests. > There are actually two AUTOFLUSH-related issues: […] You are obviously right: AUTOFLUSH should only be used for bulk operations, and avoided if persistency of data is critical. And the addition of documents will always be faster if the database is small (but it should definitely not take more than a second to add a single small document). Best, Christian
Re: [basex-talk] Gravierende Performance-Einbüße bei Persistierung von mehr als 5000, 160 KB große XML Datenstrukturen.
Possibly related, but I'm not sure: When creating millions of databases in a loop in the same session, I found that after some thousands I'd get an OOM error by BaseX. This seemed odd to me, because after each iteration, the database creation query was closed (and I'd expect GC to run at such a time?). To by-pass this I just closed the session and opened a new one each couple of thousand-th time in the loop. Maybe there is a (small) memory leak somewhere in BaseX that only becomes noticeable (and annoying) after hundreds of thousands of even millions of queries? -Oorspronkelijk bericht- Van: basex-talk-boun...@mailman.uni-konstanz.de [mailto:basex-talk-boun...@mailman.uni-konstanz.de] Namens Christian Grün Verzonden: zaterdag 14 januari 2017 12:09 Aan: Bularca, Lucian <lucian.bula...@mueller.de> CC: basex-talk@mailman.uni-konstanz.de Onderwerp: Re: [basex-talk] Gravierende Performance-Einbüße bei Persistierung von mehr als 5000, 160 KB große XML Datenstrukturen. Hi Lucian, I have a hard time reproducing the reported behavior. The attached, revised Java example (without AUTOFLUSH) required around 30 ms for the first documents and 120 ms for the last documents, which is still pretty far from what you’ve been encountering: > von Anfang ~ 10 ms auf ~ 2500 ms kommne würde But obviously something weird has been going on in your setup. Let’s see what alternatives we have… • Could you possibly try to update my example code such that it shows the reported behavior? Ideally with small input, in order to speed up the process. Maybe the runtime increase can also be demonstrated after 1.000 or 10.000 documents... • You could also send me a list of the files of your test_database directory; maybe the file sizes indicate some unusual patterns. • You could start BaseXServer with the JVM flag -Xrunhprof:cpu=samples (to be inserted in the basexserver script), start the server, run your script, stop the server directly afterwards, and send me the result file, which will be stored in the directory from where you started BaseX (java.hprof.txt). Best, Christian On Wed, Jan 11, 2017 at 4:57 PM, Christian Grün <christian.gr...@gmail.com> wrote: > Hi Lucian, > > Thanks for your analysis. Indeed I’m wondering about the monotonic > delay caused by auto flushing the data; this hasn’t always been the > case. I’m wondering even more why no one else noticed this in recent > time.. Maybe it’s not too long ago that this was introduced. It may > take some time to find the culprit, but I’ll keep you updated. > > All the best, > Christian > > > On Wed, Jan 11, 2017 at 2:46 PM, Bularca, Lucian > <lucian.bula...@mueller.de> wrote: >> Hi Christian, >> >> I've made a comparation of the persistence time series running your example >> code and mine, in all possible combinations of following scenarios: >> - with and without "set intparse on" >> - using my prepared test data and your test data >> - closing and opening the DB connection each "n"-th insertion >> operation (where n in {5, 100, 500, 1000}) >> - with and without "set autoflush on". >> >> I finally found out, that the only relevant variable that influence the >> insert operation duration is the value of the AUTOFLASH option. >> >> If AUTOFLASH = OFF when opening a database, then the persistence durations >> remains relative constant (on my machine about 43 ms) during the entire >> insert operations sequence (50.000 or 100.000 times), for all possible >> combinations named above. >> >> If AUTOFLASH = ON when opening a database, then the persistence durations >> increase monotonic, for all possible combinations named above. >> >> The persistence duration, if AUTOFLASH = ON, is directly proportional to the >> number of DB clients executing these insert operations, respectively to the >> sequence length of insert operations executed by a DB client. >> >> In my opinion, this behaviour is an issue of BaseX, because AUTOFLASH is >> implcitly set to ON (see BaseX documentation >> http://docs.basex.org/wiki/Options#AUTOFLUSH), so DB clients must explicitly >> set AUTOFLASH = OFF in order to keep the insert operation durations >> relatively constant over time. Additionally, no explicitly flushing data, >> increases the risk of data loss (see BaseX documentation >> http://docs.basex.org/wiki/Options#AUTOFLUSH), but clients how repeatedly >> execute the FLUSH command increase the durations of the subsequent insert >> operations. >> >> Regards, >> Lucian >> >> ____________ >> Von: Christian Grün [christian.gr...@gmail.com] >> Gesendet: Dienstag, 10. Januar 2017 17:33 >> A
Re: [basex-talk] Gravierende Performance-Einbüße bei Persistierung von mehr als 5000, 160 KB große XML Datenstrukturen.
Hi Lucian, I have a hard time reproducing the reported behavior. The attached, revised Java example (without AUTOFLUSH) required around 30 ms for the first documents and 120 ms for the last documents, which is still pretty far from what you’ve been encountering: > von Anfang ~ 10 ms auf ~ 2500 ms kommne würde But obviously something weird has been going on in your setup. Let’s see what alternatives we have… • Could you possibly try to update my example code such that it shows the reported behavior? Ideally with small input, in order to speed up the process. Maybe the runtime increase can also be demonstrated after 1.000 or 10.000 documents... • You could also send me a list of the files of your test_database directory; maybe the file sizes indicate some unusual patterns. • You could start BaseXServer with the JVM flag -Xrunhprof:cpu=samples (to be inserted in the basexserver script), start the server, run your script, stop the server directly afterwards, and send me the result file, which will be stored in the directory from where you started BaseX (java.hprof.txt). Best, Christian On Wed, Jan 11, 2017 at 4:57 PM, Christian Grün <christian.gr...@gmail.com> wrote: > Hi Lucian, > > Thanks for your analysis. Indeed I’m wondering about the monotonic > delay caused by auto flushing the data; this hasn’t always been the > case. I’m wondering even more why no one else noticed this in recent > time.. Maybe it’s not too long ago that this was introduced. It may > take some time to find the culprit, but I’ll keep you updated. > > All the best, > Christian > > > On Wed, Jan 11, 2017 at 2:46 PM, Bularca, Lucian > <lucian.bula...@mueller.de> wrote: >> Hi Christian, >> >> I've made a comparation of the persistence time series running your example >> code and mine, in all possible combinations of following scenarios: >> - with and without "set intparse on" >> - using my prepared test data and your test data >> - closing and opening the DB connection each "n"-th insertion operation >> (where n in {5, 100, 500, 1000}) >> - with and without "set autoflush on". >> >> I finally found out, that the only relevant variable that influence the >> insert operation duration is the value of the AUTOFLASH option. >> >> If AUTOFLASH = OFF when opening a database, then the persistence durations >> remains relative constant (on my machine about 43 ms) during the entire >> insert operations sequence (50.000 or 100.000 times), for all possible >> combinations named above. >> >> If AUTOFLASH = ON when opening a database, then the persistence durations >> increase monotonic, for all possible combinations named above. >> >> The persistence duration, if AUTOFLASH = ON, is directly proportional to the >> number of DB clients executing these insert operations, respectively to the >> sequence length of insert operations executed by a DB client. >> >> In my opinion, this behaviour is an issue of BaseX, because AUTOFLASH is >> implcitly set to ON (see BaseX documentation >> http://docs.basex.org/wiki/Options#AUTOFLUSH), so DB clients must explicitly >> set AUTOFLASH = OFF in order to keep the insert operation durations >> relatively constant over time. Additionally, no explicitly flushing data, >> increases the risk of data loss (see BaseX documentation >> http://docs.basex.org/wiki/Options#AUTOFLUSH), but clients how repeatedly >> execute the FLUSH command increase the durations of the subsequent insert >> operations. >> >> Regards, >> Lucian >> >> ________________ >> Von: Christian Grün [christian.gr...@gmail.com] >> Gesendet: Dienstag, 10. Januar 2017 17:33 >> An: Bularca, Lucian >> Cc: Dirk Kirsten; basex-talk@mailman.uni-konstanz.de >> Betreff: Re: [basex-talk] Gravierende Performance-Einbüße bei Persistierung >> von mehr als 5000, 160 KB große XML Datenstrukturen. >> >> Hi Lucian, >> >> I couldn’t run your code example out of the box. 24 hours sounds >> pretty alarming, though, so I have written my own example (attached). >> It creates 50.000 XML documents, each sized around 160 KB. It’s not as >> fast as I had expected, but the total runtime is around 13 minutes, >> and it only slow down a little when adding more documents... >> >> 1: 125279.45 ms >> 2: 128244.23 ms >> 3: 130499.9 ms >> 4: 132286.05 ms >> 5: 134814.82 ms >> >> Maybe you could compare the code with yours, and we can find out what >> causes the delay? >> >> Best, >> Christian >> >> >> On Tue, Jan 10, 2017 at 4:44 PM, Bula
Re: [basex-talk] Gravierende Performance-Einbüße bei Persistierung von mehr als 5000, 160 KB große XML Datenstrukturen.
Hi Lucian, Thanks for your analysis. Indeed I’m wondering about the monotonic delay caused by auto flushing the data; this hasn’t always been the case. I’m wondering even more why no one else noticed this in recent time.. Maybe it’s not too long ago that this was introduced. It may take some time to find the culprit, but I’ll keep you updated. All the best, Christian On Wed, Jan 11, 2017 at 2:46 PM, Bularca, Lucian <lucian.bula...@mueller.de> wrote: > Hi Christian, > > I've made a comparation of the persistence time series running your example > code and mine, in all possible combinations of following scenarios: > - with and without "set intparse on" > - using my prepared test data and your test data > - closing and opening the DB connection each "n"-th insertion operation > (where n in {5, 100, 500, 1000}) > - with and without "set autoflush on". > > I finally found out, that the only relevant variable that influence the > insert operation duration is the value of the AUTOFLASH option. > > If AUTOFLASH = OFF when opening a database, then the persistence durations > remains relative constant (on my machine about 43 ms) during the entire > insert operations sequence (50.000 or 100.000 times), for all possible > combinations named above. > > If AUTOFLASH = ON when opening a database, then the persistence durations > increase monotonic, for all possible combinations named above. > > The persistence duration, if AUTOFLASH = ON, is directly proportional to the > number of DB clients executing these insert operations, respectively to the > sequence length of insert operations executed by a DB client. > > In my opinion, this behaviour is an issue of BaseX, because AUTOFLASH is > implcitly set to ON (see BaseX documentation > http://docs.basex.org/wiki/Options#AUTOFLUSH), so DB clients must explicitly > set AUTOFLASH = OFF in order to keep the insert operation durations > relatively constant over time. Additionally, no explicitly flushing data, > increases the risk of data loss (see BaseX documentation > http://docs.basex.org/wiki/Options#AUTOFLUSH), but clients how repeatedly > execute the FLUSH command increase the durations of the subsequent insert > operations. > > Regards, > Lucian > > > Von: Christian Grün [christian.gr...@gmail.com] > Gesendet: Dienstag, 10. Januar 2017 17:33 > An: Bularca, Lucian > Cc: Dirk Kirsten; basex-talk@mailman.uni-konstanz.de > Betreff: Re: [basex-talk] Gravierende Performance-Einbüße bei Persistierung > von mehr als 5000, 160 KB große XML Datenstrukturen. > > Hi Lucian, > > I couldn’t run your code example out of the box. 24 hours sounds > pretty alarming, though, so I have written my own example (attached). > It creates 50.000 XML documents, each sized around 160 KB. It’s not as > fast as I had expected, but the total runtime is around 13 minutes, > and it only slow down a little when adding more documents... > > 1: 125279.45 ms > 2: 128244.23 ms > 3: 130499.9 ms > 4: 132286.05 ms > 5: 134814.82 ms > > Maybe you could compare the code with yours, and we can find out what > causes the delay? > > Best, > Christian > > > On Tue, Jan 10, 2017 at 4:44 PM, Bularca, Lucian > <lucian.bula...@mueller.de> wrote: >> Hi Dirk, >> >> of course, querying millions of data entries on a single database rise >> problems. This is equally problematic for all databases, not only for the >> BaseX DB and certain storing strategies will be mandatory at production >> time. >> >> The actual problem is, that adding 50.000 of 160 KB xml stuctures took 24 >> hours because that inexplicable monotonic increase of the insert operation >> durations. >> >> I'll really appreciate if someone can explain this behaviour or a >> counterexample can demonstrate, that the cause of this behaviour is test >> case but not DB inherent. >> >> Regards, >> Lucian
Re: [basex-talk] Gravierende Performance-Einbüße bei Persistierung von mehr als 5000, 160 KB große XML Datenstrukturen.
Hi Christian, I've made a comparation of the persistence time series running your example code and mine, in all possible combinations of following scenarios: - with and without "set intparse on" - using my prepared test data and your test data - closing and opening the DB connection each "n"-th insertion operation (where n in {5, 100, 500, 1000}) - with and without "set autoflush on". I finally found out, that the only relevant variable that influence the insert operation duration is the value of the AUTOFLASH option. If AUTOFLASH = OFF when opening a database, then the persistence durations remains relative constant (on my machine about 43 ms) during the entire insert operations sequence (50.000 or 100.000 times), for all possible combinations named above. If AUTOFLASH = ON when opening a database, then the persistence durations increase monotonic, for all possible combinations named above. The persistence duration, if AUTOFLASH = ON, is directly proportional to the number of DB clients executing these insert operations, respectively to the sequence length of insert operations executed by a DB client. In my opinion, this behaviour is an issue of BaseX, because AUTOFLASH is implcitly set to ON (see BaseX documentation http://docs.basex.org/wiki/Options#AUTOFLUSH), so DB clients must explicitly set AUTOFLASH = OFF in order to keep the insert operation durations relatively constant over time. Additionally, no explicitly flushing data, increases the risk of data loss (see BaseX documentation http://docs.basex.org/wiki/Options#AUTOFLUSH), but clients how repeatedly execute the FLUSH command increase the durations of the subsequent insert operations. Regards, Lucian Von: Christian Grün [christian.gr...@gmail.com] Gesendet: Dienstag, 10. Januar 2017 17:33 An: Bularca, Lucian Cc: Dirk Kirsten; basex-talk@mailman.uni-konstanz.de Betreff: Re: [basex-talk] Gravierende Performance-Einbüße bei Persistierung von mehr als 5000, 160 KB große XML Datenstrukturen. Hi Lucian, I couldn’t run your code example out of the box. 24 hours sounds pretty alarming, though, so I have written my own example (attached). It creates 50.000 XML documents, each sized around 160 KB. It’s not as fast as I had expected, but the total runtime is around 13 minutes, and it only slow down a little when adding more documents... 1: 125279.45 ms 2: 128244.23 ms 3: 130499.9 ms 4: 132286.05 ms 5: 134814.82 ms Maybe you could compare the code with yours, and we can find out what causes the delay? Best, Christian On Tue, Jan 10, 2017 at 4:44 PM, Bularca, Lucian <lucian.bula...@mueller.de> wrote: > Hi Dirk, > > of course, querying millions of data entries on a single database rise > problems. This is equally problematic for all databases, not only for the > BaseX DB and certain storing strategies will be mandatory at production > time. > > The actual problem is, that adding 50.000 of 160 KB xml stuctures took 24 > hours because that inexplicable monotonic increase of the insert operation > durations. > > I'll really appreciate if someone can explain this behaviour or a > counterexample can demonstrate, that the cause of this behaviour is test > case but not DB inherent. > > Regards, > Lucian
Re: [basex-talk] Gravierende Performance-Einbüße bei Persistierung von mehr als 5000, 160 KB große XML Datenstrukturen.
Hi Lucian, I couldn’t run your code example out of the box. 24 hours sounds pretty alarming, though, so I have written my own example (attached). It creates 50.000 XML documents, each sized around 160 KB. It’s not as fast as I had expected, but the total runtime is around 13 minutes, and it only slow down a little when adding more documents... 1: 125279.45 ms 2: 128244.23 ms 3: 130499.9 ms 4: 132286.05 ms 5: 134814.82 ms Maybe you could compare the code with yours, and we can find out what causes the delay? Best, Christian On Tue, Jan 10, 2017 at 4:44 PM, Bularca, Lucianwrote: > Hi Dirk, > > of course, querying millions of data entries on a single database rise > problems. This is equally problematic for all databases, not only for the > BaseX DB and certain storing strategies will be mandatory at production > time. > > The actual problem is, that adding 50.000 of 160 KB xml stuctures took 24 > hours because that inexplicable monotonic increase of the insert operation > durations. > > I'll really appreciate if someone can explain this behaviour or a > counterexample can demonstrate, that the cause of this behaviour is test > case but not DB inherent. > > Regards, > Lucian AddDocs.java Description: Binary data
Re: [basex-talk] Gravierende Performance-Einbüße bei Persistierung von mehr als 5000, 160 KB große XML Datenstrukturen.
Hi Dirk, of course, querying millions of data entries on a single database rise problems. This is equally problematic for all databases, not only for the BaseX DB and certain storing strategies will be mandatory at production time. The actual problem is, that adding 50.000 of 160 KB xml stuctures took 24 hours because that inexplicable monotonic increase of the insert operation durations. I'll really appreciate if someone can explain this behaviour or a counterexample can demonstrate, that the cause of this behaviour is test case but not DB inherent. Regards, Lucian Von: Dirk Kirsten [d...@basex.org] Gesendet: Dienstag, 10. Januar 2017 14:37 An: Bularca, Lucian; basex-talk@mailman.uni-konstanz.de Betreff: Re: AW: [basex-talk] Gravierende Performance-Einbüße bei Persistierung von mehr als 5000, 160 KB große XML Datenstrukturen. Hi Lucian, sorry, I obviously forgot a very important word: "not". I would NOT expect 100,000 documents to be added to be much of a problem. Sorry for the confusion. The log file is interesting, it certainly looks like the performance is degrading. I can't say much about it, but I am sure Christian (our head architect) will give you some pointers when he has time to answer. However, given the description of your problem I would advise you to in general rethink your architecture. So many consistent updates on a database seem to me to be not very performant when done on only a single database. So maybe you want to split up your data, e.g. you could put all documents of a certain day into a separate database. Or you could have one "up-to-date" database, which you always update and transfer the entries within this database into another database during low-performance times. The other database could have proper indexes and whatever you need. Because otherwise you will run into problems when querying your data. I guess you don't want to just store your data, you want to do something with it, don't you? Because just storing without using data seems a bit useless... And for this you probably want to use some indexes and having an up-to-date-index with constant updates is quite costly. To sum it up: I think you want to split up your data in some way into several databases. However, I understand that you will still have something like 100,000 documents in a database (which should be fine), so your current performance issue will still exist. My comment is more towards your general architecture. Cheers Dirk On 01/10/2017 08:24 PM, Bularca, Lucian wrote: Hi Dirk, thanks for your fast reply :) Regarding the performance measure, I've forgot to mention, that I've based my affirmations on the protocol entries from the BaseX log file (see attached basex.log). The intention of the System.out made in each iteration, is just to protocol the order number of the added xml structure, not the duration of a persist operation. This System.out indeed does have an impact on the overall performance, but cannot explain the monotonic increase of the insert operations duration (see attached basex.log file). After 24 hours of inserting xml test-structures, only the half of the 100.000 xml test-structures where added in the database, at a rate of at most 1 structure / 2 seconds. All these tests where made against the 8.5.3 version of the BaseX database. In production, we expect peaks of 2,7 * 10 ^ 5 xml structures to persist / 24 hours (~ 31 xml structures / 1 second). Do you mean with "However, I would expect 100,000 documents added to be much of a problem.", that persisting 100.000 xml structures in the BaseX database is problematic? Regards, Lucian Von: basex-talk-boun...@mailman.uni-konstanz.de<mailto:basex-talk-boun...@mailman.uni-konstanz.de> [basex-talk-boun...@mailman.uni-konstanz.de<mailto:basex-talk-boun...@mailman.uni-konstanz.de>]" im Auftrag von "Dirk Kirsten [d...@basex.org<mailto:d...@basex.org>] Gesendet: Dienstag, 10. Januar 2017 12:52 An: basex-talk@mailman.uni-konstanz.de<mailto:basex-talk@mailman.uni-konstanz.de> Betreff: Re: [basex-talk] Gravierende Performance-Einbüße bei Persistierung von mehr als 5000, 160 KB große XML Datenstrukturen. Hello Lucian, please be aware that this is an English-speaking mailing list as we have many users from all over the world and the mailing list is intended to help everyone. But as most of our team members are German (well, and Bavarians...) we of course understand it. Hence, I answer in English (for all other: Lucian seem to have same performance issues when adding many documents). First of all, are you sure your tests sufficiently test the add performance. Looking at your file TestBaseXClient.java it seems to not record the runtimes of the individual insertions, but just the overall runtime of in this case 10 insertions. Also, at least in t
Re: [basex-talk] Gravierende Performance-Einbüße bei Persistierung von mehr als 5000, 160 KB große XML Datenstrukturen.
Hi Lucian, sorry, I obviously forgot a very important word: "not". I would NOT expect 100,000 documents to be added to be much of a problem. Sorry for the confusion. The log file is interesting, it certainly looks like the performance is degrading. I can't say much about it, but I am sure Christian (our head architect) will give you some pointers when he has time to answer. However, given the description of your problem I would advise you to in general rethink your architecture. So many consistent updates on a database seem to me to be not very performant when done on only a single database. So maybe you want to split up your data, e.g. you could put all documents of a certain day into a separate database. Or you could have one "up-to-date" database, which you always update and transfer the entries within this database into another database during low-performance times. The other database could have proper indexes and whatever you need. Because otherwise you will run into problems when querying your data. I guess you don't want to just store your data, you want to do something with it, don't you? Because just storing without using data seems a bit useless... And for this you probably want to use some indexes and having an up-to-date-index with constant updates is quite costly. To sum it up: I think you want to split up your data in some way into several databases. However, I understand that you will still have something like 100,000 documents in a database (which should be fine), so your current performance issue will still exist. My comment is more towards your general architecture. Cheers Dirk On 01/10/2017 08:24 PM, Bularca, Lucian wrote: > Hi Dirk, > > thanks for your fast reply :) > > Regarding the performance measure, I've forgot to mention, that I've > based my affirmations on the protocol entries from the BaseX log file > (see attached basex.log). The intention of the System.out made in each > iteration, is just to protocol the order number of the added xml > structure, not the duration of a persist operation. This System.out > indeed does have an impact on the overall performance, but cannot > explain the monotonic increase of the insert operations duration (see > attached basex.log file). After 24 hours of inserting xml > test-structures, only the half of the 100.000 xml test-structures > where added in the database, at a rate of at most 1 structure / 2 seconds. > > All these tests where made against the 8.5.3 version of the BaseX > database. > > In production, we expect peaks of 2,7 * 10 ^ 5 xml structures to > persist / 24 hours (~ 31 xml structures / 1 second). Do you mean with > "However, I would expect 100,000 documents added to be much of a > problem.", that persisting 100.000 xml structures in the BaseX > database is problematic? > > > Regards, > Lucian > > *Von:* basex-talk-boun...@mailman.uni-konstanz.de > [basex-talk-boun...@mailman.uni-konstanz.de]" im Auftrag von "Dirk > Kirsten [d...@basex.org] > *Gesendet:* Dienstag, 10. Januar 2017 12:52 > *An:* basex-talk@mailman.uni-konstanz.de > *Betreff:* Re: [basex-talk] Gravierende Performance-Einbüße bei > Persistierung von mehr als 5000, 160 KB große XML Datenstrukturen. > > Hello Lucian, > > > please be aware that this is an English-speaking mailing list as we > have many users from all over the world and the mailing list is > intended to help everyone. But as most of our team members are German > (well, and Bavarians...) we of course understand it. Hence, I answer > in English (for all other: Lucian seem to have same performance issues > when adding many documents). > > > First of all, are you sure your tests sufficiently test the add > performance. Looking at your file TestBaseXClient.java it seems to not > record the runtimes of the individual insertions, but just the overall > runtime of in this case 10 insertions. > > Also, at least in the Example you provided you also do some other > stuff (especiall printing to sysout), which obviously also has a > performance impact. > > > Optimizing or creating indexes in between a mass update should not > increase the speed, as it builds the indexes, which will be > invalidated after the next index, so I would not expect any speed up here. > > > What version of BaseX did you use? > > > Did you set AUTOFLUSH (see > http://docs.basex.org/wiki/Options#AUTOFLUSH) to false? This should > benefit performance. > > > In general it is also a good architectural approach to split up > documents into many databases instead of having one large database. > Given that you can access as many databases as you want in one query > you will not lose any qu
Re: [basex-talk] Gravierende Performance-Einbüße bei Persistierung von mehr als 5000, 160 KB große XML Datenstrukturen.
Hi Dirk, thanks for your fast reply :) Regarding the performance measure, I've forgot to mention, that I've based my affirmations on the protocol entries from the BaseX log file (see attached basex.log). The intention of the System.out made in each iteration, is just to protocol the order number of the added xml structure, not the duration of a persist operation. This System.out indeed does have an impact on the overall performance, but cannot explain the monotonic increase of the insert operations duration (see attached basex.log file). After 24 hours of inserting xml test-structures, only the half of the 100.000 xml test-structures where added in the database, at a rate of at most 1 structure / 2 seconds. All these tests where made against the 8.5.3 version of the BaseX database. In production, we expect peaks of 2,7 * 10 ^ 5 xml structures to persist / 24 hours (~ 31 xml structures / 1 second). Do you mean with "However, I would expect 100,000 documents added to be much of a problem.", that persisting 100.000 xml structures in the BaseX database is problematic? Regards, Lucian Von: basex-talk-boun...@mailman.uni-konstanz.de [basex-talk-boun...@mailman.uni-konstanz.de]" im Auftrag von "Dirk Kirsten [d...@basex.org] Gesendet: Dienstag, 10. Januar 2017 12:52 An: basex-talk@mailman.uni-konstanz.de Betreff: Re: [basex-talk] Gravierende Performance-Einbüße bei Persistierung von mehr als 5000, 160 KB große XML Datenstrukturen. Hello Lucian, please be aware that this is an English-speaking mailing list as we have many users from all over the world and the mailing list is intended to help everyone. But as most of our team members are German (well, and Bavarians...) we of course understand it. Hence, I answer in English (for all other: Lucian seem to have same performance issues when adding many documents). First of all, are you sure your tests sufficiently test the add performance. Looking at your file TestBaseXClient.java it seems to not record the runtimes of the individual insertions, but just the overall runtime of in this case 10 insertions. Also, at least in the Example you provided you also do some other stuff (especiall printing to sysout), which obviously also has a performance impact. Optimizing or creating indexes in between a mass update should not increase the speed, as it builds the indexes, which will be invalidated after the next index, so I would not expect any speed up here. What version of BaseX did you use? Did you set AUTOFLUSH (see http://docs.basex.org/wiki/Options#AUTOFLUSH) to false? This should benefit performance. In general it is also a good architectural approach to split up documents into many databases instead of having one large database. Given that you can access as many databases as you want in one query you will not lose any query capabilities and at some point you might encounter certain limits. However, I would expect 100,000 documents added to be much of a problem. As a side node, as it seems you are evaluating BaseX and I guess you are doing this for a reason, it might be faster/easier when talking to our BaseX members, who of course can help you with evaluating your problem and identifying whether BaseX is the right choice for your given problem. Take a look at http://basexgmbh.de/ for our commercial offerings. Cheers Dirk On 01/10/2017 05:44 PM, Bularca, Lucian wrote: Guten Tag, im Rahmen einer Performance-Evaluierung der Persistierung von XML Datenstrukturen in einer BaseX Datenbank, haben wir stetig absinkende Persistierungs-Raten umgekehrt proportional zu der Datenbank-Größe festgestellt. Dieses Verhalten ist erklährbar und wäre auch einnehmbar, wenn nicht die Dauer der Persistierung einer ~ 160 KB großen XML Datenstruktur, von Anfang ~ 10 ms auf ~ 2500 ms kommne würde, nach ~ 50.000 Persistierungs-Vorgänge. Dabei versuchen wir 100.000 unterschiedliche, ungefähr 160 KB große XML Datenstrukturen mittels der Java API in eine BaseX Datenbank zu speichern um dabei die Gesammt-Dauer bzw. die Dauer der einzelnen Persistierungs-Vorgänge zu messen. Die BaseX Datenbank wurde im HTTP Modus (basexhttp) mit -Xmx 4048m gestartert. Die oben genannte Messwerte blieben gleich, unabhängig davon, ob alle XM Datenstrukturen in eine einzige Session gespeichert wurden, oder wenn alle 500 Persistierungs-Vorgänge der Socket (DB-Anbindung) geschlossen und erneut geöffnet wurde. Eine Indizierung der Datenbank (mittels der GUI "Optimize All", bzw. "Create Text Index") zwischendurch konnte die Persistierungs-Raten nicht beeinflussen bzw. optimieren. Ein Beispiel der Test-Klassen (nur exemplarisch, nicht kompilierbar !) die wir dazu benutzt haben, sind im Anhang BaseXClient.java.zip zu dieser E-Mail zu finden. Sind generell, Persistierungs-Raten von mehr als 160 KB / 2500 ms bei mehr als 30.000 vorhandene Einträge in der
Re: [basex-talk] Gravierende Performance-Einbüße bei Persistierung von mehr als 5000, 160 KB große XML Datenstrukturen.
Hello Lucian, please be aware that this is an English-speaking mailing list as we have many users from all over the world and the mailing list is intended to help everyone. But as most of our team members are German (well, and Bavarians...) we of course understand it. Hence, I answer in English (for all other: Lucian seem to have same performance issues when adding many documents). First of all, are you sure your tests sufficiently test the add performance. Looking at your file TestBaseXClient.java it seems to not record the runtimes of the individual insertions, but just the overall runtime of in this case 10 insertions. Also, at least in the Example you provided you also do some other stuff (especiall printing to sysout), which obviously also has a performance impact. Optimizing or creating indexes in between a mass update should not increase the speed, as it builds the indexes, which will be invalidated after the next index, so I would not expect any speed up here. What version of BaseX did you use? Did you set AUTOFLUSH (see http://docs.basex.org/wiki/Options#AUTOFLUSH) to false? This should benefit performance. In general it is also a good architectural approach to split up documents into many databases instead of having one large database. Given that you can access as many databases as you want in one query you will not lose any query capabilities and at some point you might encounter certain limits. However, I would expect 100,000 documents added to be much of a problem. As a side node, as it seems you are evaluating BaseX and I guess you are doing this for a reason, it might be faster/easier when talking to our BaseX members, who of course can help you with evaluating your problem and identifying whether BaseX is the right choice for your given problem. Take a look at http://basexgmbh.de/ for our commercial offerings. Cheers Dirk On 01/10/2017 05:44 PM, Bularca, Lucian wrote: > Guten Tag, > > im Rahmen einer Performance-Evaluierung der Persistierung von XML > Datenstrukturen in einer BaseX Datenbank, haben wir stetig absinkende > Persistierungs-Raten umgekehrt proportional zu der Datenbank-Größe > festgestellt. > > Dieses Verhalten ist erklährbar und wäre auch einnehmbar, wenn nicht > die Dauer der Persistierung einer ~ 160 KB großen XML Datenstruktur, > von Anfang ~ 10 ms auf ~ 2500 ms kommne würde, nach ~ 50.000 > Persistierungs-Vorgänge. > > Dabei versuchen wir 100.000 unterschiedliche, ungefähr 160 KB große > XML Datenstrukturen mittels der Java API in eine BaseX Datenbank zu > speichern um dabei die Gesammt-Dauer bzw. die Dauer der einzelnen > Persistierungs-Vorgänge zu messen. Die BaseX Datenbank wurde im HTTP > Modus (basexhttp) mit -Xmx 4048m gestartert. > > Die oben genannte Messwerte blieben gleich, unabhängig davon, ob alle > XM Datenstrukturen in eine einzige Session gespeichert wurden, oder > wenn alle 500 Persistierungs-Vorgänge der Socket (DB-Anbindung) > geschlossen und erneut geöffnet wurde. Eine Indizierung der Datenbank > (mittels der GUI "Optimize All", bzw. "Create Text Index") > zwischendurch konnte die Persistierungs-Raten nicht beeinflussen bzw. > optimieren. > > Ein Beispiel der Test-Klassen (nur exemplarisch, nicht kompilierbar !) > die wir dazu benutzt haben, sind im Anhang BaseXClient.java.zip zu > dieser E-Mail zu finden. > > Sind generell, Persistierungs-Raten von mehr als 160 KB / 2500 ms bei > mehr als 30.000 vorhandene Einträge in der BaseX zu erwarten, oder > können wir diese Persistierungs-Zeiten drastisch optimieren (und wenn > ja, wie)? > > > Mit freundlichen Grüßen, > Lucian Bularca > > -- Dirk Kirsten, BaseX GmbH, http://basexgmbh.de |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer: | Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle `-- Phone: 0049 7531 91 68 276, Fax: 0049 7531 20 05 22