Nicklas Nordborg wrote: > Pawel Sztromwasser wrote: >> Hi all, >> >> I have been trying to optimize the performance of LowessNormalization >> plugin a little bit lately. It is extremely slow on our Postgres >> database when using small 'block groups'. > > It sounds like you are using an older version of BASE. The performance > of the Lowess plug-in for small block group sizes was greatly improved > in the 2.5.0 release. The tests showed decrease in execution time to > about 10-20% of the original. This was on arrays with 48 blocks and > block group size=1. Now the execution time is more or less the same for > all values of the block group size parameter. More information about > this at: >
Yes, we are currently using 2.4.5 and 2.5.0, but the tests were run on the older one. Upgrading it is a good idea:) By the way, download links to .gz files on the web page don't work. The improvement you are writing about is approximately what we achieved by decreasing number of db queries to 1 (no mater how big blocks are). I though about posting it with a code changes after some tests, but it's great that it has been done already. >> Anyway, I came across something that surprised me. When using >> DynamicResultIterator I noticed that hasNext() does not behave in the >> way I expect. After fetching a row by next() method, invoking hasNext() >> causes iterator to move to the next row. I checked that on the following >> example (it = iterator): > > The next() method always returns the same object which is just a "proxy" > that fetches data from the underlying ResultSet object. > >> command | result >> ----------------------- >> r = it.next() | - >> r.getInt(pos) | 324 >> it.hasNext() | true >> r.getInt(pos) | 325 >> ----------------------- >> >> I looked into the code to check what is going on. >> DynamicResultIterator.hasNext() invokes next() method on >> java.sql.ResultSet object to check if there are rows to read, but it >> does not move the cursor back(ResultSet.previous()). In addition, next() >> method uses hasNext() to move the cursor forward (otherwise executing >> if(it.hasNext) r=it.next() would skip one row). > > No rows are skipped. The hasNext() method only moves to the next row if > next() has been called in the meantime. It is safe to call hasNext() any > number of times. It will just move to the next row the first time. > >> Is this how it is meant to be? > > Yes. > > > I realize that the usual way of using >> hasNext() is in the context like: >> >> while(it.hasNext) >> r=it.next() >> // process r >> >> where this issue is not a problem, but sometimes it is useful to check >> if there are next rows of data without actually moving the cursor. One >> could also benefit from previous() method in that DynamicResultIterator >> class. >> There are probably reasons behind this kind of implementation I am not >> aware of. Then mentioning in the docs that hasNext moves the pointer to >> next result would be nice. > > The only way to check if there are more rows in a ResultSet is to try to > move to the next row. For performance reasons all queries are executed > in read-only, forward-only mode so there is no possibility to move > backwards. > > And since the DynamicResultIterator is an java.util.Iterator, we are > only interested in moving forwards anyway. > > You are correct that the documentation could be clearer about the issue > that it is actually the hasNext() method that moves to the next row, not > the next() method. The next() method at least states that the returned > SqlResult is just a pointer to the current row and if results from > previous rows are needed then that information needs to be copied by the > calling code. This is in itself also a performance optimization since we > never make unnecessary copies in the BASE core. > Thanks for the explanation. It is just not what I expected by hasNext() method to do. I think also that other users expecting it to behave like java.util.Iterator.hasNext could be a bit surprised before reading docs:) Best wishes, Pawel > /Nicklas > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > basedb-devel mailing list > basedb-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/basedb-devel ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ basedb-devel mailing list basedb-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/basedb-devel
