Hi Gabriel; just getting back to you so you know I am alive. Changing over the the queue has made a *vast* difference in how well the datastore performs in uDig; everything is much better to work with and my customer is happy. I know we have some more issue to work through; but I wanted you to let you know the approach is a resounding success.
I am updating now and will try and respond to the items you listed below. Jody Gabriel Roldán wrote: > Hi, > > here's the deal with arcsde after a week of hacking: > > Saul, if you don't want to go through the whole content check item 4- at the > end. > > - command queue strategy for connection handling successfully implemented for > vector data. I even set the connection's concurrency policy to > SE_ONE_THREAD_POLICY so it fails fast if tried to be used from another thread > than the one it was created with. This helped to spot a couple places where > command should have been used but were not being. > > - all vector data tests pass, except those three that were not relevant to > this work, that still need to be fixed. There are a difference on the level > of transaction isolation ArcSDE uses depending on if the backend is mssql or > not, see <http://jira.codehaus.org/browse/GEOT-1831> > > - By using SessionTransactionState, I've turned Session into an interface, > and > am providing a decorator for non auto commit transactions that just avoids > the session to be returned to the pool if its on transaction. This provides > for a lot of code clean up, since now all the code can follow the same usage > pattern when it needs a Session: acquire/use/dispose. The actual session will > take care of returning itself to the pool or not but the client code does not > care. > > - Sessions are no longer kept as instance variables except in > ArcSDEAttributeReader and ArcSDEFeatureWriter, the only two that need to hold > on a Session until they're exhausted/closed. The only one that knows about > the pool is ArcSDEDataStore. Sessions are always dynamically acquired from > the pool in a per transaction basis. Read only code leads to > getSession(Transaction.AUTO_COMMIT). > > - Care has been taken for a single connection to be enough for any single > common usage. Moreover, thanks to the command queue stuff, sessions (and > hence SeConnections) have a much higher probability of serving multiple > requests "at a time", since the "execution units" needing a session are much > finely grained. > > All in all, performance through WMS seems good, even using a large OL preview > with tiles and maxConnections set to 2 with a quite large layer. > > What's still pending: > 1- fixing the polygon with hole creation bug > 2- testing hard with versioned tables, a user reported a lot of deadlocks in > the database > 3- backporting to 2.4.x > 4- and very importantly: go through all the gce code and fix/adapt to > commands. > 5- Test ArcSDEConnectionReference for the single connection configuration. > Jody I would try caching the connection once and just overriding getSession() > to return it over and over. > > Saul, I would be willing to help on this last one, just need to know how to > run the tests, if I need to set up something special or the tests create > their own data, etc > > Best regards, > > Gabriel > > On Wednesday 21 May 2008 09:58:35 pm Gabriel Roldán wrote: > >> On Wednesday 21 May 2008 09:39:34 pm Jody Garnett wrote: >> >>> Gabriel Roldán wrote: >>> >>>> Hey Jody, >>>> >>>> so this is what I did: >>>> - got rid of connection.commit/rollback in ArcTransactionState >>>> - added ArcSDEDataStore.getSession(Transaction). For read only use just >>>> call it with Transaction.AUTO_COMMIT, and that's what FeatureSource >>>> does, whilst FetureStore uses the transaction it was set with >>>> >>> Sounds good. >>> >>> >>>> There's something strange with the command execution though, may be you >>>> have some time to test it, since its almost bed time for me now. >>>> >>> I will update now. >>> >>> >>>> The strange is that running ArcSDEFeatureStoreTest against my server >>>> (which is close so it runs fast) I get a lot of "THIS CONNECTION IS >>>> BEING USED AT THE MOMENT BY ANOTHER THREAD" errors, while executing the >>>> same test against your server ( which is far away so most of the time >>>> is spent in network io) only the last two test cases fail. >>>> >>> That is confusing to me; I don't understand it either. >>> >>> >>>> My hope is that if you run it you'll get a lot of broken cases as your >>>> server is fast for you, and you may be able to get a clue why some >>>> command execution is breaking that way... >>>> >>>> Another thing: using issueReadCommand like you did in fetchSchema >>>> implies needing two connections at a given time for a single thing. >>>> I've commented it by now, since I guess when using >>>> ArcSDEConnectionReference you just want to return the same session over >>>> and over, so there shouldn't be a need to do anything special? or >>>> rather I'm not >>>> understanding why you want the issueReadOnly method. The idea being to >>>> pick up any available connection, but you have only one, so why not >>>> just overriding getSession to return the same one? not sure.. >>>> >>> This is why I need the code review: >>> - ArcSDEConnectionReference.issueReadCommand will try and reuse the >>> "last" Session (ie the only session), if it is not available it will >>> grab a Session from the pool like normal >>> -ArcSDEConnectionPool.issueReadCommand will grab a session from the pool >>> like normal - I did not want to change any functionality on you until >>> you had done a code review. So this method behaves exactly the same as >>> before (ie we always needed 2 connections in the past) >>> >> it did in past, then I got with working with a single one, then introducing >> this required two again. right now you can run the ArcSDEFeatureSourceTest >> with a single connection, though I won't do any statement that a single >> connection is enough for all and every code path, yet.. >> The big difference is that now we have a much finer granularity on what an >> execution unit is, connection wise, so there're higher probabilities a >> single connection serves more read only operations. >> >> >>>> Also, you'll note a lot of verbosity on the console, it was me freaking >>>> out while trying to understand about what's causing the connection to >>>> complain about threading. >>>> >>> So where does the exception about threading appear? >>> >> ArcSDEFeatureStoreTest.testUpdateAutocommit first, then everything gets >> screwed up. >> >> but I couldnt understand for the last long hours why the heck the command >> sent at ArcSdeFeatureWriter.getUpdatableColumnNames() (line 694) leads to >> this error and not any other command. >> >> any clue will be much appreciated, I seem too burned right now for a single >> day, and am starting to stop beleaving in resolving hard problems in dreams >> or shower... :) >> >> cheers, >> >> Gabriel >> >> >>> Jody >>> >> ------------------------------------------------------------------------- >> This SF.net email is sponsored by: Microsoft >> Defy all challenges. Microsoft(R) Visual Studio 2008. >> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ >> _______________________________________________ >> Geotools-devel mailing list >> Geotools-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/geotools-devel >> >> !DSPAM:4045,48347f5e26452090977483! >> > > > ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Geotools-devel mailing list Geotools-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geotools-devel
