2010/1/7 Kristian Rosenvold <kristian.rosenv...@gmail.com>: > On Thu, 2010-01-07 at 10:17 +0000, Stephen Connolly wrote: > >> Side note: >> >> Now that sounds like the concurrency code I wrote to convert from >> Accurev to Subversion.... >> >> I actually ended up creating the state of a stream at a specific >> revision because it was asked for (by a downstream stream/workspace or >> by a subsequent revision) >> >> Then I had a dynamic cache that tracked popular states... and then you >> just step through each revision one by one and solve every stream. > > Before you know it, someone will suggest that this is actually called > "functional programming". Given that any call for find(x) will always > return the same value, the underlying language can just cache the whole > call.
Yes but with a 20GB repo you still need a dynamic cache, and my cache was context specific (i.e. it had logic that allowed it to knwo when it could propagate entries forward (thereby avoiding unnecessary work completely) and when it could definitely throw away entries (thereby reducing cache churn because an entry that is really garbage but newer than a non-garbage entry would result in re-calc for a FIFO cache) > >> I was able to convert 115789 revisions in one depot with 1000's of >> streams/workspaces/snapshots on my quad core2 with 4GB of ram in only >> 3 days! The SVN dump file took 4 days to load into subversion! >> > > Everything takes 4 days with subversion ;) > >> Of course if we were to implement that kind of approach for Maven then >> you might not even build a specific module because nobody was looking >> for it... hmmm even faster builds > > Getting real, I actually tried a threading model for MNG-3004 where the > same thread would dive downwards when there was an output-dependency, > because I considered this to be "critical path". > > I actually had this running, but it had 2 very serious problems; > > there was a very real risk of deadlocking with anything but unlimited > threads, since when thread X left for child Y, I would schedule another > thread to run the rest of X. The rest is left as an exercise to the > reader. > > Additionally, using anything but 1 thread to do a single build opens up > for a whole set of concurrency issues. In it's current state, most of > the state for a build has a clear happens-before relationship to the > thread running the build. If you start zigzagging between threads you > *really* quickly see that this relationship is valuable. > > Kristian > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > For additional commands, e-mail: dev-h...@maven.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
