I don't object to changes to storage so long as we have a migration plan and a design doc. I'm also not opposed to radical revisits of storage, including overhauling what we store and where we store it. For example, instead of storing our `TaskConfig` objects could we store Mesos `TaskInfo` objects instead? Could we store data outside of the scheduler like in Cassandra? Should we have a high level 'Job' store to make querying for job level data easier?
On Thu, Mar 30, 2017 at 10:16 AM, David McLaughlin <dmclaugh...@apache.org> wrote: > Hi all, > > I'd like to start a discussion around storage in Aurora. > > I think one of the biggest mistakes we made in migrating our storage to H2 > was deleting the memory stores as we moved. We made a pretty big bet that > we could eventually make H2/relational databases work. I don't think that > bet has paid off and that we need to revisit the direction we're taking. > > My belief is that the current H2/MyBatis approach is untenable for large > production clusters, at least without changing our current single-master > architecture. At Twitter we are already having to fight to keep GC > manageable even without DbTaskStore enabled, so I don't see a path forward > where we could eventually enable that. So far experiments with H2 off-heap > storage have provided marginal (if any) gains. > > Would anyone object to restoring the in-memory stores and creating new > implementations for the missing ones (UpdateStore)? I'd even go further and > propose that we consider in-memory H2 and MyBatis a failed experiment and > we drop that storage layer completely. > > Cheers, > David > > -- > Zameer Manji >
