Thanks Alex - I need to get in the habit of posting more frequently over there :)
Adam > On Apr 10, 2019, at 3:16 AM, Alex Miller <alexmil...@apple.com.INVALID> wrote: > > >> On 2019/03/20 22:47:42, Adam Kocoloski <kocol...@apache.org> wrote: >> I don’t know in detail what optimizations FoundationDB applies to atomic >> operations (e.g. coalescing them at a layer above the storage engine). >> That’s worth checking into, as otherwise I’d be concerned about introducing >> super-hot keys here. > > > Inducing hot keys or hot shards would both be of concern. There is logic > that will try to deal with write hotspots, but it splits shard based on write > bandwidth to a particular shard, which likely wouldn’t be triggered by a > stream of atomic operations. > > Clients will merge atomic operations together if you issue multiple in a > transaction, e.g. I believe three ATOMIC_ADDs on one key will become one > ATOMIC_ADD of the sum. There isn’t particularly any logic that optimizes the > handling of atomic operations once they leave the client. Storage servers > only commit every ~250ms, so this shouldn’t translate into a full page write > on each atomic operation to the same key, but the storage server will end up > applying each atomic operation individually. > > ( And I’ve gone and filed https://github.com/apple/foundationdb/issues/1450 > to trigger more discussion within FoundationDB on this topic. ) > > >> On Mar 25, 2019, at 11:42 AM, Adam Kocoloski <kocol...@apache.org> wrote: >>> On Mar 25, 2019, at 12:48 PM, Mike Rhodes <couc...@dx13.co.uk> wrote: >>> >>> I couldn't immediately see how we cleared out older entries from this >>> potentially very large queue. For example, the worker processing the queue >>> to deduplicate might issue range deletes after processing each "batch". Is >>> this simple enough to do? >> >> Yes, that’s the (implicit) idea. Simple to implement, not clear to me how >> well the storage servers can handle the load. I think the “range clears are >> cheap” statement largely refers to the transaction management system. > > At the storage level, range clears are cheap in terms of immediate execution, > and slow in terms of total work performed. You can find the details of this > in > https://forums.foundationdb.org/t/shards-are-not-splitted-into-smaller-ones/815/4 > , and an example of what to be careful about with range clears on > https://forums.foundationdb.org/t/used-disk-space-dramatically-increases-while-sum-of-key-value-sizes-is-constant/644 > > >> On Mar 27, 2019, at 11:07 AM, Ilya Khlopotov <iil...@apache.org> wrote: >> What if FDB would support a list type for a value and would have an atomic >> operation to add the value to the list if it is missing. In this case we >> could store the data we need as follows (under separate subspace TBD). >> VersionStamp = (DbName, EventType) >> DbName = [versionstamps] >> ... >> The question is how we would implement atomic addition of a value to a list. > > If I’ve understood your proposal correctly, I think it sounds easier to just > use APPEND_IF_FITS ? > > https://apple.github.io/foundationdb/javadoc/index.html?com/apple/foundationdb/MutationType.html >
