Alex, I am making a guess. I would be better able to support this guess with data from leveldb LOG files of one server.
The performance difference between the sequential keys and the reverse of the sequential keys is informative. The reverse ordered keys essentially become more like "random keys" within the leveldb key space. Overtime, they disperse to different .sst table files scattering the "focal point" of the read-before-write operations. The sequential keys are keeping all newly written data within the same .sst table file, i.e. the same "focal point". And there is a new tuning called "timed grooming" in the Riak 2.1.3 and 2.1.4 releases that has a bug. The bug can cause more frequent compaction cycles than intended for 2.1.3 and 2.1.4 under some work loads. Your particular sequential keys might be such a load. This is what the LOG files would indicate. The biggest impact of more frequent compaction cycles is that the Linux page cache and leveldb block caches get invalidated more often causing longer read cycles to determine "nothing is there" in the read-before-write operation of Riak. Your higher ring size is likely not helping, but there is math that can prove or disprove this assumption. And again, the base numbers are within the LOG files. We do have an eleveldb patch release available. You would have to manual install it on each of the nodes. The patch disables the timed grooming and contains some critical bug fixes that lead to the Riak 2.1.4 release. You would need to tell me which operating system package you originally download for Riak, then I can send an appropriate link. Matthew > On May 5, 2016, at 10:11 AM, alexc155 <ahcl...@gmail.com> wrote: > > Hi, > > Thanks for your reply. > > I don't think that write_once is going to work for us as we have to > periodically update the data (although if we remove the data before > re-inserting it, would that work?) > > Why does read-before-write slow down new writes so much? > > Some new information we've found - it seems that if we write the data and > then update it, we get fast speeds too. It's just the initial write of the > data that is slow. > > So why is writing sequential keys so much slower than updating them or > writing non-sequential keys? > > > > -- > View this message in context: > http://riak-users.197444.n3.nabble.com/How-to-increase-Riak-write-performance-for-sequential-alpha-numeric-keys-tp4034219p4034225.html > Sent from the Riak Users mailing list archive at Nabble.com. > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
