Ingo - Unfortunately once you've got a sibling tombstone, things get a bit tricky. It's not going away until you resolve them which when using the JSONConverter in the Java client, you can't. Oddly enough, this is the first time anyone has hit this.
I've got a couple ideas on how to address this properly but I need to look at some things first. In the meantime, what I'd suggest as a workaround is to copy and paste the source for the JSONConverter into your own Converter<T> that you'll pass to the StoreObject and modify it to return null: https://github.com/basho/riak-java-client/blob/master/src/main/java/com/basho/riak/client/convert/JSONConverter.java#L141 Have it check to see if riakObject.getValue() returns null and if it does ... return null. You'll also need to modify your ConflictResolver to check for null as it iterates through the list of your POJOs that gets passed to it and act accordingly. If there's only a tombstone, just return null ... which means you will also need to modify your Mutation<T> to handle a null being passed to it in the case of there only being a tombstone. In the end this may well be what I do but I think I have a slightly more elegant solution that I want to look into. I've got an errand I need to run this morning, but I'll get to work on this as soon as I get back. Thanks, and sorry for the trouble. - Brian Roach On Thu, Jan 31, 2013 at 3:56 AM, Ingo Rockel <ingo.roc...@bluelionmobile.com> wrote: > Hi Brian, > > thanks for the detailed explaination! > > I had a look at an object which constantly fails to load even if retrying: > > lftp :~> cat "http://172.22.3.14:8091/riak/m/Um|18498012|4|0|18298081" > ---- Verbinde mit 172.22.3.14 (172.22.3.14) Port 8091 > ---> GET /riak/m/Um|18498012|4|0|18298081 HTTP/1.1 > ---> Host: 172.22.3.14:8091 > ---> User-Agent: lftp/4.3.3 > ---> Accept: */* > ---> Connection: keep-alive > ---> > <--- HTTP/1.1 300 Multiple Choices > <--- X-Riak-Vclock: a85hYGBgzGDKBVIcaZPWMQZyWttkMCWy5rEyXNhTd4ovCwA= > <--- Vary: Accept, Accept-Encoding > <--- Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue) > <--- Last-Modified: Thu, 31 Jan 2013 10:00:48 GMT > <--- ETag: "6PSreYIOL25KOpNyG0XPe7" > <--- Date: Thu, 31 Jan 2013 10:42:41 GMT > <--- Content-Type: text/plain > <--- Content-Length: 56 > <--- > <--* Siblings: > <--* 50Uz9nvQWwOUBE6USi2gki > <--* 1JsgLs3CE3k2mWsaCEiPp4 > cat: Zugriff nicht möglich: 300 Multiple Choices > (/riak/m/Um|18498012|4|0|18298081) > lftp :~> cat > "http://172.22.3.14:8091/riak/m/Um|18498012|4|0|18298081?vtag=50Uz9nvQWwOUBE6USi2gki" > ---> GET /riak/m/Um|18498012|4|0|18298081?vtag=50Uz9nvQWwOUBE6USi2gki > HTTP/1.1 > ---> Host: 172.22.3.14:8091 > ---> User-Agent: lftp/4.3.3 > ---> Accept: */* > ---> Connection: keep-alive > ---> > <--- HTTP/1.1 200 OK > <--- X-Riak-Vclock: a85hYGBgzGDKBVIcaZPWMQZyWttkMCWy5rEyXNhTd4ovCwA= > <--- Vary: Accept-Encoding > <--- Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue) > <--- Link: </riak/m>; rel="up" > <--- Last-Modified: Thu, 31 Jan 2013 09:57:38 GMT > <--- ETag: "50Uz9nvQWwOUBE6USi2gki" > <--- Date: Thu, 31 Jan 2013 10:42:49 GMT > <--- Content-Type: application/octet-stream > <--- Content-Length: 0 > <--- > lftp :~> cat > "http://172.22.3.14:8091/riak/m/Um|18498012|4|0|18298081?vtag=1JsgLs3CE3k2mWsaCEiPp4" > ---> GET /riak/m/Um|18498012|4|0|18298081?vtag=1JsgLs3CE3k2mWsaCEiPp4 > HTTP/1.1 > ---> Host: 172.22.3.14:8091 > ---> User-Agent: lftp/4.3.3 > ---> Accept: */* > ---> Connection: keep-alive > ---> > <--- HTTP/1.1 200 OK > <--- X-Riak-Vclock: a85hYGBgzGDKBVIcaZPWMQZyWttkMCWy5rEyXNhTd4ovCwA= > <--- Vary: Accept-Encoding > <--- Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue) > <--- Link: </riak/m>; rel="up" > <--- Last-Modified: Thu, 31 Jan 2013 10:00:48 GMT > <--- ETag: "1JsgLs3CE3k2mWsaCEiPp4" > <--- Date: Thu, 31 Jan 2013 10:43:01 GMT > <--- Content-Type: application/json; charset=UTF-8 > <--- Content-Length: 114 > <--- > {"sortKey":1359626448000106,"st":2,"t":4,"r":18498012,"s":18298081,"ct":1359626448000,"rv":21215685,"cv":1,"su":0} > > the object has two siblings, one the deleted empty "tombstone" and one with > the new data. And there's a gap auf 2:30min between both siblings. There's > no immediate write after the deletion. I logged the write operations and > this gap is there also. And the java client constantly fails to load this > object. > > This objects are user notifications which are identified by their type (the > key) and can be updated frequently but the user also is allowed to delete a > single notification (which he did in this case). > > Is there anything I can do to resolve this? A last write wins just for this > case? If I write a new notification to update the old one I don't care if > there's any deletion tombstone... > > Ingo > > Am 30.01.2013 22:42, schrieb Brian Roach: > >> Ingo - >> >> Riak is returning an object with no contents (which ends up being an >> empty String passed to Jackson). >> >> Unless you've somehow mangled the data yourself (which sounds unlikely >> given the bit about the 404 from the command line; more on that in a >> bit) what's happening is that you're encountering a tombstone; an >> object that has been deleted via a delete operation but hasn't been >> removed yet. This causes an "empty" object to be returned (the >> tombstone) and causes Jackson to puke (HTTP will actually return this >> as a 404 but if you look there's still a X-Riak-Vclock: header with a >> vclock). >> >> Probably the best description of how this works in Riak is a post by >> Jon Meredith which can be found here: >> >> http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-October/006048.html >> >> Unfortunately this is something the Java client doesn't know what to >> do with when using the default JSONConverter and your own POJOs. And >> it's not as simple as "just return null" because of a case where a >> tombstone could actually be a sibling and the client then needs to >> resolve the conflict which is the next step in the process. It's >> something I'm going to have to think about. >> >> As you've discovered, when the tombstone isn't a sibling simply >> retrying will often work because by then the delete has fully >> completed and the tombstones have been removed from the Riak nodes. >> >> Is there a reason you're rapidly doing a delete then a store (which >> triggers that fetch)? >> >> Thanks, >> Brian Roach >> >> On Wed, Jan 30, 2013 at 9:06 AM, Ingo Rockel >> <ingo.roc...@bluelionmobile.com> wrote: >>> >>> Hi Dmitri, >>> >>> it doesn't happen in my code and it does happen while the riak-client >>> tries >>> to deserialize a fetched object from riak in a "fetch-before-store" (see >>> the >>> stack), I also get this error randomly while trying just to fetch an >>> object >>> from the database. >>> >>> And if I try to fetch the object from the cmdline I just get a 404. So I >>> would expect the java-client just returns a null-result for this fetch >>> and >>> not to throw an exception. >>> >>> All my objects are stored using the riak-java-client and the >>> json-serializer. >>> >>> Ahh, just tested: if I retry it sometimes works, although most of the >>> time >>> still fails (haven't tried with a sleep so far). >>> >>> Ingo >>> >>> Am 30.01.2013 16:57, schrieb Dmitri Zagidulin: >>>> >>>> >>>> Hi Ingo. >>>> >>>> It's difficult to diagnose the exact reason without looking at your >>>> code. >>>> But that error is a JSON parser error. It gets thrown whenever the code >>>> tries to parse an empty string as a json object. >>>> The general-case solution is to validate your strings or input streams >>>> that you're turning into JSON objects, or to catch an exception when >>>> creating that object and deal with it accordingly. >>>> >>>> But again, it's hard to say why it's happening exactly, in your case -- >>>> try to determine where in your code that's happening and think of ways >>>> some input or result is empty, and check for that. >>>> >>>> Dmitri >>>> >>>> >>>> >>>> On Wed, Jan 30, 2013 at 10:44 AM, Ingo Rockel >>>> <ingo.roc...@bluelionmobile.com <mailto:ingo.roc...@bluelionmobile.com>> >>>> >>>> wrote: >>>> >>>> Hi, >>>> >>>> I wrote a java tool to convert part of our data from a >>>> mysql-database into riak. As this tool is running while our system >>>> is still up, it needs to replay all modifications done in the mysql >>>> database, during these modifications I sometimes get this exception >>>> from the riak client: >>>> >>>> com.basho.riak.client.convert.__ConversionException: >>>> >>>> java.io.EOFException: No content to map to Object due to end of >>>> input >>>> com.basho.riak.client.convert.__ConversionException: >>>> >>>> java.io.EOFException: No content to map to Object due to end of >>>> input >>>> at >>>> >>>> >>>> com.basho.riak.client.convert.__JSONConverter.toDomain(__JSONConverter.java:167) >>>> at >>>> >>>> >>>> com.basho.riak.client.__operations.FetchObject.__execute(FetchObject.java:110) >>>> at >>>> >>>> >>>> com.basho.riak.client.__operations.StoreObject.__execute(StoreObject.java:112) >>>> at >>>> >>>> >>>> com.bluelionmobile.qeep.__messaging.db.impl.__MessageKVImpl.__storeUniqueMessageDto(__MessageKVImpl.java:264) >>>> at >>>> >>>> >>>> com.bluelionmobile.qeep.__messaging.db.impl.__MessageKVImpl.__createDataFromDTO(__MessageKVImpl.java:138) >>>> at >>>> >>>> >>>> com.bluelionmobile.qeep.__messaging.db.impl.__MessageKVImpl.__updateDataFromDTO(__MessageKVImpl.java:205) >>>> at >>>> >>>> >>>> com.bluelionmobile.qeep.__messaging.db.utils.Replay$__ReplayRunner.run(Replay.java:__243) >>>> at java.lang.Thread.run(Thread.__java:722) >>>> >>>> Caused by: java.io.EOFException: No content to map to Object due to >>>> end of input >>>> at >>>> >>>> >>>> org.codehaus.jackson.map.__ObjectMapper._initForReading(__ObjectMapper.java:2775) >>>> at >>>> >>>> >>>> org.codehaus.jackson.map.__ObjectMapper._readMapAndClose(__ObjectMapper.java:2718) >>>> at >>>> >>>> >>>> org.codehaus.jackson.map.__ObjectMapper.readValue(__ObjectMapper.java:1863) >>>> at >>>> >>>> >>>> com.basho.riak.client.convert.__JSONConverter.toDomain(__JSONConverter.java:156) >>>> >>>> ... 7 more >>>> >>>> it only happens once every few thousand updates and if I check the >>>> object from the cmdline I only get a 404. >>>> >>>> Any ideas what might cause this and how to fix/workaround it? >>>> >>>> Ingo >>>> >>>> >>>> _________________________________________________ >>>> riak-users mailing list >>>> riak-users@lists.basho.com <mailto:riak-users@lists.basho.com> >>>> >>>> http://lists.basho.com/__mailman/listinfo/riak-users___lists.basho.com >>>> >>>> <http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com> >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> riak-users mailing list >>>> riak-users@lists.basho.com >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>> >>> >>> >>> -- >>> Software Architect >>> >>> Blue Lion mobile GmbH >>> Tel. +49 (0) 221 788 797 14 >>> Fax. +49 (0) 221 788 797 19 >>> Mob. +49 (0) 176 24 87 30 89 >>> >>> ingo.roc...@bluelionmobile.com >>>>>> >>>>>> qeep: Hefferwolf >>> >>> >>> www.bluelionmobile.com >>> www.qeep.net >>> >>> >>> _______________________________________________ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > -- > Software Architect > > Blue Lion mobile GmbH > Tel. +49 (0) 221 788 797 14 > Fax. +49 (0) 221 788 797 19 > Mob. +49 (0) 176 24 87 30 89 > > ingo.roc...@bluelionmobile.com >>>> qeep: Hefferwolf > > www.bluelionmobile.com > www.qeep.net _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
