Ingo - As an FYI, once I started looking into this I found you had stumbled into quite the can of worms. I just finished a pretty comprehensive set of changes that will now allow proper handling of tombstones in the Java client - https://github.com/basho/riak-java-client/pull/195
Sorry you had to be the one, but thanks for letting us know about this! - Roach On Thu, Jan 31, 2013 at 10:36 AM, Ingo Rockel <ingo.roc...@bluelionmobile.com> wrote: > Hi Brian, > > thanks for the suggestion but I already chose a different solution for now, > if these messages get deleted I just delete the links to the message and > mark the message as "abandoned" and available for reuse. So I don't run into > the conflict if I need to store the message again. > > I just started the replay again and let it run for a while to see if this > works for me. > > Thanks! > > Ingo > > Am 31.01.2013 16:54, schrieb Brian Roach: > >> Ingo - >> >> Unfortunately once you've got a sibling tombstone, things get a bit >> tricky. It's not going away until you resolve them which when using >> the JSONConverter in the Java client, you can't. Oddly enough, this is >> the first time anyone has hit this. >> >> I've got a couple ideas on how to address this properly but I need to >> look at some things first. >> >> In the meantime, what I'd suggest as a workaround is to copy and paste >> the source for the JSONConverter into your own Converter<T> that >> you'll pass to the StoreObject and modify it to return null: >> >> >> https://github.com/basho/riak-java-client/blob/master/src/main/java/com/basho/riak/client/convert/JSONConverter.java#L141 >> >> Have it check to see if riakObject.getValue() returns null and if it >> does ... return null. You'll also need to modify your ConflictResolver >> to check for null as it iterates through the list of your POJOs that >> gets passed to it and act accordingly. If there's only a tombstone, >> just return null ... which means you will also need to modify your >> Mutation<T> to handle a null being passed to it in the case of there >> only being a tombstone. >> >> In the end this may well be what I do but I think I have a slightly >> more elegant solution that I want to look into. >> >> I've got an errand I need to run this morning, but I'll get to work on >> this as soon as I get back. >> >> Thanks, and sorry for the trouble. >> - Brian Roach >> >> >> On Thu, Jan 31, 2013 at 3:56 AM, Ingo Rockel >> <ingo.roc...@bluelionmobile.com> wrote: >>> >>> Hi Brian, >>> >>> thanks for the detailed explaination! >>> >>> I had a look at an object which constantly fails to load even if >>> retrying: >>> >>> lftp :~> cat "http://172.22.3.14:8091/riak/m/Um|18498012|4|0|18298081" >>> ---- Verbinde mit 172.22.3.14 (172.22.3.14) Port 8091 >>> ---> GET /riak/m/Um|18498012|4|0|18298081 HTTP/1.1 >>> ---> Host: 172.22.3.14:8091 >>> ---> User-Agent: lftp/4.3.3 >>> ---> Accept: */* >>> ---> Connection: keep-alive >>> ---> >>> <--- HTTP/1.1 300 Multiple Choices >>> <--- X-Riak-Vclock: a85hYGBgzGDKBVIcaZPWMQZyWttkMCWy5rEyXNhTd4ovCwA= >>> <--- Vary: Accept, Accept-Encoding >>> <--- Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue) >>> <--- Last-Modified: Thu, 31 Jan 2013 10:00:48 GMT >>> <--- ETag: "6PSreYIOL25KOpNyG0XPe7" >>> <--- Date: Thu, 31 Jan 2013 10:42:41 GMT >>> <--- Content-Type: text/plain >>> <--- Content-Length: 56 >>> <--- >>> <--* Siblings: >>> <--* 50Uz9nvQWwOUBE6USi2gki >>> <--* 1JsgLs3CE3k2mWsaCEiPp4 >>> cat: Zugriff nicht möglich: 300 Multiple Choices >>> (/riak/m/Um|18498012|4|0|18298081) >>> lftp :~> cat >>> >>> "http://172.22.3.14:8091/riak/m/Um|18498012|4|0|18298081?vtag=50Uz9nvQWwOUBE6USi2gki" >>> ---> GET /riak/m/Um|18498012|4|0|18298081?vtag=50Uz9nvQWwOUBE6USi2gki >>> HTTP/1.1 >>> ---> Host: 172.22.3.14:8091 >>> ---> User-Agent: lftp/4.3.3 >>> ---> Accept: */* >>> ---> Connection: keep-alive >>> ---> >>> <--- HTTP/1.1 200 OK >>> <--- X-Riak-Vclock: a85hYGBgzGDKBVIcaZPWMQZyWttkMCWy5rEyXNhTd4ovCwA= >>> <--- Vary: Accept-Encoding >>> <--- Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue) >>> <--- Link: </riak/m>; rel="up" >>> <--- Last-Modified: Thu, 31 Jan 2013 09:57:38 GMT >>> <--- ETag: "50Uz9nvQWwOUBE6USi2gki" >>> <--- Date: Thu, 31 Jan 2013 10:42:49 GMT >>> <--- Content-Type: application/octet-stream >>> <--- Content-Length: 0 >>> <--- >>> lftp :~> cat >>> >>> "http://172.22.3.14:8091/riak/m/Um|18498012|4|0|18298081?vtag=1JsgLs3CE3k2mWsaCEiPp4" >>> ---> GET /riak/m/Um|18498012|4|0|18298081?vtag=1JsgLs3CE3k2mWsaCEiPp4 >>> HTTP/1.1 >>> ---> Host: 172.22.3.14:8091 >>> ---> User-Agent: lftp/4.3.3 >>> ---> Accept: */* >>> ---> Connection: keep-alive >>> ---> >>> <--- HTTP/1.1 200 OK >>> <--- X-Riak-Vclock: a85hYGBgzGDKBVIcaZPWMQZyWttkMCWy5rEyXNhTd4ovCwA= >>> <--- Vary: Accept-Encoding >>> <--- Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue) >>> <--- Link: </riak/m>; rel="up" >>> <--- Last-Modified: Thu, 31 Jan 2013 10:00:48 GMT >>> <--- ETag: "1JsgLs3CE3k2mWsaCEiPp4" >>> <--- Date: Thu, 31 Jan 2013 10:43:01 GMT >>> <--- Content-Type: application/json; charset=UTF-8 >>> <--- Content-Length: 114 >>> <--- >>> >>> {"sortKey":1359626448000106,"st":2,"t":4,"r":18498012,"s":18298081,"ct":1359626448000,"rv":21215685,"cv":1,"su":0} >>> >>> the object has two siblings, one the deleted empty "tombstone" and one >>> with >>> the new data. And there's a gap auf 2:30min between both siblings. >>> There's >>> no immediate write after the deletion. I logged the write operations and >>> this gap is there also. And the java client constantly fails to load this >>> object. >>> >>> This objects are user notifications which are identified by their type >>> (the >>> key) and can be updated frequently but the user also is allowed to delete >>> a >>> single notification (which he did in this case). >>> >>> Is there anything I can do to resolve this? A last write wins just for >>> this >>> case? If I write a new notification to update the old one I don't care if >>> there's any deletion tombstone... >>> >>> Ingo >>> >>> Am 30.01.2013 22:42, schrieb Brian Roach: >>> >>>> Ingo - >>>> >>>> Riak is returning an object with no contents (which ends up being an >>>> empty String passed to Jackson). >>>> >>>> Unless you've somehow mangled the data yourself (which sounds unlikely >>>> given the bit about the 404 from the command line; more on that in a >>>> bit) what's happening is that you're encountering a tombstone; an >>>> object that has been deleted via a delete operation but hasn't been >>>> removed yet. This causes an "empty" object to be returned (the >>>> tombstone) and causes Jackson to puke (HTTP will actually return this >>>> as a 404 but if you look there's still a X-Riak-Vclock: header with a >>>> vclock). >>>> >>>> Probably the best description of how this works in Riak is a post by >>>> Jon Meredith which can be found here: >>>> >>>> >>>> http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-October/006048.html >>>> >>>> Unfortunately this is something the Java client doesn't know what to >>>> do with when using the default JSONConverter and your own POJOs. And >>>> it's not as simple as "just return null" because of a case where a >>>> tombstone could actually be a sibling and the client then needs to >>>> resolve the conflict which is the next step in the process. It's >>>> something I'm going to have to think about. >>>> >>>> As you've discovered, when the tombstone isn't a sibling simply >>>> retrying will often work because by then the delete has fully >>>> completed and the tombstones have been removed from the Riak nodes. >>>> >>>> Is there a reason you're rapidly doing a delete then a store (which >>>> triggers that fetch)? >>>> >>>> Thanks, >>>> Brian Roach >>>> >>>> On Wed, Jan 30, 2013 at 9:06 AM, Ingo Rockel >>>> <ingo.roc...@bluelionmobile.com> wrote: >>>>> >>>>> >>>>> Hi Dmitri, >>>>> >>>>> it doesn't happen in my code and it does happen while the riak-client >>>>> tries >>>>> to deserialize a fetched object from riak in a "fetch-before-store" >>>>> (see >>>>> the >>>>> stack), I also get this error randomly while trying just to fetch an >>>>> object >>>>> from the database. >>>>> >>>>> And if I try to fetch the object from the cmdline I just get a 404. So >>>>> I >>>>> would expect the java-client just returns a null-result for this fetch >>>>> and >>>>> not to throw an exception. >>>>> >>>>> All my objects are stored using the riak-java-client and the >>>>> json-serializer. >>>>> >>>>> Ahh, just tested: if I retry it sometimes works, although most of the >>>>> time >>>>> still fails (haven't tried with a sleep so far). >>>>> >>>>> Ingo >>>>> >>>>> Am 30.01.2013 16:57, schrieb Dmitri Zagidulin: >>>>>> >>>>>> >>>>>> >>>>>> Hi Ingo. >>>>>> >>>>>> It's difficult to diagnose the exact reason without looking at your >>>>>> code. >>>>>> But that error is a JSON parser error. It gets thrown whenever the >>>>>> code >>>>>> tries to parse an empty string as a json object. >>>>>> The general-case solution is to validate your strings or input streams >>>>>> that you're turning into JSON objects, or to catch an exception when >>>>>> creating that object and deal with it accordingly. >>>>>> >>>>>> But again, it's hard to say why it's happening exactly, in your case >>>>>> -- >>>>>> try to determine where in your code that's happening and think of ways >>>>>> some input or result is empty, and check for that. >>>>>> >>>>>> Dmitri >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Jan 30, 2013 at 10:44 AM, Ingo Rockel >>>>>> <ingo.roc...@bluelionmobile.com >>>>>> <mailto:ingo.roc...@bluelionmobile.com>> >>>>>> >>>>>> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> I wrote a java tool to convert part of our data from a >>>>>> mysql-database into riak. As this tool is running while our >>>>>> system >>>>>> is still up, it needs to replay all modifications done in the >>>>>> mysql >>>>>> database, during these modifications I sometimes get this >>>>>> exception >>>>>> from the riak client: >>>>>> >>>>>> com.basho.riak.client.convert.__ConversionException: >>>>>> >>>>>> java.io.EOFException: No content to map to Object due to end of >>>>>> input >>>>>> com.basho.riak.client.convert.__ConversionException: >>>>>> >>>>>> java.io.EOFException: No content to map to Object due to end of >>>>>> input >>>>>> at >>>>>> >>>>>> >>>>>> >>>>>> com.basho.riak.client.convert.__JSONConverter.toDomain(__JSONConverter.java:167) >>>>>> at >>>>>> >>>>>> >>>>>> >>>>>> com.basho.riak.client.__operations.FetchObject.__execute(FetchObject.java:110) >>>>>> at >>>>>> >>>>>> >>>>>> >>>>>> com.basho.riak.client.__operations.StoreObject.__execute(StoreObject.java:112) >>>>>> at >>>>>> >>>>>> >>>>>> >>>>>> com.bluelionmobile.qeep.__messaging.db.impl.__MessageKVImpl.__storeUniqueMessageDto(__MessageKVImpl.java:264) >>>>>> at >>>>>> >>>>>> >>>>>> >>>>>> com.bluelionmobile.qeep.__messaging.db.impl.__MessageKVImpl.__createDataFromDTO(__MessageKVImpl.java:138) >>>>>> at >>>>>> >>>>>> >>>>>> >>>>>> com.bluelionmobile.qeep.__messaging.db.impl.__MessageKVImpl.__updateDataFromDTO(__MessageKVImpl.java:205) >>>>>> at >>>>>> >>>>>> >>>>>> >>>>>> com.bluelionmobile.qeep.__messaging.db.utils.Replay$__ReplayRunner.run(Replay.java:__243) >>>>>> at java.lang.Thread.run(Thread.__java:722) >>>>>> >>>>>> Caused by: java.io.EOFException: No content to map to Object due >>>>>> to >>>>>> end of input >>>>>> at >>>>>> >>>>>> >>>>>> >>>>>> org.codehaus.jackson.map.__ObjectMapper._initForReading(__ObjectMapper.java:2775) >>>>>> at >>>>>> >>>>>> >>>>>> >>>>>> org.codehaus.jackson.map.__ObjectMapper._readMapAndClose(__ObjectMapper.java:2718) >>>>>> at >>>>>> >>>>>> >>>>>> >>>>>> org.codehaus.jackson.map.__ObjectMapper.readValue(__ObjectMapper.java:1863) >>>>>> at >>>>>> >>>>>> >>>>>> >>>>>> com.basho.riak.client.convert.__JSONConverter.toDomain(__JSONConverter.java:156) >>>>>> >>>>>> ... 7 more >>>>>> >>>>>> it only happens once every few thousand updates and if I check >>>>>> the >>>>>> object from the cmdline I only get a 404. >>>>>> >>>>>> Any ideas what might cause this and how to fix/workaround it? >>>>>> >>>>>> Ingo >>>>>> >>>>>> >>>>>> _________________________________________________ >>>>>> riak-users mailing list >>>>>> riak-users@lists.basho.com <mailto:riak-users@lists.basho.com> >>>>>> >>>>>> http://lists.basho.com/__mailman/listinfo/riak-users___lists.basho.com >>>>>> >>>>>> <http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> riak-users mailing list >>>>>> riak-users@lists.basho.com >>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>>> >>>>> >>>>> >>>>> -- >>>>> Software Architect >>>>> >>>>> Blue Lion mobile GmbH >>>>> Tel. +49 (0) 221 788 797 14 >>>>> Fax. +49 (0) 221 788 797 19 >>>>> Mob. +49 (0) 176 24 87 30 89 >>>>> >>>>> ingo.roc...@bluelionmobile.com >>>>>>>> >>>>>>>> >>>>>>>> qeep: Hefferwolf >>>>> >>>>> >>>>> >>>>> www.bluelionmobile.com >>>>> www.qeep.net >>>>> >>>>> >>>>> _______________________________________________ >>>>> riak-users mailing list >>>>> riak-users@lists.basho.com >>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >>> >>> >>> -- >>> Software Architect >>> >>> Blue Lion mobile GmbH >>> Tel. +49 (0) 221 788 797 14 >>> Fax. +49 (0) 221 788 797 19 >>> Mob. +49 (0) 176 24 87 30 89 >>> >>> ingo.roc...@bluelionmobile.com >>>>>> >>>>>> qeep: Hefferwolf >>> >>> >>> www.bluelionmobile.com >>> www.qeep.net > > > > -- > Software Architect > > Blue Lion mobile GmbH > Tel. +49 (0) 221 788 797 14 > Fax. +49 (0) 221 788 797 19 > Mob. +49 (0) 176 24 87 30 89 > > ingo.roc...@bluelionmobile.com >>>> qeep: Hefferwolf > > www.bluelionmobile.com > www.qeep.net _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
