Re: Riak: reliable object deletion

[al so]

Implications when *delete_mode *set to 'keep'*?*


On Thu, Feb 23, 2017 at 9:53 PM, al so  wrote:

> Present both in Solr and Riak. It's a 1-way replication in MDC. MDC is not
>> the cause unless there is a bug there as well.
>>
>>
>> On Thu, Feb 23, 2017 at 6:02 AM, Fred Dushin  wrote:
>>
>>> Running a Solr query has no impact on writes -- Riak search queries are
>>> direct pass throughs to Solr query and don't touch any of the salient Riak
>>> systems (batching writes to Solr, YZ AAE, etc).  I believe the timing of
>>> the reappearance is a coincidence.
>>>
>>> Is it possible the object reappeared via MDC?  Do you have record of the
>>> reappeared object in your other cluster?
>>>
>>> Also, just to confirm, when you say the object reappeared -- it
>>> re-appeared in Riak (and subsequently in Solr), correct?  Or are you saying
>>> you just find the object in Solr (via a search)?
>>>
>>> -Fred
>>>
>>> On Feb 23, 2017, at 1:43 AM, al so  wrote:
>>>
>>>  Here is brief env:
  Riak v 2.0.8 + Solr/ 5 node cluster/ MDC

  Problem:
   Deleted object suddenly resurrected after few days. Solr search query(
 "*:*") was executed around the time of reappearance.

  Bucket property for this reappeared object
  {
   "props": {
 "name": "UsaHype",
 "allow_mult": false,
 "basic_quorum": false,
 "big_vclock": 50,
 "chash_keyfun": {
   "mod": "riak_core_util",
   "fun": "chash_std_keyfun"
 },
 "dvv_enabled": true,
 "dw": "quorum",
 "last_write_wins": false,
 "linkfun": {
   "mod": "riak_kv_wm_link_walker",
   "fun": "mapreduce_linkfun"
 },
 "n_val": 3,
 "notfound_ok": true,
 "old_vclock": 86400,
 "postcommit": [
   {
 "mod": "riak_repl_leader",
 "fun": "postcommit"
   },
   {
 "mod": "riak_repl2_rt",
 "fun": "postcommit"
   }
 ],
 "pr": 0,
 "precommit": [

 ],
 "pw": 0,
 "r": "quorum",
 "repl": true,
 "rw": "quorum",
 "small_vclock": 50,
 "w": "quorum",
 "young_vclock": 20
   }
 }


 * delete_mode is not configured at all. Defaults to 3 sec? Do Tombstone 
 still get created when no delete_mode at all in the config? Can we query 
 metadata(*X-Riak-Deleted)* for this reappeared object.*

  How would one go about finding root cause. I do know the date when the
 object was deleted from our app logs. I also seem to know when it
 reappeared. Seems like a solr search ("*:*") in code was executed just
 before this object reappeared? Look at AAE? Is there a way to find if Solr
 and Riak backends are out-of-sync (AAE?).

  I understand tuning other params (allow_mult = true, ...) will have
 its own implications.

  In summary:
   How do I find root cause of this issue.
   How to reliable delete an object.

 -Volk

>>>
>>>
>>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak: reliable object deletion

[al so]

>
> Present both in Solr and Riak. It's a 1-way replication in MDC. MDC is not
> the cause unless there is a bug there as well.
>
>
> On Thu, Feb 23, 2017 at 6:02 AM, Fred Dushin  wrote:
>
>> Running a Solr query has no impact on writes -- Riak search queries are
>> direct pass throughs to Solr query and don't touch any of the salient Riak
>> systems (batching writes to Solr, YZ AAE, etc).  I believe the timing of
>> the reappearance is a coincidence.
>>
>> Is it possible the object reappeared via MDC?  Do you have record of the
>> reappeared object in your other cluster?
>>
>> Also, just to confirm, when you say the object reappeared -- it
>> re-appeared in Riak (and subsequently in Solr), correct?  Or are you saying
>> you just find the object in Solr (via a search)?
>>
>> -Fred
>>
>> On Feb 23, 2017, at 1:43 AM, al so  wrote:
>>
>>  Here is brief env:
>>>  Riak v 2.0.8 + Solr/ 5 node cluster/ MDC
>>>
>>>  Problem:
>>>   Deleted object suddenly resurrected after few days. Solr search query(
>>> "*:*") was executed around the time of reappearance.
>>>
>>>  Bucket property for this reappeared object
>>>  {
>>>   "props": {
>>> "name": "UsaHype",
>>> "allow_mult": false,
>>> "basic_quorum": false,
>>> "big_vclock": 50,
>>> "chash_keyfun": {
>>>   "mod": "riak_core_util",
>>>   "fun": "chash_std_keyfun"
>>> },
>>> "dvv_enabled": true,
>>> "dw": "quorum",
>>> "last_write_wins": false,
>>> "linkfun": {
>>>   "mod": "riak_kv_wm_link_walker",
>>>   "fun": "mapreduce_linkfun"
>>> },
>>> "n_val": 3,
>>> "notfound_ok": true,
>>> "old_vclock": 86400,
>>> "postcommit": [
>>>   {
>>> "mod": "riak_repl_leader",
>>> "fun": "postcommit"
>>>   },
>>>   {
>>> "mod": "riak_repl2_rt",
>>> "fun": "postcommit"
>>>   }
>>> ],
>>> "pr": 0,
>>> "precommit": [
>>>
>>> ],
>>> "pw": 0,
>>> "r": "quorum",
>>> "repl": true,
>>> "rw": "quorum",
>>> "small_vclock": 50,
>>> "w": "quorum",
>>> "young_vclock": 20
>>>   }
>>> }
>>>
>>>
>>> * delete_mode is not configured at all. Defaults to 3 sec? Do Tombstone 
>>> still get created when no delete_mode at all in the config? Can we query 
>>> metadata(*X-Riak-Deleted)* for this reappeared object.*
>>>
>>>  How would one go about finding root cause. I do know the date when the
>>> object was deleted from our app logs. I also seem to know when it
>>> reappeared. Seems like a solr search ("*:*") in code was executed just
>>> before this object reappeared? Look at AAE? Is there a way to find if Solr
>>> and Riak backends are out-of-sync (AAE?).
>>>
>>>  I understand tuning other params (allow_mult = true, ...) will have its
>>> own implications.
>>>
>>>  In summary:
>>>   How do I find root cause of this issue.
>>>   How to reliable delete an object.
>>>
>>> -Volk
>>>
>>
>>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak: reliable object deletion

[al so]

Present both in Solr and Riak.

On Thu, Feb 23, 2017 at 6:02 AM, Fred Dushin  wrote:

> Running a Solr query has no impact on writes -- Riak search queries are
> direct pass throughs to Solr query and don't touch any of the salient Riak
> systems (batching writes to Solr, YZ AAE, etc).  I believe the timing of
> the reappearance is a coincidence.
>
> Is it possible the object reappeared via MDC?  Do you have record of the
> reappeared object in your other cluster?
>
> Also, just to confirm, when you say the object reappeared -- it
> re-appeared in Riak (and subsequently in Solr), correct?  Or are you saying
> you just find the object in Solr (via a search)?
>
> -Fred
>
> On Feb 23, 2017, at 1:43 AM, al so  wrote:
>
>  Here is brief env:
>>  Riak v 2.0.8 + Solr/ 5 node cluster/ MDC
>>
>>  Problem:
>>   Deleted object suddenly resurrected after few days. Solr search query(
>> "*:*") was executed around the time of reappearance.
>>
>>  Bucket property for this reappeared object
>>  {
>>   "props": {
>> "name": "UsaHype",
>> "allow_mult": false,
>> "basic_quorum": false,
>> "big_vclock": 50,
>> "chash_keyfun": {
>>   "mod": "riak_core_util",
>>   "fun": "chash_std_keyfun"
>> },
>> "dvv_enabled": true,
>> "dw": "quorum",
>> "last_write_wins": false,
>> "linkfun": {
>>   "mod": "riak_kv_wm_link_walker",
>>   "fun": "mapreduce_linkfun"
>> },
>> "n_val": 3,
>> "notfound_ok": true,
>> "old_vclock": 86400,
>> "postcommit": [
>>   {
>> "mod": "riak_repl_leader",
>> "fun": "postcommit"
>>   },
>>   {
>> "mod": "riak_repl2_rt",
>> "fun": "postcommit"
>>   }
>> ],
>> "pr": 0,
>> "precommit": [
>>
>> ],
>> "pw": 0,
>> "r": "quorum",
>> "repl": true,
>> "rw": "quorum",
>> "small_vclock": 50,
>> "w": "quorum",
>> "young_vclock": 20
>>   }
>> }
>>
>>
>> * delete_mode is not configured at all. Defaults to 3 sec? Do Tombstone 
>> still get created when no delete_mode at all in the config? Can we query 
>> metadata(*X-Riak-Deleted)* for this reappeared object.*
>>
>>  How would one go about finding root cause. I do know the date when the
>> object was deleted from our app logs. I also seem to know when it
>> reappeared. Seems like a solr search ("*:*") in code was executed just
>> before this object reappeared? Look at AAE? Is there a way to find if Solr
>> and Riak backends are out-of-sync (AAE?).
>>
>>  I understand tuning other params (allow_mult = true, ...) will have its
>> own implications.
>>
>>  In summary:
>>   How do I find root cause of this issue.
>>   How to reliable delete an object.
>>
>> -Volk
>>
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak: reliable object deletion

[Fred Dushin]

Running a Solr query has no impact on writes -- Riak search queries are direct 
pass throughs to Solr query and don't touch any of the salient Riak systems 
(batching writes to Solr, YZ AAE, etc).  I believe the timing of the 
reappearance is a coincidence.

Is it possible the object reappeared via MDC?  Do you have record of the 
reappeared object in your other cluster?

Also, just to confirm, when you say the object reappeared -- it re-appeared in 
Riak (and subsequently in Solr), correct?  Or are you saying you just find the 
object in Solr (via a search)?

-Fred

> On Feb 23, 2017, at 1:43 AM, al so  wrote:
> 
>  Here is brief env:
>  Riak v 2.0.8 + Solr/ 5 node cluster/ MDC
>  
>  Problem:
>   Deleted object suddenly resurrected after few days. Solr search 
> query("*:*") was executed around the time of reappearance.
> 
>  Bucket property for this reappeared object
>  {
>   "props": {
> "name": "UsaHype",
> "allow_mult": false,
> "basic_quorum": false,
> "big_vclock": 50,
> "chash_keyfun": {
>   "mod": "riak_core_util",
>   "fun": "chash_std_keyfun"
> },
> "dvv_enabled": true,
> "dw": "quorum",
> "last_write_wins": false,
> "linkfun": {
>   "mod": "riak_kv_wm_link_walker",
>   "fun": "mapreduce_linkfun"
> },
> "n_val": 3,
> "notfound_ok": true,
> "old_vclock": 86400,
> "postcommit": [
>   {
> "mod": "riak_repl_leader",
> "fun": "postcommit"
>   },
>   {
> "mod": "riak_repl2_rt",
> "fun": "postcommit"
>   }
> ],
> "pr": 0,
> "precommit": [
>   
> ],
> "pw": 0,
> "r": "quorum",
> "repl": true,
> "rw": "quorum",
> "small_vclock": 50,
> "w": "quorum",
> "young_vclock": 20
>   }
> }
>  
>  delete_mode is not configured at all. Defaults to 3 sec? Do Tombstone still 
> get created when no delete_mode at all in the config? Can we query 
> metadata(X-Riak-Deleted) for this reappeared object.
>  How would one go about finding root cause. I do know the date when the 
> object was deleted from our app logs. I also seem to know when it reappeared. 
> Seems like a solr search ("*:*") in code was executed just before this object 
> reappeared? Look at AAE? Is there a way to find if Solr and Riak backends are 
> out-of-sync (AAE?). 
> 
>  I understand tuning other params (allow_mult = true, ...) will have its own 
> implications.
> 
>  In summary:
>   How do I find root cause of this issue. 
>   How to reliable delete an object.
> 
> -Volk
> 
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak: reliable object deletion

[al so]

>
>  Here is brief env:
>  Riak v 2.0.8 + Solr/ 5 node cluster/ MDC
>
>  Problem:
>   Deleted object suddenly resurrected after few days. Solr search query(
> "*:*") was executed around the time of reappearance.
>
>  Bucket property for this reappeared object
>  {
>   "props": {
> "name": "UsaHype",
> "allow_mult": false,
> "basic_quorum": false,
> "big_vclock": 50,
> "chash_keyfun": {
>   "mod": "riak_core_util",
>   "fun": "chash_std_keyfun"
> },
> "dvv_enabled": true,
> "dw": "quorum",
> "last_write_wins": false,
> "linkfun": {
>   "mod": "riak_kv_wm_link_walker",
>   "fun": "mapreduce_linkfun"
> },
> "n_val": 3,
> "notfound_ok": true,
> "old_vclock": 86400,
> "postcommit": [
>   {
> "mod": "riak_repl_leader",
> "fun": "postcommit"
>   },
>   {
> "mod": "riak_repl2_rt",
> "fun": "postcommit"
>   }
> ],
> "pr": 0,
> "precommit": [
>
> ],
> "pw": 0,
> "r": "quorum",
> "repl": true,
> "rw": "quorum",
> "small_vclock": 50,
> "w": "quorum",
> "young_vclock": 20
>   }
> }
>
>
> * delete_mode is not configured at all. Defaults to 3 sec? Do Tombstone still 
> get created when no delete_mode at all in the config? Can we query 
> metadata(*X-Riak-Deleted)* for this reappeared object.*
>
>  How would one go about finding root cause. I do know the date when the
> object was deleted from our app logs. I also seem to know when it
> reappeared. Seems like a solr search ("*:*") in code was executed just
> before this object reappeared? Look at AAE? Is there a way to find if Solr
> and Riak backends are out-of-sync (AAE?).
>
>  I understand tuning other params (allow_mult = true, ...) will have its
> own implications.
>
>  In summary:
>   How do I find root cause of this issue.
>   How to reliable delete an object.
>
> -Volk
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com