Re: Atomic Multicore Operations - E.G. Move Docs
Yes, by generations I meant versioning. The problem is that you have to have a central holder of the current generation number. ZK does this very well. It is a distributed synchronized file system for very small files. If you have a more natural place to store the current generation number, that's fine also. On Mon, Aug 20, 2012 at 2:47 PM, Nicholas Ball wrote: > > hi lance, > > how would that work? generation is essentially versioning right? > i also don't see why you need to use zk to do this as it's all on a single > machine, was hoping for a simpler solution :) > > On Sun, 19 Aug 2012 19:26:41 -0700, Lance Norskog > wrote: >> I would use generation numbers on documents, and communicate a global >> generation number in ZK. >> >> On Thu, Aug 16, 2012 at 2:22 AM, Nicholas Ball >> wrote: >>> >>> I've been close to implementing a 2PC protocol before for something > else, >>> however for this it's not needed. >>> As the move operation will be done on a single node which has both the >>> cores, this could be done differently. Just not entirely sure how to do >>> it. >>> >>> When a commit is done at the moment, the core must get locked somehow, > it >>> is at this point where we should lock the other core too if a move >>> operation is being executed. >>> >>> Nick >>> >>> On Thu, 16 Aug 2012 10:32:10 +0800, Li Li wrote: >>> > http://zookeeper.apache.org/doc/r3.3.6/recipes.html#sc_recipes_twoPhasedCommit On Thu, Aug 16, 2012 at 7:41 AM, Nicholas Ball wrote: > > Haven't managed to find a good way to do this yet. Does anyone have > any > ideas on how I could implement this feature? > Really need to move docs across from one core to another atomically. > > Many thanks, > Nicholas > > On Mon, 02 Jul 2012 04:37:12 -0600, Nicholas Ball > wrote: >> That could work, but then how do you ensure commit is called on the >>> two >> cores at the exact same time? >> >> Cheers, >> Nicholas >> >> On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog > >> wrote: >>> Index all documents to both cores, but do not call commit until > both >>> report that indexing worked. If one of the cores throws an > exception, >>> call roll back on both cores. >>> >>> On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball >>> wrote: Hey all, Trying to figure out the best way to perform atomic operation > across multiple cores on the same solr instance i.e. a multi-core > environment. An example would be to move a set of docs from one core onto > another >> core and ensure that a softcommit is done as the exact same time. If > one >> were to fail so would the other. Obviously this would probably require some customization but > wanted >>> to know what the best way to tackle this would be and where should I > be looking in the source. Many thanks for the help in advance, Nicholas a.k.a. incunix -- Lance Norskog goks...@gmail.com
Re: Atomic Multicore Operations - E.G. Move Docs
hi lance, how would that work? generation is essentially versioning right? i also don't see why you need to use zk to do this as it's all on a single machine, was hoping for a simpler solution :) On Sun, 19 Aug 2012 19:26:41 -0700, Lance Norskog wrote: > I would use generation numbers on documents, and communicate a global > generation number in ZK. > > On Thu, Aug 16, 2012 at 2:22 AM, Nicholas Ball > wrote: >> >> I've been close to implementing a 2PC protocol before for something else, >> however for this it's not needed. >> As the move operation will be done on a single node which has both the >> cores, this could be done differently. Just not entirely sure how to do >> it. >> >> When a commit is done at the moment, the core must get locked somehow, it >> is at this point where we should lock the other core too if a move >> operation is being executed. >> >> Nick >> >> On Thu, 16 Aug 2012 10:32:10 +0800, Li Li wrote: >>> >> http://zookeeper.apache.org/doc/r3.3.6/recipes.html#sc_recipes_twoPhasedCommit >>> >>> On Thu, Aug 16, 2012 at 7:41 AM, Nicholas Ball >>> wrote: Haven't managed to find a good way to do this yet. Does anyone have any ideas on how I could implement this feature? Really need to move docs across from one core to another atomically. Many thanks, Nicholas On Mon, 02 Jul 2012 04:37:12 -0600, Nicholas Ball wrote: > That could work, but then how do you ensure commit is called on the >> two > cores at the exact same time? > > Cheers, > Nicholas > > On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog > wrote: >> Index all documents to both cores, but do not call commit until both >> report that indexing worked. If one of the cores throws an exception, >> call roll back on both cores. >> >> On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball >> wrote: >>> >>> Hey all, >>> >>> Trying to figure out the best way to perform atomic operation across >>> multiple cores on the same solr instance i.e. a multi-core environment. >>> >>> An example would be to move a set of docs from one core onto another > core >>> and ensure that a softcommit is done as the exact same time. If one > were >>> to >>> fail so would the other. >>> Obviously this would probably require some customization but wanted >> to >>> know what the best way to tackle this would be and where should I be >>> looking in the source. >>> >>> Many thanks for the help in advance, >>> Nicholas a.k.a. incunix
Re: Atomic Multicore Operations - E.G. Move Docs
I would use generation numbers on documents, and communicate a global generation number in ZK. On Thu, Aug 16, 2012 at 2:22 AM, Nicholas Ball wrote: > > I've been close to implementing a 2PC protocol before for something else, > however for this it's not needed. > As the move operation will be done on a single node which has both the > cores, this could be done differently. Just not entirely sure how to do it. > > When a commit is done at the moment, the core must get locked somehow, it > is at this point where we should lock the other core too if a move > operation is being executed. > > Nick > > On Thu, 16 Aug 2012 10:32:10 +0800, Li Li wrote: >> > http://zookeeper.apache.org/doc/r3.3.6/recipes.html#sc_recipes_twoPhasedCommit >> >> On Thu, Aug 16, 2012 at 7:41 AM, Nicholas Ball >> wrote: >>> >>> Haven't managed to find a good way to do this yet. Does anyone have any >>> ideas on how I could implement this feature? >>> Really need to move docs across from one core to another atomically. >>> >>> Many thanks, >>> Nicholas >>> >>> On Mon, 02 Jul 2012 04:37:12 -0600, Nicholas Ball >>> wrote: That could work, but then how do you ensure commit is called on the > two cores at the exact same time? Cheers, Nicholas On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog wrote: > Index all documents to both cores, but do not call commit until both > report that indexing worked. If one of the cores throws an exception, > call roll back on both cores. > > On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball > wrote: >> >> Hey all, >> >> Trying to figure out the best way to perform atomic operation across >> multiple cores on the same solr instance i.e. a multi-core >>> environment. >> >> An example would be to move a set of docs from one core onto another core >> and ensure that a softcommit is done as the exact same time. If one were >> to >> fail so would the other. >> Obviously this would probably require some customization but wanted > to >> know what the best way to tackle this would be and where should I be >> looking in the source. >> >> Many thanks for the help in advance, >> Nicholas a.k.a. incunix -- Lance Norskog goks...@gmail.com
Re: Atomic Multicore Operations - E.G. Move Docs
I've been close to implementing a 2PC protocol before for something else, however for this it's not needed. As the move operation will be done on a single node which has both the cores, this could be done differently. Just not entirely sure how to do it. When a commit is done at the moment, the core must get locked somehow, it is at this point where we should lock the other core too if a move operation is being executed. Nick On Thu, 16 Aug 2012 10:32:10 +0800, Li Li wrote: > http://zookeeper.apache.org/doc/r3.3.6/recipes.html#sc_recipes_twoPhasedCommit > > On Thu, Aug 16, 2012 at 7:41 AM, Nicholas Ball > wrote: >> >> Haven't managed to find a good way to do this yet. Does anyone have any >> ideas on how I could implement this feature? >> Really need to move docs across from one core to another atomically. >> >> Many thanks, >> Nicholas >> >> On Mon, 02 Jul 2012 04:37:12 -0600, Nicholas Ball >> wrote: >>> That could work, but then how do you ensure commit is called on the two >>> cores at the exact same time? >>> >>> Cheers, >>> Nicholas >>> >>> On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog >>> wrote: Index all documents to both cores, but do not call commit until both report that indexing worked. If one of the cores throws an exception, call roll back on both cores. On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball wrote: > > Hey all, > > Trying to figure out the best way to perform atomic operation across > multiple cores on the same solr instance i.e. a multi-core >> environment. > > An example would be to move a set of docs from one core onto another >>> core > and ensure that a softcommit is done as the exact same time. If one >>> were > to > fail so would the other. > Obviously this would probably require some customization but wanted to > know what the best way to tackle this would be and where should I be > looking in the source. > > Many thanks for the help in advance, > Nicholas a.k.a. incunix
Re: Atomic Multicore Operations - E.G. Move Docs
http://zookeeper.apache.org/doc/r3.3.6/recipes.html#sc_recipes_twoPhasedCommit On Thu, Aug 16, 2012 at 7:41 AM, Nicholas Ball wrote: > > Haven't managed to find a good way to do this yet. Does anyone have any > ideas on how I could implement this feature? > Really need to move docs across from one core to another atomically. > > Many thanks, > Nicholas > > On Mon, 02 Jul 2012 04:37:12 -0600, Nicholas Ball > wrote: >> That could work, but then how do you ensure commit is called on the two >> cores at the exact same time? >> >> Cheers, >> Nicholas >> >> On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog >> wrote: >>> Index all documents to both cores, but do not call commit until both >>> report that indexing worked. If one of the cores throws an exception, >>> call roll back on both cores. >>> >>> On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball >>> wrote: Hey all, Trying to figure out the best way to perform atomic operation across multiple cores on the same solr instance i.e. a multi-core > environment. An example would be to move a set of docs from one core onto another >> core and ensure that a softcommit is done as the exact same time. If one >> were to fail so would the other. Obviously this would probably require some customization but wanted to know what the best way to tackle this would be and where should I be looking in the source. Many thanks for the help in advance, Nicholas a.k.a. incunix
Re: Atomic Multicore Operations - E.G. Move Docs
do you really need this？ distributed transaction is a difficult problem. in 2pc, every node could fail, including coordinator. something like leader election needed to make sure it works. you maybe try zookeeper. but if the transaction is not very very important like transfer money in bank, you can do like this. coordinator: 在 2012-8-16 上午7:42，"Nicholas Ball" 写道： > > Haven't managed to find a good way to do this yet. Does anyone have any > ideas on how I could implement this feature? > Really need to move docs across from one core to another atomically. > > Many thanks, > Nicholas > > On Mon, 02 Jul 2012 04:37:12 -0600, Nicholas Ball > wrote: > > That could work, but then how do you ensure commit is called on the two > > cores at the exact same time? > > > > Cheers, > > Nicholas > > > > On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog > > wrote: > >> Index all documents to both cores, but do not call commit until both > >> report that indexing worked. If one of the cores throws an exception, > >> call roll back on both cores. > >> > >> On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball > >> wrote: > >>> > >>> Hey all, > >>> > >>> Trying to figure out the best way to perform atomic operation across > >>> multiple cores on the same solr instance i.e. a multi-core > environment. > >>> > >>> An example would be to move a set of docs from one core onto another > > core > >>> and ensure that a softcommit is done as the exact same time. If one > > were > >>> to > >>> fail so would the other. > >>> Obviously this would probably require some customization but wanted to > >>> know what the best way to tackle this would be and where should I be > >>> looking in the source. > >>> > >>> Many thanks for the help in advance, > >>> Nicholas a.k.a. incunix >
Re: Atomic Multicore Operations - E.G. Move Docs
在 2012-7-2 傍晚6:37，"Nicholas Ball" 写道： > > > That could work, but then how do you ensure commit is called on the two > cores at the exact same time? that may needs something like two phrase commit in relational dB. lucene has prepareCommit, but to implement 2pc, many things need to do. > Also, any way to commit a specific update rather then all the back-logged > ones? > > Cheers, > Nicholas > > On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog > wrote: > > Index all documents to both cores, but do not call commit until both > > report that indexing worked. If one of the cores throws an exception, > > call roll back on both cores. > > > > On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball > > wrote: > >> > >> Hey all, > >> > >> Trying to figure out the best way to perform atomic operation across > >> multiple cores on the same solr instance i.e. a multi-core environment. > >> > >> An example would be to move a set of docs from one core onto another > core > >> and ensure that a softcommit is done as the exact same time. If one > were > >> to > >> fail so would the other. > >> Obviously this would probably require some customization but wanted to > >> know what the best way to tackle this would be and where should I be > >> looking in the source. > >> > >> Many thanks for the help in advance, > >> Nicholas a.k.a. incunix
Re: Atomic Multicore Operations - E.G. Move Docs
Haven't managed to find a good way to do this yet. Does anyone have any ideas on how I could implement this feature? Really need to move docs across from one core to another atomically. Many thanks, Nicholas On Mon, 02 Jul 2012 04:37:12 -0600, Nicholas Ball wrote: > That could work, but then how do you ensure commit is called on the two > cores at the exact same time? > > Cheers, > Nicholas > > On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog > wrote: >> Index all documents to both cores, but do not call commit until both >> report that indexing worked. If one of the cores throws an exception, >> call roll back on both cores. >> >> On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball >> wrote: >>> >>> Hey all, >>> >>> Trying to figure out the best way to perform atomic operation across >>> multiple cores on the same solr instance i.e. a multi-core environment. >>> >>> An example would be to move a set of docs from one core onto another > core >>> and ensure that a softcommit is done as the exact same time. If one > were >>> to >>> fail so would the other. >>> Obviously this would probably require some customization but wanted to >>> know what the best way to tackle this would be and where should I be >>> looking in the source. >>> >>> Many thanks for the help in advance, >>> Nicholas a.k.a. incunix
Re: Atomic Multicore Operations - E.G. Move Docs
That could work, but then how do you ensure commit is called on the two cores at the exact same time? Also, any way to commit a specific update rather then all the back-logged ones? Cheers, Nicholas On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog wrote: > Index all documents to both cores, but do not call commit until both > report that indexing worked. If one of the cores throws an exception, > call roll back on both cores. > > On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball > wrote: >> >> Hey all, >> >> Trying to figure out the best way to perform atomic operation across >> multiple cores on the same solr instance i.e. a multi-core environment. >> >> An example would be to move a set of docs from one core onto another core >> and ensure that a softcommit is done as the exact same time. If one were >> to >> fail so would the other. >> Obviously this would probably require some customization but wanted to >> know what the best way to tackle this would be and where should I be >> looking in the source. >> >> Many thanks for the help in advance, >> Nicholas a.k.a. incunix
Re: Atomic Multicore Operations - E.G. Move Docs
Index all documents to both cores, but do not call commit until both report that indexing worked. If one of the cores throws an exception, call roll back on both cores. On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball wrote: > > Hey all, > > Trying to figure out the best way to perform atomic operation across > multiple cores on the same solr instance i.e. a multi-core environment. > > An example would be to move a set of docs from one core onto another core > and ensure that a softcommit is done as the exact same time. If one were to > fail so would the other. > Obviously this would probably require some customization but wanted to > know what the best way to tackle this would be and where should I be > looking in the source. > > Many thanks for the help in advance, > Nicholas a.k.a. incunix -- Lance Norskog goks...@gmail.com
