Re: Atomic Multicore Operations - E.G. Move Docs
hi lance, how would that work? generation is essentially versioning right? i also don't see why you need to use zk to do this as it's all on a single machine, was hoping for a simpler solution :) On Sun, 19 Aug 2012 19:26:41 -0700, Lance Norskog goks...@gmail.com wrote: I would use generation numbers on documents, and communicate a global generation number in ZK. On Thu, Aug 16, 2012 at 2:22 AM, Nicholas Ball nicholas.b...@nodelay.com wrote: I've been close to implementing a 2PC protocol before for something else, however for this it's not needed. As the move operation will be done on a single node which has both the cores, this could be done differently. Just not entirely sure how to do it. When a commit is done at the moment, the core must get locked somehow, it is at this point where we should lock the other core too if a move operation is being executed. Nick On Thu, 16 Aug 2012 10:32:10 +0800, Li Li fancye...@gmail.com wrote: http://zookeeper.apache.org/doc/r3.3.6/recipes.html#sc_recipes_twoPhasedCommit On Thu, Aug 16, 2012 at 7:41 AM, Nicholas Ball nicholas.b...@nodelay.com wrote: Haven't managed to find a good way to do this yet. Does anyone have any ideas on how I could implement this feature? Really need to move docs across from one core to another atomically. Many thanks, Nicholas On Mon, 02 Jul 2012 04:37:12 -0600, Nicholas Ball nicholas.b...@nodelay.com wrote: That could work, but then how do you ensure commit is called on the two cores at the exact same time? Cheers, Nicholas On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog goks...@gmail.com wrote: Index all documents to both cores, but do not call commit until both report that indexing worked. If one of the cores throws an exception, call roll back on both cores. On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball nicholas.b...@nodelay.com wrote: Hey all, Trying to figure out the best way to perform atomic operation across multiple cores on the same solr instance i.e. a multi-core environment. An example would be to move a set of docs from one core onto another core and ensure that a softcommit is done as the exact same time. If one were to fail so would the other. Obviously this would probably require some customization but wanted to know what the best way to tackle this would be and where should I be looking in the source. Many thanks for the help in advance, Nicholas a.k.a. incunix
Re: Atomic Multicore Operations - E.G. Move Docs
Yes, by generations I meant versioning. The problem is that you have to have a central holder of the current generation number. ZK does this very well. It is a distributed synchronized file system for very small files. If you have a more natural place to store the current generation number, that's fine also. On Mon, Aug 20, 2012 at 2:47 PM, Nicholas Ball nicholas.b...@nodelay.com wrote: hi lance, how would that work? generation is essentially versioning right? i also don't see why you need to use zk to do this as it's all on a single machine, was hoping for a simpler solution :) On Sun, 19 Aug 2012 19:26:41 -0700, Lance Norskog goks...@gmail.com wrote: I would use generation numbers on documents, and communicate a global generation number in ZK. On Thu, Aug 16, 2012 at 2:22 AM, Nicholas Ball nicholas.b...@nodelay.com wrote: I've been close to implementing a 2PC protocol before for something else, however for this it's not needed. As the move operation will be done on a single node which has both the cores, this could be done differently. Just not entirely sure how to do it. When a commit is done at the moment, the core must get locked somehow, it is at this point where we should lock the other core too if a move operation is being executed. Nick On Thu, 16 Aug 2012 10:32:10 +0800, Li Li fancye...@gmail.com wrote: http://zookeeper.apache.org/doc/r3.3.6/recipes.html#sc_recipes_twoPhasedCommit On Thu, Aug 16, 2012 at 7:41 AM, Nicholas Ball nicholas.b...@nodelay.com wrote: Haven't managed to find a good way to do this yet. Does anyone have any ideas on how I could implement this feature? Really need to move docs across from one core to another atomically. Many thanks, Nicholas On Mon, 02 Jul 2012 04:37:12 -0600, Nicholas Ball nicholas.b...@nodelay.com wrote: That could work, but then how do you ensure commit is called on the two cores at the exact same time? Cheers, Nicholas On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog goks...@gmail.com wrote: Index all documents to both cores, but do not call commit until both report that indexing worked. If one of the cores throws an exception, call roll back on both cores. On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball nicholas.b...@nodelay.com wrote: Hey all, Trying to figure out the best way to perform atomic operation across multiple cores on the same solr instance i.e. a multi-core environment. An example would be to move a set of docs from one core onto another core and ensure that a softcommit is done as the exact same time. If one were to fail so would the other. Obviously this would probably require some customization but wanted to know what the best way to tackle this would be and where should I be looking in the source. Many thanks for the help in advance, Nicholas a.k.a. incunix -- Lance Norskog goks...@gmail.com
Re: Atomic Multicore Operations - E.G. Move Docs
I would use generation numbers on documents, and communicate a global generation number in ZK. On Thu, Aug 16, 2012 at 2:22 AM, Nicholas Ball nicholas.b...@nodelay.com wrote: I've been close to implementing a 2PC protocol before for something else, however for this it's not needed. As the move operation will be done on a single node which has both the cores, this could be done differently. Just not entirely sure how to do it. When a commit is done at the moment, the core must get locked somehow, it is at this point where we should lock the other core too if a move operation is being executed. Nick On Thu, 16 Aug 2012 10:32:10 +0800, Li Li fancye...@gmail.com wrote: http://zookeeper.apache.org/doc/r3.3.6/recipes.html#sc_recipes_twoPhasedCommit On Thu, Aug 16, 2012 at 7:41 AM, Nicholas Ball nicholas.b...@nodelay.com wrote: Haven't managed to find a good way to do this yet. Does anyone have any ideas on how I could implement this feature? Really need to move docs across from one core to another atomically. Many thanks, Nicholas On Mon, 02 Jul 2012 04:37:12 -0600, Nicholas Ball nicholas.b...@nodelay.com wrote: That could work, but then how do you ensure commit is called on the two cores at the exact same time? Cheers, Nicholas On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog goks...@gmail.com wrote: Index all documents to both cores, but do not call commit until both report that indexing worked. If one of the cores throws an exception, call roll back on both cores. On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball nicholas.b...@nodelay.com wrote: Hey all, Trying to figure out the best way to perform atomic operation across multiple cores on the same solr instance i.e. a multi-core environment. An example would be to move a set of docs from one core onto another core and ensure that a softcommit is done as the exact same time. If one were to fail so would the other. Obviously this would probably require some customization but wanted to know what the best way to tackle this would be and where should I be looking in the source. Many thanks for the help in advance, Nicholas a.k.a. incunix -- Lance Norskog goks...@gmail.com
Re: Atomic Multicore Operations - E.G. Move Docs
I've been close to implementing a 2PC protocol before for something else, however for this it's not needed. As the move operation will be done on a single node which has both the cores, this could be done differently. Just not entirely sure how to do it. When a commit is done at the moment, the core must get locked somehow, it is at this point where we should lock the other core too if a move operation is being executed. Nick On Thu, 16 Aug 2012 10:32:10 +0800, Li Li fancye...@gmail.com wrote: http://zookeeper.apache.org/doc/r3.3.6/recipes.html#sc_recipes_twoPhasedCommit On Thu, Aug 16, 2012 at 7:41 AM, Nicholas Ball nicholas.b...@nodelay.com wrote: Haven't managed to find a good way to do this yet. Does anyone have any ideas on how I could implement this feature? Really need to move docs across from one core to another atomically. Many thanks, Nicholas On Mon, 02 Jul 2012 04:37:12 -0600, Nicholas Ball nicholas.b...@nodelay.com wrote: That could work, but then how do you ensure commit is called on the two cores at the exact same time? Cheers, Nicholas On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog goks...@gmail.com wrote: Index all documents to both cores, but do not call commit until both report that indexing worked. If one of the cores throws an exception, call roll back on both cores. On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball nicholas.b...@nodelay.com wrote: Hey all, Trying to figure out the best way to perform atomic operation across multiple cores on the same solr instance i.e. a multi-core environment. An example would be to move a set of docs from one core onto another core and ensure that a softcommit is done as the exact same time. If one were to fail so would the other. Obviously this would probably require some customization but wanted to know what the best way to tackle this would be and where should I be looking in the source. Many thanks for the help in advance, Nicholas a.k.a. incunix
Re: Atomic Multicore Operations - E.G. Move Docs
Haven't managed to find a good way to do this yet. Does anyone have any ideas on how I could implement this feature? Really need to move docs across from one core to another atomically. Many thanks, Nicholas On Mon, 02 Jul 2012 04:37:12 -0600, Nicholas Ball nicholas.b...@nodelay.com wrote: That could work, but then how do you ensure commit is called on the two cores at the exact same time? Cheers, Nicholas On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog goks...@gmail.com wrote: Index all documents to both cores, but do not call commit until both report that indexing worked. If one of the cores throws an exception, call roll back on both cores. On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball nicholas.b...@nodelay.com wrote: Hey all, Trying to figure out the best way to perform atomic operation across multiple cores on the same solr instance i.e. a multi-core environment. An example would be to move a set of docs from one core onto another core and ensure that a softcommit is done as the exact same time. If one were to fail so would the other. Obviously this would probably require some customization but wanted to know what the best way to tackle this would be and where should I be looking in the source. Many thanks for the help in advance, Nicholas a.k.a. incunix
Re: Atomic Multicore Operations - E.G. Move Docs
在 2012-7-2 傍晚6:37，Nicholas Ball nicholas.b...@nodelay.com写道： That could work, but then how do you ensure commit is called on the two cores at the exact same time? that may needs something like two phrase commit in relational dB. lucene has prepareCommit, but to implement 2pc, many things need to do. Also, any way to commit a specific update rather then all the back-logged ones? Cheers, Nicholas On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog goks...@gmail.com wrote: Index all documents to both cores, but do not call commit until both report that indexing worked. If one of the cores throws an exception, call roll back on both cores. On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball nicholas.b...@nodelay.com wrote: Hey all, Trying to figure out the best way to perform atomic operation across multiple cores on the same solr instance i.e. a multi-core environment. An example would be to move a set of docs from one core onto another core and ensure that a softcommit is done as the exact same time. If one were to fail so would the other. Obviously this would probably require some customization but wanted to know what the best way to tackle this would be and where should I be looking in the source. Many thanks for the help in advance, Nicholas a.k.a. incunix
Re: Atomic Multicore Operations - E.G. Move Docs
do you really need this？ distributed transaction is a difficult problem. in 2pc, every node could fail, including coordinator. something like leader election needed to make sure it works. you maybe try zookeeper. but if the transaction is not very very important like transfer money in bank, you can do like this. coordinator: 在 2012-8-16 上午7:42，Nicholas Ball nicholas.b...@nodelay.com写道： Haven't managed to find a good way to do this yet. Does anyone have any ideas on how I could implement this feature? Really need to move docs across from one core to another atomically. Many thanks, Nicholas On Mon, 02 Jul 2012 04:37:12 -0600, Nicholas Ball nicholas.b...@nodelay.com wrote: That could work, but then how do you ensure commit is called on the two cores at the exact same time? Cheers, Nicholas On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog goks...@gmail.com wrote: Index all documents to both cores, but do not call commit until both report that indexing worked. If one of the cores throws an exception, call roll back on both cores. On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball nicholas.b...@nodelay.com wrote: Hey all, Trying to figure out the best way to perform atomic operation across multiple cores on the same solr instance i.e. a multi-core environment. An example would be to move a set of docs from one core onto another core and ensure that a softcommit is done as the exact same time. If one were to fail so would the other. Obviously this would probably require some customization but wanted to know what the best way to tackle this would be and where should I be looking in the source. Many thanks for the help in advance, Nicholas a.k.a. incunix
Re: Atomic Multicore Operations - E.G. Move Docs
http://zookeeper.apache.org/doc/r3.3.6/recipes.html#sc_recipes_twoPhasedCommit On Thu, Aug 16, 2012 at 7:41 AM, Nicholas Ball nicholas.b...@nodelay.com wrote: Haven't managed to find a good way to do this yet. Does anyone have any ideas on how I could implement this feature? Really need to move docs across from one core to another atomically. Many thanks, Nicholas On Mon, 02 Jul 2012 04:37:12 -0600, Nicholas Ball nicholas.b...@nodelay.com wrote: That could work, but then how do you ensure commit is called on the two cores at the exact same time? Cheers, Nicholas On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog goks...@gmail.com wrote: Index all documents to both cores, but do not call commit until both report that indexing worked. If one of the cores throws an exception, call roll back on both cores. On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball nicholas.b...@nodelay.com wrote: Hey all, Trying to figure out the best way to perform atomic operation across multiple cores on the same solr instance i.e. a multi-core environment. An example would be to move a set of docs from one core onto another core and ensure that a softcommit is done as the exact same time. If one were to fail so would the other. Obviously this would probably require some customization but wanted to know what the best way to tackle this would be and where should I be looking in the source. Many thanks for the help in advance, Nicholas a.k.a. incunix
Re: Atomic Multicore Operations - E.G. Move Docs
That could work, but then how do you ensure commit is called on the two cores at the exact same time? Also, any way to commit a specific update rather then all the back-logged ones? Cheers, Nicholas On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog goks...@gmail.com wrote: Index all documents to both cores, but do not call commit until both report that indexing worked. If one of the cores throws an exception, call roll back on both cores. On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball nicholas.b...@nodelay.com wrote: Hey all, Trying to figure out the best way to perform atomic operation across multiple cores on the same solr instance i.e. a multi-core environment. An example would be to move a set of docs from one core onto another core and ensure that a softcommit is done as the exact same time. If one were to fail so would the other. Obviously this would probably require some customization but wanted to know what the best way to tackle this would be and where should I be looking in the source. Many thanks for the help in advance, Nicholas a.k.a. incunix
Atomic Multicore Operations - E.G. Move Docs
Hey all, Trying to figure out the best way to perform atomic operation across multiple cores on the same solr instance i.e. a multi-core environment. An example would be to move a set of docs from one core onto another core and ensure that a softcommit is done as the exact same time. If one were to fail so would the other. Obviously this would probably require some customization but wanted to know what the best way to tackle this would be and where should I be looking in the source. Many thanks for the help in advance, Nicholas a.k.a. incunix
Re: Atomic Multicore Operations - E.G. Move Docs
Index all documents to both cores, but do not call commit until both report that indexing worked. If one of the cores throws an exception, call roll back on both cores. On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball nicholas.b...@nodelay.com wrote: Hey all, Trying to figure out the best way to perform atomic operation across multiple cores on the same solr instance i.e. a multi-core environment. An example would be to move a set of docs from one core onto another core and ensure that a softcommit is done as the exact same time. If one were to fail so would the other. Obviously this would probably require some customization but wanted to know what the best way to tackle this would be and where should I be looking in the source. Many thanks for the help in advance, Nicholas a.k.a. incunix -- Lance Norskog goks...@gmail.com
