[jira] Updated: (SOLR-1724) Real Basic Core Management with Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Rutherglen updated SOLR-1724: --- Attachment: SOLR-1724.patch Fixed the unit tests that were failing due to the switch over to using CoreContainer's initZooKeeper method. ZkNodeCoresManager is instantiated in CoreContainer. There's a beginning of a UI in zkcores.jsp I think we still need a core move test. I'm thinking of adding backing up a core as an action that may be performed in a new cores version file. Real Basic Core Management with Zookeeper - Key: SOLR-1724 URL: https://issues.apache.org/jira/browse/SOLR-1724 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 1.4 Reporter: Jason Rutherglen Fix For: 1.5 Attachments: commons-lang-2.4.jar, gson-1.4.jar, hadoop-0.20.2-dev-core.jar, hadoop-0.20.2-dev-test.jar, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch Though we're implementing cloud, I need something real soon I can play with and deploy. So this'll be a patch that only deploys new cores, and that's about it. The arch is real simple: On Zookeeper there'll be a directory that contains files that represent the state of the cores of a given set of servers which will look like the following: /production/cores-1.txt /production/cores-2.txt /production/core-host-1-actual.txt (ephemeral node per host) Where each core-N.txt file contains: hostname,corename,instanceDir,coredownloadpath coredownloadpath is a URL such as file://, http://, hftp://, hdfs://, ftp://, etc and core-host-actual.txt contains: hostname,corename,instanceDir,size Everytime a new core-N.txt file is added, the listening host finds it's entry in the list and begins the process of trying to match the entries. Upon completion, it updates it's /core-host-1-actual.txt file to it's completed state or logs an error. When all host actual files are written (without errors), then a new core-1-actual.txt file is written which can be picked up by another process that can create a new core proxy. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1724) Real Basic Core Management with Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Rutherglen updated SOLR-1724: --- Attachment: SOLR-1724.patch Zipping from a Lucene directory works and has a test case A ReplicationHandler is added by default under a unique name, if one exists already, we still create our own, for the express purpose of locking an index commit point, zipping it, then uploading it to, for example, HDFS. This part will likely be written next. Real Basic Core Management with Zookeeper - Key: SOLR-1724 URL: https://issues.apache.org/jira/browse/SOLR-1724 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 1.4 Reporter: Jason Rutherglen Fix For: 1.5 Attachments: commons-lang-2.4.jar, gson-1.4.jar, hadoop-0.20.2-dev-core.jar, hadoop-0.20.2-dev-test.jar, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch Though we're implementing cloud, I need something real soon I can play with and deploy. So this'll be a patch that only deploys new cores, and that's about it. The arch is real simple: On Zookeeper there'll be a directory that contains files that represent the state of the cores of a given set of servers which will look like the following: /production/cores-1.txt /production/cores-2.txt /production/core-host-1-actual.txt (ephemeral node per host) Where each core-N.txt file contains: hostname,corename,instanceDir,coredownloadpath coredownloadpath is a URL such as file://, http://, hftp://, hdfs://, ftp://, etc and core-host-actual.txt contains: hostname,corename,instanceDir,size Everytime a new core-N.txt file is added, the listening host finds it's entry in the list and begins the process of trying to match the entries. Upon completion, it updates it's /core-host-1-actual.txt file to it's completed state or logs an error. When all host actual files are written (without errors), then a new core-1-actual.txt file is written which can be picked up by another process that can create a new core proxy. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1724) Real Basic Core Management with Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Rutherglen updated SOLR-1724: --- Attachment: SOLR-1724.patch Backing a core up works, at least according to the test case... I will probably begin to test this patch in a staging environment next, where Zookeeper is run in it's own process and a real HDFS cluster is used. Real Basic Core Management with Zookeeper - Key: SOLR-1724 URL: https://issues.apache.org/jira/browse/SOLR-1724 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 1.4 Reporter: Jason Rutherglen Fix For: 1.5 Attachments: commons-lang-2.4.jar, gson-1.4.jar, hadoop-0.20.2-dev-core.jar, hadoop-0.20.2-dev-test.jar, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch Though we're implementing cloud, I need something real soon I can play with and deploy. So this'll be a patch that only deploys new cores, and that's about it. The arch is real simple: On Zookeeper there'll be a directory that contains files that represent the state of the cores of a given set of servers which will look like the following: /production/cores-1.txt /production/cores-2.txt /production/core-host-1-actual.txt (ephemeral node per host) Where each core-N.txt file contains: hostname,corename,instanceDir,coredownloadpath coredownloadpath is a URL such as file://, http://, hftp://, hdfs://, ftp://, etc and core-host-actual.txt contains: hostname,corename,instanceDir,size Everytime a new core-N.txt file is added, the listening host finds it's entry in the list and begins the process of trying to match the entries. Upon completion, it updates it's /core-host-1-actual.txt file to it's completed state or logs an error. When all host actual files are written (without errors), then a new core-1-actual.txt file is written which can be picked up by another process that can create a new core proxy. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1724) Real Basic Core Management with Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Rutherglen updated SOLR-1724: --- Attachment: SOLR-1724.patch I added a test case that simulates attempting to install a bad core. Still need to get the backup a Solr core to HDFS working. Real Basic Core Management with Zookeeper - Key: SOLR-1724 URL: https://issues.apache.org/jira/browse/SOLR-1724 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 1.4 Reporter: Jason Rutherglen Fix For: 1.5 Attachments: commons-lang-2.4.jar, gson-1.4.jar, hadoop-0.20.2-dev-core.jar, hadoop-0.20.2-dev-test.jar, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch Though we're implementing cloud, I need something real soon I can play with and deploy. So this'll be a patch that only deploys new cores, and that's about it. The arch is real simple: On Zookeeper there'll be a directory that contains files that represent the state of the cores of a given set of servers which will look like the following: /production/cores-1.txt /production/cores-2.txt /production/core-host-1-actual.txt (ephemeral node per host) Where each core-N.txt file contains: hostname,corename,instanceDir,coredownloadpath coredownloadpath is a URL such as file://, http://, hftp://, hdfs://, ftp://, etc and core-host-actual.txt contains: hostname,corename,instanceDir,size Everytime a new core-N.txt file is added, the listening host finds it's entry in the list and begins the process of trying to match the entries. Upon completion, it updates it's /core-host-1-actual.txt file to it's completed state or logs an error. When all host actual files are written (without errors), then a new core-1-actual.txt file is written which can be picked up by another process that can create a new core proxy. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1724) Real Basic Core Management with Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Rutherglen updated SOLR-1724: --- Attachment: SOLR-1724.patch Removing cores seems to work well, on to modified cores... I checkpointing progress in case things break, I can easily roll back. Real Basic Core Management with Zookeeper - Key: SOLR-1724 URL: https://issues.apache.org/jira/browse/SOLR-1724 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 1.4 Reporter: Jason Rutherglen Fix For: 1.5 Attachments: commons-lang-2.4.jar, gson-1.4.jar, hadoop-0.20.2-dev-core.jar, hadoop-0.20.2-dev-test.jar, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch Though we're implementing cloud, I need something real soon I can play with and deploy. So this'll be a patch that only deploys new cores, and that's about it. The arch is real simple: On Zookeeper there'll be a directory that contains files that represent the state of the cores of a given set of servers which will look like the following: /production/cores-1.txt /production/cores-2.txt /production/core-host-1-actual.txt (ephemeral node per host) Where each core-N.txt file contains: hostname,corename,instanceDir,coredownloadpath coredownloadpath is a URL such as file://, http://, hftp://, hdfs://, ftp://, etc and core-host-actual.txt contains: hostname,corename,instanceDir,size Everytime a new core-N.txt file is added, the listening host finds it's entry in the list and begins the process of trying to match the entries. Upon completion, it updates it's /core-host-1-actual.txt file to it's completed state or logs an error. When all host actual files are written (without errors), then a new core-1-actual.txt file is written which can be picked up by another process that can create a new core proxy. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1724) Real Basic Core Management with Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Rutherglen updated SOLR-1724: --- Attachment: SOLR-1724.patch * No-commit * NodeCoresManagerTest.testInstallCores works * There's HDFS test cases using MiniDFSCluster Real Basic Core Management with Zookeeper - Key: SOLR-1724 URL: https://issues.apache.org/jira/browse/SOLR-1724 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 1.4 Reporter: Jason Rutherglen Fix For: 1.5 Attachments: commons-lang-2.4.jar, gson-1.4.jar, hadoop-0.20.2-dev-core.jar, hadoop-0.20.2-dev-test.jar, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch Though we're implementing cloud, I need something real soon I can play with and deploy. So this'll be a patch that only deploys new cores, and that's about it. The arch is real simple: On Zookeeper there'll be a directory that contains files that represent the state of the cores of a given set of servers which will look like the following: /production/cores-1.txt /production/cores-2.txt /production/core-host-1-actual.txt (ephemeral node per host) Where each core-N.txt file contains: hostname,corename,instanceDir,coredownloadpath coredownloadpath is a URL such as file://, http://, hftp://, hdfs://, ftp://, etc and core-host-actual.txt contains: hostname,corename,instanceDir,size Everytime a new core-N.txt file is added, the listening host finds it's entry in the list and begins the process of trying to match the entries. Upon completion, it updates it's /core-host-1-actual.txt file to it's completed state or logs an error. When all host actual files are written (without errors), then a new core-1-actual.txt file is written which can be picked up by another process that can create a new core proxy. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1724) Real Basic Core Management with Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Rutherglen updated SOLR-1724: --- Attachment: SOLR-1724.patch Updated to HEAD Real Basic Core Management with Zookeeper - Key: SOLR-1724 URL: https://issues.apache.org/jira/browse/SOLR-1724 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 1.4 Reporter: Jason Rutherglen Fix For: 1.5 Attachments: commons-lang-2.4.jar, gson-1.4.jar, hadoop-0.20.2-dev-core.jar, hadoop-0.20.2-dev-test.jar, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch Though we're implementing cloud, I need something real soon I can play with and deploy. So this'll be a patch that only deploys new cores, and that's about it. The arch is real simple: On Zookeeper there'll be a directory that contains files that represent the state of the cores of a given set of servers which will look like the following: /production/cores-1.txt /production/cores-2.txt /production/core-host-1-actual.txt (ephemeral node per host) Where each core-N.txt file contains: hostname,corename,instanceDir,coredownloadpath coredownloadpath is a URL such as file://, http://, hftp://, hdfs://, ftp://, etc and core-host-actual.txt contains: hostname,corename,instanceDir,size Everytime a new core-N.txt file is added, the listening host finds it's entry in the list and begins the process of trying to match the entries. Upon completion, it updates it's /core-host-1-actual.txt file to it's completed state or logs an error. When all host actual files are written (without errors), then a new core-1-actual.txt file is written which can be picked up by another process that can create a new core proxy. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1724) Real Basic Core Management with Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Rutherglen updated SOLR-1724: --- Attachment: SOLR-1724.patch Added a way to hold a given number of host or cores files around in ZK, after which, the oldest are deleted. Real Basic Core Management with Zookeeper - Key: SOLR-1724 URL: https://issues.apache.org/jira/browse/SOLR-1724 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 1.4 Reporter: Jason Rutherglen Fix For: 1.5 Attachments: commons-lang-2.4.jar, gson-1.4.jar, hadoop-0.20.2-dev-core.jar, hadoop-0.20.2-dev-test.jar, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch Though we're implementing cloud, I need something real soon I can play with and deploy. So this'll be a patch that only deploys new cores, and that's about it. The arch is real simple: On Zookeeper there'll be a directory that contains files that represent the state of the cores of a given set of servers which will look like the following: /production/cores-1.txt /production/cores-2.txt /production/core-host-1-actual.txt (ephemeral node per host) Where each core-N.txt file contains: hostname,corename,instanceDir,coredownloadpath coredownloadpath is a URL such as file://, http://, hftp://, hdfs://, ftp://, etc and core-host-actual.txt contains: hostname,corename,instanceDir,size Everytime a new core-N.txt file is added, the listening host finds it's entry in the list and begins the process of trying to match the entries. Upon completion, it updates it's /core-host-1-actual.txt file to it's completed state or logs an error. When all host actual files are written (without errors), then a new core-1-actual.txt file is written which can be picked up by another process that can create a new core proxy. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1724) Real Basic Core Management with Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Rutherglen updated SOLR-1724: --- Attachment: SOLR-1724.patch No-commit NodeCoresManager[Test] needs more work A CoreController matchHosts unit test was added to CoreControllerTest Real Basic Core Management with Zookeeper - Key: SOLR-1724 URL: https://issues.apache.org/jira/browse/SOLR-1724 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 1.4 Reporter: Jason Rutherglen Fix For: 1.5 Attachments: commons-lang-2.4.jar, gson-1.4.jar, hadoop-0.20.2-dev-core.jar, hadoop-0.20.2-dev-test.jar, SOLR-1724.patch, SOLR-1724.patch, SOLR-1724.patch Though we're implementing cloud, I need something real soon I can play with and deploy. So this'll be a patch that only deploys new cores, and that's about it. The arch is real simple: On Zookeeper there'll be a directory that contains files that represent the state of the cores of a given set of servers which will look like the following: /production/cores-1.txt /production/cores-2.txt /production/core-host-1-actual.txt (ephemeral node per host) Where each core-N.txt file contains: hostname,corename,instanceDir,coredownloadpath coredownloadpath is a URL such as file://, http://, hftp://, hdfs://, ftp://, etc and core-host-actual.txt contains: hostname,corename,instanceDir,size Everytime a new core-N.txt file is added, the listening host finds it's entry in the list and begins the process of trying to match the entries. Upon completion, it updates it's /core-host-1-actual.txt file to it's completed state or logs an error. When all host actual files are written (without errors), then a new core-1-actual.txt file is written which can be picked up by another process that can create a new core proxy. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1724) Real Basic Core Management with Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Rutherglen updated SOLR-1724: --- Attachment: gson-1.4.jar hadoop-0.20.2-dev-test.jar hadoop-0.20.2-dev-core.jar Hadoop and Gson dependencies Real Basic Core Management with Zookeeper - Key: SOLR-1724 URL: https://issues.apache.org/jira/browse/SOLR-1724 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 1.4 Reporter: Jason Rutherglen Fix For: 1.5 Attachments: commons-lang-2.4.jar, gson-1.4.jar, hadoop-0.20.2-dev-core.jar, hadoop-0.20.2-dev-test.jar, SOLR-1724.patch Though we're implementing cloud, I need something real soon I can play with and deploy. So this'll be a patch that only deploys new cores, and that's about it. The arch is real simple: On Zookeeper there'll be a directory that contains files that represent the state of the cores of a given set of servers which will look like the following: /production/cores-1.txt /production/cores-2.txt /production/core-host-1-actual.txt (ephemeral node per host) Where each core-N.txt file contains: hostname,corename,instanceDir,coredownloadpath coredownloadpath is a URL such as file://, http://, hftp://, hdfs://, ftp://, etc and core-host-actual.txt contains: hostname,corename,instanceDir,size Everytime a new core-N.txt file is added, the listening host finds it's entry in the list and begins the process of trying to match the entries. Upon completion, it updates it's /core-host-1-actual.txt file to it's completed state or logs an error. When all host actual files are written (without errors), then a new core-1-actual.txt file is written which can be picked up by another process that can create a new core proxy. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1724) Real Basic Core Management with Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Rutherglen updated SOLR-1724: --- Attachment: SOLR-1724.patch Here's the first cut... I agree, I'm not really into ephemeral ZK nodes for Solr hosts/nodes. The reason is contact with ZK is highly superficial and can be intermittent. I'm mostly concerned with insuring the core operations succeed on a given server. If a server goes down, there needs to be more than ZK to prove it, and if it goes down completely, I'll simply reallocate it's cores to another server using the core management mechanism provided in this patch. The issue is still being worked on, specifically the Solr server portion that downloads the cores from some location, or performs operations. The file format will move to json. Real Basic Core Management with Zookeeper - Key: SOLR-1724 URL: https://issues.apache.org/jira/browse/SOLR-1724 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 1.4 Reporter: Jason Rutherglen Fix For: 1.5 Attachments: SOLR-1724.patch Though we're implementing cloud, I need something real soon I can play with and deploy. So this'll be a patch that only deploys new cores, and that's about it. The arch is real simple: On Zookeeper there'll be a directory that contains files that represent the state of the cores of a given set of servers which will look like the following: /production/cores-1.txt /production/cores-2.txt /production/core-host-1-actual.txt (ephemeral node per host) Where each core-N.txt file contains: hostname,corename,instanceDir,coredownloadpath coredownloadpath is a URL such as file://, http://, hftp://, hdfs://, ftp://, etc and core-host-actual.txt contains: hostname,corename,instanceDir,size Everytime a new core-N.txt file is added, the listening host finds it's entry in the list and begins the process of trying to match the entries. Upon completion, it updates it's /core-host-1-actual.txt file to it's completed state or logs an error. When all host actual files are written (without errors), then a new core-1-actual.txt file is written which can be picked up by another process that can create a new core proxy. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
