Re: Persistence problem with swapped cores after Solr restart -- 4.9.1
On 7/14/2015 10:06 AM, Erick Erickson wrote: Were any errors reported in the logs? If not, this is certainly worth a JIRA. If the persistence bits are swallowing file access perms that's A Bad Thing IMO. I will find out, and file an issue if necessary. It may take me a couple of days. Thanks, Shawn
Re: Persistence problem with swapped cores after Solr restart -- 4.9.1
Shawn: Were any errors reported in the logs? If not, this is certainly worth a JIRA. If the persistence bits are swallowing file access perms that's A Bad Thing IMO. Erick On Tue, Jul 14, 2015 at 12:46 AM, Upayavira wrote: > Problems between keyboard and chair are the best kind. They are the > easiest to resolve. If I were you, I'd be feeling *glad* it wasn't a > bug. > > Upayavira > > On Tue, Jul 14, 2015, at 07:31 AM, Shawn Heisey wrote: >> On 7/13/2015 10:02 PM, Erick Erickson wrote: >> > Uggghh. Not persistence again >> > >> > I'll stay tuned.. >> >> While gathering every possible detail I could during the index rebuild, >> I came across what I believe is the cause of my problems. What I found >> is no surprise: PEBCAK. >> >> All of the core.properties files were owned by root, but I run Solr with >> a "solr" user for security purposes -- core swaps were unable to update >> the files. I know how this mistake happened, but I'm amazed that it >> hasn't been an obvious problem before now. I first set up these >> machines (handling an index for a new customer) over a year ago. >> >> I fixed the permissions, the rebuild finished, and I restarted Solr on >> both machines. Everything persisted beautifully. >> >> My dev server (which sees reboots and restarts *far* more frequently >> than the production machines) has not been having this problem, and it >> turns out that all the file permissions were correct on that machine. >> >> I am very glad when a problem like this turns out to not be a bug, but >> it IS embarrassing. >> >> If anyone needs any proof of Solr's stability, I can show you servers >> that run a Solr JVM without interruption for weeks or months at a time. >> They would probably go longer, but customers keep wanting changes, plus >> I like to make sure the operating system stays current with security >> patches. >> >> Thanks, >> Shawn >>
Re: Persistence problem with swapped cores after Solr restart -- 4.9.1
Problems between keyboard and chair are the best kind. They are the easiest to resolve. If I were you, I'd be feeling *glad* it wasn't a bug. Upayavira On Tue, Jul 14, 2015, at 07:31 AM, Shawn Heisey wrote: > On 7/13/2015 10:02 PM, Erick Erickson wrote: > > Uggghh. Not persistence again > > > > I'll stay tuned.. > > While gathering every possible detail I could during the index rebuild, > I came across what I believe is the cause of my problems. What I found > is no surprise: PEBCAK. > > All of the core.properties files were owned by root, but I run Solr with > a "solr" user for security purposes -- core swaps were unable to update > the files. I know how this mistake happened, but I'm amazed that it > hasn't been an obvious problem before now. I first set up these > machines (handling an index for a new customer) over a year ago. > > I fixed the permissions, the rebuild finished, and I restarted Solr on > both machines. Everything persisted beautifully. > > My dev server (which sees reboots and restarts *far* more frequently > than the production machines) has not been having this problem, and it > turns out that all the file permissions were correct on that machine. > > I am very glad when a problem like this turns out to not be a bug, but > it IS embarrassing. > > If anyone needs any proof of Solr's stability, I can show you servers > that run a Solr JVM without interruption for weeks or months at a time. > They would probably go longer, but customers keep wanting changes, plus > I like to make sure the operating system stays current with security > patches. > > Thanks, > Shawn >
Re: Persistence problem with swapped cores after Solr restart -- 4.9.1
On 7/13/2015 10:02 PM, Erick Erickson wrote: > Uggghh. Not persistence again > > I'll stay tuned.. While gathering every possible detail I could during the index rebuild, I came across what I believe is the cause of my problems. What I found is no surprise: PEBCAK. All of the core.properties files were owned by root, but I run Solr with a "solr" user for security purposes -- core swaps were unable to update the files. I know how this mistake happened, but I'm amazed that it hasn't been an obvious problem before now. I first set up these machines (handling an index for a new customer) over a year ago. I fixed the permissions, the rebuild finished, and I restarted Solr on both machines. Everything persisted beautifully. My dev server (which sees reboots and restarts *far* more frequently than the production machines) has not been having this problem, and it turns out that all the file permissions were correct on that machine. I am very glad when a problem like this turns out to not be a bug, but it IS embarrassing. If anyone needs any proof of Solr's stability, I can show you servers that run a Solr JVM without interruption for weeks or months at a time. They would probably go longer, but customers keep wanting changes, plus I like to make sure the operating system stays current with security patches. Thanks, Shawn
Re: Persistence problem with swapped cores after Solr restart -- 4.9.1
Uggghh. Not persistence again I'll stay tuned.. Erick On Mon, Jul 13, 2015 at 2:44 PM, Shawn Heisey wrote: > On Solr 4.9.1 with core discovery, I seem to be having trouble with core > swaps not persisting through a full Solr restart. > > I apologize for the fact that this message is lean on details ... I've > seen the problem twice now, but I don't have any concrete before/after > information about what's in each core.properties file. I am attempting > to set up the scenario again and gather that information. > > The entire directory structure is set up as a git repo, so I will be > able to tell if any files (like core.properties) are modified for the > rebuild/swap that I have started. The repo shows no changes at the > moment, but I have done several of these rebuild/swap operations, so > even if core.properties is being correctly updated, it might just have > landed back on the original configuration. > > I have another copy of my index using Solr 4.7.2 with the old solr.xml > format that seems to have no problems with core swapping and > persistence. That works differently, though -- all cores are defined in > solr.xml rather than with core.properties files. > > When I first set up these Solr instances, I don't recall having this > problem, but full Solr restarts are really rare, so it's possible I just > didn't create the right circumstances. > > Thanks, > Shawn >
Persistence problem with swapped cores after Solr restart -- 4.9.1
On Solr 4.9.1 with core discovery, I seem to be having trouble with core swaps not persisting through a full Solr restart. I apologize for the fact that this message is lean on details ... I've seen the problem twice now, but I don't have any concrete before/after information about what's in each core.properties file. I am attempting to set up the scenario again and gather that information. The entire directory structure is set up as a git repo, so I will be able to tell if any files (like core.properties) are modified for the rebuild/swap that I have started. The repo shows no changes at the moment, but I have done several of these rebuild/swap operations, so even if core.properties is being correctly updated, it might just have landed back on the original configuration. I have another copy of my index using Solr 4.7.2 with the old solr.xml format that seems to have no problems with core swapping and persistence. That works differently, though -- all cores are defined in solr.xml rather than with core.properties files. When I first set up these Solr instances, I don't recall having this problem, but full Solr restarts are really rare, so it's possible I just didn't create the right circumstances. Thanks, Shawn
