Re: 7.6.04 Full Text Search problems
We ran into some problem with FTS on server group with SQL 2008 on Windows and BMC help to tune it Fix was: Thread setting RPC socket configuration, Private Queues Removing the shared directory concept Running two instance of FTS on Primary FTS server one with 1GB and second with 2GB max memory (for Heap stack error) These settings depends on the environment so you need to figure out best working combinatin but definately you can check these varibles. We still have few issues around FTS but it did help stabilize the system. Thanks Ravi Rai Date: Thu, 31 May 2012 13:06:15 -0500 From: ddru...@rockwellcollins.com Subject: Re: 7.6.04 Full Text Search problems To: arslist@ARSLIST.ORG ** LJ, Thanks for the suggestion. I have turned it on. Thanks, Dennis "Longwing, LJ CTR MDA/IC" Sent by: "Action Request System discussion list(ARSList)" 05/31/2012 12:46 PM Please respond to arslist@ARSLIST.ORG To arslist@ARSLIST.ORG cc Subject Re: 7.6.04 Full Text Search problems I would suspect that if you ever found the time you would find that the sec= ondary worked just fine if the primary wasn't online. One suggestion I wou= ld make would be to turn on server group logging on all nodes and then star= t things up properly. The logs are VERY low volume so no problems leaving = them on full time. The logs should identify when anything in the server gr= oup changes in the range of something taking over functions, heartbeat miss= es, etc. I would suspect based on the information that you have given that= the problem is related to multiple servers trying to get a lock at the sam= e time and throwing the error. I would expect the server group logs to giv= e details about this. -Original Message- From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIS= T.ORG] On Behalf Of Dennis Ruble Sent: Thursday, May 31, 2012 10:10 AM To: arslist@ARSLIST.ORG Subject: Re: 7.6.04 Full Text Search problems **=20 LJ,=20 We've tried it both ways. BMC initially installed it with each server writ= ing to the collection directory. The problem appeared intermittent at that= time with the primary server usually being able to write and the secondary= server locked out once the problem occurred. =20 Based upon recommendations from support we reconfigured making the primary = server the only indexer. The secondary and integration servers are not in = the ranking form at all for "Full Text Index" and have the config set to "D= isable FTS Indexer". There's a paper on configuring this way due to perfor= mance, and we hoped it would take care of the lock issue too. =20 FTS does function fine when the secondary server is down. Due to test sche= dule and the amount of reconfiguration to do to make the secondary server t= he only indexer I haven't tested to see if it was any better with the secon= dary being the indexer.=20 Thanks,=20 Dennis=20 _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are" _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are"
Re: 7.6.04 Full Text Search problems
LJ, Thanks for the suggestion. I have turned it on. Thanks, Dennis "Longwing, LJ CTR MDA/IC" Sent by: "Action Request System discussion list(ARSList)" 05/31/2012 12:46 PM Please respond to arslist@ARSLIST.ORG To arslist@ARSLIST.ORG cc Subject Re: 7.6.04 Full Text Search problems I would suspect that if you ever found the time you would find that the sec= ondary worked just fine if the primary wasn't online. One suggestion I wou= ld make would be to turn on server group logging on all nodes and then star= t things up properly. The logs are VERY low volume so no problems leaving = them on full time. The logs should identify when anything in the server gr= oup changes in the range of something taking over functions, heartbeat miss= es, etc. I would suspect based on the information that you have given that= the problem is related to multiple servers trying to get a lock at the sam= e time and throwing the error. I would expect the server group logs to giv= e details about this. -Original Message- From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIS= T.ORG] On Behalf Of Dennis Ruble Sent: Thursday, May 31, 2012 10:10 AM To: arslist@ARSLIST.ORG Subject: Re: 7.6.04 Full Text Search problems **=20 LJ,=20 We've tried it both ways. BMC initially installed it with each server writ= ing to the collection directory. The problem appeared intermittent at that= time with the primary server usually being able to write and the secondary= server locked out once the problem occurred. =20 Based upon recommendations from support we reconfigured making the primary = server the only indexer. The secondary and integration servers are not in = the ranking form at all for "Full Text Index" and have the config set to "D= isable FTS Indexer". There's a paper on configuring this way due to perfor= mance, and we hoped it would take care of the lock issue too. =20 FTS does function fine when the secondary server is down. Due to test sche= dule and the amount of reconfiguration to do to make the secondary server t= he only indexer I haven't tested to see if it was any better with the secon= dary being the indexer.=20 Thanks,=20 Dennis=20 _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are" ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are"
Re: 7.6.04 Full Text Search problems
I would suspect that if you ever found the time you would find that the sec= ondary worked just fine if the primary wasn't online. One suggestion I wou= ld make would be to turn on server group logging on all nodes and then star= t things up properly. The logs are VERY low volume so no problems leaving = them on full time. The logs should identify when anything in the server gr= oup changes in the range of something taking over functions, heartbeat miss= es, etc. I would suspect based on the information that you have given that= the problem is related to multiple servers trying to get a lock at the sam= e time and throwing the error. I would expect the server group logs to giv= e details about this. -Original Message- From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIS= T.ORG] On Behalf Of Dennis Ruble Sent: Thursday, May 31, 2012 10:10 AM To: arslist@ARSLIST.ORG Subject: Re: 7.6.04 Full Text Search problems **=20 LJ,=20 We've tried it both ways. BMC initially installed it with each server writ= ing to the collection directory. The problem appeared intermittent at that= time with the primary server usually being able to write and the secondary= server locked out once the problem occurred. =20 Based upon recommendations from support we reconfigured making the primary = server the only indexer. The secondary and integration servers are not in = the ranking form at all for "Full Text Index" and have the config set to "D= isable FTS Indexer". There's a paper on configuring this way due to perfor= mance, and we hoped it would take care of the lock issue too. =20 FTS does function fine when the secondary server is down. Due to test sche= dule and the amount of reconfiguration to do to make the secondary server t= he only indexer I haven't tested to see if it was any better with the secon= dary being the indexer.=20 Thanks,=20 Dennis=20 _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are"
Re: 7.6.04 Full Text Search problems
Nate, We started in your configuration, but are currently configured per the document you attached and continue tweaking it per direction from BMC support. Our hope was it would improve performance AND clear the lock issue, but the lock problem seems more solid in this configuration. Now when the problem occurs, we have to restart arsystem on both servers to clear it. Before it only seemed to affect the secondary server. On your questions, the account Remedy is running under is in the Administrators group on each server. Both servers have been restarted repeatedly as configurations change. I don't like having the primary as the single point of failure, but I'm not fond of performance issues either. Right now we're just trying to find something that works. Luckily we are still testing in QA and working towards an early July go-live so still have a little time to test. I just got off the phone with support and will be testing some changes tonight. More tomorrow... Thanks! Dennis Nathan Aker Sent by: "Action Request System discussion list(ARSList)" 05/31/2012 10:22 AM Please respond to arslist@ARSLIST.ORG To arslist@ARSLIST.ORG cc Subject Re: 7.6.04 Full Text Search problems ** Dennis, we?re running 7.6.04 in a server group against a shared FTS collection directory. While we have had to do a lot of tuning related to plugin server performance, we?re not seeing the error you have below. A few things to validate: 1. Do you have the collection directories setup on a shared drive, with both servers pointed at the share? 2. Do you have Remedy running under a user account that also has full write access to the shared drive where the FTS collection directories are housed? 3. Did you restart both ARS servers after repointing to the share? This config is working for us, though we are facing some performance issues. We are currently evaluating the attached config changes document which effectively relocates the FTS plugin from each server in the server group to one primary server. While I?m not a fan of the fact that it basically creates a single point of failure and negates the benefits of the server group in regards to the FTS function, it is BMC?s latest recommendation to us and may get you past the locking issue as it would make all FTS plugins run on the same server. I?m happy to review any deeper configuration questions with you on this based on our experience. We?ve had to tinker a lot with FTS so I?m painfully aware of the details of our environment in this respect. Thanks. Nate. Nathan Aker ITSM Solution Architect McAfee, Inc. From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Dennis Ruble Sent: Thursday, May 31, 2012 9:46 AM To: arslist@ARSLIST.ORG Subject: 7.6.04 Full Text Search problems ** Listers, We are trying to get 7.6.04 running and are having problems with Full Text Search(FTS). We have our arsystem servers(Windows 2008 R2) in a server group and are experiencing write lock errors: FTSPlugin::initializeSearchService - Error initializing the FTSService:com.bmc.arsys.fts.impl.lucene.LuceneFTSService getIndexWriter: Error getting index writer Root Exception: Lock obtain timed out: NativeFSLock@\\crntarsqa01\collection\write.lock: java (ARERR 875) We've been working this for many weeks with BMC and have tried several different configurations with no apparent progress. I've also read evidence of other customers reporting the same issue. My questions for you are: 1. Is anyone successfully running FTS in ITSM 7.6.04 on Windows 2008 R2 in a server group? (Works fine on single server) 2. If you experienced the same error we have, did you find a fix? Any help/suggestions would be appreciated. Regards, Dennis Ruble Rockwell Collins _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ [attachment "Configuring FTS for performance in a server group.docx" deleted by Dennis D Ruble/CedarRapids/RockwellCollins] ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are"
Re: 7.6.04 Full Text Search problems
LJ, We've tried it both ways. BMC initially installed it with each server writing to the collection directory. The problem appeared intermittent at that time with the primary server usually being able to write and the secondary server locked out once the problem occurred. Based upon recommendations from support we reconfigured making the primary server the only indexer. The secondary and integration servers are not in the ranking form at all for "Full Text Index" and have the config set to "Disable FTS Indexer". There's a paper on configuring this way due to performance, and we hoped it would take care of the lock issue too. FTS does function fine when the secondary server is down. Due to test schedule and the amount of reconfiguration to do to make the secondary server the only indexer I haven't tested to see if it was any better with the secondary being the indexer. Thanks, Dennis ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are"
Re: 7.6.04 Full Text Search problems
Dennis, we're running 7.6.04 in a server group against a shared FTS collection directory. While we have had to do a lot of tuning related to plugin server performance, we're not seeing the error you have below. A few things to validate: 1. Do you have the collection directories setup on a shared drive, with both servers pointed at the share? 2. Do you have Remedy running under a user account that also has full write access to the shared drive where the FTS collection directories are housed? 3. Did you restart both ARS servers after repointing to the share? This config is working for us, though we are facing some performance issues. We are currently evaluating the attached config changes document which effectively relocates the FTS plugin from each server in the server group to one primary server. While I'm not a fan of the fact that it basically creates a single point of failure and negates the benefits of the server group in regards to the FTS function, it is BMC's latest recommendation to us and may get you past the locking issue as it would make all FTS plugins run on the same server. I'm happy to review any deeper configuration questions with you on this based on our experience. We've had to tinker a lot with FTS so I'm painfully aware of the details of our environment in this respect. Thanks. Nate. Nathan Aker ITSM Solution Architect McAfee, Inc. From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Dennis Ruble Sent: Thursday, May 31, 2012 9:46 AM To: arslist@ARSLIST.ORG Subject: 7.6.04 Full Text Search problems ** Listers, We are trying to get 7.6.04 running and are having problems with Full Text Search(FTS). We have our arsystem servers(Windows 2008 R2) in a server group and are experiencing write lock errors: FTSPlugin::initializeSearchService - Error initializing the FTSService:com.bmc.arsys.fts.impl.lucene.LuceneFTSService getIndexWriter: Error getting index writer Root Exception: Lock obtain timed out: NativeFSLock@\\crntarsqa01\collection\write.lock<mailto:NativeFSLock@\\crntarsqa01\collection\write.lock>: java (ARERR 875) We've been working this for many weeks with BMC and have tried several different configurations with no apparent progress. I've also read evidence of other customers reporting the same issue. My questions for you are: 1. Is anyone successfully running FTS in ITSM 7.6.04 on Windows 2008 R2 in a server group? (Works fine on single server) 2. If you experienced the same error we have, did you find a fix? Any help/suggestions would be appreciated. Regards, Dennis Ruble Rockwell Collins _attend WWRUG12 www.wwrug.com<http://www.wwrug.com> ARSlist: "Where the Answers Are"_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are" Configuring FTS for performance in a server group.docx Description: Configuring FTS for performance in a server group.docx
Re: 7.6.04 Full Text Search problems
Dennis, One question...I don't use FTS myself, so I'm sorry if this is covered in the documentation. Is your FTS Collection 'shared' between the servers in the server group? If so, do you have the server ranking setup properly so that only one server at a time is trying to create/update the index? I ask this simply because if you have two remedy servers writing to the same collection, only one of them is able to get a lock on the directory at a timelike I said though...I don't use FTS and have never set it up so I'm not intricately familiar with the inner workings and configuration. So...another question just from a troubleshooting perspective. If you have one of the nodes shut down in the group...does FTS function properly? What if you swap it and bring that one down and startup the other one...does that work then? Or in other words, are you only having problems when BOTH nodes are online, and then only having it on one node? -Original Message- From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Dennis Ruble Sent: Thursday, May 31, 2012 8:46 AM To: arslist@ARSLIST.ORG Subject: 7.6.04 Full Text Search problems ** Listers, We are trying to get 7.6.04 running and are having problems with Full Text Search(FTS). We have our arsystem servers(Windows 2008 R2) in a server group and are experiencing write lock errors: FTSPlugin::initializeSearchService - Error initializing the FTSService:com.bmc.arsys.fts.impl.lucene.LuceneFTSService getIndexWriter: Error getting index writer Root Exception: Lock obtain timed out: NativeFSLock@\\crntarsqa01\collection\write.lock: java (ARERR 875) We've been working this for many weeks with BMC and have tried several different configurations with no apparent progress. I've also read evidence of other customers reporting the same issue. My questions for you are: 1. Is anyone successfully running FTS in ITSM 7.6.04 on Windows 2008 R2 in a server group? (Works fine on single server) 2. If you experienced the same error we have, did you find a fix? Any help/suggestions would be appreciated. Regards, Dennis Ruble Rockwell Collins _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are"
7.6.04 Full Text Search problems
Listers, We are trying to get 7.6.04 running and are having problems with Full Text Search(FTS). We have our arsystem servers(Windows 2008 R2) in a server group and are experiencing write lock errors: FTSPlugin::initializeSearchService - Error initializing the FTSService:com.bmc.arsys.fts.impl.lucene.LuceneFTSService getIndexWriter: Error getting index writer Root Exception: Lock obtain timed out: NativeFSLock@\\crntarsqa01\collection\write.lock: java (ARERR 875) We've been working this for many weeks with BMC and have tried several different configurations with no apparent progress. I've also read evidence of other customers reporting the same issue. My questions for you are: 1. Is anyone successfully running FTS in ITSM 7.6.04 on Windows 2008 R2 in a server group? (Works fine on single server) 2. If you experienced the same error we have, did you find a fix? Any help/suggestions would be appreciated. Regards, Dennis Ruble Rockwell Collins ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are"
