Hi Kyle, You’re spot on with the temp files being the cause of the problem.
Firstly, to explain the temp (guard) files: they’ve been part of Thinking Sphinx since 3.1.0 and are a simple locking mechanism to ensure Thinking Sphinx doesn’t try to process the same index concurrently. Since 3.1.2, the files should be cleaned up on exceptions, but yes, as you’ve said more serious crashes might skip that logic? I can’t think of other situations, to be honest… When indexing is requested and the index is already locked, there will be a ‘warn’ level log to note this in the Rails logs: Guard file for index advisor_delta exists, not indexing … so, perhaps hunting through those logs and anything Sidekiq outputs might provide clues? If there are segfaults or similar, surely they’d be in the logs? One related thing that may help is adding a uniqueness constraint to Sidekiq, at least for the ts_delta queue, to avoid concurrent indexing of any given delta. If I think of anything else that may help, I’ll let you know. — Pat > On 17 Feb 2017, at 11:19 am, Kyle Ginavan <k...@onehq.com > <mailto:k...@onehq.com>> wrote: > > We are experiencing some issues with deltas on our production server. Once a > record has been updated it can no longer be queried. We are unable to query > it using the updated record by the old or the new name. The record appears to > be gone from sphinx. When the issue starts happening it doesn't happen to all > record types, just a few of them... mostly the larger ones. Furthermore, once > the issue starts happening a full index and rotate does not fix it. We have > to delete these *.temp files for the indices that stopped working. Once the > files are removed everything starts working again. Unfortunately we do not > have steps to recreate it, however, one hypothesis could be due to a large > load. We are also using sidekiq to queue up the deltas and another hypothesis > is that unrelated code in other sidekiq jobs has a segfault error or other > really bad crash other than an exception. Obviously those crashes need > resolved too. > > When this error starts the sphinx.log file will stop outputting the below and > once we remove the *.temp files they show up again. > [Thu Feb 16 18:10:23.185 2017] [17066] rotating index 'advisor_delta': started > [Thu Feb 16 18:10:23.186 2017] [17066] rotating index 'advisor_delta': success > > Therefore, all the evidence appears that items are being removed from the > core index and not added to the delta index. One idea is that it happens once > these *.temp files show up in the indices_location. What are these *.temp > files? Is there another log file or log file setting that would help us > identify the issue here? Any other data points we should be looking at? > > ======TECH & SETTINGS====== > > sphinx: 2.1.5 > thinking sphinx: 3.1.3 > rails: 4.2.7.1 > ruby: 2.2.3 > > mem_limit: 1024M > 64bit_timestamps: true > enable_star: true > min_infix_len: 2 > indices_location: /data/sphinx/indexes > configuration_file: /data/agencieshq/shared/config/sphinx.conf > log: /data/agencieshq/shared/log/sphinx.log > query_log: /data/agencieshq/shared/log/sphinx.query.log > pid_file: /data/agencieshq/shared/pids/sphinx.pid > binlog_path: /data/sphinx/binlog > > > -- > You received this message because you are subscribed to the Google Groups > "Thinking Sphinx" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to thinking-sphinx+unsubscr...@googlegroups.com > <mailto:thinking-sphinx+unsubscr...@googlegroups.com>. > To post to this group, send email to thinking-sphinx@googlegroups.com > <mailto:thinking-sphinx@googlegroups.com>. > Visit this group at https://groups.google.com/group/thinking-sphinx > <https://groups.google.com/group/thinking-sphinx>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. To unsubscribe from this group and stop receiving emails from it, send an email to thinking-sphinx+unsubscr...@googlegroups.com. To post to this group, send email to thinking-sphinx@googlegroups.com. Visit this group at https://groups.google.com/group/thinking-sphinx. For more options, visit https://groups.google.com/d/optout.
