We are experiencing some issues with deltas on our production server. Once a record has been updated it can no longer be queried. We are unable to query it using the updated record by the old or the new name. The record appears to be gone from sphinx. When the issue starts happening it doesn't happen to all record types, just a few of them... mostly the larger ones. Furthermore, once the issue starts happening a full index and rotate does not fix it. We have to delete these *.temp files for the indices that stopped working. Once the files are removed everything starts working again. Unfortunately we do not have steps to recreate it, however, one hypothesis could be due to a large load. We are also using sidekiq to queue up the deltas and another hypothesis is that unrelated code in other sidekiq jobs has a segfault error or other really bad crash other than an exception. Obviously those crashes need resolved too.
When this error starts the sphinx.log file will stop outputting the below and once we remove the *.temp files they show up again. [Thu Feb 16 18:10:23.185 2017] [17066] rotating index 'advisor_delta': started [Thu Feb 16 18:10:23.186 2017] [17066] rotating index 'advisor_delta': success Therefore, all the evidence appears that items are being removed from the core index and not added to the delta index. One idea is that it happens once these *.temp files show up in the indices_location. What are these *.temp files? Is there another log file or log file setting that would help us identify the issue here? Any other data points we should be looking at? ======TECH & SETTINGS====== sphinx: 2.1.5 thinking sphinx: 3.1.3 rails: 4.2.7.1 ruby: 2.2.3 mem_limit: 1024M 64bit_timestamps: true enable_star: true min_infix_len: 2 indices_location: /data/sphinx/indexes configuration_file: /data/agencieshq/shared/config/sphinx.conf log: /data/agencieshq/shared/log/sphinx.log query_log: /data/agencieshq/shared/log/sphinx.query.log pid_file: /data/agencieshq/shared/pids/sphinx.pid binlog_path: /data/sphinx/binlog -- You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. To unsubscribe from this group and stop receiving emails from it, send an email to thinking-sphinx+unsubscr...@googlegroups.com. To post to this group, send email to thinking-sphinx@googlegroups.com. Visit this group at https://groups.google.com/group/thinking-sphinx. For more options, visit https://groups.google.com/d/optout.
