Hi Jill. I think reindex is currently broken. We are working on a PR that switches away from Tire (used for ES), and I discovered that reindex happens to be broken yesterday as part of that work. This PR is prep for an ES upgrade, but is compatible with the current version of ES.
Here is the PR, but it is not complete yet. It is probably larger than what you need to fix your issue. One problem that I know you will have is partially fixed in this commit: https://github.com/edx/cs_comments_service/pull/216/commits/2be90968e96af553fa9e15a6181a9d051d065a40. The issue was that Rake did not have some fixes that were in app.rb. I can discuss more if needed. But for now, I'm going to concentrate on landing this work. Robert On Monday, January 9, 2017 at 10:35:21 PM UTC-5, [email protected] wrote: > > Hello, > > We're setting up a new edxapp instance for a client with existing courses > and forum posts, and are having trouble getting ElasticSearch to return > search results for old posts. New posts/comments are indexed just fine > though. > > The new edxapp instance is running Eucalyptus.2, with data migrated from > Dogwood. > > Here's what I've tried: > > - Ran rake search:rebuild as recommended by > cs_comment_service/wiki/Search-Indexes > <https://github.com/edx/cs_comments_service/wiki/Search-Indexes>. > - Had to apply this patch > > <https://github.com/open-craft/cs_comments_service/commit/d14b0284814acb7b451d497cc9bb8c8cea65a8d0> > to > get the task to run without error (will submit that upstream). > - Output looked promising; there were a lot of CommentThread > entries read from mongodb: > > forum@c:~/cs_comments_service$ rake search:rebuild > /edx/app/forum/cs_comments_service/lib/tasks/flags.rake:6: warning: > already initialized constant ROOT > /edx/app/forum/cs_comments_service/lib/tasks/kpis.rake:7: warning: > already initialized constant ROOT > W, [2017-01-09T04:21:07.666963 #24546] WARN -- : Overwriting existing > field _id in class User. > I, [2017-01-09T04:21:07.675829 #24546] INFO -- : Creating new index: > content_20170109042107... > I, [2017-01-09T04:21:07.787122 #24546] INFO -- : Applying index > mappings for CommentThread > I, [2017-01-09T04:21:07.833666 #24546] INFO -- : Applying index > mappings for Comment > I, [2017-01-09T04:21:07.848420 #24546] INFO -- : ...done! > I, [2017-01-09T04:21:07.851193 #24546] INFO -- : preparing to > (re)build index > D, [2017-01-09T04:21:07.852430 #24546] DEBUG -- : MONGODB | Adding > xx.xx.xx.xx:27017 to the cluster. > D, [2017-01-09T04:21:07.856389 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.getnonce | STARTED | {} > D, [2017-01-09T04:21:07.857681 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.getnonce | SUCCEEDED | > 0.00074798s > D, [2017-01-09T04:21:07.858003 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.authenticate | STARTED | {} > D, [2017-01-09T04:21:07.859127 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.authenticate | SUCCEEDED | > 0.000842973s > D, [2017-01-09T04:21:07.912745 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | STARTED | > {"count"=>"contents", "query"=>{"_type"=>{"$in"=>["Comment", > "CommentThread"]}, "updated_at"=>{"$lte"=>2017-01-09 04:21:07 UTC}}} > D, [2017-01-09T04:21:07.916484 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | SUCCEEDED | 0.003127465s > D, [2017-01-09T04:21:07.918459 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | STARTED | > {"count"=>"contents", "query"=>{"_type"=>{"$in"=>["Comment", > "CommentThread"]}, "updated_at"=>{"$lte"=>2017-01-09 04:21:07 UTC}}} > D, [2017-01-09T04:21:07.920659 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | SUCCEEDED | > 0.0019071720000000002s > D, [2017-01-09T04:21:07.922103 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | > {"find"=>"contents", "filter"=>{"_type"=>{"$in"=>["Comment", > "CommentThread"]}, "updated_at"=>{"$lte"=>2017-01-09 04:21:07 UTC}}, > "skip"=>0, "limit"=>500} > D, [2017-01-09T04:21:07.963584 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.041098094s > I, [2017-01-09T04:21:07.976211 #24546] INFO -- : > content_20170109042107: imported 0 of 280 (0.0% complete after 0.06 seconds) > D, [2017-01-09T04:21:07.978606 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | > {"find"=>"contents", > "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), > "_type"=>{"$in"=>["CommentThread"]}}} > D, [2017-01-09T04:21:07.979844 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | > 0.0009901159999999999s > D, [2017-01-09T04:21:07.984179 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | > {"find"=>"contents", > "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), > "_type"=>{"$in"=>["CommentThread"]}}} > D, [2017-01-09T04:21:07.985220 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | > 0.0007970890000000001s > D, [2017-01-09T04:21:07.986733 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | > {"find"=>"contents", > "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), > "_type"=>{"$in"=>["CommentThread"]}}} > D, [2017-01-09T04:21:07.987790 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.000801455s > D, [2017-01-09T04:21:07.989637 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | > {"find"=>"contents", > "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), > "_type"=>{"$in"=>["CommentThread"]}}} > D, [2017-01-09T04:21:07.990676 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.000781482s > D, [2017-01-09T04:21:07.994360 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | > {"find"=>"contents", > "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), > "_type"=>{"$in"=>["CommentThread"]}}} > D, [2017-01-09T04:21:07.995228 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.000639859s > D, [2017-01-09T04:21:07.996512 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | > {"find"=>"contents", > "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), > "_type"=>{"$in"=>["CommentThread"]}}} > D, [2017-01-09T04:21:07.997424 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.000684286s > D, [2017-01-09T04:21:07.999114 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | > {"find"=>"contents", > "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), > "_type"=>{"$in"=>["CommentThread"]}}} > <snip 987 DEBUG lines similar to the last two> > D, [2017-01-09T04:21:10.053846 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | SUCCEEDED | 0.003505191s > D, [2017-01-09T04:21:10.055157 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | > {"find"=>"contents", "filter"=>{"_type"=>{"$in"=>["Comment", > "CommentThread"]}, "updated_at"=>{"$lte"=>2017-01-09 04:21:07 UTC}}, > "skip"=>500, "limit"=>500} > D, [2017-01-09T04:21:10.057621 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.002178538s > I, [2017-01-09T04:21:10.057859 #24546] INFO -- : > content_20170109042107: finished importing 280 documents > I, [2017-01-09T04:21:10.063743 #24546] INFO -- : alias already exists > (will move): content_20170109042021 > I, [2017-01-09T04:21:10.072781 #24546] INFO -- : alias "content" now > points to index content_20170109042107. > I, [2017-01-09T04:21:10.072937 #24546] INFO -- : importing any > documents that changed between 2017-01-09 04:21:07 UTC and now > D, [2017-01-09T04:21:10.074877 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | STARTED | > {"count"=>"contents", "query"=>{"_type"=>{"$in"=>["Comment", > "CommentThread"]}, "updated_at"=>{"$gte"=>2017-01-09 04:21:07 UTC}}} > D, [2017-01-09T04:21:10.077733 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | SUCCEEDED | 0.002418113s > D, [2017-01-09T04:21:10.079052 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | STARTED | > {"count"=>"contents", "query"=>{"_type"=>{"$in"=>["Comment", > "CommentThread"]}, "updated_at"=>{"$gte"=>2017-01-09 04:21:07 UTC}}} > D, [2017-01-09T04:21:10.081516 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | SUCCEEDED | > 0.0021388329999999997s > D, [2017-01-09T04:21:10.082713 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | > {"find"=>"contents", "filter"=>{"_type"=>{"$in"=>["Comment", > "CommentThread"]}, "updated_at"=>{"$gte"=>2017-01-09 04:21:07 UTC}}, > "skip"=>0, "limit"=>500} > D, [2017-01-09T04:21:10.085106 #24546] DEBUG -- : MONGODB | > xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.002042407s > I, [2017-01-09T04:21:10.085363 #24546] INFO -- : > content_20170109042107: finished importing 0 documents > > - And sure enough, the new content_20170109042107 index now has the > main content alias. The other indexes must be my previous attempts > (though I don't remember trying that many times): > > forum@c:~/cs_comments_service$ curl http://xx.xx.xx.xx:9200/_aliases > { > "content_20170108125330":{"aliases":{}}, > "content_20170108125001":{"aliases":{}}, > "content_20170108125747":{"aliases":{}}, > "content_20170108125315":{"aliases":{}}, > "courseware_index":{"aliases":{}}, > "content_20170109041935":{"aliases":{}}, > "content_20170109042107":{"aliases":{}}, > "content_20170108125440":{"aliases":{}}, > "content_20170108125305":{"aliases":{}}, > "content_20170108125433":{"aliases":{}}, > "content_20170109045546":{"aliases":{"content":{}}}, > "content_20170109042021":{"aliases":{}}, > "content_20170108125518":{"aliases":{}} > } > > - But there's only two entries in in the content index, and they only > appeared after I added a new comment to the post: > > forum@c:~/cs_comments_service$ curl http://xx.xx.xx.xx:9200/content/_search > > {"took":2,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0}, > "hits":{"total":2,"max_score":1.0, > "hits":[ > > {"_index":"content_20170109045546","_type":"comment","_id":"5873153298615406ec000000","_score":1.0, > "_source" : > {"body":"foxes","course_id":"course-v1:edX+DemoX+Demo_Course","created_at":"2017-01-09T04:44:34Z","updated_at":"2017-01-09T04:44:34Z","commentable_id":"cba3e4cd91d0466b9ac50926e495b76f","comment_thread_id":"587203839861540683000000","group_id":{},"context":"course","id":"5873153298615406ec000000","_type":"comment"}}, > > {"_index":"content_20170109045546","_type":"comment_thread","_id":"587203839861540683000000","_score":1.0, > "_source" : {"title":"Aeroplane videotape stomach","body":"Random words for > the forum search > test","course_id":"course-v1:edX+DemoX+Demo_Course","commentable_id":"cba3e4cd91d0466b9ac50926e495b76f","created_at":"2017-01-08T09:16:51Z","updated_at":"2017-01-09T04:44:34Z","context":"course","last_activity_at":"2017-01-09T04:44:34Z","id":"587203839861540683000000","group_id":{},"votes_point":0,"author_id":"2582","thread_id":"587203839861540683000000","_type":"comment_thread"}} > ] > }} > > > > Has anyone seen this before? Any ideas? > > -- > Jill > @OpenCraft <http://opencraft.com/> > -- You received this message because you are subscribed to the Google Groups "General Open edX discussion" group. To view this discussion on the web visit https://groups.google.com/d/msgid/edx-code/a4a9ace8-676b-495b-9ea2-3132f53ac21d%40googlegroups.com.
