Hello, We're setting up a new edxapp instance for a client with existing courses and forum posts, and are having trouble getting ElasticSearch to return search results for old posts. New posts/comments are indexed just fine though.
The new edxapp instance is running Eucalyptus.2, with data migrated from Dogwood. Here's what I've tried: - Ran rake search:rebuild as recommended by cs_comment_service/wiki/Search-Indexes <https://github.com/edx/cs_comments_service/wiki/Search-Indexes>. - Had to apply this patch <https://github.com/open-craft/cs_comments_service/commit/d14b0284814acb7b451d497cc9bb8c8cea65a8d0> to get the task to run without error (will submit that upstream). - Output looked promising; there were a lot of CommentThread entries read from mongodb: forum@c:~/cs_comments_service$ rake search:rebuild /edx/app/forum/cs_comments_service/lib/tasks/flags.rake:6: warning: already initialized constant ROOT /edx/app/forum/cs_comments_service/lib/tasks/kpis.rake:7: warning: already initialized constant ROOT W, [2017-01-09T04:21:07.666963 #24546] WARN -- : Overwriting existing field _id in class User. I, [2017-01-09T04:21:07.675829 #24546] INFO -- : Creating new index: content_20170109042107... I, [2017-01-09T04:21:07.787122 #24546] INFO -- : Applying index mappings for CommentThread I, [2017-01-09T04:21:07.833666 #24546] INFO -- : Applying index mappings for Comment I, [2017-01-09T04:21:07.848420 #24546] INFO -- : ...done! I, [2017-01-09T04:21:07.851193 #24546] INFO -- : preparing to (re)build index D, [2017-01-09T04:21:07.852430 #24546] DEBUG -- : MONGODB | Adding xx.xx.xx.xx:27017 to the cluster. D, [2017-01-09T04:21:07.856389 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.getnonce | STARTED | {} D, [2017-01-09T04:21:07.857681 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.getnonce | SUCCEEDED | 0.00074798s D, [2017-01-09T04:21:07.858003 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.authenticate | STARTED | {} D, [2017-01-09T04:21:07.859127 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.authenticate | SUCCEEDED | 0.000842973s D, [2017-01-09T04:21:07.912745 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | STARTED | {"count"=>"contents", "query"=>{"_type"=>{"$in"=>["Comment", "CommentThread"]}, "updated_at"=>{"$lte"=>2017-01-09 04:21:07 UTC}}} D, [2017-01-09T04:21:07.916484 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | SUCCEEDED | 0.003127465s D, [2017-01-09T04:21:07.918459 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | STARTED | {"count"=>"contents", "query"=>{"_type"=>{"$in"=>["Comment", "CommentThread"]}, "updated_at"=>{"$lte"=>2017-01-09 04:21:07 UTC}}} D, [2017-01-09T04:21:07.920659 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | SUCCEEDED | 0.0019071720000000002s D, [2017-01-09T04:21:07.922103 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | {"find"=>"contents", "filter"=>{"_type"=>{"$in"=>["Comment", "CommentThread"]}, "updated_at"=>{"$lte"=>2017-01-09 04:21:07 UTC}}, "skip"=>0, "limit"=>500} D, [2017-01-09T04:21:07.963584 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.041098094s I, [2017-01-09T04:21:07.976211 #24546] INFO -- : content_20170109042107: imported 0 of 280 (0.0% complete after 0.06 seconds) D, [2017-01-09T04:21:07.978606 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | {"find"=>"contents", "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), "_type"=>{"$in"=>["CommentThread"]}}} D, [2017-01-09T04:21:07.979844 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.0009901159999999999s D, [2017-01-09T04:21:07.984179 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | {"find"=>"contents", "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), "_type"=>{"$in"=>["CommentThread"]}}} D, [2017-01-09T04:21:07.985220 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.0007970890000000001s D, [2017-01-09T04:21:07.986733 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | {"find"=>"contents", "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), "_type"=>{"$in"=>["CommentThread"]}}} D, [2017-01-09T04:21:07.987790 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.000801455s D, [2017-01-09T04:21:07.989637 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | {"find"=>"contents", "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), "_type"=>{"$in"=>["CommentThread"]}}} D, [2017-01-09T04:21:07.990676 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.000781482s D, [2017-01-09T04:21:07.994360 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | {"find"=>"contents", "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), "_type"=>{"$in"=>["CommentThread"]}}} D, [2017-01-09T04:21:07.995228 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.000639859s D, [2017-01-09T04:21:07.996512 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | {"find"=>"contents", "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), "_type"=>{"$in"=>["CommentThread"]}}} D, [2017-01-09T04:21:07.997424 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.000684286s D, [2017-01-09T04:21:07.999114 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | {"find"=>"contents", "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), "_type"=>{"$in"=>["CommentThread"]}}} <snip 987 DEBUG lines similar to the last two> D, [2017-01-09T04:21:10.053846 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | SUCCEEDED | 0.003505191s D, [2017-01-09T04:21:10.055157 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | {"find"=>"contents", "filter"=>{"_type"=>{"$in"=>["Comment", "CommentThread"]}, "updated_at"=>{"$lte"=>2017-01-09 04:21:07 UTC}}, "skip"=>500, "limit"=>500} D, [2017-01-09T04:21:10.057621 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.002178538s I, [2017-01-09T04:21:10.057859 #24546] INFO -- : content_20170109042107: finished importing 280 documents I, [2017-01-09T04:21:10.063743 #24546] INFO -- : alias already exists (will move): content_20170109042021 I, [2017-01-09T04:21:10.072781 #24546] INFO -- : alias "content" now points to index content_20170109042107. I, [2017-01-09T04:21:10.072937 #24546] INFO -- : importing any documents that changed between 2017-01-09 04:21:07 UTC and now D, [2017-01-09T04:21:10.074877 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | STARTED | {"count"=>"contents", "query"=>{"_type"=>{"$in"=>["Comment", "CommentThread"]}, "updated_at"=>{"$gte"=>2017-01-09 04:21:07 UTC}}} D, [2017-01-09T04:21:10.077733 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | SUCCEEDED | 0.002418113s D, [2017-01-09T04:21:10.079052 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | STARTED | {"count"=>"contents", "query"=>{"_type"=>{"$in"=>["Comment", "CommentThread"]}, "updated_at"=>{"$gte"=>2017-01-09 04:21:07 UTC}}} D, [2017-01-09T04:21:10.081516 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | SUCCEEDED | 0.0021388329999999997s D, [2017-01-09T04:21:10.082713 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | {"find"=>"contents", "filter"=>{"_type"=>{"$in"=>["Comment", "CommentThread"]}, "updated_at"=>{"$gte"=>2017-01-09 04:21:07 UTC}}, "skip"=>0, "limit"=>500} D, [2017-01-09T04:21:10.085106 #24546] DEBUG -- : MONGODB | xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.002042407s I, [2017-01-09T04:21:10.085363 #24546] INFO -- : content_20170109042107: finished importing 0 documents - And sure enough, the new content_20170109042107 index now has the main content alias. The other indexes must be my previous attempts (though I don't remember trying that many times): forum@c:~/cs_comments_service$ curl http://xx.xx.xx.xx:9200/_aliases { "content_20170108125330":{"aliases":{}}, "content_20170108125001":{"aliases":{}}, "content_20170108125747":{"aliases":{}}, "content_20170108125315":{"aliases":{}}, "courseware_index":{"aliases":{}}, "content_20170109041935":{"aliases":{}}, "content_20170109042107":{"aliases":{}}, "content_20170108125440":{"aliases":{}}, "content_20170108125305":{"aliases":{}}, "content_20170108125433":{"aliases":{}}, "content_20170109045546":{"aliases":{"content":{}}}, "content_20170109042021":{"aliases":{}}, "content_20170108125518":{"aliases":{}} } - But there's only two entries in in the content index, and they only appeared after I added a new comment to the post: forum@c:~/cs_comments_service$ curl http://xx.xx.xx.xx:9200/content/_search {"took":2,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0}, "hits":{"total":2,"max_score":1.0, "hits":[ {"_index":"content_20170109045546","_type":"comment","_id":"5873153298615406ec000000","_score":1.0, "_source" : {"body":"foxes","course_id":"course-v1:edX+DemoX+Demo_Course","created_at":"2017-01-09T04:44:34Z","updated_at":"2017-01-09T04:44:34Z","commentable_id":"cba3e4cd91d0466b9ac50926e495b76f","comment_thread_id":"587203839861540683000000","group_id":{},"context":"course","id":"5873153298615406ec000000","_type":"comment"}}, {"_index":"content_20170109045546","_type":"comment_thread","_id":"587203839861540683000000","_score":1.0, "_source" : {"title":"Aeroplane videotape stomach","body":"Random words for the forum search test","course_id":"course-v1:edX+DemoX+Demo_Course","commentable_id":"cba3e4cd91d0466b9ac50926e495b76f","created_at":"2017-01-08T09:16:51Z","updated_at":"2017-01-09T04:44:34Z","context":"course","last_activity_at":"2017-01-09T04:44:34Z","id":"587203839861540683000000","group_id":{},"votes_point":0,"author_id":"2582","thread_id":"587203839861540683000000","_type":"comment_thread"}} ] }} Has anyone seen this before? Any ideas? -- Jill @OpenCraft <http://opencraft.com/> -- You received this message because you are subscribed to the Google Groups "General Open edX discussion" group. To view this discussion on the web visit https://groups.google.com/d/msgid/edx-code/c6f6340c-8f04-4c52-96d4-3aa307e5ce1a%40googlegroups.com.
