I added this (untested) PR with a bit more details: https://github.com/edx/cs_comments_service/pull/220
On Wednesday, January 11, 2017 at 10:00:01 AM UTC-5, Robert R wrote: > > Hi Jill. > > I think reindex is currently broken. We are working on a PR that switches > away from Tire (used for ES), and I discovered that reindex happens to be > broken yesterday as part of that work. This PR is prep for an ES upgrade, > but is compatible with the current version of ES. > > Here is the PR, but it is not complete yet. It is probably larger than > what you need to fix your issue. One problem that I know you will have is > partially fixed in this commit: > https://github.com/edx/cs_comments_service/pull/216/commits/2be90968e96af553fa9e15a6181a9d051d065a40. > > The issue was that Rake did not have some fixes that were in app.rb. > > I can discuss more if needed. But for now, I'm going to concentrate on > landing this work. > > Robert > > On Monday, January 9, 2017 at 10:35:21 PM UTC-5, [email protected] > wrote: >> >> Hello, >> >> We're setting up a new edxapp instance for a client with existing courses >> and forum posts, and are having trouble getting ElasticSearch to return >> search results for old posts. New posts/comments are indexed just fine >> though. >> >> The new edxapp instance is running Eucalyptus.2, with data migrated from >> Dogwood. >> >> Here's what I've tried: >> >> - Ran rake search:rebuild as recommended by >> cs_comment_service/wiki/Search-Indexes >> <https://github.com/edx/cs_comments_service/wiki/Search-Indexes>. >> - Had to apply this patch >> >> <https://github.com/open-craft/cs_comments_service/commit/d14b0284814acb7b451d497cc9bb8c8cea65a8d0> >> to >> get the task to run without error (will submit that upstream). >> - Output looked promising; there were a lot of CommentThread >> entries read from mongodb: >> >> forum@c:~/cs_comments_service$ rake search:rebuild >> /edx/app/forum/cs_comments_service/lib/tasks/flags.rake:6: warning: >> already initialized constant ROOT >> /edx/app/forum/cs_comments_service/lib/tasks/kpis.rake:7: warning: >> already initialized constant ROOT >> W, [2017-01-09T04:21:07.666963 #24546] WARN -- : Overwriting existing >> field _id in class User. >> I, [2017-01-09T04:21:07.675829 #24546] INFO -- : Creating new index: >> content_20170109042107... >> I, [2017-01-09T04:21:07.787122 #24546] INFO -- : Applying index >> mappings for CommentThread >> I, [2017-01-09T04:21:07.833666 #24546] INFO -- : Applying index >> mappings for Comment >> I, [2017-01-09T04:21:07.848420 #24546] INFO -- : ...done! >> I, [2017-01-09T04:21:07.851193 #24546] INFO -- : preparing to >> (re)build index >> D, [2017-01-09T04:21:07.852430 #24546] DEBUG -- : MONGODB | Adding >> xx.xx.xx.xx:27017 to the cluster. >> D, [2017-01-09T04:21:07.856389 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.getnonce | STARTED | {} >> D, [2017-01-09T04:21:07.857681 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.getnonce | SUCCEEDED | >> 0.00074798s >> D, [2017-01-09T04:21:07.858003 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.authenticate | STARTED | {} >> D, [2017-01-09T04:21:07.859127 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.authenticate | SUCCEEDED | >> 0.000842973s >> D, [2017-01-09T04:21:07.912745 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | STARTED | >> {"count"=>"contents", "query"=>{"_type"=>{"$in"=>["Comment", >> "CommentThread"]}, "updated_at"=>{"$lte"=>2017-01-09 04:21:07 UTC}}} >> D, [2017-01-09T04:21:07.916484 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | SUCCEEDED | >> 0.003127465s >> D, [2017-01-09T04:21:07.918459 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | STARTED | >> {"count"=>"contents", "query"=>{"_type"=>{"$in"=>["Comment", >> "CommentThread"]}, "updated_at"=>{"$lte"=>2017-01-09 04:21:07 UTC}}} >> D, [2017-01-09T04:21:07.920659 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | SUCCEEDED | >> 0.0019071720000000002s >> D, [2017-01-09T04:21:07.922103 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | >> {"find"=>"contents", "filter"=>{"_type"=>{"$in"=>["Comment", >> "CommentThread"]}, "updated_at"=>{"$lte"=>2017-01-09 04:21:07 UTC}}, >> "skip"=>0, "limit"=>500} >> D, [2017-01-09T04:21:07.963584 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.041098094s >> I, [2017-01-09T04:21:07.976211 #24546] INFO -- : >> content_20170109042107: imported 0 of 280 (0.0% complete after 0.06 seconds) >> D, [2017-01-09T04:21:07.978606 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | >> {"find"=>"contents", >> "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), >> "_type"=>{"$in"=>["CommentThread"]}}} >> D, [2017-01-09T04:21:07.979844 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | >> 0.0009901159999999999s >> D, [2017-01-09T04:21:07.984179 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | >> {"find"=>"contents", >> "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), >> "_type"=>{"$in"=>["CommentThread"]}}} >> D, [2017-01-09T04:21:07.985220 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | >> 0.0007970890000000001s >> D, [2017-01-09T04:21:07.986733 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | >> {"find"=>"contents", >> "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), >> "_type"=>{"$in"=>["CommentThread"]}}} >> D, [2017-01-09T04:21:07.987790 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.000801455s >> D, [2017-01-09T04:21:07.989637 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | >> {"find"=>"contents", >> "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), >> "_type"=>{"$in"=>["CommentThread"]}}} >> D, [2017-01-09T04:21:07.990676 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.000781482s >> D, [2017-01-09T04:21:07.994360 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | >> {"find"=>"contents", >> "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), >> "_type"=>{"$in"=>["CommentThread"]}}} >> D, [2017-01-09T04:21:07.995228 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.000639859s >> D, [2017-01-09T04:21:07.996512 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | >> {"find"=>"contents", >> "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), >> "_type"=>{"$in"=>["CommentThread"]}}} >> D, [2017-01-09T04:21:07.997424 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.000684286s >> D, [2017-01-09T04:21:07.999114 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | >> {"find"=>"contents", >> "filter"=>{"_id"=>BSON::ObjectId('5771941e6fa57b9dee000003'), >> "_type"=>{"$in"=>["CommentThread"]}}} >> <snip 987 DEBUG lines similar to the last two> >> D, [2017-01-09T04:21:10.053846 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | SUCCEEDED | >> 0.003505191s >> D, [2017-01-09T04:21:10.055157 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | >> {"find"=>"contents", "filter"=>{"_type"=>{"$in"=>["Comment", >> "CommentThread"]}, "updated_at"=>{"$lte"=>2017-01-09 04:21:07 UTC}}, >> "skip"=>500, "limit"=>500} >> D, [2017-01-09T04:21:10.057621 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.002178538s >> I, [2017-01-09T04:21:10.057859 #24546] INFO -- : >> content_20170109042107: finished importing 280 documents >> I, [2017-01-09T04:21:10.063743 #24546] INFO -- : alias already exists >> (will move): content_20170109042021 >> I, [2017-01-09T04:21:10.072781 #24546] INFO -- : alias "content" now >> points to index content_20170109042107. >> I, [2017-01-09T04:21:10.072937 #24546] INFO -- : importing any >> documents that changed between 2017-01-09 04:21:07 UTC and now >> D, [2017-01-09T04:21:10.074877 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | STARTED | >> {"count"=>"contents", "query"=>{"_type"=>{"$in"=>["Comment", >> "CommentThread"]}, "updated_at"=>{"$gte"=>2017-01-09 04:21:07 UTC}}} >> D, [2017-01-09T04:21:10.077733 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | SUCCEEDED | >> 0.002418113s >> D, [2017-01-09T04:21:10.079052 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | STARTED | >> {"count"=>"contents", "query"=>{"_type"=>{"$in"=>["Comment", >> "CommentThread"]}, "updated_at"=>{"$gte"=>2017-01-09 04:21:07 UTC}}} >> D, [2017-01-09T04:21:10.081516 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.count | SUCCEEDED | >> 0.0021388329999999997s >> D, [2017-01-09T04:21:10.082713 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | STARTED | >> {"find"=>"contents", "filter"=>{"_type"=>{"$in"=>["Comment", >> "CommentThread"]}, "updated_at"=>{"$gte"=>2017-01-09 04:21:07 UTC}}, >> "skip"=>0, "limit"=>500} >> D, [2017-01-09T04:21:10.085106 #24546] DEBUG -- : MONGODB | >> xx.xx.xx.xx:27017 | c_opencraft_hosting_forum.find | SUCCEEDED | 0.002042407s >> I, [2017-01-09T04:21:10.085363 #24546] INFO -- : >> content_20170109042107: finished importing 0 documents >> >> - And sure enough, the new content_20170109042107 index now has >> the main content alias. The other indexes must be my previous >> attempts (though I don't remember trying that many times): >> >> forum@c:~/cs_comments_service$ curl http://xx.xx.xx.xx:9200/_aliases >> { >> "content_20170108125330":{"aliases":{}}, >> "content_20170108125001":{"aliases":{}}, >> "content_20170108125747":{"aliases":{}}, >> "content_20170108125315":{"aliases":{}}, >> "courseware_index":{"aliases":{}}, >> "content_20170109041935":{"aliases":{}}, >> "content_20170109042107":{"aliases":{}}, >> "content_20170108125440":{"aliases":{}}, >> "content_20170108125305":{"aliases":{}}, >> "content_20170108125433":{"aliases":{}}, >> "content_20170109045546":{"aliases":{"content":{}}}, >> "content_20170109042021":{"aliases":{}}, >> "content_20170108125518":{"aliases":{}} >> } >> >> - But there's only two entries in in the content index, and they only >> appeared after I added a new comment to the post: >> >> forum@c:~/cs_comments_service$ curl >> http://xx.xx.xx.xx:9200/content/_search >> >> {"took":2,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0}, >> "hits":{"total":2,"max_score":1.0, >> "hits":[ >> >> {"_index":"content_20170109045546","_type":"comment","_id":"5873153298615406ec000000","_score":1.0, >> "_source" : >> {"body":"foxes","course_id":"course-v1:edX+DemoX+Demo_Course","created_at":"2017-01-09T04:44:34Z","updated_at":"2017-01-09T04:44:34Z","commentable_id":"cba3e4cd91d0466b9ac50926e495b76f","comment_thread_id":"587203839861540683000000","group_id":{},"context":"course","id":"5873153298615406ec000000","_type":"comment"}}, >> >> {"_index":"content_20170109045546","_type":"comment_thread","_id":"587203839861540683000000","_score":1.0, >> "_source" : {"title":"Aeroplane videotape stomach","body":"Random words for >> the forum search >> test","course_id":"course-v1:edX+DemoX+Demo_Course","commentable_id":"cba3e4cd91d0466b9ac50926e495b76f","created_at":"2017-01-08T09:16:51Z","updated_at":"2017-01-09T04:44:34Z","context":"course","last_activity_at":"2017-01-09T04:44:34Z","id":"587203839861540683000000","group_id":{},"votes_point":0,"author_id":"2582","thread_id":"587203839861540683000000","_type":"comment_thread"}} >> ] >> }} >> >> >> >> Has anyone seen this before? Any ideas? >> >> -- >> Jill >> @OpenCraft <http://opencraft.com/> >> > -- You received this message because you are subscribed to the Google Groups "General Open edX discussion" group. To view this discussion on the web visit https://groups.google.com/d/msgid/edx-code/1011dbab-cd51-4026-922e-0a6700943fea%40googlegroups.com.
