Update:
# From https://openedx.slack.com/archives/general/p1481721224000199
I ran `sudo service elasticsearch start` but it came back with
```* Starting ElasticSearch Server
[ OK
]
```
but when I check the status `sudo service elasticsearch status` it says
``` * elasticsearch is not running
```
[9:38]
@tobz I checked the log files located in `/var/log/elasticsearch/*` but
couldn't find anything relevant to an error. I'm going to revisit the
`/etc/init.d/elasticsearch` script to see if it exists early for something.
[9:41]
@tobz Actually I was looking at the wrong log file. When you look at
`/etc/default/elasticsearch` it sets the log path to
`LOG_DIR=/edx/var/log/elasticsearch`. I found some more relevant logs.
2016-12-14 14:46:52,635][WARN ][common.jna ] Unknown mlockall
error 0
[2016-12-14 14:46:52,703][INFO ][node ] [Wild Child]
version[0.90.13], pid[8898], build[249c9c5/2014-03-25T15:27:12Z]
[2016-12-14 14:46:52,703][INFO ][node ] [Wild Child]
initializing ...
[2016-12-14 14:46:52,708][INFO ][plugins ] [Wild Child]
loaded [], sites []
[2016-12-14 14:46:54,214][WARN ][transport ] [Wild Child]
Registered two transport handlers for action discovery/zen/multicast,
handlers:
org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Multicast$
[2016-12-14 14:46:54,485][WARN ][transport ] [Wild Child]
Registered two transport handlers for action discovery/zen/multicast,
handlers:
org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Multicast$
< A bunch of these warnings >
[2016-12-14 14:46:54,839][WARN ][transport ] [Wild Child]
Registered two transport handlers for action discovery/zen/multicast,
handlers:
org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$MulticastPingResponseRequestHandler@20d3e3b1,
org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$MulticastPingResponseRequestHandler@4389d21
[2016-12-14 14:46:54,856][ERROR][bootstrap ] {0.90.13}:
Initialization Failed ...
1) IllegalStateException[This is a proxy used to support circular
references involving constructors. The object we're proxying is not
constructed yet. Please wait until after injection has completed to use
this object.]2) ElasticSearchIllegalArgumentException[Failed to resolve
address for [[]]
StringIndexOutOfBoundsException[String index out of range: -2]
Zach
On Tuesday, December 13, 2016 at 10:17:04 AM UTC-5, Zachary Trabookis wrote:
>
> I setup an *OPENEDX_RELEASE="open-release/eucalyptus.2" *using these
> instructions
> http://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/open-release-eucalyptus.master/installation/analytics/install_analytics.html.
>
> After starting the LMS devstack and halting the virtual machine while that
> application was running I had to following these directions to unlock
> MongoDB.
>
>
> http://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/latest/installation/devstack/troubleshooting_vm_installation.html#starting-open-edx-servers-after-a-mongodb-failure
>
> sudo rm /edx/var/mongo/mongodb/mongod.lock
> sudo mongod -repair --config /etc/mongod.conf
> sudo chown -R mongodb:mongodb /edx/var/mongo/.
> sudo service mongod start
>
>
> After restarting the LMS I came across this issue with *elasticsearch*.
> Now I also noticed the http://localhost:8000/courses page shows this
> error "There was an error, try searching again". I've closed out the
> running LMS application and halted the VM and restarted it with no luck.
> How can I get past these elasticsearch issues below.
>
> edxapp@precise64:~/edx-platform$ paver devstack lms
> ---> pavelib.servers.devstack
> ---> pavelib.prereqs.install_prereqs
> ---> pavelib.prereqs.install_node_prereqs
> Node prereqs unchanged, skipping...
> ---> pavelib.prereqs.install_python_prereqs
> ---> pavelib.prereqs.uninstall_python_packages
> Python uninstalls unchanged, skipping...
> Python prereqs unchanged, skipping...
> python manage.py cms --settings=devstack reindex_course --setup
> /edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/require/helpers.py:4:
>
> RemovedInDjango19Warning: django.utils.importlib will be removed in Django
> 1.9.
> from django.utils.importlib import import_module
>
> 2016-12-13 09:57:46,803 WARNING 3304 [py.warnings] importlib.py:9 -
> /edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/require/helpers.py:4:
>
> RemovedInDjango19Warning: django.utils.importlib will be removed in Django
> 1.9.
> from django.utils.importlib import import_module
>
> 2016-12-13 09:57:50,460 INFO 3304 [urllib3.connectionpool]
> connectionpool.py:257 - Starting new HTTP connection (1): localhost
> 2016-12-13 09:57:50,461 WARNING 3304 [urllib3.connectionpool]
> connectionpool.py:603 - Retrying (3 attempts remain) after connection
> broken by 'error(111, 'Connection refused')': /courseware_index
> 2016-12-13 09:57:50,461 INFO 3304 [urllib3.connectionpool]
> connectionpool.py:257 - Starting new HTTP connection (2): localhost
> 2016-12-13 09:57:50,462 WARNING 3304 [urllib3.connectionpool]
> connectionpool.py:603 - Retrying (2 attempts remain) after connection
> broken by 'error(111, 'Connection refused')': /courseware_index
> 2016-12-13 09:57:50,462 INFO 3304 [urllib3.connectionpool]
> connectionpool.py:257 - Starting new HTTP connection (3): localhost
> 2016-12-13 09:57:50,463 WARNING 3304 [urllib3.connectionpool]
> connectionpool.py:603 - Retrying (1 attempts remain) after connection
> broken by 'error(111, 'Connection refused')': /courseware_index
> 2016-12-13 09:57:50,463 INFO 3304 [urllib3.connectionpool]
> connectionpool.py:257 - Starting new HTTP connection (4): localhost
> 2016-12-13 09:57:50,464 WARNING 3304 [elasticsearch] base.py:68 - HEAD
> http://localhost:9200/courseware_index [status:N/A request:0.005s]
> Traceback (most recent call last):
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py",
>
> line 46, in perform_request
> response = self.pool.urlopen(method, url, body, **kw)
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py",
>
> line 607, in urlopen
> release_conn=release_conn, **response_kw)
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py",
>
> line 607, in urlopen
> release_conn=release_conn, **response_kw)
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py",
>
> line 607, in urlopen
> release_conn=release_conn, **response_kw)
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py",
>
> line 591, in urlopen
> raise MaxRetryError(self, url, e)
> MaxRetryError: HTTPConnectionPool(host='localhost', port=9200): Max
> retries exceeded with url: /courseware_index (Caused by <class
> 'socket.error'>: [Errno 111] Connection refused)
> 2016-12-13 09:57:50,469 INFO 3304 [elasticsearch] base.py:70 - > None
> 2016-12-13 09:57:50,470 WARNING 3304 [elasticsearch]
> connection_pool.py:141 - Connection <Urllib3HttpConnection:
> http://localhost:9200> has failed for 1 times in a row, putting on 60
> second timeout.
> 2016-12-13 09:57:50,470 INFO 3304 [elasticsearch] connection_pool.py:185 -
> Resurrecting connection <Urllib3HttpConnection: http://localhost:9200>
> (force=True).
> 2016-12-13 09:57:50,471 INFO 3304 [urllib3.connectionpool]
> connectionpool.py:257 - Starting new HTTP connection (5): localhost
> 2016-12-13 09:57:50,472 WARNING 3304 [urllib3.connectionpool]
> connectionpool.py:603 - Retrying (3 attempts remain) after connection
> broken by 'error(111, 'Connection refused')': /courseware_index
> 2016-12-13 09:57:50,472 INFO 3304 [urllib3.connectionpool]
> connectionpool.py:257 - Starting new HTTP connection (6): localhost
> 2016-12-13 09:57:50,472 WARNING 3304 [urllib3.connectionpool]
> connectionpool.py:603 - Retrying (2 attempts remain) after connection
> broken by 'error(111, 'Connection refused')': /courseware_index
> 2016-12-13 09:57:50,473 INFO 3304 [urllib3.connectionpool]
> connectionpool.py:257 - Starting new HTTP connection (7): localhost
> 2016-12-13 09:57:50,473 WARNING 3304 [urllib3.connectionpool]
> connectionpool.py:603 - Retrying (1 attempts remain) after connection
> broken by 'error(111, 'Connection refused')': /courseware_index
> 2016-12-13 09:57:50,474 INFO 3304 [urllib3.connectionpool]
> connectionpool.py:257 - Starting new HTTP connection (8): localhost
> 2016-12-13 09:57:50,474 WARNING 3304 [elasticsearch] base.py:68 - HEAD
> http://localhost:9200/courseware_index [status:N/A request:0.003s]
> Traceback (most recent call last):
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py",
>
> line 46, in perform_request
> response = self.pool.urlopen(method, url, body, **kw)
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py",
>
> line 607, in urlopen
> release_conn=release_conn, **response_kw)
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py",
>
> line 607, in urlopen
> release_conn=release_conn, **response_kw)
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py",
>
> line 607, in urlopen
> release_conn=release_conn, **response_kw)
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py",
>
> line 591, in urlopen
> raise MaxRetryError(self, url, e)
> MaxRetryError: HTTPConnectionPool(host='localhost', port=9200): Max
> retries exceeded with url: /courseware_index (Caused by <class
> 'socket.error'>: [Errno 111] Connection refused)
> 2016-12-13 09:57:50,477 INFO 3304 [elasticsearch] base.py:70 - > None
> 2016-12-13 09:57:50,477 WARNING 3304 [elasticsearch]
> connection_pool.py:141 - Connection <Urllib3HttpConnection:
> http://localhost:9200> has failed for 2 times in a row, putting on 120
> second timeout.
> 2016-12-13 09:57:50,478 INFO 3304 [elasticsearch] connection_pool.py:185 -
> Resurrecting connection <Urllib3HttpConnection: http://localhost:9200>
> (force=True).
> 2016-12-13 09:57:50,478 INFO 3304 [urllib3.connectionpool]
> connectionpool.py:257 - Starting new HTTP connection (9): localhost
> 2016-12-13 09:57:50,479 WARNING 3304 [urllib3.connectionpool]
> connectionpool.py:603 - Retrying (3 attempts remain) after connection
> broken by 'error(111, 'Connection refused')': /courseware_index
> 2016-12-13 09:57:50,479 INFO 3304 [urllib3.connectionpool]
> connectionpool.py:257 - Starting new HTTP connection (10): localhost
> 2016-12-13 09:57:50,479 WARNING 3304 [urllib3.connectionpool]
> connectionpool.py:603 - Retrying (2 attempts remain) after connection
> broken by 'error(111, 'Connection refused')': /courseware_index
> 2016-12-13 09:57:50,480 INFO 3304 [urllib3.connectionpool]
> connectionpool.py:257 - Starting new HTTP connection (11): localhost
> 2016-12-13 09:57:50,480 WARNING 3304 [urllib3.connectionpool]
> connectionpool.py:603 - Retrying (1 attempts remain) after connection
> broken by 'error(111, 'Connection refused')': /courseware_index
> 2016-12-13 09:57:50,481 INFO 3304 [urllib3.connectionpool]
> connectionpool.py:257 - Starting new HTTP connection (12): localhost
> 2016-12-13 09:57:50,481 WARNING 3304 [elasticsearch] base.py:68 - HEAD
> http://localhost:9200/courseware_index [status:N/A request:0.003s]
> Traceback (most recent call last):
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py",
>
> line 46, in perform_request
> response = self.pool.urlopen(method, url, body, **kw)
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py",
>
> line 607, in urlopen
> release_conn=release_conn, **response_kw)
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py",
>
> line 607, in urlopen
> release_conn=release_conn, **response_kw)
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py",
>
> line 607, in urlopen
> release_conn=release_conn, **response_kw)
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py",
>
> line 591, in urlopen
> raise MaxRetryError(self, url, e)
> MaxRetryError: HTTPConnectionPool(host='localhost', port=9200): Max
> retries exceeded with url: /courseware_index (Caused by <class
> 'socket.error'>: [Errno 111] Connection refused)
> 2016-12-13 09:57:50,483 INFO 3304 [elasticsearch] base.py:70 - > None
> 2016-12-13 09:57:50,484 WARNING 3304 [elasticsearch]
> connection_pool.py:141 - Connection <Urllib3HttpConnection:
> http://localhost:9200> has failed for 3 times in a row, putting on 240
> second timeout.
> 2016-12-13 09:57:50,484 INFO 3304 [elasticsearch] connection_pool.py:185 -
> Resurrecting connection <Urllib3HttpConnection: http://localhost:9200>
> (force=True).
> 2016-12-13 09:57:50,484 INFO 3304 [urllib3.connectionpool]
> connectionpool.py:257 - Starting new HTTP connection (13): localhost
> 2016-12-13 09:57:50,485 WARNING 3304 [urllib3.connectionpool]
> connectionpool.py:603 - Retrying (3 attempts remain) after connection
> broken by 'error(111, 'Connection refused')': /courseware_index
> 2016-12-13 09:57:50,485 INFO 3304 [urllib3.connectionpool]
> connectionpool.py:257 - Starting new HTTP connection (14): localhost
> 2016-12-13 09:57:50,486 WARNING 3304 [urllib3.connectionpool]
> connectionpool.py:603 - Retrying (2 attempts remain) after connection
> broken by 'error(111, 'Connection refused')': /courseware_index
> 2016-12-13 09:57:50,486 INFO 3304 [urllib3.connectionpool]
> connectionpool.py:257 - Starting new HTTP connection (15): localhost
> 2016-12-13 09:57:50,487 WARNING 3304 [urllib3.connectionpool]
> connectionpool.py:603 - Retrying (1 attempts remain) after connection
> broken by 'error(111, 'Connection refused')': /courseware_index
> 2016-12-13 09:57:50,487 INFO 3304 [urllib3.connectionpool]
> connectionpool.py:257 - Starting new HTTP connection (16): localhost
> 2016-12-13 09:57:50,487 WARNING 3304 [elasticsearch] base.py:68 - HEAD
> http://localhost:9200/courseware_index [status:N/A request:0.003s]
> Traceback (most recent call last):
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py",
>
> line 46, in perform_request
> response = self.pool.urlopen(method, url, body, **kw)
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py",
>
> line 607, in urlopen
> release_conn=release_conn, **response_kw)
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py",
>
> line 607, in urlopen
> release_conn=release_conn, **response_kw)
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py",
>
> line 607, in urlopen
> release_conn=release_conn, **response_kw)
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py",
>
> line 591, in urlopen
> raise MaxRetryError(self, url, e)
> MaxRetryError: HTTPConnectionPool(host='localhost', port=9200): Max
> retries exceeded with url: /courseware_index (Caused by <class
> 'socket.error'>: [Errno 111] Connection refused)
> 2016-12-13 09:57:50,490 INFO 3304 [elasticsearch] base.py:70 - > None
> 2016-12-13 09:57:50,490 WARNING 3304 [elasticsearch]
> connection_pool.py:141 - Connection <Urllib3HttpConnection:
> http://localhost:9200> has failed for 4 times in a row, putting on 480
> second timeout.
> 2016-12-13 09:57:50,491 ERROR 3304 [root] reindex_course.py:86 - Search
> Engine error - ConnectionError(HTTPConnectionPool(host='localhost',
> port=9200): Max retries exceeded with url: /courseware_index (Caused by
> <class 'socket.error'>: [Errno 111] Connection refused)) caused by:
> MaxRetryError(HTTPConnectionPool(host='localhost', port=9200): Max retries
> exceeded with url: /courseware_index (Caused by <class 'socket.error'>:
> [Errno 111] Connection refused))
> Traceback (most recent call last):
> File
> "/edx/app/edxapp/edx-platform/cms/djangoapps/contentstore/management/commands/reindex_course.py",
>
> line 84, in handle
> searcher = SearchEngine.get_search_engine(index_name)
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/search/search_engine_base.py",
>
> line 50, in get_search_engine
> return search_engine_class(index=index) if search_engine_class else
> None
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/search/elastic.py",
>
> line 275, in __init__
> if not self._es.indices.exists(index=self.index_name):
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/elasticsearch/client/utils.py",
>
> line 70, in _wrapped
> return func(*args, params=params, **kwargs)
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/elasticsearch/client/indices.py",
>
> line 149, in exists
> self.transport.perform_request('HEAD', _make_path(index),
> params=params)
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/elasticsearch/transport.py",
>
> line 274, in perform_request
> status, headers, data = connection.perform_request(method, url,
> params, body, ignore=ignore)
> File
> "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py",
>
> line 51, in perform_request
> raise ConnectionError('N/A', str(e), e)
> ConnectionError: ConnectionError(HTTPConnectionPool(host='localhost',
> port=9200): Max retries exceeded with url: /courseware_index (Caused by
> <class 'socket.error'>: [Errno 111] Connection refused)) caused by:
> MaxRetryError(HTTPConnectionPool(host='localhost', port=9200): Max retries
> exceeded with url: /courseware_index (Caused by <class 'socket.error'>:
> [Errno 111] Connection refused))
> ---> pavelib.assets.update_assets
> python manage.py lms --settings=devstack preprocess_assets
> lms/static/sass/*.scss lms/static/themed_sass
> Finished preprocessing lms assets.
> xmodule_assets common/static/xmodule
> Finished processing xmodule assets.
> mkdir_p path('common/static/common/js/vendor')
> /bin/cp -rf node_modules/jquery/dist/jquery.js
> common/static/common/js/vendor
> /bin/cp -rf node_modules/jquery-migrate/dist/jquery-migrate.js
> common/static/common/js/vendor
> /bin/cp -rf node_modules/jquery.scrollto/jquery.scrollTo.js
> common/static/common/js/vendor
> /bin/cp -rf node_modules/underscore/underscore.js
> common/static/common/js/vendor
> /bin/cp -rf node_modules/underscore.string/dist/underscore.string.js
> common/static/common/js/vendor
> /bin/cp -rf node_modules/picturefill/dist/picturefill.js
> common/static/common/js/vendor
> /bin/cp -rf node_modules/backbone/backbone.js
> common/static/common/js/vendor
> /bin/cp -rf
> node_modules/edx-ui-toolkit/node_modules/backbone.paginator/lib/backbone.paginator.js
>
> common/static/common/js/vendor
> ---> pavelib.assets.compile_coffeescript
> node_modules/.bin/coffee --compile `find /edx/app/edxapp/edx-platform/lms
> /edx/app/edxapp/edx-platform/cms /edx/app/edxapp/edx-platform/common -type
> f -name "*.coffee"`
> python manage.py lms --settings=devstack compile_sass lms
> Skipping theme asset compilation: enable theming to process themed assets
> ---> pavelib.assets.compile_sass
> Started compiling Sass:
> Finished compiling 'common' sass.
> Started compiling 'lms' Sass for 'system'.
> Finished compiling 'lms' Sass for 'system'.
> Finished compiling Sass:
> >> common/static/sass -> common/static/css in 0:00:00.018948s
> >> lms/static/sass -> lms/static/css in 0:00:50.096556s
> >> lms/static/certificates/sass -> lms/static/certificates/css in
> 0:00:12.936575s
>
> Successful compilations:
> --- "common" sass files.
> --- lms sass for system.
>
> ---> pavelib.assets.watch_assets
> find /edx/app/edxapp/edx-platform/lms /edx/app/edxapp/edx-platform/cms
> /edx/app/edxapp/edx-platform/common -type f -name "*.coffee"
> Starting asset watcher...
> Enter CTL-C to end
> Performing system checks...
>
> System check identified no issues (0 silenced).
> December 13, 2016 - 09:59:19
> Django version 1.8.13, using settings 'lms.envs.devstack'
> Starting development server at http://0.0.0.0:8000/
> Quit the server with CONTROL-C.
>
>
>
>
--
You received this message because you are subscribed to the Google Groups
"General Open edX discussion" group.
To view this discussion on the web visit
https://groups.google.com/d/msgid/edx-code/bf53e675-3998-4f38-a789-6e859642f9a1%40googlegroups.com.
