Re: "too many open files" problems and suggestions on cluster configuration
How to calculate the best amount of shards? пятница, 1 мая 2015 г., 18:21:47 UTC+3 пользователь David Pilato написал: > > Add more nodes or reduce the number of shards per node. > > -- > David ;-) > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 1 mai 2015 à 17:05, Ann Yablunovskaya > > a écrit : > > I am looking for suggestions on cluster configuration. > > I have 2 nodes (master/data and data), 544 indices, about 800 mil > documents. > > If I try to insert more documents and create more indices, I will > catch error "too many open files". > > My node's configuration: > > CentOS 7 > Intel(R) Xeon(R) CPU x16 > RAM 62 Gb > > # ulimit -n > 10 > > In future I will have a lot of indices (about 2000) and a lot of documents > (~5 bil or maybe more) > > How can I avoid the error "too many open files"? > > > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearc...@googlegroups.com . > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/c5d45b95-b3d7-4b6a-80fa-111d66f3f65a%40googlegroups.com > > <https://groups.google.com/d/msgid/elasticsearch/c5d45b95-b3d7-4b6a-80fa-111d66f3f65a%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7c2e1952-e718-4563-ac5c-bb92b45b0aa5%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
"too many open files" problems and suggestions on cluster configuration
I am looking for suggestions on cluster configuration. I have 2 nodes (master/data and data), 544 indices, about 800 mil documents. If I try to insert more documents and create more indices, I will catch error "too many open files". My node's configuration: CentOS 7 Intel(R) Xeon(R) CPU x16 RAM 62 Gb # ulimit -n 10 In future I will have a lot of indices (about 2000) and a lot of documents (~5 bil or maybe more) How can I avoid the error "too many open files"? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c5d45b95-b3d7-4b6a-80fa-111d66f3f65a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout. es_config.pp Description: Binary data
Re: Elasticsearch crashed after start
[2015-04-24 03:38:04,501][INFO ][node ] [server] version[1.5.1], pid[29957], build[5e38401/2015-04-09T13:41:35Z] [2015-04-24 03:38:04,502][INFO ][node ] [server] initializing ... [2015-04-24 03:38:04,536][INFO ][plugins ] [server] loaded [marvel, shield, license], sites [marvel] [2015-04-24 03:38:04,562][ERROR][bootstrap] Exception org.elasticsearch.ElasticsearchIllegalStateException: Failed to created node environment at org.elasticsearch.node.internal.InternalNode.(InternalNode.java:162) at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159) at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:70) at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:213) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32) Caused by: java.nio.file.AccessDeniedException: /opt/elasticsearch/data/server/cluster/nodes/1 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:383) at java.nio.file.Files.createDirectory(Files.java:630) at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734) at java.nio.file.Files.createDirectories(Files.java:720) at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:105) at org.elasticsearch.node.internal.InternalNode.(InternalNode.java:160) ... 4 more I fix this problem, sorry. The reason was in deleting elasticsearch user and missing file privileges. So, I changed file permissions and it works пятница, 24 апреля 2015 г., 5:03:39 UTC+3 пользователь Ann Yablunovskaya написал: > > Hi! > > I don't understand, what happend. > OS CentOS 7.1 > I have ES cluster with two servers. > It have the same configuration. > > I tried to configure shield and marvel but my second ES instanse have > suddenly crashed. > > > [root@server bin]# ./elasticsearch -v > Version: 1.5.1, Build: 5e38401/2015-04-09T13:41:35Z, JVM: 1.7.0_79 > > [root@server bin]# service elasticsearch-server start > Redirecting to /bin/systemctl start elasticsearch-server.service > > [root@server bin]# service elasticsearch-server status > Redirecting to /bin/systemctl status elasticsearch-server.service > elasticsearch-server.service - Starts and stops a single elasticsearch > instance on this system >Loaded: loaded (/usr/lib/systemd/system/elasticsearch-server.service; > enabled) >Active: active (running) since Thu 2015-04-23 21:50:09 EDT; 5s ago > Docs: http://www.elasticsearch.org > Process: 15958 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -d > -p /var/run/elasticsearch/elasticsearch-server.pid > -Des.default.config=$CONF_FILE -Des.default.path.home=$ES_HOME > -Des.default.path.logs=$LOG_DIR -Des.default.path.data=$DATA_DIR > -Des.default.path.work=$WORK_DIR -Des.default.path.conf=$CONF_DIR > (code=exited, status=0/SUCCESS) > Main PID: 15964 (java) >CGroup: /system.slice/elasticsearch-server.service >└─15964 java -Xms30g -Xmx30g -Djava.awt.headless=true > -XX:+UseParNewGC -XX:+UseConcMarkSweepGC > -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiating... > > Apr 23 21:50:08 server.local systemd[1]: PID file > /var/run/elasticsearch/elasticsearch-server.pid not readable (yet?) after > start. > Apr 23 21:50:09 server.local systemd[1]: Started Starts and stops a single > elasticsearch instance on this system. > > [root@server bin]# telnet localhost 9200 > Trying ::1... > telnet: connect to address ::1: Connection refused > Trying 127.0.0.1... > telnet: connect to address 127.0.0.1: Connection refused > > [root@server bin]# service elasticsearch-server status > Redirecting to /bin/systemctl status elasticsearch-server.service > elasticsearch-server.service - Starts and stops a single elasticsearch > instance on this system >Loaded: loaded (/usr/lib/systemd/system/elasticsearch-server.service; > enabled) >Active: failed (Result: exit-code) since Thu 2015-04-23 21:50:15 EDT; > 16s ago > Docs: http://www.elasticsearch.org > Process: 15958 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -d > -p /var/run/elasticsearch/elasticsearch-server.pid > -Des.default.config=$CONF_FILE -Des.default.path.home=$ES_HOME > -Des.default.path.logs=$LOG_DIR -Des.default.path.data=$DATA_DIR > -Des.default.path.work=$WORK_DIR -Des.default.path.conf=$CONF_DIR > (code=exited, status=0/SUCCESS) > Main PID: 15964 (code=exited, status=3) > > Apr 23 21:50:
Elasticsearch crashed after start
Hi! I don't understand, what happend. OS CentOS 7.1 I have ES cluster with two servers. It have the same configuration. I tried to configure shield and marvel but my second ES instanse have suddenly crashed. [root@server bin]# ./elasticsearch -v Version: 1.5.1, Build: 5e38401/2015-04-09T13:41:35Z, JVM: 1.7.0_79 [root@server bin]# service elasticsearch-server start Redirecting to /bin/systemctl start elasticsearch-server.service [root@server bin]# service elasticsearch-server status Redirecting to /bin/systemctl status elasticsearch-server.service elasticsearch-server.service - Starts and stops a single elasticsearch instance on this system Loaded: loaded (/usr/lib/systemd/system/elasticsearch-server.service; enabled) Active: active (running) since Thu 2015-04-23 21:50:09 EDT; 5s ago Docs: http://www.elasticsearch.org Process: 15958 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -d -p /var/run/elasticsearch/elasticsearch-server.pid -Des.default.config=$CONF_FILE -Des.default.path.home=$ES_HOME -Des.default.path.logs=$LOG_DIR -Des.default.path.data=$DATA_DIR -Des.default.path.work=$WORK_DIR -Des.default.path.conf=$CONF_DIR (code=exited, status=0/SUCCESS) Main PID: 15964 (java) CGroup: /system.slice/elasticsearch-server.service └─15964 java -Xms30g -Xmx30g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiating... Apr 23 21:50:08 server.local systemd[1]: PID file /var/run/elasticsearch/elasticsearch-server.pid not readable (yet?) after start. Apr 23 21:50:09 server.local systemd[1]: Started Starts and stops a single elasticsearch instance on this system. [root@server bin]# telnet localhost 9200 Trying ::1... telnet: connect to address ::1: Connection refused Trying 127.0.0.1... telnet: connect to address 127.0.0.1: Connection refused [root@server bin]# service elasticsearch-server status Redirecting to /bin/systemctl status elasticsearch-server.service elasticsearch-server.service - Starts and stops a single elasticsearch instance on this system Loaded: loaded (/usr/lib/systemd/system/elasticsearch-server.service; enabled) Active: failed (Result: exit-code) since Thu 2015-04-23 21:50:15 EDT; 16s ago Docs: http://www.elasticsearch.org Process: 15958 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -d -p /var/run/elasticsearch/elasticsearch-server.pid -Des.default.config=$CONF_FILE -Des.default.path.home=$ES_HOME -Des.default.path.logs=$LOG_DIR -Des.default.path.data=$DATA_DIR -Des.default.path.work=$WORK_DIR -Des.default.path.conf=$CONF_DIR (code=exited, status=0/SUCCESS) Main PID: 15964 (code=exited, status=3) Apr 23 21:50:08 server.local systemd[1]: PID file /var/run/elasticsearch/elasticsearch-server.pid not readable (yet?) after start. Apr 23 21:50:09 server.local.local systemd[1]: Started Starts and stops a single elasticsearch instance on this system. Apr 23 21:50:15 server.local.local systemd[1]: elasticsearch-server.service: main process exited, code=exited, status=3/NOTIMPLEMENTED Apr 23 21:50:15 server.local.local systemd[1]: Unit elasticsearch-server.service entered failed state. I will be really appreciate any help! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b8029de2-a170-49d2-81fd-9f8ec963d1b9%40googlegroups.com. For more options, visit https://groups.google.com/d/optout. es.yml Description: Binary data
