Index corrupted.
Dear All, We configured path.data to point to 12 locations. One of the disc got issue.. So we removed that from path.data list. Now indexes are not starting. Please suggest me how to recover from this problem.. below is what I am continuously getting from log files. Exception: gb_es090_0.tip, _1k2ow.fdx, _2rd5_es090_0.tip, _948t.si, _118sb.nvd, _1t6sf.nvm] at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:114) ... 4 more Caused by: java.io.FileNotFoundException: segments_5p8p at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:514) at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:114) at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:329) at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:416) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:864) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:710) at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:412) at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:121) at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:105) ... 4 more [2015-03-03 13:29:16,297][WARN ][cluster.action.shard ] [172.18.182.181] [fw6][10] sending failed shard for [fw6][10], node[WkI61Si4QZ2wB1vqvtTdfQ], [P], s[INITIALIZING], indexUUID [mmmWL-8WSjOrAOcuh7rhRg], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[fw6][10] failed to fetch index version after copying it over]; nested: IndexShardGateway Thanks Prasath Rajan -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/259f5a81-6184-4bb3-b418-39a01fe8bf66%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
client Details
Hi All, Is there any way to list the client ips those all connected to cluster and can list what query they are running? Thanks & Regards Prasath Rajan -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/35fe2233-ada2-43ab-bdaa-8193b7fa72cb%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Disk Decommission
Dear Mark, Thanks for your answers... So.. The only way I have is deleting old docs... Thanks Prasanth Rajan On Mar 1, 2015 1:04 PM, "Mark Walkom" wrote: > It doesn't work like that though :) > > On 1 March 2015 at 17:30, Prasanth R wrote: > >> Dear Mark, >> >> Each node roughly have more than One TB data.. So moving to another node >> is not easy.. If elasticsearch stop indexing on disks those are reaching >> certain percentage will be more useful.. >> >> Thanks >> Prasanth Rajan >> On Mar 1, 2015 11:53 AM, "Mark Walkom" wrote: >> >>> Are the disks not the same size? >>> >>> You will need to make sure the shards that this node has are on other >>> nodes, or move shards to other nodes, then replace the disk. >>> >>> On 1 March 2015 at 16:32, Prasanth R wrote: >>> >>>> Dear Mark, >>>> >>>> Our path.data pointing to different directories and each is different >>>> hard disk.. One of the hard disc reached more than 90%.. Please suggest me >>>> what to do.. >>>> >>>> Thanks >>>> Prasanth Rajan >>>> On Mar 1, 2015 9:42 AM, "Mark Walkom" wrote: >>>> >>>>> If you are using multiple data paths and you replace a disk, you will >>>>> lose all data on that node. >>>>> >>>>> "Data can be saved to multiple directories, and if each directory is >>>>> mounted on a different hard drive, this is a simple and effective way to >>>>> set up a software RAID 0. Elasticsearch will automatically stripe data >>>>> between the different directories, boosting performance" >>>>> >>>>> http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_important_configuration_changes.html#_paths >>>>> >>>>> On 27 February 2015 at 21:48, Prasanth R >>>>> wrote: >>>>> >>>>>> Dear All, >>>>>> >>>>>> Is it possible to decommission a disc from nodes? >>>>>> is it possible to stop indexing/storing data on particular disc?. >>>>>> >>>>>> Thanks >>>>>> Prasanth Rajan >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "elasticsearch" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to elasticsearch+unsubscr...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/elasticsearch/04017b82-a798-4258-bb94-079e7b2c6267%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/elasticsearch/04017b82-a798-4258-bb94-079e7b2c6267%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "elasticsearch" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/elasticsearch/RY8OanGRV_c/unsubscribe >>>>> . >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> elasticsearch+unsubscr...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-Jyn8bN4HwiEiS2PsY7E5c9U_06sF6EPSMAsAakhNHOA%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-Jyn8bN4HwiEiS2PsY7E5c9U_06sF6EPSMAsAakhNHOA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to elasticsearch+unsubscr...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/CAJLGCR2GFiFMowYUtEc35g_PFcRLDAqV_M27gx7GJN3BMBeQcg%40mail.gmail.com >>>> <https://g
Re: Disk Decommission
Dear Mark, Each node roughly have more than One TB data.. So moving to another node is not easy.. If elasticsearch stop indexing on disks those are reaching certain percentage will be more useful.. Thanks Prasanth Rajan On Mar 1, 2015 11:53 AM, "Mark Walkom" wrote: > Are the disks not the same size? > > You will need to make sure the shards that this node has are on other > nodes, or move shards to other nodes, then replace the disk. > > On 1 March 2015 at 16:32, Prasanth R wrote: > >> Dear Mark, >> >> Our path.data pointing to different directories and each is different >> hard disk.. One of the hard disc reached more than 90%.. Please suggest me >> what to do.. >> >> Thanks >> Prasanth Rajan >> On Mar 1, 2015 9:42 AM, "Mark Walkom" wrote: >> >>> If you are using multiple data paths and you replace a disk, you will >>> lose all data on that node. >>> >>> "Data can be saved to multiple directories, and if each directory is >>> mounted on a different hard drive, this is a simple and effective way to >>> set up a software RAID 0. Elasticsearch will automatically stripe data >>> between the different directories, boosting performance" >>> >>> http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_important_configuration_changes.html#_paths >>> >>> On 27 February 2015 at 21:48, Prasanth R >>> wrote: >>> >>>> Dear All, >>>> >>>> Is it possible to decommission a disc from nodes? >>>> is it possible to stop indexing/storing data on particular disc?. >>>> >>>> Thanks >>>> Prasanth Rajan >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to elasticsearch+unsubscr...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/04017b82-a798-4258-bb94-079e7b2c6267%40googlegroups.com >>>> <https://groups.google.com/d/msgid/elasticsearch/04017b82-a798-4258-bb94-079e7b2c6267%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "elasticsearch" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/elasticsearch/RY8OanGRV_c/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> elasticsearch+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-Jyn8bN4HwiEiS2PsY7E5c9U_06sF6EPSMAsAakhNHOA%40mail.gmail.com >>> <https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-Jyn8bN4HwiEiS2PsY7E5c9U_06sF6EPSMAsAakhNHOA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearch+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAJLGCR2GFiFMowYUtEc35g_PFcRLDAqV_M27gx7GJN3BMBeQcg%40mail.gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/CAJLGCR2GFiFMowYUtEc35g_PFcRLDAqV_M27gx7GJN3BMBeQcg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/RY8OanGRV_c/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-RZJWO8ewpR1tQoSbChD5a0knBV4d8Je1w-WM%3DUT9t3g%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-RZJWO8ewpR1tQoSbChD5a0knBV4d8Je1w-WM%3DUT9t3g%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJLGCR2s45-G8%2BROAKt75hbZy_ZYdsP9d5ucMULjj%3DXNbEQydA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Disk Decommission
Dear Mark, Our path.data pointing to different directories and each is different hard disk.. One of the hard disc reached more than 90%.. Please suggest me what to do.. Thanks Prasanth Rajan On Mar 1, 2015 9:42 AM, "Mark Walkom" wrote: > If you are using multiple data paths and you replace a disk, you will lose > all data on that node. > > "Data can be saved to multiple directories, and if each directory is > mounted on a different hard drive, this is a simple and effective way to > set up a software RAID 0. Elasticsearch will automatically stripe data > between the different directories, boosting performance" > > http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_important_configuration_changes.html#_paths > > On 27 February 2015 at 21:48, Prasanth R > wrote: > >> Dear All, >> >> Is it possible to decommission a disc from nodes? >> is it possible to stop indexing/storing data on particular disc?. >> >> Thanks >> Prasanth Rajan >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearch+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/04017b82-a798-4258-bb94-079e7b2c6267%40googlegroups.com >> <https://groups.google.com/d/msgid/elasticsearch/04017b82-a798-4258-bb94-079e7b2c6267%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/RY8OanGRV_c/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-Jyn8bN4HwiEiS2PsY7E5c9U_06sF6EPSMAsAakhNHOA%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-Jyn8bN4HwiEiS2PsY7E5c9U_06sF6EPSMAsAakhNHOA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJLGCR2GFiFMowYUtEc35g_PFcRLDAqV_M27gx7GJN3BMBeQcg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Disk Decommission
Dear All, Is it possible to decommission a disc from nodes? is it possible to stop indexing/storing data on particular disc?. Thanks Prasanth Rajan -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/04017b82-a798-4258-bb94-079e7b2c6267%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Disabling Scoring feature.
Dear All, We are dealing with logs and do not bother about scoring. is there any way to disable scoring entirely. Thanks Prasath Rajan -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5e06f79b-c6c4-44d8-aab4-3d5ba671657d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Disk awarnes on Indexing.
Thanks for the suggestion... Let me try this... On Feb 22, 2015 2:02 AM, "Mark Walkom" wrote: > Reduce your shard size and increase the count, add more nodes, reduce your > heap to 31GB. > > On 22 February 2015 at 00:55, Prasanth R > wrote: > >> We have 10 Node cluster and each with 32 GB Ram allocated.. 9 indexes.. >> 50 shards...25TB data... Queries are roughly taking 1 to 2 minutes...facing >> frequent node downs... >> >> Any suggestion to improve? >> On Feb 21, 2015 2:27 AM, "Mark Walkom" wrote: >> >>> You have 600GB of data to move if you want to reallocate or recover, you >>> have 600GB of data to query, which means you have a massive amount of data >>> to load into memory to do that. It's a big risk. >>> >>> On 21 February 2015 at 06:35, Nikolas Everett wrote: >>> >>>> I have 30GB shards and the biggest problem I have is that they take a >>>> long time to replicate to other machines. I believe there are memory >>>> issues for very large shards as well but I don't know them that well. >>>> >>>> Nik >>>> On Feb 20, 2015 7:31 PM, "Prasanth R" >>>> wrote: >>>> >>>>> Could you please elaborate about the problem? >>>>> On Feb 20, 2015 4:13 PM, "Prasanth R" >>>>> wrote: >>>>> >>>>>> Problem? >>>>>> On Feb 20, 2015 3:07 PM, "Mark Walkom" wrote: >>>>>> >>>>>>> You can change the watermark levels for them, but that is about it. >>>>>>> >>>>>>> Also if you have 600GB shards then you have a problem. >>>>>>> >>>>>>> On 20 February 2015 at 18:24, Prasanth R >>>>>> > wrote: >>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>>Few of our nodes filled disks up to 90%. Is there any option to >>>>>>>> avoid indexing on those disks instead of relocating shards. Because >>>>>>>> each >>>>>>>> shard contains roughly 600G data, so relocating is costly as ours is >>>>>>>> very >>>>>>>> busy cluster. >>>>>>>> >>>>>>>> Thanks >>>>>>>> Prasath Rajan >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "elasticsearch" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to elasticsearch+unsubscr...@googlegroups.com. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com >>>>>>>> <https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to a topic in >>>>>>> the Google Groups "elasticsearch" group. >>>>>>> To unsubscribe from this topic, visit >>>>>>> https://groups.google.com/d/topic/elasticsearch/oLZBiQIUsM8/unsubscribe >>>>>>> . >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> elasticsearch+unsubscr...@googlegroups.com. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_w8kimuc8Od6YJ_HHDKAbyBrbTu7j7p3AfmD7tOtKiCw%40mail.gmail.com >>>>>>> <https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_w8kimuc8Od6YJ_HHDKAbyBrbTu7j7p3AfmD7tOtKiCw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "elasticsearch" group
Re: Disk awarnes on Indexing.
We have 10 Node cluster and each with 32 GB Ram allocated.. 9 indexes.. 50 shards...25TB data... Queries are roughly taking 1 to 2 minutes...facing frequent node downs... Any suggestion to improve? On Feb 21, 2015 2:27 AM, "Mark Walkom" wrote: > You have 600GB of data to move if you want to reallocate or recover, you > have 600GB of data to query, which means you have a massive amount of data > to load into memory to do that. It's a big risk. > > On 21 February 2015 at 06:35, Nikolas Everett wrote: > >> I have 30GB shards and the biggest problem I have is that they take a >> long time to replicate to other machines. I believe there are memory >> issues for very large shards as well but I don't know them that well. >> >> Nik >> On Feb 20, 2015 7:31 PM, "Prasanth R" wrote: >> >>> Could you please elaborate about the problem? >>> On Feb 20, 2015 4:13 PM, "Prasanth R" >>> wrote: >>> >>>> Problem? >>>> On Feb 20, 2015 3:07 PM, "Mark Walkom" wrote: >>>> >>>>> You can change the watermark levels for them, but that is about it. >>>>> >>>>> Also if you have 600GB shards then you have a problem. >>>>> >>>>> On 20 February 2015 at 18:24, Prasanth R >>>>> wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>>Few of our nodes filled disks up to 90%. Is there any option to >>>>>> avoid indexing on those disks instead of relocating shards. Because each >>>>>> shard contains roughly 600G data, so relocating is costly as ours is very >>>>>> busy cluster. >>>>>> >>>>>> Thanks >>>>>> Prasath Rajan >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "elasticsearch" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to elasticsearch+unsubscr...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "elasticsearch" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/elasticsearch/oLZBiQIUsM8/unsubscribe >>>>> . >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> elasticsearch+unsubscr...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_w8kimuc8Od6YJ_HHDKAbyBrbTu7j7p3AfmD7tOtKiCw%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_w8kimuc8Od6YJ_HHDKAbyBrbTu7j7p3AfmD7tOtKiCw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to elasticsearch+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAJLGCR2wPhxw0ps_%2B-hJzpH%2B9Bt94AM2k4wkS8-EkzCYNyvjZQ%40mail.gmail.com >>> <https://groups.google.com/d/msgid/elasticsearch/CAJLGCR2wPhxw0ps_%2B-hJzpH%2B9Bt94AM2k4wkS8-EkzCYNyvjZQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearch+unsubscr...@googlegroups.com. >> To view this discussion on the
Re: Disk awarnes on Indexing.
Could you please elaborate about the problem? On Feb 20, 2015 4:13 PM, "Prasanth R" wrote: > Problem? > On Feb 20, 2015 3:07 PM, "Mark Walkom" wrote: > >> You can change the watermark levels for them, but that is about it. >> >> Also if you have 600GB shards then you have a problem. >> >> On 20 February 2015 at 18:24, Prasanth R >> wrote: >> >>> Hi All, >>> >>>Few of our nodes filled disks up to 90%. Is there any option to avoid >>> indexing on those disks instead of relocating shards. Because each shard >>> contains roughly 600G data, so relocating is costly as ours is very busy >>> cluster. >>> >>> Thanks >>> Prasath Rajan >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to elasticsearch+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com >>> <https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "elasticsearch" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/elasticsearch/oLZBiQIUsM8/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> elasticsearch+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_w8kimuc8Od6YJ_HHDKAbyBrbTu7j7p3AfmD7tOtKiCw%40mail.gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_w8kimuc8Od6YJ_HHDKAbyBrbTu7j7p3AfmD7tOtKiCw%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJLGCR2wPhxw0ps_%2B-hJzpH%2B9Bt94AM2k4wkS8-EkzCYNyvjZQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Disk awarnes on Indexing.
Problem? On Feb 20, 2015 3:07 PM, "Mark Walkom" wrote: > You can change the watermark levels for them, but that is about it. > > Also if you have 600GB shards then you have a problem. > > On 20 February 2015 at 18:24, Prasanth R > wrote: > >> Hi All, >> >>Few of our nodes filled disks up to 90%. Is there any option to avoid >> indexing on those disks instead of relocating shards. Because each shard >> contains roughly 600G data, so relocating is costly as ours is very busy >> cluster. >> >> Thanks >> Prasath Rajan >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearch+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com >> <https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/oLZBiQIUsM8/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_w8kimuc8Od6YJ_HHDKAbyBrbTu7j7p3AfmD7tOtKiCw%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_w8kimuc8Od6YJ_HHDKAbyBrbTu7j7p3AfmD7tOtKiCw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJLGCR3fqG_SKZr7PK%3DT%2Bs1%2B-dqtBWDb7D%2B2VNFJ7dFxVjm0UQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Disk awarnes on Indexing.
Hi All, Few of our nodes filled disks up to 90%. Is there any option to avoid indexing on those disks instead of relocating shards. Because each shard contains roughly 600G data, so relocating is costly as ours is very busy cluster. Thanks Prasath Rajan -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f0eeb39f-de10-4db1-9db7-838c8e4d21c3%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: ElasticSearch- IndexReaders cannot exceed 2147483647
Dear All, Thanks for your replies. Conclusion is, we can not store more than 2147483647 records per shard as of now. The only option is we need to increase the shard count. Thanks Prasath Rajan On Tuesday, October 14, 2014 9:34:33 PM UTC+5:30, Jörg Prante wrote: > > You can not store more than 2G docs per shard in "Lucene 4.x" codecs. This > is a documented Lucene limit: > > "Similarly, Lucene uses a Java int to refer to document numbers, and the > index file format uses an Int32 on-disk to store document numbers. This is > a limitation of both the index file format and the current implementation. > Eventually these should be replaced with either UInt64 values, or better > yet, VInt values which have no limit." > > > https://lucene.apache.org/core/4_9_1/core/org/apache/lucene/codecs/lucene49/package-summary.html#Limitations > > Jörg > > > On Tue, Oct 14, 2014 at 5:37 PM, Prasanth R > wrote: > >> Thanks for the reply. >> >> My scenario here is, >> 1) No nested docs. >> 2) I don't have any limit per shard.. >> >> I didn't know about internal limit of ES. >> On Oct 14, 2014 8:23 PM, "Alexandre Rafalovitch" > > wrote: >> >>> On 14 October 2014 10:33, Prasanth R >> > wrote: >>> > There is no upper limit... >>> >>> Well, then you must have an infinitely scalable architecture and a >>> decision when the content starts getting shared. So, then the question >>> is what is your individual shard allowed to grow to. Which is "how >>> many documents - including nested - you are expecting to have in a >>> single shard". >>> >>> Because, ElasticSearch has an internal limit and you just hit it. So, >>> the question is whether it is intentional, unintentional or a result >>> of a bug. >>> >>> Regards, >>>Alex. >>> >>> Personal: http://www.outerthoughts.com/ and @arafalov >>> Solr resources and newsletter: http://www.solr-start.com/ and @solrstart >>> Solr popularizers community: https://www.linkedin.com/groups?gid=6713853 >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "elasticsearch" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/elasticsearch/H74KAYmGtoc/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> elasticsearc...@googlegroups.com . >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G8GsLZpeO9-b5_RGs%3D-tGBNp39QgeY2rywjRnOQfcfnw%40mail.gmail.com >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearc...@googlegroups.com . >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAJLGCR1LRDnv9Nw71py%3DPFksSqLnXOG6x6-K0kUW1j26%3D6_pYA%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/elasticsearch/CAJLGCR1LRDnv9Nw71py%3DPFksSqLnXOG6x6-K0kUW1j26%3D6_pYA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d85eb58f-2c9d-4fba-88ca-55b64a74be3a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: ElasticSearch- IndexReaders cannot exceed 2147483647
Thanks for the reply. My scenario here is, 1) No nested docs. 2) I don't have any limit per shard.. I didn't know about internal limit of ES. On Oct 14, 2014 8:23 PM, "Alexandre Rafalovitch" wrote: > On 14 October 2014 10:33, Prasanth R wrote: > > There is no upper limit... > > Well, then you must have an infinitely scalable architecture and a > decision when the content starts getting shared. So, then the question > is what is your individual shard allowed to grow to. Which is "how > many documents - including nested - you are expecting to have in a > single shard". > > Because, ElasticSearch has an internal limit and you just hit it. So, > the question is whether it is intentional, unintentional or a result > of a bug. > > Regards, >Alex. > > Personal: http://www.outerthoughts.com/ and @arafalov > Solr resources and newsletter: http://www.solr-start.com/ and @solrstart > Solr popularizers community: https://www.linkedin.com/groups?gid=6713853 > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/H74KAYmGtoc/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G8GsLZpeO9-b5_RGs%3D-tGBNp39QgeY2rywjRnOQfcfnw%40mail.gmail.com > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJLGCR1LRDnv9Nw71py%3DPFksSqLnXOG6x6-K0kUW1j26%3D6_pYA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: ElasticSearch- IndexReaders cannot exceed 2147483647
There is no upper limit... On Tuesday, October 14, 2014 8:02:03 PM UTC+5:30, Alexandre Rafalovitch wrote: > > Obvious question: how many documents are you expecting to have in > there? Including the nested ones if you are storing them as separate > documents. > > Regards, >Alex. > Personal: http://www.outerthoughts.com/ and @arafalov > Solr resources and newsletter: http://www.solr-start.com/ and @solrstart > Solr popularizers community: https://www.linkedin.com/groups?gid=6713853 > > > On 14 October 2014 10:25, Prasanth R > > wrote: > > Getting following exception in one of our ElasticSearch Index, > > > > [Failed to start shard, message > [IndexShardGatewayRecoveryException[[fw][4] > > failed recovery]; nested: IllegalArgumentException[Too many documents, > > composite IndexReaders cannot exceed 2147483647]; ]] > > > > Please somebody help. > > > > Thanks > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to elasticsearc...@googlegroups.com . > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/elasticsearch/f60845f2-476d-489a-936f-291884986dbf%40googlegroups.com. > > > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3646a7e4-7246-43df-8970-01b776a0720f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
ElasticSearch- IndexReaders cannot exceed 2147483647
Getting following exception in one of our ElasticSearch Index, [Failed to start shard, message [IndexShardGatewayRecoveryException[[fw][4] failed recovery]; nested: IllegalArgumentException[Too many documents, composite IndexReaders cannot exceed 2147483647]; ]] Please somebody help. Thanks -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f60845f2-476d-489a-936f-291884986dbf%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: How to index a .JSON file
Hi david,
My json file contains data like this
{ "field1" : "value1" }
{ "field1" : "value2" }
{ "field1" : "value3" }
Is it possible to index with fsriver?.
My river/index creation as below,
curl -XPUT 'localhost:9200/_river/security/_meta' -d '{
"type":"fs",
"fs": {
"url": "/Projects/elasticsearch-1.0.1/JSON/",
"update_rate": 1000,
"json_support" : "true"
},
"index":{
"index":"security",
"type":"logs",
"bulk_size":50
}
}
I am keep getting error saying that error 'Error while indexing content
from /Projects/elasticsearch-1.0.1/JSON/'
Could you please help me out?
Thanks
Prasanth.R
On Thursday, November 21, 2013 11:01:40 PM UTC+5:30, David Pilato wrote:
>
> Did you add a return after the last body?
>
> --
> *David Pilato* | *Technical Advocate* | *Elasticsearch.com*
> @dadoonet |
> @elasticsearchfr
>
>
> Le 21 novembre 2013 at 15:14:15, cml (funny1...@gmail.com )
> a écrit:
>
> Hi Rafal,
> what I do is
> {"index":{"_index":"test","_type":"keywords","_id":"1"}}
> {"keywords":"red"}
>
> and i post as:
> curl -s -XPOST 'http://192.168.213.158:9200/_bulk' --data-binary
> @keywords.json
> Get an error:
> {"error":"ActionRequestValidationException[Validation Failed: 1: no
> requests added;]","status":500}
>
> any idea?
>
>
>
>
>
>
> 在 2013年4月10日星期三UTC+2下午11时49分56秒，Rafał Kuć写道：
>>
>> Hello!
>>
>> In order to send two documents in the same JSON file you need to use
>> ElasticSearch Bulk API. You can find more information about it here:
>> http://www.elasticsearch.org/guide/reference/api/bulk/
>>
>> Basically your document would have to look like this:
>>
>> {"index":{"_index":"test","_type":"test","_id":"1"}}
>> {"lane":"M05","routes":160}
>> {"index":{"_index":"test","_type":"test","_id":"2"}}
>> {"lane":"M04","routes":170}
>>
>> If the above content would be stored in a file called bulk.json, you can
>> send it using the following command:
>>
>> curl -s -XPOST 'http://jfblouvmlxecs01:9200/_bulk' --data-binary
>> @bulk.json
>>
>> --
>> Regards,
>> Rafał Kuć
>> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch -
>> ElasticSearch
>>
>>
>> Thanks. I've tried adding the @ and that works for a very simple JSON
>> document. This is the one I used:
>> {"lane":"M05","routes":160}
>>
>> I also tried adding this JSON doc with 2 rows (2 records) like this,
>> however it put both records under the same id:
>>
>> {"lane":"M05","routes":160},{"lane":"M04","routes":170}
>>
>> How can I do this when I have multiple records in the same .json file?
>>
>> On Wednesday, April 10, 2013 5:03:43 PM UTC-4, Rafał Kuć wrote:
>> Hello!
>>
>> You can find more information about indexing data here:
>> http://www.elasticsearch.org/guide/reference/api/index_/
>>
>> One thing I've noticed in your command is that you lack the @ character
>> before the file name. Your command should look like this:
>>
>> curl -XPOST 'http://jfblouvmlxecs01:9200/test/test/1' -d @lane.json
>>
>> instead of the one you've pasted.
>>
>> --
>> Regards,
>> Rafał Kuć
>> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch -
>> ElasticSearch
>>
>>
>> I'm completely new to Elasticsearch and I've been importing data to
>> Elasticsearch up to this point by manually entering the JSON.
>>
>> I'd like to begin loading in .json files to make things faster and
>> possibly to bulk load in the future. I'm not sure how to do this
>> currently. The .json would be generated from a SQL Server query and
>> formatted as JSON by another layer, and then placed in a folder on the
>> server.
>>
>> I've attempted to run this code to index the JSON, however I get an error
>> message:
>>
>> [root@JFBLOUVMLXECS01 ~]# curl -XPOST '
>> http://jfblouvmlxecs01:9200/test/test/1' -d lane.json
>> {"error":"ElasticSearchParseException[Failed to derive xcontent from
>> (offset=0, length=9): [108, 97, 110, 101, 46, 106, 115, 111,
>> 110]]","status":400}[root@JFBLOUVMLXECS01 ~]#
>>
>> Any help you can provide would be much appreciated. Please keep in mind
>> I'm a total noob with this.
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearc...@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearc...@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearc...@googlegroups.com .
> For more options, visit h
