Newbie question about Spark and Elasticsearch
am trying to understand how spark and ES work... could someone please help me answer this question.. val conf = new Configuration() conf.set(es.resource, radio/artists) conf.set(es.query, ?q=me*) val esRDD = sc.newHadoopRDD(conf, classOf[EsInputFormat[Text, MapWritable]], classOf[Text], classOf[MapWritable])) val docCount = esRDD.count(); When and where is data being transferred from ES? is it all collected on the Spark master node, then partitioned and sent to the worker nodes? or is each worker node talking to ES to somehow get a partition of the data? How does this effectively work? Thanks a lot, Mohamed. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEU_gmf9Nt0xn_0NbzDn_moRWUT96uWYf4cicJdZik3r0Zz8XA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Newbie question about Spark and Elasticsearch
Great Thanks a lot Costin. Are people supposed to deploy the Spark workers on the same ES cluster? I guess it would make sense for data to remain local and avoid network transfers altogether? Thanks a lot, Mohamed. On Monday, December 8, 2014 10:19:12 AM UTC-5, Costin Leau wrote: Hi, First off I recommend using the native integration (aka the Java/Scala APIs) instead of MapReduce. The latter works but the former is better performing and more flexible. ES works in a similar fashion to the HDFS store - the data doesn't go through the master rather, each task has its own partition on works on its own set of data. Behind the scenes we map each worker to an index shard (if there aren't enough workers, then some will work across multiple shards). On 12/8/14 4:59 PM, Mohamed Lrhazi wrote: am trying to understand how spark and ES work... could someone please help me answer this question.. val conf = new Configuration() conf.set(es.resource, radio/artists) conf.set(es.query, ?q=me*) val esRDD = sc.newHadoopRDD(conf, classOf[EsInputFormat[Text, MapWritable]], classOf[Text], classOf[MapWritable])) val docCount = esRDD.count(); When and where is data being transferred from ES? is it all collected on the Spark master node, then partitioned and sent to the worker nodes? or is each worker node talking to ES to somehow get a partition of the data? How does this effectively work? Thanks a lot, Mohamed. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com javascript: mailto: elasticsearch+unsubscr...@googlegroups.com javascript:. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEU_gmf9Nt0xn_0NbzDn_moRWUT96uWYf4cicJdZik3r0Zz8XA%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAEU_gmf9Nt0xn_0NbzDn_moRWUT96uWYf4cicJdZik3r0Zz8XA%40mail.gmail.com?utm_medium=emailutm_source=footer. For more options, visit https://groups.google.com/d/optout. -- Costin -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/26361977-b5e1-45fa-b305-e59310e2ce3f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Failed to start shard
Sorry dont recall! but I find this in an email sent on the same thread think: For me the change that seemed to have worked, I keep my fingers crossed, is to add node.master=False to all nodes except one... black magic! Other things we did involve: Adding more nodes and deleting indexes with zero documents. HTH, Mohamed. On Mon, Oct 27, 2014 at 8:12 AM, Anirudh Perugu anir...@unbxd.com wrote: Hi Mohamed, Did you find a solution to this problem? On Wednesday, July 23, 2014 10:06:16 AM UTC+5:30, Mohamed Lrhazi wrote: I went ahead and upgraded to elasticsearch-1.2.2-1.noarch, but the same errors continues after cluster start. I notice that the node inquestion is the master, and is supposed to be a non data node. it should not be starting any shards, right? Thanks a lot, Mohamed. On Tue, Jul 22, 2014 at 11:55 PM, Mohamed Lrhazi ml...@georgetown.edu wrote: Hello, Our master is constantly logging this error, about twice per second I saw similar error mentioned in the release notes for a previous verion, this cluster is running 1.0.1. Is it the same issue? What work arounds do we have? [2014-07-22 23:50:28,533][WARN ][cluster.action.shard ] [rap-es2] [de_2007][0] received shard failed for [de_2007][0], node[EtDIFKT7Qi-nAuvG47tGYQ], [P], s[INITIALIZING], indexUUID [yLrHy5k7RqO6rLnl_b4kww], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[de_2007][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[de_2007][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/data/elasticsearch/data/Odine2/nodes/0/indices/de_2007/0/index), type=MERGE, rate=20.0)]): files: []]; ]] -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/ topic/elasticsearch/OVc_0Kqm2yk/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearc...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/ msgid/elasticsearch/fd9dfaef-7179-4abf-bb60-b8a93982d76a% 40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/fd9dfaef-7179-4abf-bb60-b8a93982d76a%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/OVc_0Kqm2yk/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52f5a6c8-847c-4294-a2eb-308eb9aa3fe7%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/52f5a6c8-847c-4294-a2eb-308eb9aa3fe7%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEU_gmdtPD8Cxg%2BJmqJhtZRgMv%3DQ8SNo7qds1GT%2BOdegkGKQ-w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Unstable cluster - suspect illegal state: trying to move shard from primary mode to replica mode
Thanks Robin. For me the change that seemed to have worked, I keep my fingers crossed, is to add node.master=False to all nodes except one... black magic! Thanks, Mohamed. On Sun, Jul 27, 2014 at 7:35 AM, Robin Clarke ro...@robinclarke.net wrote: I adjusted the required master nodes to N-1, where N was the total number of master nodes I have. On 23 July 2014 15:36, Mohamed Lrhazi ml...@georgetown.edu wrote: I think am running into this same issue, even after upgrading to 1.2.2. Did you stabilize your cluster? Thanks, Mohamed. On Saturday, May 24, 2014 5:05:55 AM UTC-4, Robin Clarke wrote: And found this error too in one of the nodes which left the cluster: java.lang.NullPointerException at org.elasticsearch.gateway.local.state.meta. LocalGatewayMetaState.clusterChanged(LocalGatewayMetaState.java:185) at org.elasticsearch.gateway.local.LocalGateway. clusterChanged(LocalGateway.java:207) at org.elasticsearch.cluster.service.InternalClusterService$ UpdateTask.run(InternalClusterService.java:431) at org.elasticsearch.common.util.concurrent. PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run( PrioritizedEsThreadPoolExecutor.java:134) at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) -Robin- -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/9XP5MwOkgk0/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f7d09651-0f97-4529-b4e3-4cee752539e0%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/f7d09651-0f97-4529-b4e3-4cee752539e0%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- Best winds, -Robin- ~:) -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/9XP5MwOkgk0/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CACX78vZyV19goRnBv4RR%3Dm_z%3DAFoFaxU67T0ndzWJWr%3Dru6Jbg%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CACX78vZyV19goRnBv4RR%3Dm_z%3DAFoFaxU67T0ndzWJWr%3Dru6Jbg%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEU_gmfX8hMRD-BpvMAzhGoATmBTULjKA3TE7yAzGFsvs8j2bw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Unstable cluster - suspect illegal state: trying to move shard from primary mode to replica mode
I think am running into this same issue, even after upgrading to 1.2.2. Did you stabilize your cluster? Thanks, Mohamed. On Saturday, May 24, 2014 5:05:55 AM UTC-4, Robin Clarke wrote: And found this error too in one of the nodes which left the cluster: java.lang.NullPointerException at org.elasticsearch.gateway.local.state.meta.LocalGatewayMetaState.clusterChanged(LocalGatewayMetaState.java:185) at org.elasticsearch.gateway.local.LocalGateway.clusterChanged(LocalGateway.java:207) at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:431) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:134) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) -Robin- -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f7d09651-0f97-4529-b4e3-4cee752539e0%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Failed to start shard
Hello, Our master is constantly logging this error, about twice per second I saw similar error mentioned in the release notes for a previous verion, this cluster is running 1.0.1. Is it the same issue? What work arounds do we have? [2014-07-22 23:50:28,533][WARN ][cluster.action.shard ] [rap-es2] [de_2007][0] received shard failed for [de_2007][0], node[EtDIFKT7Qi-nAuvG47tGYQ], [P], s[INITIALIZING], indexUUID [yLrHy5k7RqO6rLnl_b4kww], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[de_2007][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[de_2007][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/data/elasticsearch/data/Odine2/nodes/0/indices/de_2007/0/index), type=MERGE, rate=20.0)]): files: []]; ]] -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fd9dfaef-7179-4abf-bb60-b8a93982d76a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Failed to start shard
I went ahead and upgraded to elasticsearch-1.2.2-1.noarch, but the same errors continues after cluster start. I notice that the node inquestion is the master, and is supposed to be a non data node. it should not be starting any shards, right? Thanks a lot, Mohamed. On Tue, Jul 22, 2014 at 11:55 PM, Mohamed Lrhazi ml...@georgetown.edu wrote: Hello, Our master is constantly logging this error, about twice per second I saw similar error mentioned in the release notes for a previous verion, this cluster is running 1.0.1. Is it the same issue? What work arounds do we have? [2014-07-22 23:50:28,533][WARN ][cluster.action.shard ] [rap-es2] [de_2007][0] received shard failed for [de_2007][0], node[EtDIFKT7Qi-nAuvG47tGYQ], [P], s[INITIALIZING], indexUUID [yLrHy5k7RqO6rLnl_b4kww], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[de_2007][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[de_2007][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/data/elasticsearch/data/Odine2/nodes/0/indices/de_2007/0/index), type=MERGE, rate=20.0)]): files: []]; ]] -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/OVc_0Kqm2yk/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fd9dfaef-7179-4abf-bb60-b8a93982d76a%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/fd9dfaef-7179-4abf-bb60-b8a93982d76a%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEU_gmfNi6nowVSM1FMxWn6XKZkk_szcy1%3DCVZ%3Dz%2BZWoZC%3D8HA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: There were no results because no indices were found that match your selected time span
I have the same issue... I upgraded to 0.9.10, still seems to find, and display, no data. I have plugin head installed and it finds the indexes fine, but marvel does not. There is no configuration step, right? where do I check for the plugin's logs? Thanks, Mohamed. On Tuesday, January 28, 2014 2:01:21 PM UTC-5, Thomas Ardal wrote: When trying out Marvel on my Elasticsearch installation, I get the error There were no results because no indices were found that match your selected time span in the top of the page. If I understand the documentation, Marvel automatically collects statistics from all indexes on the node. What am I doing wrong? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9c3ea1b1-f134-4ea2-9584-6319e6b94cb7%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: There were no results because no indices were found that match your selected time span
I found out some of my nodes were not upgraded, and I also did not have the marvel plugin installed on all nodes... data is now appearing in the marvel dashboard. Thanks, Mohamed. On Wednesday, January 29, 2014 12:35:00 PM UTC-5, Mohamed Lrhazi wrote: I have the same issue... I upgraded to 0.9.10, still seems to find, and display, no data. I have plugin head installed and it finds the indexes fine, but marvel does not. There is no configuration step, right? where do I check for the plugin's logs? Thanks, Mohamed. On Tuesday, January 28, 2014 2:01:21 PM UTC-5, Thomas Ardal wrote: When trying out Marvel on my Elasticsearch installation, I get the error There were no results because no indices were found that match your selected time span in the top of the page. If I understand the documentation, Marvel automatically collects statistics from all indexes on the node. What am I doing wrong? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4cc562a1-c952-4881-8be9-1fcae821209c%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
