The error indicates the ES nodes don't reply in a timely fashion and thus the connection drops. Based on your logs it
seems to be either a GC or a network issue.
You could try turning on logging in package 'org.elasticsearch.hadoop.rest' to
DEBUG.
How many tasks do you have and what's your bulk size/entries number?
On 10/1/14 2:15 PM, Zach Cox wrote:
Hi Costin - we updated our dependencies to use elasticsearch-hadoop
2.0.2.BUILD-SNAPSHOT, but that didn't seem to change
anything. We're still seeing the same task failures while trying to write to
Elasticsearch. The only difference in the
logs is that now I don't see the SimpleHttpConnectionManager warnings.
Any ideas what we could try next?
Thanks,
Zach
On Tuesday, September 30, 2014 10:54:27 AM UTC-5, Costin Leau wrote:
Can you please try the 2.0.2.BUILD-SNAPSHOT? I think you might be running
into issue #256 which was fixed some time ago
and will be part of the upcoming
2.0.2, 2.1 Beta2.
Cheers,
On 9/30/14 6:43 PM, Zach Cox wrote:
> Hi Costin:
>
> elasticsearch-hadoop 2.0.0
> cascading 2.5.4
> scalding 0.10.0
>
> Thanks,
> Zach
>
>
> On Tuesday, September 30, 2014 10:25:10 AM UTC-5, Costin Leau wrote:
>
> What version of es-hadoop/es/cascading are you using?
>
> On 9/30/14 6:16 PM, Zach Cox wrote:
> > Hi - we're having problems with one of our map-reduce jobs that
writes to Elasticsearch. Lots of map tasks are failing
> > due to ES being "unavailable", with logs like this:
> >
>
>https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt
<https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt>
>
<https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt
<https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt>>
>
> >
> > Seems like elasticsearch-hadoop tries talking to an ES node, it
times out, tries the next one, it times out, etc until
> > all nodes in the cluster are exhausted and then it gives up.
> >
> > As far as I can tell, the ES cluster is healthy while this is
occurring. May map tasks are succeeding - probably about
> > 10% of the attempts are killed due to this issue. The main problem
is that these killed tasks waste a lot of time, and
> > slow down the overall job execution.
> >
> > I'm not sure where to troubleshoot this next. Does anyone have any idea
what would cause all of these time outs & failures?
> >
> > I'm also curious about the lines like this:
> >
> > 2014-09-30 12:49:20,469 WARN
org.apache.commons.httpclient.SimpleHttpConnectionManager:
SimpleHttpConnectionManager being used incorrectly. Be sure that
> HttpMethod.releaseConnection() is always called and that only one
thread and/or method is using this connection
> manager at a time.
> >
> >
> > Would that be related to the timeout problem we're seeing?
> >
> > Thanks,
> > Zach
> >
> > --
> > You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it,
send an email to
> >elasticsearc...@googlegroups.com <javascript:>
<mailto:elasticsearch+unsubscr...@googlegroups.com <javascript:>
<javascript:>>.
> > To view this discussion on the web visit
>
>https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com>
>
<https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com>>
> >
<https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer>
>
<https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer>>>.
>
> > For more options, visithttps://groups.google.com/d/optout
<http://groups.google.com/d/optout> <https://groups.google.com/d/optout
<https://groups.google.com/d/optout>>.
> --
> Costin
>
> --
> You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
email to
>elasticsearc...@googlegroups.com <javascript:>
<mailto:elasticsearch+unsubscr...@googlegroups.com <javascript:>>.
> To view this discussion on the web visit
>https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com>
>
<https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm_medium=email&utm_source=footer>>.
> For more options, visithttps://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
Costin
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
elasticsearch+unsubscr...@googlegroups.com
<mailto:elasticsearch+unsubscr...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
Costin
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/542BF16B.6010103%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
