Re: Kibana4 doesn't recognize my timestamp; ES is mapping it as a string.
By using this statement 'es.mapping.names’ = ‘time:@timestamp’ You've changed the name of the 'time' field in your hive table to '@timestamp' in your elasticsearch index. See if you have a can find a field called @timestamp in kibana and use that as your time field in ES. Alternatively try removing that line from your hive creation table see if that changes anything. On Sat, Mar 21, 2015 at 2:39 PM, BEN SALEM Omar omar.bensa...@esprit.tn wrote: I want to push data I have in my hadoop cluster to ES and then visualize the hole thing in kibana4. this is what I’ve done : 1) CREATE TABLE xx(traffic_type_id INT, caller INT, time STRING, tranche_horaire INT, called INT, call_duration INT, code_type_trafic STRING, code_destination_trafic STRING, location_number STRING, id_offre INT, id_service INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘\t'; LOAD DATA INPATH ‘/user/hive/outt.csv’ OVERWRITE INTO TABLE xx; __ 2) CREATE EXTERNAL TABLE esxx (caller INT, time STRING, tranche INT, called_number INT, duration INT, code_type STRING, code_destination STRING, location STRING, offre INT, service INT) STORED BY ‘org.elasticsearch.hadoop.hive.EsStorageHandler’ TBLPROPERTIES(‘es.resource’ = ‘xx/xx’, ‘es.nodes’=’192.168.238.130:9200′, ‘es.mapping.names’ = ‘time:@timestamp’); INSERT OVERWRITE TABLE escdr SELECT s.caller, s.time, s.tranche_horaire, s.called, s.call_duration, s.code_type_trafic, s.code_destination_trafic, s.location_number, s.id_offre, s.id_service FROM xx s; _ 3) CREATE EXTERNAL TABLE xx ( caller INT, time TIMESTAMP, tranche INT, called_number INT, duration INT, code_type STRING, code_destination STRING, location STRING, offre INT, service INT) STORED BY ‘org.elasticsearch.hadoop.hive.EsStorageHandler’ TBLPROPERTIES(‘es.resource’ = ‘xx/xx/’, ‘es.nodes’=’192.168.238.130:9200′, ‘es.mapping.names’ = ‘time:@timestamp’); But Kibana doesn’t seem to recognize my timestamp “time”, ES keeps on mapping it as a string (the time field in my csv file is as so : exp : 01AUG2014:19:02:11 ! What should I do and change to let ES do the appropriate mapping and thus recognize my timestamp? Best regards, Omar, -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/30597efd-b762-473d-94aa-9c1671fb0ff4%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/30597efd-b762-473d-94aa-9c1671fb0ff4%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAO9TxdPRV%2BaGkScp58hRfhSzAK4ojDO2NV0MTWt7LGGU9cb-Ow%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Geohash Filter in kibana and Custom layers in maps
Hello Everyone, I'm trying to find a way to implement a geohash filter in kibana ? Also how would one go about adding custom layers to the maps ? Would really appreciate any resources you could point me to. Thanks -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAO9TxdMaN1c1RGDLxf6H7sdoozk%3DSE-KjVMVs-J9ufaXe7tHSQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Hive to elasticsearch Parsing exception.
Ignoring both null values and null strings worked. Will open a issue about this. Thanks a lot Costin. On Fri, Mar 13, 2015 at 12:08 AM, Costin Leau costin.l...@gmail.com wrote: Likely the issue is caused by the fact that in your manual mapping, the NULL value is not actually mapped to null but actually to a string value. You should be able to get around it by converting NULL to a proper NULL value which es-hadoop can recognized; additionally you can 'translate' it to a default one. As for understanding what field caused the exception, unfortunately Elasticsearch doesn't provide enough information about this yet but it should. Can you please raise a quick issue on es-hadoop about this? Thanks, On Thu, Mar 12, 2015 at 10:12 PM, P lva ruvi...@gmail.com wrote: Hello Everyone, I'm loading data from a a hive table (0.13) in to elasticsearch (1.4.4). With the auto create index option turned on , I don't face any problems and I can see all the data in ES. However, I get the following error when i create the index manually. Caused by: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: Found unrecoverable error [Bad Request(400) - [MapperParsingException[failed to parse]; nested: NumberFormatException[For input string: NULL]; ]]; Bailing out.. at org.elasticsearch.hadoop.rest.RestClient.retryFailedEntries(RestClient.java:199) at org.elasticsearch.hadoop.rest.RestClient.bulk(RestClient.java:165) at org.elasticsearch.hadoop.rest.RestRepository.sendBatch(RestRepository.java:170) at org.elasticsearch.hadoop.rest.RestRepository.doWriteToIndex(RestRepository.java:152) at org.elasticsearch.hadoop.rest.RestRepository.writeProcessedToIndex(RestRepository.java:146) at org.elasticsearch.hadoop.hive.EsHiveOutputFormat$EsHiveRecordWriter.write(EsHiveOutputFormat.java:63) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:621) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793) at org.apache.hadoop.hive.ql.exec.LimitOperator.processOp(LimitOperator.java:51) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793) at org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45) at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:262) To create the index manually, I've used the same mappings from the first auto create step and changed one field to geo point type. Changing the field type is the only change I made. The column that I wanted to be geo fields had a few nulls, so i selected rows without nulls and still have the same error. Is there any way to identify which column is causing the issue ? There's about 70 columns in my table. Tl;dr Hive table to elasticsearch Auto create index works fine Fails when I manually created index with almost same mapping (except one field changed from string to geopoint) Thanks -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAO9TxdO22hy2%3Dcz1S_DJgvtd0rsw%2Bu0WL8SqLFR8GTbbGJr9EQ%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAO9TxdO22hy2%3Dcz1S_DJgvtd0rsw%2Bu0WL8SqLFR8GTbbGJr9EQ%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJogdmderE6q3w0mJytbmfKkYHyegs7zwi9x5wtOe9G_MWKEyw%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAJogdmderE6q3w0mJytbmfKkYHyegs7zwi9x5wtOe9G_MWKEyw%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAO9TxdOwO41e3NXm-pmohyGY8TjSF-RnB4kc1S%2B7U3Hm3cZkuQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Hive to elasticsearch Parsing exception.
Hello Everyone, I'm loading data from a a hive table (0.13) in to elasticsearch (1.4.4). With the auto create index option turned on , I don't face any problems and I can see all the data in ES. However, I get the following error when i create the index manually. Caused by: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: Found unrecoverable error [Bad Request(400) - [MapperParsingException[failed to parse]; nested: NumberFormatException[For input string: NULL]; ]]; Bailing out.. at org.elasticsearch.hadoop.rest.RestClient.retryFailedEntries(RestClient.java:199) at org.elasticsearch.hadoop.rest.RestClient.bulk(RestClient.java:165) at org.elasticsearch.hadoop.rest.RestRepository.sendBatch(RestRepository.java:170) at org.elasticsearch.hadoop.rest.RestRepository.doWriteToIndex(RestRepository.java:152) at org.elasticsearch.hadoop.rest.RestRepository.writeProcessedToIndex(RestRepository.java:146) at org.elasticsearch.hadoop.hive.EsHiveOutputFormat$EsHiveRecordWriter.write(EsHiveOutputFormat.java:63) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:621) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793) at org.apache.hadoop.hive.ql.exec.LimitOperator.processOp(LimitOperator.java:51) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793) at org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45) at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:262) To create the index manually, I've used the same mappings from the first auto create step and changed one field to geo point type. Changing the field type is the only change I made. The column that I wanted to be geo fields had a few nulls, so i selected rows without nulls and still have the same error. Is there any way to identify which column is causing the issue ? There's about 70 columns in my table. Tl;dr Hive table to elasticsearch Auto create index works fine Fails when I manually created index with almost same mapping (except one field changed from string to geopoint) Thanks -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAO9TxdO22hy2%3Dcz1S_DJgvtd0rsw%2Bu0WL8SqLFR8GTbbGJr9EQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Notifications for a query
I'd really appreciate any ideas for this. On Monday, July 21, 2014 10:28:31 AM UTC-5, P lva wrote: Hello Everyone, Started working with Elasticsearch recently. Just wanted to know if there's any way of being notified when a document matches a query. (essentially create a monitoring system) Can I use percolator to do this ? Thanks -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0c74ac3a-3a05-4dd4-b91d-0ee028e60f3f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Notifications for a query
Hello Everyone, Started working with Elasticsearch recently. Just wanted to know if there's any way of being notified when a document matches a query. (essentially create a monitoring system) Can I use percolator to do this ? Thanks -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAO9TxdPOJGbY8nEj4Hj31LMr%3DaVLTwc_%3DwJW%3Drc7Ly2LAYc8nw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Elastic Search -Hive mapping
Hello, I have an event timestamp in hive table which I'm breaking into evnt_time(timestamp) and an evnt_date(string) columns. The idea is to use the evnt_time as timestamp in kibana and be able to search for date and stack one day on top of another in a histogram plot in kibana. Eventually giving me comparison between two days as a stacked chart. As per elasticsearch-hive documentation, strings in hive will be of type string in elasticsearch also. However when I load data into elasticsearch the date column is being considered as timestamp. How do I prevent this ? Please share if you have any ideas on how to get a stacked chart. Thanks -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/37ebc775-09c1-4ccf-9063-18dc8b959c3d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Micro Analysis in Kibana
The sql query I would use would be select count(name) from table Elasticsearch terms would give number of 'John' but I'd like to know number of 'John Watson' and number of 'John Lennon'. On Thu, Apr 3, 2014 at 8:37 PM, Wang cnwangy...@gmail.com wrote: how to count how many terms in docs? I mean , in Iva's example, how to get the result of 3 , which mean there is 3 terms? On Friday, April 4, 2014, Binh Ly binhly...@yahoo.com wrote: Sounds like a terms panel on that field that contains the values 1,2,3. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9b1a3779-f5a3-45ed-a58d-7cdb23eb717b%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/9b1a3779-f5a3-45ed-a58d-7cdb23eb717b%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/t8opboYG7UA/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPwW3GV_0Y3LJ985GQG%3DQqPFcchQGRc1hFZH8NXTNwTsgiGATQ%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAPwW3GV_0Y3LJ985GQG%3DQqPFcchQGRc1hFZH8NXTNwTsgiGATQ%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAO9TxdM_7z17vsQjwaSinpY6w3nSHzfLikQDLSk3crJQJwr0rg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Micro Analysis in Kibana
Awesome Thanks Binh Ly
On Friday, April 4, 2014 2:55:20 PM UTC-5, Binh Ly wrote:
You'll need to set the field name to not_analyzed so that you can get a
distinct value for the whole field (instead of tokenized values):
{
mappings: {
doc: {
properties: {
name: {
type: string,
index: not_analyzed
}
}
}
}
}
After that, you can do a terms facet on name and you'll get the count that
you want.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/efc12955-99e5-4bac-b7e8-e8f04441a01a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Micro Analysis in Kibana
Hello everyone, I'm using kibana and would like to add a panel that basically counts the values in column. (Similar to micro analysis when you select the field). value 1 value 2 value 2 value 3 value 1 Result value 1 = 2 value 2 = 2 value 3 = 1 Which panel can one use to get this information ? Thanks -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/75d0a046-2332-4814-9bc8-2b239df5e836%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: ES-Hive
I got it to work by replacing
'es.resource.write'='http://eshost:9200'
'es.resource.node'='http://eshost:9200'
as one of the tbl properties in hive.
Thanks for your help Costin.
On Friday, March 14, 2014 1:12:00 PM UTC-5, Costin Leau wrote:
You should not change the configuration in Elasticsearch. By default, ES
binds on all available interfaces - specifying
an Ip:port is likely to restrict access rather than extend it.
To find out the configuration options in es-hadoop, look no further than
the docs [1]. If the hive job is running on the
same machine as elasticsearch, the default (localhost:9200) should be
enough. As an alternative you can specify the
public ip of your host: x.y.z.w
As a last step, enable TRACE logging (through log4j) on
org.elasticsearch.hadoop package and see what comes out
including network connectivity.
By the way, make sure you don't have any firewall or proxy set on your
system which might be picked up automatically by
the JVM.
[1]
http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/configuration.html
On 3/14/2014 6:50 PM, P lva wrote:
Costin,
About what you asked,
1) Hive server is running on the same machine as elasticsearch,
2) I get a response when I do a curl http://elasticsearchivehost:9200
I feel I'm missing something simple. This is what I've got until now.
1) First error (connection refused) is because i left the default
settings as is.
I changed the 'network.host' in elasticsearch.yml to the hostname
2) Second error (cannot discover elasticsearch version) is when I
changed the 'network.host' in elasticsearch.yml to the
hostname.
Looked at the source code and figured I'm expected to pass eshost:port
to the hive table at table creation.
(
https://github.com/elasticsearch/elasticsearch-hadoop/blob/master/src/main/java/org/elasticsearch/hadoop/hive/EsStorageHandler.java#L48)
.
So I included 'es.resource.write'='http://elasticsearchhost:9200' as
one of the table properties during table creation step.
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'esdemo/hive',
'es.mapping.name'='time:@timestamp',
'es.resource.write'='http://eshost:9200');
Now I get connection refused again. Is that the right way to pass that
information ?
Thanks for you patience and help
On Friday, March 14, 2014 10:33:14 AM UTC-5, Costin Leau wrote:
Without any extra information I'm afraid I can only guess what might
be the issue.
Make sure you have the latest Elasticsearch 0.90 or 1.x available on
port 9200 with the HTTP/REST port open.
Also make sure that Hive actually runs on the same machine - not
just the client but also the server (meaning Hadoop
itself).
You indicate that if you change the network configuration you get an
error regarding the version - this suggests that:
1. Hive is actually running on a different machine than ES - hence
the network error
2. After pointing Hive to the actual ES machine, you get an error
since you're using an old Elasticsearch version
(0.20)
Cheers,
On 3/14/2014 12:19 AM, P lva wrote:
I have a simple query
insert into table eslogs select * from eslogs_ext;
Hive and elasticsearch are running on the same host.
To execute the script I'm following the directions from the link.
http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html
http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html
There are two elasticsearch nodes, and they can recognize each
other (as indicated by start up process) , but why would
hive not be able to pick them up ? Can you explain what could have
gone wrong ?
On Thursday, March 13, 2014 4:28:14 PM UTC-5, Costin Leau wrote:
What does your Hive script look like? Can you confirm the
ip/address of your Hive and Elasticsearch ? How are you
executing the script?
The error indicates an error in your network configuration.
Cheers,
P.S. Feel free to post a gist or whatever it's convenient.
On 3/13/2014 10:38 PM, P lva wrote:
Hi, I have few weblogs in a hive table that I'd like to
visualize in kibana.
ES is on the same node as hive server.
Followed directions from this pagehttp://
www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html
http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html
http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html
http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html
Re: Elasticsearch Startup issues.
It worked fine after manually setting all the environment variables. I would say this though. Server a : ES out of box works from home dir. Server b : ES out of box neither works from /usr/lib/ nor does it work from home dir. Only way is to manually set env parameters. Both servers are built identically. On Wednesday, March 12, 2014 2:57:57 PM UTC-5, Jörg Prante wrote: The script elasticsearch, how you call it with absolute path under system dir /usr/lib, has problems in traversing the path to find the parent folder and set the variable ES_HOME. You should use an init script, like provided by the distributions or the service wrapper, with preconfigured environment, so ES_HOME and ES_CLASSPATH (i.e. the $ES_HOME/lib folder) can be used. Jörg On Wed, Mar 12, 2014 at 8:44 PM, P lva ruv...@gmail.com javascript:wrote: I didnt want elasticsearch to be under a user's directory so I have it in /usr/lib/elasticsearch. All the files are owned by the non root user that is running the command bash /usr/lib/elasticsearch/bin/elasticsearch On Wednesday, March 12, 2014 2:21:41 PM UTC-5, Jörg Prante wrote: And do you start Elasticsearch by the same user you installed Elasticsearch under? Jörg On Wed, Mar 12, 2014 at 8:19 PM, joerg...@gmail.com joerg...@gmail.comwrote: How do you start Elasticsearch, by $ cd $ES_HOME $ ./bin/elasticsearch or $ cd $ES_HOME/bin $ ./elasticsearch or ... ? Jörg On Wed, Mar 12, 2014 at 8:12 PM, P lva ruv...@gmail.com wrote: I have RHEL 6. Java 1.7 default shell is bash. I tried it on a server, it worked fine right out of the box, but when I moved to the actual set of servers it failed. Its got something to do with environment for sure, but I can't figure out what. On Wednesday, March 12, 2014 5:20:56 AM UTC-5, Alexander Reelsen wrote: Hey, can you be more verbose, how you are actually starting elasticsearch, which version you are using, what operating system you are running on, which java version you are running, what is your default shell? Did you modify anything from the standard installation? Thanks. --Alex On Tue, Mar 11, 2014 at 11:38 PM, P lva ruv...@gmail.com wrote: I get this error at elasticsearch startup Exception in thread main java.lang.NoClassDefFoundError: org.apache.lucene.util.Version at org.elasticsearch.Version.clinit(Version.java:42) at java.lang.Class.initializeClass(libgcj.so.10) at org.elasticsearch.bootstrap.Bootstrap.buildErrorMessage(Boot strap.java:252) at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:178) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch .java:32) The jar containing this class is present in the lib directory and echo $ES_CLASSPATH shows this jar. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fd314c1e-a78 a-4d93-a1e4-5c442fab1bee%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/fd314c1e-a78a-4d93-a1e4-5c442fab1bee%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/ msgid/elasticsearch/ccb52426-5681-4505-b08f-8233ba0d78ef% 40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/ccb52426-5681-4505-b08f-8233ba0d78ef%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com javascript:. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/412cb050-aecf-4309-b230-db2d538ddd12%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/412cb050-aecf-4309-b230-db2d538ddd12%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/add0ad6a-83d3-46c7-a440-9d3a5dbdc8a5%40googlegroups.com. For more options, visit https://groups.google.com/d
ES-Hive
Hi, I have few weblogs in a hive table that I'd like to visualize in kibana.
ES is on the same node as hive server.
Followed directions from this
page
http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html
I can create a table using esstorage handler, but when I tried to ingest
data into this table I got
Error: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row {***first row of my table**}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:175)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row {*** first row of my table**}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.io.IOException: Out of nodes and retries; caught exception
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:652)
at
org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:88)
at
org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91)
at
org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
... 9 more
Caused by: java.io.IOException: Out of nodes and retries; caught exception
at
org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:81)
at
org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:221)
at
org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:205)
at
org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:209)
at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:103)
at
org.elasticsearch.hadoop.rest.RestClient.discoverNodes(RestClient.java:85)
at
org.elasticsearch.hadoop.rest.InitializationUtils.discoverNodesIfNeeded(InitializationUtils.java:60)
at
org.elasticsearch.hadoop.mr.EsOutputFormat$ESRecordWriter.init(EsOutputFormat.java:165)
at
org.elasticsearch.hadoop.hive.EsHiveOutputFormat$ESHiveRecordWriter.write(EsHiveOutputFormat.java:50)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:638)
... 18 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.init(Socket.java:425)
at java.net.Socket.init(Socket.java:280)
at
org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)
at
org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122)
at
org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at
org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.execute(CommonsHttpTransport.java:160)
at
org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:74)
... 27 more
Now, I changed the config network.host to the ipadress of the server. Now
when I run
Re: ES-Hive
I have a simple query
insert into table eslogs select * from eslogs_ext;
Hive and elasticsearch are running on the same host.
To execute the script I'm following the directions from the link.
http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html
There are two elasticsearch nodes, and they can recognize each other (as
indicated by start up process) , but why would hive not be able to pick
them up ? Can you explain what could have gone wrong ?
On Thursday, March 13, 2014 4:28:14 PM UTC-5, Costin Leau wrote:
What does your Hive script look like? Can you confirm the ip/address of
your Hive and Elasticsearch ? How are you
executing the script?
The error indicates an error in your network configuration.
Cheers,
P.S. Feel free to post a gist or whatever it's convenient.
On 3/13/2014 10:38 PM, P lva wrote:
Hi, I have few weblogs in a hive table that I'd like to visualize in
kibana.
ES is on the same node as hive server.
Followed directions from this page
http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html
I can create a table using esstorage handler, but when I tried to
ingest data into this table I got
Error: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing
row {***first row of my table**}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:175)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive
Runtime Error while processing row {*** first row of
my table**}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.io.IOException: Out of nodes and retries; caught exception
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:652)
at
org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:88)
at
org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91)
at
org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
... 9 more
Caused by: java.io.IOException: Out of nodes and retries; caught
exception
at
org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:81)
at
org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:221)
at
org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:205)
at
org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:209)
at
org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:103)
at
org.elasticsearch.hadoop.rest.RestClient.discoverNodes(RestClient.java:85)
at
org.elasticsearch.hadoop.rest.InitializationUtils.discoverNodesIfNeeded(InitializationUtils.java:60)
at
org.elasticsearch.hadoop.mr.EsOutputFormat$ESRecordWriter.init(EsOutputFormat.java:165)
at
org.elasticsearch.hadoop.hive.EsHiveOutputFormat$ESHiveRecordWriter.write(EsHiveOutputFormat.java:50)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:638)
... 18 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528
Re: Elasticsearch Startup issues.
Apart from cluster name and node name I didnt change anything. Version 1.0.1 Commands I use to start bin/elasticsearch On Wednesday, March 12, 2014 5:20:56 AM UTC-5, Alexander Reelsen wrote: Hey, can you be more verbose, how you are actually starting elasticsearch, which version you are using, what operating system you are running on, which java version you are running, what is your default shell? Did you modify anything from the standard installation? Thanks. --Alex On Tue, Mar 11, 2014 at 11:38 PM, P lva ruv...@gmail.com javascript:wrote: I get this error at elasticsearch startup Exception in thread main java.lang.NoClassDefFoundError: org.apache.lucene.util.Version at org.elasticsearch.Version.clinit(Version.java:42) at java.lang.Class.initializeClass(libgcj.so.10) at org.elasticsearch.bootstrap.Bootstrap.buildErrorMessage(Bootstrap.java:252) at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:178) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32) The jar containing this class is present in the lib directory and echo $ES_CLASSPATH shows this jar. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com javascript:. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fd314c1e-a78a-4d93-a1e4-5c442fab1bee%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/fd314c1e-a78a-4d93-a1e4-5c442fab1bee%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b489cd4a-cb33-45f6-9ff5-fb99dd4429eb%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Elasticsearch Startup issues.
I have RHEL 6. Java 1.7 default shell is bash. I tried it on a server, it worked fine right out of the box, but when I moved to the actual set of servers it failed. Its got something to do with environment for sure, but I can't figure out what. On Wednesday, March 12, 2014 5:20:56 AM UTC-5, Alexander Reelsen wrote: Hey, can you be more verbose, how you are actually starting elasticsearch, which version you are using, what operating system you are running on, which java version you are running, what is your default shell? Did you modify anything from the standard installation? Thanks. --Alex On Tue, Mar 11, 2014 at 11:38 PM, P lva ruv...@gmail.com javascript:wrote: I get this error at elasticsearch startup Exception in thread main java.lang.NoClassDefFoundError: org.apache.lucene.util.Version at org.elasticsearch.Version.clinit(Version.java:42) at java.lang.Class.initializeClass(libgcj.so.10) at org.elasticsearch.bootstrap.Bootstrap.buildErrorMessage(Bootstrap.java:252) at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:178) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32) The jar containing this class is present in the lib directory and echo $ES_CLASSPATH shows this jar. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com javascript:. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fd314c1e-a78a-4d93-a1e4-5c442fab1bee%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/fd314c1e-a78a-4d93-a1e4-5c442fab1bee%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ccb52426-5681-4505-b08f-8233ba0d78ef%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Elasticsearch Startup issues.
I didnt want elasticsearch to be under a user's directory so I have it in /usr/lib/elasticsearch. All the files are owned by the non root user that is running the command bash /usr/lib/elasticsearch/bin/elasticsearch On Wednesday, March 12, 2014 2:21:41 PM UTC-5, Jörg Prante wrote: And do you start Elasticsearch by the same user you installed Elasticsearch under? Jörg On Wed, Mar 12, 2014 at 8:19 PM, joerg...@gmail.com javascript: joerg...@gmail.com javascript: wrote: How do you start Elasticsearch, by $ cd $ES_HOME $ ./bin/elasticsearch or $ cd $ES_HOME/bin $ ./elasticsearch or ... ? Jörg On Wed, Mar 12, 2014 at 8:12 PM, P lva ruv...@gmail.com javascript:wrote: I have RHEL 6. Java 1.7 default shell is bash. I tried it on a server, it worked fine right out of the box, but when I moved to the actual set of servers it failed. Its got something to do with environment for sure, but I can't figure out what. On Wednesday, March 12, 2014 5:20:56 AM UTC-5, Alexander Reelsen wrote: Hey, can you be more verbose, how you are actually starting elasticsearch, which version you are using, what operating system you are running on, which java version you are running, what is your default shell? Did you modify anything from the standard installation? Thanks. --Alex On Tue, Mar 11, 2014 at 11:38 PM, P lva ruv...@gmail.com wrote: I get this error at elasticsearch startup Exception in thread main java.lang.NoClassDefFoundError: org.apache.lucene.util.Version at org.elasticsearch.Version.clinit(Version.java:42) at java.lang.Class.initializeClass(libgcj.so.10) at org.elasticsearch.bootstrap.Bootstrap.buildErrorMessage( Bootstrap.java:252) at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:178) at org.elasticsearch.bootstrap.Elasticsearch.main( Elasticsearch.java:32) The jar containing this class is present in the lib directory and echo $ES_CLASSPATH shows this jar. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/ msgid/elasticsearch/fd314c1e-a78a-4d93-a1e4-5c442fab1bee% 40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/fd314c1e-a78a-4d93-a1e4-5c442fab1bee%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com javascript:. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ccb52426-5681-4505-b08f-8233ba0d78ef%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/ccb52426-5681-4505-b08f-8233ba0d78ef%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/412cb050-aecf-4309-b230-db2d538ddd12%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Elasticsearch Startup issues.
I get this error at elasticsearch startup Exception in thread main java.lang.NoClassDefFoundError: org.apache.lucene.util.Version at org.elasticsearch.Version.clinit(Version.java:42) at java.lang.Class.initializeClass(libgcj.so.10) at org.elasticsearch.bootstrap.Bootstrap.buildErrorMessage(Bootstrap.java:252) at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:178) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32) The jar containing this class is present in the lib directory and echo $ES_CLASSPATH shows this jar. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fd314c1e-a78a-4d93-a1e4-5c442fab1bee%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
