date:20140320

Include -Djava.net.preferIPv4Stack=true to your JAVA_OPTS environment
variable before starting Elasticsearch. Also read up on the network page
that David referred to:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-network.html

If you want a control wrapper around Elasticsearch, you might be better off
installing via rpm/yum and not use the service wrapper.

Cheers,

Ivan


On Thu, Mar 20, 2014 at 10:26 AM, Anikessh Jain wrote:

> thnks for the info
>
> how to run the service on ipv4 bcoz when i am starting ti is starting on
> ipv6 instead on ipv4 ,how do i change that
>
>
> On Thursday, March 20, 2014 10:30:35 PM UTC+5:30, David Pilato wrote:
>
>> I don't know.
>>
>> So let me help you for elasticsearch:
>>
>> wget https://download.elasticsearch.org/elasticsearch/elasticsearch/
>> elasticsearch-1.0.1.tar.gz
>> tar xzf elasticsearch-1.0.1.tar.gz
>> ./elasticsearch-1.0.1/bin/elasticsearch
>>
>> And that's all!
>> This is basically what is written here: http://www.
>> elasticsearch.org/guide/en/elasticsearch/reference/
>> current/setup.html#setup-installation
>>
>>
>> --
>> *David Pilato* | *Technical Advocate* | *Elasticsearch.com*
>> @dadoonet  | 
>> @elasticsearchfr
>>
>>
>> Le 20 mars 2014 à 17:49:49, Anikessh Jain (anikess...@gmail.com) a écrit:
>>
>> Hi David,
>>
>> But why do people download and install it through
>> elasticsearch-servicewrapper ,does the leatest version supports it all bcoz
>> i want to install kibana with logstash and redis with elasticsearch and i
>> am a little bit confused will it work bcoz i have wasted my whole day on on
>> error.please guide me what to do as i dont know what to do and doing it for
>> first time
>>
>> On Thursday, March 20, 2014 10:11:52 PM UTC+5:30, David Pilato wrote:
>>>
>>>  First download a recent version. If you are starting with
>>> elasticsearch, you don't need to start with a so old version.
>>>  1.0.1 is fine.
>>>
>>>  Then read the documentation here: http://www.
>>> elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html
>>>
>>>  No need to install so many pieces.
>>>
>>>  Unzip and launch. That's all.
>>>
>>>
>>>  --
>>> *David Pilato* | *Technical Advocate* | *Elasticsearch.com*
>>>  @dadoonet  | 
>>> @elasticsearchfr
>>>
>>>
>>> Le 20 mars 2014 à 17:06:12, Anikessh Jain (anikess...@gmail.com) a
>>> écrit:
>>>
>>>  please help on the above error
>>>
>>> On Thursday, March 20, 2014 9:03:46 PM UTC+5:30, Anikessh Jain wrote:

   wget 
 https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.2.tar.gz
 tar xvf elasticsearch-0.20.2.tar.gz

 mv elasticsearch-0.20.2 elasticsearch

wget 
 http://github.com/elasticsearch/elasticsearch-servicewrapper/archive/master.zip
 unzip master
 mv elasticsearch-servicewrapper-master/service/ elasticsearch/bin
 rm -rf master
 rm -rf elasticsearch-servicewrapper-master/


 Configure 'elasticsearch.conf'

  set.default.ES_HOME=/usr/local/elasticsearch
 set.default.ES_HEAP_SIZE=4096
 wrapper.java.additional.10=-Des.max-open-files=true

 wrapper.logfile.maxsize=5m
 wrapper.logfile.maxfiles=5


 Add ES home to root user's '.bash_profile'

  # ElasticSearch
 export ES_HOME=/usr/local/elasticsearch


 Create elasticsearch user

  useradd -d /home/elasticsearch -s /bin/sh elasticsearch
 chown -R elasticsearch:elasticsearch $ES_HOME
 chown -R elasticsearch:elasticsearch /mnt/storage/es-data


 Edit elasticsearch user's '.bash_profile'

  # .bash_profile

 # Get the aliases and functions
 if [ -f ~/.bashrc ]; then
 . ~/.bashrc
 fi

 # User specific environment and startup programs

 PATH=$PATH:$HOME/bin

 export PATH

 # JAVA_HOME needs to be the latest 1.7 JDK on the system
 JAVA_HOME=/usr/local/jdk7
 export JAVA_HOME

 #Add JAVA_HOME to the PATH
 PATH=$JAVA_HOME/bin:$PATH

 # ElasticSearch
 export ES_HOME=/usr/local/elasticsearch

 unset USERNAME


 Edit '/etc/security/limits.conf' (optional as this will be set in the 
 service script, too)

  elasticsearchsoftnofile  65535
 elasticsearchhardnofile  65535


 Verify the file descriptor limit

  sudo -u elasticsearch -s ulimit -Sn


 Install the service

  bin/service/elasticsearch install


 Edit '/etc/init.d/elasticsearch'

  # Java
 JAVA_HOME=/usr/local/jdk7
 export JAVA_HOME
 PATH=$JAVA_HOME/bin:$PATH
 RUN_AS_USER=elasticsearch
 ULIMIT_N=65535


 Run the service

  bin/service/elasticsearch start
 or
 service elasticsearch start   -> here the

Re: 2 clusters versus 1 big cluster?

2014-03-20 Thread Mauri

Hi Brad

I agree with what Mark and Zachary have said and will expand on these.

Firstly, shard and index level operations in ElasticSearch are 
peer-to-peer. Single-shard operations will affect at most 2 nodes, the node 
receiving the request and the node hosting an instance of the shard 
(primary or replica depending on the operation). Multi-shard operations 
(such a searches) will affect from one to (N  +1) nodes where N is the 
number of shards in the index. 

So from an index/shard operation perspective there is no reason to split 
into two clusters. The key issue with index/shard operations is that the 
cluster is able to handle the traffic volume. So if you do decide to split 
out into to two clusters you will need to look at the load profile for each 
of your client types to determine how much raw processing power you need in 
each cluster. It may be that a 10:20 split is more optimum than a 15:15 
split between clusters to balance request traffic, and therefore CPU 
utilisation, across all nodes. If you go with one cluster this is not an 
issue because you can move shards between nodes to balance the request 
traffic.

Larger clusters also imply more work for the cluster master in managing the 
cluster. This comes down to the number of nodes that the master has to 
communicate with, and manage, and the size of the cluster state. A cluster 
with 30 nodes is not too large for a master to keep track of. There will be 
an increase in network traffic associated with the increase in volume of 
master-to-worker and worker-to-master pings used to detect the 
presence/absence of nodes. This can be offset by reducing the respective 
ping intervals.

In a large cluster it is good practice to have a group of dedicated master 
nodes, say 3, from which the master is elected. These nodes do not host any 
user data meaning that cluster management is not compromised by high user 
request traffic. 

The size of the cluster state may be more of an issue. The cluster state 
comprises all of the information about the cluster configuration. The 
cluster state has records for each node, index, document mapping, shard, 
etc. Whenever there is a change to the cluster state it is first made by 
the master which then sends the updated cluster state to each worker node. 
Note that the entire cluster state is sent, not just the changes! It is 
therefore highly desirable to limit that frequency of changes to the 
cluster state, primarily by minimizing dynamic field mapping updates, and 
the overall size of the cluster state, primarily by minimizing the number 
of indices. 

In your proposed model the size of the cluster state associated the set of 
60 shared month indices will be larger than that of one set of 60 dedicated 
month indices by virtue of having 100 shards to 6. However, it may not be 
much bigger because there will be much more metadata associated with 
defining the index structure, notably the field mappings for all document 
types in the index, than the metadata defining the shards of the index. So 
it may well be that the size of the cluster state associated with 60 
"shared" month indices plus N sets of 60 "dedicated" indices is not much 
more than that of (N + 1) sets of 60 "dedicated" indices. So there may not 
be much point in splitting to two clusters. A quick way to look at this for 
your actual data model is to:
  1. Set up an index in ES with mappings for all document types and 6 
shards and 0 replicas,
  2. Retrieve the index metadata JSON using ES admin API,  
  3. Increase the number of replicas to 16 (102 shards total),
  4. Retrieve the index metadata JSON using ES admin API,
  5. Compare the two JSON documents from 2 and 4. 

As state above it is desirable to minimize the number of indices. Each 
shard is a Lucene index which consumes memory and requires open file 
descriptors from the OS for segment data files and Lucene index level 
files. You may find yourself running out of memory and/or file descriptors 
if you are not careful. 

I understand you are looking for a design that will cater for on disc data 
volume. Given that your data is split into monthly indices it may well be 
that no one index, either "shared" or "dedicated" will reach that volume in 
one month. There may also be seasonal factors to consider whereby one or 
two months have much higher volumes than others. I have read/heard about 
cases where a monthly index architecture was implemented but later scraped 
for a single index approach because the month-to-month variation in volume 
was detrimental to overall system resource utilisation and performance.

In you case think about whether monthly indices are really appropriate. An 
alternative model is to partition one years worth of data into a set of 
indices bounded by size rather than time. In this model a new index is 
started on Jan 01 and data is added to it until it reaches some predefined 
size limit, at which point a new index is created to accept new data from 
that

Re: Help setting up logging

The logging configuration specifies how and what to log, but it does not
specify when or what actually constitutes a slow query/index. Not all
queries/index requests are logged, just the slow ones. You need to define
the threshold in the main elasticsearch.yml config file.

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-slowlog.html

-- 
Ivan


On Thu, Mar 20, 2014 at 6:05 PM, Raphael Miranda
wrote:

> ES is creating the log files upon startup but they are empty? I switched
> every log level to DEBUG and it started pouring more log into
> elasticsearch.log still, no query or indexing is logged.
>
> -rw-r--r-- 1 elasticsearch elasticsearch 0 Mar 21 00:54
> elasticsearch_index_indexing_slowlog.log
> -rw-r--r-- 1 elasticsearch elasticsearch 0 Mar 21 00:54
> elasticsearch_index_search_slowlog.log
> -rw-r--r-- 1 elasticsearch elasticsearch 83910 Mar 21 01:00
> elasticsearch.log
>
> Heres my logging.yml
>
> # you can override this using by setting a system property, for example
> -Des.logger.level=DEBUG
> es.logger.level: DEBUG
> rootLogger: ${es.logger.level}, console, file
> logger:
>   # log action execution errors for easier debugging
>   action: DEBUG
>   # reduce the logging for aws, too much is logged under the default INFO
>   com.amazonaws: DEBUG
>
>   # gateway
>   gateway: DEBUG
>   index.gateway: DEBUG
>
>   # peer shard recovery
>   indices.recovery: DEBUG
>
>   # discovery
>   discovery: DEBUG
>
>   index.search.slowlog: DEBUG, index_search_slow_log_file
>   index.indexing.slowlog: DEBUG, index_indexing_slow_log_file
>
> additivity:
>   index.search.slowlog: true
>   index.indexing.slowlog: true
> appender:
>   console:
> type: console
> layout:
>   type: consolePattern
>   conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
>
>   file:
> type: dailyRollingFile
> file: ${path.logs}/${cluster.name}.log
> datePattern: "'.'-MM-dd"
> layout:
>   type: pattern
>   conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
>
>   index_search_slow_log_file:
> type: dailyRollingFile
> file: ${path.logs}/${cluster.name}_index_search_slowlog.log
> datePattern: "'.'-MM-dd"
> layout:
>   type: pattern
>   conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
>
>   index_indexing_slow_log_file:
> type: dailyRollingFile
> file: ${path.logs}/${cluster.name}_index_indexing_slowlog.log
> datePattern: "'.'-MM-dd"
> layout:
>   type: pattern
>   conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/39a035f9-752c-47a5-9a5c-61d4aeb643ee%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBwy_yL_wif20GNtk3z_B%3Djt2%2BQCxxjQ-CCPh4WX9mqdw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

EC2 Discovery

2014-03-20 Thread Raphael Miranda

are both machines in the same security group?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/eb8bb939-3b9d-4f5b-a45c-3d529f75983e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Help setting up logging

2014-03-20 Thread Raphael Miranda

ES is creating the log files upon startup but they are empty? I switched 
every log level to DEBUG and it started pouring more log into 
elasticsearch.log still, no query or indexing is logged.

-rw-r--r-- 1 elasticsearch elasticsearch 0 Mar 21 00:54 
elasticsearch_index_indexing_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch 0 Mar 21 00:54 
elasticsearch_index_search_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch 83910 Mar 21 01:00 
elasticsearch.log

Heres my logging.yml

# you can override this using by setting a system property, for example 
-Des.logger.level=DEBUG
es.logger.level: DEBUG
rootLogger: ${es.logger.level}, console, file
logger:
  # log action execution errors for easier debugging
  action: DEBUG
  # reduce the logging for aws, too much is logged under the default INFO
  com.amazonaws: DEBUG

  # gateway
  gateway: DEBUG
  index.gateway: DEBUG

  # peer shard recovery
  indices.recovery: DEBUG

  # discovery
  discovery: DEBUG

  index.search.slowlog: DEBUG, index_search_slow_log_file
  index.indexing.slowlog: DEBUG, index_indexing_slow_log_file

additivity:
  index.search.slowlog: true
  index.indexing.slowlog: true
appender:
  console:
type: console
layout:
  type: consolePattern
  conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

  file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}.log
datePattern: "'.'-MM-dd"
layout:
  type: pattern
  conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

  index_search_slow_log_file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}_index_search_slowlog.log
datePattern: "'.'-MM-dd"
layout:
  type: pattern
  conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

  index_indexing_slow_log_file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}_index_indexing_slowlog.log
datePattern: "'.'-MM-dd"
layout:
  type: pattern
  conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/39a035f9-752c-47a5-9a5c-61d4aeb643ee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

EC2 Discovery

2014-03-20 Thread ZenMaster80

Any clues to what i am missing, i turned discovery trace on, but dont't see any 
useful info.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7f3dce1e-53d1-4c38-804f-6262896d43d6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Datadog metrics

2014-03-20 Thread Alexis Lê-Quôc

On Wednesday, March 19, 2014 1:30:44 PM UTC-4, Bjornar wrote:
>
> We've gotten the elasticsearch plugin for Datadog up and running and are 
> now looking through tons of very useful and some not-so-useful metrics. Has 
> anyone by chance gone through and annotated what the different metrics are 
> ? 
>
> Thanks,
> Bjornar
>

Hi Bjornar,

Internally (at Datadog) we have put together a dashboard that reveals what 
we think are useful metrics to monitor our internal clusters. I have put 
together the dashboard json description here 
(https://gist.github.com/alq666/9675918) so that you can re-import it in 
your account. I hope this is helpful.

Alexis

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5ee6682d-66d1-4145-98fc-53c5cb7cb301%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: how to modify term frequency formula?

You can provide your own similarity to be used at the field level, but
recent version of elasticsearch allows you to access the tf-idf values in
order to do custom scoring [1]. Also look at Britta's recent talk on the
subject [2].

That said, either your custom similarity or custom scoring would need
access to what exactly are the terms which are repeated many times. Have
you looked into omitting term frequencies? It would completely bypass using
term frequencies, which might be an overkill in your case. Look into the
index options [3].

Finally, perhaps the common terms query can help [4].

[1]
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-advanced-scripting.html

[2] https://speakerdeck.com/elasticsearch/scoring-for-human-beings

[3]
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#string

[4]
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-common-terms-query.html

Cheers,

Ivan


On Thu, Mar 20, 2014 at 8:08 AM, geantbrun  wrote:

> Hi,
> If I understand well, the formula used for the term frequency part in the
> default similarity module is the square root of the actual frequency. Is it
> possible to modify that formula to include something like a
> min(my_max_value,sqrt(frequency))? I would like to avoid huge tf's for
> documents that have the same term repeated many times. It seems that BM25
> similarity has a parameter to control saturation but I would prefer to
> stick with the simple tf/idf similarity module.
> Thank you for your help
> Patrick
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/9a12b611-d08d-41f9-8fd4-b74ad75a6a5c%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBKkA9-gBOYZau%3DDWn-O0f_XVqNmXJa67zSCnC1uLmV4A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Remember

2014-03-20 Thread war house

*Muslim population is now Greater than non-Muslims. If anywhere Muslims are
tortured by non-Muslims, they should revolt. Before be killed, Muslim must
kill at least 10 non-Muslims. Almighty ALLAH must help Muslims and make
them win. Remember, this work is compulsory in INDIA, MYANMAR, CENTRAL
AFRICAN REPUBLIC, PALESTINE, IRAQ, AFGANISTAN and others.*

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CA%2BkKsm4U7%3DULo%3DC5A%2BuV9TKs%2BSzJg9Fk8YwORpQg8VY5fZmRNA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Sort documents by inner json fields (string and date)

2014-03-20 Thread Dionysis Lorentzos

Assuming that this is my mapping:

{
   "event": {
  "properties": {
 "name": {
"type": "string",
"index": "analyzed",
"fields": {
   "raw": {
  "type": "string",
  "index": "not_analyzed"
   }
}
 },
 "more": {
"properties": {
   "title": {
  "type": "string",
  "index": "analyzed",
  "fields": {
 "raw": {
"type": "string",
"index": "not_analyzed"
 }
  }
   },
   "start_date": {
  "type": "date",
  "format": "-MM-dd'T'HH:mm:ssZ"
   }
}
 }
  }
   }
}



I can successfully get sorted results for `name.raw` like:

GET event/event/_search
> {
>"sort": [ {  "name.raw": "asc" } ]
> }


However I have hard time when using `more.title.raw` and `more.start_date` 
as the results will display null in the returned sort field e.g.:

  "hits": [
 {
"_index": "event",
"_type": "event",
"_id": "52f70fb1c36b446c340e93bf",
"_score": null,
"_source": {
   "name": "S. Walter Stewart Library",  
   "more": {
  "title": "Parent Led Sing Along Group", 
  "start_date": "2014-02-12T20:15:00+00:00",   
   
   }
},
"sort": [
   null
]
 },
 
  ]


Is it possible with that schema to sort 1) by `more.title` 2) by 
`more.start_date` ?+how

Best,
D


PS: I guess I can also use `"format":"date_time_no_millis"` in 
`"start_date"` field.





-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/065e61bc-f9e3-49aa-b53e-ba0f73780717%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Yet another "WARNING: Message not fully read" thread

2014-03-20 Thread tufi

Hi Jörg, 

yes, more or less... The full warning line is: 

Mrz 20, 2014 11:55:11 AM org.elasticsearch.transport.netty
WARNING: [Stick] Message not fully read (response) for [5507] handler future
(org.elasticsearch.client.transport.
TransportClientNodesService$SimpleNodeSampler$1@4893ccd0), error 
[false],resetting

This is happening in the client. 
I do use elasticsearch-head and HQ plug-ins installed on the master. 

Thanks. 
Tufi.


Am Donnerstag, 20. März 2014 20:28:55 UTC+1 schrieb Jörg Prante:
>
> Thanks for the very commendable description. 
>
> Do you have also a full stack trace of the "Message not fully read" 
> exception? Is it happening in the client?
>
> Do you use any plugins?
>
> Jörg
>
>
>
>
> On Thu, Mar 20, 2014 at 3:37 PM, tufi >wrote:
>
>> Hi, 
>>
>> also I am getting the warning "Message not fully read" from one of my ES 
>> Nodes.
>> I certainly googled a lot and found out that, different versions of JVM 
>> as well as different versions of ES server and clients can cause this. 
>>
>> Well, I'm pretty sure, I have the same versions everywhere. 
>>
>> My use case: 
>> Programming a multi-thread java-app to import about 30G data out of 
>> approx. 1000 csv files to Elasticsearch. I use transport client for that, 
>> to send bulkImport requests with <10.000 bulk objects for each request.
>>
>> ES Cluster: 
>> *1 Master: *4 Cores + 16 GB RAM
>> Overwritten properties in elasticsearch.yml
>>
>> cluster.name: $CLUSTERNAME
>> node.name: $NODENAME
>> node.master: true
>> #node.data: false
>> node.max_local_storage_nodes: 1
>> index.number_of_shards: 8 # 4 x $numberOfNodes since I have 2 data nodes 
>> 8
>> index.number_of_replicas: 1
>>
>>
>>
>> Versions: 
>>
>> $bin/elasticsearch -v
>> Version: 1.0.0, Build: a46900e/2014-02-12T16:18:34Z, JVM: 1.7.0_25
>>
>>
>> *1st Slave*: 4 Cores + 16 GB RAM
>> Overwritten properties in elasticsearch.yml
>>
>> cluster.name: $CLUSTERNAME
>> node.name: $NODENAME
>> node.master: false
>>
>> node.max_local_storage_nodes: 1
>>
>> # thread pool
>> threadpool.bulk.queue_size: -1
>>
>> path.data: /opt/elasticsearch/data
>> path.plugins: /opt/elasticsearch/plugins
>> path.conf: /opt/elasticsearch/config
>>
>>
>>
>> Versions: 
>> $bin/elasticsearch -v
>> Version: 1.0.0, Build: a46900e/2014-02-12T16:18:34Z, JVM: 1.7.0_25
>>
>>
>>
>> *2nd Slave*: 4 Cores + 16 GB RAM
>> Overwritten properties in elasticsearch.yml
>>
>> cluster.name: $CLUSTERNAME
>> node.name: $NODENAME
>> node.master: false
>>
>> node.max_local_storage_nodes: 1
>>
>> # thread pool
>> threadpool.bulk.queue_size: -1
>>
>> path.data: /opt/elasticsearch/data
>> path.plugins: /opt/elasticsearch/plugins
>> path.conf: /opt/elasticsearch/config
>>
>>
>>
>> Versions: 
>> $bin/elasticsearch -v
>> Version: 1.0.0, Build: a46900e/2014-02-12T16:18:34Z, JVM: 1.7.0_25
>>
>>
>> *My Client: *4 Cores + 16 GB RAM
>> pom.xml
>> 
>>   1.0.0
>> 
>> 
>> 
>> org.elasticsearch
>> elasticsearch
>> ${es.version}
>> 
>>
>> Maven: org.elasticsearch:elasticsearch:1.0.0 
>> is indeed under libs/ :
>>
>> elasticsearch-1.0.0.jar
>>
>> Java Code to initialize the transport client: 
>> 
>> XMLConfiguration config = ConfigLoader.getInstance().getConfig();
>> config.setExpressionEngine(new XPathExpressionEngine());
>> HierarchicalConfiguration hConf = config.configurationAt(
>> "/database/es/node[@master='true']");
>>
>> String clusterName = hConf.getString("clusterName");
>> String url = hConf.getString("url");
>> int port = hConf.getInt("port/transport");
>>
>> System.out.println("clusterName = " + clusterName);
>> Settings settings = ImmutableSettings.settingsBuilder()
>> .put("cluster.name", clusterName).build();
>> client = new TransportClient(settings).addTransportAddress(
>> new InetSocketTransportAddress(url, port));
>>
>> Last but not least; java version on App-Runner: 
>>
>> System.out.println(System.getProperty("java.version"));
>>
>> 1.7.0_25
>>
>>
>> Is there anything else, which I can check for getting rid of these 
>> warnings? 
>> What exactly does this warning mean? Do I lose data?
>>
>> Since I'm new to ES; do I have to set the ES config values 
>> *node.max_local_storage_nodes, 
>> index.number_of_shards, index.number_of_replicas* also in the slave 
>> configs? 
>>
>> Would you need additional information?
>>
>> Thank you very much for your help. 
>>
>> Tufi.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/c64630fb-768f-47c8-9225-4d1290580703%40googlegroups.com

Re: Optimal number of Shards per node

2014-03-20 Thread Zachary Tong

Unfortunately, there is no way that we can tell you an optimal number.  But 
there is a way that you can perform some capacity tests, and arrive at 
usable numbers that you can extrapolate from.  The process is very simple:


   - Create a single index, with a single shard, on a single 
   production-style machine
   - Start indexing *real, production-style *data.  "Fake" or "dummy" data 
   won't work here, it needs to mimic real-world data
   - Periodically, run real-world queries that you would expect users to 
   enter
   - At some point, you'll find that performance is no longer acceptable to 
   you.  Perhaps the indexing rate becomes too slow.  Or perhaps query latency 
   is too slow.  Or perhaps your node just runs out of memory
   - Write down the number of documents in the shard, and the physical size 
   of the shard

Now you know the limit of a single shard given your hardware + queries + 
data.  Using that knowledge, you can extrapolate given your expected 
search/indexing load, and how many documents you expect to index over the 
next few years, etc.

-Zach



On Thursday, March 20, 2014 3:29:47 PM UTC-5, Rajan Bhatt wrote:
>
> Hello,
>
> I would appreciate if someone can suggest optimal number of shards per ES 
> node for optimal performance or any recommended way to arrive at number of 
> shards given number of core and memory foot print.
>
> Thanks in advance
> Reagards
> Rajan
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/94bd5180-1198-4cfd-9b3d-f532d3fea5d2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Optimal number of Shards per node

2014-03-20 Thread Rajan Bhatt

Hello,

I would appreciate if someone can suggest optimal number of shards per ES 
node for optimal performance or any recommended way to arrive at number of 
shards given number of core and memory foot print.

Thanks in advance
Reagards
Rajan

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/57774626-f484-48c6-9b84-408df9ced896%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Yet another "WARNING: Message not fully read" thread

2014-03-20 Thread joergpra...@gmail.com

Thanks for the very commendable description.

Do you have also a full stack trace of the "Message not fully read"
exception? Is it happening in the client?

Do you use any plugins?

Jörg




On Thu, Mar 20, 2014 at 3:37 PM, tufi  wrote:

> Hi,
>
> also I am getting the warning "Message not fully read" from one of my ES
> Nodes.
> I certainly googled a lot and found out that, different versions of JVM as
> well as different versions of ES server and clients can cause this.
>
> Well, I'm pretty sure, I have the same versions everywhere.
>
> My use case:
> Programming a multi-thread java-app to import about 30G data out of
> approx. 1000 csv files to Elasticsearch. I use transport client for that,
> to send bulkImport requests with <10.000 bulk objects for each request.
>
> ES Cluster:
> *1 Master: *4 Cores + 16 GB RAM
> Overwritten properties in elasticsearch.yml
>
> cluster.name: $CLUSTERNAME
> node.name: $NODENAME
> node.master: true
> #node.data: false
> node.max_local_storage_nodes: 1
> index.number_of_shards: 8 # 4 x $numberOfNodes since I have 2 data nodes 8
> index.number_of_replicas: 1
>
>
>
> Versions:
>
> $bin/elasticsearch -v
> Version: 1.0.0, Build: a46900e/2014-02-12T16:18:34Z, JVM: 1.7.0_25
>
>
> *1st Slave*: 4 Cores + 16 GB RAM
> Overwritten properties in elasticsearch.yml
>
> cluster.name: $CLUSTERNAME
> node.name: $NODENAME
> node.master: false
>
> node.max_local_storage_nodes: 1
>
> # thread pool
> threadpool.bulk.queue_size: -1
>
> path.data: /opt/elasticsearch/data
> path.plugins: /opt/elasticsearch/plugins
> path.conf: /opt/elasticsearch/config
>
>
>
> Versions:
> $bin/elasticsearch -v
> Version: 1.0.0, Build: a46900e/2014-02-12T16:18:34Z, JVM: 1.7.0_25
>
>
>
> *2nd Slave*: 4 Cores + 16 GB RAM
> Overwritten properties in elasticsearch.yml
>
> cluster.name: $CLUSTERNAME
> node.name: $NODENAME
> node.master: false
>
> node.max_local_storage_nodes: 1
>
> # thread pool
> threadpool.bulk.queue_size: -1
>
> path.data: /opt/elasticsearch/data
> path.plugins: /opt/elasticsearch/plugins
> path.conf: /opt/elasticsearch/config
>
>
>
> Versions:
> $bin/elasticsearch -v
> Version: 1.0.0, Build: a46900e/2014-02-12T16:18:34Z, JVM: 1.7.0_25
>
>
> *My Client: *4 Cores + 16 GB RAM
> pom.xml
> 
>   1.0.0
> 
> 
> 
> org.elasticsearch
> elasticsearch
> ${es.version}
> 
>
> Maven: org.elasticsearch:elasticsearch:1.0.0
> is indeed under libs/ :
>
> elasticsearch-1.0.0.jar
>
> Java Code to initialize the transport client:
>
> XMLConfiguration config = ConfigLoader.getInstance().getConfig();
> config.setExpressionEngine(new XPathExpressionEngine());
> HierarchicalConfiguration hConf = config.configurationAt(
> "/database/es/node[@master='true']");
>
> String clusterName = hConf.getString("clusterName");
> String url = hConf.getString("url");
> int port = hConf.getInt("port/transport");
>
> System.out.println("clusterName = " + clusterName);
> Settings settings = ImmutableSettings.settingsBuilder()
> .put("cluster.name", clusterName).build();
> client = new TransportClient(settings).addTransportAddress(
> new InetSocketTransportAddress(url, port));
>
> Last but not least; java version on App-Runner:
>
> System.out.println(System.getProperty("java.version"));
>
> 1.7.0_25
>
>
> Is there anything else, which I can check for getting rid of these
> warnings?
> What exactly does this warning mean? Do I lose data?
>
> Since I'm new to ES; do I have to set the ES config values 
> *node.max_local_storage_nodes,
> index.number_of_shards, index.number_of_replicas* also in the slave
> configs?
>
> Would you need additional information?
>
> Thank you very much for your help.
>
> Tufi.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/c64630fb-768f-47c8-9225-4d1290580703%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH_cQvgf3SWM%2BJEPLzEdJoAw8Lfp2c-OEG0mtQKTn5Z-g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

EC2 Discovery

I can't seem to make my EC2 cluster of 2 nodes/instances work.

- If I comment out the section below, I can connect to each instance 
individually and query it, I am using Sense plugin to query Elastic Search 
Instance.

- With the sections included as below, I cannot query neither node, I get 
"Request failed to get to the server (status code: 0):"


 In /instance 1/ elasticsearch.yml
cluster.name: mycluster
node.name: "node_1"

cloud:

   aws:

   access_key: 

   secret_key: 

discovery:

   type: ec2

 In /instance 2/ elasticsearch.yml
cluster.name: mycluster
node.name: "node_2"

cloud:

  aws:

  access_key: 

  secret_key: 

discovery:

  type: ec2


I also tried with:

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts: ["Instance_1 IP:9300", "instance_2 
IP:9300"]


Here is "mycluster.log" from both instances:

$ more /var/log/elasticsearch/mycluster.log 

[2014-03-20 19:00:37,635][INFO ][node ] [node_1] version
[0.90.10], pid[3520], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:00:37,635][INFO ][node ] 
[node_1]initializing 
...

[2014-03-20 19:00:37,698][INFO ][plugins  ] [node_1] loaded 
[mapper-attachments, cloud-aws], sites []

[2014-03-20 19:01:17,898][INFO ][node ] [node_1] version
[0.90.10], pid[3594], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:01:17,898][INFO ][node ] 
[node_1]initializing 
...

[2014-03-20 19:01:17,961][INFO ][plugins  ] [node_1] loaded 
[mapper-attachments, cloud-aws], sites []

[2014-03-20 19:03:50,048][INFO ][node ] [node_1] version
[0.90.10], pid[3671], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:03:50,048][INFO ][node ] 
[node_1]initializing 
...

[2014-03-20 19:03:50,111][INFO ][plugins  ] [node_1] loaded 
[mapper-attachments, cloud-aws], sites []


$more /var/log/elasticsearch/mycluster.log 

[2014-03-20 19:00:29,465][INFO ][node ] [node_2] version
[0.90.10], pid[2800], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:00:29,466][INFO ][node ] 
[node_2]initializing 
...

[2014-03-20 19:00:29,528][INFO ][plugins  ] [node_2] loaded 
[mapper-attachments, cloud-aws], sites []

[2014-03-20 19:01:26,645][INFO ][node ] [node_2] version
[0.90.10], pid[2874], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:01:26,646][INFO ][node ] 
[node_2]initializing 
...

[2014-03-20 19:01:26,709][INFO ][plugins  ] [node_2] loaded 
[mapper-attachments, cloud-aws], sites []

[2014-03-20 19:03:58,357][INFO ][node ] [node_2] version
[0.90.10], pid[2951], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:03:58,357][INFO ][node ] 
[node_2]initializing 
...

[2014-03-20 19:03:58,420][INFO ][plugins  ] [node_2] loaded 
[mapper-attachments, cloud-aws], sites []



-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/69b43625-f560-4e42-a987-2ba2310d462b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Creating dynamic fields from a field

2014-03-20 Thread Pablo Musa

Ok, I tried all the hints but now I have can't solve my original problem.

I need to do an update to a value of type custom_foo.
In my previous approach I would do ctx._source.custom_foo.value+=1.
But now, there is a vector and I dont know which index is custom_foo.

Is there any fast method to get an nested object by value?

Thanks,
Pablo



2014-03-17 16:29 GMT-03:00 Ivan Brusic :

> There is an REST API:
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-analyze.html
>
> If you are using Java, you can avoid the network roundtrip by creating the
> AnalysisService locally. For hints, see the test class:
> https://github.com/elasticsearch/elasticsearch/blob/master/src/test/java/org/elasticsearch/index/analysis/AnalysisModuleTests.java
>
> --
> Ivan
>
>
> On Mon, Mar 17, 2014 at 12:11 PM,  wrote:
>
>> Thank you very much for the hints ;)
>>
>>
>> > If it matters, you can call the _analyze API to produce the tokens
>> before you inject your fields.
>>
>> Is there an URL there I can call? Or only using the internal API?
>>
>>
>> > Your cluster state will eventually become enormous.
>>
>> Yes, I saw it coming but was postponing in dev phase. Thanks for the
>> solution, it will help very much!!
>>
>> Thanks again guys!
>>
>> --Pablo
>>
>>
>> On Saturday, March 15, 2014 8:59:09 AM UTC-3, Clinton Gormley wrote:
>>
>>> To add to what Binh said, you really shouldn't add field names like this:
>>> On 14 March 2014 21:20, Pablo Musa  wrote:
>>>
 {
   "title":"The greatest band ever - Urban Legion",
   "greatest_x" : 1,
   "band_x" : 1,
   "ever_x" : 1,
   "Urban_x": 1,
   "Legion_x" : 1,
   "greatest_y" : [],
   "band_y" : [],
   "ever_y" : [],
   "Urban_y": [],
   "Legion_y" : []
 }

>>>
>>> You end up with an explosion of fields, and each field has an inverted
>>> index associated with it.  Your cluster state will eventually become
>>> enormous. Any change to the cluster state (eg adding a field, changing an
>>> index, changes to nodes etc) results in the cluster state being copied to
>>> every node in the cluster. If the state is very large you will experience a
>>> significant slow down.
>>>
>>> Instead of:
>>>
>>> [{ custom_foo: xxx }, { custom_bar: yyy }]
>>>
>>> Use nested fields with eg
>>> [
>>>{ type: "custom_foo", value: "xxx" },
>>>{ type: "custom_bar", value: "yyy" }
>>> ]
>>>
>>> That way you have only two fields.
>>>
>>> clint
>>>
>>>   --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/a5e3f166-7c40-4bdf-b1cc-4854abdb4595%40googlegroups.com
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/TYLV9Leqfg8/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAuVXV9Cu_XY5sVuwU3J04iQcC%3Durw2tHnZRbdPG4WBxg%40mail.gmail.com
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAF6PhFJjiiTEKt-Y0vWOd9dVDQNtywbDiWE%3Do%3D_hgTLRBbV%3DKg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: EC2 Discovery



/etc/elasticsearch$ more /var/log/elasticsearch/mycluster.log 

[2014-03-20 19:00:37,635][INFO ][node ] [node_1] version
[0.90.10], pid[3520], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:00:37,635][INFO ][node ] 
[node_1]initializing 
...

[2014-03-20 19:00:37,698][INFO ][plugins  ] [node_1] loaded 
[mapper-attachments, cloud-aws], sites []

[2014-03-20 19:01:17,898][INFO ][node ] [node_1] version
[0.90.10], pid[3594], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:01:17,898][INFO ][node ] 
[node_1]initializing 
...

[2014-03-20 19:01:17,961][INFO ][plugins  ] [node_1] loaded 
[mapper-attachments, cloud-aws], sites []

[2014-03-20 19:03:50,048][INFO ][node ] [node_1] version
[0.90.10], pid[3671], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:03:50,048][INFO ][node ] 
[node_1]initializing 
...

[2014-03-20 19:03:50,111][INFO ][plugins  ] [node_1] loaded 
[mapper-attachments, cloud-aws], sites []




 more /var/log/elasticsearch/mycluster.log 

[2014-03-20 19:00:29,465][INFO ][node ] [node_2] version
[0.90.10], pid[2800], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:00:29,466][INFO ][node ] 
[node_2]initializing 
...

[2014-03-20 19:00:29,528][INFO ][plugins  ] [node_2] loaded 
[mapper-attachments, cloud-aws], sites []

[2014-03-20 19:01:26,645][INFO ][node ] [node_2] version
[0.90.10], pid[2874], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:01:26,646][INFO ][node ] 
[node_2]initializing 
...

[2014-03-20 19:01:26,709][INFO ][plugins  ] [node_2] loaded 
[mapper-attachments, cloud-aws], sites []

[2014-03-20 19:03:58,357][INFO ][node ] [node_2] version
[0.90.10], pid[2951], build[0a5781f/2014-01-10T10:18:37Z]

[2014-03-20 19:03:58,357][INFO ][node ] 
[node_2]initializing 
...

[2014-03-20 19:03:58,420][INFO ][plugins  ] [node_2] loaded 
[mapper-attachments, cloud-aws], sites []















On Thursday, March 20, 2014 2:50:46 PM UTC-4, IronMan2014 wrote:
>
>  can't seem to make my EC2 cluster of 2 nodes work.
>
> - If I comment out the section below, I can connect to each instance 
> individually and query it.
> - With the sections included as below, I cannot query neither node, I get 
> "Request failed to get to the server (status code: 0):"
>
>
> cluster.name: myCluster
> node.name: "node 1"
>
> cloud:
>
>aws:
>
>access_key: 
>
>secret_key: 
>
> discovery:
>
>type: ec2
>
> 
> cluster.name: myCluster
> node.name: "node 2"
>
> cloud:
>
>   aws:
>
>   access_key: 
>
>   secret_key: 
>
> discovery:
>
>   type: ec2
>
>
> I also tried with:
>
>  discovery.zen.ping.multicast.enabled: false
>
> discovery.zen.ping.unicast.hosts: ["Instance_1 IP:9300", "instance_2 
> IP:9300"]
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a73dfef8-23e6-4c6a-a780-0b7722ae45b3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

EC2 Discovery

 can't seem to make my EC2 cluster of 2 nodes work.

- If I comment out the section below, I can connect to each instance 
individually and query it.
- With the sections included as below, I cannot query neither node, I get 
"Request failed to get to the server (status code: 0):"


cluster.name: myCluster
node.name: "node 1"

cloud:

   aws:

   access_key: 

   secret_key: 

discovery:

   type: ec2


cluster.name: myCluster
node.name: "node 2"

cloud:

  aws:

  access_key: 

  secret_key: 

discovery:

  type: ec2


I also tried with:

 discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts: ["Instance_1 IP:9300", "instance_2 
IP:9300"]

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/22a0c3cc-dc6f-46a5-926b-878cec6f6c79%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: elasticsearch error of class not found

On the same page I gave you, you have a chapter "configuration".
I think you should at least read it. 

Also on the guide, there is a search bar. Very useful to search for some 
information. Try with "network".

I'm sure you will find information you are looking for.

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 20 mars 2014 à 18:26, Anikessh Jain  a écrit :

thnks for the info

how to run the service on ipv4 bcoz when i am starting ti is starting on ipv6 
instead on ipv4 ,how do i change that 

> On Thursday, March 20, 2014 10:30:35 PM UTC+5:30, David Pilato wrote:
> I don't know.
> 
> So let me help you for elasticsearch:
> 
> wget 
> https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.0.1.tar.gz
> tar xzf elasticsearch-1.0.1.tar.gz
> ./elasticsearch-1.0.1/bin/elasticsearch
> 
> And that's all!
> This is basically what is written here: 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html#setup-installation
> 
> 
> -- 
> David Pilato | Technical Advocate | Elasticsearch.com
> @dadoonet | @elasticsearchfr
> 
> 
>> Le 20 mars 2014 à 17:49:49, Anikessh Jain (anikess...@gmail.com) a écrit:
>> 
>> Hi David,
>> 
>> But why do people download and install it through 
>> elasticsearch-servicewrapper ,does the leatest version supports it all bcoz 
>> i want to install kibana with logstash and redis with elasticsearch and i am 
>> a little bit confused will it work bcoz i have wasted my whole day on on 
>> error.please guide me what to do as i dont know what to do and doing it for 
>> first time
>> 
>>> On Thursday, March 20, 2014 10:11:52 PM UTC+5:30, David Pilato wrote:
>>> First download a recent version. If you are starting with elasticsearch, 
>>> you don't need to start with a so old version.
>>> 1.0.1 is fine.
>>> 
>>> Then read the documentation here: 
>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html
>>> 
>>> No need to install so many pieces.
>>> 
>>> Unzip and launch. That's all.
>>> 
>>> 
>>> -- 
>>> David Pilato | Technical Advocate | Elasticsearch.com
>>> @dadoonet | @elasticsearchfr
>>> 
>>> 
 Le 20 mars 2014 à 17:06:12, Anikessh Jain (anikess...@gmail.com) a écrit:
 
 please help on the above error 
 
> On Thursday, March 20, 2014 9:03:46 PM UTC+5:30, Anikessh Jain wrote:
>> wget 
>> https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.2.tar.gz
>> tar xvf elasticsearch-0.20.2.tar.gz
>> 
>> mv elasticsearch-0.20.2 elasticsearch
>> 
>> wget 
>> http://github.com/elasticsearch/elasticsearch-servicewrapper/archive/master.zip
>> unzip master
>> mv elasticsearch-servicewrapper-master/service/ elasticsearch/bin
>> rm -rf master
>> rm -rf elasticsearch-servicewrapper-master/
>> 
>> 
>> Configure ‘elasticsearch.conf’
>> 
>> set.default.ES_HOME=/usr/local/elasticsearch
>> set.default.ES_HEAP_SIZE=4096
>> wrapper.java.additional.10=-Des.max-open-files=true
>> 
>> wrapper.logfile.maxsize=5m
>> wrapper.logfile.maxfiles=5
>> 
>> 
>> Add ES home to root user’s ‘.bash_profile’
>> 
>> # ElasticSearch
>> export ES_HOME=/usr/local/elasticsearch
>> 
>> 
>> Create elasticsearch user
>> 
>> useradd -d /home/elasticsearch -s /bin/sh elasticsearch
>> chown -R elasticsearch:elasticsearch $ES_HOME
>> chown -R elasticsearch:elasticsearch /mnt/storage/es-data
>> 
>> 
>> Edit elasticsearch user’s ‘.bash_profile’
>> 
>> # .bash_profile
>> 
>> # Get the aliases and functions
>> if [ -f ~/.bashrc ]; then
>> . ~/.bashrc
>> fi
>> 
>> # User specific environment and startup programs
>> 
>> PATH=$PATH:$HOME/bin
>> 
>> export PATH
>> 
>> # JAVA_HOME needs to be the latest 1.7 JDK on the system
>> JAVA_HOME=/usr/local/jdk7
>> export JAVA_HOME
>> 
>> #Add JAVA_HOME to the PATH
>> PATH=$JAVA_HOME/bin:$PATH
>> 
>> # ElasticSearch
>> export ES_HOME=/usr/local/elasticsearch
>> 
>> unset USERNAME
>> 
>> 
>> Edit ‘/etc/security/limits.conf’ (optional as this will be set in the 
>> service script, too)
>> 
>> elasticsearchsoftnofile  65535
>> elasticsearchhardnofile  65535
>> 
>> 
>> Verify the file descriptor limit
>> 
>> sudo -u elasticsearch -s ulimit -Sn
>> 
>> 
>> Install the service
>> 
>> bin/service/elasticsearch install
>> 
>> Edit ‘/etc/init.d/elasticsearch’
>> # Java
>> JAVA_HOME=/usr/local/jdk7
>> export JAVA_HOME
>> PATH=$JAVA_HOME/bin:$PATH
>> RUN_AS_USER=elasticsearch
>> ULIMIT_N=65535
>> 
>> Run the service
>> bin/service/elasticsearch start
>> or
>> service elasticsearch start   -> here the above error i am getting can 
>> you please help me
> ple

Chef cookbook tutorial: What AWS permissions are needed to run this cookbook?

2014-03-20 Thread Dan Testa

Hi,

I am following the elasticsearch chef cookbook tutorial here:
http://www.elasticsearch.org/tutorials/deploying-elasticsearch-with-chef-solo/

I am getting stuck on this step:

time ssh -t $SSH_OPTIONS $HOST "sudo chef-solo --node-name elasticsearch-test-1 
-j /tmp/node.json"

Here is the error I am getting.  It seems that my AWS key is lacking the 
necessary permissions to create an EBS volume.  I added "ec2:CreateVolume" 
and "ec2:AttachVolume" to no avail.  I cannot find any info on what 
permissions I need to run this.  Can anyone point me to this info?

[testadadmin@uspetddebspoc01 elasticsearch-test]$ time ssh -t $SSH_OPTIONS 
$HOST "sudo chef-solo --node-name elasticsearch-test-1 -j /tmp/node.json"
Warning: Permanently added 
'ec2-23-20-247-88.compute-1.amazonaws.com,23.20.247.88' (RSA) to the list 
of known hosts.
Starting Chef Client, version 11.6.0
Compiling Cookbooks...
Recipe: elasticsearch::ebs
  * package[gcc] action upgrade (up to date)
  * package[gcc-c++] action upgrade (up to date)
  * package[make] action upgrade (up to date)
  * package[libxslt-devel] action upgrade (up to date)
  * package[libxml2-devel] action upgrade (up to date)
  * chef_gem[fog] action install (up to date)
[2014-03-20T18:29:18+00:00] WARN: Cloning resource attributes for 
directory[/usr/local/var/data/elasticsearch/disk1] from prior resource 
(CHEF-3694)
[2014-03-20T18:29:18+00:00] WARN: Previous 
directory[/usr/local/var/data/elasticsearch/disk1]: 
/var/chef-solo/cookbooks/elasticsearch/recipes/data.rb:16:in `block in 
from_file'
[2014-03-20T18:29:18+00:00] WARN: Current 
 directory[/usr/local/var/data/elasticsearch/disk1]: 
/var/chef-solo/cookbooks/elasticsearch/recipes/data.rb:39:in `block in 
from_file'
Converging 50 resources
Recipe: monit::default
  * package[monit] action install (up to date)
  * service[monit] action start (up to date)
  * template[/etc/monit/monitrc] action create (up to date)
Recipe: elasticsearch::curl
  * package[curl] action install (up to date)
Recipe: ark::default
  * package[unzip] action install (up to date)
  * package[libtool] action install (up to date)
  * package[autoconf] action install (up to date)
Recipe: elasticsearch::default
  * group[elasticsearch] action create (up to date)
  * user[elasticsearch] action create (up to date)
  * bash[remove the elasticsearch user home] action run (skipped due to 
not_if)
  * directory[/usr/local/etc/elasticsearch] action create (up to date)
  * directory[/usr/local/var/log/elasticsearch] action create (up to date)
  * directory[/usr/local/var/run] action create (up to date)
  * directory[/usr/local/var/data/elasticsearch] action create (up to date)
  * template[/etc/init.d/elasticsearch] action create (up to date)
  * service[elasticsearch] action enable (up to date)
  * ark[elasticsearch] action install (skipped due to not_if)
  * bash[enable user limits] action run (skipped due to not_if)
  * log[increase limits for the elasticsearch user] action write

  * file[/etc/security/limits.d/10-elasticsearch.conf] action create (up to 
date)
  * template[elasticsearch-env.sh] action create (up to date)
  * template[elasticsearch.yml] action create (up to date)
  * template[logging.yml] action create (up to date)
Recipe: elasticsearch::plugins
  * directory[/usr/local/elasticsearch-0.90.12/plugins/] action create (up 
to date)
  * ruby_block[Install plugin: karmi/elasticsearch-paramedic] action run 
(skipped due to not_if)
Recipe: elasticsearch::ebs
  * package[gcc] action nothing (skipped due to action :nothing)
  * package[gcc-c++] action nothing (skipped due to action :nothing)
  * package[make] action nothing (skipped due to action :nothing)
  * package[libxslt-devel] action nothing (skipped due to action :nothing)
  * package[libxml2-devel] action nothing (skipped due to action :nothing)
  * chef_gem[fog] action install (up to date)
  * ruby_block[Create EBS volume on /dev/sda2 (size: 25GB)] action run

Error executing action `run` on resource 'ruby_block[Create EBS volume on 
/dev/sda2 (size: 25GB)]'



Fog::Compute::AWS::Error

UnauthorizedOperation => You are not authorized to perform this operation.


Cookbook Trace:
---
/var/chef-solo/cookbooks/elasticsearch/libraries/create_ebs.rb:41:in `block 
(2 levels) in create_ebs'


Resource Declaration:
-
# In /var/chef-solo/cookbooks/elasticsearch/libraries/create_ebs.rb

 16: ruby_block "Create EBS volume on #{device} (size: 
#{params[:ebs][:size]}GB)" do
 17: 
 18:   block do
 19: require 'fog'
 20: require 'open-uri'
 21: 
 22: region  = params[:region] || 
node.elasticsearch[:cloud][:aws][:region]
 23: instance_id = 
open('http://169.254.169.254/latest/meta-data/instance-id'){|f| f.gets}
 24: raise "[!] Cannot get insta

Autocomplete rest of query: Do not propose shorter queries?

2014-03-20 Thread Steven

Hi Everyone, 

I have put together some auto complete functionality based around the blog 
posted on ES page (http://www.elasticsearch.org/blog/you-complete-me/). I 
use the following to create the index.

PUT test_index
{
"mappings": {
"query" : {
  "properties" : {
"product" : { "type" : "string" },
"name_suggest" : {
  "type" : "completion"
}
  }
}
  }

Everything works fine except for 1 experience issue. That is for example if 
I search for xbox 360, xbox one is also considered a relevant suggestion. I 
would prefer if the autocomplete terms returned only contained the previous 
characters entered for the search. For example xbox 360 xbox 360 controller 
etc. The index contains a list of search terms. 

Can anyone suggest ways of doing this? Very unfamiliar with the ES API. 


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/eece9c6f-75be-4046-9af1-b279a23a3ed6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ElasticSearch to SQL sync

2014-03-20 Thread Andrew Cholakian

This is a complex issue with no single, simple, answer. Performing 
synchronization between two databases is a non-trivial undertaking for any 
technology. For an overview of strategies and techniques please see my 
article on the topic Keeping Elasticsearch in 
Sync
. 

---
Andrew Cholakian
Developer, Found Hosted Elasticsearch
www.found.no, @foundsays

On Thursday, March 20, 2014 2:29:30 AM UTC-5, Komal Parekh wrote:
>
> Hello,
>
> We are running one application which has very large amount of data 
> updating very frequently. And we are using SQL for any command like 
> update/delete/add records. And for querying part we are using 
> ElasticSearch. So is there any way which syncs my data very frequently say 
> in 5 to 10 seconds. Is there any push mechanism available with 
> ElasticSearch which pushes data from SQL to ES? And more what is the best 
> option for synching? Please reply urgently with the best option :). 
>
>
> Thanks
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/24c46bf2-910d-462a-b639-1f9586544dab%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

EC2 Discovery

I can't seem to make my EC2 cluster of 2 nodes work.

- If I comment out the section below, I can connect to each instance 
individually.
- With the sections included as below, I can connect to node 1, but node 2 
gives me "Request failed to get to the server (status code: 0):"


cluster.name: myCluster
node.name: "node 1"

cloud:

   aws:

   access_key: 

   secret_key: 

discovery:

   type: ec2


cluster.name: myCluster
node.name: "node 2"

cloud:

  aws:

  access_key: 

  secret_key: 

discovery:

  type: ec2


I also tried with:

 discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts: ["Instance_1 IP:9300", "instance_2 
IP:9300"]


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2877c25a-c2f9-4f26-96b4-1b02ba1c3fd0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES attempting to parse dates automatically (possibly Hadoop related)

Sure - take a look at dynamic_templates - you define one under your index/type
and specify the match for your field.
You can either define the mapping for the fields that you want and leave the so-called catch-all (*) directive last or,
if you have type of naming convention, use that:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-root-object-type.html#_dynamic_templates

On 3/20/14 6:32 PM, Brian Stempin wrote:

That's the problem -- it's a web log that contains a URL that could have
literally anything in it. Anyone could put a
base64 value as a random query parameter. I could have the M/R job ignore all
fields that I don't explicitly expect,
but that's not very flexible and prevents me from spotting possible abuse or
user-error. Is there any way for me to
disable ES's type-guessing or to provide a default guess? I'd rather have ES
default to a string than to fail a M/R job
because its type-guess was wrong.

Brian

On Thu, Mar 20, 2014 at 12:26 PM, Costin Leau mailto:costin.l...@gmail.com>> wrote:

Then what you could do is to minimize the bulk size to say 100 documents,
turn on logging and run your data through.
This way you can catch the 'special' document in the act.

As for expectations - Elasticsearch tries to guess the field type by
looking at its value - it seems the base64
entry looks like a date, hence the error. You can avoid this by defining
the field (either directly or through a
template) in your mapping so it always gets mapped to a string.
As a rule of thumb, whenever you want full control over the index, mapping
is the way to do it.

On 3/20/14 6:10 PM, Brian Stempin wrote:

I have unit tests for this MR job, and they show that the JSON output
is a string as I'd expect, so Gson is most
likely
not the cause.

I'm hesitant to show more code (owned by the work-place), but I can
describe it a little bit further:

* The mapper gets a W3C log entry
* The log entry is broken into its components and put into document X
* The request URL is then taken and broken down into its query
parameters and the key-value pairs are put
into document Y
* Some elements are then explicitly filtered from X and Y
* Those two documents are placed inside of document Z, which is
ultimately what is serialized and sent to ES

We do have a base64 encoded param that we expect and handle
appropriately. In this case, someone most likely
sent it as
the wrong param name, hence why its making its way into document Y
without further processing. Since its being
sent as
a name that's not listed in the mapping, I expect it to just be treated
as a string.

The only reason that I chose to go the Gson route vs building
MapWritables is that building MapWritables is terribly
verbose. Also, it comes with the overhead of having to wrap each
String with a Text type, which just seems silly.
Using the built-in JSON serializer is just not convenient in this
case.

Brian

On Thu, Mar 20, 2014 at 11:18 AM, Costin Leau mailto:costin.l...@gmail.com>
>__> wrote:

My guess is that GSON adds the said field in its result. The
base64 suggests that there's some binary data
in the mix.

By the way, can you show up more of your code - any reason why you
create the JSON yourself rather than
just pass
logEntryMap to Es-Hadoop?
It can create the json for you - which is what I recommend; unless
you have the JSON in HDFS, it's best to
rely on
es-hadoop to do it instead of an external tool.

Cheers,

On 3/20/14 4:48 PM, Brian Stempin wrote:

Hi,
All I'm doing is building a map and passing that to Gson for
serialization. A snippet from my map method:

logEntryMap.put("cs(User-Agent)", values[9]);
context.write(NullWritable.get(), new
Text(gson.toJson(logEntryMap)));

values[] is a String array. Everything that goes into the map
that gets serialized is a string.

I do have es.input.json set to true. This failure doesn't occur
until >100,000,000 records are in the
index, so its
happening late in the load process. The part that I find
strange is that the field in question isn't in my
mapping, and
I've not touched the default mapping. I'm not sure why it
would try to parse it as anything other than
a string.

I'll turn on TRACE logging and see what happens.

Brian

On Wed, Mar 19, 2014 at 5:35 PM, Costin Leau mailto:costin.l...@gmail.com>

Re: Elasticsearch backend?

2014-03-20 Thread Joshua P

Thank you both! This is what I wanted to know.

On Thursday, March 20, 2014 1:05:00 PM UTC-4, Nikolas Everett wrote:
>
> You might be wondering if it is using Tomcat or Glassfish or something 
> too.  The answer is not usually.  There is a plugin that will let you 
> install it in a servlet container but most folks just run it as a 
> standalone service.  It has an init script and stuff.
>
>
> Nik
>
>
> On Thu, Mar 20, 2014 at 1:02 PM, Adrien Grand 
> 
> > wrote:
>
>> It depends on what you mean by serve, but transport is handled by Netty 
>> and storage by Lucene. On top of them and a few more libraries, 
>> Elasticsearch adds distribution management, search, percolation, 
>> aggregations, etc.
>>
>>
>> On Thu, Mar 20, 2014 at 3:51 PM, Joshua P 
>> > wrote:
>>
>>> What does Elasticsearch use to serve up responses?
>>>
>>> Thanks! 
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to elasticsearc...@googlegroups.com .
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/dce3e112-06fa-4e00-9279-59d88a676748%40googlegroups.com
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> -- 
>> Adrien Grand
>>  
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j50c4vkwyii1JkwecWYsQq6ZwjUNEueZyVBgC2fv39sRw%40mail.gmail.com
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7e08bccb-acfb-4411-9770-7b959a64259d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: elasticsearch error of class not found

thnks for the info

how to run the service on ipv4 bcoz when i am starting ti is starting on 
ipv6 instead on ipv4 ,how do i change that 

On Thursday, March 20, 2014 10:30:35 PM UTC+5:30, David Pilato wrote:
>
> I don't know.
>
> So let me help you for elasticsearch:
>
> wget 
> https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.0.1.tar.gz
> tar xzf elasticsearch-1.0.1.tar.gz
> ./elasticsearch-1.0.1/bin/elasticsearch
>
> And that's all!
> This is basically what is written here: 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html#setup-installation
>
>
> -- 
> *David Pilato* | *Technical Advocate* | *Elasticsearch.com*
> @dadoonet  | 
> @elasticsearchfr
>
>
> Le 20 mars 2014 à 17:49:49, Anikessh Jain (anikess...@gmail.com) 
> a écrit:
>
> Hi David, 
>
> But why do people download and install it through 
> elasticsearch-servicewrapper ,does the leatest version supports it all bcoz 
> i want to install kibana with logstash and redis with elasticsearch and i 
> am a little bit confused will it work bcoz i have wasted my whole day on on 
> error.please guide me what to do as i dont know what to do and doing it for 
> first time
>
> On Thursday, March 20, 2014 10:11:52 PM UTC+5:30, David Pilato wrote: 
>>
>>  First download a recent version. If you are starting with 
>> elasticsearch, you don't need to start with a so old version.
>>  1.0.1 is fine.
>>  
>>  Then read the documentation here: 
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html
>>  
>>  No need to install so many pieces.
>>  
>>  Unzip and launch. That's all.
>>  
>>
>>  -- 
>> *David Pilato* | *Technical Advocate* | *Elasticsearch.com* 
>>  @dadoonet  | 
>> @elasticsearchfr
>>  
>>
>> Le 20 mars 2014 à 17:06:12, Anikessh Jain (anikess...@gmail.com) a écrit:
>>
>>  please help on the above error 
>>
>> On Thursday, March 20, 2014 9:03:46 PM UTC+5:30, Anikessh Jain wrote: 
>>>
>>>   wget 
>>> https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.2.tar.gz
>>> tar xvf elasticsearch-0.20.2.tar.gz
>>>
>>> mv elasticsearch-0.20.2 elasticsearch
>>>
>>>wget 
>>> http://github.com/elasticsearch/elasticsearch-servicewrapper/archive/master.zip
>>> unzip master
>>> mv elasticsearch-servicewrapper-master/service/ elasticsearch/bin
>>> rm -rf master
>>> rm -rf elasticsearch-servicewrapper-master/
>>>
>>>
>>> Configure ‘elasticsearch.conf’
>>>
>>>  set.default.ES_HOME=/usr/local/elasticsearch
>>> set.default.ES_HEAP_SIZE=4096
>>> wrapper.java.additional.10=-Des.max-open-files=true
>>>
>>> wrapper.logfile.maxsize=5m
>>> wrapper.logfile.maxfiles=5
>>>
>>>  
>>> Add ES home to root user’s ‘.bash_profile’
>>>
>>>  # ElasticSearch
>>> export ES_HOME=/usr/local/elasticsearch
>>>
>>>  
>>> Create elasticsearch user
>>>
>>>  useradd -d /home/elasticsearch -s /bin/sh elasticsearch
>>> chown -R elasticsearch:elasticsearch $ES_HOME
>>> chown -R elasticsearch:elasticsearch /mnt/storage/es-data
>>>
>>>  
>>> Edit elasticsearch user’s ‘.bash_profile’
>>>
>>>  # .bash_profile
>>>
>>> # Get the aliases and functions
>>> if [ -f ~/.bashrc ]; then
>>> . ~/.bashrc
>>> fi
>>>
>>> # User specific environment and startup programs
>>>
>>> PATH=$PATH:$HOME/bin
>>>
>>> export PATH
>>>
>>> # JAVA_HOME needs to be the latest 1.7 JDK on the system
>>> JAVA_HOME=/usr/local/jdk7
>>> export JAVA_HOME
>>>
>>> #Add JAVA_HOME to the PATH
>>> PATH=$JAVA_HOME/bin:$PATH
>>>
>>> # ElasticSearch
>>> export ES_HOME=/usr/local/elasticsearch
>>>
>>> unset USERNAME
>>>
>>>  
>>> Edit ‘/etc/security/limits.conf’ (optional as this will be set in the 
>>> service script, too)
>>>
>>>  elasticsearchsoftnofile  65535
>>> elasticsearchhardnofile  65535
>>>
>>>  
>>> Verify the file descriptor limit
>>>
>>>  sudo -u elasticsearch -s ulimit -Sn
>>>
>>>  
>>> Install the service
>>>
>>>  bin/service/elasticsearch install
>>>
>>>  
>>> Edit ‘/etc/init.d/elasticsearch’
>>>
>>>  # Java
>>> JAVA_HOME=/usr/local/jdk7
>>> export JAVA_HOME
>>> PATH=$JAVA_HOME/bin:$PATH
>>> RUN_AS_USER=elasticsearch
>>> ULIMIT_N=65535
>>>
>>>  
>>> Run the service
>>>
>>>  bin/service/elasticsearch start
>>> or
>>> service elasticsearch start   -> here the above error i am getting can you 
>>> please help me
>>>
>>> please help me what wrong i have done   
>>>
>>>On Thursday, March 20, 2014 8:10:22 PM UTC+5:30, David Pilato wrote: 

  What did you download?

 --
 David ;-)
 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

  
 Le 20 mars 2014 à 14:54, Anikessh Jain  a écrit :

  nobody there to help me , help me please i am in need

 On Thursday, March 20, 2014 3:31:42 PM UTC+5:30, Anikessh Jain wrote: 
>
> Can anybody help me it is  a bit urgent .
>
>

Re: Elasticsearch backend?

2014-03-20 Thread Nikolas Everett

You might be wondering if it is using Tomcat or Glassfish or something
too.  The answer is not usually.  There is a plugin that will let you
install it in a servlet container but most folks just run it as a
standalone service.  It has an init script and stuff.


Nik


On Thu, Mar 20, 2014 at 1:02 PM, Adrien Grand <
adrien.gr...@elasticsearch.com> wrote:

> It depends on what you mean by serve, but transport is handled by Netty
> and storage by Lucene. On top of them and a few more libraries,
> Elasticsearch adds distribution management, search, percolation,
> aggregations, etc.
>
>
> On Thu, Mar 20, 2014 at 3:51 PM, Joshua P  wrote:
>
>> What does Elasticsearch use to serve up responses?
>>
>> Thanks!
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/dce3e112-06fa-4e00-9279-59d88a676748%40googlegroups.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Adrien Grand
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j50c4vkwyii1JkwecWYsQq6ZwjUNEueZyVBgC2fv39sRw%40mail.gmail.com
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0K0yDgzux8L4vcs-R1t-KxJenZvc5mSespFBAd4X8jGw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch backend?

2014-03-20 Thread Adrien Grand

It depends on what you mean by serve, but transport is handled by Netty and
storage by Lucene. On top of them and a few more libraries, Elasticsearch
adds distribution management, search, percolation, aggregations, etc.


On Thu, Mar 20, 2014 at 3:51 PM, Joshua P  wrote:

> What does Elasticsearch use to serve up responses?
>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/dce3e112-06fa-4e00-9279-59d88a676748%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adrien Grand

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j50c4vkwyii1JkwecWYsQq6ZwjUNEueZyVBgC2fv39sRw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: elasticsearch error of class not found

I don't know.

So let me help you for elasticsearch:

wget 
https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.0.1.tar.gz
tar xzf elasticsearch-1.0.1.tar.gz
./elasticsearch-1.0.1/bin/elasticsearch

And that's all!
This is basically what is written here: 
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html#setup-installation


-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr


Le 20 mars 2014 à 17:49:49, Anikessh Jain (anikesshjai...@gmail.com) a écrit:

Hi David,

But why do people download and install it through elasticsearch-servicewrapper 
,does the leatest version supports it all bcoz i want to install kibana with 
logstash and redis with elasticsearch and i am a little bit confused will it 
work bcoz i have wasted my whole day on on error.please guide me what to do as 
i dont know what to do and doing it for first time

On Thursday, March 20, 2014 10:11:52 PM UTC+5:30, David Pilato wrote:
First download a recent version. If you are starting with elasticsearch, you 
don't need to start with a so old version.
1.0.1 is fine.

Then read the documentation here: 
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html

No need to install so many pieces.

Unzip and launch. That's all.


-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr


Le 20 mars 2014 à 17:06:12, Anikessh Jain (anikess...@gmail.com) a écrit:

please help on the above error 

On Thursday, March 20, 2014 9:03:46 PM UTC+5:30, Anikessh Jain wrote:

wget 
https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.2.tar.gz
tar xvf elasticsearch-0.20.2.tar.gz

mv elasticsearch-0.20.2 elasticsearch



wget 
http://github.com/elasticsearch/elasticsearch-servicewrapper/archive/master.zip
unzip master
mv elasticsearch-servicewrapper-master/service/ elasticsearch/bin
rm -rf master
rm -rf elasticsearch-servicewrapper-master/



Configure ‘elasticsearch.conf’


set.default.ES_HOME=/usr/local/elasticsearch
set.default.ES_HEAP_SIZE=4096
wrapper.java.additional.10=-Des.max-open-files=true

wrapper.logfile.maxsize=5m
wrapper.logfile.maxfiles=5


Add ES home to root user’s ‘.bash_profile’


# ElasticSearch
export ES_HOME=/usr/local/elasticsearch


Create elasticsearch user


useradd -d /home/elasticsearch -s /bin/sh elasticsearch
chown -R elasticsearch:elasticsearch $ES_HOME
chown -R elasticsearch:elasticsearch /mnt/storage/es-data


Edit elasticsearch user’s ‘.bash_profile’


# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

# JAVA_HOME needs to be the latest 1.7 JDK on the system
JAVA_HOME=/usr/local/jdk7
export JAVA_HOME

#Add JAVA_HOME to the PATH
PATH=$JAVA_HOME/bin:$PATH

# ElasticSearch
export ES_HOME=/usr/local/elasticsearch

unset USERNAME


Edit ‘/etc/security/limits.conf’ (optional as this will be set in the service 
script, too)


elasticsearchsoftnofile  65535
elasticsearchhardnofile  65535


Verify the file descriptor limit


sudo -u elasticsearch -s ulimit -Sn


Install the service


bin/service/elasticsearch install

Edit ‘/etc/init.d/elasticsearch’

# Java
JAVA_HOME=/usr/local/jdk7
export JAVA_HOME
PATH=$JAVA_HOME/bin:$PATH
RUN_AS_USER=elasticsearch
ULIMIT_N=65535

Run the service

bin/service/elasticsearch start
or
service elasticsearch start   -> here the above error i am getting can you 
please help me

please help me what wrong i have done   
On Thursday, March 20, 2014 8:10:22 PM UTC+5:30, David Pilato wrote:
What did you download?

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 20 mars 2014 à 14:54, Anikessh Jain  a écrit :

nobody there to help me , help me please i am in need

On Thursday, March 20, 2014 3:31:42 PM UTC+5:30, Anikessh Jain wrote:
Can anybody help me it is  a bit urgent .



On Thursday, March 20, 2014 3:09:56 PM UTC+5:30, Anikessh Jain wrote:
Hi All.

I am a newbie to Elastic search and I am configuring Kibana with Logstash and 
Redis  and Elasticsearch  in  Centos 32 Bit and when i am trying to start the 
service of elastic search i am getting the below error


WrapperSimpleApp Error: Unable

 to locate the class org.elasticsearch.bootstrap.ElasticsearchF : 
java.lang.ClassNotFoundException: org.elasticsearch.bootstrap.ElasticsearchF




How to solve the above error ,i tried many steps but no use  but of no help, 
have anybody face the same issue please help me.
--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearc...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6e075a41-21b4-46f9-a9b3-a9beae1210b4%40googlegroups.com.
For more options, visit http

Re: Complete cluster failure

Don't bother trying digging deeper since I suspect network.

I tried many different configurations while trying to pinpoint the problem,
so I did not write down the various states, just the successes/failures.
Using the described methods, IPV4 was indeed working, but multicast was
still not cooperating on the test cluster (bound_address
{inet[/192.168.50.124:9300]}, publish_address {inet[/192.168.50.124:9300]}).
Two out of the eight nodes refused to see the other six, but were able to
talk to each other. Same subnet with iptables disabled.

I am transitioning the unicast, just writing things down in case someone
has similar problems in the future.

-- 
Ivan


On Thu, Mar 20, 2014 at 6:04 AM, Zachary Tong wrote:

> Nice post-mortem, thanks for the writeup.  Hopefully someone will stumble
> on this in the future and avoid the same headache you had :)
>
> How would you force IPV4? I tried using preferIPv4Stack and setting
>> network.host to _eth0:ipv4_, but it still did not work. Even switched off
>> iptables at a point!
>>
>
> Hmm...that's interesting.  I would have recommended those two exact
> methods.  I'll do some digging and see why they didn't work...
>
> -Z
>
>
>
> On Thursday, March 20, 2014 1:23:48 AM UTC-5, Ivan Brusic wrote:
>
>> Responses inline.
>>
>>
>> On Wed, Mar 19, 2014 at 7:25 PM, Zachary Tong wrote:
>>
>>> Yeah, in case anyone reads this thread in the future, this log output is
>>> a good indicator of multicast problems.  You can see that the the nodes are
>>> pinging and talking to each other on this log line:
>>>
>>> --> target [[search6][T3tINFmqREK9W6oqZV0r7A][inet[/192.168.50.106:9300]]],
>>> master [null]
>>>
>>>
>>> That's basically a ping response from a node saying "Hey!  I'm alive,
>>> but I dunno who the master is yet".  If the nodes were unable to
>>> communicate, you'd see failed ping responsese instead.  But they are unable
>>> to fully connect and start to establish/elect a master.  So they all just
>>> sit around and ping each other for ages until giving up.  It isn't always,
>>> but usually, a multicast issue when you see logs like this.  You can
>>> usually diagnose this by manually telnetting between nodes on port
>>> 9300...if the connection isn't refused, its probably a multicast discovery
>>> issue.
>>>
>>
>> Once everything was stabilized and I was able to get some sleep, I looked
>> at the code (ZenDiscovery.java) the next day and realized exactly what you
>> said. Everything has been running on the existing infrastructure since
>> 2012, so I did not initially put blame on multicast discovery.
>>
>> The other common culprit is IPv6 issue, often when you have multiple
>>> NICs.  You'll see the publish address bind to IPv6 and the bind address on
>>> IPv4...and the whole cluster goes to hell because the nodes can see each
>>> other but not communicate.
>>>
>>
>> Once I realized it was a multicast issue, I blamed the sysadmins because
>> it wasn't my fault. :) I assumed it was an issue communicating between the
>> underlying VM hosts, but I was able to replicate the issue on physical
>> machines as well. The sysadmin suspected IPV6. We are indeed running IPV6
>> and looking at the logs in the initial post the bound and publish address
>> are indeed IPV4 and IPV6 respectively. Tried setting 
>> java.net.preferIPv4Stack=true
>> in JAVA_OPTS, but it did not make a difference. Ultimately I decided that
>> even if we discovered and fix the culprit, we might get bitten again in the
>> future, so I switched to unicast. I know have to keep track of different
>> config files in source control instead of one.
>>
>>
>>> Discovery issues are generally fixed by disabling multicast and forcing
>>> IPv4 in my experience.
>>>
>>
>> How would you force IPV4? I tried using preferIPv4Stack and setting
>> network.host to _eth0:ipv4_, but it still did not work. Even switched off
>> iptables at a point!
>>
>>
>>> Glad you got it working again Ivan! :)
>>>
>>
>> You and me both!
>>
>> Cheers,
>>
>> Ivan
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/5750058e-5305-4408-93c2-ee9ddf3ea300%40googlegroups.com
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDpQTgT29WyELH%2Bcf_%3DwgZpV0BJM-VgdJeTirvJ5vTCJg%40mail.gmail.com.
For more

Re: elasticsearch error of class not found

Hi David,

But why do people download and install it through 
elasticsearch-servicewrapper ,does the leatest version supports it all bcoz 
i want to install kibana with logstash and redis with elasticsearch and i 
am a little bit confused will it work bcoz i have wasted my whole day on on 
error.please guide me what to do as i dont know what to do and doing it for 
first time

On Thursday, March 20, 2014 10:11:52 PM UTC+5:30, David Pilato wrote:
>
> First download a recent version. If you are starting with elasticsearch, 
> you don't need to start with a so old version.
> 1.0.1 is fine.
>
> Then read the documentation here: 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html
>
> No need to install so many pieces.
>
> Unzip and launch. That's all.
>
>
> -- 
> *David Pilato* | *Technical Advocate* | *Elasticsearch.com*
> @dadoonet  | 
> @elasticsearchfr
>
>
> Le 20 mars 2014 à 17:06:12, Anikessh Jain (anikess...@gmail.com) 
> a écrit:
>
> please help on the above error 
>
> On Thursday, March 20, 2014 9:03:46 PM UTC+5:30, Anikessh Jain wrote: 
>>
>>   wget 
>> https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.2.tar.gz
>> tar xvf elasticsearch-0.20.2.tar.gz
>>
>> mv elasticsearch-0.20.2 elasticsearch
>>
>>wget 
>> http://github.com/elasticsearch/elasticsearch-servicewrapper/archive/master.zip
>> unzip master
>> mv elasticsearch-servicewrapper-master/service/ elasticsearch/bin
>> rm -rf master
>> rm -rf elasticsearch-servicewrapper-master/
>>
>>
>> Configure ‘elasticsearch.conf’
>>
>>  set.default.ES_HOME=/usr/local/elasticsearch
>> set.default.ES_HEAP_SIZE=4096
>> wrapper.java.additional.10=-Des.max-open-files=true
>>
>> wrapper.logfile.maxsize=5m
>> wrapper.logfile.maxfiles=5
>>
>>  
>> Add ES home to root user’s ‘.bash_profile’
>>
>>  # ElasticSearch
>> export ES_HOME=/usr/local/elasticsearch
>>
>>  
>> Create elasticsearch user
>>
>>  useradd -d /home/elasticsearch -s /bin/sh elasticsearch
>> chown -R elasticsearch:elasticsearch $ES_HOME
>> chown -R elasticsearch:elasticsearch /mnt/storage/es-data
>>
>>  
>> Edit elasticsearch user’s ‘.bash_profile’
>>
>>  # .bash_profile
>>
>> # Get the aliases and functions
>> if [ -f ~/.bashrc ]; then
>> . ~/.bashrc
>> fi
>>
>> # User specific environment and startup programs
>>
>> PATH=$PATH:$HOME/bin
>>
>> export PATH
>>
>> # JAVA_HOME needs to be the latest 1.7 JDK on the system
>> JAVA_HOME=/usr/local/jdk7
>> export JAVA_HOME
>>
>> #Add JAVA_HOME to the PATH
>> PATH=$JAVA_HOME/bin:$PATH
>>
>> # ElasticSearch
>> export ES_HOME=/usr/local/elasticsearch
>>
>> unset USERNAME
>>
>>  
>> Edit ‘/etc/security/limits.conf’ (optional as this will be set in the 
>> service script, too)
>>
>>  elasticsearchsoftnofile  65535
>> elasticsearchhardnofile  65535
>>
>>  
>> Verify the file descriptor limit
>>
>>  sudo -u elasticsearch -s ulimit -Sn
>>
>>  
>> Install the service
>>
>>  bin/service/elasticsearch install
>>
>>  
>> Edit ‘/etc/init.d/elasticsearch’
>>
>>  # Java
>> JAVA_HOME=/usr/local/jdk7
>> export JAVA_HOME
>> PATH=$JAVA_HOME/bin:$PATH
>> RUN_AS_USER=elasticsearch
>> ULIMIT_N=65535
>>
>>  
>> Run the service
>>
>>  bin/service/elasticsearch start
>> or
>> service elasticsearch start   -> here the above error i am getting can you 
>> please help me   
>>
>> please help me what wrong i have done  
>>
>>On Thursday, March 20, 2014 8:10:22 PM UTC+5:30, David Pilato wrote: 
>>>
>>>  What did you download?
>>>
>>> --
>>> David ;-)
>>> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>>>
>>>  
>>> Le 20 mars 2014 à 14:54, Anikessh Jain  a écrit :
>>>
>>>  nobody there to help me , help me please i am in need
>>>
>>> On Thursday, March 20, 2014 3:31:42 PM UTC+5:30, Anikessh Jain wrote: 

 Can anybody help me it is  a bit urgent .



 On Thursday, March 20, 2014 3:09:56 PM UTC+5:30, Anikessh Jain wrote: 
>
> Hi All. 
>
> I am a newbie to Elastic search and I am configuring Kibana with 
> Logstash and Redis  and Elasticsearch  in  Centos 32 Bit and when i am 
> trying to start the service of elastic search i am getting the below error
>
>  WrapperSimpleApp Error: Unable
>
>  to locate the class org.elasticsearch.bootstrap.ElasticsearchF : 
> java.lang.ClassNotFoundException: 
> org.elasticsearch.bootstrap.ElasticsearchF
>
>
>  How to solve the above error ,i tried many steps but no use  but of no 
> help, have anybody face the same issue please help me.
>
>--
>>> You received this message because you are subscribed to the Google 
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/6e075a41-21

Re: elasticsearch error of class not found

First download a recent version. If you are starting with elasticsearch, you 
don't need to start with a so old version.
1.0.1 is fine.

Then read the documentation here: 
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html

No need to install so many pieces.

Unzip and launch. That's all.


-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr


Le 20 mars 2014 à 17:06:12, Anikessh Jain (anikesshjai...@gmail.com) a écrit:

please help on the above error 

On Thursday, March 20, 2014 9:03:46 PM UTC+5:30, Anikessh Jain wrote:

wget 
https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.2.tar.gz
tar xvf elasticsearch-0.20.2.tar.gz

mv elasticsearch-0.20.2 elasticsearch


wget 
http://github.com/elasticsearch/elasticsearch-servicewrapper/archive/master.zip
unzip master
mv elasticsearch-servicewrapper-master/service/ elasticsearch/bin
rm -rf master
rm -rf elasticsearch-servicewrapper-master/


Configure ‘elasticsearch.conf’

set.default.ES_HOME=/usr/local/elasticsearch
set.default.ES_HEAP_SIZE=4096
wrapper.java.additional.10=-Des.max-open-files=true

wrapper.logfile.maxsize=5m
wrapper.logfile.maxfiles=5

Add ES home to root user’s ‘.bash_profile’

# ElasticSearch
export ES_HOME=/usr/local/elasticsearch

Create elasticsearch user

useradd -d /home/elasticsearch -s /bin/sh elasticsearch
chown -R elasticsearch:elasticsearch $ES_HOME
chown -R elasticsearch:elasticsearch /mnt/storage/es-data

Edit elasticsearch user’s ‘.bash_profile’

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

# JAVA_HOME needs to be the latest 1.7 JDK on the system
JAVA_HOME=/usr/local/jdk7
export JAVA_HOME

#Add JAVA_HOME to the PATH
PATH=$JAVA_HOME/bin:$PATH

# ElasticSearch
export ES_HOME=/usr/local/elasticsearch

unset USERNAME

Edit ‘/etc/security/limits.conf’ (optional as this will be set in the service 
script, too)

elasticsearchsoftnofile  65535
elasticsearchhardnofile  65535

Verify the file descriptor limit

sudo -u elasticsearch -s ulimit -Sn

Install the service


bin/service/elasticsearch install

Edit ‘/etc/init.d/elasticsearch’

# Java
JAVA_HOME=/usr/local/jdk7
export JAVA_HOME
PATH=$JAVA_HOME/bin:$PATH
RUN_AS_USER=elasticsearch
ULIMIT_N=65535

Run the service

bin/service/elasticsearch start
or
service elasticsearch start   -> here the above error i am getting can you 
please help me   

please help me what wrong i have done  
On Thursday, March 20, 2014 8:10:22 PM UTC+5:30, David Pilato wrote:
What did you download?

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 20 mars 2014 à 14:54, Anikessh Jain  a écrit :

nobody there to help me , help me please i am in need

On Thursday, March 20, 2014 3:31:42 PM UTC+5:30, Anikessh Jain wrote:
Can anybody help me it is  a bit urgent .



On Thursday, March 20, 2014 3:09:56 PM UTC+5:30, Anikessh Jain wrote:
Hi All.

I am a newbie to Elastic search and I am configuring Kibana with Logstash and 
Redis  and Elasticsearch  in  Centos 32 Bit and when i am trying to start the 
service of elastic search i am getting the below error


WrapperSimpleApp Error: Unable

 to locate the class org.elasticsearch.bootstrap.ElasticsearchF : 
java.lang.ClassNotFoundException: org.elasticsearch.bootstrap.ElasticsearchF




How to solve the above error ,i tried many steps but no use  but of no help, 
have anybody face the same issue please help me.
--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearc...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6e075a41-21b4-46f9-a9b3-a9beae1210b4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/221d4092-88ec-405d-a9b1-4cae1ce1d3cf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/etPan.532b1a51.333ab105.c9bd%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

Re: ES attempting to parse dates automatically (possibly Hadoop related)

2014-03-20 Thread Brian Stempin

That's the problem -- it's a web log that contains a URL that could have
literally anything in it.  Anyone could put a base64 value as a random
query parameter.  I could have the M/R job ignore all fields that I don't
explicitly expect, but that's not very flexible and prevents me from
spotting possible abuse or user-error.  Is there any way for me to disable
ES's type-guessing or to provide a default guess?  I'd rather have ES
default to a string than to fail a M/R job because its type-guess was wrong.

Brian


On Thu, Mar 20, 2014 at 12:26 PM, Costin Leau  wrote:

> Then what you could do is to minimize the bulk size to say 100 documents,
> turn on logging and run your data through.
> This way you can catch the 'special' document in the act.
>
> As for expectations - Elasticsearch tries to guess the field type by
> looking at its value - it seems the base64 entry looks like a date, hence
> the error. You can avoid this by defining the field (either directly or
> through a template) in your mapping so it always gets mapped to a string.
> As a rule of thumb, whenever you want full control over the index, mapping
> is the way to do it.
>
>
>
> On 3/20/14 6:10 PM, Brian Stempin wrote:
>
>> I have unit tests for this MR job, and they show that the JSON output is
>> a string as I'd expect, so Gson is most likely
>> not the cause.
>>
>> I'm hesitant to show more code (owned by the work-place), but I can
>> describe it a little bit further:
>>
>>   * The mapper gets a W3C log entry
>>   * The log entry is broken into its components and put into document X
>>   * The request URL is then taken and broken down into its query
>> parameters and the key-value pairs are put into document Y
>>   * Some elements are then explicitly filtered from X and Y
>>   * Those two documents are placed inside of document Z, which is
>> ultimately what is serialized and sent to ES
>>
>>
>> We do have a base64 encoded param that we expect and handle
>> appropriately.  In this case, someone most likely sent it as
>> the wrong param name, hence why its making its way into document Y
>> without further processing.  Since its being sent as
>> a name that's not listed in the mapping, I expect it to just be treated
>> as a string.
>>
>> The only reason that I chose to go the Gson route vs building
>> MapWritables is that building MapWritables is terribly
>> verbose.  Also, it comes with the overhead of having to wrap each String
>> with a Text type, which just seems silly.
>>   Using the built-in JSON serializer is just not convenient in this case.
>>
>> Brian
>>
>>
>> On Thu, Mar 20, 2014 at 11:18 AM, Costin Leau > costin.l...@gmail.com>> wrote:
>>
>> My guess is that GSON adds the said field in its result. The base64
>> suggests that there's some binary data in the mix.
>>
>> By the way, can you show up more of your code - any reason why you
>> create the JSON yourself rather than just pass
>> logEntryMap to Es-Hadoop?
>> It can create the json for you - which is what I recommend; unless
>> you have the JSON in HDFS, it's best to rely on
>> es-hadoop to do it instead of an external tool.
>>
>> Cheers,
>>
>>
>> On 3/20/14 4:48 PM, Brian Stempin wrote:
>>
>> Hi,
>> All I'm doing is building a map and passing that to Gson for
>> serialization.  A snippet from my map method:
>>
>> logEntryMap.put("cs(User-__Agent)", values[9]);
>> context.write(NullWritable.__get(), new
>> Text(gson.toJson(logEntryMap))__);
>>
>>
>> values[] is a String array.  Everything that goes into the map
>> that gets serialized is a string.
>>
>> I do have es.input.json set to true.  This failure doesn't occur
>> until >100,000,000 records are in the index, so its
>> happening late in the load process.  The part that I find strange
>> is that the field in question isn't in my
>> mapping, and
>> I've not touched the default mapping.  I'm not sure why it would
>> try to parse it as anything other than a string.
>>
>> I'll turn on TRACE logging and see what happens.
>>
>> Brian
>>
>>
>> On Wed, Mar 19, 2014 at 5:35 PM, Costin Leau <
>> costin.l...@gmail.com 
>> >__>
>> wrote:
>>
>>  Hi,
>>
>>  How do you pass the json to es-hadoop? Do you have an
>> example? By the way, you can enable TRACE logging on
>>  org.elasticsearch.hadoop and see everything that es-hadoop
>> does, including the data that goes over the wire.
>>  My guess is that the conversion of logs to JSON creates some
>> extra artifacts which are later on interpreted as
>>  Writable object (instead of raw JSON) by ES Hadoop.
>>  Make sure you tell es-hadoop that its source it's json
>> (through es.input.json set to true).
>>  The logs will likely confirm (or not) the above :)
>>
>>  Cheers,
>>
>>
>>

Re: ES attempting to parse dates automatically (possibly Hadoop related)

Then what you could do is to minimize the bulk size to say 100 documents, turn
on logging and run your data through.
This way you can catch the 'special' document in the act.

As for expectations - Elasticsearch tries to guess the field type by looking at its value - it seems the base64 entry
looks like a date, hence the error. You can avoid this by defining the field (either directly or through a template) in
your mapping so it always gets mapped to a string.

As a rule of thumb, whenever you want full control over the index, mapping is
the way to do it.

On 3/20/14 6:10 PM, Brian Stempin wrote:

I have unit tests for this MR job, and they show that the JSON output is a
string as I'd expect, so Gson is most likely
not the cause.

I'm hesitant to show more code (owned by the work-place), but I can describe it
a little bit further:

* The mapper gets a W3C log entry
* The log entry is broken into its components and put into document X
* The request URL is then taken and broken down into its query parameters and
the key-value pairs are put into document Y
* Some elements are then explicitly filtered from X and Y
* Those two documents are placed inside of document Z, which is ultimately
what is serialized and sent to ES

We do have a base64 encoded param that we expect and handle appropriately. In
this case, someone most likely sent it as
the wrong param name, hence why its making its way into document Y without
further processing. Since its being sent as
a name that's not listed in the mapping, I expect it to just be treated as a
string.

The only reason that I chose to go the Gson route vs building MapWritables is
that building MapWritables is terribly
verbose. Also, it comes with the overhead of having to wrap each String with a
Text type, which just seems silly.
Using the built-in JSON serializer is just not convenient in this case.

Brian

On Thu, Mar 20, 2014 at 11:18 AM, Costin Leau mailto:costin.l...@gmail.com>> wrote:

My guess is that GSON adds the said field in its result. The base64
suggests that there's some binary data in the mix.

By the way, can you show up more of your code - any reason why you create
the JSON yourself rather than just pass
logEntryMap to Es-Hadoop?
It can create the json for you - which is what I recommend; unless you have
the JSON in HDFS, it's best to rely on
es-hadoop to do it instead of an external tool.

Cheers,

On 3/20/14 4:48 PM, Brian Stempin wrote:

Hi,
All I'm doing is building a map and passing that to Gson for
serialization. A snippet from my map method:

logEntryMap.put("cs(User-__Agent)", values[9]);
context.write(NullWritable.__get(), new
Text(gson.toJson(logEntryMap))__);

values[] is a String array. Everything that goes into the map that
gets serialized is a string.

I do have es.input.json set to true. This failure doesn't occur until
>100,000,000 records are in the index, so its
happening late in the load process. The part that I find strange is
that the field in question isn't in my
mapping, and
I've not touched the default mapping. I'm not sure why it would try to
parse it as anything other than a string.

I'll turn on TRACE logging and see what happens.

Brian

On Wed, Mar 19, 2014 at 5:35 PM, Costin Leau mailto:costin.l...@gmail.com>
>__> wrote:

Hi,

How do you pass the json to es-hadoop? Do you have an example? By
the way, you can enable TRACE logging on
org.elasticsearch.hadoop and see everything that es-hadoop does,
including the data that goes over the wire.
My guess is that the conversion of logs to JSON creates some extra
artifacts which are later on interpreted as
Writable object (instead of raw JSON) by ES Hadoop.
Make sure you tell es-hadoop that its source it's json (through
es.input.json set to true).
The logs will likely confirm (or not) the above :)

Cheers,

On 3/19/14 11:14 PM, Brian Stempin wrote:

Hi List,
I have an ES cluster that takes in some data from our logs.
We use Hadoop to parse the individual log
entries
into JSON
strings, which does a bulk insert using ES's output format.
For whatever reason, ES attempts to parse
base64
strings as
a dates and fails. Here's a line from one of my Hadoop logs:

java.lang.IllegalStateException: Found unrecoverable
error [Bad Request(400) -

MapperParsingException[failed to parse [csUriParams.d]];
nested: MapperParsingException[failed to parse
date
field [REDACTED BASE64 STRING], tried both date format
[dateOptionalTime], and timestamp number with

Re: elasticsearch error of class not found

please help on the above error 

On Thursday, March 20, 2014 9:03:46 PM UTC+5:30, Anikessh Jain wrote:
>
> wget 
> https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.2.tar.gz
> tar xvf elasticsearch-0.20.2.tar.gz
>
> mv elasticsearch-0.20.2 elasticsearch
>
> wget 
> http://github.com/elasticsearch/elasticsearch-servicewrapper/archive/master.zip
> unzip master
> mv elasticsearch-servicewrapper-master/service/ elasticsearch/bin
> rm -rf master
> rm -rf elasticsearch-servicewrapper-master/
>
>
> Configure ‘elasticsearch.conf’
>
> set.default.ES_HOME=/usr/local/elasticsearch
> set.default.ES_HEAP_SIZE=4096
> wrapper.java.additional.10=-Des.max-open-files=true
>
> wrapper.logfile.maxsize=5m
> wrapper.logfile.maxfiles=5
>
>
> Add ES home to root user’s ‘.bash_profile’
>
> # ElasticSearch
> export ES_HOME=/usr/local/elasticsearch
>
>
> Create elasticsearch user
>
> useradd -d /home/elasticsearch -s /bin/sh elasticsearch
> chown -R elasticsearch:elasticsearch $ES_HOME
> chown -R elasticsearch:elasticsearch /mnt/storage/es-data
>
>
> Edit elasticsearch user’s ‘.bash_profile’
>
> # .bash_profile
>
> # Get the aliases and functions
> if [ -f ~/.bashrc ]; then
> . ~/.bashrc
> fi
>
> # User specific environment and startup programs
>
> PATH=$PATH:$HOME/bin
>
> export PATH
>
> # JAVA_HOME needs to be the latest 1.7 JDK on the system
> JAVA_HOME=/usr/local/jdk7
> export JAVA_HOME
>
> #Add JAVA_HOME to the PATH
> PATH=$JAVA_HOME/bin:$PATH
>
> # ElasticSearch
> export ES_HOME=/usr/local/elasticsearch
>
> unset USERNAME
>
>
> Edit ‘/etc/security/limits.conf’ (optional as this will be set in the service 
> script, too)
>
> elasticsearchsoftnofile  65535
> elasticsearchhardnofile  65535
>
>
> Verify the file descriptor limit
>
> sudo -u elasticsearch -s ulimit -Sn
>
>
> Install the service
>
> bin/service/elasticsearch install
>
>
> Edit ‘/etc/init.d/elasticsearch’
>
> # Java
> JAVA_HOME=/usr/local/jdk7
> export JAVA_HOME
> PATH=$JAVA_HOME/bin:$PATH
> RUN_AS_USER=elasticsearch
> ULIMIT_N=65535
>
>
> Run the service
>
> bin/service/elasticsearch start
> or
> service elasticsearch start   -> here the above error i am getting can you 
> please help me  
>
> please help me what wrong i have done 
>
> On Thursday, March 20, 2014 8:10:22 PM UTC+5:30, David Pilato wrote:
>>
>> What did you download?
>>
>> --
>> David ;-)
>> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>>
>>
>> Le 20 mars 2014 à 14:54, Anikessh Jain  a écrit :
>>
>> nobody there to help me , help me please i am in need
>>
>> On Thursday, March 20, 2014 3:31:42 PM UTC+5:30, Anikessh Jain wrote:
>>>
>>> Can anybody help me it is  a bit urgent .
>>>
>>>
>>>
>>> On Thursday, March 20, 2014 3:09:56 PM UTC+5:30, Anikessh Jain wrote:

 Hi All.

 I am a newbie to Elastic search and I am configuring Kibana with 
 Logstash and Redis  and Elasticsearch  in  Centos 32 Bit and when i am 
 trying to start the service of elastic search i am getting the below error

 WrapperSimpleApp Error: Unable

  to locate the class org.elasticsearch.bootstrap.ElasticsearchF : 
 java.lang.ClassNotFoundException: 
 org.elasticsearch.bootstrap.ElasticsearchF


 How to solve the above error ,i tried many steps but no use  but of no 
 help, have anybody face the same issue please help me.

  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/6e075a41-21b4-46f9-a9b3-a9beae1210b4%40googlegroups.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/221d4092-88ec-405d-a9b1-4cae1ce1d3cf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES attempting to parse dates automatically (possibly Hadoop related)

2014-03-20 Thread Brian Stempin

I have unit tests for this MR job, and they show that the JSON output is a
string as I'd expect, so Gson is most likely not the cause.

I'm hesitant to show more code (owned by the work-place), but I can
describe it a little bit further:

   - The mapper gets a W3C log entry
   - The log entry is broken into its components and put into document X
   - The request URL is then taken and broken down into its query
   parameters and the key-value pairs are put into document Y
   - Some elements are then explicitly filtered from X and Y
   - Those two documents are placed inside of document Z, which is
   ultimately what is serialized and sent to ES

We do have a base64 encoded param that we expect and handle appropriately.
 In this case, someone most likely sent it as the wrong param name, hence
why its making its way into document Y without further processing.  Since
its being sent as a name that's not listed in the mapping, I expect it to
just be treated as a string.

The only reason that I chose to go the Gson route vs building MapWritables
is that building MapWritables is terribly verbose.  Also, it comes with the
overhead of having to wrap each String with a Text type, which just seems
silly.  Using the built-in JSON serializer is just not convenient in this
case.

Brian


On Thu, Mar 20, 2014 at 11:18 AM, Costin Leau  wrote:

> My guess is that GSON adds the said field in its result. The base64
> suggests that there's some binary data in the mix.
>
> By the way, can you show up more of your code - any reason why you create
> the JSON yourself rather than just pass logEntryMap to Es-Hadoop?
> It can create the json for you - which is what I recommend; unless you
> have the JSON in HDFS, it's best to rely on es-hadoop to do it instead of
> an external tool.
>
> Cheers,
>
>
> On 3/20/14 4:48 PM, Brian Stempin wrote:
>
>> Hi,
>> All I'm doing is building a map and passing that to Gson for
>> serialization.  A snippet from my map method:
>>
>> logEntryMap.put("cs(User-Agent)", values[9]);
>> context.write(NullWritable.get(), new Text(gson.toJson(logEntryMap)));
>>
>> values[] is a String array.  Everything that goes into the map that gets
>> serialized is a string.
>>
>> I do have es.input.json set to true.  This failure doesn't occur until
>> >100,000,000 records are in the index, so its
>> happening late in the load process.  The part that I find strange is that
>> the field in question isn't in my mapping, and
>> I've not touched the default mapping.  I'm not sure why it would try to
>> parse it as anything other than a string.
>>
>> I'll turn on TRACE logging and see what happens.
>>
>> Brian
>>
>>
>> On Wed, Mar 19, 2014 at 5:35 PM, Costin Leau > costin.l...@gmail.com>> wrote:
>>
>> Hi,
>>
>> How do you pass the json to es-hadoop? Do you have an example? By the
>> way, you can enable TRACE logging on
>> org.elasticsearch.hadoop and see everything that es-hadoop does,
>> including the data that goes over the wire.
>> My guess is that the conversion of logs to JSON creates some extra
>> artifacts which are later on interpreted as
>> Writable object (instead of raw JSON) by ES Hadoop.
>> Make sure you tell es-hadoop that its source it's json (through
>> es.input.json set to true).
>> The logs will likely confirm (or not) the above :)
>>
>> Cheers,
>>
>>
>> On 3/19/14 11:14 PM, Brian Stempin wrote:
>>
>> Hi List,
>> I have an ES cluster that takes in some data from our logs.  We
>> use Hadoop to parse the individual log entries
>> into JSON
>> strings, which does a bulk insert using ES's output format.  For
>> whatever reason, ES attempts to parse base64
>> strings as
>> a dates and fails.  Here's a line from one of my Hadoop logs:
>>
>>  java.lang.__IllegalStateException: Found unrecoverable
>> error [Bad Request(400) -
>>
>> MapperParsingException[failed to parse [csUriParams.d]]; nested:
>> MapperParsingException[failed to parse date
>> field [REDACTED BASE64 STRING], tried both date format
>> [dateOptionalTime], and timestamp number with locale []];
>> nested: IllegalArgumentException[__Invalid format: "__
>> Y2lkPURFJml0ZW1zPWE2NTJjLXgxZT__Fj..."]; ]; Bailing out..
>>
>>  at org.elasticsearch.hadoop.rest.__RestClient.
>> retryFailedEntries(__RestClient.java:145)
>>
>>  at org.elasticsearch.hadoop.rest.
>> __RestClient.bulk(RestClient.__java:120)
>>
>>  at org.elasticsearch.hadoop.rest.
>> __RestRepository.sendBatch(__RestRepository.java:147)
>>
>>
>>  
>>
>>
>> csUriParams.d does not appear in my mapping, so I never
>> explicitly asked for it to be treated as a date.
>>
>> Any idea why ES is trying to treat it as a date?
>>
>> Thanks,
>> Brian
>>
>> --
>> You received this message because you are subscribed to the
>> Google Groups "elasticsearch" group.
>> To unsubscr

Re: elasticsearch error of class not found



On Thursday, March 20, 2014 3:09:56 PM UTC+5:30, Anikessh Jain wrote:
>
> Hi All.
>
> I am a newbie to Elastic search and I am configuring Kibana with Logstash 
> and Redis  and Elasticsearch  in  Centos 32 Bit and when i am trying to 
> start the service of elastic search i am getting the below error
>
> WrapperSimpleApp Error: Unable
>
>  to locate the class org.elasticsearch.bootstrap.ElasticsearchF : 
> java.lang.ClassNotFoundException: org.elasticsearch.bootstrap.ElasticsearchF
>
>
> How to solve the above error ,i tried many steps but no use  but of no help, 
> have anybody face the same issue please help me.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a8275483-ac50-4f0a-a150-61481c2f8d94%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Fwd: elasticsearch error of class not found



wget 
https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.2.tar.gz
tar xvf elasticsearch-0.20.2.tar.gz

mv elasticsearch-0.20.2 elasticsearch

wget 
http://github.com/elasticsearch/elasticsearch-servicewrapper/archive/master.zip
unzip master
mv elasticsearch-servicewrapper-master/service/ elasticsearch/bin
rm -rf master
rm -rf elasticsearch-servicewrapper-master/


Configure ‘elasticsearch.conf’

set.default.ES_HOME=/usr/local/elasticsearch
set.default.ES_HEAP_SIZE=4096
wrapper.java.additional.10=-Des.max-open-files=true

wrapper.logfile.maxsize=5m
wrapper.logfile.maxfiles=5


Add ES home to root user’s ‘.bash_profile’

# ElasticSearch
export ES_HOME=/usr/local/elasticsearch


Create elasticsearch user

useradd -d /home/elasticsearch -s /bin/sh elasticsearch
chown -R elasticsearch:elasticsearch $ES_HOME
chown -R elasticsearch:elasticsearch /mnt/storage/es-data


Edit elasticsearch user’s ‘.bash_profile’

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

# JAVA_HOME needs to be the latest 1.7 JDK on the system
JAVA_HOME=/usr/local/jdk7
export JAVA_HOME

#Add JAVA_HOME to the PATH
PATH=$JAVA_HOME/bin:$PATH

# ElasticSearch
export ES_HOME=/usr/local/elasticsearch

unset USERNAME


Edit ‘/etc/security/limits.conf’ (optional as this will be set in the service 
script, too)

elasticsearchsoftnofile  65535
elasticsearchhardnofile  65535


Verify the file descriptor limit

sudo -u elasticsearch -s ulimit -Sn


Install the service

bin/service/elasticsearch install


Edit ‘/etc/init.d/elasticsearch’

# Java
JAVA_HOME=/usr/local/jdk7
export JAVA_HOME
PATH=$JAVA_HOME/bin:$PATH
RUN_AS_USER=elasticsearch
ULIMIT_N=65535


Run the service

bin/service/elasticsearch start
or
service elasticsearch start   -> here the above error i am getting can you 
please help me  

please help me what wrong i have done 

On Thursday, March 20, 2014 8:10:22 PM UTC+5:30, David Pilato wrote:
>
> What did you download?
>
> --
> David ;-)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
>
> Le 20 mars 2014 à 14:54, Anikessh Jain > 
> a écrit :
>
> nobody there to help me , help me please i am in need
>
> On Thursday, March 20, 2014 3:31:42 PM UTC+5:30, Anikessh Jain wrote:
>>
>> Can anybody help me it is  a bit urgent .
>>
>>
>>
>> On Thursday, March 20, 2014 3:09:56 PM UTC+5:30, Anikessh Jain wrote:
>>>
>>> Hi All.
>>>
>>> I am a newbie to Elastic search and I am configuring Kibana with 
>>> Logstash and Redis  and Elasticsearch  in  Centos 32 Bit and when i am 
>>> trying to start the service of elastic search i am getting the below error
>>>
>>> WrapperSimpleApp Error: Unable
>>>
>>>  to locate the class org.elasticsearch.bootstrap.ElasticsearchF : 
>>> java.lang.ClassNotFoundException: org.elasticsearch.bootstrap.ElasticsearchF
>>>
>>>
>>> How to solve the above error ,i tried many steps but no use  but of no 
>>> help, have anybody face the same issue please help me.
>>>
>>>  -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/6e075a41-21b4-46f9-a9b3-a9beae1210b4%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6d8b8875-5c20-41de-bb37-87c9c263a176%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES attempting to parse dates automatically (possibly Hadoop related)

My guess is that GSON adds the said field in its result. The base64 suggests
that there's some binary data in the mix.

By the way, can you show up more of your code - any reason why you create the JSON yourself rather than just pass
logEntryMap to Es-Hadoop?
It can create the json for you - which is what I recommend; unless you have the JSON in HDFS, it's best to rely on
es-hadoop to do it instead of an external tool.

Cheers,

On 3/20/14 4:48 PM, Brian Stempin wrote:

Hi,
All I'm doing is building a map and passing that to Gson for serialization. A
snippet from my map method:

logEntryMap.put("cs(User-Agent)", values[9]);
context.write(NullWritable.get(), new Text(gson.toJson(logEntryMap)));

values[] is a String array. Everything that goes into the map that gets
serialized is a string.

I do have es.input.json set to true. This failure doesn't occur until
>100,000,000 records are in the index, so its
happening late in the load process. The part that I find strange is that the
field in question isn't in my mapping, and
I've not touched the default mapping. I'm not sure why it would try to parse
it as anything other than a string.

I'll turn on TRACE logging and see what happens.

Brian

On Wed, Mar 19, 2014 at 5:35 PM, Costin Leau mailto:costin.l...@gmail.com>> wrote:

Hi,

How do you pass the json to es-hadoop? Do you have an example? By the way,
you can enable TRACE logging on
org.elasticsearch.hadoop and see everything that es-hadoop does, including
the data that goes over the wire.
My guess is that the conversion of logs to JSON creates some extra
artifacts which are later on interpreted as
Writable object (instead of raw JSON) by ES Hadoop.
Make sure you tell es-hadoop that its source it's json (through
es.input.json set to true).
The logs will likely confirm (or not) the above :)

Cheers,

On 3/19/14 11:14 PM, Brian Stempin wrote:

Hi List,
I have an ES cluster that takes in some data from our logs. We use
Hadoop to parse the individual log entries
into JSON
strings, which does a bulk insert using ES's output format. For
whatever reason, ES attempts to parse base64
strings as
a dates and fails. Here's a line from one of my Hadoop logs:

java.lang.__IllegalStateException: Found unrecoverable error [Bad
Request(400) -
MapperParsingException[failed to parse [csUriParams.d]]; nested:
MapperParsingException[failed to parse date
field [REDACTED BASE64 STRING], tried both date format
[dateOptionalTime], and timestamp number with locale []];
nested: IllegalArgumentException[__Invalid format:
"__Y2lkPURFJml0ZW1zPWE2NTJjLXgxZT__Fj..."]; ]; Bailing out..

at
org.elasticsearch.hadoop.rest.__RestClient.retryFailedEntries(__RestClient.java:145)

at
org.elasticsearch.hadoop.rest.__RestClient.bulk(RestClient.__java:120)

at
org.elasticsearch.hadoop.rest.__RestRepository.sendBatch(__RestRepository.java:147)

csUriParams.d does not appear in my mapping, so I never explicitly
asked for it to be treated as a date.

Any idea why ES is trying to treat it as a date?

Thanks,
Brian

To view this discussion on the web visit

https://groups.google.com/d/__msgid/elasticsearch/49e5fe0b-__cec3-4914-b8d6-99440dd5fb69%__40googlegroups.com

For more options, visit https://groups.google.com/d/__optout
.

--
Costin

--
You received this message because you are subscribed to a topic in the Google Groups
"elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/__topic/elasticsearch/___iE0t92CUzA/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@__googlegroups.com
.
To view this discussion on the web visit

https://groups.g

Re: elasticsearch java interaction

2014-03-20 Thread Georgi Ivanov

There is something wrong with your set-up

How many ES node s you have ?
On which IP addresses are ES hosts listening ?

I understood you have 2 hosts , but it seems you have only one on your 
local machine .

This is the code (a bit modified) I am using at the moment 


public void connectES() {
Set hosts = new HashSet();
  hosts.add("host1.mydomain.com");
  hosts.add("host2.host1.mydomain.com"); // Make sure this resolvs to 
proper IP address
  Settings settings = 
ImmutableSettings.settingsBuilder().put("cluster.name", 
"vesseltrackerES").build();

  TransportClient transportClient = new TransportClient(settings);
  for (String host : this.hosts) {
transportClient = transportClient.addTransportAddress(new 
InetSocketTransportAddress(host, 9300));
  }

  System.out.print("Connected to nodes : ");
  for (DiscoveryNode node : transportClient.connectedNodes()) {
System.out.print(node.getHostName() + " , ");
  }
  System.out.println("");

  this.client = (Client) transportClient;
}


On Thursday, March 20, 2014 2:51:50 PM UTC+1, Venu Krishna wrote:
>
> Actually this is my elasticsearch index  http://localhost:9200/, as you 
> told i have replaced 9200 with 9300 in the above code ,then i executed the 
> application i am getting following exceptions.
>
> Mar 20, 2014 7:17:45 PM org.elasticsearch.client.transport
> WARNING: [Bailey, Gailyn] failed to get node info for 
> [#transport#-1][inet[localhost/127.0.0.1:9300]]
> org.elasticsearch.transport.NodeDisconnectedException: 
> [][inet[localhost/127.0.0.1:9300]][/cluster/nodes/info] disconnected
>
> Connected
> Mar 20, 2014 7:17:50 PM org.elasticsearch.client.transport
> WARNING: [Bailey, Gailyn] failed to get node info for 
> [#transport#-1][inet[localhost/127.0.0.1:9300]]
> org.elasticsearch.transport.NodeDisconnectedException: 
> [][inet[localhost/127.0.0.1:9300]][/cluster/nodes/info] disconnected
>
> Mar 20, 2014 7:17:50 PM org.elasticsearch.client.transport
> WARNING: [Bailey, Gailyn] failed to get node info for 
> [#transport#-1][inet[localhost/127.0.0.1:9300]]
> org.elasticsearch.transport.NodeDisconnectedException: 
> [][inet[localhost/127.0.0.1:9300]][/cluster/nodes/info] disconnected
>
> Thankyou
>
> On Thursday, March 20, 2014 7:12:14 PM UTC+5:30, David Pilato wrote:
>>
>> Use port 9300
>>
>> --
>> David ;-)
>> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>>
>>
>> Le 20 mars 2014 à 14:34, Venu Krishna  a écrit :
>>
>> Thankyou for the reply.I am not getting any errors,but i am not able to 
>> connect to my elasticsearch using java.Here my code.
>>
>> import java.net.InetSocketAddress;
>>
>> import org.elasticsearch.client.Client;
>> import org.elasticsearch.client.transport.TransportClient;
>> import org.elasticsearch.common.transport.InetSocketTransportAddress;
>>
>>
>> public class JavaES_Client {
>>
>> void function()
>> {
>> //on StartUp
>> System.out.println("In Function");
>> 
>> Client client = new TransportClient()
>> .addTransportAddress(new InetSocketTransportAddress("localhost", 
>> 9200));   // This is where my control is getting stuck,without any 
>> exceptions or errors.
>> 
>> 
>> System.out.println("Connected");
>> //on ShutDown
>> client.close();
>> }
>> 
>> 
>> public static void main(String[] args) {
>> 
>> System.out.println("In Main Method");
>> JavaES_Client jc = new JavaES_Client();
>> System.out.println("Object Created");
>> jc.function();
>> 
>> }
>> 
>> }
>>
>>
>> On Thursday, March 20, 2014 2:20:25 PM UTC+5:30, Georgi Ivanov wrote:
>>>
>>> On Linux the file is  /etc/hosts
>>> On Windows c:\windows\system32\drivers\etc\hosts
>>>
>>> Open the file in text editor
>>>
>>> Add following lines:
>>> 192.168.1.100 host1
>>> 192.168.1.101 host2
>>>
>>> Make sure that 192.168.1.100/101 is the right IP address of the 
>>> host1/host2
>>>
>>>
>>>
>>> 2014-03-20 8:35 GMT+01:00 Venu Krishna :
>>>
 Hi Georgi Ivanov,
   yes,i am able to understand the Exception i.e. 
 UnresolvedAddressException,but you are telling that to make sure host1 and 
 host2 are resolved by adding entries to /etc/hosts to wherever the file in 
 on Windows,for this can you give me the steps how to approach this.Sorry i 
 am new to this and am learning i am unable to get the proper 
 example.Thanks 
 in advance for the help.


 On Thursday, March 20, 2014 2:36:10 AM UTC+5:30, Georgi Ivanov wrote:
>
> Well
> I think UnresolvedAddressException obviously means that your Java 
> client can not resolve host1 and host2
>
> make sure host1 and host2 are resolvable  by adding entries to 
> /etc/hosts ot wherever the file in on Windows
>
>
>
> On Friday, March 14, 2014 1:47:02 PM UTC+1, Venu Krishna wrote:
>>
>> Hi,
>>I am Y.Venu,i am totally new to this elasticsearch,now i am

how to modify term frequency formula?

2014-03-20 Thread geantbrun

Hi,
If I understand well, the formula used for the term frequency part in the 
default similarity module is the square root of the actual frequency. Is it 
possible to modify that formula to include something like a 
min(my_max_value,sqrt(frequency))? I would like to avoid huge tf's for 
documents that have the same term repeated many times. It seems that BM25 
similarity has a parameter to control saturation but I would prefer to 
stick with the simple tf/idf similarity module.
Thank you for your help
Patrick

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9a12b611-d08d-41f9-8fd4-b74ad75a6a5c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Elasticsearch backend?

2014-03-20 Thread Joshua P

What does Elasticsearch use to serve up responses?

Thanks! 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/dce3e112-06fa-4e00-9279-59d88a676748%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES attempting to parse dates automatically (possibly Hadoop related)

2014-03-20 Thread Brian Stempin

Hi,
All I'm doing is building a map and passing that to Gson for serialization.
 A snippet from my map method:

logEntryMap.put("cs(User-Agent)", values[9]);
context.write(NullWritable.get(), new Text(gson.toJson(logEntryMap)));

values[] is a String array.  Everything that goes into the map that gets
serialized is a string.

I do have es.input.json set to true.  This failure doesn't occur until
>100,000,000 records are in the index, so its happening late in the load
process.  The part that I find strange is that the field in question isn't
in my mapping, and I've not touched the default mapping.  I'm not sure why
it would try to parse it as anything other than a string.

I'll turn on TRACE logging and see what happens.

Brian


On Wed, Mar 19, 2014 at 5:35 PM, Costin Leau  wrote:

> Hi,
>
> How do you pass the json to es-hadoop? Do you have an example? By the way,
> you can enable TRACE logging on org.elasticsearch.hadoop and see everything
> that es-hadoop does, including the data that goes over the wire.
> My guess is that the conversion of logs to JSON creates some extra
> artifacts which are later on interpreted as Writable object (instead of raw
> JSON) by ES Hadoop.
> Make sure you tell es-hadoop that its source it's json (through
> es.input.json set to true).
> The logs will likely confirm (or not) the above :)
>
> Cheers,
>
>
> On 3/19/14 11:14 PM, Brian Stempin wrote:
>
>> Hi List,
>> I have an ES cluster that takes in some data from our logs.  We use
>> Hadoop to parse the individual log entries into JSON
>> strings, which does a bulk insert using ES's output format.  For whatever
>> reason, ES attempts to parse base64 strings as
>> a dates and fails.  Here's a line from one of my Hadoop logs:
>>
>> java.lang.IllegalStateException: Found unrecoverable error [Bad
>> Request(400) - MapperParsingException[failed to parse [csUriParams.d]];
>> nested: MapperParsingException[failed to parse date field [REDACTED BASE64
>> STRING], tried both date format [dateOptionalTime], and timestamp number
>> with locale []]; nested: IllegalArgumentException[Invalid format: "
>> Y2lkPURFJml0ZW1zPWE2NTJjLXgxZTFj..."]; ]; Bailing out..
>>
>> at org.elasticsearch.hadoop.rest.RestClient.retryFailedEntries(
>> RestClient.java:145)
>>
>> at org.elasticsearch.hadoop.rest.RestClient.bulk(RestClient.
>> java:120)
>>
>> at org.elasticsearch.hadoop.rest.RestRepository.sendBatch(
>> RestRepository.java:147)
>>
>> 
>>
>>
>> csUriParams.d does not appear in my mapping, so I never explicitly asked
>> for it to be treated as a date.
>>
>> Any idea why ES is trying to treat it as a date?
>>
>> Thanks,
>> Brian
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to
>> elasticsearch+unsubscr...@googlegroups.com > unsubscr...@googlegroups.com>.
>>
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/49e5fe0b-
>> cec3-4914-b8d6-99440dd5fb69%40googlegroups.com
>> > cec3-4914-b8d6-99440dd5fb69%40googlegroups.com?utm_medium=
>> email&utm_source=footer>.
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> Costin
>
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/elasticsearch/_iE0t92CUzA/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/elasticsearch/532A0D9C.7010401%40gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CANB1ciCdBYj_68DCxEcDxfYucuyhJ7NPWrmEWtV2CypqGp0dSA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: elasticsearch error of class not found

What did you download?

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 20 mars 2014 à 14:54, Anikessh Jain  a écrit :

nobody there to help me , help me please i am in need

> On Thursday, March 20, 2014 3:31:42 PM UTC+5:30, Anikessh Jain wrote:
> Can anybody help me it is  a bit urgent .
> 
> 
> 
>> On Thursday, March 20, 2014 3:09:56 PM UTC+5:30, Anikessh Jain wrote:
>> Hi All.
>> 
>> I am a newbie to Elastic search and I am configuring Kibana with Logstash 
>> and Redis  and Elasticsearch  in  Centos 32 Bit and when i am trying to 
>> start the service of elastic search i am getting the below error
>> 
>> WrapperSimpleApp Error: Unable
>>  to locate the class org.elasticsearch.bootstrap.ElasticsearchF : 
>> java.lang.ClassNotFoundException: org.elasticsearch.bootstrap.ElasticsearchF
>> 
>> How to solve the above error ,i tried many steps but no use  but of no help, 
>> have anybody face the same issue please help me.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6e075a41-21b4-46f9-a9b3-a9beae1210b4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/63E9FEAC-ABB9-4127-AA9B-66FFF4F99338%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Yet another "WARNING: Message not fully read" thread

2014-03-20 Thread tufi

Hi, 

also I am getting the warning "Message not fully read" from one of my ES 
Nodes.
I certainly googled a lot and found out that, different versions of JVM as 
well as different versions of ES server and clients can cause this. 

Well, I'm pretty sure, I have the same versions everywhere. 

My use case: 
Programming a multi-thread java-app to import about 30G data out of approx. 
1000 csv files to Elasticsearch. I use transport client for that, to send 
bulkImport requests with <10.000 bulk objects for each request.

ES Cluster: 
*1 Master: *4 Cores + 16 GB RAM
Overwritten properties in elasticsearch.yml

cluster.name: $CLUSTERNAME
node.name: $NODENAME
node.master: true
#node.data: false
node.max_local_storage_nodes: 1
index.number_of_shards: 8 # 4 x $numberOfNodes since I have 2 data nodes 8
index.number_of_replicas: 1



Versions: 

$bin/elasticsearch -v
Version: 1.0.0, Build: a46900e/2014-02-12T16:18:34Z, JVM: 1.7.0_25


*1st Slave*: 4 Cores + 16 GB RAM
Overwritten properties in elasticsearch.yml

cluster.name: $CLUSTERNAME
node.name: $NODENAME
node.master: false

node.max_local_storage_nodes: 1

# thread pool
threadpool.bulk.queue_size: -1

path.data: /opt/elasticsearch/data
path.plugins: /opt/elasticsearch/plugins
path.conf: /opt/elasticsearch/config



Versions: 
$bin/elasticsearch -v
Version: 1.0.0, Build: a46900e/2014-02-12T16:18:34Z, JVM: 1.7.0_25



*2nd Slave*: 4 Cores + 16 GB RAM
Overwritten properties in elasticsearch.yml

cluster.name: $CLUSTERNAME
node.name: $NODENAME
node.master: false

node.max_local_storage_nodes: 1

# thread pool
threadpool.bulk.queue_size: -1

path.data: /opt/elasticsearch/data
path.plugins: /opt/elasticsearch/plugins
path.conf: /opt/elasticsearch/config



Versions: 
$bin/elasticsearch -v
Version: 1.0.0, Build: a46900e/2014-02-12T16:18:34Z, JVM: 1.7.0_25


*My Client: *4 Cores + 16 GB RAM
pom.xml

  1.0.0



org.elasticsearch
elasticsearch
${es.version}


Maven: org.elasticsearch:elasticsearch:1.0.0 
is indeed under libs/ :

elasticsearch-1.0.0.jar

Java Code to initialize the transport client: 

XMLConfiguration config = ConfigLoader.getInstance().getConfig();
config.setExpressionEngine(new XPathExpressionEngine());
HierarchicalConfiguration hConf = config.configurationAt(
"/database/es/node[@master='true']");

String clusterName = hConf.getString("clusterName");
String url = hConf.getString("url");
int port = hConf.getInt("port/transport");

System.out.println("clusterName = " + clusterName);
Settings settings = ImmutableSettings.settingsBuilder()
.put("cluster.name", clusterName).build();
client = new TransportClient(settings).addTransportAddress(
new InetSocketTransportAddress(url, port));

Last but not least; java version on App-Runner: 

System.out.println(System.getProperty("java.version"));

1.7.0_25


Is there anything else, which I can check for getting rid of these 
warnings? 
What exactly does this warning mean? Do I lose data?

Since I'm new to ES; do I have to set the ES config values 
*node.max_local_storage_nodes, 
index.number_of_shards, index.number_of_replicas* also in the slave 
configs? 

Would you need additional information?

Thank you very much for your help. 

Tufi.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c64630fb-768f-47c8-9225-4d1290580703%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Kibana Histogram unexpected line curve with cumulative value

On Thursday, March 20, 2014 3:18:35 PM UTC+1, Xwilly Azel wrote:
> On Thursday, March 20, 2014 2:29:24 PM UTC+1, Xwilly Azel wrote:
> 
> > On Thursday, March 20, 2014 1:15:44 PM UTC+1, Isaac Hazan wrote:
> 
> > 
> 
> > > The attached as well.
> 
> > 
> 
> > > It’s another way to circumvent the problem
> 
> > 
> 
> > >  
> 
> > 
> 
> > > From: elasti...@googlegroups.com [mailto:elasti...@googlegroups.com] On 
> > > Behalf Of Xwilly Azel
> 
> > 
> 
> > > Sent: Thursday, March 20, 2014 1:06 PM
> 
> > 
> 
> > > To: elasti...@googlegroups.com
> 
> > 
> 
> > > Subject: Re: Kibana Histogram unexpected line curve with cumulative value
> 
> > 
> 
> > >  
> 
> > 
> 
> > > 
> 
> > 
> 
> > > here my inspect from kibana  :
> 
> > 
> 
> > > 
> 
> > 
> 
> > >  
> 
> > 
> 
> > > 
> 
> > 
> 
> > >  
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > curl -XGET 
> > > 'http://x:9200/logstash-2014.03.20,logstash-2014.03.19/_search?pretty'
> > >  -d '{
> 
> > 
> 
> > > 
> 
> > 
> 
> > >   "facets": {
> 
> > 
> 
> > > 
> 
> > 
> 
> > >     "0": {
> 
> > 
> 
> > > 
> 
> > 
> 
> > >       "date_histogram": {
> 
> > 
> 
> > > 
> 
> > 
> 
> > >         "key_field": "@timestamp",
> 
> > 
> 
> > > 
> 
> > 
> 
> > >         "value_field": "gclog.gc.mem.after",
> 
> > 
> 
> > > 
> 
> > 
> 
> > >         "interval": "1h"
> 
> > 
> 
> > > 
> 
> > 
> 
> > >       },
> 
> > 
> 
> > > 
> 
> > 
> 
> > >       "global": true,
> 
> > 
> 
> > > 
> 
> > 
> 
> > >       "facet_filter": {
> 
> > 
> 
> > > 
> 
> > 
> 
> > >         "fquery": {
> 
> > 
> 
> > > 
> 
> > 
> 
> > >           "query": {
> 
> > 
> 
> > > 
> 
> > 
> 
> > >             "filtered": {
> 
> > 
> 
> > > 
> 
> > 
> 
> > >               "query": {
> 
> > 
> 
> > > 
> 
> > 
> 
> > >                 "query_string": {
> 
> > 
> 
> > > 
> 
> > 
> 
> > >                   "query": "gc.type:GC OR gc.type:Full"
> 
> > 
> 
> > > 
> 
> > 
> 
> > >                 }
> 
> > 
> 
> > > 
> 
> > 
> 
> > >               },
> 
> > 
> 
> > > 
> 
> > 
> 
> > >               "filter": {
> 
> > 
> 
> > > 
> 
> > 
> 
> > >                 "bool": {
> 
> > 
> 
> > > 
> 
> > 
> 
> > >                   "must": [
> 
> > 
> 
> > > 
> 
> > 
> 
> > >                     {
> 
> > 
> 
> > > 
> 
> > 
> 
> > >                       "range": {
> 
> > 
> 
> > > 
> 
> > 
> 
> > >                         "@timestamp": {
> 
> > 
> 
> > > 
> 
> > 
> 
> > >                           "from": 1395227019780,
> 
> > 
> 
> > > 
> 
> > 
> 
> > >                           "to": "now"
> 
> > 
> 
> > > 
> 
> > 
> 
> > >                         }
> 
> > 
> 
> > > 
> 
> > 
> 
> > >                       }
> 
> > 
> 
> > > 
> 
> > 
> 
> > >                     }
> 
> > 
> 
> > > 
> 
> > 
> 
> > >                   ]
> 
> > 
> 
> > > 
> 
> > 
> 
> > >                 }
> 
> > 
> 
> > > 
> 
> > 
> 
> > >               }
> 
> > 
> 
> > > 
> 
> > 
> 
> > >             }
> 
> > 
> 
> > > 
> 
> > 
> 
> > >           }
> 
> > 
> 
> > > 
> 
> > 
> 
> > >         }
> 
> > 
> 
> > > 
> 
> > 
> 
> > >       }
> 
> > 
> 
> > > 
> 
> > 
> 
> > >     }
> 
> > 
> 
> > > 
> 
> > 
> 
> > >   },
> 
> > 
> 
> > > 
> 
> > 
> 
> > >   "size": 0
> 
> > 
> 
> > > 
> 
> > 
> 
> > > }'
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > On Thursday, March 20, 2014 12:00:29 PM UTC+1, Xwilly Azel wrote:
> 
> > 
> 
> > > 
> 
> > 
> 
> > > I've upgrader to kibana 3.0.0.
> 
> > 
> 
> > > 
> 
> > 
> 
> > >  
> 
> > 
> 
> > > 
> 
> > 
> 
> > > Still the same issue, when I 'm doing an histogram, I don't have option 
> > > to disable zero-fill checkbox.
> 
> > 
> 
> > > 
> 
> > 
> 
> > >  
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > On Thursday, March 20, 2014 8:22:35 AM UTC+1, Isaac Hazan wrote:
> 
> > 
> 
> > > 
> 
> > 
> 
> > > 
> 
> > 
> 
> > > Yes, the problem is because when the histogram encounters buckets with no 
> > > data it assumes zero values instead of joining the points between the 2 
> > > interval.
> 
> > 
> 
> > > I solved it by using a different kibana version, I effectively found 2 
> > > patches one with a zero-fill checkbox and another one with an option box. 
> > > I found just one of them:
> 
> > 
> 
> > > https://github.com/elasticsearch/kibana/issues/741
> 
> > 
> 
> > >  
> 
> > 
> 
> > > Both versions solve the problem.
> 
> > 
> 
> > >  
> 
> > 
> 
> > > That’s the way I solved it.
> 
> > 
> 
> > >  
> 
> > 
> 
> > > Hope that helps
> 
> > 
> 
> > >  
> 
> > 
> 
> > > /Itzhak
> 
> > 
> 
> > >  
> 
> > 
> 
> > > From: elasti...@googlegroups.com [mailto:elasti...@googlegroups.com] On 
> > > Behalf Of Xwilly Azel
> 
> > 
> 
> > > Sent: Thursday, March 20, 2014 9:12 AM
> 
> > 
> 
> > > To: elasti...@googlegroups.com
> 
> > 
> 
> > > Subject: Re: Kibana Histogram unexpected line curve with cumulative value
> 
> > 
> 
> > >  
> 
> > 
> 
> > > 
> 
> > 
> 
> > > I've got the same kind of issue.
> 
> > 
> 
> > > 
> 
> > 
> 
> > >  
> 
> > 
> 
> > > 
> 
> > 
> 
> > > Did you find a solution ?
> 
> > 
> 
> > > 
> 
> > 
> 
> > >

Re: fuzziness & score computation

2014-03-20 Thread Zachary Tong

You are correct in your analysis of the fuzzy scoring.  Fuzzy variants are 
scored (relatively) the same as the exact match, because they are treated 
the same when executed internally.

If you want to score exact matches higher, I would use a boolean 
combination of an exact match and a fuzzy match.  Semi-pseudo-query here:

{
"query": {
"bool": {
"should": [
   {
"match" : {
"my_field" : {
"query" : "car renting london",
"operator" : "and"
},
"boost" : 2
}
},
{
"fuzzy_like_this": {}   
}
]
}
}
}

Basically, the match query is set to AND operator (so all terms are 
required) and it is given a boost of 2.  That means that exact matches will 
be boosted preferentially over the fuzzy matches, which will have the 
default boost of 1.

Also I get results with more terms getting the same score, like "cheap car 
> renting London", "offers car renting London". 
>

The reason you are seeing results like this is because you are using the 
fuzzy_like_this query.  It's a combination of more_like_this and fuzzy. 
 The way MLT works is that it takes all the individual terms in your query, 
builds a big boolean and searches the index for the boolean.  Docs just 
need the terms, in no particular order.  The Fuzzy Like This works the 
same, except terms are allowed to fuzzily match.   With MLT and FLT, you're 
bound to find "off-target" results because these queries are sorta like 
shotguns, looking for a wide spread of terms.

*2) fuzzy query*
>
> That doesn't make what I want since it does not analyze the query (I 
> think) and so it will treat the query in an unexpected way for my purposes 
> of "free text" search
>

As an alternative, you can use the Match query and set the "fuzziness" 
parameter.  You'll get fuzzy like the fuzzy query, but analysis from the 
Match query.

As a general comment, trying to deal with misspellings and fuzziness is 
always a game between precision (number of returned results that are 
correct) and recall (number of correct results that are returned).  As you 
increase fuzziness, you increase recall -- more of your correct results are 
in your search hits...but you lose precision...they may be at position 200. 
 You'll always be battling the precision/recall fight.

I would instead search for exact matches, and prompt user to fix 
mispellings with suggesters.  This makes your search and relevancy *vastly* 
simpler, 
and tends to provide a better user experience because they can just click 
the as-you-type suggestion or the "Did you mean?" link.  Win win for 
everyone.

-Zach

On Thursday, March 20, 2014 4:46:49 AM UTC-5, Adrian Luna wrote:
>
> Hi, 
>
> Sorry that I am relatively fresh to elasticsearch so please don't be too 
> harsh.
>
> I feel like I'm not being able to understand the behaviour of any of the 
> fuzzy queries in ES.
>
> *1) match with fuzziness enabled*
>
> {
>   "query": {
> "fuzzy_like_this_field": {
>   "field_name": {
> "like_text": "car renting London",
> "fuzziness": "0.5"
>   }
> }
>   }
> }
>
> As I see it from my tests, this kind of query will give same score to 
> documents with field_name="car renting London" and "car ranting London" or 
> "car renting Londen" for example. That means, it will not give any 
> negatively score misspellings. I can imagine that first the possible 
> variants are computed and then the score is just computed with a 
> "representative score" which is the same for every variant that match the 
> requirements. 
>
> Am I right? If I am, is it any way to boost the exact match over the fuzzy 
> match?
>
> Also I get results with more terms getting the same score, like "cheap car 
> renting London", "offers car renting London". That's something I cannot get 
> to understand. When I use the explain API, it seems that the resulting 
> score is a sum of the different matches with its internal weightings, 
> tf-idf, etc. but it seems to not be considering the terms outside the 
> query, while I would expect the exact match to score at least slightly 
> higher. 
>
> Am I missing something here? Is it just the expected result and I am just 
> being too demanding?
>
> *2) fuzzy query*
>
> That doesn't make what I want since it does not analyze the query (I 
> think) and so it will treat the query in an unexpected way for my purposes 
> of "free text" search
>
> *3) fuzzy_like_this or fuzzy_like_this_field*
>
> This other search takes rid of the first problem in point 1, since as I 
> read from the documentation, it seems to use some tricks to avoid favouring 
> rare terms (misspellings will be here) over more frequent terms, etc. but 
> it's still giving the same score to exact match and matches where other 
> terms are pres

Re: has_child queries fail if queryName is set

2014-03-20 Thread Riyaz

Thanks Martijn

Tried in v0.90.12 and it works. Thank You!

On Thursday, March 20, 2014 7:21:57 AM UTC-4, Martijn v Groningen wrote:
>
> This is a bug and has been fixed. Can you try using the latest 0.90.x 
> release or maybe upgrade to the latest 1.0.x release?
>
>
> On 20 March 2014 00:02, Riyaz > wrote:
>
>> Hi,
>>
>> I am using elasticsearch v0.90.5 and trying to set QueryName (_name) for 
>> a has_child query as described here: 
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/0.90/search-request-named-queries-and-filters.html
>>
>> But the has_child queries are failing if queryName is set. They also fail 
>> if queryName is set on a wrapping query. You can find the curl recreation 
>> here: https://gist.github.com/hmrizin/9645816
>>
>> What am I missing? Any help appreciated.
>>
>> Thanks
>> Riyaz
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/7bc36970-9bcd-4fc1-b3ab-fe16ca07df93%40googlegroups.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> -- 
> Met vriendelijke groet,
>
> Martijn van Groningen 
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/fe889a90-7afe-4574-ab2d-dc1b89dd601d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Fwd: elasticsearch error of class not found

nobody there to help me , help me please i am in need

On Thursday, March 20, 2014 3:31:42 PM UTC+5:30, Anikessh Jain wrote:
>
> Can anybody help me it is  a bit urgent .
>
>
>
> On Thursday, March 20, 2014 3:09:56 PM UTC+5:30, Anikessh Jain wrote:
>>
>> Hi All.
>>
>> I am a newbie to Elastic search and I am configuring Kibana with Logstash 
>> and Redis  and Elasticsearch  in  Centos 32 Bit and when i am trying to 
>> start the service of elastic search i am getting the below error
>>
>> WrapperSimpleApp Error: Unable
>>
>>  to locate the class org.elasticsearch.bootstrap.ElasticsearchF : 
>> java.lang.ClassNotFoundException: org.elasticsearch.bootstrap.ElasticsearchF
>>
>>
>> How to solve the above error ,i tried many steps but no use  but of no help, 
>> have anybody face the same issue please help me.
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6e075a41-21b4-46f9-a9b3-a9beae1210b4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: elasticsearch java interaction

Actually this is my elasticsearch index  http://localhost:9200/, as you 
told i have replaced 9200 with 9300 in the above code ,then i executed the 
application i am getting following exceptions.

Mar 20, 2014 7:17:45 PM org.elasticsearch.client.transport
WARNING: [Bailey, Gailyn] failed to get node info for 
[#transport#-1][inet[localhost/127.0.0.1:9300]]
org.elasticsearch.transport.NodeDisconnectedException: 
[][inet[localhost/127.0.0.1:9300]][/cluster/nodes/info] disconnected

Connected
Mar 20, 2014 7:17:50 PM org.elasticsearch.client.transport
WARNING: [Bailey, Gailyn] failed to get node info for 
[#transport#-1][inet[localhost/127.0.0.1:9300]]
org.elasticsearch.transport.NodeDisconnectedException: 
[][inet[localhost/127.0.0.1:9300]][/cluster/nodes/info] disconnected

Mar 20, 2014 7:17:50 PM org.elasticsearch.client.transport
WARNING: [Bailey, Gailyn] failed to get node info for 
[#transport#-1][inet[localhost/127.0.0.1:9300]]
org.elasticsearch.transport.NodeDisconnectedException: 
[][inet[localhost/127.0.0.1:9300]][/cluster/nodes/info] disconnected

Thankyou

On Thursday, March 20, 2014 7:12:14 PM UTC+5:30, David Pilato wrote:
>
> Use port 9300
>
> --
> David ;-)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
>
> Le 20 mars 2014 à 14:34, Venu Krishna > a 
> écrit :
>
> Thankyou for the reply.I am not getting any errors,but i am not able to 
> connect to my elasticsearch using java.Here my code.
>
> import java.net.InetSocketAddress;
>
> import org.elasticsearch.client.Client;
> import org.elasticsearch.client.transport.TransportClient;
> import org.elasticsearch.common.transport.InetSocketTransportAddress;
>
>
> public class JavaES_Client {
>
> void function()
> {
> //on StartUp
> System.out.println("In Function");
> 
> Client client = new TransportClient()
> .addTransportAddress(new InetSocketTransportAddress("localhost", 
> 9200));   // This is where my control is getting stuck,without any 
> exceptions or errors.
> 
> 
> System.out.println("Connected");
> //on ShutDown
> client.close();
> }
> 
> 
> public static void main(String[] args) {
> 
> System.out.println("In Main Method");
> JavaES_Client jc = new JavaES_Client();
> System.out.println("Object Created");
> jc.function();
> 
> }
> 
> }
>
>
> On Thursday, March 20, 2014 2:20:25 PM UTC+5:30, Georgi Ivanov wrote:
>>
>> On Linux the file is  /etc/hosts
>> On Windows c:\windows\system32\drivers\etc\hosts
>>
>> Open the file in text editor
>>
>> Add following lines:
>> 192.168.1.100 host1
>> 192.168.1.101 host2
>>
>> Make sure that 192.168.1.100/101 is the right IP address of the 
>> host1/host2
>>
>>
>>
>> 2014-03-20 8:35 GMT+01:00 Venu Krishna :
>>
>>> Hi Georgi Ivanov,
>>>   yes,i am able to understand the Exception i.e. 
>>> UnresolvedAddressException,but you are telling that to make sure host1 and 
>>> host2 are resolved by adding entries to /etc/hosts to wherever the file in 
>>> on Windows,for this can you give me the steps how to approach this.Sorry i 
>>> am new to this and am learning i am unable to get the proper example.Thanks 
>>> in advance for the help.
>>>
>>>
>>> On Thursday, March 20, 2014 2:36:10 AM UTC+5:30, Georgi Ivanov wrote:

 Well
 I think UnresolvedAddressException obviously means that your Java 
 client can not resolve host1 and host2

 make sure host1 and host2 are resolvable  by adding entries to 
 /etc/hosts ot wherever the file in on Windows



 On Friday, March 14, 2014 1:47:02 PM UTC+1, Venu Krishna wrote:
>
> Hi,
>I am Y.Venu,i am totally new to this elasticsearch,now i am 
> trying to communicate java elastisearch,i have gone through the 
> elasticsearch java api's 
>
> 1st i came across maven repositry.
> i have created pom.xml in my eclipse and in the dependency tag i 
> have just placed the code that i found in maven repositry 
>
>  i.e.  
>  
>
> 
> org.elasticsearch
>
> elasticsearch
>
> ${es.version}
> 
>
> After that i have created one class with the main method and i copied and 
> placed the code that i found in the client api of elasticsearch i.e.
>
>  TransportClient.
>
> main()
> {
> Client client = new TransportClient()
>   .addTransportAddress(new 
> InetSocketTransportAddress("host1", 9200))
>   .addTransportAddress(new 
> InetSocketTransportAddress("host2", 9200));
>
>   // on shutdown
>
>   client.close();
>   
>   Settings settings = ImmutableSettings.settingsBuilder()
>   .put("client.transport.sniff", true).build();
>   TransportClient client1 = new TransportClie

Re: elasticsearch java interaction

Use port 9300

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 20 mars 2014 à 14:34, Venu Krishna  a écrit :

Thankyou for the reply.I am not getting any errors,but i am not able to connect 
to my elasticsearch using java.Here my code.

import java.net.InetSocketAddress;

import org.elasticsearch.client.Client;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.transport.InetSocketTransportAddress;


public class JavaES_Client {

void function()
{
//on StartUp
System.out.println("In Function");

Client client = new TransportClient()
.addTransportAddress(new InetSocketTransportAddress("localhost", 
9200));   // This is where my control is getting stuck,without any exceptions 
or errors.


System.out.println("Connected");
//on ShutDown
client.close();
}


public static void main(String[] args) {

System.out.println("In Main Method");
JavaES_Client jc = new JavaES_Client();
System.out.println("Object Created");
jc.function();

}

}


> On Thursday, March 20, 2014 2:20:25 PM UTC+5:30, Georgi Ivanov wrote:
> On Linux the file is  /etc/hosts
> On Windows c:\windows\system32\drivers\etc\hosts
> 
> Open the file in text editor
> 
> Add following lines:
> 192.168.1.100 host1
> 192.168.1.101 host2
> 
> Make sure that 192.168.1.100/101 is the right IP address of the host1/host2
> 
> 
> 
> 2014-03-20 8:35 GMT+01:00 Venu Krishna :
>> Hi Georgi Ivanov,
>>   yes,i am able to understand the Exception i.e. 
>> UnresolvedAddressException,but you are telling that to make sure host1 and 
>> host2 are resolved by adding entries to /etc/hosts to wherever the file in 
>> on Windows,for this can you give me the steps how to approach this.Sorry i 
>> am new to this and am learning i am unable to get the proper example.Thanks 
>> in advance for the help.
>> 
>> 
>>> On Thursday, March 20, 2014 2:36:10 AM UTC+5:30, Georgi Ivanov wrote:
>>> Well
>>> I think UnresolvedAddressException obviously means that your Java client 
>>> can not resolve host1 and host2
>>> 
>>> make sure host1 and host2 are resolvable  by adding entries to /etc/hosts 
>>> ot wherever the file in on Windows
>>> 
>>> 
>>> 
 On Friday, March 14, 2014 1:47:02 PM UTC+1, Venu Krishna wrote:
 Hi,
I am Y.Venu,i am totally new to this elasticsearch,now i am trying 
 to communicate java elastisearch,i have gone through the elasticsearch 
 java api's 
 
 1st i came across maven repositry.
 i have created pom.xml in my eclipse and in the dependency tag i have 
 just placed the code that i found in maven repositry 
 
  i.e.  
  
 
 org.elasticsearch
 
 elasticsearch
 
 ${es.version}
 
 
 
 After that i have created one class with the main method and i copied and 
 placed the code that i found in the client api of elasticsearch i.e.
 
  TransportClient.
 
 main()
 {
 Client client = new TransportClient()
.addTransportAddress(new 
 InetSocketTransportAddress("host1", 9200))
.addTransportAddress(new 
 InetSocketTransportAddress("host2", 9200));
 
 
// on shutdown
 
client.close();

Settings settings = ImmutableSettings.settingsBuilder()
.put("client.transport.sniff", true).build();
TransportClient client1 = new TransportClient(settings);
 
 
 }
 
 After running this app javapplication,i am getting the errors like this
 
 
 
 In Main Method
 Mar 14, 2014 6:05:24 PM org.elasticsearch.node
 INFO: [Mister Machine] {elasticsearch/0.16.1}[11016]: initializing ...
 
 Mar 14, 2014 6:05:24 PM org.elasticsearch.plugins
 INFO: [Mister Machine] loaded []
 org.elasticsearch.common.inject.internal.ComputationException: 
 org.elasticsearch.common.inject.internal.ComputationException: 
 java.lang.NoClassDefFoundError: Lorg/apache/lucene/store/Lock;
 
at 
 org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:553)
at 
 org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:419)
 
at 
 org.elasticsearch.common.inject.internal.CustomConcurrentHashMap$ComputingImpl.get(CustomConcurrentHashMap.java:2041)
at 
 org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:46)
 
at 
 org.elasticsearch.common.inject.ConstructorInjectorStore.get(ConstructorInjectorStore.java:52)
at 
 org.elasticsearch.common.inject.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:57)
 
at

Re: Confusing highlight result when creating many tokens

2014-03-20 Thread Jon-Paul Lussier

Hi Elasticsearch, still waiting to see if this is a known issue, possibly 
that's resolved in a future release, or if this is something I did? I'd 
appreciate knowing, at least, if anyone can help. Thanks much.

On Friday, March 14, 2014 5:29:10 PM UTC-4, Jon-Paul Lussier wrote:
>
> Hey Elasticsearch, hopefully someone can at least explain if this is 
> intentional and how it happens(I have had other fragment highlighting 
> issues not unlike this)
>
> The problem seems simple, I have a 64 character string that I generate 62 
> tokens for. Whenever I search for the entire string, I end up getting the 
> highlight applied to the 50th fragment instead of the one that actually 
> most nearly matches my search query.
>
> Also confusing is if I try a very similar search, trying to use an exact 
> match on the SHA1 or MD5 attributes -- highlighting works like I'd expect 
> it to.
>
>
> Please see the gist here: 
> https://gist.github.com/jonpaul/d4a9aa7f9c8741933cf5
>
>
> Currently I'm using 1.0.0-BETA2 so this *may* be a fixed bug, sorry if 
> that's the case, I couldn't find anything that matches my problem per se.
>
> Thanks very much in advance for help anyone can provide!
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e2a9657d-e5df-4e0c-b1dc-78b13457827c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: elasticsearch java interaction

Thankyou for the reply.I am not getting any errors,but i am not able to 
connect to my elasticsearch using java.Here my code.

import java.net.InetSocketAddress;

import org.elasticsearch.client.Client;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.transport.InetSocketTransportAddress;


public class JavaES_Client {

void function()
{
//on StartUp
System.out.println("In Function");

Client client = new TransportClient()
.addTransportAddress(new InetSocketTransportAddress("localhost", 
9200));   // This is where my control is getting stuck,without any 
exceptions or errors.


System.out.println("Connected");
//on ShutDown
client.close();
}


public static void main(String[] args) {

System.out.println("In Main Method");
JavaES_Client jc = new JavaES_Client();
System.out.println("Object Created");
jc.function();

}

}


On Thursday, March 20, 2014 2:20:25 PM UTC+5:30, Georgi Ivanov wrote:
>
> On Linux the file is  /etc/hosts
> On Windows c:\windows\system32\drivers\etc\hosts
>
> Open the file in text editor
>
> Add following lines:
> 192.168.1.100 host1
> 192.168.1.101 host2
>
> Make sure that 192.168.1.100/101 is the right IP address of the 
> host1/host2
>
>
>
> 2014-03-20 8:35 GMT+01:00 Venu Krishna >:
>
>> Hi Georgi Ivanov,
>>   yes,i am able to understand the Exception i.e. 
>> UnresolvedAddressException,but you are telling that to make sure host1 and 
>> host2 are resolved by adding entries to /etc/hosts to wherever the file in 
>> on Windows,for this can you give me the steps how to approach this.Sorry i 
>> am new to this and am learning i am unable to get the proper example.Thanks 
>> in advance for the help.
>>
>>
>> On Thursday, March 20, 2014 2:36:10 AM UTC+5:30, Georgi Ivanov wrote:
>>>
>>> Well
>>> I think UnresolvedAddressException obviously means that your Java 
>>> client can not resolve host1 and host2
>>>
>>> make sure host1 and host2 are resolvable  by adding entries to 
>>> /etc/hosts ot wherever the file in on Windows
>>>
>>>
>>>
>>> On Friday, March 14, 2014 1:47:02 PM UTC+1, Venu Krishna wrote:

 Hi,
I am Y.Venu,i am totally new to this elasticsearch,now i am 
 trying to communicate java elastisearch,i have gone through the 
 elasticsearch java api's 

 1st i came across maven repositry.
 i have created pom.xml in my eclipse and in the dependency tag i 
 have just placed the code that i found in maven repositry 

  i.e.  
  

 
 org.elasticsearch

 elasticsearch

 ${es.version}
 

 After that i have created one class with the main method and i copied and 
 placed the code that i found in the client api of elasticsearch i.e.

  TransportClient.

 main()
 {
 Client client = new TransportClient()
.addTransportAddress(new 
 InetSocketTransportAddress("host1", 9200))
.addTransportAddress(new 
 InetSocketTransportAddress("host2", 9200));

// on shutdown

client.close();

Settings settings = ImmutableSettings.settingsBuilder()
.put("client.transport.sniff", true).build();
TransportClient client1 = new TransportClient(settings);

 }

 After running this app javapplication,i am getting the errors like this



 In Main Method
 Mar 14, 2014 6:05:24 PM org.elasticsearch.node
 INFO: [Mister Machine] {elasticsearch/0.16.1}[11016]: initializing ...

 Mar 14, 2014 6:05:24 PM org.elasticsearch.plugins
 INFO: [Mister Machine] loaded []
 org.elasticsearch.common.inject.internal.ComputationException: 
 org.elasticsearch.common.inject.internal.ComputationException: 
 java.lang.NoClassDefFoundError: Lorg/apache/lucene/store/Lock;

at 
 org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:553)
at 
 org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:419)

at 
 org.elasticsearch.common.inject.internal.CustomConcurrentHashMap$ComputingImpl.get(CustomConcurrentHashMap.java:2041)
at 
 org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:46)

at 
 org.elasticsearch.common.inject.ConstructorInjectorStore.get(ConstructorInjectorStore.java:52)
at 
 org.elasticsearch.common.inject.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:57)

at 
 org.elasticsearch.common.inject.InjectorImpl.initializeBinding(InjectorImpl.java:377)
at 
 org.elasticsearch.common.inject.BindingProcessor$1$1.run(Bind

Re: Install issues with Kibana3 vs elasticsearch 0.19.11

2014-03-20 Thread Alex at Ikanow

Never mind, I'm an idiot, it clearly mentions it needs 0.90.x in the README 
:(

On Wednesday, March 19, 2014 12:49:46 PM UTC-4, Alex at Ikanow wrote:
>
>
> I downloaded the latest Kibana3, popped it on a tomcat instance sharing 
> space with my elasticsearch (0.19.11) instance and tried to connect (both: 
> using an ssh tunnel to connect localhost:9200 back to the server, and 
> opening port 9200 in the firewall)
>
> In both cases, the browser makes a call to _nodes (eg returns 
> "{"ok":true,"cluster_name":"infinite-dev","nodes":{"Yup-Cmn0QwCrkYI6l7SdRw":{"name":"Firefrost","transport_address":"inet[/
> 10.113.42.186:9300]","hostname":"ip-10-113-42-186","http_address":"inet[/
> 10.113.42.186:9200]"}}}")
>
> and then returns the following error:
>
> TypeError: Cannot call method 'split' of undefined at 
> http://SERVER/kibana-3.0.0/app/app.js:22:11260at
>  he (
> http:// 
> SERVER/kibana-3.0.0/app/app.js:7:20041)
>  
> at Function.Yb (http:// 
> SERVER 
> /kibana-3.0.0/app/app.js:7:7025)
>  
> at http:// 
> SERVER/kibana-3.0.0/app/app.js:22:11204
>  
> at i (http:// 
> SERVER/kibana-3.0.0/app/app.js:9:458)
>  
> at i (http:// 
> SERVER/kibana-3.0.0/app/app.js:9:458)
>  
> at http:// 
> SERVER/kibana-3.0.0/app/app.js:9:1014
>  
> at Object.f.$eval (http:// 
> SERVER 
> /kibana-3.0.0/app/app.js:9:6963)
>  
> at Object.f.$digest (http://
> SERVER 
> /kibana-3.0.0/app/app.js:9:5755)
>  
> at Object.f.$apply (http://
> SERVER 
> /kibana-3.0.0/app/app.js:9:7111) 
>
> I don't see any other calls back to elasticseach
>
> I couldn't find a statement anywhere of which versions Kibana3 is 
> compatible with - does it just need a later version (anyone know the 
> earliest with which it is compatible, out of curiosity; though I'm planning 
> to move to 1.0 anyway soon), or am I doing something wrong
>
> Thanks for any insight/help anyone can provide!
>
> Alex
>
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c6a2f998-89a9-4794-bdf8-15d1dcd26aae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Kibana Histogram unexpected line curve with cumulative value

I have the same issue with your version and I don't see where in kibana i 
can say : disable zero-fill checkbox.

On Thursday, March 20, 2014 2:29:24 PM UTC+1, Xwilly Azel wrote:
>
> On Thursday, March 20, 2014 1:15:44 PM UTC+1, Isaac Hazan wrote: 
> > The attached as well. 
> > It’s another way to circumvent the problem 
> >   
> > From: elasti...@googlegroups.com [mailto:elasti...@googlegroups.com] On 
> Behalf Of Xwilly Azel 
> > Sent: Thursday, March 20, 2014 1:06 PM 
> > To: elasti...@googlegroups.com 
> > Subject: Re: Kibana Histogram unexpected line curve with cumulative 
> value 
> >   
> > 
> > here my inspect from kibana  : 
> > 
> >   
> > 
> >   
> > 
> > 
> > curl -XGET '
> http://x:9200/logstash-2014.03.20,logstash-2014.03.19/_search?pretty' 
> -d '{ 
> > 
> >   "facets": { 
> > 
> > "0": { 
> > 
> >   "date_histogram": { 
> > 
> > "key_field": "@timestamp", 
> > 
> > "value_field": "gclog.gc.mem.after", 
> > 
> > "interval": "1h" 
> > 
> >   }, 
> > 
> >   "global": true, 
> > 
> >   "facet_filter": { 
> > 
> > "fquery": { 
> > 
> >   "query": { 
> > 
> > "filtered": { 
> > 
> >   "query": { 
> > 
> > "query_string": { 
> > 
> >   "query": "gc.type:GC OR gc.type:Full" 
> > 
> > } 
> > 
> >   }, 
> > 
> >   "filter": { 
> > 
> > "bool": { 
> > 
> >   "must": [ 
> > 
> > { 
> > 
> >   "range": { 
> > 
> > "@timestamp": { 
> > 
> >   "from": 1395227019780, 
> > 
> >   "to": "now" 
> > 
> > } 
> > 
> >   } 
> > 
> > } 
> > 
> >   ] 
> > 
> > } 
> > 
> >   } 
> > 
> > } 
> > 
> >   } 
> > 
> > } 
> > 
> >   } 
> > 
> > } 
> > 
> >   }, 
> > 
> >   "size": 0 
> > 
> > }' 
> > 
> > 
> > On Thursday, March 20, 2014 12:00:29 PM UTC+1, Xwilly Azel wrote: 
> > 
> > I've upgrader to kibana 3.0.0. 
> > 
> >   
> > 
> > Still the same issue, when I 'm doing an histogram, I don't have option 
> to disable zero-fill checkbox. 
> > 
> >   
> > 
> > 
> > On Thursday, March 20, 2014 8:22:35 AM UTC+1, Isaac Hazan wrote: 
> > 
> > 
> > Yes, the problem is because when the histogram encounters buckets with 
> no data it assumes zero values instead of joining the points between the 2 
> interval. 
> > I solved it by using a different kibana version, I effectively found 2 
> patches one with a zero-fill checkbox and another one with an option box. I 
> found just one of them: 
> > https://github.com/elasticsearch/kibana/issues/741 
> >   
> > Both versions solve the problem. 
> >   
> > That’s the way I solved it. 
> >   
> > Hope that helps 
> >   
> > /Itzhak 
> >   
> > From: elasti...@googlegroups.com [mailto:elasti...@googlegroups.com] On 
> Behalf Of Xwilly Azel 
> > Sent: Thursday, March 20, 2014 9:12 AM 
> > To: elasti...@googlegroups.com 
> > Subject: Re: Kibana Histogram unexpected line curve with cumulative 
> value 
> >   
> > 
> > I've got the same kind of issue. 
> > 
> >   
> > 
> > Did you find a solution ? 
> > 
> > On Monday, March 10, 2014 3:58:21 PM UTC+1, Isaac Hazan wrote: 
> > 
> > Kibana cannot do a the Histogram of the cumulative value of a field as 
> describe at:https://github.com/elasticsearch/kibana/issues/740 
> > To overcome that I created a separate index where I calculate myself the 
> total and saved it to Elasticsearch. 
> > The mapping looks as follows:curl -XPOST localhost:9200/first_install -d 
> '{"settings" : {"number_of_shards" : 5},"mappings" : 
> {"fi" : {"properties" : {"evtTime" : { 
> "type" : "date", "index": "not_analyzed", "format": "dd/MMM/:HH:mm:ss" 
> },"cumulativeValue" : { "type" : "integer", "index": 
> "not_analyzed" }}}}}' 
> > The values are saved properly but unexpectedly Kibana does not draw the 
> line i would expect, instead it joins between point that do not exist. 
> > Following is the Kibana sreenshot: 
> > The line curve should always be increasing since my data set is always 
> increasing, that i can prove by the following events as seen by kibana 
> itself: 
> > Could it be related to the data formatting I did? 
> > Thx in advance. 
> > -- 
> > You received this message because you are subscribed to a topic in the 
> Google Groups "elasticsearch" group. 
> > To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/elasticsearch/mp2hUkR8U6w/unsubscribe. 
> > To unsubscribe from this group and all its topics, send an email to 
> elasticsearc...@googlegroups.com. 
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/9fe1f350-c5fb-47f2-8925-2d52b7376d94%4

How to filter a value from multivalue field in Elasticsearch

2014-03-20 Thread Anurag Tripathi

 

Hi, I have a document indexed in this format.

"hits": [
 {
"_index": "temp",
"_type": "test",
"_source": {
   "brand": [
  "A",
  "B",
  "C",
  "D",
  "E",
  "F",
  "G"
   ],
   "search_term": "",
   "age": "19-35 Years",
   "dob": "1981-06-01T00:00:00.000Z",
   "deal": [
  "P",
  "Q",
  "R",
  "S",
   ]
}
 }

I want to extract information related to "brand" : "C".I applied a term 
filter on the created alias but It's not giving me the particular 
value,however it gives me all the values (brand : 
"A","B","C","D","E","F","G".Any trick to get "C" and all the values related 
to "C" ? Here is the term filter which I am applying.Not sure,If I have to 
apply a custom CSV analyzer ? 

POST /_aliases
{
"actions" : [
{
"add" : {
 "index" : "temp",
 "alias" : "temp_alias",
"filter" : { 
  "or" : [
{
"term" : { "brand" : "C" }
}
]
 }
}
}
]
}


brand and deal are having this mapping.

"brand": { "type": "string", "index": "not_analyzed" }


Thanks,

Anurag

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f62d1ac3-0614-4034-a664-eab868cb1fd7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Complete cluster failure

2014-03-20 Thread Zachary Tong

Nice post-mortem, thanks for the writeup.  Hopefully someone will stumble 
on this in the future and avoid the same headache you had :)

How would you force IPV4? I tried using preferIPv4Stack and setting 
> network.host to _eth0:ipv4_, but it still did not work. Even switched off 
> iptables at a point!
>
 
Hmm...that's interesting.  I would have recommended those two exact 
methods.  I'll do some digging and see why they didn't work... 

-Z



On Thursday, March 20, 2014 1:23:48 AM UTC-5, Ivan Brusic wrote:
>
> Responses inline.
>
> On Wed, Mar 19, 2014 at 7:25 PM, Zachary Tong 
> 
> > wrote:
>
>> Yeah, in case anyone reads this thread in the future, this log output is 
>> a good indicator of multicast problems.  You can see that the the nodes are 
>> pinging and talking to each other on this log line:
>>
>> --> target [[search6][T3tINFmqREK9W6oqZV0r7A][inet[/192.168.50.106:9300]]], 
>> master [null]
>>
>>
>> That's basically a ping response from a node saying "Hey!  I'm alive, but 
>> I dunno who the master is yet".  If the nodes were unable to communicate, 
>> you'd see failed ping responsese instead.  But they are unable to fully 
>> connect and start to establish/elect a master.  So they all just sit around 
>> and ping each other for ages until giving up.  It isn't always, but 
>> usually, a multicast issue when you see logs like this.  You can usually 
>> diagnose this by manually telnetting between nodes on port 9300...if the 
>> connection isn't refused, its probably a multicast discovery issue.
>>
>
> Once everything was stabilized and I was able to get some sleep, I looked 
> at the code (ZenDiscovery.java) the next day and realized exactly what you 
> said. Everything has been running on the existing infrastructure since 
> 2012, so I did not initially put blame on multicast discovery.
>
> The other common culprit is IPv6 issue, often when you have multiple NICs. 
>>  You'll see the publish address bind to IPv6 and the bind address on 
>> IPv4...and the whole cluster goes to hell because the nodes can see each 
>> other but not communicate.
>>
>
> Once I realized it was a multicast issue, I blamed the sysadmins because 
> it wasn't my fault. :) I assumed it was an issue communicating between the 
> underlying VM hosts, but I was able to replicate the issue on physical 
> machines as well. The sysadmin suspected IPV6. We are indeed running IPV6 
> and looking at the logs in the initial post the bound and publish address 
> are indeed IPV4 and IPV6 respectively. Tried 
> setting java.net.preferIPv4Stack=true in JAVA_OPTS, but it did not make a 
> difference. Ultimately I decided that even if we discovered and fix the 
> culprit, we might get bitten again in the future, so I switched to unicast. 
> I know have to keep track of different config files in source control 
> instead of one.
>
>
>> Discovery issues are generally fixed by disabling multicast and forcing 
>> IPv4 in my experience.
>>
>
> How would you force IPV4? I tried using preferIPv4Stack and setting 
> network.host to _eth0:ipv4_, but it still did not work. Even switched off 
> iptables at a point!
>
>
>> Glad you got it working again Ivan! :)
>>
>
> You and me both!
>
> Cheers,
>
> Ivan
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5750058e-5305-4408-93c2-ee9ddf3ea300%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [hadoop] Getting elasticsearch-hadoop working with Shark

I recommend using master - there are several improvements done in this area. Also using the latest Shark (0.9.0) and 
Hive (0.12) will help.


On 3/20/14 12:00 PM, Nick Pentreath wrote:

Hi

I am struggling to get this working too. I'm just trying locally for now, 
running Shark 0.8.1, Hive 0.9.0 and ES 1.0.1
with ES-hadoop 1.3.0.M2.

I managed to get a basic example working with WRITING into an index. But I'm 
really after READING and index.

I believe I have set everything up correctly, I've added the jar to Shark:
ADD JAR /path/to/es-hadoop.jar;

created a table:
CREATE EXTERNAL TABLE test_read (name string, price double)

STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'

TBLPROPERTIES('es.resource' = 'test_index/test_type/_search?q=*');


And then trying to 'SELECT * FROM test _read' gives me :

org.apache.spark.SparkException: Job aborted: Task 3.0:0 failed more than 0 
times; aborting job
java.lang.ClassCastException: 
org.elasticsearch.hadoop.hive.EsHiveInputFormat$ESHiveSplit cannot be cast to
org.elasticsearch.hadoop.hive.EsHiveInputFormat$ESHiveSplit

at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:827)

at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:825)

at 
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)

at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)

at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:825)

at org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:440)

at 
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$run(DAGScheduler.scala:502)

at org.apache.spark.scheduler.DAGScheduler$$anon$1.run(DAGScheduler.scala:157)

FAILED: Execution Error, return code -101 from shark.execution.SparkTask


In fact I get the same error thrown when trying to READ from the table that I 
successfully WROTE to...

On Saturday, 22 February 2014 12:31:21 UTC+2, Costin Leau wrote:

Yeah, it might have been some sort of network configuration issue where 
services where running on different machines
and
localhost pointed to a different location.

Either way, I'm glad to hear things have are moving forward.

Cheers,

On 22/02/2014 1:06 AM, Max Lang wrote:
> I managed to get it working on ec2 without issue this time. I'd say the 
biggest difference was that this time I set up a
> dedicated ES machine. Is it possible that, because I was using a cluster with 
slaves, when I used "localhost" the slaves
> couldn't find the ES instance running on the master? Or do all the 
requests go through the master?
>
>
> On Wednesday, February 19, 2014 2:35:40 PM UTC-8, Costin Leau wrote:
>
> Hi,
>
> Setting logging in Hive/Hadoop can be tricky since the log4j needs to 
be picked up by the running JVM otherwise you
> won't see anything.
> Take a look at this link on how to tell Hive to use your logging 
settings [1].
>
> For the next release, we might introduce dedicated exceptions for the 
simple fact that some libraries, like Hive,
> swallow the stack trace and it's unclear what the issue is which 
makes the exception (IllegalStateException) ambiguous.
>
> Let me know how it goes and whether you will encounter any issues 
with Shark. Or if you don't :)
>
> Thanks!
>
> 
[1]https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-ErrorLogs


> 
>
>
> On 20/02/2014 12:02 AM, Max Lang wrote:
> > Hey Costin,
> >
> > Thanks for the swift reply. I abandoned EC2 to take that out of the 
equation and managed to get everything working
> > locally using the latest version of everything (though I realized 
just now I'm still on hive 0.9). I'm guessing you're
> > right about some port connection issue because I definitely had ES 
running on that machine.
> >
> > I changed hive-log4j.properties and added
> > |
> > #custom logging levels
> > #log4j.logger.xxx=DEBUG
> > log4j.logger.org.elasticsearch.hadoop.rest=TRACE
> >log4j.logger.org.elasticsearch.hadoop.mr 

>=TRACE
> > |
> >
> > But I didn't see any trace logging. Hopefully I can get it working 
on EC2 without issue, but, for the future, is this
> > the correct way to set TRACE logging?
> >
> > Oh and, for reference,

How to Upgrade ElasticSearch from 0.20.4 to 0.90.11 or RC1 ?

2014-03-20 Thread Rok Inroks


Hi,

 I've been assigned to upgrade a single node production Elasticsearch 
server.

The ideal would be to have an  upgrade process description witch avoid 
losing data .  So my question taking this requirement into account is it 
possible to upgrade such a server knowing from ElasticSearch version 0.20.4 
straight to 090.11  ( or even 1.0.0 RC1)  and what the instruction to 
accomplish it ?  


Thanks for the helps

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b3427233-2d4c-438b-aed4-14e925e347d9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Percolator sometimes doesn't refresh queries

2014-03-20 Thread Tomasz Romanczuk

I indexed 10 queries in percolator index. Next 9 was deleted. 
Sometimes it looks like index didn't refresh (I repeated step many times), 
deleted queries are still matched and returned in  resposne. I tried to 
clear cache and refresh index, but sometimes it doesn't work, my code:

BulkRequestBuilder bulkRequest = client.prepareBulk();
while (some condition) {
bulkRequest.add(client.prepareIndex("_percolator", INDEX_NAME, 
id).setSource(...));
}
while (some condition) {
bulkRequest.add(client.prepareDelete("_percolator", INDEX_NAME, id));
}
BulkResponse response = bulkRequest.setRefresh(true).execute().actionGet();
client.admin().indices().prepareClearCache(INDEX_NAME).execute().actionGet();
client.admin().indices().prepareClearCache(PERCOLATOR).execute().actionGet();
client.admin().indices().prepareRefresh(INDEX_NAME).execute().actionGet();
client.admin().indices().prepareRefresh(PERCOLATOR).execute().actionGet();

How to make sure that after bulk request (response.hasFailures() always 
returns false) percolator index will be refreshed?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4315acb8-8e7e-40bf-a89c-18d8fa476408%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Disable gateway completely, now we have snapshot/restore?

2014-03-20 Thread madsmartin

Hello together,

I'm looking into getting rid of the s3 gateway. I'm snapshotting to s3 and 
use extensive replication (each shard replicated to 4 machines). I the 
unlikely case that I loose a shard completely, then I can recover from 
backup, and I even have to possibility to repopulate the data from scratch 
to get data lost since last snapshot repopulated.

Why use a gateway at all then? How do I completely disable gateway?

Sincerely,
Mads Martin

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/926d9daf-9ea4-4fb3-a7bc-882ac0340811%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: has_child queries fail if queryName is set

2014-03-20 Thread Martijn v Groningen

This is a bug and has been fixed. Can you try using the latest 0.90.x
release or maybe upgrade to the latest 1.0.x release?


On 20 March 2014 00:02, Riyaz  wrote:

> Hi,
>
> I am using elasticsearch v0.90.5 and trying to set QueryName (_name) for a
> has_child query as described here:
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/0.90/search-request-named-queries-and-filters.html
>
> But the has_child queries are failing if queryName is set. They also fail
> if queryName is set on a wrapping query. You can find the curl recreation
> here: https://gist.github.com/hmrizin/9645816
>
> What am I missing? Any help appreciated.
>
> Thanks
> Riyaz
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/7bc36970-9bcd-4fc1-b3ab-fe16ca07df93%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Met vriendelijke groet,

Martijn van Groningen

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CA%2BA76TzEDDc8snopE_pgv%2BpTfUsm0XEpH2ZncB7K8O%3DkAEnb1A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Kibana Histogram unexpected line curve with cumulative value

here my inspect from kibana  :


curl -XGET 
'http://x:9200/logstash-2014.03.20,logstash-2014.03.19/_search?pretty' 
-d '{
  "facets": {
"0": {
  "date_histogram": {
"key_field": "@timestamp",
"value_field": "gclog.gc.mem.after",
"interval": "1h"
  },
  "global": true,
  "facet_filter": {
"fquery": {
  "query": {
"filtered": {
  "query": {
"query_string": {
  "query": "gc.type:GC OR gc.type:Full"
}
  },
  "filter": {
"bool": {
  "must": [
{
  "range": {
"@timestamp": {
  "from": 1395227019780,
  "to": "now"
}
  }
}
  ]
}
  }
}
  }
}
  }
}
  },
  "size": 0
}'

On Thursday, March 20, 2014 12:00:29 PM UTC+1, Xwilly Azel wrote:
>
> I've upgrader to kibana 3.0.0.
>
> Still the same issue, when I 'm doing an histogram, I don't have option to 
> disable zero-fill checkbox.
>
>
> On Thursday, March 20, 2014 8:22:35 AM UTC+1, Isaac Hazan wrote:
>>
>> Yes, the problem is because when the histogram encounters buckets with no 
>> data it assumes zero values instead of joining the points between the 2 
>> interval.
>>
>> I solved it by using a different kibana version, I effectively found 2 
>> patches one with a zero-fill checkbox and another one with an option box. I 
>> found just one of them:
>>
>> https://github.com/elasticsearch/kibana/issues/741
>>
>>  
>>
>> Both versions solve the problem.
>>
>>  
>>
>> That’s the way I solved it.
>>
>>  
>>
>> Hope that helps
>>
>>  
>>
>> /Itzhak
>>
>>  
>>
>> *From:* elasti...@googlegroups.com [mailto:elasti...@googlegroups.com] *On 
>> Behalf Of *Xwilly Azel
>> *Sent:* Thursday, March 20, 2014 9:12 AM
>> *To:* elasti...@googlegroups.com
>> *Subject:* Re: Kibana Histogram unexpected line curve with cumulative 
>> value
>>
>>  
>>
>> I've got the same kind of issue.
>>
>>  
>>
>> Did you find a solution ?
>>
>> On Monday, March 10, 2014 3:58:21 PM UTC+1, Isaac Hazan wrote:
>>
>> Kibana cannot do a the Histogram of the cumulative value of a field as 
>> describe at:https://github.com/elasticsearch/kibana/issues/740
>>
>> To overcome that I created a separate index where I calculate myself the 
>> total and saved it to Elasticsearch.
>>
>> The mapping looks as follows:
>>
>> curl -XPOST localhost:9200/first_install -d '{
>>
>> "settings" : {
>>
>> "number_of_shards" : 5
>>
>> },
>>
>> "mappings" : {
>>
>> "fi" : {
>>
>> "properties" : {
>>
>> "evtTime" : { "type" : "date", "index": "not_analyzed", 
>> "format": "dd/MMM/:HH:mm:ss" },
>>
>> "cumulativeValue" : { "type" : "integer", "index": 
>> "not_analyzed" }
>>
>> }
>>
>> }
>>
>> }
>>
>> }'
>>
>> The values are saved properly but unexpectedly Kibana does not draw the 
>> line i would expect, instead it joins between point that do not exist.
>>
>> Following is the Kibana sreenshot:
>>
>> The line curve should always be increasing since my data set is always 
>> increasing, that i can prove by the following events as seen by kibana 
>> itself:
>>
>> Could it be related to the data formatting I did?
>>
>> Thx in advance.
>>
>> -- 
>> You received this message because you are subscribed to a topic in the 
>> Google Groups "elasticsearch" group.
>> To unsubscribe from this topic, visit 
>> https://groups.google.com/d/topic/elasticsearch/mp2hUkR8U6w/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to 
>> elasticsearc...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/9fe1f350-c5fb-47f2-8925-2d52b7376d94%40googlegroups.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/71e13a70-e6a6-4896-a050-62f2462ab085%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Kibana Histogram unexpected line curve with cumulative value

I've upgrader to kibana 3.0.0.

Still the same issue, when I 'm doing an histogram, I don't have option to 
disable zero-fill checkbox.


On Thursday, March 20, 2014 8:22:35 AM UTC+1, Isaac Hazan wrote:
>
> Yes, the problem is because when the histogram encounters buckets with no 
> data it assumes zero values instead of joining the points between the 2 
> interval.
>
> I solved it by using a different kibana version, I effectively found 2 
> patches one with a zero-fill checkbox and another one with an option box. I 
> found just one of them:
>
> https://github.com/elasticsearch/kibana/issues/741
>
>  
>
> Both versions solve the problem.
>
>  
>
> That’s the way I solved it.
>
>  
>
> Hope that helps
>
>  
>
> /Itzhak
>
>  
>
> *From:* elasti...@googlegroups.com  [mailto:
> elasti...@googlegroups.com ] *On Behalf Of *Xwilly Azel
> *Sent:* Thursday, March 20, 2014 9:12 AM
> *To:* elasti...@googlegroups.com 
> *Subject:* Re: Kibana Histogram unexpected line curve with cumulative 
> value
>
>  
>
> I've got the same kind of issue.
>
>  
>
> Did you find a solution ?
>
> On Monday, March 10, 2014 3:58:21 PM UTC+1, Isaac Hazan wrote:
>
> Kibana cannot do a the Histogram of the cumulative value of a field as 
> describe at:https://github.com/elasticsearch/kibana/issues/740
>
> To overcome that I created a separate index where I calculate myself the 
> total and saved it to Elasticsearch.
>
> The mapping looks as follows:
>
> curl -XPOST localhost:9200/first_install -d '{
>
> "settings" : {
>
> "number_of_shards" : 5
>
> },
>
> "mappings" : {
>
> "fi" : {
>
> "properties" : {
>
> "evtTime" : { "type" : "date", "index": "not_analyzed", 
> "format": "dd/MMM/:HH:mm:ss" },
>
> "cumulativeValue" : { "type" : "integer", "index": 
> "not_analyzed" }
>
> }
>
> }
>
> }
>
> }'
>
> The values are saved properly but unexpectedly Kibana does not draw the 
> line i would expect, instead it joins between point that do not exist.
>
> Following is the Kibana sreenshot:
>
> The line curve should always be increasing since my data set is always 
> increasing, that i can prove by the following events as seen by kibana 
> itself:
>
> Could it be related to the data formatting I did?
>
> Thx in advance.
>
> -- 
> You received this message because you are subscribed to a topic in the 
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/elasticsearch/mp2hUkR8U6w/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to 
> elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/9fe1f350-c5fb-47f2-8925-2d52b7376d94%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e0bded3c-2a37-4d6f-af45-178dc9ecc053%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Ann] Elasticsearch Image Plugin 1.2.0 released

Awesome demo! Very well done!

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 20 mars 2014 à 11:41:24, Kevin Wang (kevin807...@gmail.com) a écrit:

Hi All,

I've released version 1.2.0 of Elasticsearch Image Plugin.
The Image Plugin is an Content Based Image Retrieval Plugin for Elasticsearch
using LIRE (Lucene Image Retrieval). It allows users to index images and search
for similar images.

Changes in 1.2.0:
Use multi-thread when multiple features are required to improve index speed
Allow index metadata
Allow query by existing image in index

https://github.com/kzwang/elasticsearch-image

Also I've created a demo website for this plugin
(http://demo.elasticsearch-image.com/), it has 1,000,000 images from
MIRFLICKR-1M collection (http://press.liacs.nl/mirflickr)

Thanks,
Kevin

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e43d34ff-e0e6-4d3a-a3d2-565bb6867e67%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/etPan.532ac7bf.238e1f29.c9bd%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

[Ann] Elasticsearch Image Plugin 1.2.0 released

2014-03-20 Thread Kevin Wang

Hi All,

I've released version 1.2.0 of Elasticsearch Image Plugin.
The Image Plugin is an Content Based Image Retrieval Plugin for 
Elasticsearch using LIRE (Lucene Image Retrieval). It allows users to index 
images and search for similar images.

Changes in 1.2.0:

   - Use multi-thread when multiple features are required to improve index 
   speed
   - Allow index metadata
   - Allow query by existing image in index
   


https://github.com/kzwang/elasticsearch-image

Also I've created a demo website for this plugin 
(http://demo.elasticsearch-image.com/), it has 1,000,000 images  from 
MIRFLICKR-1M collection (http://press.liacs.nl/mirflickr)


Thanks,
Kevin

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e43d34ff-e0e6-4d3a-a3d2-565bb6867e67%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ElasticSearch to SQL sync

2014-03-20 Thread joergpra...@gmail.com

As David said, for push, you must modify your middleware that performs the
insert/update/delete - there is nothing ES can do for you. You must add an
ES client that can execute the respective operations on your data against
an ES cluster.

Poll method does not scale, push does scale.

I do not understand JDBC river is not mature, I happily accept bug reports
and patches. Maybe you discovered that river instances do not scale and are
kind of limited.

Jörg



On Thu, Mar 20, 2014 at 10:48 AM, Komal Parekh wrote:

> Yes we have done some implementation on river but it does not give real
> time data. And also it is not as much mature. So for real time data we need
> to use some push mechanism which can help up to have realtime data.
>
> On Thursday, March 20, 2014 3:13:13 PM UTC+5:30, Jörg Prante wrote:
>
>> Have you tried the JDBC river for poll mechanism?
>>
>> https://github.com/jprante/elasticsearch-river-jdbc/
>>
>> Jörg
>>
>>
>> On Thu, Mar 20, 2014 at 9:50 AM, Komal Parekh wrote:
>>
>>>
>>> Thanks David for your prompt response. But we want some automatic Push
>>> or poll mechanism for this.
>>>
>>>
>>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/d1e6-c85b-46cc-8d7d-b41a763ae3f0%40googlegroups.com
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFmQ_w8iJo0Bdw%2BvS9s8sEfajgxi1MTeZabpSdUNkHaMQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to delete queries from percolator?

2014-03-20 Thread Tomasz Romanczuk

Got it! Seems that clear cache is workaround :)
client.admin().indices().prepareClearCache(INDEX_NAME).execute().actionGet();

W dniu czwartek, 20 marca 2014 11:17:02 UTC+1 użytkownik Tomasz Romanczuk 
napisał:
>
> I have indexed 1 queries in percolator. Next I want to update some of 
> them and delete 9000 queries. I use bulk request, operation seems to finish 
> with success (without fails). But after all deleted queries still are 
> returned. Below code refreshing index:
>
> BulkRequestBuilder bulkRequest = client.prepareBulk();
> while (some condition) {
> bulkRequest.add(client.prepareIndex("_percolator", INDEX_NAME, 
> id).setSource(...));
> }
> while (some condition) {
> bulkRequest.add(client.prepareDelete("_percolator", INDEX_NAME, id));
> }
> BulkResponse response = bulkRequest.setRefresh(true).execute().actionGet();
>
> response.hasFailures() returns *false*. Is there any bug in elasticsearch 
> or am I doing something wrong? Application restart helps, index is 
> refreshed but I want do it online without restarts.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/57b3c75b-e90b-4ef4-9a91-ceb09375c3bd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Can aggregation return the documents in each bucket?

2014-03-20 Thread Adrien Grand

Hi,

Unfortunately, this is not possible, aggregations cannot return whole
documents today.


On Wed, Mar 19, 2014 at 1:07 AM, Erich Lin  wrote:

> It seems aggregation will return the count of documents for each bucket.
>
> Can I also retrieve the documents within each bucket that matched that
> bucket criteria?
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/0e5c1001-e3bb-4345-849e-3b77816411d8%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adrien Grand

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j54QLzGup4xTfjoYBf21vKd31g2OBBsjREYs0NuPfg3Ag%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

How to delete queries from percolator?

2014-03-20 Thread Tomasz Romanczuk

I have indexed 1 queries in percolator. Next I want to update some of 
them and delete 9000 queries. I use bulk request, operation seems to finish 
with success (without fails). But after all deleted queries still are 
returned. Below code refreshing index:

BulkRequestBuilder bulkRequest = client.prepareBulk();
while (some condition) {
bulkRequest.add(client.prepareIndex("_percolator", INDEX_NAME, 
id).setSource(...));
}
while (some condition) {
bulkRequest.add(client.prepareDelete("_percolator", INDEX_NAME, id));
}
BulkResponse response = bulkRequest.setRefresh(true).execute().actionGet();

response.hasFailures() returns *false*. Is there any bug in elasticsearch 
or am I doing something wrong? Application restart helps, index is 
refreshed but I want do it online without restarts.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b7930318-82ff-46ec-a838-2cecad943fff%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Fwd: elasticsearch error of class not found

Can anybody help me it is  a bit urgent .



On Thursday, March 20, 2014 3:09:56 PM UTC+5:30, Anikessh Jain wrote:
>
> Hi All.
>
> I am a newbie to Elastic search and I am configuring Kibana with Logstash 
> and Redis  and Elasticsearch  in  Centos 32 Bit and when i am trying to 
> start the service of elastic search i am getting the below error
>
> WrapperSimpleApp Error: Unable
>
>  to locate the class org.elasticsearch.bootstrap.ElasticsearchF : 
> java.lang.ClassNotFoundException: org.elasticsearch.bootstrap.ElasticsearchF
>
>
> How to solve the above error ,i tried many steps but no use  but of no help, 
> have anybody face the same issue please help me.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c2e6f892-9d61-4ecf-9624-b180842bac11%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [hadoop] Getting elasticsearch-hadoop working with Shark

2014-03-20 Thread Nick Pentreath

Hi

I am struggling to get this working too. I'm just trying locally for now, 
running Shark 0.8.1, Hive 0.9.0 and ES 1.0.1 with ES-hadoop 1.3.0.M2.

I managed to get a basic example working with WRITING into an index. But 
I'm really after READING and index. 

I believe I have set everything up correctly, I've added the jar to Shark:
ADD JAR /path/to/es-hadoop.jar;

created a table:
CREATE EXTERNAL TABLE test_read (name string, price double)

STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'

TBLPROPERTIES('es.resource' = 'test_index/test_type/_search?q=*');


And then trying to 'SELECT * FROM test _read' gives me :

org.apache.spark.SparkException: Job aborted: Task 3.0:0 failed more than 0 
times; aborting job java.lang.ClassCastException: 
org.elasticsearch.hadoop.hive.EsHiveInputFormat$ESHiveSplit cannot be cast 
to org.elasticsearch.hadoop.hive.EsHiveInputFormat$ESHiveSplit

at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:827)

at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:825)

at 
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)

at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)

at 
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:825)

at 
org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:440)

at 
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$run(DAGScheduler.scala:502)

at 
org.apache.spark.scheduler.DAGScheduler$$anon$1.run(DAGScheduler.scala:157)

FAILED: Execution Error, return code -101 from shark.execution.SparkTask


In fact I get the same error thrown when trying to READ from the table that 
I successfully WROTE to...
On Saturday, 22 February 2014 12:31:21 UTC+2, Costin Leau wrote:
>
> Yeah, it might have been some sort of network configuration issue where 
> services where running on different machines and 
> localhost pointed to a different location. 
>
> Either way, I'm glad to hear things have are moving forward. 
>
> Cheers, 
>
> On 22/02/2014 1:06 AM, Max Lang wrote: 
> > I managed to get it working on ec2 without issue this time. I'd say the 
> biggest difference was that this time I set up a 
> > dedicated ES machine. Is it possible that, because I was using a cluster 
> with slaves, when I used "localhost" the slaves 
> > couldn't find the ES instance running on the master? Or do all the 
> requests go through the master? 
> > 
> > 
> > On Wednesday, February 19, 2014 2:35:40 PM UTC-8, Costin Leau wrote: 
> > 
> > Hi, 
> > 
> > Setting logging in Hive/Hadoop can be tricky since the log4j needs 
> to be picked up by the running JVM otherwise you 
> > won't see anything. 
> > Take a look at this link on how to tell Hive to use your logging 
> settings [1]. 
> > 
> > For the next release, we might introduce dedicated exceptions for 
> the simple fact that some libraries, like Hive, 
> > swallow the stack trace and it's unclear what the issue is which 
> makes the exception (IllegalStateException) ambiguous. 
> > 
> > Let me know how it goes and whether you will encounter any issues 
> with Shark. Or if you don't :) 
> > 
> > Thanks! 
> > 
> > [1] 
> https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-ErrorLogs
>  
> > <
> https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-ErrorLogs>
>  
>
> > 
> > On 20/02/2014 12:02 AM, Max Lang wrote: 
> > > Hey Costin, 
> > > 
> > > Thanks for the swift reply. I abandoned EC2 to take that out of 
> the equation and managed to get everything working 
> > > locally using the latest version of everything (though I realized 
> just now I'm still on hive 0.9). I'm guessing you're 
> > > right about some port connection issue because I definitely had ES 
> running on that machine. 
> > > 
> > > I changed hive-log4j.properties and added 
> > > | 
> > > #custom logging levels 
> > > #log4j.logger.xxx=DEBUG 
> > > log4j.logger.org.elasticsearch.hadoop.rest=TRACE 
> > >log4j.logger.org.elasticsearch.hadoop.mr <
> http://log4j.logger.org.elasticsearch.hadoop.mr>=TRACE 
> > > | 
> > > 
> > > But I didn't see any trace logging. Hopefully I can get it working 
> on EC2 without issue, but, for the future, is this 
> > > the correct way to set TRACE logging? 
> > > 
> > > Oh and, for reference, I tried running without ES up and I got the 
> following, exceptions: 
> > > 
> > > 2014-02-19 13:46:08,803 ERROR shark.SharkDriver 
> (Logging.scala:logError(64)) - FAILED: Hive Internal Error: 
> > > java.lang.IllegalStateException(Cannot discover Elasticsearch 
> version) 
> > > java.lang.IllegalStateException: Cannot discover Elasticsearch 
> version 
> > > at 
> org.elasticsearch.hadoop.hive.EsStorageHandler.init(EsStorageHandler.java:101)
>  
>
> > > at

Re: Running aggregations on two different nested objects

2014-03-20 Thread Adrien Grand

Hi,

The aggregation doesn't work because today, when you enter the context of a
nested field in an aggregation, it is not possible to escape it. I don't
think there is an easy way to modify your data model in order to work
around this issue, however this is an issue that we plan to fix in the
future (not in the upcoming 1.1 release however, rather in a few months).


On Wed, Mar 19, 2014 at 10:00 AM, Jean-Noël Rivasseau wrote:

> Hello,
>
> I just started using ElasticSearch 1.0.1. I am trying to find the ideal
> data model and query for my exact needs, which I will explain below (I
> changed just the terms of the data model corresponding to my real use case,
> in order to see if I was able to formulate it differently, which was
> useful).
>
> I am indexing documents corresponding to BookedStay. A BookedStay has a
> nested array (named places) containing map objects corresponding to visited
> places during the stay. An object has an id (place id) and a category
> corresponding to the time of day of the visited place. A BookedStay then
> has a second nested array, corresponding to the amenities used during the
> stay. The objects in the array have an id (of type string) and a count.
>
> So a BookedStay can be represented as : {"date": 03/03/2014,
> "placesVisited": [{"id": 3, "category": "MORNING"}, {"id": 5,
> "category":  "AFTERNOON"}, {"id": 7, "category": "EVENING"}], "amenities":
> [{"amenityId": "restaurant", "count": 3}, {"amenityId": "dvdPlayer",
> "count": 1}] }
>
> What I want to run is a query over a given room number, and find for
> all BookedStay that have this given place number in their places array, an
> aggregate over all amenities used, per time of day.
>
> This amounts to finding, for all documents that have a place id of (for
> instance) 5, the number of times the restaurant was used, or the dvd player
> in the lounge, broken down by time of day. The ultimate goal is to
> understand better how the visit of a place in a given time of day affects
> the services sold by the hotel.
>
> I am unable to achieve this query, as when I run a first nested aggregate
> over the category, I cannot nest the second one over the amenities as it is
> in the "parent" document. Is it possible to do that? In that case, how do I
> specify that the nested aggregation will take place over the parent object
> of the current aggregation?
>
> Here is a tentative query with the Java driver (obviously not working,
> because of the above problem):
>
> SearchRequestBuilder srb =
> elasticSearchService.getClient().prepareSearch("test_index").setSearchType(SearchType.COUNT).setTypes("test_stay").setQuery(QueryBuilders.nestedQuery("placesVisited",
> QueryBuilders.termQuery("id", 5)))
>
> .addAggregation(AggregationBuilders.nested("nestedPlaceVisited").path("placesVisited")
> .subAggregation(AggregationBuilders.filter("currentPlaceFilter").filter(FilterBuilders.termFilter("id",
> 5))
>
> .subAggregation(AggregationBuilders.terms("countPerTimeCategory").field("category")
> .subAggregation(AggregationBuilders.nested("nestedAmenities").path("amenities")
>  // HERE THIS subaggregation should run over the original document... and I
> dont know how to achieve that
>
> subAggregation(AggregationBuilders.terms("amenitiesUsed").field("amenities.amenityId"))
>
> Thanks for your help over this difficult problem! If it's not possible
> with "parent aggregations", how should I refactor my data model?
>
> Jean-Noel
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/e8a752fc-0a96-437d-b071-4009c0f39d33%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adrien Grand

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6JhVsi6tr2r54sFxnAPpip7gzhvmvX1hje1ueynPr50w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Cascading in elasticsearch

Not exactly but I think you can use 
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-delete-by-query.html#docs-delete-by-query
 to remove all children corresponding to a given parent and then remove the 
parent?

I suppose you are talking about parent / child feature right? 

-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr


Le 20 mars 2014 à 10:45:26, Komal Parekh (komaldpar...@gmail.com) a écrit:

Hello,


Like we have cascade on update and cascade on delete in SQL , do we have any 
such functionality in ElasticSearch?
--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4a72c347-d62d-4a4c-a230-242369241df7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/etPan.532ab9e5.1d9f6e5f.97ca%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

Re: GROUP BY query in ElasticSearch

2014-03-20 Thread Nguyen Manh Tien

Check terms aggregation, it allow return specified top field in a group,
can i return the whole doc there?


On Thu, Mar 20, 2014 at 4:36 PM, David Pilato  wrote:

> Terms aggregation should be what you are looking for.
>
> --
> *David Pilato* | *Technical Advocate* | *Elasticsearch.com*
> @dadoonet  | 
> @elasticsearchfr
>
>
> Le 20 mars 2014 à 10:24:02, Nguyen Manh Tien (tien.nguyenm...@gmail.com)
> a écrit:
>
> Thanks David,
>
> If you check my sample above, i don't want any aggregate info (min, max,
> count, ...) for each group, just want to get top N result from each group,
> is this possible?
>
>
> On Thu, Mar 20, 2014 at 1:53 PM, David Pilato  wrote:
>
>>  Have a look at aggregations.
>>
>> --
>> David ;-)
>> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>>
>>
>> Le 20 mars 2014 à 03:55, Nguyen Manh Tien  a
>> écrit :
>>
>>   Hello,
>>
>> In SQL i can use GROUP BY to limit number of rows in each group like
>> bellow.
>> Is there any way to do that in ElasticSearch?
>>
>> Thanks.
>>
>>1.
>>
>>*Find the first 10 groups of cars, with 2 hits in each group:*
>>
>>SELECT category, city, makemodel
>>FROM cars
>>GROUP BY category TOP 2
>>
>>*Results:*
>>
>>
>> 
>>| category  | city| makemodel 
>>  |
>>
>> 
>>| compact   | u.s.a./texas/houston| asian/acura/tl
>>  |
>>| compact   | u.s.a./florida/tampa| asian/acura/1.6el 
>>  |
>>
>> +---+-++
>>| exotic| u.s.a./florida/tampa| 
>> european/bentley/azure |
>>| exotic| u.s.a./california/san francisco | 
>> european/bentley/azure |
>>
>> 
>>
>>2.
>>
>>*Find the numbers of cars in different categories:*
>>
>>SELECT category, grouphitscount
>>FROM cars
>>GROUP BY category top 1
>>
>>*Results:*
>>
>>+---++
>>| category  | grouphitscount |
>>+---++
>>| compact   | 4169   |
>>| exotic| 1902   |
>>
>>
>>  --
>>  You received this message because you are subscribed to the Google
>> Groups "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>>
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/c1e2d776-e006-430c-82a0-389e3fc6731c%40googlegroups.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>  --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "elasticsearch" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/elasticsearch/8GZHBLh1vsc/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/FA25DC0F-39A9-4131-B6FF-F0EEC4E6C99C%40pilato.fr.
>>
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CANihJfWFQZb_rbUHdEG8X_jQDA%2BgJHhsz_m0Erjd4h-sZJmp7Q%40mail.gmail.com
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/8GZHBLh1vsc/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/etPan.532ab696.3fa62aca.97ca%40MacBook-Air-de-David.local

Re: ElasticSearch to SQL sync

Yes we have done some implementation on river but it does not give real 
time data. And also it is not as much mature. So for real time data we need 
to use some push mechanism which can help up to have realtime data. 

On Thursday, March 20, 2014 3:13:13 PM UTC+5:30, Jörg Prante wrote:
>
> Have you tried the JDBC river for poll mechanism?
>
> https://github.com/jprante/elasticsearch-river-jdbc/
>
> Jörg
>
>
> On Thu, Mar 20, 2014 at 9:50 AM, Komal Parekh 
> 
> > wrote:
>
>>
>> Thanks David for your prompt response. But we want some automatic Push or 
>> poll mechanism for this. 
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d1e6-c85b-46cc-8d7d-b41a763ae3f0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

fuzziness & score computation

2014-03-20 Thread Adrian Luna

Hi,

Sorry that I am relatively fresh to elasticsearch so please don't be too
harsh.

I feel like I'm not being able to understand the behaviour of any of the
fuzzy queries in ES.

*1) match with fuzziness enabled*

{
"query": {
"fuzzy_like_this_field": {
"field_name": {
"like_text": "car renting London",
"fuzziness": "0.5"
}
}
}
}

As I see it from my tests, this kind of query will give same score to
documents with field_name="car renting London" and "car ranting London" or
"car renting Londen" for example. That means, it will not give any
negatively score misspellings. I can imagine that first the possible
variants are computed and then the score is just computed with a
"representative score" which is the same for every variant that match the
requirements.

Am I right? If I am, is it any way to boost the exact match over the fuzzy
match?

Also I get results with more terms getting the same score, like "cheap car
renting London", "offers car renting London". That's something I cannot get
to understand. When I use the explain API, it seems that the resulting
score is a sum of the different matches with its internal weightings,
tf-idf, etc. but it seems to not be considering the terms outside the
query, while I would expect the exact match to score at least slightly
higher.

Am I missing something here? Is it just the expected result and I am just
being too demanding?

*2) fuzzy query*

That doesn't make what I want since it does not analyze the query (I think)
and so it will treat the query in an unexpected way for my purposes of
"free text" search

*3) fuzzy_like_this or fuzzy_like_this_field*

This other search takes rid of the first problem in point 1, since as I
read from the documentation, it seems to use some tricks to avoid favouring
rare terms (misspellings will be here) over more frequent terms, etc. but
it's still giving the same score to exact match and matches where other
terms are present.

Is there any way to get the expected behaviour?. By this I mean to be able
to execute almost free-text queries with some fuzziness to take rid of
possible misspellings in the query terms, but with an (at least for me)
more exhaustive score computation. If not, is there any other more complex
query or a function_score to get such a performance.

Thank you very much, any comment will be pretty much appreciated. Also, if
I am not right in my suppositions, any clarification will be very welcome.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/916f5408-ecfd-4676-8d48-db4467a9d839%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Cascading in elasticsearch

Hello,


Like we have cascade on update and cascade on delete in SQL , do we have 
any such functionality in ElasticSearch?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4a72c347-d62d-4a4c-a230-242369241df7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: searching pdf files by content with Mongodb-river

I think I'm starting to understand what you are trying to get…
You don't want original content but only extracted content, right?

I think that if you store content it should work.

Something like this (in mapping):

{
    "person" : {
        "properties" : {
            "file" : {
                "type" : "attachment",
                "fields" : {
                    "file" : {"index" : "no", "store" : "yes"}
                }
            }
        }
    }
}

And then when search, ask for field "file.file" instead of _source (default): 
curl -XGET 
'http://localhost:9200/index/person/_search?q=whatever&fields=file.file'

Should work I guess.

-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr


Le 20 mars 2014 à 10:12:01, sAs59 (mr.akmu...@gmail.com) a écrit:

It's still unclear, I've decoded my whole text and instead I'm getting this 
kind of text.
Where should I see my actual text?
I also tried using different charset, but still unclear.

<>
stream
xœXKoÛF ¾ ð Ð‘ â –\.Ék€8MÑ^
÷ $=Ð % –-—”ìôßwfvgw–‘" ( 8Ü÷7¯ofôáîúêýži£æfv·º¾Ò³9üÓ³¦R ¦êºP• 
Ý=]_Ígküóéúêkv—›ì!¿)³~–ßh“½Áx‡ã!o²-~,ñ ,VÙ ¿Æ0\À9“u°ï q~ að o,² 'ø xa èEw >Ö°Á 
¤ ßÿB06 !ØÓv„3c¼xµC< ,í‘b-aÜ¿âzOrù;_àã)o³þ —öñ.Z]ÑU#o^ ”ž6ý“ë2SN¾?avd8³ü¯ÙÝ¯×W 
Á î~4BUªÖ ¾Æ7J[EùWp‹“÷)×uÖí ^áÏŽ·Ð C2ö`„ÒÍâr l PúÍÝbÑoQ«ˆrèèìˆBãz% 
¶aqüATÑ@šEÃõ#/+Z/²Ïh^¯ú ±9 Ø›±wï/ù}ëÜH>Û] Ì²RÆze. Ú’@ì‚çz—au¼;q§®
U¦Wžz^WVÙ"ÝÛ‘ …P©£§ŽqÎ©qËn 3Rj ºÿ.•E¼Dj^}—×Ñ GŽÂª¢¸ ö• ’H ñ+Œ;Úp@ ¹ÉàªôÞ…žjÎ 
P[Õ6^ƒKFMaß;Ò ®¨Ý[Ïqœ §1¿Ox¼^L` 3 ”³$t8•Ü ã Iå ÞO^_¹oTÁ^’¡G3 c“éà}Á) 
+µàZrn|mÍ!A×¿åÆãatáÕ€ŒÅ#59C~÷ü™x Jë ò¬!lÛ¨’
Ñå7 p¼ «‘u d PÕæ¿ WíµÓ= 3 Õ&5 Œÿ†ñ!qå½—sÇ ÜF‰fÅ hùC:r Gÿ wìqÄs,B ’”Ì1 ä. 
‘U)âŒÜ´ñf<§õºU-+ ¡M1I^¥WÃ(g‚Ì8p¼Š’ ©' | G¡KÕ´)Ž-ç@¾·wª0ç’ œ= ~“¤?\Þ ?ÀñVÚ’.ë 
ÿô¤h8¢ G’£pÌT/p&PÊ+ $‰_ Äy[YLá•4:MxŸßsäv b³Ö;‰ i+”¡# †à@à?Nm" DN¿ ª 
]l™}„ñw6û(} «|‚ »E’ëéz ÔU_¤äWVÖÒg k½7v Â ˆ§þ¿ä`M K¥‘ R$>è¼Ùm#Ì^O2 
NÐÎÎ‘rØÃ*pé†jÕ:I“ ^ý §E Þ‰6å ][BI·cÌô Y–*E †[Héî¤‡AÔÝMùœÁœ· >8 – ¤åWºñ 5 
F•¬æ/¹‘•Fy jëì ‡ô>" h¥É>!È i J¿L÷>È¨Àù–kËÄÃŽ£-‹Bé*EK†™Ï…ÏáUGü-f x3TG©ï¶Z '~ cÒ 
U®Ý=w>iåö f8§úy¥šÒ óH ± Ñ‚- Zˆ À0pÖy‘ µLI IÊ Kú!÷þßqGõ V 
½X¦üþÛO\§,¬2uŠÿæÔÞR“áäÞ“÷–FÕ“½$`· í
zT™šÆBÞ‰% J²C*hB)Õû>.a +IöHûr9SUMÊÊãý–u‡¼Œ‰x'â'åÑ Ïøà“ÜCsÂk[O#,åà] :€ 
ðµt_[DþqÁì¶^fÚªEÝ'" 45ªÒéÞ“÷ÚV™É½lZW šì[î¥YzÑq~
½"É Ëˆ ÐCHóƒŒÆ6):` uu>@+Û ?:´Ÿ}9 ¤þ îCoPÎÁ ï„è ÅâÁ»Q·d ± î¹j£ ¡h|“`Ò 
[€þ"%;²ÇÁ…ÐÌ—“ž "Ð ˆ£ä " Ý*= ù•I Ñ/ø®Ø ÁÓÄSo! ! … ý\íÕ\ õ´-tÆÝú$òÂi®¨D¯B ˜.lÖ¯ 
_lüéçH âP eÇa9Š=±†Á M ¹‰æ¥ŽïÀ¿ŒˆjK ÅEY¼ - ¾ƒ:‡ÎbÌ£ àôžIÉŸYF7 ?®ÐÌ}îÊð}ô±ó< 
T]s#àlê\m—ûò1h²÷MrlLf¹Ö'ÊÖæØOBj‚åým1ÓzúÛeQ¶jÞ¦È¤ ÿ òÂˆ©
endstream
endobj
5 0 obj

Re: ElasticSearch to SQL sync

2014-03-20 Thread joergpra...@gmail.com

Have you tried the JDBC river for poll mechanism?

https://github.com/jprante/elasticsearch-river-jdbc/

Jörg


On Thu, Mar 20, 2014 at 9:50 AM, Komal Parekh wrote:

>
> Thanks David for your prompt response. But we want some automatic Push or
> poll mechanism for this.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHDJOf1ja28VWHf98ru0kzEJ9-ZydKzZyFY2Uk%2B4_dZHA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

elasticsearch error of class not found

Hi All.

I am a newbie to Elastic search and I am configuring Kibana with Logstash 
and Redis  and Elasticsearch  in  Centos 32 Bit and when i am trying to 
start the service of elastic search i am getting the below error

WrapperSimpleApp Error: Unable

 to locate the class org.elasticsearch.bootstrap.ElasticsearchF : 
java.lang.ClassNotFoundException: org.elasticsearch.bootstrap.ElasticsearchF


How to solve the above error ,i tried many steps but no use  but of no help, 
have anybody face the same issue please help me.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/bd7b47b5-46f1-4066-89a9-ba5c4929a25c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: GROUP BY query in ElasticSearch

Terms aggregation should be what you are looking for.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 20 mars 2014 à 10:24:02, Nguyen Manh Tien (tien.nguyenm...@gmail.com) a
écrit:

Thanks David,

If you check my sample above, i don't want any aggregate info (min, max, count,
...) for each group, just want to get top N result from each group, is this
possible?

On Thu, Mar 20, 2014 at 1:53 PM, David Pilato wrote:
Have a look at aggregations.

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 20 mars 2014 à 03:55, Nguyen Manh Tien a écrit :

Hello,

In SQL i can use GROUP BY to limit number of rows in each group like bellow.
Is there any way to do that in ElasticSearch?

Thanks.
Find the first 10 groups of cars, with 2 hits in each group:

SELECT category, city, makemodel
FROM cars
GROUP BY category TOP 2
Results:

| category | city| makemodel
|

Find the numbers of cars in different categories:

SELECT category, grouphitscount
FROM cars
GROUP BY category top 1
Results:

+---++
| category | grouphitscount |
+---++
| compact | 4169 |
| exotic| 1902 |

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c1e2d776-e006-430c-82a0-389e3fc6731c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to a topic in the Google
Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/8GZHBLh1vsc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/FA25DC0F-39A9-4131-B6FF-F0EEC4E6C99C%40pilato.fr.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANihJfWFQZb_rbUHdEG8X_jQDA%2BgJHhsz_m0Erjd4h-sZJmp7Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/etPan.532ab696.3fa62aca.97ca%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

Re: GROUP BY query in ElasticSearch

2014-03-20 Thread Nguyen Manh Tien

Thanks David,

If you check my sample above, i don't want any aggregate info (min, max,
count, ...) for each group, just want to get top N result from each group,
is this possible?


On Thu, Mar 20, 2014 at 1:53 PM, David Pilato  wrote:

> Have a look at aggregations.
>
> --
> David ;-)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
>
> Le 20 mars 2014 à 03:55, Nguyen Manh Tien  a
> écrit :
>
> Hello,
>
> In SQL i can use GROUP BY to limit number of rows in each group like
> bellow.
> Is there any way to do that in ElasticSearch?
>
> Thanks.
>
>1.
>
>*Find the first 10 groups of cars, with 2 hits in each group:*
>
>SELECT category, city, makemodel
>FROM cars
>GROUP BY category TOP 2
>
>*Results:*
>
>
> 
>| category  | city| makemodel  
> |
>
> 
>| compact   | u.s.a./texas/houston| asian/acura/tl 
> |
>| compact   | u.s.a./florida/tampa| asian/acura/1.6el  
> |
>
> +---+-++
>| exotic| u.s.a./florida/tampa| european/bentley/azure 
> |
>| exotic| u.s.a./california/san francisco | european/bentley/azure 
> |
>
> 
>
>2.
>
>*Find the numbers of cars in different categories:*
>
>SELECT category, grouphitscount
>FROM cars
>GROUP BY category top 1
>
>*Results:*
>
>+---++
>| category  | grouphitscount |
>+---++
>| compact   | 4169   |
>| exotic| 1902   |
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
>
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/c1e2d776-e006-430c-82a0-389e3fc6731c%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/8GZHBLh1vsc/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/FA25DC0F-39A9-4131-B6FF-F0EEC4E6C99C%40pilato.fr
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CANihJfWFQZb_rbUHdEG8X_jQDA%2BgJHhsz_m0Erjd4h-sZJmp7Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Sorting results with Elastic Search, Mongo Db and PHP

2014-03-20 Thread Marc Witteveen

title and ti have the same duplicate dataset, I used ti for testing to 
reduce the array elements to just two, but no luck.

I'll get some test data and post it in a bit.

/M


On Thursday, 20 March 2014 03:12:47 UTC+1, Zachary Tong wrote:
>
> Could you provide a small recreation of the problem in a gist?  Just a few 
> sample documents and two searches with different sorts that don't work?
>
> It could be your mapping that is incorrect, or a syntax issue with the 
> query/doc/mapping.  It may be a typo, but you are sorting on 
> "title.", but the document you posted has the inner object named 
> "ti", not "title".  
>
> But, assuming that is just a typo while making the post, a gist recreation 
> would be very helpful to debugging this.
>
> -Zach
>
>
>
> On Wednesday, March 19, 2014 2:43:48 PM UTC-5, Marc Witteveen wrote:
>>
>> Hi,
>>
>> I hope somebody can help me, though my question mentions PHP the sort 
>> problem happens when I use the console to fetch data.
>>
>> I am having a MongoDb database with data and use elastic search and PHP 
>> to search and display the search results.
>>
>> The document contains multilingual, see the example below:
>>
>> "data": {
>> "ti": {
>> "en": "Poland-Gdansk: Network Equipment",
>> "nl": "Polen-Gdansk: Netwerk Apparatuur",
>> "si": "Poljska-Gdansk: Oprema za omrežje"
>> }
>> }
>>
>>
>> *Note: the document has obvious more data but I don't believe it is 
>> relevent at the moment to make my question understandable. If you want me 
>> to post a complete document, please let me know and I will be happy to do 
>> so.*
>> I am getting the search result fine but when I do a sort om ti field for 
>> an ascending or descending sort the returned dataset is missed up.
>>
>> I am using the official elastic search PHP library available on 
>> https://github.com/elasticsearch/elasticsearch-php.
>>
>> The sort order I set as followed:
>>
>> $sitelanguage = "en";
>> $order = "asc";
>> $sortOrder = array(
>> "title." . $sitelanguage => array(
>> "order" => $order
>> )
>> );
>>
>> These array then is put to the code that build the search query:
>>
>> $searchParams = array(
>> 'body' => array(
>> 'query' => array(
>> 'bool' => array(
>> 'must' => array(
>>  array(
>>  'query_string' => array(
>>   'default_field' => '_all',
>>   'query' => 'Gdansk'
>>   )
>>  ), 
>>  )
>> )
>>  ),
>> 'sort' => $sortOrder
>>  )
>> );
>>
>> It looks to me that with ascending sort order the sort that is executed 
>> is done with the "en" key while descending is done with the "si" key 
>> instead of the value "Poland-Gdansk: Network Equipment".
>>
>> How could I solve is so that sorting is done correctly, so when 
>> $sitelanguage is set to "en" it sorts only on the value in this array and 
>> obvious if any other value is set in the variable the sort is done using 
>> that value.
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7f7410c2-2205-41ea-a633-21a7574502d7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: searching pdf files by content with Mongodb-river

2014-03-20 Thread sAs59

It's still unclear, I've decoded my whole text and instead I'm getting this
kind of text.
Where should I see my actual text?
I also tried using different charset, but still unclear.

<>
stream
xœXKoÛF ¾ ð Ð‘ â –\.Ék€8MÑ^
÷ $=Ð %
–-—”ìôßwfvgw–‘" ( 8Ü÷7¯ofôáîúêýži£æfv·º¾Ò³9üÓ³¦R ¦êºP• 
Ý=]_Ígküóéúêkv—›ì!¿)³~–ßh“½Áx‡ã!o²-~,ñ ,VÙ ¿Æ0\À9“u°ï  q~  að o,² 'ø xa  èEw 
>Ö°Á ¤ ßÿB06  !ØÓv„
3c¼xµC<  ,í‘b-aÜ¿âzOrù;_àã)o³þ —öñ.Z]ÑU#o^
”ž6ý“ë2SN¾?avd8³ü¯ÙÝ¯×W Á  î~4BUªÖ  ¾Æ7J[EùWp‹“÷)×uÖí ^áÏŽ·Ð C2ö`„ÒÍâr l 
PúÍÝbÑoQ«ˆrèèìˆBãz% ¶aqüATÑ@šEÃõ#/+Z/²Ïh^¯ú ±9 Ø›±wï/ù}ëÜH>Û]
Ì²RÆze.  Ú’@ì‚çz—au¼;q§®
U¦Wžz^WVÙ"ÝÛ‘ …P©£§ŽqÎ©qËn 3Rj ºÿ.•E¼Dj^
}—×Ñ GŽÂª¢¸ ö• ’H ñ+Œ;Úp@ ¹ÉàªôÞ…žjÎ P[Õ6^ƒKFMaß;Ò ®¨Ý[Ïqœ
§1¿Ox¼^L` 3 ”³$t8•Ü ã Iå ÞO^_¹oTÁ^’¡G3
c“éà}Á) +µàZrn|mÍ!A×¿åÆãatáÕ€ŒÅ#59C~÷ü™x Jë ò¬!lÛ¨’
Ñå7 p¼ «‘u d PÕæ¿ WíµÓ=  3 Õ&5 Œÿ†ñ!qå½—sÇ ÜF‰fÅ hùC:r  Gÿ  wìqÄs,B ’”Ì1 ä. 
‘U)âŒÜ´ñf<§õºU-+  ¡M1I^¥WÃ(g‚Ì8p¼Š’ ©' | G¡KÕ´)Ž-ç@¾·wª0ç’
  œ= ~“¤?\Þ ?ÀñVÚ’.ë ÿô¤h8¢ G’£pÌT/p&PÊ+ $‰_ Äy[YLá•4:MxŸßsäv b³Ö;‰ i+”¡#  
†à@à?Nm" DN¿
ª ]l™}„ñw6û(} «|‚ »E’ëéz ÔU_¤äWVÖÒg k½7v Â ˆ§þ¿ä`M K¥‘ R$
>è¼Ùm#Ì^O2 NÐÎÎ‘rØÃ*pé†jÕ:I“ ^ý  §E Þ‰6å ][BI·cÌô Y–*E
†[Héî¤‡AÔÝMùœÁœ· >8 – ¤åWºñ 5 F•¬æ/¹‘• F y  jëì ‡ô>" h¥É>!È  i J
¿L÷>È¨Àù–kËÄÃŽ£-‹Bé*EK†™Ï…ÏáUGü-f x3TG©ï¶Z   '~ cÒ U®Ý=w>iåö f8§úy¥šÒ
óH  ± Ñ‚- Zˆ À0pÖy‘ µLI  IÊ Kú!÷þßqGõ V  ½X¦üþÛO\§,¬2uŠÿæÔÞR“áäÞ“÷–FÕ“½$`· í
zT™šÆBÞ‰% J²C*hB)Õû>.a +IöHûr9SUMÊÊãý–u‡¼Œ‰x'â'åÑ Ïøà“ÜCsÂk[O#,åà]  :€ 
ðµt_[DþqÁì¶^fÚªEÝ'" 45ªÒéÞ“÷ÚV™É½lZW šì[î¥
YzÑq~
½"É Ëˆ ÐCHóƒŒÆ6):` uu>@+Û ?:´Ÿ}9 ¤þ
 îCoPÎÁ  ï„è ÅâÁ»Q·d  ± î¹j£ ¡h|“`Ò [€þ"%;
²ÇÁ…ÐÌ—“ž "Ð ˆ£ä " Ý*= ù•I Ñ/ø®Ø  ÁÓÄSo! 
! …   ý\íÕ\ õ´-tÆÝú$òÂi®¨D¯B ˜.lÖ¯   _lüéçH âP  eÇa9Š=±†Á
M ¹‰æ¥ŽïÀ¿ŒˆjK  ÅEY¼ - ¾ƒ:‡ÎbÌ£ àôžIÉŸYF7 ?®ÐÌ}îÊð}ô±ó< 
T]s#àlê\m—ûò1h²÷MrlLf¹Ö'ÊÖæØOBj‚åým1ÓzúÛeQ¶jÞ¦È¤ ÿ òÂˆ©
endstream
endobj
5 0 obj

Re: ElasticSearch to SQL sync


Thanks David for your prompt response. But we want some automatic Push or 
poll mechanism for this. 


On Thursday, March 20, 2014 1:04:28 PM UTC+5:30, David Pilato wrote:
>
> Best: when you push in your application to MySQL, push as well to 
> elasticsearch.
>
> --
> David ;-)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
>
> Le 20 mars 2014 à 08:29, Komal Parekh > 
> a écrit :
>
> Hello,
>
> We are running one application which has very large amount of data 
> updating very frequently. And we are using SQL for any command like 
> update/delete/add records. And for querying part we are using 
> ElasticSearch. So is there any way which syncs my data very frequently say 
> in 5 to 10 seconds. Is there any push mechanism available with 
> ElasticSearch which pushes data from SQL to ES? And more what is the best 
> option for synching? Please reply urgently with the best option :). 
>
>
> Thanks
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/34e2f8a0-9592-4cb6-8d86-8fbb03242488%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4f632383-b489-4893-b27d-0b93b174df48%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: elasticsearch java interaction

2014-03-20 Thread Georgi Ivanov

On Linux the file is  /etc/hosts
On Windows c:\windows\system32\drivers\etc\hosts

Open the file in text editor

Add following lines:
192.168.1.100 host1
192.168.1.101 host2

Make sure that 192.168.1.100/101 is the right IP address of the host1/host2



2014-03-20 8:35 GMT+01:00 Venu Krishna :

> Hi Georgi Ivanov,
>   yes,i am able to understand the Exception i.e.
> UnresolvedAddressException,but you are telling that to make sure host1 and
> host2 are resolved by adding entries to /etc/hosts to wherever the file in
> on Windows,for this can you give me the steps how to approach this.Sorry i
> am new to this and am learning i am unable to get the proper example.Thanks
> in advance for the help.
>
>
> On Thursday, March 20, 2014 2:36:10 AM UTC+5:30, Georgi Ivanov wrote:
>>
>> Well
>> I think UnresolvedAddressException obviously means that your Java client
>> can not resolve host1 and host2
>>
>> make sure host1 and host2 are resolvable  by adding entries to /etc/hosts
>> ot wherever the file in on Windows
>>
>>
>>
>> On Friday, March 14, 2014 1:47:02 PM UTC+1, Venu Krishna wrote:
>>>
>>> Hi,
>>>I am Y.Venu,i am totally new to this elasticsearch,now i am
>>> trying to communicate java elastisearch,i have gone through the
>>> elasticsearch java api's
>>>
>>> 1st i came across maven repositry.
>>> i have created pom.xml in my eclipse and in the dependency tag i
>>> have just placed the code that i found in maven repositry
>>>
>>>  i.e.
>>>
>>>
>>> 
>>> org.elasticsearch
>>> elasticsearch
>>> ${es.version}
>>> 
>>>
>>> After that i have created one class with the main method and i copied and 
>>> placed the code that i found in the client api of elasticsearch i.e.
>>>  TransportClient.
>>>
>>> main()
>>> {
>>> Client client = new TransportClient()
>>> .addTransportAddress(new 
>>> InetSocketTransportAddress("host1", 9200))
>>> .addTransportAddress(new 
>>> InetSocketTransportAddress("host2", 9200));
>>>
>>> // on shutdown
>>>
>>> client.close();
>>> 
>>> Settings settings = ImmutableSettings.settingsBuilder()
>>> .put("client.transport.sniff", true).build();
>>> TransportClient client1 = new TransportClient(settings);
>>>
>>> }
>>>
>>> After running this app javapplication,i am getting the errors like this
>>>
>>>
>>>
>>> In Main Method
>>> Mar 14, 2014 6:05:24 PM org.elasticsearch.node
>>> INFO: [Mister Machine] {elasticsearch/0.16.1}[11016]: initializing ...
>>> Mar 14, 2014 6:05:24 PM org.elasticsearch.plugins
>>> INFO: [Mister Machine] loaded []
>>> org.elasticsearch.common.inject.internal.ComputationException: 
>>> org.elasticsearch.common.inject.internal.ComputationException: 
>>> java.lang.NoClassDefFoundError: Lorg/apache/lucene/store/Lock;
>>> at 
>>> org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:553)
>>> at 
>>> org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:419)
>>> at 
>>> org.elasticsearch.common.inject.internal.CustomConcurrentHashMap$ComputingImpl.get(CustomConcurrentHashMap.java:2041)
>>> at 
>>> org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:46)
>>> at 
>>> org.elasticsearch.common.inject.ConstructorInjectorStore.get(ConstructorInjectorStore.java:52)
>>> at 
>>> org.elasticsearch.common.inject.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:57)
>>> at 
>>> org.elasticsearch.common.inject.InjectorImpl.initializeBinding(InjectorImpl.java:377)
>>> at 
>>> org.elasticsearch.common.inject.BindingProcessor$1$1.run(BindingProcessor.java:169)
>>> at 
>>> org.elasticsearch.common.inject.BindingProcessor.initializeBindings(BindingProcessor.java:224)
>>> at 
>>> org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(InjectorBuilder.java:120)
>>> at 
>>> org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:105)
>>> at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:92)
>>> at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:69)
>>> at 
>>> org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:58)
>>> at 
>>> org.elasticsearch.node.internal.InternalNode.(InternalNode.java:146)
>>> at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
>>> at org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:166)
>>> at ES_Client.main(ES_Client.java:64)
>>> Caused by: org.elasticsearch.common.inject.internal.ComputationException: 
>>> java.lang.NoClassDefFoundError: Lorg/apache/lucene/store/Lock;
>>> at 
>>> org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:553)
>>> at 
>>> org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:419)
>>> at 
>>> org.elasticsearch.common.inject.internal.CustomC

Re: ElasticSearch to SQL sync

Thanks David for your prompt response. Actually we are using MSSQL not 
MySQL. So this solution will not work for us.

On Thursday, March 20, 2014 1:04:28 PM UTC+5:30, David Pilato wrote:
>
> Best: when you push in your application to MySQL, push as well to 
> elasticsearch.
>
> --
> David ;-)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
>
> Le 20 mars 2014 à 08:29, Komal Parekh > 
> a écrit :
>
> Hello,
>
> We are running one application which has very large amount of data 
> updating very frequently. And we are using SQL for any command like 
> update/delete/add records. And for querying part we are using 
> ElasticSearch. So is there any way which syncs my data very frequently say 
> in 5 to 10 seconds. Is there any push mechanism available with 
> ElasticSearch which pushes data from SQL to ES? And more what is the best 
> option for synching? Please reply urgently with the best option :). 
>
>
> Thanks
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/34e2f8a0-9592-4cb6-8d86-8fbb03242488%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/80291df4-8681-4079-befc-e2db8160bf20%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Error in Elasticsearch NullPointerException at org.elasticsearch.plugins.PluginsService.loadSitePlugins

Hi Preeti Jain,
i am venu,i am new to this technology i.e. 
elasticsearch,i am trying to do communicate between the java and 
elasticsearch like ccommuniccation between java and oracle,i am surfing for 
the examples but unable to get the required way.Can help to achieve the 
Java ElasticSearch comunication(using Node or TransportClient).

On Wednesday, March 19, 2014 2:14:55 PM UTC+5:30, Preeti Jain wrote:
>
> Hi ,
>
> We suddenly started getting this error in Elasticsearch log
>
> [2014-03-19 09:40:57,419][DEBUG][action.admin.cluster.node.info] [North] 
> failed to execute on node [B5a2wTMvQpGHOpO5oIjnug]
> java.lang.NullPointerException
> at 
> org.elasticsearch.plugins.PluginsService.loadSitePlugins(PluginsService.java:441)
> at org.elasticsearch.plugins.PluginsService.info
> (PluginsService.java:308)
> at org.elasticsearch.node.service.NodeService.info
> (NodeService.java:122)
> at 
> org.elasticsearch.action.admin.cluster.node.info.TransportNodesInfoAction.nodeOperation(TransportNodesInfoAction.java:100)
> at 
> org.elasticsearch.action.admin.cluster.node.info.TransportNodesInfoAction.nodeOperation(TransportNodesInfoAction.java:43)
> at 
> org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$2.run(TransportNodesOperationAction.java:146)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
>
> We are unable to perform any action on elasticsearch.
>
> Any idea what could be the issue?
>
> Regards,
> Preeti
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7db6f15b-14b3-46c3-ae62-96b8253b903f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Mapper Attachments Plugin and SMILE

2014-03-20 Thread Paweł Młynarczyk

Hello

I've got a problem working with Mapper Attachments Plugin.

This code


val json = XContentFactory.jsonBuilder().startObject()
.field("file").startObject()
  .field("content").value(Base64.encodeBytes(bytes))
.endObject()
  .endObject()

client.prepareIndex(indexName, typeName)
  .setSource(json).execute().actionGet()


works perfectly well, but when I change jsonBuilder to smileBuilder, ES 
throws

org.elasticsearch.index.mapper.MapperParsingException: failed to parse
at 
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:540)
at 
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:462)
at 
org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:371)
at 
org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:215)
at 
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556)
at 
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.common.jackson.core.JsonParseException: 
Current token (VALUE_STRING) not VALUE_EMBEDDED_OBJECT, can not access as 
binary
 at [Source: [B@3cb66faa; line: -1, column: 98325]
at 
org.elasticsearch.common.jackson.core.JsonParser._constructError(JsonParser.java:1524)
at 
org.elasticsearch.common.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:557)
at 
org.elasticsearch.common.jackson.dataformat.smile.SmileParser.getBinaryValue(SmileParser.java:1212)
at 
org.elasticsearch.common.jackson.core.JsonParser.getBinaryValue(JsonParser.java:1131)
at 
org.elasticsearch.common.xcontent.json.JsonXContentParser.binaryValue(JsonXContentParser.java:183)
at 
org.elasticsearch.index.mapper.attachment.AttachmentMapper.parse(AttachmentMapper.java:321)
at 
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:517)
at 
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:459)
at 
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:515)
... 8 more

and when I try not to encode the file

val json = XContentFactory.smileBuilder().startObject()
.field("file").value(bytes)
  .endObject()

or

val json = XContentFactory.smileBuilder().startObject()
.field("file").startObject()
  .field("content").value(bytes)
.endObject()
  .endObject()

ES throws 

org.elasticsearch.index.mapper.MapperParsingException: No content is 
provided.
at 
org.elasticsearch.index.mapper.attachment.AttachmentMapper.parse(AttachmentMapper.java:337)
at 
org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:616)
at 
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:469)
at 
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:515)
at 
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:462)
at 
org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:371)
at 
org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:215)
at 
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556)
at 
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)




I am using SMILE so I am curious if there is some workaround or am I doing 
something wrong?

Thanks in advance

Paweł Młynarczyk

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/59a46c68-8703-4254-9869-1ac6e1bac883%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Does ES support to use the analyzer which decided by _analyzer field in search?

2014-03-20 Thread Ivan Ji

Hi all,

Assume my schema is 

{
> "settings": {
> "index": {
> "analysis": {
> "analyzer": {
> "ik_analyzer": {
> "tokenizer": "ik",
> "filter": ["engram"]
> }
> },
> "filter": {
> "engram": {
> "type": "edgeNGram",
> "min_gram": 2,
> "max_gram": 10
> }
> }
> }
> }
> },
> "mappings": {
> "main": {
> "_analyzer": {
> "path": "analyzer_name"
> },
> "properties": {
> "id": { "type": "string", "index": "analyzed"},
> "id2": { "type": "string", "index": "analyzed", "analyzer": "ik_analyzer"},
> "analyzer_name": { "type": "string", "index": "not_analyzed"}
>
> }
> }
> }
> }



I use the "_analyzer" to use different analyzer which is decided by the 
"analyzer_name" field between documents. 

When I use the "match query" during search for the "id" field, whose 
analyzer is decided by the field of "analyzer_name", what's the search 
analyzer to be used in this situation without explicitly specifying the 
search analyzer? 
And how to use the analyzer decided by the "analyzer_name" field to be the 
search analyzer?

We know if I queried for the "id2" field, the search analyzer is decided by 
the analyzer it used, that is "ik_analyzer". 
But does ES support the functionality, using the same analyzer to search 
with the analyzer at indexing, after setting the "_analyzer" ?

Ideas?

cheers,

Ivan



-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b20e70e1-33e1-43d3-888d-17119d49121f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Kibana Histogram unexpected line curve with cumulative value

I don't find the option a zero-fill checkbox  in my kibana version.

I've got logstash v1.3.3

I'll try to get the last version of kibana.

On Thursday, March 20, 2014 8:22:35 AM UTC+1, Isaac Hazan wrote:
>
> Yes, the problem is because when the histogram encounters buckets with no 
> data it assumes zero values instead of joining the points between the 2 
> interval.
>
> I solved it by using a different kibana version, I effectively found 2 
> patches one with a zero-fill checkbox and another one with an option box. I 
> found just one of them:
>
> https://github.com/elasticsearch/kibana/issues/741
>
>  
>
> Both versions solve the problem.
>
>  
>
> That’s the way I solved it.
>
>  
>
> Hope that helps
>
>  
>
> /Itzhak
>
>  
>
> *From:* elasti...@googlegroups.com  [mailto:
> elasti...@googlegroups.com ] *On Behalf Of *Xwilly Azel
> *Sent:* Thursday, March 20, 2014 9:12 AM
> *To:* elasti...@googlegroups.com 
> *Subject:* Re: Kibana Histogram unexpected line curve with cumulative 
> value
>
>  
>
> I've got the same kind of issue.
>
>  
>
> Did you find a solution ?
>
> On Monday, March 10, 2014 3:58:21 PM UTC+1, Isaac Hazan wrote:
>
> Kibana cannot do a the Histogram of the cumulative value of a field as 
> describe at:https://github.com/elasticsearch/kibana/issues/740
>
> To overcome that I created a separate index where I calculate myself the 
> total and saved it to Elasticsearch.
>
> The mapping looks as follows:
>
> curl -XPOST localhost:9200/first_install -d '{
>
> "settings" : {
>
> "number_of_shards" : 5
>
> },
>
> "mappings" : {
>
> "fi" : {
>
> "properties" : {
>
> "evtTime" : { "type" : "date", "index": "not_analyzed", 
> "format": "dd/MMM/:HH:mm:ss" },
>
> "cumulativeValue" : { "type" : "integer", "index": 
> "not_analyzed" }
>
> }
>
> }
>
> }
>
> }'
>
> The values are saved properly but unexpectedly Kibana does not draw the 
> line i would expect, instead it joins between point that do not exist.
>
> Following is the Kibana sreenshot:
>
> The line curve should always be increasing since my data set is always 
> increasing, that i can prove by the following events as seen by kibana 
> itself:
>
> Could it be related to the data formatting I did?
>
> Thx in advance.
>
> -- 
> You received this message because you are subscribed to a topic in the 
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/elasticsearch/mp2hUkR8U6w/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to 
> elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/9fe1f350-c5fb-47f2-8925-2d52b7376d94%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1fc19ee2-e1aa-433c-9239-96415c93ceff%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: elasticsearch java interaction

Hi Georgi Ivanov,
  yes,i am able to understand the Exception i.e. 
UnresolvedAddressException,but you are telling that to make sure host1 and 
host2 are resolved by adding entries to /etc/hosts to wherever the file in 
on Windows,for this can you give me the steps how to approach this.Sorry i 
am new to this and am learning i am unable to get the proper example.Thanks 
in advance for the help.

On Thursday, March 20, 2014 2:36:10 AM UTC+5:30, Georgi Ivanov wrote:
>
> Well
> I think UnresolvedAddressException obviously means that your Java client 
> can not resolve host1 and host2
>
> make sure host1 and host2 are resolvable  by adding entries to /etc/hosts 
> ot wherever the file in on Windows
>
>
>
> On Friday, March 14, 2014 1:47:02 PM UTC+1, Venu Krishna wrote:
>>
>> Hi,
>>I am Y.Venu,i am totally new to this elasticsearch,now i am trying 
>> to communicate java elastisearch,i have gone through the elasticsearch java 
>> api's 
>>
>> 1st i came across maven repositry.
>> i have created pom.xml in my eclipse and in the dependency tag i have 
>> just placed the code that i found in maven repositry 
>>
>>  i.e.  
>>  
>>
>> 
>> org.elasticsearch
>> elasticsearch
>> ${es.version}
>> 
>>
>> After that i have created one class with the main method and i copied and 
>> placed the code that i found in the client api of elasticsearch i.e.
>>  TransportClient.
>>
>> main()
>> {
>> Client client = new TransportClient()
>>  .addTransportAddress(new 
>> InetSocketTransportAddress("host1", 9200))
>>  .addTransportAddress(new 
>> InetSocketTransportAddress("host2", 9200));
>>
>>  // on shutdown
>>
>>  client.close();
>>  
>>  Settings settings = ImmutableSettings.settingsBuilder()
>>  .put("client.transport.sniff", true).build();
>>  TransportClient client1 = new TransportClient(settings);
>>
>> }
>>
>> After running this app javapplication,i am getting the errors like this
>>
>>
>>
>> In Main Method
>> Mar 14, 2014 6:05:24 PM org.elasticsearch.node
>> INFO: [Mister Machine] {elasticsearch/0.16.1}[11016]: initializing ...
>> Mar 14, 2014 6:05:24 PM org.elasticsearch.plugins
>> INFO: [Mister Machine] loaded []
>> org.elasticsearch.common.inject.internal.ComputationException: 
>> org.elasticsearch.common.inject.internal.ComputationException: 
>> java.lang.NoClassDefFoundError: Lorg/apache/lucene/store/Lock;
>>  at 
>> org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:553)
>>  at 
>> org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:419)
>>  at 
>> org.elasticsearch.common.inject.internal.CustomConcurrentHashMap$ComputingImpl.get(CustomConcurrentHashMap.java:2041)
>>  at 
>> org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:46)
>>  at 
>> org.elasticsearch.common.inject.ConstructorInjectorStore.get(ConstructorInjectorStore.java:52)
>>  at 
>> org.elasticsearch.common.inject.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:57)
>>  at 
>> org.elasticsearch.common.inject.InjectorImpl.initializeBinding(InjectorImpl.java:377)
>>  at 
>> org.elasticsearch.common.inject.BindingProcessor$1$1.run(BindingProcessor.java:169)
>>  at 
>> org.elasticsearch.common.inject.BindingProcessor.initializeBindings(BindingProcessor.java:224)
>>  at 
>> org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(InjectorBuilder.java:120)
>>  at 
>> org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:105)
>>  at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:92)
>>  at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:69)
>>  at 
>> org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:58)
>>  at 
>> org.elasticsearch.node.internal.InternalNode.(InternalNode.java:146)
>>  at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
>>  at org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:166)
>>  at ES_Client.main(ES_Client.java:64)
>> Caused by: org.elasticsearch.common.inject.internal.ComputationException: 
>> java.lang.NoClassDefFoundError: Lorg/apache/lucene/store/Lock;
>>  at 
>> org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:553)
>>  at 
>> org.elasticsearch.common.inject.internal.MapMaker$StrategyImpl.compute(MapMaker.java:419)
>>  at 
>> org.elasticsearch.common.inject.internal.CustomConcurrentHashMap$ComputingImpl.get(CustomConcurrentHashMap.java:2041)
>>  at 
>> org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:46)
>>  at 
>> org.elasticsearch.common.inject.MembersInjectorStore.get(MembersInjectorStore.java:66)
>>  at 
>> org.elasticsearch.common.inject.ConstructorInjectorStore.crea

Re: ElasticSearch to SQL sync

Best: when you push in your application to MySQL, push as well to elasticsearch.

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 20 mars 2014 à 08:29, Komal Parekh a écrit :

Hello,

We are running one application which has very large amount of data updating
very frequently. And we are using SQL for any command like update/delete/add
records. And for querying part we are using ElasticSearch. So is there any way
which syncs my data very frequently say in 5 to 10 seconds. Is there any push
mechanism available with ElasticSearch which pushes data from SQL to ES? And
more what is the best option for synching? Please reply urgently with the best
option :).

Thanks
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/34e2f8a0-9592-4cb6-8d86-8fbb03242488%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/E544B701-5E84-4092-A50E-D3A28CEC66B4%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

ElasticSearch to SQL sync