Re: Need help with Java API

[Alain Désilets]

On Tuesday, 22 July 2014 14:45:28 UTC-4, Jörg Prante wrote:
>
> Noo bulk request API is fantistic and a must!
>
> Just take a bit care to not pass nulls or empty requests to it.
>

Thx for the encouragement. I perservered a bit more and then realized that 
I had misunderstood how to create an IndexRequest. I fixed the problem and 
it's now working.

I have updated the gist demo code accordingly.

Alain

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f40afe21-42a9-40af-a6a0-0779eb6a9d41%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Need help with Java API

[joergpra...@gmail.com]

Noo bulk request API is fantistic and a must!

Just take a bit care to not pass nulls or empty requests to it.

Jörg


On Tue, Jul 22, 2014 at 8:26 PM, Alain Désilets 
wrote:

>
>
> On Tuesday, 22 July 2014 14:21:29 UTC-4, Jörg Prante wrote:
>>
>> To Issue 5: unfortunately, there are some hidden NPEs in the bulk request
>> API.
>>
>
> Ah, OK.
>
> I guess I'll stay away from the bulk request API then.
>
> Alain
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/f6a4d5df-52fd-4ad7-899f-964d7b7d451a%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGwsTwOBEaR2ih3fdd%2B6TmVUc7LoVMYrAFpEjOtK-dWkg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Need help with Java API

[Alain Désilets]

On Tuesday, 22 July 2014 14:21:29 UTC-4, Jörg Prante wrote:
>
> To Issue 5: unfortunately, there are some hidden NPEs in the bulk request 
> API.
>

Ah, OK. 

I guess I'll stay away from the bulk request API then.

Alain

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f6a4d5df-52fd-4ad7-899f-964d7b7d451a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Need help with Java API

[joergpra...@gmail.com]

To Issue 5: unfortunately, there are some hidden NPEs in the bulk request
API.

Check this

https://github.com/jprante/elasticsearch-support/blob/master/src/test/java/org/xbib/elasticsearch/support/various/NPETest.java

if you find a pattern that matches your case.

Jörg



On Tue, Jul 22, 2014 at 8:00 PM, Alain Désilets 
wrote:

> Thx Jörg,
>
> Your comments helped me get quite a bit further. Here is an updated
> version of my code:
>
>https://gist.github.com/alaindesilets/aec9492890c37075fa4e
>
>
> On Monday, 21 July 2014 13:13:58 UTC-4, Jörg Prante wrote:
>>
>> To issue 1: you create a single node cluster without index, and a client
>> of it.
>>
>
> Duh! Don't know how I could have missed that. So now, I added a
> method createIndex() which creates the index if it doesn't exist, and sets
> number_of_replicas to 2. It doesn't define mappings since the ES doc says
> that default mappings will be automatically generated if none are specified.
>
> Note however that even with that change, I still couldn't see the new
> index in Marvel/Sense, when I instantiated the client with
> method makeClientFromEmbeddedNode(). But if I instantiate the client
> through a TransportClient, i.e. by invoking
> method makeClientFromTransportClient(), then Marvel/Sense sees the new
> index.
>
> Using a TransportClient instead of a client obtained from an embedded node
> also greatly accelerated the client creation. Instead of taking > 8 secs to
> create the client, it takes < 1 sec. Not sure why.
>
>
>>
>> To issue 2: you see the UnavailableShardsException caused by a timeout
>> while indexing to a replica shard. This means, you may have set up a single
>> node cluster, but with replica level 1 (default) which needs 2 nodes for
>> indexing. Maybe there was once another node joining the cluster and ES
>> wants it back abut it never came (after 60 secs). Then ES returns the
>> timeout error. Maybe replica level 0 helps. You should also check the
>> cluster health. A color of green shows everything works, yellow means there
>> are too few nodes to satisfy the replica condition, and read means the
>> cluster is not in a consistent/usable state.
>>
>
> I tried setting number_replica=0 and number_replica=1 in createIndex(),
> but I still got the org.elasticsearch.action.UnavailableShardsException
> error if I instantiate the client with makeClientFromEmbeddedNode(). But I
> don't get the error if I instantiate the client with
> makeClientFromTransportClient(), independantly of the number of replicas I
> specify.
>
>
>> To issue 3: not sure what clusterName() means. I would use settings and
>> add a field "cluster.name". Maybe it is equivalent. You must ensure you
>> use the same "cluster.name" setting throughout all nodes and clients.
>> You also can not reuse data from clusters that had other names (look into
>> the "data" folder)
>>
>
> I have now moved that code to a method called
> makeClientFromNamedClusterNode(). I haven't been able to make that work at
> all.
>
> To issue 4: ES takes ~5 secs for discovery, the zen modules pings and
>> waits for responding master nodes by default. If you just test locally on
>> your developer machine, you should disable zen. Most easily by disabling
>> networking at all, by using NodeBuilder.nodeBuilder().local(true)...
>>
>
> Not sure I understand what that means, but in any case, using a
> TransportClient seems to address that issue.
>
> So, all in all, I feel I am in pretty good shape now. Thanks for the help.
>
> There is one new Issue I an now encountering, when I try to do a bulk
> indexing, namely:
>
> Issue 5: If I uncomment the call to index100NewBeersInOneBatch(), I get
> the following:
>
> == Indexing 100 new beer objects as in one batch... (Elapsed so far: 1.0
> seconds)
> Adding beer no 0 to the bulk request. (Elapsed so far: 1.0 seconds)
> Exception in thread "main" java.lang.NullPointerException
> at org.elasticsearch.action.bulk.BulkRequest.internalAdd(
> BulkRequest.java:129)
> at org.elasticsearch.action.bulk.BulkRequest.add(BulkRequest.java:
> 118)
> at org.elasticsearch.action.bulk.BulkRequestBuilder.add(
> BulkRequestBuilder.java:52)
> at ca.nrc.ElasticSearch.ElasticSearchDemo.
> index100NewBeersInOneBatch(ElasticSearchDemo.java:158)
> at ca.nrc.ElasticSearch.ElasticSearchDemo.main(ElasticSearchDemo.
> java:72)
>
>
> Any thoughts on what might be going on there?
>
> Thx.
>
> Alain
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/c3f6c329-4dc4-4714-8e6b-bae1b7fa1f51%40googlegroups.com
> 
> .
>
> For more options, v

Re: Need help with Java API

[Alain Désilets]

Thx Jörg,

Your comments helped me get quite a bit further. Here is an updated version 
of my code:

   https://gist.github.com/alaindesilets/aec9492890c37075fa4e

On Monday, 21 July 2014 13:13:58 UTC-4, Jörg Prante wrote:
>
> To issue 1: you create a single node cluster without index, and a client 
> of it. 
>

Duh! Don't know how I could have missed that. So now, I added a 
method createIndex() which creates the index if it doesn't exist, and sets 
number_of_replicas to 2. It doesn't define mappings since the ES doc says 
that default mappings will be automatically generated if none are specified.

Note however that even with that change, I still couldn't see the new index 
in Marvel/Sense, when I instantiated the client with 
method makeClientFromEmbeddedNode(). But if I instantiate the client 
through a TransportClient, i.e. by invoking 
method makeClientFromTransportClient(), then Marvel/Sense sees the new 
index.

Using a TransportClient instead of a client obtained from an embedded node 
also greatly accelerated the client creation. Instead of taking > 8 secs to 
create the client, it takes < 1 sec. Not sure why.
 

>
> To issue 2: you see the UnavailableShardsException caused by a timeout 
> while indexing to a replica shard. This means, you may have set up a single 
> node cluster, but with replica level 1 (default) which needs 2 nodes for 
> indexing. Maybe there was once another node joining the cluster and ES 
> wants it back abut it never came (after 60 secs). Then ES returns the 
> timeout error. Maybe replica level 0 helps. You should also check the 
> cluster health. A color of green shows everything works, yellow means there 
> are too few nodes to satisfy the replica condition, and read means the 
> cluster is not in a consistent/usable state.
>

I tried setting number_replica=0 and number_replica=1 in createIndex(), but 
I still got the org.elasticsearch.action.UnavailableShardsException error 
if I instantiate the client with makeClientFromEmbeddedNode(). But I don't 
get the error if I instantiate the client with 
makeClientFromTransportClient(), independantly of the number of replicas I 
specify.
 

> To issue 3: not sure what clusterName() means. I would use settings and 
> add a field "cluster.name". Maybe it is equivalent. You must ensure you 
> use the same "cluster.name" setting throughout all nodes and clients. You 
> also can not reuse data from clusters that had other names (look into the 
> "data" folder)
>

I have now moved that code to a method called 
makeClientFromNamedClusterNode(). I haven't been able to make that work at 
all. 

To issue 4: ES takes ~5 secs for discovery, the zen modules pings and waits 
> for responding master nodes by default. If you just test locally on your 
> developer machine, you should disable zen. Most easily by disabling 
> networking at all, by using NodeBuilder.nodeBuilder().local(true)...
>

Not sure I understand what that means, but in any case, using a 
TransportClient seems to address that issue.

So, all in all, I feel I am in pretty good shape now. Thanks for the help.

There is one new Issue I an now encountering, when I try to do a bulk 
indexing, namely:

Issue 5: If I uncomment the call to index100NewBeersInOneBatch(), I get the 
following:

== Indexing 100 new beer objects as in one batch... (Elapsed so far: 1.0 
seconds)
Adding beer no 0 to the bulk request. (Elapsed so far: 1.0 seconds)
Exception in thread "main" java.lang.NullPointerException
at org.elasticsearch.action.bulk.BulkRequest.internalAdd(BulkRequest
.java:129)
at org.elasticsearch.action.bulk.BulkRequest.add(BulkRequest.java:
118)
at org.elasticsearch.action.bulk.BulkRequestBuilder.add(
BulkRequestBuilder.java:52)
at ca.nrc.ElasticSearch.ElasticSearchDemo.index100NewBeersInOneBatch
(ElasticSearchDemo.java:158)
at ca.nrc.ElasticSearch.ElasticSearchDemo.main(ElasticSearchDemo.
java:72)


Any thoughts on what might be going on there?

Thx.

Alain

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c3f6c329-4dc4-4714-8e6b-bae1b7fa1f51%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Need help with Java API

[joergpra...@gmail.com]

To issue 1: you create a single node cluster without index, and a client of
it.

To issue 2: you see the UnavailableShardsException caused by a timeout
while indexing to a replica shard. This means, you may have set up a single
node cluster, but with replica level 1 (default) which needs 2 nodes for
indexing. Maybe there was once another node joining the cluster and ES
wants it back abut it never came (after 60 secs). Then ES returns the
timeout error. Maybe replica level 0 helps. You should also check the
cluster health. A color of green shows everything works, yellow means there
are too few nodes to satisfy the replica condition, and read means the
cluster is not in a consistent/usable state.

To issue 3: not sure what clusterName() means. I would use settings and add
a field "cluster.name". Maybe it is equivalent. You must ensure you use the
same "cluster.name" setting throughout all nodes and clients. You also can
not reuse data from clusters that had other names (look into the "data"
folder)

To issue 4: ES takes ~5 secs for discovery, the zen modules pings and waits
for responding master nodes by default. If you just test locally on your
developer machine, you should disable zen. Most easily by disabling
networking at all, by using NodeBuilder.nodeBuilder().local(true)...

Jörg




On Mon, Jul 21, 2014 at 6:53 PM, Alain Désilets 
wrote:

> I am trying to get started with the Java API, using the excellent tutorial
> found here:
>
>http://www.slideshare.net/dadoonet/hands-on-lab-elasticsearch
>
> But I am still having a lot of trouble.
>
> Below is a sample of code that I have written:
>
> package ca.nrc.ElasticSearch;
>
> import org.codehaus.jackson.map.ObjectMapper;
> import org.elasticsearch.action.get.GetResponse;
> import org.elasticsearch.action.index.IndexResponse;
> import org.elasticsearch.client.Client;
> import org.elasticsearch.node.NodeBuilder;
>
> public class ElasticSearchRunner {
>
> static ObjectMapper mapper;
> static Client client;
> static String indexName = "meal5";
> static String typeName = "beer";
> static long startTimeMSecs;
>
> public static void main(String[] args) throws Exception {
> startTimeMSecs = System.currentTimeMillis();
> mapper = new ObjectMapper(); // create once, reuse
>  echo("Creating the ElasticSearch client...");
> client = NodeBuilder.nodeBuilder().node().client(); // Does this create a
> brand new cluster?
> // client =
> NodeBuilder.nodeBuilder().clusterName("handson").client(true).node().client();
> // Joins existing cluster called "handson"
> echo("DONE creating the ElasticSearch client... Elapsed time =
> "+elapsedSecs()+" secs.");
>  echo("Creating a beer object...");
> Beer beer = new Beer("Heineken", Colour.PALE, 0.33, 3);
> String jsonString = mapper.writeValueAsString(beer);
> echo("DONE Creating a beer object...");
>
> echo("Indexing the beer object...");
> IndexResponse ir = null;
> ir = client.prepareIndex(indexName, typeName).setSource(jsonString)
> .execute().actionGet();
> echo("DONE Indexing the beer object...");
>
> echo("Retrieving the beer object...");
> GetResponse gr = null;
> gr = client.prepareGet(indexName, typeName, ir.getId()).execute()
> .actionGet();
> echo("DONE Retrieving the beer object...");
> }
>
>  public static float elapsedSecs() {
> float elapsed = (System.currentTimeMillis() - startTimeMSecs)/1000;
> return elapsed;
> }
>  public static void echo(String mess) {
> mess = mess + " (Elapsed so far: "+elapsedSecs()+" seconds)";
> System.out.println(mess);
> }
> }
>
>
> It works, "sort of"...
>
> If I use the first method for creating the client:
>
> client = NodeBuilder.nodeBuilder().node().client();
>
> Then it works fin the first time I run it. However:
>
> *** ISSUE 1: If I try to inspect the meal index with Marvel, I don't find
> it.
>
> Also,
>
> *** ISSUE 2: If I run the application a second time, I get the following
> output:
>
> Creating the ElasticSearch client... (Elapsed so far: 0.0 seconds)
> log4j:WARN No appenders could be found for logger (org.elasticsearch.node).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
> DONE creating the ElasticSearch client... Elapsed time = 9.0 secs.
> (Elapsed so far: 9.0 seconds)
> Creating a beer object... (Elapsed so far: 9.0 seconds)
> DONE Creating a beer object... (Elapsed so far: 9.0 seconds)
> Indexing the beer object... (Elapsed so far: 9.0 seconds)
> Exception in thread "main"
> org.elasticsearch.action.UnavailableShardsException: [meal5][0] [2]
> shardIt, [0] active : Timeout waiting for [1m], request: index
> {[meal5][beer][B3F5ZEmSTruqdnlxhYviFg],
> source[{"brand":"Heineken","colour":"PALE","size":0.33,"price":3.0}]}
> at
> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.raiseTimeoutFailure(TransportShardReplicationOperationAction.java:526)
> at
> org.elasticsearch.action.support.replication.TransportS

Need help with Java API

[Alain Désilets]

I am trying to get started with the Java API, using the excellent tutorial 
found here:

   http://www.slideshare.net/dadoonet/hands-on-lab-elasticsearch

But I am still having a lot of trouble.

Below is a sample of code that I have written:

package ca.nrc.ElasticSearch;

import org.codehaus.jackson.map.ObjectMapper;
import org.elasticsearch.action.get.GetResponse;
import org.elasticsearch.action.index.IndexResponse;
import org.elasticsearch.client.Client;
import org.elasticsearch.node.NodeBuilder;

public class ElasticSearchRunner {

static ObjectMapper mapper;
static Client client;
static String indexName = "meal5";
static String typeName = "beer";
static long startTimeMSecs;

public static void main(String[] args) throws Exception {
startTimeMSecs = System.currentTimeMillis(); 
mapper = new ObjectMapper(); // create once, reuse
 echo("Creating the ElasticSearch client..."); 
client = NodeBuilder.nodeBuilder().node().client(); // Does this create a 
brand new cluster?
// client = 
NodeBuilder.nodeBuilder().clusterName("handson").client(true).node().client(); 
// Joins existing cluster called "handson"
echo("DONE creating the ElasticSearch client... Elapsed time = 
"+elapsedSecs()+" secs."); 
 echo("Creating a beer object...");
Beer beer = new Beer("Heineken", Colour.PALE, 0.33, 3);
String jsonString = mapper.writeValueAsString(beer);
echo("DONE Creating a beer object...");

echo("Indexing the beer object...");
IndexResponse ir = null;
ir = client.prepareIndex(indexName, typeName).setSource(jsonString)
.execute().actionGet();
echo("DONE Indexing the beer object...");

echo("Retrieving the beer object...");
GetResponse gr = null;
gr = client.prepareGet(indexName, typeName, ir.getId()).execute()
.actionGet();
echo("DONE Retrieving the beer object..."); 
}

 public static float elapsedSecs() {
float elapsed = (System.currentTimeMillis() - startTimeMSecs)/1000;
return elapsed;
}
 public static void echo(String mess) {
mess = mess + " (Elapsed so far: "+elapsedSecs()+" seconds)";
System.out.println(mess);
}
}


It works, "sort of"...

If I use the first method for creating the client:

client = NodeBuilder.nodeBuilder().node().client();

Then it works fin the first time I run it. However:

*** ISSUE 1: If I try to inspect the meal index with Marvel, I don't find 
it.

Also, 

*** ISSUE 2: If I run the application a second time, I get the following 
output:

Creating the ElasticSearch client... (Elapsed so far: 0.0 seconds)
log4j:WARN No appenders could be found for logger (org.elasticsearch.node).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for 
more info.
DONE creating the ElasticSearch client... Elapsed time = 9.0 secs. (Elapsed 
so far: 9.0 seconds)
Creating a beer object... (Elapsed so far: 9.0 seconds)
DONE Creating a beer object... (Elapsed so far: 9.0 seconds)
Indexing the beer object... (Elapsed so far: 9.0 seconds)
Exception in thread "main" 
org.elasticsearch.action.UnavailableShardsException: [meal5][0] [2] 
shardIt, [0] active : Timeout waiting for [1m], request: index 
{[meal5][beer][B3F5ZEmSTruqdnlxhYviFg], 
source[{"brand":"Heineken","colour":"PALE","size":0.33,"price":3.0}]}
at 
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.raiseTimeoutFailure(TransportShardReplicationOperationAction.java:526)
at 
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$3.onTimeout(TransportShardReplicationOperationAction.java:516)
at 
org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:239)
at 
org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:494)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)


For me to be able to run the application again, I have to change the 
 indexName variable to a different value (ex: "meal6").

Also: 

*** ISSUE 3: If I try using the second method for creating a Client, then 
it doesn't work. More specifically, if I use this method:

client = 
NodeBuilder.nodeBuilder().clusterName("handson").client(true).node().client(); 

This yields:

Creating the ElasticSearch client... (Elapsed so far: 0.0 seconds)
log4j:WARN No appenders could be found for logger (org.elasticsearch.node).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for 
more info.
DONE creating the ElasticSearch client... Elapsed time = 34.0 secs. 
(Elapsed so far: 34.0 seconds)
Creating a beer object... (Elapsed so far: 34.0 seconds)
DONE Creating a beer object... (Elapsed so far: 34.0 seconds)
Indexing the beer object... (Elapsed so far: 34.0 seconds)
Exception in thread "main"