Re: Writing unit tests to test complex solr queries

2019-05-16 Thread Pratik Patel
Thanks a lot for the response Mikhail and Angie!

I did go through most of the test classes in solr before posting here but
couldn't find anything which is close to what I want to do which is to load
pre-created index files and configuration or at least index files.
However, the class HelloWorldSolrCloudTestCase.java class pointed out by
Angie put together with his code that he has shared seems to be completing
the picture and looks spot on! Thanks a lot.

I will try to re-write my unit tests with this approach and will post an
update soon.

@Angie, can you please share the format of data in your
"testdata/test-data.json" file? I want to be sure about using the correct
format.

Thanks!
Pratik

On Tue, May 14, 2019 at 1:14 PM Angie Rabelero 
wrote:

> Hi, I’ll advised you to extend the class SolrCloudTestCase, which extends
> the MiniSolrCloudCluster. Theres a hello world example in the solr source
> at
> https://github.com/apache/lucene-solr/blob/master/solr/core/src/test/org/apache/solr/HelloWorldSolrCloudTestCase.java
> .
>
> Here’s how I setup a cluster, create a collection with my ConfigSet, and
> index a file.
>
> @BeforeClass
> public static void setupCluster() throws Exception {
>
> // Create and configure cluster
> configureCluster(nodeCount)
> .addConfig(CONFIG_NAME, getFile(CONFIG_DIR).toPath())
> .configure();
>
> // Create an empty collection
> Create.createCollection(COLLECTION, CONFIG_NAME, numShards,
> numReplicas)
>   .setMaxShardsPerNode(maxShardsPerNode)
>   .process(cluster.getSolrClient(), COLLECTION);
> AbstractDistribZkTestBase
> .waitForRecoveriesToFinish(COLLECTION,
> cluster.getSolrClient().getZkStateReader(), true, true, 120);
>
> // Set default collection
> cluster.getSolrClient().setDefaultCollection(COLLECTION);
>
> // Add documents to collection
> ContentStreamUpdateRequest up = new
> ContentStreamUpdateRequest("/update");
> up.addFile(getFile("testdata/test-data.json"), "application/json");
> up.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
> NamedList result = cluster.getSolrClient().request(up);
>
> // Print cluster status
> System.out.println("Default Collection: " +
> cluster.getSolrClient().getDefaultCollection());
> System.out.println("Cluster State: " +
> cluster.getSolrClient().getZkStateReader().getClusterState());
> System.out.println("Update Result: " + result);
>
> }
>
>  I copy the configset to the resources dir in the pom using a mauven
> plugin. And the test file is already in the resources dir.
>
>
>
>
> > On May 14, 2019, at 04:01, Mikhail Khludnev  wrote:
> >
> > Hello, Pratick.
> > Welcome to mysterious world of Solr testing. The best way is to find
> > existing test closest to your problem field, copy in and amend
> necessarily.
> > What about
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_lucene-2Dsolr_blob_master_solr_solrj_src_test_org_apache_solr_client_solrj_io_stream_StreamExpressionTest.java=DwIBaQ=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE=lUsTzFRk0CX38HvagQ0wd52D67dA0fx_D6M6F3LHzAU=9tFliF4KA1tiG2lGmDJWO34hyq9-Sz1inAxRPVKkz78=KjveDzxzQAKRmvzPYk2y1FQ-w6yAGWuwfTVGHMQP2ZA=
> > ?
> >
> > On Fri, May 10, 2019 at 11:36 PM Pratik Patel 
> wrote:
> >
> >> Hello Everyone,
> >>
> >> I want to write unit tests for some solr queries which are being
> triggered
> >> through java code. These queries includes complex streaming expressions
> and
> >> faceting queries which requires large number of documents to be present
> in
> >> solr index. I can not create and push so many documents programmatically
> >> through my tests.
> >>
> >> I am trying to find a way to test these queries without depending on
> >> externally running solr instance. I found following approach which is
> using
> >> classes like EmbeddedSolrServer and CoreContainer. We can put index
> files
> >> and solr configuration on classpath and run the tests against them.
> >>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__dzone.com_articles_junit-2Dtesting-2Dfor-2Dsolr-2D6=DwIBaQ=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE=lUsTzFRk0CX38HvagQ0wd52D67dA0fx_D6M6F3LHzAU=9tFliF4KA1tiG2lGmDJWO34hyq9-Sz1inAxRPVKkz78=K4vPwvz9h9H8s-nsZTbkmCvTh002RP3CHcpbb9IOrpw=
> >>
> >> However, this seems to be an old approach and I am trying to find a way
> to
> >> do it using latest solr-test-framework. I also can not use old approach
> >> because I want to test Streaming Expressions as well and I need
> >> SolrCloudClient for that.
> >> In solr-test-framework, I found MiniSolrCloudCluster class but I don't
> know
> >> how to use pre-created index files and configuration with that.
> >>
> >> Does anyone know how we can use pre-created index files and
> configuration
> >> with latest test-framework? What is the recommended way to do such kind
> of
> >> 

Re: Writing unit tests to test complex solr queries

2019-05-14 Thread Angie Rabelero
Hi, I’ll advised you to extend the class SolrCloudTestCase, which extends the 
MiniSolrCloudCluster. Theres a hello world example in the solr source at 
https://github.com/apache/lucene-solr/blob/master/solr/core/src/test/org/apache/solr/HelloWorldSolrCloudTestCase.java.

Here’s how I setup a cluster, create a collection with my ConfigSet, and index 
a file.

@BeforeClass
public static void setupCluster() throws Exception {

// Create and configure cluster
configureCluster(nodeCount)
.addConfig(CONFIG_NAME, getFile(CONFIG_DIR).toPath())
.configure();

// Create an empty collection
Create.createCollection(COLLECTION, CONFIG_NAME, numShards, numReplicas)
  .setMaxShardsPerNode(maxShardsPerNode)
  .process(cluster.getSolrClient(), COLLECTION);
AbstractDistribZkTestBase
.waitForRecoveriesToFinish(COLLECTION, 
cluster.getSolrClient().getZkStateReader(), true, true, 120);

// Set default collection
cluster.getSolrClient().setDefaultCollection(COLLECTION);

// Add documents to collection
ContentStreamUpdateRequest up = new 
ContentStreamUpdateRequest("/update");
up.addFile(getFile("testdata/test-data.json"), "application/json");
up.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
NamedList result = cluster.getSolrClient().request(up);

// Print cluster status
System.out.println("Default Collection: " + 
cluster.getSolrClient().getDefaultCollection());
System.out.println("Cluster State: " + 
cluster.getSolrClient().getZkStateReader().getClusterState());
System.out.println("Update Result: " + result);

}

 I copy the configset to the resources dir in the pom using a mauven plugin. 
And the test file is already in the resources dir.




> On May 14, 2019, at 04:01, Mikhail Khludnev  wrote:
> 
> Hello, Pratick.
> Welcome to mysterious world of Solr testing. The best way is to find
> existing test closest to your problem field, copy in and amend necessarily.
> What about
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_lucene-2Dsolr_blob_master_solr_solrj_src_test_org_apache_solr_client_solrj_io_stream_StreamExpressionTest.java=DwIBaQ=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE=lUsTzFRk0CX38HvagQ0wd52D67dA0fx_D6M6F3LHzAU=9tFliF4KA1tiG2lGmDJWO34hyq9-Sz1inAxRPVKkz78=KjveDzxzQAKRmvzPYk2y1FQ-w6yAGWuwfTVGHMQP2ZA=
> ?
> 
> On Fri, May 10, 2019 at 11:36 PM Pratik Patel  wrote:
> 
>> Hello Everyone,
>> 
>> I want to write unit tests for some solr queries which are being triggered
>> through java code. These queries includes complex streaming expressions and
>> faceting queries which requires large number of documents to be present in
>> solr index. I can not create and push so many documents programmatically
>> through my tests.
>> 
>> I am trying to find a way to test these queries without depending on
>> externally running solr instance. I found following approach which is using
>> classes like EmbeddedSolrServer and CoreContainer. We can put index files
>> and solr configuration on classpath and run the tests against them.
>> 
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__dzone.com_articles_junit-2Dtesting-2Dfor-2Dsolr-2D6=DwIBaQ=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE=lUsTzFRk0CX38HvagQ0wd52D67dA0fx_D6M6F3LHzAU=9tFliF4KA1tiG2lGmDJWO34hyq9-Sz1inAxRPVKkz78=K4vPwvz9h9H8s-nsZTbkmCvTh002RP3CHcpbb9IOrpw=
>> 
>> However, this seems to be an old approach and I am trying to find a way to
>> do it using latest solr-test-framework. I also can not use old approach
>> because I want to test Streaming Expressions as well and I need
>> SolrCloudClient for that.
>> In solr-test-framework, I found MiniSolrCloudCluster class but I don't know
>> how to use pre-created index files and configuration with that.
>> 
>> Does anyone know how we can use pre-created index files and configuration
>> with latest test-framework? What is the recommended way to do such kind of
>> testing? Any direction with this would be really helpful.
>> 
>> Thanks!
>> Pratik
>> 
> 
> 
> -- 
> Sincerely yours
> Mikhail Khludnev