Re: The Streaming API (Solrj.io) : id must have DocValues?

2016-04-26 Thread Joel Bernstein
My blog is pretty out of date at this point unfortunately. I need to get some better examples published. Also there is huge amount of work that went into Solr 6 Streaming API and Streaming Expressions that make them much easier to work with. In Solr 6.1 you'll be able to test Streaming

Re: The Streaming API (Solrj.io) : id must have DocValues?

2016-04-26 Thread sudsport s
I see that some work was done to remove stream handler form config. so enabling stream handler is still security issue? https://issues.apache.org/jira/browse/SOLR-8262 On Tue, Apr 26, 2016 at 11:14 AM, sudsport s wrote: > I am using solr 5.3.1 server & solr5.5 on client (

Re: The Streaming API (Solrj.io) : id must have DocValues?

2016-04-26 Thread sudsport s
I am using solr 5.3.1 server & solr5.5 on client ( solrj) . I will try with solrj 6.0 On Tue, Apr 26, 2016 at 11:12 AM, Susmit Shukla wrote: > Which solrj version are you using? could you try with solrj 6.0 > > On Tue, Apr 26, 2016 at 10:36 AM, sudsport s

Re: The Streaming API (Solrj.io) : id must have DocValues?

2016-04-26 Thread Susmit Shukla
Which solrj version are you using? could you try with solrj 6.0 On Tue, Apr 26, 2016 at 10:36 AM, sudsport s wrote: > @Joel > >Can you describe how you're planning on using Streaming? > > I am mostly using it for distirbuted join case. We were planning to use > similar

Re: The Streaming API (Solrj.io) : id must have DocValues?

2016-04-26 Thread sudsport s
@Joel >Can you describe how you're planning on using Streaming? I am mostly using it for distirbuted join case. We were planning to use similar logic (hash id and join) in Spark for our usecase. but since data is stored in solr , I will be using solr stream to perform same operation. I have

Re: The Streaming API (Solrj.io) : id must have DocValues?

2016-04-26 Thread sudsport s
Thanks @Reth yes that was my one of the concern. I will look at JIRA you mentioned. Thanks Joel I used some of examples for streaming client from your blog. I got basic tuple stream working but I get following exception while running parallel string. java.io.IOException:

Re: The Streaming API (Solrj.io) : id must have DocValues?

2016-04-25 Thread Joel Bernstein
Can you describe how you're planning on using Streaming? I can provide some feedback on how it will perform for your use use. When scaling out Streaming you'll get large performance boosts when you increase the number of shards, replicas and workers. This is particularly true if you're doing

Re: The Streaming API (Solrj.io) : id must have DocValues?

2016-04-25 Thread Reth RM
Hi, So, is the concern related to same field value being stored twice: with stored=true and docValues=true? If that is the case, there is a jira relevant to this, fixed[1]. If you upgrade to 5.5/6.0 version, it is possible to read non-stored fields from docValues index., check out. [1]

Re: The Streaming API (Solrj.io) : id must have DocValues?

2016-04-24 Thread sudsport s
Thanks Erik for reply, Since I was storing Id (its stored field) and after enabling docValues my guess is it will be stored in 2 places. also as per my understanding docValues are great when you have values which repeat. I am not sure how beneficial it would be for uniqueId field. I am looking at

Re: The Streaming API (Solrj.io) : id must have DocValues?

2016-04-24 Thread Erick Erickson
In a word, "yes". DocValues aren't particularly expensive, or expensive at all. The idea is that when you sort by a field or facet, the field has to be "uninverted" which builds the entire structure in Java's JVM (this is when the field is _not_ DocValues). DocValues essentially serialize this

The Streaming API (Solrj.io) : id must have DocValues?

2016-04-24 Thread sudsport s
I was trying to use Streaming for reading basic tuple stream. I am using sort by id asc , I am getting following exception I am using export search handler as per https://cwiki.apache.org/confluence/display/solr/Exporting+Result+Sets null:java.io.IOException: id must have DocValues to use this