Re: Fast autocomplete for large dataset

[Olivier Austina]

Thank you Eric for your replies and the link.

Regards
Olivier


2015-08-02 3:47 GMT+02:00 Erick Erickson :

> Here's some background:
>
> http://lucidworks.com/blog/solr-suggester/
>
> Basically, the limitation is that to build the suggester all docs in
> the index need to be read to pull out the stored field and build
> either the FST or the sidecar Lucene index, which can be a _very_
> costly operation (as in minutes/hours for a large dataset).
>
> bq: The requirement is that the autocomplete should be fast (not
> slowdown by the volume of data as dataset become bigger)
>
> Well, in some alternate universe this may be possible. But the larger
> the corpus the slower the processing will be, there's just no way
> around that. Whether it's fast enough for your application is a better
> question ;).
>
> Best,
> Erick
>
>
> On Sat, Aug 1, 2015 at 2:05 PM, Olivier Austina
>  wrote:
> > Thank you Eric,
> >
> > I would like to implement an autocomplete for large dataset.  The
> > autocomplete should show the phrase or the question the user want as the
> > user types. The requirement is that the autocomplete should be fast (not
> > slowdown by the volume of data as dataset become bigger), and easy to
> > maintain. The autocomplete can have its own Solr server.  It is an
> > autocomplete like others but it should be only fast and easy to maintain.
> >
> > What is the limitations of suggesters mentioned in the article? Thank
> you.
> >
> > Regards
> > Olivier
> >
> >
> > 2015-08-01 19:41 GMT+02:00 Erick Erickson :
> >
> >> Not really. There's no need to use ngrams as the article suggests if the
> >> terms component does what you need. Which is why I asked you about what
> >> autocomplete means in your context. Which you have not clarified. Have
> you
> >> even looked at terms component?  Especially the terms.prefix option?
> >>
> >> Terms component has it's limitations, but performance isn't one of them.
> >> The suggesters mentioned in the article have other limitations. It's
> really
> >> useless to discuss those limitations, though, until the problem you're
> >> trying to solve is clearly stated.
> >> On Aug 1, 2015 1:01 PM, "Olivier Austina" 
> >> wrote:
> >>
> >> > Thank you Eric for your reply.
> >> > If I understand it seems that these approaches are using index to hold
> >> > terms. As the index grows bigger, it can be a performance issues.
> >> > Is it right? Please can you check this article
> >> > <http://www.norconex.com/serving-autocomplete-suggestions-fast/> to
> see
> >> > what I mean?   Thank you.
> >> >
> >> > Regards
> >> > Olivier
> >> >
> >> >
> >> > 2015-08-01 17:42 GMT+02:00 Erick Erickson :
> >> >
> >> > > Well, defining what you mean by "autocomplete" would be a start. If
> >> it's
> >> > > just
> >> > > a user types some letters and you suggest the next N terms in the
> list,
> >> > > TermsComponent will fix you right up.
> >> > >
> >> > > If it's more complicated, the AutoSuggest functionality might help.
> >> > >
> >> > > If it's correcting spelling, there's the spellchecker.
> >> > >
> >> > > Best,
> >> > > Erick
> >> > >
> >> > > On Sat, Aug 1, 2015 at 10:00 AM, Olivier Austina
> >> > >  wrote:
> >> > > > Hi,
> >> > > >
> >> > > > I am looking for a fast and easy to maintain way to do
> autocomplete
> >> for
> >> > > > large dataset in solr. I heard about Ternary Search Tree (TST)
> >> > > > <https://en.wikipedia.org/wiki/Ternary_search_tree>.
> >> > > > But I would like to know if there is something I missed such as
> best
> >> > > > practice, Solr new feature. Any suggestion is welcome. Thank you.
> >> > > >
> >> > > > Regards
> >> > > > Olivier
> >> > >
> >> >
> >>
>

Re: Fast autocomplete for large dataset

[Olivier Austina]

Thank you Eric,

I would like to implement an autocomplete for large dataset.  The
autocomplete should show the phrase or the question the user want as the
user types. The requirement is that the autocomplete should be fast (not
slowdown by the volume of data as dataset become bigger), and easy to
maintain. The autocomplete can have its own Solr server.  It is an
autocomplete like others but it should be only fast and easy to maintain.

What is the limitations of suggesters mentioned in the article? Thank you.

Regards
Olivier


2015-08-01 19:41 GMT+02:00 Erick Erickson :

> Not really. There's no need to use ngrams as the article suggests if the
> terms component does what you need. Which is why I asked you about what
> autocomplete means in your context. Which you have not clarified. Have you
> even looked at terms component?  Especially the terms.prefix option?
>
> Terms component has it's limitations, but performance isn't one of them.
> The suggesters mentioned in the article have other limitations. It's really
> useless to discuss those limitations, though, until the problem you're
> trying to solve is clearly stated.
> On Aug 1, 2015 1:01 PM, "Olivier Austina" 
> wrote:
>
> > Thank you Eric for your reply.
> > If I understand it seems that these approaches are using index to hold
> > terms. As the index grows bigger, it can be a performance issues.
> > Is it right? Please can you check this article
> > <http://www.norconex.com/serving-autocomplete-suggestions-fast/> to see
> > what I mean?   Thank you.
> >
> > Regards
> > Olivier
> >
> >
> > 2015-08-01 17:42 GMT+02:00 Erick Erickson :
> >
> > > Well, defining what you mean by "autocomplete" would be a start. If
> it's
> > > just
> > > a user types some letters and you suggest the next N terms in the list,
> > > TermsComponent will fix you right up.
> > >
> > > If it's more complicated, the AutoSuggest functionality might help.
> > >
> > > If it's correcting spelling, there's the spellchecker.
> > >
> > > Best,
> > > Erick
> > >
> > > On Sat, Aug 1, 2015 at 10:00 AM, Olivier Austina
> > >  wrote:
> > > > Hi,
> > > >
> > > > I am looking for a fast and easy to maintain way to do autocomplete
> for
> > > > large dataset in solr. I heard about Ternary Search Tree (TST)
> > > > <https://en.wikipedia.org/wiki/Ternary_search_tree>.
> > > > But I would like to know if there is something I missed such as best
> > > > practice, Solr new feature. Any suggestion is welcome. Thank you.
> > > >
> > > > Regards
> > > > Olivier
> > >
> >
>

Re: Fast autocomplete for large dataset

[Olivier Austina]

Thank you Eric for your reply.
If I understand it seems that these approaches are using index to hold
terms. As the index grows bigger, it can be a performance issues.
Is it right? Please can you check this article
<http://www.norconex.com/serving-autocomplete-suggestions-fast/> to see
what I mean?   Thank you.

Regards
Olivier


2015-08-01 17:42 GMT+02:00 Erick Erickson :

> Well, defining what you mean by "autocomplete" would be a start. If it's
> just
> a user types some letters and you suggest the next N terms in the list,
> TermsComponent will fix you right up.
>
> If it's more complicated, the AutoSuggest functionality might help.
>
> If it's correcting spelling, there's the spellchecker.
>
> Best,
> Erick
>
> On Sat, Aug 1, 2015 at 10:00 AM, Olivier Austina
>  wrote:
> > Hi,
> >
> > I am looking for a fast and easy to maintain way to do autocomplete for
> > large dataset in solr. I heard about Ternary Search Tree (TST)
> > <https://en.wikipedia.org/wiki/Ternary_search_tree>.
> > But I would like to know if there is something I missed such as best
> > practice, Solr new feature. Any suggestion is welcome. Thank you.
> >
> > Regards
> > Olivier
>

Fast autocomplete for large dataset

[Olivier Austina]

Hi,

I am looking for a fast and easy to maintain way to do autocomplete for
large dataset in solr. I heard about Ternary Search Tree (TST)
.
But I would like to know if there is something I missed such as best
practice, Solr new feature. Any suggestion is welcome. Thank you.

Regards
Olivier

Re: How to implement Auto complete, suggestion client side

[Olivier Austina]

Hi,

Thank you Dan Davis and Alexandre Rafalovitch. This is very helpful for me.

Regards
Olivier


2015-01-27 0:51 GMT+01:00 Alexandre Rafalovitch :

> You've got a lot of options depending on what you want. But since you
> seem to just want _an_ example, you can use mine from
> http://www.solr-start.com/javadoc/solr-lucene/index.html (gray search
> box there).
>
> You can see the source for the test screen (using Spring Boot and
> Spring Data Solr as a middle-layer) and Select2 for the UI at:
> https://github.com/arafalov/Solr-Javadoc/tree/master/SearchServer.
> The Solr definition is at:
>
> https://github.com/arafalov/Solr-Javadoc/tree/master/JavadocIndex/JavadocCollection/conf
>
> Other implementation pieces are in that (and another) public
> repository as well, but it's all in Java. You'll probably want to do
> something similar in PHP.
>
> Regards,
>Alex.
> 
> Sign up for my Solr resources newsletter at http://www.solr-start.com/
>
>
> On 26 January 2015 at 17:11, Olivier Austina 
> wrote:
> > Hi All,
> >
> > I would say I am new to web technology.
> >
> > I would like to implement auto complete/suggestion in the user search box
> > as the user type in the search box (like Google for example). I am using
> > Solr as database. Basically I am  familiar with Solr and I can formulate
> > suggestion queries.
> >
> > But now I don't know how to implement suggestion in the User Interface.
> > Which technologies should I need. The website is in PHP. Any suggestions,
> > examples, basic tutorial is welcome. Thank you.
> >
> >
> >
> > Regards
> > Olivier
>

How to implement Auto complete, suggestion client side

[Olivier Austina]

Hi All,

I would say I am new to web technology.

I would like to implement auto complete/suggestion in the user search box
as the user type in the search box (like Google for example). I am using
Solr as database. Basically I am  familiar with Solr and I can formulate
suggestion queries.

But now I don't know how to implement suggestion in the User Interface.
Which technologies should I need. The website is in PHP. Any suggestions,
examples, basic tutorial is welcome. Thank you.



Regards
Olivier

Architecture for PHP web site, Solr and an application

[Olivier Austina]

Hi,

I would like to query only some fields in Solr depend on the user input as
I know the fields.

The user send an HTML form to the PHP website. The application get the
fields and their content from the PHP web site. The application then
formulate a query to Solr based on this fields and other contextual
information. Only fields from the HTML form are used. The forms don't have
the same fields. The application is not yet developed. It could be in C++,
Java or other language using a database. It uses more resources.

I am wondering which architecture is suitable for this case:
-How to make the architecture scalable (to support more users)
-How to make PHP communicate with the application if this application is
not in PHP.

Any suggestion is welcome. Thank you.

 Regards
Olivier

Re: UI for Solr

[Olivier Austina]

Hi Alex,

Thank you for prompt reply. I am not aware of Spring.io's Spring Data Solr.

Regards
Olivier


2014-12-23 16:50 GMT+01:00 Alexandre Rafalovitch :

> You don't expose Solr directly to the user, it is not setup for
> full-proof security out of the box. So you would need a client to talk
> to Solr.
>
> Something like Spring.io's Spring Data Solr could be one of the things
> to check. You can see an auto-complete example for it at:
> https://github.com/arafalov/Solr-Javadoc/tree/master/SearchServer/src/main
> and embedded in action at
> http://www.solr-start.com/javadoc/solr-lucene/index.html (search box
> on the top)
>
> Regards,
>Alex.
> 
> Sign up for my Solr resources newsletter at http://www.solr-start.com/
>
>
> On 23 December 2014 at 10:45, Olivier Austina 
> wrote:
> > Hi,
> >
> > I would like to build a User Interface on top of Solr for PC and mobile.
> I
> > am wondering if there is a framework, best practice commonly used. I want
> > Solr features such as suggestion, auto complete, facet to be available
> for
> > UI. Any suggestion is welcome. Than you.
> >
> > Regards
> > Olivier
>

UI for Solr

[Olivier Austina]

Hi,

I would like to build a User Interface on top of Solr for PC and mobile. I
am wondering if there is a framework, best practice commonly used. I want
Solr features such as suggestion, auto complete, facet to be available for
UI. Any suggestion is welcome. Than you.

Regards
Olivier

Re: Indexing documents/files for production use

[Olivier Austina]

Thank you Alexandre, Jürgen and Erick for your replies. It is clear for me.

Regards
Olivier


2014-10-28 23:35 GMT+01:00 Erick Erickson :

> And one other consideration in addition to the two excellent responses
> so far
>
> In a SolrCloud environment, SolrJ via CloudSolrServer will automatically
> route the documents to the correct shard leader, saving some additional
> overhead. Post.jar and cURL send the docs to a node, which in turn
> forward the docs to the correct shard leader which lowers
> throughput
>
> Best,
> Erick
>
> On Tue, Oct 28, 2014 at 2:32 PM, "Jürgen Wagner (DVT)"
>  wrote:
> > Hello Olivier,
> >   for real production use, you won't really want to use any toys like
> > post.jar or curl. You want a decent connector to whatever data source
> there
> > is, that fetches data, possibly massages it a bit, and then feeds it into
> > Solr - by means of SolrJ or directly into the web service of Solr via
> binary
> > protocols. This way, you can properly handle incremental feeding,
> processing
> > of data from remote locations (with the connector being closer to the
> data
> > source), and also source data security. Also think about what happens if
> you
> > do processing of incoming documents in Solr. What happens if Tika runs
> out
> > of memory because of PDF problems? What if this crashes your Solr node?
> In
> > our Solr projects, we generally do not do any sizable processing within
> Solr
> > as document processing and document indexing or querying have all
> different
> > scaling properties.
> >
> > "Production use" most typically is not achieved by deploying a vanilla
> Solr,
> > but rather having a bit more glue and wrappage, so the whole will fit
> your
> > requirements in terms of functionality, scaling, monitoring and
> robustness.
> > Some similar platforms like Elasticsearch try to alleviate these pains of
> > going to a production-style infrastructure, but that's at the expense of
> > flexibility and comes with limitations.
> >
> > For proof-of-concept or demonstrator-style applications, the plain tools
> out
> > of the box will be fine. For production applications, you want to have
> more
> > robust components.
> >
> > Best regards,
> > --Jürgen
> >
> >
> > On 28.10.2014 22:12, Olivier Austina wrote:
> >
> > Hi All,
> >
> > I am reading the solr documentation. I have understood that post.jar
> > <
> http://wiki.apache.org/solr/ExtractingRequestHandler#SimplePostTool_.28post.jar.29
> >
> > is not meant for production use, cURL
> > <
> https://cwiki.apache.org/confluence/display/solr/Introduction+to+Solr+Indexing
> >
> > is not recommanded. Is SolrJ better for production?  Thank you.
> > Regards
> > Olivier
> >
> >
> >
> > --
> >
> > Mit freundlichen Grüßen/Kind regards/Cordialement vôtre/Atentamente/С
> > уважением
> > i.A. Jürgen Wagner
> > Head of Competence Center "Intelligence"
> > & Senior Cloud Consultant
> >
> > Devoteam GmbH, Industriestr. 3, 70565 Stuttgart, Germany
> > Phone: +49 6151 868-8725, Fax: +49 711 13353-53, Mobile: +49 171 864
> 1543
> > E-Mail: juergen.wag...@devoteam.com, URL: www.devoteam.de
> >
> > 
> > Managing Board: Jürgen Hatzipantelis (CEO)
> > Address of Record: 64331 Weiterstadt, Germany; Commercial Register:
> > Amtsgericht Darmstadt HRB 6450; Tax Number: DE 172 993 071
> >
> >
>

Indexing documents/files for production use

[Olivier Austina]

Hi All,

I am reading the solr documentation. I have understood that post.jar

is not meant for production use, cURL

is not recommanded. Is SolrJ better for production?  Thank you.
Regards
Olivier

Re: OpenExchangeRates.Org rates in solr

[Olivier Austina]

Hi Will,

I am learning Solr now. I can use it  later for business or for free
access. Thank you.

Regards
Olivier


2014-10-26 17:32 GMT+01:00 Will Martin :

> Hi Olivier:
>
> Can you clarify this message? Are you using Solr at the business? Or are
> you giving free access to solr installations?
>
> Thanks,
> Will
>
>
> -Original Message-
> From: Olivier Austina [mailto:olivier.aust...@gmail.com]
> Sent: Sunday, October 26, 2014 10:57 AM
> To: solr-user@lucene.apache.org
> Subject: OpenExchangeRates.Org rates in solr
>
> Hi,
>
> There is a way to see the OpenExchangeRates.Org <
> http://www.OpenExchangeRates.Org> rates used in Solr somewhere. I have
> changed the configuration to use these rates. Thank you.
> Regards
> Olivier
>
>

OpenExchangeRates.Org rates in solr

[Olivier Austina]

Hi,

There is a way to see the OpenExchangeRates.Org
 rates used in Solr somewhere. I have
changed the configuration to use these rates. Thank you.
Regards
Olivier

Re: Remove indexes of XML file

[Olivier Austina]

Thank you Alex, I think I can use the file to delete corresponding indexes.

Regards
Olivier


2014-10-24 21:51 GMT+02:00 Alexandre Rafalovitch :

> You can delete individually, all (*:* query) or by specific query. So,
> if there is no common query pattern you may need to do a multi-id
> query - something like "id:(id1 id2 id3 id4)" which does require you
> knowing the IDs.
>
> Regards,
>Alex.
> Personal: http://www.outerthoughts.com/ and @arafalov
> Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
> Solr popularizers community: https://www.linkedin.com/groups?gid=6713853
>
>
> On 24 October 2014 15:44, Olivier Austina 
> wrote:
> > Hi,
> >
> > This is newbie question. I have indexed some documents using some XML
> files
> > as indicating in the tutorial
> > <http://lucene.apache.org/solr/4_10_1/tutorial.html> with the command :
> >
> > java -jar post.jar *.xml
> >
> > I have seen how to delete an index for one document but how to delete
> > all indexes
> > for documents within an XML file. For example if I have indexed some
> > files A, B, C, D etc.,
> > how to delete indexes of documents from file C. Is there a command
> > like above or other
> > solution without using individual ID? Thank you.
> >
> >
> > Regards
> > Olivier
>

Remove indexes of XML file

[Olivier Austina]

Hi,

This is newbie question. I have indexed some documents using some XML files
as indicating in the tutorial
 with the command :

java -jar post.jar *.xml

I have seen how to delete an index for one document but how to delete
all indexes
for documents within an XML file. For example if I have indexed some
files A, B, C, D etc.,
how to delete indexes of documents from file C. Is there a command
like above or other
solution without using individual ID? Thank you.


Regards
Olivier

Website running Solr

[Olivier Austina]

Hi All,
Is there a way to know if a website use Solr? Thanks.
Regards
Olivier

Re: Topology of Solr use

[Olivier Austina]

Thank you Markus, the link is very useful.


Regards
Olivier



2014-04-17 18:24 GMT+02:00 Markus Jelsma :

> This may help a bit:
>
> https://wiki.apache.org/solr/PublicServers
>
> -Original message-
> From:Olivier Austina 
> Sent:Thu 17-04-2014 18:16
> Subject:Topology of Solr use
> To:solr-user@lucene.apache.org;
> Hi All,
> I would to have an idea about Solr usage: number of users, industry,
> countries or any helpful information. Thank you.
> Regards
> Olivier
>

Topology of Solr use

[Olivier Austina]

Hi All,
I would to have an idea about Solr usage: number of users, industry,
countries or any helpful information. Thank you.
Regards
Olivier

Querying specific database attributes or table

[Olivier Austina]

Hi,
I am new to Solr.

I would like to index and querying a relational database. Is it possible to
query a specific table or attribute of the database. Example if I have 2
tables A and B both have the attribute "name" and I want to have only the
results form the table A and not from table B. Is it possible?
Can I restrict the query to only one table without having result from
others table?
Is it possible to query a specific attribute of a table?
Is it possible to do join query like SQL?
Any suggestion is welcome. Thank you.

Regards
Olivier