Re: Pointing to Hbase for Docuements or Directly Saving Documents at Hbase
ghts ahead of time because they are query > >>> >> >> dependent. You could store documents in HBase and use Solr just > >>> for > >>> >> >> indexing. Is that what you want to do? If so, a custom > >>> >> >> SearchComponent executed after QueryComponent could fetch data > >>> from > >>> >> >> external store like HBase. I'm not sure if I'd recommend that. > >>> >> >> > >>> >> >> Otis > >>> >> >> -- > >>> >> >> Solr & ElasticSearch Support > >>> >> >> http://sematext.com/ > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> On Thu, Apr 11, 2013 at 10:01 AM, Furkan KAMACI < > >>> > > > furkankamaci@ > > >>> >> > > >>> >> >> wrote: > >>> >> >> > Actually I don't think to store documents at Solr. I want to > >>> store > >>> >> just > >>> >> >> > highlights (snippets) at Hbase and I want to retrieve them from > >>> Hbase > >>> >> >> when > >>> >> >> > needed. > >>> >> >> > What do you think about separating just highlights from Solr > and > >>> >> storing > >>> >> >> > them into Hbase at Solrclod. By the way if you explain at which > >>> >> process > >>> >> >> and > >>> >> >> > how highlights are genareted at Solr you are welcome. > >>> >> >> > > >>> >> >> > > >>> >> >> > 2013/4/9 Otis Gospodnetic < > > > otis.gospodnetic@ > > > > > >>> >> >> > > >>> >> >> >> You may also be interested in looking at things like solrbase > >>> (on > >>> >> >> Github). > >>> >> >> >> > >>> >> >> >> Otis > >>> >> >> >> -- > >>> >> >> >> Solr & ElasticSearch Support > >>> >> >> >> http://sematext.com/ > >>> >> >> >> > >>> >> >> >> > >>> >> >> >> > >>> >> >> >> > >>> >> >> >> > >>> >> >> >> On Sat, Apr 6, 2013 at 6:01 PM, Furkan KAMACI < > >>> >> > > > furkankamaci@ > > >> > >>> >> >> >> wrote: > >>> >> >> >> > Hi; > >>> >> >> >> > > >>> >> >> >> > First of all should mention that I am new to Solr and making > >>> a > >>> >> >> research > >>> >> >> >> > about it. What I am trying to do that I will crawl some > >>> websites > >>> >> with > >>> >> >> >> Nutch > >>> >> >> >> > and then I will index them with Solr. (Nutch 2.1, > >>> Solr-SolrCloud > >>> >> 4.2 ) > >>> >> >> >> > > >>> >> >> >> > I wonder about something. I have a cloud of machines that > >>> crawls > >>> >> >> websites > >>> >> >> >> > and stores that documents. Then I send that documents into > >>> >> SolrCloud. > >>> >> >> >> Solr > >>> >> >> >> > indexes that documents and generates indexes and save them. > I > >>> know > >>> >> >> that > >>> >> >> >> > from Information Retrieval theory: it *may* not be efficient > >>> to > >>> >> store > >>> >> >> >> > indexes at a NoSQL database (they are something like linked > >>> lists > >>> >> and > >>> >> >> if > >>> >> >> >> > you store them in such kind of database you *may* have a > >>> sparse > >>> >> >> >> > representation -by the way there may be some solutions for > >>> it. > >>> If > >>> >> you > >>> >> >> >> > explain them you are welcome.) > >>> >> >> >> > > >>> >> >> >> > However Solr stores some documents too (i.e. highlights) So > >>> some > >>> >> of my > >>> >> >> >> > documents will be doubled somehow. If I consider that I will > >>> have > >>> >> many > >>> >> >> >> > documents, that dobuled documents may cause a problem for > me. > >>> So is > >>> >> >> there > >>> >> >> >> > any way not storing that documents at Solr and pointing to > >>> them > >>> at > >>> >> >> >> > Hbase(where I save my crawled documents) or instead of > >>> pointing > >>> >> >> directly > >>> >> >> >> > storing them at Hbase (is it efficient or not)? > >>> >> >> >> > >>> >> >> > >>> >> > >>> > > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Pointing-to-Hbase-for-Docuements-or-Directly-Saving-Documents-at-Hbase-tp4054277p4056599.html > Sent from the Solr - User mailing list archive at Nabble.com. >
Re: Pointing to Hbase for Docuements or Directly Saving Documents at Hbase
olr. I want to >>> store >>> >> just >>> >> >> > highlights (snippets) at Hbase and I want to retrieve them from >>> Hbase >>> >> >> when >>> >> >> > needed. >>> >> >> > What do you think about separating just highlights from Solr and >>> >> storing >>> >> >> > them into Hbase at Solrclod. By the way if you explain at which >>> >> process >>> >> >> and >>> >> >> > how highlights are genareted at Solr you are welcome. >>> >> >> > >>> >> >> > >>> >> >> > 2013/4/9 Otis Gospodnetic < > otis.gospodnetic@ > > >>> >> >> > >>> >> >> >> You may also be interested in looking at things like solrbase >>> (on >>> >> >> Github). >>> >> >> >> >>> >> >> >> Otis >>> >> >> >> -- >>> >> >> >> Solr & ElasticSearch Support >>> >> >> >> http://sematext.com/ >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> On Sat, Apr 6, 2013 at 6:01 PM, Furkan KAMACI < >>> >> > furkankamaci@ >> >>> >> >> >> wrote: >>> >> >> >> > Hi; >>> >> >> >> > >>> >> >> >> > First of all should mention that I am new to Solr and making >>> a >>> >> >> research >>> >> >> >> > about it. What I am trying to do that I will crawl some >>> websites >>> >> with >>> >> >> >> Nutch >>> >> >> >> > and then I will index them with Solr. (Nutch 2.1, >>> Solr-SolrCloud >>> >> 4.2 ) >>> >> >> >> > >>> >> >> >> > I wonder about something. I have a cloud of machines that >>> crawls >>> >> >> websites >>> >> >> >> > and stores that documents. Then I send that documents into >>> >> SolrCloud. >>> >> >> >> Solr >>> >> >> >> > indexes that documents and generates indexes and save them. I >>> know >>> >> >> that >>> >> >> >> > from Information Retrieval theory: it *may* not be efficient >>> to >>> >> store >>> >> >> >> > indexes at a NoSQL database (they are something like linked >>> lists >>> >> and >>> >> >> if >>> >> >> >> > you store them in such kind of database you *may* have a >>> sparse >>> >> >> >> > representation -by the way there may be some solutions for >>> it. >>> If >>> >> you >>> >> >> >> > explain them you are welcome.) >>> >> >> >> > >>> >> >> >> > However Solr stores some documents too (i.e. highlights) So >>> some >>> >> of my >>> >> >> >> > documents will be doubled somehow. If I consider that I will >>> have >>> >> many >>> >> >> >> > documents, that dobuled documents may cause a problem for me. >>> So is >>> >> >> there >>> >> >> >> > any way not storing that documents at Solr and pointing to >>> them >>> at >>> >> >> >> > Hbase(where I save my crawled documents) or instead of >>> pointing >>> >> >> directly >>> >> >> >> > storing them at Hbase (is it efficient or not)? >>> >> >> >> >>> >> >> >>> >> >>> -- View this message in context: http://lucene.472066.n3.nabble.com/Pointing-to-Hbase-for-Docuements-or-Directly-Saving-Documents-at-Hbase-tp4054277p4056599.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Pointing to Hbase for Docuements or Directly Saving Documents at Hbase
Use Solr. It's pretty clear you don't yet have any problems that would make you think about alternatives. Using Solr to store and not just index will make your life simpler (and your app simpler and likely faster). Otis -- Solr & ElasticSearch Support http://sematext.com/ On Tue, Apr 16, 2013 at 6:31 PM, Furkan KAMACI wrote: > Thanks again for your answer. If I find any document about such comparisons > that I would like to read. > > By the way, is there any advantage for using Lucene instead of anything > else as like that: > > Using Lucene is naturally supported at Solr and if I use anything else I > may face with some compatibility problems or communicating issues? > > > 2013/4/17 Otis Gospodnetic > >> People do use other data stores to retrieve data sometimes. e.g. Mongo >> is popular for that. Like I hinted in another email, I wouldn't >> necessarily recommend this for common cases. Don't do it unless you >> really know you need it. Otherwise, just store in Solr. >> >> Otis >> -- >> Solr & ElasticSearch Support >> http://sematext.com/ >> >> >> >> >> >> On Tue, Apr 16, 2013 at 5:32 PM, Furkan KAMACI >> wrote: >> > Hi Otis and Jack; >> > >> > I have made a research about highlights and debugged code. I see that >> > highlight are query dependent and not stored. Why Solr uses Lucene for >> > storing text, I mean i.e. content of a web page. Is there any comparison >> > about to store texts at Hbase or any other databases versus Lucene. >> > >> > Also I want to learn that is there anybody who has used anything else >> from >> > Lucene to store text of document at our solr user list? >> > >> > 2013/4/11 Otis Gospodnetic >> > >> >> Source code is your best bet. Wiki has info about how to use it, but >> >> not how highlighting is implemented. But you don't need to understand >> >> the implementation details to understand that they are dynamic, >> >> computed specifically for each query for each matching document, so >> >> you cannot store them anywhere ahead of time. >> >> >> >> Otis >> >> -- >> >> Solr & ElasticSearch Support >> >> http://sematext.com/ >> >> >> >> >> >> >> >> >> >> >> >> On Thu, Apr 11, 2013 at 11:22 AM, Furkan KAMACI > > >> >> wrote: >> >> > Hi Otis; >> >> > >> >> > It seems that I should read more about highlights. Is there any where >> >> that >> >> > explains in detail how highlights are generated at Solr? >> >> > >> >> > 2013/4/11 Otis Gospodnetic >> >> > >> >> >> Hi, >> >> >> >> >> >> You can't store highlights ahead of time because they are query >> >> >> dependent. You could store documents in HBase and use Solr just for >> >> >> indexing. Is that what you want to do? If so, a custom >> >> >> SearchComponent executed after QueryComponent could fetch data from >> >> >> external store like HBase. I'm not sure if I'd recommend that. >> >> >> >> >> >> Otis >> >> >> -- >> >> >> Solr & ElasticSearch Support >> >> >> http://sematext.com/ >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Thu, Apr 11, 2013 at 10:01 AM, Furkan KAMACI < >> furkankam...@gmail.com >> >> > >> >> >> wrote: >> >> >> > Actually I don't think to store documents at Solr. I want to store >> >> just >> >> >> > highlights (snippets) at Hbase and I want to retrieve them from >> Hbase >> >> >> when >> >> >> > needed. >> >> >> > What do you think about separating just highlights from Solr and >> >> storing >> >> >> > them into Hbase at Solrclod. By the way if you explain at which >> >> process >> >> >> and >> >> >> > how highlights are genareted at Solr you are welcome. >> >> >> > >> >> >> > >> >> >> > 2013/4/9 Otis Gospodnetic >> >> >> > >> >> >> >> You may also be interested in looking at things like solrbase (on >> >> >> Github). >> >> >> >> >> >> >> >> Otis >> >> >> >> -- >> >> >> >> Solr & ElasticSearch Support >> >> >> >> http://sematext.com/ >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Sat, Apr 6, 2013 at 6:01 PM, Furkan KAMACI < >> >> furkankam...@gmail.com> >> >> >> >> wrote: >> >> >> >> > Hi; >> >> >> >> > >> >> >> >> > First of all should mention that I am new to Solr and making a >> >> >> research >> >> >> >> > about it. What I am trying to do that I will crawl some websites >> >> with >> >> >> >> Nutch >> >> >> >> > and then I will index them with Solr. (Nutch 2.1, Solr-SolrCloud >> >> 4.2 ) >> >> >> >> > >> >> >> >> > I wonder about something. I have a cloud of machines that crawls >> >> >> websites >> >> >> >> > and stores that documents. Then I send that documents into >> >> SolrCloud. >> >> >> >> Solr >> >> >> >> > indexes that documents and generates indexes and save them. I >> know >> >> >> that >> >> >> >> > from Information Retrieval theory: it *may* not be efficient to >> >> store >> >> >> >> > indexes at a NoSQL database (they are something like linked >> lists >> >> and >> >> >> if >> >> >> >> > you store them in such kind of database you *may* have a sparse >> >> >> >> > representation -by the way there may be some solutions for it. >> If >>
Re: Pointing to Hbase for Docuements or Directly Saving Documents at Hbase
Thanks again for your answer. If I find any document about such comparisons that I would like to read. By the way, is there any advantage for using Lucene instead of anything else as like that: Using Lucene is naturally supported at Solr and if I use anything else I may face with some compatibility problems or communicating issues? 2013/4/17 Otis Gospodnetic > People do use other data stores to retrieve data sometimes. e.g. Mongo > is popular for that. Like I hinted in another email, I wouldn't > necessarily recommend this for common cases. Don't do it unless you > really know you need it. Otherwise, just store in Solr. > > Otis > -- > Solr & ElasticSearch Support > http://sematext.com/ > > > > > > On Tue, Apr 16, 2013 at 5:32 PM, Furkan KAMACI > wrote: > > Hi Otis and Jack; > > > > I have made a research about highlights and debugged code. I see that > > highlight are query dependent and not stored. Why Solr uses Lucene for > > storing text, I mean i.e. content of a web page. Is there any comparison > > about to store texts at Hbase or any other databases versus Lucene. > > > > Also I want to learn that is there anybody who has used anything else > from > > Lucene to store text of document at our solr user list? > > > > 2013/4/11 Otis Gospodnetic > > > >> Source code is your best bet. Wiki has info about how to use it, but > >> not how highlighting is implemented. But you don't need to understand > >> the implementation details to understand that they are dynamic, > >> computed specifically for each query for each matching document, so > >> you cannot store them anywhere ahead of time. > >> > >> Otis > >> -- > >> Solr & ElasticSearch Support > >> http://sematext.com/ > >> > >> > >> > >> > >> > >> On Thu, Apr 11, 2013 at 11:22 AM, Furkan KAMACI > > >> wrote: > >> > Hi Otis; > >> > > >> > It seems that I should read more about highlights. Is there any where > >> that > >> > explains in detail how highlights are generated at Solr? > >> > > >> > 2013/4/11 Otis Gospodnetic > >> > > >> >> Hi, > >> >> > >> >> You can't store highlights ahead of time because they are query > >> >> dependent. You could store documents in HBase and use Solr just for > >> >> indexing. Is that what you want to do? If so, a custom > >> >> SearchComponent executed after QueryComponent could fetch data from > >> >> external store like HBase. I'm not sure if I'd recommend that. > >> >> > >> >> Otis > >> >> -- > >> >> Solr & ElasticSearch Support > >> >> http://sematext.com/ > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> On Thu, Apr 11, 2013 at 10:01 AM, Furkan KAMACI < > furkankam...@gmail.com > >> > > >> >> wrote: > >> >> > Actually I don't think to store documents at Solr. I want to store > >> just > >> >> > highlights (snippets) at Hbase and I want to retrieve them from > Hbase > >> >> when > >> >> > needed. > >> >> > What do you think about separating just highlights from Solr and > >> storing > >> >> > them into Hbase at Solrclod. By the way if you explain at which > >> process > >> >> and > >> >> > how highlights are genareted at Solr you are welcome. > >> >> > > >> >> > > >> >> > 2013/4/9 Otis Gospodnetic > >> >> > > >> >> >> You may also be interested in looking at things like solrbase (on > >> >> Github). > >> >> >> > >> >> >> Otis > >> >> >> -- > >> >> >> Solr & ElasticSearch Support > >> >> >> http://sematext.com/ > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> On Sat, Apr 6, 2013 at 6:01 PM, Furkan KAMACI < > >> furkankam...@gmail.com> > >> >> >> wrote: > >> >> >> > Hi; > >> >> >> > > >> >> >> > First of all should mention that I am new to Solr and making a > >> >> research > >> >> >> > about it. What I am trying to do that I will crawl some websites > >> with > >> >> >> Nutch > >> >> >> > and then I will index them with Solr. (Nutch 2.1, Solr-SolrCloud > >> 4.2 ) > >> >> >> > > >> >> >> > I wonder about something. I have a cloud of machines that crawls > >> >> websites > >> >> >> > and stores that documents. Then I send that documents into > >> SolrCloud. > >> >> >> Solr > >> >> >> > indexes that documents and generates indexes and save them. I > know > >> >> that > >> >> >> > from Information Retrieval theory: it *may* not be efficient to > >> store > >> >> >> > indexes at a NoSQL database (they are something like linked > lists > >> and > >> >> if > >> >> >> > you store them in such kind of database you *may* have a sparse > >> >> >> > representation -by the way there may be some solutions for it. > If > >> you > >> >> >> > explain them you are welcome.) > >> >> >> > > >> >> >> > However Solr stores some documents too (i.e. highlights) So some > >> of my > >> >> >> > documents will be doubled somehow. If I consider that I will > have > >> many > >> >> >> > documents, that dobuled documents may cause a problem for me. > So is > >> >> there > >> >> >> > any way not storing that documents at Solr and pointing to them > at > >> >> >> > Hbase(where I save my crawled documents) or instead o
Re: Pointing to Hbase for Docuements or Directly Saving Documents at Hbase
People do use other data stores to retrieve data sometimes. e.g. Mongo is popular for that. Like I hinted in another email, I wouldn't necessarily recommend this for common cases. Don't do it unless you really know you need it. Otherwise, just store in Solr. Otis -- Solr & ElasticSearch Support http://sematext.com/ On Tue, Apr 16, 2013 at 5:32 PM, Furkan KAMACI wrote: > Hi Otis and Jack; > > I have made a research about highlights and debugged code. I see that > highlight are query dependent and not stored. Why Solr uses Lucene for > storing text, I mean i.e. content of a web page. Is there any comparison > about to store texts at Hbase or any other databases versus Lucene. > > Also I want to learn that is there anybody who has used anything else from > Lucene to store text of document at our solr user list? > > 2013/4/11 Otis Gospodnetic > >> Source code is your best bet. Wiki has info about how to use it, but >> not how highlighting is implemented. But you don't need to understand >> the implementation details to understand that they are dynamic, >> computed specifically for each query for each matching document, so >> you cannot store them anywhere ahead of time. >> >> Otis >> -- >> Solr & ElasticSearch Support >> http://sematext.com/ >> >> >> >> >> >> On Thu, Apr 11, 2013 at 11:22 AM, Furkan KAMACI >> wrote: >> > Hi Otis; >> > >> > It seems that I should read more about highlights. Is there any where >> that >> > explains in detail how highlights are generated at Solr? >> > >> > 2013/4/11 Otis Gospodnetic >> > >> >> Hi, >> >> >> >> You can't store highlights ahead of time because they are query >> >> dependent. You could store documents in HBase and use Solr just for >> >> indexing. Is that what you want to do? If so, a custom >> >> SearchComponent executed after QueryComponent could fetch data from >> >> external store like HBase. I'm not sure if I'd recommend that. >> >> >> >> Otis >> >> -- >> >> Solr & ElasticSearch Support >> >> http://sematext.com/ >> >> >> >> >> >> >> >> >> >> >> >> On Thu, Apr 11, 2013 at 10:01 AM, Furkan KAMACI > > >> >> wrote: >> >> > Actually I don't think to store documents at Solr. I want to store >> just >> >> > highlights (snippets) at Hbase and I want to retrieve them from Hbase >> >> when >> >> > needed. >> >> > What do you think about separating just highlights from Solr and >> storing >> >> > them into Hbase at Solrclod. By the way if you explain at which >> process >> >> and >> >> > how highlights are genareted at Solr you are welcome. >> >> > >> >> > >> >> > 2013/4/9 Otis Gospodnetic >> >> > >> >> >> You may also be interested in looking at things like solrbase (on >> >> Github). >> >> >> >> >> >> Otis >> >> >> -- >> >> >> Solr & ElasticSearch Support >> >> >> http://sematext.com/ >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Sat, Apr 6, 2013 at 6:01 PM, Furkan KAMACI < >> furkankam...@gmail.com> >> >> >> wrote: >> >> >> > Hi; >> >> >> > >> >> >> > First of all should mention that I am new to Solr and making a >> >> research >> >> >> > about it. What I am trying to do that I will crawl some websites >> with >> >> >> Nutch >> >> >> > and then I will index them with Solr. (Nutch 2.1, Solr-SolrCloud >> 4.2 ) >> >> >> > >> >> >> > I wonder about something. I have a cloud of machines that crawls >> >> websites >> >> >> > and stores that documents. Then I send that documents into >> SolrCloud. >> >> >> Solr >> >> >> > indexes that documents and generates indexes and save them. I know >> >> that >> >> >> > from Information Retrieval theory: it *may* not be efficient to >> store >> >> >> > indexes at a NoSQL database (they are something like linked lists >> and >> >> if >> >> >> > you store them in such kind of database you *may* have a sparse >> >> >> > representation -by the way there may be some solutions for it. If >> you >> >> >> > explain them you are welcome.) >> >> >> > >> >> >> > However Solr stores some documents too (i.e. highlights) So some >> of my >> >> >> > documents will be doubled somehow. If I consider that I will have >> many >> >> >> > documents, that dobuled documents may cause a problem for me. So is >> >> there >> >> >> > any way not storing that documents at Solr and pointing to them at >> >> >> > Hbase(where I save my crawled documents) or instead of pointing >> >> directly >> >> >> > storing them at Hbase (is it efficient or not)? >> >> >> >> >> >>
Re: Pointing to Hbase for Docuements or Directly Saving Documents at Hbase
Hi Otis and Jack; I have made a research about highlights and debugged code. I see that highlight are query dependent and not stored. Why Solr uses Lucene for storing text, I mean i.e. content of a web page. Is there any comparison about to store texts at Hbase or any other databases versus Lucene. Also I want to learn that is there anybody who has used anything else from Lucene to store text of document at our solr user list? 2013/4/11 Otis Gospodnetic > Source code is your best bet. Wiki has info about how to use it, but > not how highlighting is implemented. But you don't need to understand > the implementation details to understand that they are dynamic, > computed specifically for each query for each matching document, so > you cannot store them anywhere ahead of time. > > Otis > -- > Solr & ElasticSearch Support > http://sematext.com/ > > > > > > On Thu, Apr 11, 2013 at 11:22 AM, Furkan KAMACI > wrote: > > Hi Otis; > > > > It seems that I should read more about highlights. Is there any where > that > > explains in detail how highlights are generated at Solr? > > > > 2013/4/11 Otis Gospodnetic > > > >> Hi, > >> > >> You can't store highlights ahead of time because they are query > >> dependent. You could store documents in HBase and use Solr just for > >> indexing. Is that what you want to do? If so, a custom > >> SearchComponent executed after QueryComponent could fetch data from > >> external store like HBase. I'm not sure if I'd recommend that. > >> > >> Otis > >> -- > >> Solr & ElasticSearch Support > >> http://sematext.com/ > >> > >> > >> > >> > >> > >> On Thu, Apr 11, 2013 at 10:01 AM, Furkan KAMACI > > >> wrote: > >> > Actually I don't think to store documents at Solr. I want to store > just > >> > highlights (snippets) at Hbase and I want to retrieve them from Hbase > >> when > >> > needed. > >> > What do you think about separating just highlights from Solr and > storing > >> > them into Hbase at Solrclod. By the way if you explain at which > process > >> and > >> > how highlights are genareted at Solr you are welcome. > >> > > >> > > >> > 2013/4/9 Otis Gospodnetic > >> > > >> >> You may also be interested in looking at things like solrbase (on > >> Github). > >> >> > >> >> Otis > >> >> -- > >> >> Solr & ElasticSearch Support > >> >> http://sematext.com/ > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> On Sat, Apr 6, 2013 at 6:01 PM, Furkan KAMACI < > furkankam...@gmail.com> > >> >> wrote: > >> >> > Hi; > >> >> > > >> >> > First of all should mention that I am new to Solr and making a > >> research > >> >> > about it. What I am trying to do that I will crawl some websites > with > >> >> Nutch > >> >> > and then I will index them with Solr. (Nutch 2.1, Solr-SolrCloud > 4.2 ) > >> >> > > >> >> > I wonder about something. I have a cloud of machines that crawls > >> websites > >> >> > and stores that documents. Then I send that documents into > SolrCloud. > >> >> Solr > >> >> > indexes that documents and generates indexes and save them. I know > >> that > >> >> > from Information Retrieval theory: it *may* not be efficient to > store > >> >> > indexes at a NoSQL database (they are something like linked lists > and > >> if > >> >> > you store them in such kind of database you *may* have a sparse > >> >> > representation -by the way there may be some solutions for it. If > you > >> >> > explain them you are welcome.) > >> >> > > >> >> > However Solr stores some documents too (i.e. highlights) So some > of my > >> >> > documents will be doubled somehow. If I consider that I will have > many > >> >> > documents, that dobuled documents may cause a problem for me. So is > >> there > >> >> > any way not storing that documents at Solr and pointing to them at > >> >> > Hbase(where I save my crawled documents) or instead of pointing > >> directly > >> >> > storing them at Hbase (is it efficient or not)? > >> >> > >> >
Re: Pointing to Hbase for Docuements or Directly Saving Documents at Hbase
Source code is your best bet. Wiki has info about how to use it, but not how highlighting is implemented. But you don't need to understand the implementation details to understand that they are dynamic, computed specifically for each query for each matching document, so you cannot store them anywhere ahead of time. Otis -- Solr & ElasticSearch Support http://sematext.com/ On Thu, Apr 11, 2013 at 11:22 AM, Furkan KAMACI wrote: > Hi Otis; > > It seems that I should read more about highlights. Is there any where that > explains in detail how highlights are generated at Solr? > > 2013/4/11 Otis Gospodnetic > >> Hi, >> >> You can't store highlights ahead of time because they are query >> dependent. You could store documents in HBase and use Solr just for >> indexing. Is that what you want to do? If so, a custom >> SearchComponent executed after QueryComponent could fetch data from >> external store like HBase. I'm not sure if I'd recommend that. >> >> Otis >> -- >> Solr & ElasticSearch Support >> http://sematext.com/ >> >> >> >> >> >> On Thu, Apr 11, 2013 at 10:01 AM, Furkan KAMACI >> wrote: >> > Actually I don't think to store documents at Solr. I want to store just >> > highlights (snippets) at Hbase and I want to retrieve them from Hbase >> when >> > needed. >> > What do you think about separating just highlights from Solr and storing >> > them into Hbase at Solrclod. By the way if you explain at which process >> and >> > how highlights are genareted at Solr you are welcome. >> > >> > >> > 2013/4/9 Otis Gospodnetic >> > >> >> You may also be interested in looking at things like solrbase (on >> Github). >> >> >> >> Otis >> >> -- >> >> Solr & ElasticSearch Support >> >> http://sematext.com/ >> >> >> >> >> >> >> >> >> >> >> >> On Sat, Apr 6, 2013 at 6:01 PM, Furkan KAMACI >> >> wrote: >> >> > Hi; >> >> > >> >> > First of all should mention that I am new to Solr and making a >> research >> >> > about it. What I am trying to do that I will crawl some websites with >> >> Nutch >> >> > and then I will index them with Solr. (Nutch 2.1, Solr-SolrCloud 4.2 ) >> >> > >> >> > I wonder about something. I have a cloud of machines that crawls >> websites >> >> > and stores that documents. Then I send that documents into SolrCloud. >> >> Solr >> >> > indexes that documents and generates indexes and save them. I know >> that >> >> > from Information Retrieval theory: it *may* not be efficient to store >> >> > indexes at a NoSQL database (they are something like linked lists and >> if >> >> > you store them in such kind of database you *may* have a sparse >> >> > representation -by the way there may be some solutions for it. If you >> >> > explain them you are welcome.) >> >> > >> >> > However Solr stores some documents too (i.e. highlights) So some of my >> >> > documents will be doubled somehow. If I consider that I will have many >> >> > documents, that dobuled documents may cause a problem for me. So is >> there >> >> > any way not storing that documents at Solr and pointing to them at >> >> > Hbase(where I save my crawled documents) or instead of pointing >> directly >> >> > storing them at Hbase (is it efficient or not)? >> >> >>
Re: Pointing to Hbase for Docuements or Directly Saving Documents at Hbase
Hi Otis; It seems that I should read more about highlights. Is there any where that explains in detail how highlights are generated at Solr? 2013/4/11 Otis Gospodnetic > Hi, > > You can't store highlights ahead of time because they are query > dependent. You could store documents in HBase and use Solr just for > indexing. Is that what you want to do? If so, a custom > SearchComponent executed after QueryComponent could fetch data from > external store like HBase. I'm not sure if I'd recommend that. > > Otis > -- > Solr & ElasticSearch Support > http://sematext.com/ > > > > > > On Thu, Apr 11, 2013 at 10:01 AM, Furkan KAMACI > wrote: > > Actually I don't think to store documents at Solr. I want to store just > > highlights (snippets) at Hbase and I want to retrieve them from Hbase > when > > needed. > > What do you think about separating just highlights from Solr and storing > > them into Hbase at Solrclod. By the way if you explain at which process > and > > how highlights are genareted at Solr you are welcome. > > > > > > 2013/4/9 Otis Gospodnetic > > > >> You may also be interested in looking at things like solrbase (on > Github). > >> > >> Otis > >> -- > >> Solr & ElasticSearch Support > >> http://sematext.com/ > >> > >> > >> > >> > >> > >> On Sat, Apr 6, 2013 at 6:01 PM, Furkan KAMACI > >> wrote: > >> > Hi; > >> > > >> > First of all should mention that I am new to Solr and making a > research > >> > about it. What I am trying to do that I will crawl some websites with > >> Nutch > >> > and then I will index them with Solr. (Nutch 2.1, Solr-SolrCloud 4.2 ) > >> > > >> > I wonder about something. I have a cloud of machines that crawls > websites > >> > and stores that documents. Then I send that documents into SolrCloud. > >> Solr > >> > indexes that documents and generates indexes and save them. I know > that > >> > from Information Retrieval theory: it *may* not be efficient to store > >> > indexes at a NoSQL database (they are something like linked lists and > if > >> > you store them in such kind of database you *may* have a sparse > >> > representation -by the way there may be some solutions for it. If you > >> > explain them you are welcome.) > >> > > >> > However Solr stores some documents too (i.e. highlights) So some of my > >> > documents will be doubled somehow. If I consider that I will have many > >> > documents, that dobuled documents may cause a problem for me. So is > there > >> > any way not storing that documents at Solr and pointing to them at > >> > Hbase(where I save my crawled documents) or instead of pointing > directly > >> > storing them at Hbase (is it efficient or not)? > >> >
Re: Pointing to Hbase for Docuements or Directly Saving Documents at Hbase
Hi, You can't store highlights ahead of time because they are query dependent. You could store documents in HBase and use Solr just for indexing. Is that what you want to do? If so, a custom SearchComponent executed after QueryComponent could fetch data from external store like HBase. I'm not sure if I'd recommend that. Otis -- Solr & ElasticSearch Support http://sematext.com/ On Thu, Apr 11, 2013 at 10:01 AM, Furkan KAMACI wrote: > Actually I don't think to store documents at Solr. I want to store just > highlights (snippets) at Hbase and I want to retrieve them from Hbase when > needed. > What do you think about separating just highlights from Solr and storing > them into Hbase at Solrclod. By the way if you explain at which process and > how highlights are genareted at Solr you are welcome. > > > 2013/4/9 Otis Gospodnetic > >> You may also be interested in looking at things like solrbase (on Github). >> >> Otis >> -- >> Solr & ElasticSearch Support >> http://sematext.com/ >> >> >> >> >> >> On Sat, Apr 6, 2013 at 6:01 PM, Furkan KAMACI >> wrote: >> > Hi; >> > >> > First of all should mention that I am new to Solr and making a research >> > about it. What I am trying to do that I will crawl some websites with >> Nutch >> > and then I will index them with Solr. (Nutch 2.1, Solr-SolrCloud 4.2 ) >> > >> > I wonder about something. I have a cloud of machines that crawls websites >> > and stores that documents. Then I send that documents into SolrCloud. >> Solr >> > indexes that documents and generates indexes and save them. I know that >> > from Information Retrieval theory: it *may* not be efficient to store >> > indexes at a NoSQL database (they are something like linked lists and if >> > you store them in such kind of database you *may* have a sparse >> > representation -by the way there may be some solutions for it. If you >> > explain them you are welcome.) >> > >> > However Solr stores some documents too (i.e. highlights) So some of my >> > documents will be doubled somehow. If I consider that I will have many >> > documents, that dobuled documents may cause a problem for me. So is there >> > any way not storing that documents at Solr and pointing to them at >> > Hbase(where I save my crawled documents) or instead of pointing directly >> > storing them at Hbase (is it efficient or not)? >>
Re: Pointing to Hbase for Docuements or Directly Saving Documents at Hbase
Actually I don't think to store documents at Solr. I want to store just highlights (snippets) at Hbase and I want to retrieve them from Hbase when needed. What do you think about separating just highlights from Solr and storing them into Hbase at Solrclod. By the way if you explain at which process and how highlights are genareted at Solr you are welcome. 2013/4/9 Otis Gospodnetic > You may also be interested in looking at things like solrbase (on Github). > > Otis > -- > Solr & ElasticSearch Support > http://sematext.com/ > > > > > > On Sat, Apr 6, 2013 at 6:01 PM, Furkan KAMACI > wrote: > > Hi; > > > > First of all should mention that I am new to Solr and making a research > > about it. What I am trying to do that I will crawl some websites with > Nutch > > and then I will index them with Solr. (Nutch 2.1, Solr-SolrCloud 4.2 ) > > > > I wonder about something. I have a cloud of machines that crawls websites > > and stores that documents. Then I send that documents into SolrCloud. > Solr > > indexes that documents and generates indexes and save them. I know that > > from Information Retrieval theory: it *may* not be efficient to store > > indexes at a NoSQL database (they are something like linked lists and if > > you store them in such kind of database you *may* have a sparse > > representation -by the way there may be some solutions for it. If you > > explain them you are welcome.) > > > > However Solr stores some documents too (i.e. highlights) So some of my > > documents will be doubled somehow. If I consider that I will have many > > documents, that dobuled documents may cause a problem for me. So is there > > any way not storing that documents at Solr and pointing to them at > > Hbase(where I save my crawled documents) or instead of pointing directly > > storing them at Hbase (is it efficient or not)? >
Re: Pointing to Hbase for Docuements or Directly Saving Documents at Hbase
You may also be interested in looking at things like solrbase (on Github). Otis -- Solr & ElasticSearch Support http://sematext.com/ On Sat, Apr 6, 2013 at 6:01 PM, Furkan KAMACI wrote: > Hi; > > First of all should mention that I am new to Solr and making a research > about it. What I am trying to do that I will crawl some websites with Nutch > and then I will index them with Solr. (Nutch 2.1, Solr-SolrCloud 4.2 ) > > I wonder about something. I have a cloud of machines that crawls websites > and stores that documents. Then I send that documents into SolrCloud. Solr > indexes that documents and generates indexes and save them. I know that > from Information Retrieval theory: it *may* not be efficient to store > indexes at a NoSQL database (they are something like linked lists and if > you store them in such kind of database you *may* have a sparse > representation -by the way there may be some solutions for it. If you > explain them you are welcome.) > > However Solr stores some documents too (i.e. highlights) So some of my > documents will be doubled somehow. If I consider that I will have many > documents, that dobuled documents may cause a problem for me. So is there > any way not storing that documents at Solr and pointing to them at > Hbase(where I save my crawled documents) or instead of pointing directly > storing them at Hbase (is it efficient or not)?
Re: Pointing to Hbase for Docuements or Directly Saving Documents at Hbase
Solr would not be storing the original source form of the documents in any case. Whether you use Tika or SolrCell, only the text stream of the content and the metadata would ever get indexed or stored in Solr. Solr completely decouples "indexing" and "storing" of data values. If you don't want to "store" the text stream in Solr, then don't. If you want to "store" the original blob of the source documents in some other data store, that's your choice. You can store the original URL or a document ID or URL for some alternate document store. That's your choice to make. Solr in no way forces you one way or the other. And whether that URL or document ID refers to HBase or a web site, doesn't matter to Solr either. Whether or not you could more efficiently store the original document bytes in Lucene/Solr DocValues vs. HBase is a separate matter - I don't know one way or the other whether DocValues help or not. Or whether a Solr BinaryField might be suitable for store the original bytes of a document (but without indexing the bytes.) In other words, maybe you could just use two separate Solr servers, one for text index and metadata store, and the other for raw store of the original document bytes. -- Jack Krupansky -Original Message- From: Furkan KAMACI Sent: Saturday, April 06, 2013 6:01 PM To: solr-user@lucene.apache.org Subject: Pointing to Hbase for Docuements or Directly Saving Documents at Hbase Hi; First of all should mention that I am new to Solr and making a research about it. What I am trying to do that I will crawl some websites with Nutch and then I will index them with Solr. (Nutch 2.1, Solr-SolrCloud 4.2 ) I wonder about something. I have a cloud of machines that crawls websites and stores that documents. Then I send that documents into SolrCloud. Solr indexes that documents and generates indexes and save them. I know that from Information Retrieval theory: it *may* not be efficient to store indexes at a NoSQL database (they are something like linked lists and if you store them in such kind of database you *may* have a sparse representation -by the way there may be some solutions for it. If you explain them you are welcome.) However Solr stores some documents too (i.e. highlights) So some of my documents will be doubled somehow. If I consider that I will have many documents, that dobuled documents may cause a problem for me. So is there any way not storing that documents at Solr and pointing to them at Hbase(where I save my crawled documents) or instead of pointing directly storing them at Hbase (is it efficient or not)?
Pointing to Hbase for Docuements or Directly Saving Documents at Hbase
Hi; First of all should mention that I am new to Solr and making a research about it. What I am trying to do that I will crawl some websites with Nutch and then I will index them with Solr. (Nutch 2.1, Solr-SolrCloud 4.2 ) I wonder about something. I have a cloud of machines that crawls websites and stores that documents. Then I send that documents into SolrCloud. Solr indexes that documents and generates indexes and save them. I know that from Information Retrieval theory: it *may* not be efficient to store indexes at a NoSQL database (they are something like linked lists and if you store them in such kind of database you *may* have a sparse representation -by the way there may be some solutions for it. If you explain them you are welcome.) However Solr stores some documents too (i.e. highlights) So some of my documents will be doubled somehow. If I consider that I will have many documents, that dobuled documents may cause a problem for me. So is there any way not storing that documents at Solr and pointing to them at Hbase(where I save my crawled documents) or instead of pointing directly storing them at Hbase (is it efficient or not)?
