RE: SolrCloud Indexing question
Thank you so much for the suggestion, Is the same recommended for querying too i found it very slow when i do query using clousolrserver Kalyan Date: Tue, 6 Aug 2013 13:25:37 -0600 From: s...@elyograg.org To: solr-user@lucene.apache.org Subject: Re: SolrCloud Indexing question On 8/6/2013 12:55 PM, Kalyan Kuram wrote: Hi AllI need suggestion on how to send indexing commands to 2 different solr server,Basically i want to mirror my index,here is the scenarioi have 2 cluster, each cluster has one master and 2 slaves with external zookeeper in the fronti need suggestion on what solr api class i should use to send indexing commands to 2 masters,will LBHttpSolrServer do the indexing or is this only used for querying If there is a better approach please suggest Kalyan If you're using zookeeper, then your index is SolrCloud, and you don't have masters and slaves. The traditional master/slave replication model does not apply to SolrCloud. With SolrCloud, there is no need to have two independent clusters. If a server dies, the other servers in the cloud will keep the cluster operational. When you bring the dead server back with the proper config, it will automatically be synchronized with the cluster. For a Java program with SolrJ, use a CloudSolrServer object for each cluster. The constructor for CloudSolrServer accepts the same zkHost parameter that you give to each Solr server when starting in SolrCloud mode. You cannot index to independent clusters at the same time through one object - if they truly are independent SolrCloud installs, you have to manage updates to both of them independently. Thanks, Shawn
SolrCloud Indexing question
Hi AllI need suggestion on how to send indexing commands to 2 different solr server,Basically i want to mirror my index,here is the scenarioi have 2 cluster, each cluster has one master and 2 slaves with external zookeeper in the fronti need suggestion on what solr api class i should use to send indexing commands to 2 masters,will LBHttpSolrServer do the indexing or is this only used for querying If there is a better approach please suggest Kalyan
URLDatasource Authentication
Hi I am trying to access xml files which are stored in our cms,how do i pass username/passwd to dih so i can get all xml files its throwing exception java.io.IOException: Server returned HTTP response code: 401 for URL: http://admin:admin...@cms1.zinio.com.com/PATH/articles/100850443.xml Is there anything else i can use to pull data from cms Kalyan
Solr Shards and ZooKeeper
Hi allI am trying to configure external zookeeper with solr instances which has to have 2 shards.I tried the introductory solrcloud wiki page and lucidworks solrcloud page it works just fine(embedded zookeeper),The problem i have is start solr with 2 shards when i have external zookeeper,i cant get solr to start with 2 shards Steps followed1.start zookeeper 2.Start First solr instance with args nohup java -Dbootstrap_confdir=./solr/Articles/conf -Dcollection.configName=Articles -DzkHost=dev-core-solr1:2181 -DnumShards=2 -jar start.jar 3.Start the second solr instancenohup java -DzkHost=dev-core-solr1:2181 -jar start.jar And when i navigate to the cloud page i see shard1 connected to 2 solr instances instead of shard1 connected to solr1instance1 and shard2 to solrinstance2 Behaviour is not the same when i start embedded zookeeper from solr,i see shard1 connected to solrinstance1 and shard2 connected to solrinstance2 Am i doing something wrong or have i missed any steps.Please help Kalyan
RE: Solr Shards and ZooKeeper
It worked ,i followed steps only difference i erased everything and started from scratch again From: kalyan.ku...@live.com To: solr-user@lucene.apache.org Subject: Solr Shards and ZooKeeper Date: Wed, 12 Jun 2013 14:51:41 -0400 Hi allI am trying to configure external zookeeper with solr instances which has to have 2 shards.I tried the introductory solrcloud wiki page and lucidworks solrcloud page it works just fine(embedded zookeeper),The problem i have is start solr with 2 shards when i have external zookeeper,i cant get solr to start with 2 shards Steps followed1.start zookeeper 2.Start First solr instance with args nohup java -Dbootstrap_confdir=./solr/Articles/conf -Dcollection.configName=Articles -DzkHost=dev-core-solr1:2181 -DnumShards=2 -jar start.jar 3.Start the second solr instancenohup java -DzkHost=dev-core-solr1:2181 -jar start.jar And when i navigate to the cloud page i see shard1 connected to 2 solr instances instead of shard1 connected to solr1instance1 and shard2 to solrinstance2 Behaviour is not the same when i start embedded zookeeper from solr,i see shard1 connected to solrinstance1 and shard2 connected to solrinstance2 Am i doing something wrong or have i missed any steps.Please help Kalyan
RE: Strip HTML Tags and Store
Thanks it worked..!!
From: j...@basetechnology.com
To: solr-user@lucene.apache.org
Subject: Re: Strip HTML Tags and Store
Date: Thu, 30 May 2013 22:53:37 -0400
Update Request Processors to the rescue again. Namely, the HTML Strip Field
Update processor:
Add to your solrconfig:
updateRequestProcessorChain name=html-strip-features
processor class=solr.HTMLStripFieldUpdateProcessorFactory
str name=fieldNamefeatures/str
/processor
processor class=solr.LogUpdateProcessorFactory /
processor class=solr.RunUpdateProcessorFactory /
/updateRequestProcessorChain
See:
http://lucene.apache.org/solr/4_3_0/solr-core/org/apache/solr/update/processor/HTMLStripFieldUpdateProcessorFactory.html
Index content:
curl
http://localhost:8983/solr/update?commit=trueupdate.chain=html-strip-features;
\
-H 'Content-type:application/json' -d '
[{id: doc-1,
title: lt;Hello Worldgt;,
features: pThis is a atest/a line gt;.,
other_t: pOther btext/b/p,
more_t: Some bmore itext/i./b The end}]'
Results:
id:doc-1,
title:[lt;Hello Worldgt;],
features:[\nThis is a test line .],
other_t:pOther btext/b/p,
more_t:Some bmore itext/i./b The end,
That stripped the HTML only from the features field, and expanded the
named character entity as well.
Add multiple str for multiple fields, or use fieldRegex, or... some
other options. See:
http://lucene.apache.org/solr/4_3_0/solr-core/org/apache/solr/update/processor/FieldMutatingUpdateProcessorFactory.html
-- Jack Krupansky
-Original Message-
From: Kalyan Kuram
Sent: Thursday, May 30, 2013 8:18 PM
To: solr-user@lucene.apache.org
Subject: Strip HTML Tags and Store
Hi AllI am trying to understand what gets stored when i configure a field
indexed and stored for example i have this in my schema.xmlfield
name=articleBody type=text_general indexed=true stored=true /and
fieldType name=text_general class=solr.TextField
positionIncrementGap=100
analyzer type=index
tokenizer class=solr.StandardTokenizerFactory/
charFilter class=solr.HTMLStripCharFilterFactory/
filter class=solr.StopFilterFactory ignoreCase=true
words=stopwords.txt enablePositionIncrements=true /
filter class=solr.LowerCaseFilterFactory/
/analyzer
analyzer type=query
tokenizer class=solr.StandardTokenizerFactory/
filter class=solr.StopFilterFactory ignoreCase=true
words=stopwords.txt enablePositionIncrements=true /
filter class=solr.SynonymFilterFactory synonyms=synonyms.txt
ignoreCase=true expand=true/
filter class=solr.LowerCaseFilterFactory/
/analyzer
/fieldType
I was expecting that solr will index store html strip content when i
invoke query i get some thing like this str
name=articleBodyxhtml:h1xhtml:bSouth African Miners Are Trapped by
Debt/xhtml:b/xhtml:h1 xhtml:pxhtml:b▸ A surge in high-interest
lending contributes to mine violence/xhtml:b/xhtml:p xhtml:pxhtml:b▸
At least one bank “may have reckless lending problems”/xhtml:b/xhtml:p
xhtml:pIn 2008, platinum miner James Ntseane borrowed 8,000 rand ($886)
from xhtml:bAfrican Bank Investments/xhtml:b to pay for his
grandmother's funeral. Soon after, he took out two more loans, totaling
10,000 rand, for a sofa and house extension. Four years later he owes at
least 30,515 rand, according to text messages he gets from African Bank,
South Africa's biggest provider of unsecured loans. Under a court-ordered
payment plan, his employer garnishes about 13 percent of his monthly
12,600-rand salary for the lender. He doesn't know how much interest he's
paying. “They are taking too much money,” says Ntseane, 41./xhtml:p
xhtml:pNtseane is one of more than 9 million South Africans mired in debt.
African Bank, xhtml:bBayport Financial Services, Capitec Bank
Holdings/xhtml:b, and other firms have led a boom in unsecured lending,
charging interest as high as 80 percent a year, as is allowed there. Last
year a series of strikes led to at least 46 deaths, the country's worst
mining violence since the end of apartheid. “One of the contributing factors
to all of these strikes has been this surge in unsecured lending,” says Mike
Schussler, chief economist at the research group a
href=http://economists.co.za/;Economists.co.za/a, echoing an October
statement by Trade and Industry Minister Rob Davies./xhtml:p xhtml:pThe
value of consumer loans not backed by assets such as homes rose 39 percent
in the year through September, to 140 billion rand, reports the National
Credit Regulator. The loans made up 10 percent of consumer credit on Sept.
30, up from 8 percent a year earlier. In November, South Africa's National
Treasury and the Banking Association of South Africa agreed to review
lending affordability rules, improve client education, and reduce wage
garnishing after the number of people with bad credit rose
Strip HTML Tags and Store
Hi AllI am trying to understand what gets stored when i configure a field indexed and stored for example i have this in my schema.xmlfield name=articleBody type=text_general indexed=true stored=true /and fieldType name=text_general class=solr.TextField positionIncrementGap=100 analyzer type=index tokenizer class=solr.StandardTokenizerFactory/ charFilter class=solr.HTMLStripCharFilterFactory/ filter class=solr.StopFilterFactory ignoreCase=true words=stopwords.txt enablePositionIncrements=true / filter class=solr.LowerCaseFilterFactory/ /analyzer analyzer type=query tokenizer class=solr.StandardTokenizerFactory/ filter class=solr.StopFilterFactory ignoreCase=true words=stopwords.txt enablePositionIncrements=true / filter class=solr.SynonymFilterFactory synonyms=synonyms.txt ignoreCase=true expand=true/ filter class=solr.LowerCaseFilterFactory/ /analyzer /fieldType I was expecting that solr will index store html strip content when i invoke query i get some thing like this str name=articleBodyxhtml:h1xhtml:bSouth African Miners Are Trapped by Debt/xhtml:b/xhtml:h1 xhtml:pxhtml:b▸ A surge in high-interest lending contributes to mine violence/xhtml:b/xhtml:p xhtml:pxhtml:b▸ At least one bank “may have reckless lending problems”/xhtml:b/xhtml:p xhtml:pIn 2008, platinum miner James Ntseane borrowed 8,000 rand ($886) from xhtml:bAfrican Bank Investments/xhtml:b to pay for his grandmother's funeral. Soon after, he took out two more loans, totaling 10,000 rand, for a sofa and house extension. Four years later he owes at least 30,515 rand, according to text messages he gets from African Bank, South Africa's biggest provider of unsecured loans. Under a court-ordered payment plan, his employer garnishes about 13 percent of his monthly 12,600-rand salary for the lender. He doesn't know how much interest he's paying. “They are taking too much money,” says Ntseane, 41./xhtml:p xhtml:pNtseane is one of more than 9 million South Africans mired in debt. African Bank, xhtml:bBayport Financial Services, Capitec Bank Holdings/xhtml:b, and other firms have led a boom in unsecured lending, charging interest as high as 80 percent a year, as is allowed there. Last year a series of strikes led to at least 46 deaths, the country's worst mining violence since the end of apartheid. “One of the contributing factors to all of these strikes has been this surge in unsecured lending,” says Mike Schussler, chief economist at the research group a href=http://economists.co.za/;Economists.co.za/a, echoing an October statement by Trade and Industry Minister Rob Davies./xhtml:p xhtml:pThe value of consumer loans not backed by assets such as homes rose 39 percent in the year through September, to 140 billion rand, reports the National Credit Regulator. The loans made up 10 percent of consumer credit on Sept. 30, up from 8 percent a year earlier. In November, South Africa's National Treasury and the Banking Association of South Africa agreed to review lending affordability rules, improve client education, and reduce wage garnishing after the number of people with bad credit rose to a record. Finance Minister Pravin Gordhan called the rise “worrying” a week earlier./xhtml:p xhtml:pGeorge Roussos, an executive for central support services at African Bank, says miner Ntseane borrowed more than he claims and took out a credit card. (The bank received permission from Ntseane, who denies the bank's figures, to discuss his account with xhtml:iBloomberg Businessweek/xhtml:i.) The bank says it stopped charging interest in 2011 and has no record of Ntseane making contact after he was injured in a home robbery in 2010. “The bank attempts to communicate clearly and transparently, employing multilingual consultants,” says Roussos./xhtml:p xhtml:pSouth African lenders have re sorted to court-ordered wage garnishing in more than 3 million active cases, according to the National Debt Mediation Association, a credit industry group that provides consumer debt counseling. Kem Westdyk, chief executive of xhtml:bSummit Garnishee Solutions/xhtml:b, which helps mining companies review bank requests, says at some companies up to 15 percent of workers have wages garnished; at one, more than a quarter of those cases involve African Bank. “They may have reckless lending problems,” says Westdyk, adding that some workers have five or six garnishee orders against them./xhtml:p xhtml:pNtseane says his loan agent didn't mention garnishment when she agreed to delay his loan payments. Although Davies and the country's credit regulator have pledged to clamp down on unsecured lending, Ntseane doesn't have high hopes. “I don't know when I will stop paying,” he says./xhtml:p xhtml:p prism:class=bylinexhtml:i—Franz Wild, Mike Cohen, and Renee Bonorchis/xhtml:i/xhtml:p xhtml:pxhtml:ixhtml:bThe bottom line/xhtml:b
RE: Export Index and Re-Index XML
Thanks for the help,i could successfully export the file as csv and import it into my local box successfully ,now i have a different problem i tried to re-index the content using post.sh anc chaging URL=http://dev-core-solr1:8983/solr/ZinioArticles/update/csv this is now i see this error Before this i deleted all documents and then tried to re-index .$ sh post.sh output1.csvPosting file output1.csv to http://dev-core-solr1.zinio.com:8983/solr/ZinioArticles/update/csv?xml version=1.0 encoding=UTF-8?responselst name=responseHeaderint name=status409/intint name=QTime19/int/lstlst name=errorstr name=msgversion conflict for 100845239 expected=1432420345067864064 actual=-1/strint name=code409/int/lst/response ?xml version=1.0 encoding=UTF-8?responselst name=responseHeaderint name=status0/intint name=QTime5/int/lst/response Can somebody help me how to do this Subject: Re: Export Index and Re-Index XML From: jan@cominvent.com Date: Tue, 23 Apr 2013 15:46:36 +0200 To: solr-user@lucene.apache.org Hi, I have done this many times. First use a curl job or something to download the complete index as CSV q=*:*rows=999wt=csv Then use post.jar to push that csv into the new node. Alternatively you can query with XML and use xslt update request handler with parm tr=updateXml which is a stylesheet for indexing response XML directly. -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com Solr Training - www.solrtraining.com 23. apr. 2013 kl. 02:11 skrev Kalyan Kuram kalyan.ku...@live.com: Thank you all very much for your help.I do have field configured as stored and index,i did read the FAQ from wiki,I think SolrEntityProcessor is what i think needed.I am trying to index the data from Adobe CQ and its a push based indexing and pain to index data from a very large repository.I think i can manage this with SolrEntityProcessor for now and will think of modelling data for re-indexing purposes Kalyan From: j...@basetechnology.com To: solr-user@lucene.apache.org Subject: Re: Export Index and Re-Index XML Date: Mon, 22 Apr 2013 19:54:26 -0400 Any fields which have stored values can be read and output, but indexed-only, non-stored fields cannot be read or exported. Even if they could be, their values are post-analysis, which means that there is a good chance that they cannot be run through term analysis again. It is always best to keep a copy of your raw source data separate from the data you add to Solr. Or, at least make sure any important data is stored. In short, you need to model your data for reindexing, which is a fact of life in Solr land. -- Jack Krupansky -Original Message- From: Kalyan Kuram Sent: Monday, April 22, 2013 7:07 PM To: solr-user@lucene.apache.org Subject: Export Index and Re-Index XML Hi AllI am new to solr and i wanted to know if i can export the Index as XML and then re-index back into Solr,The reason i need to do this is i misconfigured fieldtype and to make it work i need to re-index the content Kalyan
Export Index and Re-Index XML
Hi AllI am new to solr and i wanted to know if i can export the Index as XML and then re-index back into Solr,The reason i need to do this is i misconfigured fieldtype and to make it work i need to re-index the content Kalyan
RE: Export Index and Re-Index XML
Thank you all very much for your help.I do have field configured as stored and index,i did read the FAQ from wiki,I think SolrEntityProcessor is what i think needed.I am trying to index the data from Adobe CQ and its a push based indexing and pain to index data from a very large repository.I think i can manage this with SolrEntityProcessor for now and will think of modelling data for re-indexing purposes Kalyan From: j...@basetechnology.com To: solr-user@lucene.apache.org Subject: Re: Export Index and Re-Index XML Date: Mon, 22 Apr 2013 19:54:26 -0400 Any fields which have stored values can be read and output, but indexed-only, non-stored fields cannot be read or exported. Even if they could be, their values are post-analysis, which means that there is a good chance that they cannot be run through term analysis again. It is always best to keep a copy of your raw source data separate from the data you add to Solr. Or, at least make sure any important data is stored. In short, you need to model your data for reindexing, which is a fact of life in Solr land. -- Jack Krupansky -Original Message- From: Kalyan Kuram Sent: Monday, April 22, 2013 7:07 PM To: solr-user@lucene.apache.org Subject: Export Index and Re-Index XML Hi AllI am new to solr and i wanted to know if i can export the Index as XML and then re-index back into Solr,The reason i need to do this is i misconfigured fieldtype and to make it work i need to re-index the content Kalyan
