: So I've fallen back to electronics, which has an advantage of product
: names and jargon that shows off the need for text analysis, such as
: word splitting and combining, and synonyms.
:
: Here is an example of what I might limit the fields to:
A few suggestions:
1) Demonstrating dynamic
i've been migrating any general SOLAR docs i can find in CNET's wiki to
the SOLR wiki... If anyone with access to the CNET wiki has an requests
for documents to be migrated (or sections of less general documents)
please let me know.
Even if you don't have access to CNET's wiki, let me know what
I hate to bring up something so inane, but if we are ever going to care,
now is the time to do so.
what kind of consistency do we want to try and apply to the uses of the
Solr name in terms of capitalization?
Should SOLR allways be an acronym? should it always be lowercased? ..
Should we take
: - Query Parameters for the standard query handler (nothing in the CNET wiki)
: - XML format (nothing in the CNET wiki)
The XML for add/commit/delete were documnted in Collection
Building which I imported. The XML format for search results is fairly
self documenting i think ... but yeah, we
:source = $HeadURL:$
:sourceid = $Revision:$ (aka: $LastChangedRevision:$)
:
: Revision or Id? Revision doesn't have the date, but Id has redundant
: and unneeded info.
Er, sorry ... right, $Id:$ ... i was getting SolrInfoMBean.getCvsId() in
the confused with SolrInfoMBean.getVersion()
: The downside to solr/conf, solr/data and solr/logs is that we'll have
: to jump through hoops to make them web-accessible. It'd be nice if
Solr already jumps through this hoop with get-file.jsp, so it doesn't
seem like that big of a deal (heo nly catch is that they have to be
readable by the
be run localy on the server, people
may want to configure their server with host based rules, or access
controls by path -- they should be able to set ENV variables (or command
line args) to override the entire URL.
: On Mar 23, 2006, at 4:32 PM, Chris Hostetter wrote:
: regardless
: There is a ServletContextListener that can be registered in web.xml
: as a listener. This existed in Servlet 2.3 also.
Ah .. i see, I new about listeners, i just didn't realize how varied they
are (i wasn't aware of ServletContextListener at all).
-Hoss
: [ http://issues.apache.org/jira/browse/SOLR-3?page=all ]
: The only hitch with all of this, is that i seem to have a filehandle
: leak somewhere. Or at least i think i do ... running ant legacyTest
I figured out the problem. SolrQueryRequestBase maintains a handle to the
IndexSearcher
: I've been looking into how to have multiple solr webapps, and have
: them default to looking into different spots for config (and looking
: at a different property to override that).
:
: Unfortunately, it doesn't look like there is a way to do this from the
: servlet API.
Can't we just use
: Unfortunately not the javadoc says it comes from the web.xml
:
: java.lang.String getServletContextName()
: Returns the name of this web application corresponding to
: this ServletContext as specified in the deployment descriptor for this
: web application by the display-name
: The easiest way I can think of to get that effect is to store all the
: fields so you can re-create the Document and change the field being
: updated.
My brief reading of hte GData URL Doug sent suggestes that the overall
theme is content storage -- if that's the goal, mandating that modify
: For LocalSolrQueryRequest, since it is using a Map for the arguments
: anyway, I simply added the new getParams method to return a single
: item array.
I would suggest that LocalSolrRequest be modified a little more so that if
the value of an entry in the Map is an Array, then getParams
: In using Solr I've found the need to have a reload command in addition
: to a commit. The reason for this is sometimes updates are made but are
: not available via the server. The commit makes a snapshot which on a
: large index is a potentially expensive operation. Is there a way to do
:
: I'm curious why this approach was taken rather than a single well-
: formed XML response being returned, and whether changing this is
: acceptable. It would perhaps also be nice to get some context back
: for each document added, like its unique key, so a response can
: easily be correlated
: Where would this go in the code? I want to add a new XML based request
: handler so that spans and range filters can be handled. Thanks.
Since RangeFilter is included in lucene 1.9 (as is SpanQuery) it's
automatically inlcuded in Solr .. in fact, the default SolrQueryParser
(used by the
: This is a little bit confusing for me, the documentation at
: http://lucene.apache.org/java/docs/queryparsersyntax.html states they
: query format you described is a Range Query however this document in the
: wiki indicates there is a difference between filters and range queries
:
: I've been thinking that Solr could benefit from auto-valued fields,
: such as a datestamp. This would be a field defined in the schema,
: but its value comes from the Solr server itself rather than the
: client POSTing documents. The client wouldn't post such a value at
: all (it'd be an
: Well, I guess that just goes to show that I ought to catch up on all
: the great notes and TOOD items that all of you fantastic Solr
: creators have made so I don't go reinventing the wheels. Thanks Hoss!
Don't get me wrong ... the wheel hasn't been invented yet, i don't even
think anyone is
: way to get the name at the time. Many parts of Solr were done in an
: extreme rapid-apps type environment... I implemented it as fast as I
: could, no peer review, often past midnight, etc ;-)
Ah the good old days, when I'd send Yonik mail ~5PM Pacific requesting a
feature that i needed in my
I was so preoccupied with trying to understand why your cache wasn't
working, that i didnt' even register what you said about how you are using
it...
: My cache is really just a static cache of BitSet's for a fixed set of
: fields and their values. With my current index size, creating the
:
I almost forgot ... if/when you want to apply some of those facets to a
query provided by your user, put the queries for each facet into a list
and use...
ListQuery facetToApply = ...
DocList result = searcher.getDocList(mainQuery, facetsToApply,
yourSort, 0,
: The next step would be to actually use it... replacing BitSet with
: OpenBitSet in BitDocSet (an alternative would be to create another
: DocSet type, but that gets more complicated).
The one reason i can think of why having both types of DocSet could be
advantagous, is if the memory footprint
: Keeping both would complicate code that tries to find the most
: efficient way to take intersections, etc. And an
: intersection(BitSet,OpenBitSet) would be much slower than either
: intersection(BitSet,Bitset) or intersection(OpenBitSet,OpenBitSet).
I didn't like the idea of having both ..
: That is currently how I have it set up. The agent field is not
: tokenized. However, I need it to be. Here's a concrete example.
: Dante Gabriel Rossetti is one of the agents in our system. Users
: should be able to find him by typing either d, g, or r (case
: insensitive) and they'd see
: Yes I would want to return the docFreq for each term, in the header or
: something of the /select XML result?
the response is totally customizable, RequestHandlers can add any
primitive data that they want (Strings, Integers, Floats, Dates, Lists,
Maps). I would imagine you'd want to make a
: it has is case sensitivity. I could lowercase everything, but then
: the terms the user sees will be in all lowercase and that simply
: won't do for my scholarly audience :)
picky, picky users.
: It seems like what I really need is simply a separate index (or
: rather a partition of the main
: I just wanted to make sure that JNDI was really the right way to go...
: Using those Context xml files in Catalina/localhost would also work
: with init-params AFAIK.
Aren't init-params and context-params required to be specified in
the web.xml?
: Is r/o JINDI really required by the servlet
: Sounds like we need a client with the code that is not shared within
: the Solr server, a Solr JAR that is shared across both the client and
: server, and then whatever other dependencies we need like the pull
: parser.
I think the org.apache.solr.util package may be the right boundray ... the
: implementation super-simple, though arguably it would have been better to
: skip getLastServerException() and just throw an exception on the client side
: when a non-zero server side status came back. That would improve thread
: safety.
yeah .. if i remember ight,you have a
: Ah, thanks... I had been meaning to add that.
yeah .. the impetus was a coworker who wanted a string field that would
sort in a case insensative way ... i thought about writing a new
SortComparatorSource to do this ... but then figured this would be easier
(and more generally usefull).
I just
: A factory is certainly necessary sometimes. You don't want to incur
: setup time for creating a SynonymMap or a StopSet for every instance
: you create.
Sure -- i'm not saying we should eliminate the Factoires completely, just
that i wish it wasn't neccessary to write a factory for every
: Yonik, Erik...
: What would be the right way to put ObenBitSet into Lucene?
I think the right thing to do is put it in Solr since there's a
demonstrated use for it there.
if/when it makes sense to promote it into Lucene core we can do that --
but at the moment there really isn't ... the only
: Output: Currently, the summary data is output as a separate element in
: the response element (like the debug data is currently). This is
: not hard to parse, but perhaps it would be more consistent to add it
: to the doc elements (seems like that would require a bit of
: hackery).
:
: It
: Something along these lines seems reasonable (that we came up with
: near-identical schema reinforces that). I originally had a list per
: field for multiple fragments as well, though scrapped it for
: simplicity.
:
: Does breaking down the highlit segments give significantly more power
: to
Mike, I've only had a chance to skim your patch so far, but on the
surface it looks very good to me. One little thing scares me a little
bit...
: 5. SolrPluginUtils.getParam() only uses the default parameter if it is null,
not blank.
...in principal, I agree with the sentiment that should
: With option 3 you get
: [ {:1}, {:2}, {:3}, { 1:4} ]
:
: With modified option 3, not using objects if the name is null, we get
: [ 1, 2, {:3}, { 1:4} ]
The thing that scares me about all of this, is that it becomes very hard
to tell the difference between a named list and a list of values
: Exaple of what not to do:
agreed.
you've convinced me, JSON is about having a simple response format ... if
you need to deal with complex data, you need a more complex response
format.
-Hoss
: response.setContentType(
: responseWriter.getContentType(solrReq, solrRsp) );
+1
-Hoss
: Modified:
: incubator/solr/trunk/CHANGES.txt
:
incubator/solr/trunk/src/java/org/apache/solr/request/DisMaxRequestHandler.java
:
incubator/solr/trunk/src/java/org/apache/solr/request/StandardRequestHandler.java
:
:DocSet valueDocSet = req.getSearcher().getDocSet(item.getQuery());
:long count = valueDocSet.intersectionSize(results);
:
: Is this the preferred way to obtain such a count, or ithere another
that's a very good way to do it. You could also use
SolrIndexSearcher.numDocs -- it is
: Maybe it belongs in solrconfig.xml as an option on the standard
: request handler init()... after all, it wouldn't apply to the dismax
: handler right (or could it...)? If we ever got a handler that
: accepted XML with explicit optional/required/prohibited, it wouldn't
: apply there either.
: my one suggestion would be that we should support schema.xml
: declarations
: for all of the setable properties of the Lucene QueryParser...
:
:solrQueryParser defaultOperator=AND
: fuzzyMinSim=0.5
: fuzzyPrefixLength=2
:
: thanks to Hoss Yonik for your support, attached is the BinaryField
: implementation if anyone else is interested.
I'm glad it worked out for you Ben ... if you are interested in
contributing this code to Apache so that it can be included in the main
Solr code base, can you open a Jira New
)
: at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
:
:
: Thanks,
: Greg
:
: On 7/25/06, Chris Hostetter [EMAIL PROTECTED] wrote:
:
: : Thanks Greg, I just committed this.
:
: I'm all in favor of the patch as commited, but the NPE still concerens me
: ... the OutputWriter should be able to cleanly deal
: Would it be easier to just add a setDefaultParam(String,String[]) method
: to SolrQueryRequest which would affect all subsequent calls to getParam ?
: ... that would solve that disadvantage.
:
: Yes, with a couple of downsides:
: - no way to set all the defaults at once, request hander
: Here is a slice of what RequestParams or SolrParams might look like:
I like it ... we can even add more helpers to the base class like
getInt(String param) and such.
(the NamedList one should probably extend MultiMapRequestParams though
right ... multiple items with the same name and all
: + * NTFS w/ WinXP or later does support hard links for files (and cygwin
: ln works for files). The current replication scripts could be ported
: to cygwin.
Interesting. THe one other potential hitch that comes to mind is the file
locking/deletion issue that forces Lucene to use a deletable
: I am interested in solr for the RESTful interface, it fits into work I
: am doing as part of an open source GIS project. I have a couple of
: questions that I would be grateful for advice on;
:
: 1) Do spatial queries (such as documents within a bounding box) exist
If your bounding boxes are
: thanks for the answer, I am also interested in the jdbc connectivity.
Sorry, i thought that was and if not clause on your question.
I've heard of some attempts at extending Lucene's Directory with a RDBMS
backed implimentation -- from what i'm told they tend to focus on modeling
lucene files
: - if no factory can be found, an attempt will be made to construct
: one dynamically (easiest would be to create a generic factory that
: works via reflection). People could use simple filters w/o creating a
: factory for it.
I think i mentioned this before ... my opinion depends on what the
: Thanks. Yes I came up with a hacked solution to the problem. Takes a
: Query and rewrites the Terms using the Analyzer. If the Analyzer
typcially the Analisys happens before you construct a Query object --
where are these Queries comping from that they are already objects but
haven't been
: Does anyone else have comments on the two issues I raised, namely:
No comment -- I've never used highlighting so i dont' realy have enough
context/experience to have an opinion.
-Hoss
be
*easier* to macke sense of in a Servlet.
:
: Bill
:
: On 8/28/06, Chris Hostetter [EMAIL PROTECTED] wrote:
:
: : + * refactor all of the JSP pages into servlets so a JDK/JSP compiler
:
: : It seems like a simpler option would be to just move to Tomcat 5.5 or
: : Jetty 6, both of which handle JSPs
: I haven't looked at the patch yet but in terms of the parameters, might
: it make sense to use a group name similar to the highlighter params?
: e.g., facet, facet.fl, facet.query, facet.limit, etc.
:
: Also, now that we have per-field override capability for params, we
: should document which
: I guess I need to number my patch files as they don't seem to get listed
: in the order they were added and I can't remove old ones. The new patch
: appears to be number 3.
FYI: if one patch is intended to replace an existing patch, just reuse the
name -- the old version will appear greyed
Again, i dodn't know a lot about highlighting, but going back to an
earlier point about making the params indicate what they do, not what
specific API options they trigger...
: 1) hl.scoring=simple (the default) - construct with Query only. May have
some matches from other terms, but allows
custom code in the admin pages expectingthem
to recompile doesn't seem like much of a stretch.
:
: Bill
:
: On 8/29/06, Chris Hostetter [EMAIL PROTECTED] wrote:
:
:
: : Leaving them as JSPs would make it easier to customize the admin pages.
:
: It depends on the customization. CSS is still easy
: These comments are probably just confusion since the approach differs
not at all confusing, you seem to have grasped everything just fine.
The one big thing to keep in mind is that this is an attmpt at very
simple faceted searching support, my goals for these changes were:
1) provide
: One thing that my facet code does is compute the count for all items
: that have _no_ terms in a particular field, and makes an
: unspecified count as well. It does this by putting all documents
: found into a DocSet as it iterates through all terms for a field, and
: then .andNot'ing it away
: Getting back to what you said about the 80/20 rule, you certainly have
: hit that sweet spot. It may be that in just about every use case (or
: at least 80% of them :) ) the client can, at worst, extract the field
: name from the lst name attribute, and use that for grouping. While
: explicit
: Following up with what the ideal faceted browsing info might look like
: (ignoring how label info is obtained for the moment):
: - include grouping and group labels
: - include constraint label, count, and the exact query needed to
: filter by the constraint
: - have the same structure
: : One thing that my facet code does is compute the count for all items
: : that have _no_ terms in a particular field, and makes an
: : unspecified count as well. It does this by putting all documents
: : found into a DocSet as it iterates through all terms for a field, and
: : then
: I implemented lazy field loading in Solr, but then get messed up results
: or EOFExceptions. The index is fine, I turned off lazy loading and it
: looks fine. Is this known behavior?
I think it's safe to say you are in uncharted territory -- i certianly
wouldn't categorize an EOFException as
: What's the usecase here, and the downside to having to provide the
: full fq list in the URL?
: Is this simply to shorten the request URLs?
The main use case is to allow configuration of a requestHandler that the
clients don't have to be aware of at all it just works ... in the case
of fq it
: The main use case is to allow configuration of a requestHandler that the
: clients don't have to be aware of at all it just works ..
:
: That sounds fine then for things that are optinally appended, but not
: being able to override that doesn't make sense unless it's for some
: security
i've started to get anoyed enough by the long list of ? that show up
next to all of the artifcats that are produced by our build.xml that i
finally read up on how exactly to do the equivilent of a .cvsignore and
found that it's entirely controled by a subversion property (which are
pretty easy to
: The usecase you gave of setting reasonable append defaults for lists
: seems like the most common one. If the only configuration option
: available to people is to have non-overridable appends, it will get
: used as convenience (or error) and end up being a pain.
:
: I guess I'd like to see
: you can also edit your .subversion/config file
: and uncomment the 'global-ignores' line if there are lot of temp
: files in various places.
temp files aren't the issue ... it's files that are generated/copied as
part of the build.
I'll commit the props when i get to work tomorow.
-Hoss
: global-ignores is not just for temp files, but any common pattern:
:
: global-ignores = *~ #*# *.class *.o *.pyc ...
sure, but these aren't common patterns ... setting a global-ignore on
all files named commit isn't a good idea, but in the solr project, in
the path example/solr/bin, it is a
: We did notice on thing that appears incorrect. The sort mechanism being
: performed by the client adds a request parameter before sending to
: lucene.
:
: /solr/select?q=termsort=name+asc
...
: According to lucene docs
: (http://incubator.apache.org/solr/tutorial.html#Sorting),
: Should Solr have a tab on Lucene's home page? Other incubating
: Lucene-related projects do. I think it would be appropriate.
Oh yeah ... I forgot Lucene4c is in incubation too.
+1, i'm all in favorof more solr visibility.
-Hoss
: Since I was already there, I went ahead and committed a description
: for test-reports, and javadoc. But the junit target doesn't need a
: description since test is the main testing entry point.
actually, something i've been thinking would be nice is if there was a way
to run the tests and
: '''
: 1) The Lucene QueryParser tokenizes on white space before giving any
: text to the Analyzer, so if a person searches for the words sea biscit
: the analyzer will be given the words sea and biscit seperately,
: and will not know that they match a synonym.
: '''
:
: But the SynonymFilter
am i smoking crack of is post.sh mistakenly sending every doc twice in a
row? ...
for f in $FILES; do
echo Posting file $f to $URL
curl $URL --data-binary @$f
curl $URL --data-binary @$f -H 'Content-type:text/xml; charset=utf-8'
echo
done
...is there any reason not to delete that first
: Here's the changes I'll commit shortly unless there are objections
: (sorry about the false-changes... I think it must be trailing
one thing confuses me: why the boolean params for missing/zero in the
method declarations for getFieldCacheCounts and getFieldTermEnumCounts ?
... for
: The content-type should also be selectable by a request parameter, with
: text/html as the default I guess.
isn't that what the media-type attribute of xsl:output/ is for?
-Hoss
: ...but is there anyway to let
: people add web.xml for things like ServletFilters without manully
: unpacking and modifyng hte war?..
:
: The way we handle that in Cocoon 2.1.x is with ant tasks which patch
: web.xml (and other similar things) at build time, based on options
: coming from
: Depends on what one things the entry point should be, or the unit of re-use.
: If someone is implementing their own custom handler, they may very
: well be doing param parsing differently. The lowest level routines
but the bueaty of hte SolrParam class is that it isn't inheriently tied
to
: How far is Solr from what Hoss described below - single solr.war with
: multiple instances/collections/indices?
: I know it has been discussed in the past, but I don't recall anything
: after that.
It's already possible to do this with solr out of the box using tomcat...
: ...What do people think about a Map getContext() for a
: SolrQueryRequest? It could be instantiated on-demand to avoid any
: penalty for those not using it...
:
: This would be useful in my case.
no objections ... the other thought that occured to me for this specific
case is that both the
: I'll try to update my patch to pull out a getFieldFacets method that
: encapsulates parameter parsing and the decision of what faceting
: strategy to call... that way soneone could add their own method and
: override getFieldFacets() to call it in certain cases. I'm not sure
: it's really a
: Attachment: solr-XSLTResponseWriter-20060922.tar.gz
I finally got a chance to look at this tonight, i like it, but the big
thing thta jumped out at me was the configuration...
Since this QueryResponseWriter has use of configuration options, now is
probably the time to add an
Thanks for tackling this Mike ... I've been dreading the whole issue of
Lazy Loading but your patch gives me hope. I haven't had a chance to try
it out, but reading through it, it seems a lot more straight forward then
I'd feared.
A couple of concerns jump out at me though, starting with the
: - stabilize/review external api (query parameters,
: schema.xml/solrconfig.xml format, XML response format). (for
Right ... it's not something that i've thought about lately, but doing
something with the XSD in SOLR-17 so that the XML output format can be
validated would probably be a good
: http://issues.apache.org/jira/browse/SOLR-49 (XSLTResponseWriter)
: could be committed. It has no impact on existing code and can be
: useful for simple setups, demos, etc.
On the subject of stablizing the external APIs, the one thing about your
patch in it's current format that I rememebr
: Chris: What would you like to see vis-a-vis the many field issues before
: committing? Should we put in a global lazy-field-disable option?
Yeah, a simple solrconfig option that lets you disable it completley is
probably a good idea (especailly in light of LUCENE-683) and i don't see
any
: Thanks again Bertrand.
Would you like to do the honors of writing up some wiki docs about the new
Writer?
-Hoss
: What do people think... should this (optionally) go under
: responseHeader, to be added by the ResponseWriter when it sees the
: flag to do so,
: OR
: Should it be added in the response body, at the same level as
: highlighting info, faceting info, etc.
I think it should be done by the
: I think this puts everything in the right place - comments are welcome of
course.
I don't have a strong opinion about the QTime, but all the rest sounds
great.
-Hoss
: Any thought on switching to a more powerful scripting language for
: these things? Were this python, frinstance, a basic xml parser could
: be thrown together in a couple lines. Python is also more
: cross-platform than bash scripting.
I have no strong opinion about what language they be
your API params look good to me ... there are some cases where params
could be specified that contradict that we'd need to generate erors for
(ie: header=false and no field list) but that's not a bad thing.
another sticky wicket is multi value fields ... do we want to say the bulk
uploading
: Help is welcomed! Let us know if you see changes needed to satisfy
: ASF release requirements procedures.
: http://incubator.apache.org/guides/releasemanagement.html
(i'm kinda tired so this may ramble...)
ant nightly currenlty includes servlet-api-2.4.jar in the lib dir .. but
we don't
: Any ideas on what the interface should look like?
/solr/upload/xml?xsl=foo.xsl
:
: Seems to run into some of the same questions... how should we allow
: POST and specify params about the post at the same time?
alternately, the API could just require that the XML file be POSTed, and
we could
: http://wiki.apache.org/incubator/November2006
+1
-Hoss
: It would be nice to a nice HTML page allowing the user to query and display
: 1) some faceting info such as hit counts with links that allowed the user to
narrow their search results.
: 2) highlighted summaries
: 3) easy way to query the dismax handler as well as the standard request
handler
: Probably. Unless you previously set your custom loader to be the
: thread's context class loader. If you do that, you'd have to ensure
: that your loader is aware of the context classpath (which could be
: accomplished by making it a child of the old context class loader).
: Otherwise, you might
: 2. Adding a phonetic token filter and relying on the per-field analyzer
: support.
:
: Option 2 seems like it would be a lot faster in production, and
: probably easier to implement. Does that seem right?
yep, just write your Analyzer (or TokenFilter) and drop it in.
: How do I specify the
: 1. Adding fuzzy to the DisMax specs.
:
: What do you envisage the implementation looking like?
:
: Probably continue with the template-like patterns already there.
:
: title^2.0 (search title field with boost of 2.0)
: title~ (search title field with fuzzy matching)
Interesting
: A naming convention question: should the class names end in
: Filter or TokenFilter (and FilterFactory or TokenFilterFactory)?
: I see both in org.apache.solr.analysis.
Ummm yes :)
I don't think it makes a big difference ... i'd never noticed the
inconsistency untill now.
: I'm a bit
1 - 100 of 860 matches
Mail list logo
