Re: twitter questions
I've also found the question on SO, I'll give it a shot later today. On Aug 20, 2012, at 10:18 AM, Michael Herndon mhern...@wickedsoftware.net wrote: I went to update the twitter feed and saw these questions in case anyone wants to answer these: @Nick_Craver @LuceneDotNet - is anyone already working on a NIOFSDirectory port? Unless I misunderstand, it would be a huge boost to our SSD environment' @intelligibabble would love it if some #lucene or @LuceneDotNet gurus took a look at my SO question stackoverflow.com/questions/1135… http://stackoverflow.com/questions/11354455/using-lucene-net-thread-safe-from-asp-net-web-application
RE: Lucene.Net 3.0.3 and medium trust
Chris, All you have to do is create a new AppDomain properly and marshal the call into that app domain. You'll need a wrapper that derives from MarshalByRefObject so the call is marshaled; it's in the marshaled call that you do your test work and then marshal the results for comparison (or just a Boolean value, it's in the return value you can serialize things and send them across the app domain boundary by value). See here for greater detail on how to do this: http://stackoverflow.com/q/987332/50776 - Nick -Original Message- From: Christopher Currens [mailto:currens.ch...@gmail.com] Sent: Tuesday, August 14, 2012 12:16 PM To: lucene-net-dev@lucene.apache.org Subject: Re: Lucene.Net 3.0.3 and medium trust We should probably try and have unit tests that will spin up medium trust app domains and run some basic tests in them, so we don't accidentally ship with something big like this. That is what I initially tried to do to reproduce this bug on my end, but the code that I would which should have created a medium trust app domain wasn't working. I had to create an ASP.NET project and give it medium trust to repro. I'm sure there's a way to do it, though. Thanks, Christopher On Tue, Aug 14, 2012 at 9:08 AM, Christopher Currens currens.ch...@gmail.com wrote: Inheriting from WeakReference was the problem. I'm just going to remove the wrapper entirely, it doesn't give us any real benefit. The code has already been pushed. Thanks, Christopher On Tue, Aug 14, 2012 at 8:46 AM, Prescott Nasser geobmx...@hotmail.com wrote: It'd be nice to understand where this issue is coming from as it wasn't in 2.9.4 rather than jumping to a quick fix imo. I'll dig a bit myself ~P Date: Tue, 14 Aug 2012 17:32:29 +0200 From: si...@devhost.se To: lucene-net-dev@lucene.apache.org Subject: Re: Lucene.Net 3.0.3 and medium trust Hi, It got worse. Building a custom AttributeFactory can only handle a few cases, I needed to subclass _every_ TokenStream/TokenFilter to override all attribute-handling methods (AddAttribute, GetAttribute, etc), and all analyzers to use these streams, to use a dictionary with strong references just for medium trust... I gave up before doing that. ;) Is there any use cases where we need to garbage collect attributes before the stream/attribute source is closed? Changing to a normal dictionary seems like a quick fix. // Simon On 2012-08-14 17:21, Christopher Currens wrote: It must be something we've missed, as we want to target medium trust locations in the future. I can't think of anything off the top of my head that would require medium trust, though, let alone unmanaged code. I'll dive into this. Thanks, Christopher On Mon, Aug 13, 2012 at 10:01 PM, Simon Svensson si...@devhost.se wrote: Hi, I'm having trouble upgrading a web application running under medium trust from 2.9.4 to 3.0.3. Code that previously worked now throws a SecurityException. [SecurityException: Request for the permission of type 'System.Security.Permissions.SecurityPermission, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' failed.] Lucene.Net.Support.WeakKey`1..ctor(T key) +0 Lucene.Net.Support.WeakDictionary`2.get_Item(TKey key) +113 Lucene.Net.Util.DefaultAttributeFactory.GetClassForInterface() +178 Lucene.Net.Util.DefaultAttributeFactory.CreateAttributeInstance() +95 Lucene.Net.Util.AttributeSource.AddAttribute() +375 Lucene.Net.Analysis.CharTokenizer..ctor(TextReader input) +126 Lucene.Net.Analysis.WhitespaceTokenizer..ctor(TextReader in) +37 The DefaultAttributeFactory, via WeakReference, requires SecurityPermissionFlag.UnmanagedCode which is not present under medium trust. There's an AttributeFactory.DEFAULT_ATTRIBUTE_FACTORY which I could to replace the DefaultAttributeFactory, but it's readonly. I'm rewriting my code to call the constructor overload (on tokenizers) accepting an AttributeFactory, but this means that I cannot use any existing Analyzer since they don't provide an extension points to change the AttributeFactory. Is medium trust [using default classes] dropped in 3.0.3, or is this something we've missed? // Simon
Re: [Lucene.Net] 2.9.4
Prescott,
You really don't need to do that; reads and writes of reference fields are
guaranteed to be atomic as per section 5.5 of the C# Language Specufication
(Atomicity of variable references)
If you were doing other operations beyond the read and write that you wanted to
be atomic, then the lock statement is appropriate, but in this case it's not.
The volatile keyword in this case (assuming no lock) would absolutely be needed
to guarantee that the variable has the most up-to-date value at the time of
access; using lock does this for you and makes volatile unnecessary.
- Nick
On Sep 22, 2011, at 11:14 PM, Prescott Nasser geobmx...@hotmail.com wrote:
Before I go replacing all the volatile fields I wanted to run this past the
list:
private System.IO.StreamWriter infoStream;
into
private object o = new object();
private System.IO.StreamWriter _infoStream;
private System.IO.StreamWriter infoStream
{
get
{
lock (o)
{
return _infoStream;
}
}
set
{
lock (o)
{
_infoStream = value;
}
}
}
Sorry, I don't normally deal with locks..
Thanks for any guidance
~P
@Prescott,
Can the volatile fields be wrapped in a lock statement and code that access
those fields with replaced with call to a property /method that wraps access
to that field?
On Wed, Sep 21, 2011 at 1:36 PM, Troy Howard thowar...@gmail.com wrote:
I thought it was:
2.9.2 and before are 2.0 compatible
2.9.4 and before are 3.5 compatible
After 2.9.4 are 4.0 compatible
Thanks,
Troy
On Wed, Sep 21, 2011 at 10:15 AM, Michael Herndon
mhern...@wickedsoftware.net wrote:
if thats the case, then well need conditional statements for including
ThreadLocalT
On Wed, Sep 21, 2011 at 12:47 PM, Prescott Nasser geobmx...@hotmail.com
wrote:
I thought this was after 2.9.4
Sent from my Windows Phone
-Original Message-
From: Michael Herndon
Sent: Wednesday, September 21, 2011 8:30 AM
To: lucene-net-dev@lucene.apache.org
Cc: lucene-net-...@incubator.apache.org
Subject: Re: [Lucene.Net] 2.9.4
@Robert,
I believe the overwhelming consensus on the mailing list vote was to
move
to
.NET 4.0 and drop support for previous versions.
I'll take care of build scripts issue while they being refactored into
smaller chunks this week.
@Troy, Agreed.
On Wed, Sep 21, 2011 at 8:08 AM, Robert Jordan robe...@gmx.net wrote:
On 20.09.2011 23:48, Prescott Nasser wrote:
Hey all seems like we are set with 2.9.4? Feedback has been positive
and
its been quiet. Do we feel ready to vote for a new release?
I don't know if the build infrastructure is part of the
release. If yes, then there is an open issue:
Contrib doesn't build right now because there
are some assembly name mismatches between certain *.csproj
files and build/scripts/contrib.targets.
The following patches should fix the issue:
https://github.com/robert-j/**lucene.net/commit/**
c5218bca56c19b3407648224781eec**7316994a39
https://github.com/robert-j/lucene.net/commit/c5218bca56c19b3407648224781eec7316994a39
https://github.com/robert-j/**lucene.net/commit/**
50bad187655d59968d51d472b57c2a**40e201d663
https://github.com/robert-j/lucene.net/commit/50bad187655d59968d51d472b57c2a40e201d663
Also, the fix for [LUCENENET-358] is basically making
Lucene.Net.dll a .NET 4.0-only assembly:
https://github.com/apache/**lucene.net/commit/**
23ea6f52362fc7dbce48fd012cea12**9a7350c73c
https://github.com/apache/lucene.net/commit/23ea6f52362fc7dbce48fd012cea129a7350c73c
Did we agree about abandoning .NET = 3.5?
Robert
[Lucene.Net] OT: Wyatt's expression was - RE: [Lucene.Net] VOTE: .NET 2.0 Framework Support After Apache Lucene.Net 2.9.4
I've cast my vote already, but +1 to Wyatt's expression -Original Message- From: Wyatt Barnett [mailto:wyatt.barn...@gmail.com] Sent: Tuesday, May 10, 2011 12:46 PM To: lucene-net-dev@lucene.apache.org Subject: Re: [Lucene.Net] VOTE: .NET 2.0 Framework Support After Apache Lucene.Net 2.9.4 +1, burn the ships. On Tue, May 10, 2011 at 12:43 PM, Christopher Currens currens.ch...@gmail.com wrote: +1, but I'm partial to 0 if the demand is there for it. I don't mind keeping up support for 2.0, in a separate branch, for a set amount of time. On Tue, May 10, 2011 at 2:01 AM, Moray McConnachie mmcco...@oxford-analytica.com wrote: PS: If you are supporting .NET 3.5 then you get .NET 2.0 support anyway, you just have to bin-deploy the .NET 3.5 dependencies (System.Core, etc) since they are all the same CLR Aaron Powell Aaron, I think the move to 4.0 is actually to stop supporting 3.5 as well judging by later emails... Moray - Moray McConnachie Director of IT +44 1865 261 600 Oxford Analytica http://www.oxan.com - Disclaimer This message and any attachments are confidential and/or privileged. If this has been sent to you in error, please do not use, retain or disclose them, and contact the sender as soon as possible. Oxford Analytica Ltd Registered in England: No. 1196703 5 Alfred Street, Oxford United Kingdom, OX1 4EH -
RE: [Lucene.Net] VOTE: .NET 2.0 Framework Support After Apache Lucene.Net 2.9.4
+1 -Original Message- From: Troy Howard [mailto:thowar...@gmail.com] Sent: Monday, May 09, 2011 4:05 PM To: lucene-net-dev@lucene.apache.org; lucene-net-u...@lucene.apache.org Subject: [Lucene.Net] VOTE: .NET 2.0 Framework Support After Apache Lucene.Net 2.9.4 All, Please cast your votes regarding the topic of .Net Framework support. The question on the table is: Should Apache Lucene.Net 2.9.4 be the last release which supports the .Net 2.0 Framework? Some options are: [+1] - Yes, move forward to the latest .Net Framework version, and drop support for 2.0 completely. New features and performance are more important than backwards compatibility. [0] - Yes, focus on the latest .Net Framework, but also include patches and/or preprocessor directives and conditional compilation blocks to include support for 2.0 when needed. New features, performance, and backwards compatibility are all equally important and it's worth the additional complexity and coding work to meet all of those goals. [-1] No, .Net Framework 2.0 should remain our target platform. Backwards compatibility is more important than new features and performance. This vote is not limited to the Apache Lucene.Net IPMC. All users/contributors/committers/mailing list lurkers are welcome to cast their votes with an equal weight. This has been cross posted to both the dev and user mailing lists. Thanks, Troy
RE: [Lucene.Net] Medium trust security issue
Richard,
This is because the call to FileSupport.Sync ultimately calls
SupportClass.Sync(FileStream) which ends up calling the FlushFileBuffers API
function through the P/Invoke layer, which is disallowed in medium trust
environment.
However, this should be mitigated by the fact that you have set the
assembly to allow partially trusted callers (are you doing this as a check
in to the tree? If so, have you done a full security analysis? Setting
this attribute on an assembly as big as Lucene.NET has major security
implications).
It would seem to me that you might not have given Lucene.NET a
strong name; this is required for AllowPartiallyTrustedCallers to take
effect.
This issue was seen by Simone Chiaretta and was discussed in the
group a while ago:
http://web.archiveorange.com/archive/v/3k9XU33O4yJyW15fWfMd
However, at the time, Lucene.NET was built on .NET 2.0 (IIRC) and
didn't have access to the overload of the Flush method which was used to
guarantee everything was flushed to disk:
http://web.archiveorange.com/archive/v/3k9XU33O4yJyW15fWfMd#MhNDlmKgnUj5fOj
Since you are now working in .NET 4.0, you should be able to replace
the following code in SupportClass.cs
(https://svn.apache.org/repos/asf/incubator/lucene.net/trunk/C%23/src/Lucene
.Net/SupportClass.cs):
public static void Sync(System.IO.FileStream fileStream)
{
if (fileStream == null)
throw new
ArgumentNullException(fileStream);
fileStream.Flush();
if (OS.IsWindows)
{
if (!FlushFileBuffers(fileStream.Handle))
throw new System.IO.IOException();
}
else if (OS.IsUnix)
{
if (fsync(fileStream.Handle) != IntPtr.Zero)
throw new System.IO.IOException();
}
else
{
throw new NotImplementedException();
}
}
With this:
public static void Sync(System.IO.FileStream fileStream)
{
if (fileStream == null)
throw new
ArgumentNullException(fileStream);
fileStream.Flush(true);
}
One could make the argument that this should be taken out of SupportClass
and moved into FSDirectory, but that might break some of your line-for-line
port code, so best to keep it in SupportClass.
- Nicholas Paldino [.NET/C# MVP]
-Original Message-
From: Richard Wilde [mailto:rich...@wildesoft.net]
Sent: Sunday, May 01, 2011 6:01 AM
To: lucene-net-...@incubator.apache.org
Subject: [Lucene.Net] Medium trust security issue
Hi
I am running into problems using Lucence 2.9.2 in a medium trust
environment, namely Rackspace cloud. I have added the following line to
assembleyinfo.cs
[assembly: AllowPartiallyTrustedCallers()]
However the following code produces the error below
FSDirectory directory = FSDirectory.Open(new
DirectoryInfo(Server.MapPath(~/App_Data/LuceneIndex)));
Analyzer analyzer = new
StandardAnalyzer(Version.LUCENE_29);
var writer = new IndexWriter(directory, analyzer, true,
IndexWriter.MaxFieldLength.LIMITED);
writer.AddDocument(...);
writer.Optimize();
writer.Close();
The directory LuceneIndex is being created, does anyone have a fix for
this?
Security Exception
Description: The application attempted to perform an operation not allowed
by the security policy. To grant this application the required permission
please contact your system administrator or change the application's trust
level in the configuration file.
Exception Details: System.Security.SecurityException: Request failed.
Source Error:
An unhandled exception was generated during the execution of the current web
request. Information regarding the origin and location of the exception can
be identified using the exception stack trace below.
Stack Trace:
[SecurityException: Request failed.]
FileSupport.Sync(FileStream fileStream) +0
Lucene.Net.Store.FSDirectory.Sync(String name) +157
Lucene.Net.Index.SegmentInfos.FinishCommit(Directory dir) +184
Lucene.Net.Index.IndexWriter.Init(Directory d, Analyzer a, Boolean
create, Boolean closeDir, IndexDeletionPolicy deletionPolicy, Boolean
autoCommit, Int32 maxFieldLength, IndexingChain indexingChain, IndexCommit
commit) +293
Lucene.Net.Index.IndexWriter..ctor(Directory d, Analyzer a, Boolean
create, MaxFieldLength mfl) +413
Mvc.Cms.Controllers.LuceneController.Index() +1066
lambda_method(Closure
RE: Stefan's Newbie Questions (was Re: Proposal Status, Initial Committors List, Contributors List)
As a consumer (and I think that most consumers would agree), I'd have to disagree STRONGLY on trading off performance for ease of conversion. Lucene and Lucene.NET is predicated on performance, compromising that, IMO, runs contrary to the core principals of Lucene. - Nick -Original Message- From: Hans Merkl [mailto:h...@hmerkl.com] Sent: Wednesday, January 26, 2011 3:18 PM To: lucene-net-dev@lucene.apache.org Subject: Re: Stefan's Newbie Questions (was Re: Proposal Status, Initial Committors List, Contributors List) Personally I am willing to trade some performance for always being up to date with the latest Java releases and also being able to use other Java code. Although as far as I have seen most people say it's at the same speed or even slightly faster than the .NET version. I personally would be more likely to contribute to IKVM if there are any issues since it would also benefit other Java code I use like TIKA. I wonder if anybody has ever tried Lucene with IKVM in a heavy load production environment. I use it in a one thread per index desktop app and wouldn't notice if there were any issues under heavy load. On Wed, Jan 26, 2011 at 14:01, Troy Howard thowar...@gmail.com wrote: I'm on the fence about IKVM. It has some significant benefits and some significant drawbacks: Benefits: - Allows us to get to a commoditized line-by-line .NET DLL in the fastest and easiest manner. No porting. - Reasonable performance profile - Well tested Java environment equivalence Drawbacks: - Blackbox, can't improve on it or tweak behaviour. If there are bugs or other issues, related to IKVM (ie thread safety, memory handling, etc) we can't fix those without dropping IKVM as our solution. - Adds an additional dependency - May not be the best possible performance profile. As DIGY said, it's roughly equivalent, but that doesn't mean that current Lucene.Net is fully optimized for .NET. In fact, it has been proven not to be by folks who have made custom builds/forks, realizing significant speedups using generics, etc.. Also, that's a significant change in the library, which will introduce breaking API changes, and require us to beef up the unit tests to ensure that concerns like thread safety continue to behave as expected. Thanks, Troy On Wed, Jan 26, 2011 at 10:39 AM, Hans Merkl h...@hmerkl.com wrote: A .NET wrapper around the IKVM classes may be a good idea. I like the idea that IKVM would also allow use of tons of other useful Java/Lucene code that's out there. There are some filters and analyzers in Java that might be very useful for my work. That's not really possible with the line-by-line port. It may be possible with Sharpen though. On Wed, Jan 26, 2011 at 13:04, Digy digyd...@gmail.com wrote: In theory you can use ikvmc to compile the Java source files into a .NET DLL that references some IKVM DLLs and an ikmvc'ed version of OpenJDK's classlib. After that it is a plain .NET DLL and one could write a .NET centric API using that DLL. I haven't really tried it on anything serious and it may become tricky if reflection gets involved. And there is some layer of indirection you wouldn't have by a line by line translation that may lead to decreased performance. I'd be game to try it out, though. A few yers ago, I tried IKVM with ~300M (200-300 bytes) documents. It was surprisingly as fast as Lucene.Net. That may mean that we should fix something in the code. Reflection is another nice thing in IKVM. You can even load and execute Java classes J DIGY
RE: Question
Erik, It's the fact that the API is exactly the same (as well as the lines of code, practically) which causes many of the issues in Lucene.NET (not only in use but in implementation), as while Java and C# are very similar, that doesn't guarantee the same results. But that's an issue for another email, one which many (including myself) have dealt with. That being said, I would go with a Solr provider if such a thing existed. I'm debating whether or not to use the Lucene.NET library in my application, or to try and find a preexisting Solr provider. However, it doesn't seem that there are many, and I really don't have the luxury of setting up the environment myself (although I'm interested in using it, since I can very easily talk whatever language it does over the wire with .NET). - Nick -Original Message- From: Erik Hatcher [mailto:erik.hatc...@gmail.com] Sent: Thursday, January 07, 2010 4:49 AM To: lucene-net-dev@lucene.apache.org Subject: Re: Question Ed - that's a reasonable critique, but the API is practically the same between the Lucene.Net and Lucene Java. There is a section contributed by George in the upcoming 2nd edition of Lucene in Action - it's short and says basically that. But, rather than buy a commercial search engine, consider Solr! I don't want to come here and steal any of Lucene.Net's thunder by mentioning Solr, as no doubt Lucene.Net is the right fit for many projects. Solr, though, is so much more than just Lucene, providing enterprisey features (replication, distributed search, facets, and more) that just can't be trivially/naively built on top of any flavor of Lucene. And Solr is easy interfaced with .NET as a client. Of course the hurdle then is does Solr, a Java-based app, fit into the operations of your deployment environment?. It's another technology to add if the shop is purely .NET currently. But then again, it literally does run everywhere quite easily. Erik On Jan 7, 2010, at 4:27 AM, Ed Jones wrote: My problem with Lucene in Action and all the examples on the internet is that they were all in Java and you have to understand exactly what Java is doing to understand it all properly. It's for this very reason we had to shun using Lucene.net in major projects. I wanted dearly to use it but the learning curve was far too steep and there appears to be very very few .net examples of code or help. Instead we have invested a significant amount of money in buying in a much more commercial search engine. I am keeping an eye on the Lucene.net project though in-case it can be used in other parts of our business, but again the same will apply, we will need more non Java examples. Ed -Original Message- From: Roger Chapman [mailto:ro...@stormid.com] Sent: 07 January 2010 09:21 To: lucene-net-dev@lucene.apache.org Subject: RE: Question From what I can remember the book Lucene in Action has a good section on indexing documents and PDFs http://www.manning.com/hatcher2/ Roger. -Original Message- From: Ben Martz [mailto:benma...@gmail.com] Sent: 06 January 2010 19:51 To: lucene-net-dev@lucene.apache.org Cc: lucene-net-dev@lucene.apache.org Subject: Re: Question Todd, I would definitely take Michael's advice to learn more about the overall issue before you get too far. A quick answer that may help is Windows does not ship with an iFilter for PDF built-in. Installing Adobe Reader 8 or higher will install a decent PDF iFilter. I am a little surprised by your question though - I assume that you have access to your own source code and could examine the result from the iFilter that's being fed to the IndexWriter and compare the behavior in the TXT case with the behavior in the PDF case? Cheers, Ben Sent from my iPhone On Jan 6, 2010, at 10:13, Michael Garski mgar...@myspace-inc.com wrote: Todd, You'll need some way to extract the text from the PDF prior to indexing. I'm not familiar with any packages that can do that but I have heard of them. You may want to try searching the mailing list to see if there has been mention of one previously. Lucid Imagination hosts a great mailing list search tool at http://www.lucidimagination.com/search/ Michael -Original Message- From: Todd McIndoo [mailto:tmcin...@speedyscan.biz] Sent: Wednesday, January 06, 2010 10:11 AM To: lucene-net-dev@lucene.apache.org Subject: Question Sorry if this is duplicate We are using Lucene.net of version 2.0.0.4. I am trying to search a document which contains lots of PDFs. I want to search a document, which contains a specific word, using Lucene.net. We are yielding results in text documents but not in PDF. Is there something we have to do to be able to search in PDF Documents. All ifilters have been installed on the computer so I do not think
