Shad, wow, you're fast as hell! Thanks for the explanation!
Even with the newest code, the culprit is the assembly.GetTypes() method which is not working in sandboxed mode. What I now did, was implementing the function from the Microsoft reference source directly (https://referencesource.microsoft.com/#mscorlib/system/reflection/assembly.cs,5ef0afb56c4252c4). But this function has a GetTypes() call itself, which crashes again. When I look into this function, the source tells me "throw NotImplementedException()" What the hell?! Anyways, I have the workaround I explained earlier, so I can ignore this issue for now. Pushing my simple search app in sandboxed mode further, I get the next error in the following location: new SearcherManager() --> Get DirectoryReader() --> new StandardDirectoryReader() The code follows some base constructors deeper into the object and finally fails at CompositeReader() : base(). The error message I get after it goes through some finally blocks: 'Request for the permission of type 'System.Security.Permissions.SecurityPermission, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=...' failed.' If you wanna know more about my environmental limitations, you can read them here: https://msdn.microsoft.com/en-us/library/gg334752.aspx Thanks and best Regards, Matthias -----Original Message----- From: Shad Storhaug [mailto:[email protected]] Sent: Donnerstag, 18. Mai 2017 17:48 To: [email protected] Subject: RE: Problems when running a search in my production environment Matthais, > In general, reflection is working in partial trust mode, but not completely > it seems. I'm not sure about what is/isn't working exactly. Thanks. That confirms my suspicions. ASP.NET applications typically use IIS integrated mode, which means that certain operations (i.e. Reflection) are not allowed during the application startup phase, especially when using partial trust. Lucene was originally setup to initialize in the Codec, DocValuesFormat, and PostingsFormat in static constructors, which meant that *any* call to a static method or instantiating a Codec would call the Reflection code. I have modified it so it now lazy-loads upon the first call to Codec.ForName(), DocValuesFormat.ForName(), or PostingsFormat.ForName() and also waits until the first call to Codec.DefaultCodec to load that one as well. Therefore, the Reflection code happens during the application's runtime instead of during application startup, but since it only happens once upon the first call, this isn't that expensive. If anyone wants it to initialize at application startup, it is as simple as calling one of the above methods or, better yet, inherit the DefaultCodecFactory (and the other 2 factories) and call EnsureInitialized() in the constructor - at least now there is a choice. > Do you need to search through all the system assemblies (list of > 4000 > elements)? The scan filters out anything that is not a Codec type so the list is relatively short. In addition, we only scan the 2 assemblies in Lucene.Net where codecs live. The original Lucene code actually scanned every module in the application (tens of thousands of classes), so this is quite a bit more efficient than that. We sacrifice the convenience of being able to just inherit Codec without any additional setup code, but since it is not very likely that many people will need to write their own codec this is a reasonable tradeoff. > Can't we just search for the attributes e.g. "[CodecName("Lucene46")]"? We could, but then we would lose the convention-based naming, which means the attribute could no longer be optional. I followed the same convention-based approach that is used in MVC for scanning for controllers. Just calling a codec MyCoolCodec and subclassing Codec is enough - then when it is added to the codec factory's PutCodecType() method (or ScanForCodecs()) the codec will automatically be registered with the name "MyCool". The attribute is only necessary if you want to override an existing codec and reuse the same name. Anyway, if all goes well with the tests, the fix will be on the CI feed (https://www.myget.org/gallery/lucene-net-ci) in about an hour and I will have a beta version for voting on shortly after. Thanks, Shad Storhaug (NightOwl888) -----Original Message----- From: Matthias Strauss - xRM1 Business Solutions [mailto:[email protected]] Sent: Thursday, May 18, 2017 10:05 PM To: [email protected] Subject: RE: Problems when running a search in my production environment Shad, I'd like to share some information of my recent investigation with you: 1. First it crashed at assembly.GetTypes() in the function ScanForCodecs(): foreach (var c in assembly.GetTypes()) To resolve this issue I built up a list of the CodecNames manually, got their types and put them into "PutCodecTypeImpl()" Btw. Do you need to search through all the system assemblies (list of > 4000 elements)? Can't we just search for the attributes e.g. "[CodecName("Lucene46")]"? 2. Then it instantiated the Lucene46Codec but crashed at: "PostingsFormat defaultFormat = Codecs.PostingsFormat.ForName("Lucene41");" I think the crash originated from "ScanForPostingsFormats()" which has a very smiliar structure to the ScanForCodecs() function and also uses assembly.GetTypes(). So I guess I have to rewrite this function as well. Same for the DocValues... In general, reflection is working in partial trust mode, but not completely it seems. I'm not sure about what is/isn't working exactly. Best Regards, Matthias -----Original Message----- From: Shad Storhaug [mailto:[email protected]] Sent: Mittwoch, 17. Mai 2017 11:37 To: [email protected]; [email protected] Subject: RE: Problems when running a search in my production environment Matthias, There could be a couple of things happening here. The DefaultCodecFactory uses reflection first to get a list of all of the codecs, then it uses it again to instantiate the requested codec. To narrow it down, is the list at Codec.AvailableCodecs() populated with codecs or not? Could you post the values that you see there? Thanks, Shad Storhaug (NightOwl888) -----Original Message----- From: Matthias Strauss - xRM1 Business Solutions [mailto:[email protected]] Sent: Wednesday, May 17, 2017 4:21 PM To: [email protected] Cc: [email protected] Subject: RE: Problems when running a search in my production environment Hey Itamar, I'm still fighting with it. I got it to work in non-sandboxed mode by changing the following line in AttributeSource.cs from string name = attClass.FullName.Replace(attClass.Name, attClass.Name.Substring(1)) + ", " + attClass.GetTypeInfo().Assembly.FullName; to string name = attClass.FullName.Replace(attClass.Name, attClass.Name.Substring(1)); My simple search is working now, but not in sandboxed (partial trust) mode. There it fails to get the default Codec when initializing the IndexWriter. Do you have any ideas? Best Regards, Matthias -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Itamar Syn-Hershko Sent: Montag, 15. Mai 2017 17:45 To: [email protected] Subject: Re: Problems when running a search in my production environment Does this work on production for you without ILMerging? if yes, then that'd be the problem -- Itamar Syn-Hershko Freelance Developer & Consultant Elasticsearch Partner Microsoft MVP | Lucene.NET PMC http://code972.com | @synhershko <https://twitter.com/synhershko> http://BigDataBoutique.co.il/ On Mon, May 15, 2017 at 6:43 PM, Matthias Strauss - xRM1 Business Solutions <[email protected]> wrote: > 1. "Could not instantiate implementing class for Lucene.Net.Analysis. > TokenAttributes.ICharTermAttribute" > > For Sandbox mode: > 2. "The type initializer for 'Lucene.Net.Codecs.Codec' threw an > exception." at Lucene.Net.Codecs.Codec.get_Default() > "Unable to load one or more of the requested types. Retrieve > the LoaderExceptions property for more information." > > > -----Original Message----- > From: [email protected] [mailto:[email protected]] > On Behalf Of Itamar Syn-Hershko > Sent: Montag, 15. Mai 2017 17:20 > To: [email protected] > Subject: Re: Problems when running a search in my production > environment > > What is the exact exception(s) ? > > -- > > Itamar Syn-Hershko > Freelance Developer & Consultant > Elasticsearch Partner > Microsoft MVP | Lucene.NET PMC > http://code972.com | @synhershko <https://twitter.com/synhershko> > http://BigDataBoutique.co.il/ > > On Mon, May 15, 2017 at 6:02 PM, Matthias Strauss - xRM1 Business > Solutions <[email protected]> wrote: > > > Hey guys, > > > > I'm not sure if this is a thing for the user or dev mailing list, > > but > I've > > got a problem running a basic search in my production environment (CRM). > > > > > > 1. When adding a simple document to my IndexWriter I will get a > > runtime error in the following location: > > Lucene.Net\Util\AttributeSource.cs > > internal static Type GetClassForInterface<T>() where T : IAttribute > > It crashes at: "attClass.GetTypeInfo()" > > > > So I think this method is a feature of mscorlib.dll, but why is it > > crashing on my server and not on my local machine? > > > > > > 2. When running the code in sandbox mode (partially trusted code) > my > > sample app crashes even earlier when initializing the IndexWriter > > (with > > RamDirectory) at: > > Lucene.Net\Index\LiveIndexWriterConfig.cs > > internal > > LiveIndexWriterConfig(Analyzer analyzer, LuceneVersion matchVersion) > > Exception: "Unable to load one or more of the requested types. > > Retrieve the LoaderExceptions property for more information." at > > source > "mscorlib". > > > > Locally the code is working fine. > > I'm merging my code with ILMerge. Could this be the cause of the problem? > > When I merge the 4 basic Lucene.Net dll's into a single one, it's > > still working fine locally. > > >
