Re: [Dspace-tech] Error in importing Solr stats

[George Stanley Kozak]

Peter:

Thank you.  I made the setting changes you suggested.  I have another question 
for you and/or the community.  When I got the error in converting my dspace log 
files,  I ran the importer several times (since the error said “try again 
later”).  I am wondering if that messed up my statistics.  If so, is there a 
way to clear the stats and redo the importing?

George Kozak
Digital Library Specialist
Cornell University Library Information Technologies (CUL-IT)
501 Olin Library
Cornell University
Ithaca, NY 14853
607-255-8924

From: pdiet...@gmail.com [mailto:pdiet...@gmail.com] On Behalf Of Peter Dietz
Sent: Monday, September 20, 2010 10:55 PM
To: George Stanley Kozak
Cc: dspace-tech@lists.sourceforge.net
Subject: Re: [Dspace-tech] Error in importing Solr stats

Hi George,

I'm thinking that this was SOLR having an issue.

For some background, check out this thread: 
http://www.mail-archive.com/dspace-tech@lists.sourceforge.net/msg11062.html

The two imbalances currently going on are writing to solr:

 *   too frequently: usage events, which end user traffic hitting your site. 
possibly 10+ commits per second
 *   too infrequently: statistics importer. possibly too large at potentially 
millions of records from log importing in a single commit.

solr.add(document) is a cheap operation, as is just puts something in queue to 
be processed later.
solr.commit() is an expensive operation, which you want to call "just right".

Since we don't really know whats right, I suggest we let solr do whats good 
enough for it, and that would be autoCommit.

You can set that up with some defaults: 
/dspace/solr/statistics/conf/solrconfig.xml






maxDocs - number of updates since last commit is greater than this

maxTime - oldest uncommited update (in ms) is this long ago



  1

  1000



If you rely on autoCommit, we can then remove some of the 
handholding/lockstepping of solr.add(doc), immediately followed by 
solr.commit() which is terrible for solr performance (on busy instances).


So, check out activating autoCommit, and see if things improve. I would suggest 
that solr autocommit becomes the default for future releases of DSpace, so I'll 
need some agreement on this.

Peter Dietz
Systems Developer/Engineer
Ohio State University Libraries


On Mon, Sep 20, 2010 at 10:28 PM, George Stanley Kozak 
mailto:g...@cornell.edu>> wrote:
Hi...

I just migrated my production system to DSpace 1.6.2 from DSpace 1.5.2.  
Everything went well, until I did the stats-log-importer for the solr 
statistics.
It ran for a long time and then I received this error:

About to commit data to solr...Exception: Error opening new searcher.  exceeded 
limit of maxWarmingSearchers=2.  Try again later.

Does anyone know what caused this error?  I didn't see this when I upgraded my 
test system so I am assuming it has to do with the size of my production logs.  
IF so, what can I do to fix this?



George Kozak
Digital Library Specialist
Division of Library Information Technologies (DLIT)
501 Olin Library
Cornell University
Ithaca, NY 14853
607-255-8924
--
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech

--
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech

Re: [Dspace-tech] Mathematical formulas/equations in metadata fields of dspace

[Peter Dietz]

We have our science materials entered as LaTeX, so they might enter:
THE $1 ^{1}B^{+}_{u} \leftarrow 1 ^{1}A^{-}_{g}$ ABSORPTION OF JET-COOLED
TRANS,TRANS-1,3,5,7-OCTATETRAENE


Visitors could still type latex when searching for content, but on display
in the browser, a theme in development has javascript render everything
between dollar signs, and replace it with an image generated on the fly.
[image: screenshot_044.png]






Peter Dietz
Systems Developer/Engineer
Ohio State University Libraries



2010/9/21 Vlastimil Krejcir 

>   Hi Vishal,
>
>   we have already ecounter this and our mathematicians say, that inserting
> all the formulas etc. is good to be in TeX (or LaTeX) format. We are just
> working on converting these into MathML a on indexing it for fulltext
> searching. See http://dml.cz/.
>
>   Regards
>
>
> 
> Mgr. Vlastimil Krejčíř
> Library and Information Centre, Institute of Computer Science
> Masaryk University in Brno, Czech Republic
> Email: krejcir (at) ics (dot) muni (dot) cz
> Phone: +420 549 49 3872
> ICQ: 163963217
> Jabber: kre...@jabber.org
>
> 
>
> On Wed, 8 Sep 2010, vishal jindal wrote:
>
> > Hi,
> >
> > I want to insert mathematical formulas/equations, symbols in metadata
> fields of dspace. Is there any way to do this?
> >
> > Thanks and Regards,
> > Vishal Jindal
> >
> >
>
>
> --
> Start uncovering the many advantages of virtual appliances
> and start using them to simplify application deployment and
> accelerate your shift to cloud computing.
> http://p.sf.net/sfu/novell-sfdev2dev
> ___
> DSpace-tech mailing list
> DSpace-tech@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspace-tech
>
--
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech

Re: [Dspace-tech] tomcat reporting memory leak?

[Pottinger, Hardy J.]

Hi, Graham, for what it's worth, I'll stand with you. :-) I think addressing 
the issues you've discovered is really important. Here's an idea: how about 
some new unit and/or performance tests that check if a class and/or app is 
unloading cleanly? In other words, would it be possible to express the tests 
you have in such a way that they could be part of the new testing framework? 
Are there JIRA issues, and/or patches for what you have already found/fixed?

--Hardy 

> -Original Message-
> From: Graham Triggs [mailto:grahamtri...@gmail.com]
> Sent: Tuesday, September 21, 2010 6:52 AM
> To: Tom De Mulder
> Cc: dspace-tech@lists.sourceforge.net; Damian Marinaccio
> Subject: Re: [Dspace-tech] tomcat reporting memory leak?
> 
> On 20 September 2010 15:59, Tom De Mulder  wrote:
> 
> 
>   On Mon, 20 Sep 2010, Damian Marinaccio wrote:
> 
>   > I'm seeing the following log messages in catalina.out:
> 
>   > [...]
> 
>   > SEVERE: The web application [] appears to have started a thread
> named [FinalizableReferenceQueue] but has failed to stop it.
>   > This is very likely to create a memory leak.
> 
> 
>   There are quite a few memory leaks in DSpace. We have a cronjob to
> restart
>   Tomcat nightly, because otherwise it'll break the next day.
> 
> 
> 
> 
> Hi all,
> 
> Oh, welcome to my world!!
> 
> I'm going to start off by pointing out that the majority of DSpace code
> is actually quite well behaved. Going back to the codebase circa 1.4.2 /
> 1.5, and using the JSP user interface - I've got *thirty* spearate
> DSpace repositories / applications running in a single Tomcat instance,
> which has operated without a restart in over 90 days. And whilst be able
> to undeploy and redeploy any of those applications at will - or just
> reload them so that they pick up new configuration.
> 
> That does require a bit of careful setup / teardown in the context
> listeners (that wasn't always part of the DSpace code), and you need to
> get certain JARs - particularly the database/pooling drivers - out of
> the web applications entirely and into the shared level of Tomcat. Most
> of that is actually just good / recommended practise for systems
> administration of a Java application server anyway.
> 
> I was careful to point out that I have achieved that with pre-1.6 code
> and JSP only. Both 1.6 and XML ui (of any age) change the landscape. XML
> ui has always taken a large chunk of resources, although whilst it was
> still based on Cocoon 2.1, I managed to at least clean up it's startup /
> shutdown behaviour by repairing it's logging handler. This behaviour has
> changed with Cocoon 2.2, and I'll come back to that shortly.
> 
> So, 1.6 - I've been doing some work on the resource usage and clean
> loading/unloading of both JSP and XML using 1.6.2 recently, and neither
> are clean out of the box.
> 
> The first issue you run into is the FinalizableReferenceQueue noted in
> the stack trace above. This is coming from a reference map in
> reflectutils - and was found to be a cleanup problem in course of DSpace
> 2 development (the kernel / services framework was backported from that
> work). I added a LifecycleManager to reflectutils that was released as
> version 0.9.11 that allows the internal structures to be shutdown
> cleanly, and implemented this as part of DSpace 2, however this appears
> to have been ignored in the backport.
> 
> So, with the reflectutils/Lifecycle changes, and careful placement of
> JARs, etc. I did get the JSP ui to unload cleanly last week. I would
> note that I didn't stress the application too heavily, so there may be
> some operations that might trigger different code paths that are still a
> problem, but at the baseline it was working correctly.
> 
> XML ui has proven to be a somewhat more challenging beast. I first ran
> into two problems that are inside Cocoon 2.2 itself - 1) in the sitemap
> processing, it's using a stack inside a ThreadLocal, but it never
> removes the stack when it empties it, and 2) in one class relating to
> flowscript handling, it does not clean up the Mozilla Rhino engine
> correctly when it's finished using it (curiously, it's used in a number
> of places, and everywhere else it appears to be structured correctly to
> clean up - just this one class is screwed up).
> 
> With locally patched versions of the sitemap and flowscript JARs from
> Cocoon (the ThreadLocal patch isn't really guaranteed to not leak in
> unexpected circumstances - but it was sufficient to remove the problem
> in the scope of this testing. Basically, ThreadLocal is really dangerous
> to use), I then ran into another issue, this time with the
> CachingService that was backported.
> 
> With XML ui, it's using the RequestScope function of the caching service
> (it didn't appear to be exercising this part with JSP - that may just be
> because I only ran through limited code paths). For the RequestScope,
> it's tying the cache not to the request object... but to a 

[Dspace-tech] Recent submissions

[Biblio Sud]

Hi everybody.

I have a problem with xmlui interface, i would like to insert the "resent
submission" on the home page of my repository, but I don't know how put this
"recent submission" there..



Thanks for your help

-- 
Atte.
bibliosud
--
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech

[Dspace-tech] Browse For Dspace 1.5.2

[Hardik Mishra]

Hello Everyone

I am working on Dspace 1.5.2 Jspui.

I have some basic working knowledge of Dspace 1.4.2.

Please can u tell me that
What is the equivalent class of

import org.dspace.browse.Browse;

in Dspace 1.5.2


Thanks &  Regards
Hardik Mishra
Software Engineer
Webinito Networks

Give a man a fish, and you feed him for a day. Teach a man to catch fish and
you feed him for a lifetime.
--
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech

Re: [Dspace-tech] tomcat reporting memory leak?

[Graham Triggs]

On 20 September 2010 15:59, Tom De Mulder  wrote:

> On Mon, 20 Sep 2010, Damian Marinaccio wrote:
>
> > I'm seeing the following log messages in catalina.out:
> > [...]
> > SEVERE: The web application [] appears to have started a thread named
> [FinalizableReferenceQueue] but has failed to stop it.
> > This is very likely to create a memory leak.
>
> There are quite a few memory leaks in DSpace. We have a cronjob to restart
> Tomcat nightly, because otherwise it'll break the next day.
>


Hi all,

Oh, welcome to my world!!

I'm going to start off by pointing out that the majority of DSpace code is
actually quite well behaved. Going back to the codebase circa 1.4.2 / 1.5,
and using the JSP user interface - I've got *thirty* spearate DSpace
repositories / applications running in a single Tomcat instance, which has
operated without a restart in over 90 days. And whilst be able to undeploy
and redeploy any of those applications at will - or just reload them so that
they pick up new configuration.

That does require a bit of careful setup / teardown in the context listeners
(that wasn't always part of the DSpace code), and you need to get certain
JARs - particularly the database/pooling drivers - out of the web
applications entirely and into the shared level of Tomcat. Most of that is
actually just good / recommended practise for systems administration of a
Java application server anyway.

I was careful to point out that I have achieved that with pre-1.6 code and
JSP only. Both 1.6 and XML ui (of any age) change the landscape. XML ui has
always taken a large chunk of resources, although whilst it was still based
on Cocoon 2.1, I managed to at least clean up it's startup / shutdown
behaviour by repairing it's logging handler. This behaviour has changed with
Cocoon 2.2, and I'll come back to that shortly.

So, 1.6 - I've been doing some work on the resource usage and clean
loading/unloading of both JSP and XML using 1.6.2 recently, and neither are
clean out of the box.

The first issue you run into is the FinalizableReferenceQueue noted in the
stack trace above. This is coming from a reference map in reflectutils - and
was found to be a cleanup problem in course of DSpace 2 development (the
kernel / services framework was backported from that work). I added a
LifecycleManager to reflectutils that was released as version 0.9.11 that
allows the internal structures to be shutdown cleanly, and implemented this
as part of DSpace 2, however this appears to have been ignored in the
backport.

So, with the reflectutils/Lifecycle changes, and careful placement of JARs,
etc. I did get the JSP ui to unload cleanly last week. I would note that I
didn't stress the application too heavily, so there may be some operations
that might trigger different code paths that are still a problem, but at the
baseline it was working correctly.

XML ui has proven to be a somewhat more challenging beast. I first ran into
two problems that are inside Cocoon 2.2 itself - 1) in the sitemap
processing, it's using a stack inside a ThreadLocal, but it never removes
the stack when it empties it, and 2) in one class relating to flowscript
handling, it does not clean up the Mozilla Rhino engine correctly when it's
finished using it (curiously, it's used in a number of places, and
everywhere else it appears to be structured correctly to clean up - just
this one class is screwed up).

With locally patched versions of the sitemap and flowscript JARs from Cocoon
(the ThreadLocal patch isn't really guaranteed to not leak in unexpected
circumstances - but it was sufficient to remove the problem in the scope of
this testing. Basically, ThreadLocal is really dangerous to use), I then ran
into another issue, this time with the CachingService that was backported.

With XML ui, it's using the RequestScope function of the caching service (it
didn't appear to be exercising this part with JSP - that may just be because
I only ran through limited code paths). For the RequestScope, it's tying the
cache not to the request object... but to a ThreadLocal. And that
ThreadLocal isn't being cleaned up at the end of the request. (The shutdown
code is also incapable of doing the job it's intended for, as it will only
ever execute on a single thread, and not see all the other threads that may
have processed requests).

There is a high probability of this leaking memory all over the place, and
there is also the nasty potential of leak information across requests that
is undesirable.

I made another hacked version that removes the ThreadLocal, but replicates a
lot of it's thread affinity behaviour (so, it still has the nasty side
effects of the implementation, but at least removed the hold the system had
over the application resources). XML ui was *still* not unloading correctly,
and at this point the profiler stopped giving me pointers to strong
references that were being held. So right now I'm not sure what else is up -
but there is at least one more troubling part of the c

Re: [Dspace-tech] Mathematical formulas/equations in metadata fields of dspace

[Vlastimil Krejcir]

   Hi Vishal,

   we have already ecounter this and our mathematicians say, that inserting 
all the formulas etc. is good to be in TeX (or LaTeX) format. We are just 
working on converting these into MathML a on indexing it for fulltext 
searching. See http://dml.cz/.

   Regards


Mgr. Vlastimil Krejčíř
Library and Information Centre, Institute of Computer Science
Masaryk University in Brno, Czech Republic
Email: krejcir (at) ics (dot) muni (dot) cz
Phone: +420 549 49 3872
ICQ: 163963217
Jabber: kre...@jabber.org


On Wed, 8 Sep 2010, vishal jindal wrote:

> Hi,
>  
> I want to insert mathematical formulas/equations, symbols in metadata fields 
> of dspace. Is there any way to do this?
>  
> Thanks and Regards,
> Vishal Jindal
>
>

--
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech

[Dspace-tech] how subject search works ?

[AMJAD USMAN]

hi every bosy,
can anybody pliz tell me how subject search works ?
i mean to say that how DSpace uses taxonomy in subject searching ? 

From:
Amjad

  --
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech