[Dspace-devel] [DuraSpace JIRA] Commented: (DS-659) Collection.findAll performance issue - offset and limit improvement
[ https://jira.duraspace.org/browse/DS-659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18904#action_18904 ] Bojan Suzic commented on DS-659: Ok thanks for the comment. In meantime I have found there are similar issues with some other functions in DSpace API in relation to REST API translation. The problem is that it first loads a bunch of objects (whole collection) and display only first 20 or the number user requires. Although there is caching etc present, it is still not optimal in the regards to the performance and resources consumption. I got some complaints wrt this issue by REST API testers. I will consider then how it could reflect to other databases like Oracle. I am considering also to resolve this issue on the REST API level e.g. by writing new functions which will do DB querying (although it would be better to be handled by DSpace API). > Collection.findAll performance issue - offset and limit improvement > --- > > Key: DS-659 > URL: https://jira.duraspace.org/browse/DS-659 > Project: DSpace > Issue Type: Improvement > Components: DSpace API >Affects Versions: 1.7.0 >Reporter: Bojan Suzic >Priority: Trivial > Attachments: collection.java.add.txt > > Original Estimate: 1 hour > Remaining Estimate: 1 hour > > During the REST API development it has been noticed that in some cases > Collection.findAll may introduce performance issues, e.g. by working on big > collections. > For REST based browsing it is inefficient to preload big collection of say > 2 items and then return only first 100 items to the user. There are also > other examples. > In order to prevent those issues and optimize response I created two > additional findall functions making possible to use offset and limit clauses > during collections browsing. Also the simple function for returning number of > collections in the system is added. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.duraspace.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira -- The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE: Pinpoint memory and threading errors before they happen. Find and fix more than 250 security defects in the development cycle. Locate bottlenecks in serial and parallel code that limit performance. http://p.sf.net/sfu/intel-dev2devfeb ___ Dspace-devel mailing list Dspace-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-devel
[Dspace-devel] [DuraSpace JIRA] Commented: (DS-672) Nullpointer exception when bogus information is entered in the Embargo date field during submission
[ https://jira.duraspace.org/browse/DS-672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18906#action_18906 ] Mark H. Wood commented on DS-672: - What constitutes valid embargo terms is completely determined by the EmbargoSetter implementation configured. It doesn't have to look anything like a date. This makes it really difficult to do validation on the client side, as suggested in #duraspace. See https://wiki.duraspace.org/display/DSDOC/Embargo#Embargo-Embargomodelandlifecycle > Nullpointer exception when bogus information is entered in the Embargo date > field during submission > --- > > Key: DS-672 > URL: https://jira.duraspace.org/browse/DS-672 > Project: DSpace > Issue Type: Bug > Components: XMLUI >Affects Versions: 1.6.2 > Environment: The XMLUI on dspace.demo.org >Reporter: Bram Luyten (@mire) >Assignee: Mark H. Wood > > Under the submitter demo account I submitted an item in the Sample Items > collection. > I was able to enter something like "qsdf" in the embargo date field and > proceed until the last step of the submission. > When finishing the submission, following nullpointer is thrown: > An error has occurred > java.lang.NullPointerException: > Cocoon stacktrace [show] > Error calling continuation > context:/resource://aspects/Submission/submission.js - 538:-1 > processPage > context:/resource://aspects/Submission/submission.js - 474:-1 > doNextPage > context:/resource://aspects/Submission/sitemap.xmap - 129:44 > context:/resource://aspects/Submission/sitemap.xmap - 128:61 > context:/resource://aspects/Submission/sitemap.xmap - 96:48type="AuthenticatedSelector"> > context:/jndi:/localhost/xmlui/aspects/aspects.xmap - 119:72 > context:/jndi:/localhost/xmlui/aspects/aspects.xmap - 109:34 type="parameter"> > context:/jndi:/localhost/xmlui/aspects/aspects.xmap - 108:36 type="AspectMatcher"> > context:/jndi:/localhost/xmlui/sitemap.xmap - 430:108 > context:/jndi:/localhost/xmlui/sitemap.xmap - 429:53 > context:/resource://aspects/Statistics/sitemap.xmap - 70:31type="xml"> > context:/resource://aspects/Statistics/sitemap.xmap - 38:55type="Navigation"> > context:/resource://aspects/Statistics/sitemap.xmap - 33:19 > context:/jndi:/localhost/xmlui/aspects/aspects.xmap - 115:34 type="xml"> > context:/jndi:/localhost/xmlui/aspects/aspects.xmap - 114:43 type="PageNotFound"> > context:/jndi:/localhost/xmlui/aspects/aspects.xmap - 113:22 > context:/jndi:/localhost/xmlui/themes/Reference/sitemap.xmap - 184:62 > > context:/jndi:/localhost/xmlui/themes/Reference/sitemap.xmap - 181:112 > > context:/jndi:/localhost/xmlui/themes/Reference/sitemap.xmap - 180:106 > > context:/jndi:/localhost/xmlui/themes/Reference/sitemap.xmap - 174:68 > > context:/jndi:/localhost/xmlui/themes/Reference/sitemap.xmap - 170:72 > > context:/jndi:/localhost/xmlui/themes/Reference/sitemap.xmap - 155:87 > > context:/jndi:/localhost/xmlui/themes/Reference/sitemap.xmap - 131:83 > > context:/jndi:/localhost/xmlui/themes/Reference/sitemap.xmap - 128:49 > > context:/jndi:/localhost/xmlui/themes/themes.xmap - 63:45 > context:/jndi:/localhost/xmlui/themes/themes.xmap - 62:35 type="ThemeMatcher"> > context:/jndi:/localhost/xmlui/sitemap.xmap - 550:94 > Java stacktrace [hide] > java.lang.NullPointerException > at > org.dspace.embargo.EmbargoManager.getEmbargoDate(EmbargoManager.java:166) > at org.dspace.content.InstallItem.installItem(InstallItem.java:94) > at org.dspace.content.InstallItem.installItem(InstallItem.java:71) > at org.dspace.workflow.WorkflowManager.archive(WorkflowManager.java:660) > at org.dspace.workflow.WorkflowManager.doState(WorkflowManager.java:614) > at org.dspace.workflow.WorkflowManager.doState(WorkflowManager.java:590) > at org.dspace.workflow.WorkflowManager.doState(WorkflowManager.java:554) > at org.dspace.workflow.WorkflowManager.doState(WorkflowManager.java:514) > at org.dspace.workflow.WorkflowManager.start(WorkflowManager.java:199) > at > org.dspace.submit.step.CompleteStep.doProcessing(CompleteStep.java:116) > So either the submitter should be forced to either enter a correct date in > the submission step, or the bogus embargo date should be thrown out upon > archiving. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.duraspace.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira -- The ulti
[Dspace-devel] [DuraSpace JIRA] Commented: (DS-720) Solr statistics documentation in DSpace manual and DSDOC is out-of-date, wrong, and inconsistent with dspace.cfg
[ https://jira.duraspace.org/browse/DS-720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18907#action_18907 ] Kim Shepherd commented on DS-720: - Thanks Bram, the statistics documentation is looking great. I still can't see any reference to the following properties, and where/how they get used: statistics.items.dc.1=dc.identifier statistics.items.dc.2=dc.date.accessioned statistics.items.type.1=dcinput statistics.items.type.2=date statistics.default.start.datepick = 01/01/1977 > Solr statistics documentation in DSpace manual and DSDOC is out-of-date, > wrong, and inconsistent with dspace.cfg > > > Key: DS-720 > URL: https://jira.duraspace.org/browse/DS-720 > Project: DSpace > Issue Type: Documentation > Components: Documentation >Affects Versions: 1.6.0, 1.6.1, 1.6.2 >Reporter: Kim Shepherd >Priority: Major > Fix For: 1.7.1 > > > DSDOC: > == > - refers to solr.spidersfile instead of solr.spiderips.urls and gives an > invalid usage instruction (should be replaced with solr.spiderips.urls with > usage of org.dspace.statistics.util.StatisticsClient -u to download an update > of files from URLs given) > - misses out documentation for solr.statististics.query.filter.isBot and > solr.statistics.query.filter.spiderIp configuration properties > - misses out documentation for solr.statistics.logBots > - misses out documentation for solr.resolver.timeout > - misses out any documentation/usage examples for > org.dspace.statistics.util.StatisticsClient CLI utility > DSpace 1.6.2 Manual: > = > - Section "5.2.49. DSpace SOLR Statistics Configuration" suffers the same > problem as DSDOC wiki area > - Section 3.4.6. DSpace Statistics > (http://www.dspace.org/1_6_2Documentation/ch03.html#N10CF2) has correct > spider URLs config property usage, but also misses out the query filter, > logBots and resolver documentation > - Section 3.4.6. DSpace Statistics > (http://www.dspace.org/1_6_2Documentation/ch03.html#N10CF2) mentions the > following extra properties and claims they are commented-out in dspace.cfg, > but they aren't: > statistics.items.dc.1=dc.identifier > statistics.items.dc.2=dc.date.accessioned > statistics.items.type.1=dcinput > statistics.items.type.2=date > statistics.default.start.datepick = 01/01/1977 > - misses out any documentation/usage examples for > org.dspace.statistics.util.StatisticsClient CLI utility > Inconsistent dspace.cfg property naming > == > Some properties prefixed with statistics.* (particularly confusing when > configuring legacy stats reports) > Some properties prefixed with solr.* (ie. not solr.statistics.*) > Some properties prefixed with solr.statistics.* (this seems sensible and I > think should be the primary prefix for solr statistics properties) > Some properties have no prefix at all (eg. useProxies -- unless this is used > for something other than solr stats?) > The term "Spider" is used to denote a spider/crawler robot such as Google or > Yahoo, not "Bot", but terms are then mixed with property names like 'logBots' > and 'isBot' instead. (also appears in places in code) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.duraspace.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira -- The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE: Pinpoint memory and threading errors before they happen. Find and fix more than 250 security defects in the development cycle. Locate bottlenecks in serial and parallel code that limit performance. http://p.sf.net/sfu/intel-dev2devfeb ___ Dspace-devel mailing list Dspace-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-devel
