Re: [Dspace-tech] Solr stats data loss

[Peter Dietz]

Hi Anja,

One idea I have is that with solr, for performance reasons, we have an
auto-commit process where UsageEvents don't write/commit/persist into SOLR
until the commit gets triggered, so they live only in memory until
triggered to write.

...so... If these periods had a higher than normal, or perhaps even normal
occurrence of tomcat restarts, then perhaps pending documents are never
written, thus lost, upon restart.

Perhaps in the servlet container shutdown process, we could add something
to have it signal for dspace/solr to write/save/flush/persist the documents
before shutdown.

Off the top of my head I don't recall how I've written to the elastic
search API, but I'm assuming I never made these auto-commit / bulk / batch
submit changes since I never encountered performance issues with elastic
search. I'm guessing one UsageEvent equals one commit to Elastic Search, so
no data loss on shutdown.

This is just my guess of what could be happening. I suppose there could be
other explanations too, such as corrupt solr index, but I would guess that
would lose a greater amount of data. Another guess would be a server
migration that didn't sync all data properly... An unguarded solr index
that a mischievous user did a delete query... It's possibly possible that
solr and elastic search dspace-stats could have slightly different robot
rule processing (unlikely), so if your usage baseline was entirely robots,
then GoogleBot taking a few days off from crawling you could cause a
valley...

Stats is tricky, part of me wishes I just leveraged Google analytics for
everything, just to have one less system to manage. However I do like the
flexibility when you build it yourself.
On Apr 11, 2014 9:54 AM, "Anja Le Blanc" 
wrote:

> Hello All,
>
> (We are running on DSpace 1.8.2)
>
> I was looking at our stats data for the last year and a half and I
> noticed periodical drops in views/downloads which are inconsistent with
> the overall usage pattern. (I did not filter out bots for that
> exercise.) Numbers dropped for 1 to 5 days to below 10 and even to 0
> sometimes (from an average of about 5000 per day). I counted about 8
> such events since Jan 2013. (There are possibly more which don't stand
> out as much.) Our DSpace was always running and being monitored during
> that period.
>
> In our set-up we record stats in both Solr and ElasticSearch (at least
> we have done for the last half year). The data for ElasticSearch do not
> show drops for the days where Solr has data gaps. ElsaticSearch stats
> recording is triggered by the same DSpace events as Solr is.
>
> Unfortunately we have not kept log files for the periods with Solr data
> gaps.
>
> Has anyone else seen unexpected fluctuations in their stats?
> Anyone any idea of what could cause it. DSpace and Solr were running at
> the time since there are some data just not enough.
>
> To look at the data I use for views
>
> http://localhost:8080/solr/statistics/select/?q=type+%3A+2+&version=2.2&start=0&rows=0&indent=on&facet=true&facet.range=time&f.time.facet.range.start=2013-01-01T00:00:00Z&f.time.facet.range.gap=%2B1DAY&f.time.facet.range.end=2014-04-11T00:00:00Z
>
>
> downloads
>
> http://localhost:8080/solr/statistics/select/?q=type+%3A+0+&version=2.2&start=0&rows=0&indent=on&facet=true&facet.range=time&f.time.facet.range.start=2013-01-01T00:00:00Z&f.time.facet.range.gap=%2B1DAY&f.time.facet.range.end=2014-04-11T00:00:00Z
>
> Interestingly we can prove that there were more events.
>
> Any comments welcome :-)
>
> Best regards,
> Anja
>
>
> --
> Put Bad Developers to Shame
> Dominate Development with Jenkins Continuous Integration
> Continuously Automate Build, Test & Deployment
> Start a new project now. Try Jenkins in the cloud.
> http://p.sf.net/sfu/13600_Cloudbees
> ___
> DSpace-tech mailing list
> DSpace-tech@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspace-tech
> List Etiquette:
> https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
>
--
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment 
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

[Dspace-tech] Embargo bug?

[Bill Tantzen]

All,

In DSpace 4.1 xmlui, in a collection WITHOUT the embargo steps
enabled, I am able to upload a file as usual -- the only input field
being the description.  But during the review step, when editing the
just-uploaded file, the embargo options appear on the form.  Entering
data in either of these fields does nothing, but they should not be
there in any case, right?

A quick search of jira did not find a similar report.  Is this a bug,
or a mis-configuration of some kind?  I do have another collection in
which both embargo steps are enabled...

Comments?
Bill

--
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment 
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

Re: [Dspace-tech] Trouble with MoreLikeThis and Spellcheck in DSpace 4.1

[Brown, Jacob]

Hi all,

I've kind of figured out my Spellcheck issue, although I'm still very confused. 
I had my default pipe set to "noncaching" (in webapps/xmlui/sitemap.xmap):

 ...

When I switch this back to "caching", Spellcheck works as expected. Why would 
Spellcheck need to depend on a caching pipeline? Any ideas?

Thanks,

Jacob

From: Brown, Jacob [mailto:j.h.br...@tcu.edu]
Sent: Thursday, April 10, 2014 11:39 AM
To: dspace-tech@lists.sourceforge.net
Subject: [Dspace-tech] Trouble with MoreLikeThis and Spellcheck in DSpace 4.1

Hi all,

I'm having trouble getting the Discovery "More Like This" and "Spellcheck" 
features working with DSpace 4.1.

Discovery is enabled in xmlui.xconf, and Spellcheck and MoreLikeThis are 
enabled and properly configured in discovery.xml. Sidebar facets and the normal 
discovery search functions are working as expected.

Whenever I type in a misspelled search term (close to a match), I just get the 
"no results" message. When I look at the Solr requests, I don't see any 
"*spellcheck*" parameters in the URL (not sure if this is relevant).

When I visit an item page that should have related items, I don't see them, and 
there is no "item-related-container" div in the DRI XML document. I know that 
the item should have "MoreLikeThis" results, because when I load DSpace in a 
jRuby console and do something like:

item = org.dspace.handle.HandleManager.resolveToObject(context, 
'116099117/63')
dc = org.dspace.discovery.SearchUtils.getDiscoveryConfiguration(item)
related = 
org.dspace.discovery.SearchUtils.getSearchService().getRelatedItems(context, 
item, dc.getMoreLikeThisConfiguration())

I get `related.size == 3`.

I took a look at the Discovery sitemap.xmap and everything looks correct.

I'm very confused. Has anyone had similar issues or does anyone have any ideas 
about what could be going wrong?

Thanks,

Jacob Brown
Digital Services Librarian
j.h.br...@tcu.edu
817-257-5339

--
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment 
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

[Dspace-tech] Solr stats data loss

[Anja Le Blanc]

Hello All,

(We are running on DSpace 1.8.2)

I was looking at our stats data for the last year and a half and I 
noticed periodical drops in views/downloads which are inconsistent with 
the overall usage pattern. (I did not filter out bots for that 
exercise.) Numbers dropped for 1 to 5 days to below 10 and even to 0 
sometimes (from an average of about 5000 per day). I counted about 8 
such events since Jan 2013. (There are possibly more which don't stand 
out as much.) Our DSpace was always running and being monitored during 
that period.

In our set-up we record stats in both Solr and ElasticSearch (at least 
we have done for the last half year). The data for ElasticSearch do not 
show drops for the days where Solr has data gaps. ElsaticSearch stats 
recording is triggered by the same DSpace events as Solr is.

Unfortunately we have not kept log files for the periods with Solr data 
gaps.

Has anyone else seen unexpected fluctuations in their stats?
Anyone any idea of what could cause it. DSpace and Solr were running at 
the time since there are some data just not enough.

To look at the data I use for views
http://localhost:8080/solr/statistics/select/?q=type+%3A+2+&version=2.2&start=0&rows=0&indent=on&facet=true&facet.range=time&f.time.facet.range.start=2013-01-01T00:00:00Z&f.time.facet.range.gap=%2B1DAY&f.time.facet.range.end=2014-04-11T00:00:00Z
 


downloads
http://localhost:8080/solr/statistics/select/?q=type+%3A+0+&version=2.2&start=0&rows=0&indent=on&facet=true&facet.range=time&f.time.facet.range.start=2013-01-01T00:00:00Z&f.time.facet.range.gap=%2B1DAY&f.time.facet.range.end=2014-04-11T00:00:00Z

Interestingly we can prove that there were more events.

Any comments welcome :-)

Best regards,
Anja

--
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment 
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

Re: [Dspace-tech] Date of Collection in PostgreSQL

[Ruben]

Hi helix84,

I've tried to search "add_collection" in dspace.log and it doesn't show 
all added collections. For example, in January my repository have 6 new 
collections added (I extract this info with "select count(*) from 
collections " in postgresql database ) and searching for collection's id 
(the last collection id added in Decembre is 928 and the first in 
February is 934). If i search all "add_collection" in logs between 
1-01-2014 and 31-01-2014 only I obtain 2 new collections (932,933), so I 
can't believe the statistics extracted from logs.


Do you know why DSpace doesn't write all changes in logs?

Thanks for your help

On 13/03/14 13:40, helix84 wrote:

Hi Ruben,

you won't find the data in the database, DSpace simply doesn't keep 
it. You may find it in dspace.log if you keep those. Here's what it 
looks like (just grep for "add_collection"):


2014-03-13 13:38:07,979 INFO  org.dspace.content.Community @ 
ima...@knihovna.utb.cz:session_id=0EF722770C2BF281FAD8542AA68A0502:ip_addr=1.2.3.4:add_collection:community_id=91,collection_id=154



Regards,
~~helix84

Compulsory reading: DSpace Mailing List Etiquette
https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette



--
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech


___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette


--
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment 
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

[Dspace-tech] How to edit themes.xmap file for mobile theme?

[badboy]

I am trying to install mobile theme with Dspace4.1. according to the
introduction of readme.txt as following:

Installation:

0.  Get a new domain name that is an alias of the existing domain name
for your DSpace installation.
e.g. if your current domain is yoursite.edu your new domain name
might be mobile.yoursite.edu
These instructions assume that the new domain name starts with 'mobile.'
If it is something else, you will need to make a change in Step 5.

yes, I get a new domain which starts 'mobile'.

1.  Copy the mobile theme folder into your XMLUI theme folder
e.g. ../dspace/webapps/xmlui/themes/

in Dspace4.1, there is no need to copy. the files has already at the
directory.

2.  Add a call for the detectmobile.js and cookies.js file in the header
of your current main theme.xsl file.
It should look like:
 
 
* In this file, we also add a "View mobile site" link in the footer
section, which allows users to view the full site on their
mobile devices.  The cookies.js file saves this preference
but it's erased when the session is closed.
If you want such a link, the code for this should look like the
following
(with the appropriate URL for your mobile site):
http://mobile.
yoursite.edu';">
View mobile site

yes, i can edit the
webapps/xmlui/themes/Mirage/lib/xsl/core/page-structure.xsl to get

3.  Open the detectmobile.js file and enter your new mobile domain
at the end of the function call e.g. mobile.yoursite.edu
* if you choose a different domain name or theme name other than
"mobile" make sure to update the settings in the sitemap.xmap

4.  In mobile.xsl, find the link "View full website" and replace the
references to yoursite.edu with the domain name for your main site.
* lines 255-257

5.  Replace or edit the themes.xmap file located in your default theme
folder e.g. ../dspace/webapps/xmlui/themes/
* The code for setting up the properties for the domain is in
lines 32-37.  This will need to be changed if the domain name
for your mobile site starts with something other than 'mobile.'

6.  Restart tomcat and now you should be able to see the mobile theme
in action; to change the look-and-feel, you can go to
http://jquerymobile.com/themeroller/ and either create your own
files or import/upgrade the uncompressed sc-mobile.css file.


-- 
¤�q⌒�r �q⌒�r 相识也是缘份；
�u�� �q⌒�r再见亦是朋友。
�蛱铴蛱� 田| �t-
�p�p�p�p�p�p�p�p�p�p�p�p�p�p
--
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment 
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette