Re: [Dspace-tech] Solr stats queries - limiting by date

[Gary Browne]

Thanks a lot Terry, I'll check it out.

Gary Browne | Technical Manager Developments
University of Sydney Library
THE UNIVERSITY OF SYDNEY
Level 1, Fisher Library F03 | The University of Sydney | NSW | 2006
T +61 2 9351 5946 | M +61 405 647 868
E gary.bro...@sydney.edu.au
On 14/07/15 05:04, Terry Brady wrote:
Gary,

If it is helpful, I have several SOLR stats queries in the following code:

https://github.com/Georgetown-University-Libraries/batch-tools/blob/master/web/stats/solrFacets.php

I ran into some challenges knowing when to encode and when not to encode 
special characters.

Terry


On Wed, Jul 1, 2015 at 11:10 PM, Gary Browne 
mailto:gary.bro...@sydney.edu.au>> wrote:
Hi all,

I'm trying to figure out how to limit solr queries by date. I found this 
snippet online:

http://mydspace.org/solr/statistics/select?bundleName:ORIGINAL&facet.date.start=NOW/MONTH/DAY-6MONTHS&facet.date.end=NOW&facet.date.gap=1MONTH+AND+type:0+bundleName:ORIGINAL

it fails in my browser (yes, other solr searches work).

I'm trying to come up, for starters, with a solr search query for top X items 
visited in the last X months.

Can anyone point me in the right direction here, even to some documentation 
(I've done a lot of googling but nothing I've found so far helps with this 
question).

THanks,
Gary

--
Gary Browne | Development Programmer
University of Sydney Library
THE UNIVERSITY OF SYDNEY
Level 1, Fisher Library F03 | The University of Sydney | NSW | 2006
T +61 2 9351 5946 | M +61 405 647 
868
E gary.bro...@sydney.edu.au

--
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette



--
Terry Brady
Applications Programmer Analyst
Georgetown University Library Information Technology
https://www.library.georgetown.edu/lit/code
425-298-5498 (Seattle, WA)

--
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

Re: [Dspace-tech] Solr stats queries - limiting by date

[Gary Browne]

Thanks Monika,

I can see what this particular query is doing, however I'm wondering if you 
know of any resources for someone like me who is just starting out with solr 
queries - especially with reference to DSpace. I know the wiki has some 
examples but I thought there'd be some comprehensive documentation...somewhere?

For example, how do you know what the allowable values are for facet.field (and 
where can these be configured, and how do you configure them)? The same with 
"type" etc. And how do you know what the parameter names actually are (isBot, 
owningComm etc)?

Any clues would be very much appreciated.

Thanks again,
Gary


Gary Browne | Technical Manager Developments
University of Sydney Library
THE UNIVERSITY OF SYDNEY
Level 1, Fisher Library F03 | The University of Sydney | NSW | 2006
T +61 2 9351 5946 | M +61 405 647 868
E gary.bro...@sydney.edu.au
On 03/07/15 00:14, Monika C. Mevenkamp wrote:
use the fq parameter to restrict to a time range

eg   fq=time:[2015-01-01T00%3A00%3A00.000Z+TO+NOW]

the query below looks at ORIGINAL  bitstreams in a give collection downloaded 
at least once
excluding bot queries as well as query from the given ips and downloads from 
users that were logged in


http://localhost:18082/solr/statistics/select?wt=json&indent=true&rows=0&facet=true&facet.mincount=1&facet.field=id&fq=time:[2015-01-01T00%3A00%3A00.000Z+TO+NOW]&q=NOT+epersonid%3A%5B%22%22+TO+*%5D+isBot:false+owningComm:67+type:0+bundleName:ORIGINAL+-ip:%2810.6.61.26+OR+128.112.66.66%29

the response header


responseHeader":{
"status":0,
"QTime":1857,
"params":{
  "facet":"true",
  "facet.mincount":"1",
  "indent":"true",
  "q":"NOT epersonid:[\"\" TO *] isBot:false owningComm:67 type:0 
bundleName:ORIGINAL -ip:(10.6.61.26 OR 128.112.66.66)",
  "facet.field":"id",
  "wt":"json",
  "fq":"time:[2015-01-01T00:00:00.000Z TO NOW]",
  "rows":"0"}},
  "response":{"numFound":205152,"start":0,"docs":[]
  },

I wrote a jruby script to do stats for a particular community - if you are 
interested see https://github.com/akinom/dscriptor/tree/master/statistics

Monika

—
Monika Mevenkamp
Digital Repository Infrastructure Developer
Phone: 609-258-4161
333C 701 Carnegie, Princeton University, Princeton, NJ 08544

On Jul 2, 2015, at 2:10 AM, Gary Browne 
mailto:gary.bro...@sydney.edu.au>> wrote:

Hi all,

I'm trying to figure out how to limit solr queries by date. I found this 
snippet online:

http://mydspace.org/solr/statistics/select?bundleName:ORIGINAL&facet.date.start=NOW/MONTH/DAY-6MONTHS&facet.date.end=NOW&facet.date.gap=1MONTH+AND+type:0+bundleName:ORIGINAL

it fails in my browser (yes, other solr searches work).

I'm trying to come up, for starters, with a solr search query for top X items 
visited in the last X months.

Can anyone point me in the right direction here, even to some documentation 
(I've done a lot of googling but nothing I've found so far helps with this 
question).

THanks,
Gary

--
Gary Browne | Development Programmer
University of Sydney Library
THE UNIVERSITY OF SYDNEY
Level 1, Fisher Library F03 | The University of Sydney | NSW | 2006
T +61 2 9351 5946 | M +61 405 647 868
E gary.bro...@sydney.edu.au
--
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette


--
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

Re: [Dspace-tech] Solr stats queries - limiting by date

[Terry Brady]

Gary,

If it is helpful, I have several SOLR stats queries in the following code:

https://github.com/Georgetown-University-Libraries/batch-tools/blob/master/web/stats/solrFacets.php

I ran into some challenges knowing when to encode and when not to encode
special characters.

Terry


On Wed, Jul 1, 2015 at 11:10 PM, Gary Browne 
wrote:

>  Hi all,
>
> I'm trying to figure out how to limit solr queries by date. I found this
> snippet online:
>
>
> http://mydspace.org/solr/statistics/select?bundleName:ORIGINAL&facet.date.start=NOW/MONTH/DAY-6MONTHS&facet.date.end=NOW&facet.date.gap=1MONTH+AND+type:0+bundleName:ORIGINAL
>
> it fails in my browser (yes, other solr searches work).
>
> I'm trying to come up, for starters, with a solr search query for top X
> items visited in the last X months.
>
> Can anyone point me in the right direction here, even to some
> documentation (I've done a lot of googling but nothing I've found so far
> helps with this question).
>
> THanks,
> Gary
>
> --
> *Gary Browne* | Development Programmer
> University of Sydney Library
> THE UNIVERSITY OF SYDNEY
> Level 1, Fisher Library F03 | The University of Sydney | NSW | 2006
> *T* +61 2 9351 5946 | *M* +61 405 647 868
> *E* gary.bro...@sydney.edu.au
>
>
> --
> Don't Limit Your Business. Reach for the Cloud.
> GigeNET's Cloud Solutions provide you with the tools and support that
> you need to offload your IT needs and focus on growing your business.
> Configured For All Businesses. Start Your Cloud Today.
> https://www.gigenetcloud.com/
> ___
> DSpace-tech mailing list
> DSpace-tech@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspace-tech
> List Etiquette:
> https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
>



-- 
Terry Brady
Applications Programmer Analyst
Georgetown University Library Information Technology
https://www.library.georgetown.edu/lit/code
425-298-5498 (Seattle, WA)
--
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

Re: [Dspace-tech] Solr stats queries - limiting by date

[alastair.duncan]

Have a look at the differences between q and fq queries in solr. Monika suggest 
using the addition of fq and this can boost performance as it uses solr caches 
with no scoring and the q query uses a complex scoring mechanism to determine 
the relevance of a result so ordering of the result set may be different. The 
performance gains may be significant with fq and a combination of both of these 
can give the best performance and ordering that is required.

Alastair

From: Gary Browne [mailto:gary.bro...@sydney.edu.au]
Sent: 03 July 2015 00:06
To: Duncan, Alastair (STFC,RAL,SC); dspace-tech@lists.sourceforge.net
Subject: RE: Solr stats queries - limiting by date

Thanks a lot Alastair,

I'll try that.

Cheers,
Gary

From: alastair.dun...@stfc.ac.uk 
[alastair.dun...@stfc.ac.uk]
Sent: Thursday, 2 July 2015 6:32 PM
To: Gary Browne; 
dspace-tech@lists.sourceforge.net
Subject: RE: Solr stats queries - limiting by date
Hi Gary,

I'm not familiar with solr within DSpace but I've done some implementations 
with solr so the type of query you are looking to do will be something like 
this:

time:[NOW-6MONTH/MONTH TO NOW] AND bundleName:ORIGINAL

If you look at the schema fields the only one which is a date field is time so 
I'm guessing that this is the timestamp of the stats event and this gives me 
results that look about right.

http://localhost/solr/statistics/select?q=time%3A%5BNOW-6MONTH%2FMONTH+TO+NOW%5D+AND+bundleName%3AORIGINAL

I've no idea what field bundleName stores ;-)

Alastair

From: Gary Browne [mailto:gary.bro...@sydney.edu.au]
Sent: 02 July 2015 07:11
To: dspace-tech@lists.sourceforge.net
Subject: [Dspace-tech] Solr stats queries - limiting by date

Hi all,

I'm trying to figure out how to limit solr queries by date. I found this 
snippet online:

http://mydspace.org/solr/statistics/select?bundleName:ORIGINAL&facet.date.start=NOW/MONTH/DAY-6MONTHS&facet.date.end=NOW&facet.date.gap=1MONTH+AND+type:0+bundleName:ORIGINAL

it fails in my browser (yes, other solr searches work).

I'm trying to come up, for starters, with a solr search query for top X items 
visited in the last X months.

Can anyone point me in the right direction here, even to some documentation 
(I've done a lot of googling but nothing I've found so far helps with this 
question).

THanks,
Gary
--
Gary Browne | Development Programmer
University of Sydney Library
THE UNIVERSITY OF SYDNEY
Level 1, Fisher Library F03 | The University of Sydney | NSW | 2006
T +61 2 9351 5946 | M +61 405 647 868
E gary.bro...@sydney.edu.au
--
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

Re: [Dspace-tech] Solr stats queries - limiting by date

[Gary Browne]

Thanks a lot Alastair,

I'll try that.

Cheers,
Gary


From: alastair.dun...@stfc.ac.uk [alastair.dun...@stfc.ac.uk]
Sent: Thursday, 2 July 2015 6:32 PM
To: Gary Browne; dspace-tech@lists.sourceforge.net
Subject: RE: Solr stats queries - limiting by date

Hi Gary,

I’m not familiar with solr within DSpace but I’ve done some implementations 
with solr so the type of query you are looking to do will be something like 
this:

time:[NOW-6MONTH/MONTH TO NOW] AND bundleName:ORIGINAL

If you look at the schema fields the only one which is a date field is time so 
I’m guessing that this is the timestamp of the stats event and this gives me 
results that look about right.

http://localhost/solr/statistics/select?q=time%3A%5BNOW-6MONTH%2FMONTH+TO+NOW%5D+AND+bundleName%3AORIGINAL

I’ve no idea what field bundleName stores ;-)

Alastair

From: Gary Browne [mailto:gary.bro...@sydney.edu.au]
Sent: 02 July 2015 07:11
To: dspace-tech@lists.sourceforge.net
Subject: [Dspace-tech] Solr stats queries - limiting by date

Hi all,

I'm trying to figure out how to limit solr queries by date. I found this 
snippet online:

http://mydspace.org/solr/statistics/select?bundleName:ORIGINAL&facet.date.start=NOW/MONTH/DAY-6MONTHS&facet.date.end=NOW&facet.date.gap=1MONTH+AND+type:0+bundleName:ORIGINAL

it fails in my browser (yes, other solr searches work).

I'm trying to come up, for starters, with a solr search query for top X items 
visited in the last X months.

Can anyone point me in the right direction here, even to some documentation 
(I've done a lot of googling but nothing I've found so far helps with this 
question).

THanks,
Gary
--
Gary Browne | Development Programmer
University of Sydney Library
THE UNIVERSITY OF SYDNEY
Level 1, Fisher Library F03 | The University of Sydney | NSW | 2006
T +61 2 9351 5946 | M +61 405 647 868
E gary.bro...@sydney.edu.au
--
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

Re: [Dspace-tech] Solr stats queries - limiting by date

[Gary Browne]

Thanks very much Monika,

I'll give that a go. Thanks also for the link to your script, very useful.

Cheers,
Gary


From: Monika C. Mevenkamp [moni...@princeton.edu]
Sent: Friday, 3 July 2015 12:14 AM
To: Gary Browne
Cc: dspace-tech@lists.sourceforge.net
Subject: Re: [Dspace-tech] Solr stats queries - limiting by date

use the fq parameter to restrict to a time range

eg   fq=time:[2015-01-01T00%3A00%3A00.000Z+TO+NOW]

the query below looks at ORIGINAL  bitstreams in a give collection downloaded 
at least once
excluding bot queries as well as query from the given ips and downloads from 
users that were logged in


http://localhost:18082/solr/statistics/select?wt=json&indent=true&rows=0&facet=true&facet.mincount=1&facet.field=id&fq=time:[2015-01-01T00%3A00%3A00.000Z+TO+NOW]&q=NOT+epersonid%3A%5B%22%22+TO+*%5D+isBot:false+owningComm:67+type:0+bundleName:ORIGINAL+-ip:%2810.6.61.26+OR+128.112.66.66%29

the response header


responseHeader":{
"status":0,
"QTime":1857,
"params":{
  "facet":"true",
  "facet.mincount":"1",
  "indent":"true",
  "q":"NOT epersonid:[\"\" TO *] isBot:false owningComm:67 type:0 
bundleName:ORIGINAL -ip:(10.6.61.26 OR 128.112.66.66)",
  "facet.field":"id",
  "wt":"json",
  "fq":"time:[2015-01-01T00:00:00.000Z TO NOW]",
  "rows":"0"}},
  "response":{"numFound":205152,"start":0,"docs":[]
  },

I wrote a jruby script to do stats for a particular community - if you are 
interested see https://github.com/akinom/dscriptor/tree/master/statistics

Monika

—
Monika Mevenkamp
Digital Repository Infrastructure Developer
Phone: 609-258-4161
333C 701 Carnegie, Princeton University, Princeton, NJ 08544

On Jul 2, 2015, at 2:10 AM, Gary Browne 
mailto:gary.bro...@sydney.edu.au>> wrote:

Hi all,

I'm trying to figure out how to limit solr queries by date. I found this 
snippet online:

http://mydspace.org/solr/statistics/select?bundleName:ORIGINAL&facet.date.start=NOW/MONTH/DAY-6MONTHS&facet.date.end=NOW&facet.date.gap=1MONTH+AND+type:0+bundleName:ORIGINAL

it fails in my browser (yes, other solr searches work).

I'm trying to come up, for starters, with a solr search query for top X items 
visited in the last X months.

Can anyone point me in the right direction here, even to some documentation 
(I've done a lot of googling but nothing I've found so far helps with this 
question).

THanks,
Gary

--
Gary Browne | Development Programmer
University of Sydney Library
THE UNIVERSITY OF SYDNEY
Level 1, Fisher Library F03 | The University of Sydney | NSW | 2006
T +61 2 9351 5946 | M +61 405 647 868
E gary.bro...@sydney.edu.au<mailto:gary.bro...@sydney.edu.au>
--
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

--
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

Re: [Dspace-tech] Solr stats queries - limiting by date

[Monika C. Mevenkamp]

use the fq parameter to restrict to a time range

eg   fq=time:[2015-01-01T00%3A00%3A00.000Z+TO+NOW]

the query below looks at ORIGINAL  bitstreams in a give collection downloaded 
at least once
excluding bot queries as well as query from the given ips and downloads from 
users that were logged in


http://localhost:18082/solr/statistics/select?wt=json&indent=true&rows=0&facet=true&facet.mincount=1&facet.field=id&fq=time:[2015-01-01T00%3A00%3A00.000Z+TO+NOW]&q=NOT+epersonid%3A%5B%22%22+TO+*%5D+isBot:false+owningComm:67+type:0+bundleName:ORIGINAL+-ip:%2810.6.61.26+OR+128.112.66.66%29

the response header


responseHeader":{
"status":0,
"QTime":1857,
"params":{
  "facet":"true",
  "facet.mincount":"1",
  "indent":"true",
  "q":"NOT epersonid:[\"\" TO *] isBot:false owningComm:67 type:0 
bundleName:ORIGINAL -ip:(10.6.61.26 OR 128.112.66.66)",
  "facet.field":"id",
  "wt":"json",
  "fq":"time:[2015-01-01T00:00:00.000Z TO NOW]",
  "rows":"0"}},
  "response":{"numFound":205152,"start":0,"docs":[]
  },

I wrote a jruby script to do stats for a particular community - if you are 
interested see https://github.com/akinom/dscriptor/tree/master/statistics

Monika

—
Monika Mevenkamp
Digital Repository Infrastructure Developer
Phone: 609-258-4161
333C 701 Carnegie, Princeton University, Princeton, NJ 08544

On Jul 2, 2015, at 2:10 AM, Gary Browne 
mailto:gary.bro...@sydney.edu.au>> wrote:

Hi all,

I'm trying to figure out how to limit solr queries by date. I found this 
snippet online:

http://mydspace.org/solr/statistics/select?bundleName:ORIGINAL&facet.date.start=NOW/MONTH/DAY-6MONTHS&facet.date.end=NOW&facet.date.gap=1MONTH+AND+type:0+bundleName:ORIGINAL

it fails in my browser (yes, other solr searches work).

I'm trying to come up, for starters, with a solr search query for top X items 
visited in the last X months.

Can anyone point me in the right direction here, even to some documentation 
(I've done a lot of googling but nothing I've found so far helps with this 
question).

THanks,
Gary

--
Gary Browne | Development Programmer
University of Sydney Library
THE UNIVERSITY OF SYDNEY
Level 1, Fisher Library F03 | The University of Sydney | NSW | 2006
T +61 2 9351 5946 | M +61 405 647 868
E gary.bro...@sydney.edu.au
--
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

--
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

Re: [Dspace-tech] Solr stats queries - limiting by date

[alastair.duncan]

Hi Gary,

I'm not familiar with solr within DSpace but I've done some implementations 
with solr so the type of query you are looking to do will be something like 
this:

time:[NOW-6MONTH/MONTH TO NOW] AND bundleName:ORIGINAL

If you look at the schema fields the only one which is a date field is time so 
I'm guessing that this is the timestamp of the stats event and this gives me 
results that look about right.

http://localhost/solr/statistics/select?q=time%3A%5BNOW-6MONTH%2FMONTH+TO+NOW%5D+AND+bundleName%3AORIGINAL

I've no idea what field bundleName stores ;-)

Alastair

From: Gary Browne [mailto:gary.bro...@sydney.edu.au]
Sent: 02 July 2015 07:11
To: dspace-tech@lists.sourceforge.net
Subject: [Dspace-tech] Solr stats queries - limiting by date

Hi all,

I'm trying to figure out how to limit solr queries by date. I found this 
snippet online:

http://mydspace.org/solr/statistics/select?bundleName:ORIGINAL&facet.date.start=NOW/MONTH/DAY-6MONTHS&facet.date.end=NOW&facet.date.gap=1MONTH+AND+type:0+bundleName:ORIGINAL

it fails in my browser (yes, other solr searches work).

I'm trying to come up, for starters, with a solr search query for top X items 
visited in the last X months.

Can anyone point me in the right direction here, even to some documentation 
(I've done a lot of googling but nothing I've found so far helps with this 
question).

THanks,
Gary
--
Gary Browne | Development Programmer
University of Sydney Library
THE UNIVERSITY OF SYDNEY
Level 1, Fisher Library F03 | The University of Sydney | NSW | 2006
T +61 2 9351 5946 | M +61 405 647 868
E gary.bro...@sydney.edu.au
--
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

Re: [Dspace-tech] SOLR: stats-log-importer NullPointerException

[Anna Goslen]

Hi,

I'm having the same issue as Michael. Has anyone had success resolving the
issue?

Thanks,
Anna


On Mon, Sep 23, 2013 at 9:50 AM, helix84  wrote:

> While waiting for someone else to respond, you may want to check out
> ElasticSearch stats [1], which use a similar procedure, but the second
> step differs (stats-log-converter, stats-log-importer-elasticsearch).
> I've been using them successfully.
>
> Make sure to read my comments down on that page, especially use the
> workaround script for importing in smaller batches because importing
> many items is very slow. That should be fixed in 4.0.
>
> [1]
> https://wiki.duraspace.org/display/DSDOC3x/Elastic+Search+Usage+Statistics
>
>
> Regards,
> ~~helix84
>
> Compulsory reading: DSpace Mailing List Etiquette
> https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
>
>
> --
> LIMITED TIME SALE - Full Year of Microsoft Training For Just $49.99!
> 1,500+ hours of tutorials including VisualStudio 2012, Windows 8,
> SharePoint
> 2013, SQL 2012, MVC 4, more. BEST VALUE: New Multi-Library Power Pack
> includes
> Mobile, Cloud, Java, and UX Design. Lowest price ever! Ends 9/20/13.
> http://pubads.g.doubleclick.net/gampad/clk?id=58041151&iu=/4140/ostg.clktrk
> ___
> DSpace-tech mailing list
> DSpace-tech@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspace-tech
> List Etiquette:
> https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
>



-- 
Anna Goslen
Media Coordinator and Digital Initiatives Librarian
Swarthmore College Library
(610) 690-5733
agosl...@swarthmore.edu
--
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

Re: [Dspace-tech] Solr stats data loss

[Anja Le Blanc]

Hi Peter,

Thank you for your ideas to this problem.

I don't think the auto-commit can be the problem. There is some data 
just not enough. I looked at one of those problem-periods and looked at 
three stats occurrences which are one after the other in Solr
(e.g. 2014-04-04T08:53:38.027Z,  2014-04-04T08:53:38.112Z, 
2014-04-04T08:56:17.714Z) and compared this data with ElasticSearch 
stats. ES got 4 additional entries between the first and the second and 
67 additional ones between the second and the third entry. There could 
have been maximal one commit between the three Solr entries. (Auto 
commit is set to the default 15 min in our case.)

There were not any Tomcat restarts in the time frame and period of data 
loss lasted for three days that time.

There was not any data loss during the weekend, so I still have not got 
any log files.

Solr is running on one machine only and users can't access it directly - 
STATUS of solr confirms that there are no deletions in statistics core. 
If the index were corrupt I would not expect Solr to sort itself out on 
its own. Solr and ES use the same code to detect robots and there are 
still plenty of them in ES data. The load on the machine was not high 
enough to trigger any nagios errors.

I really have no idea what is happening.

We are working on the upgrade to DSpace 4 but are not there yet.
It is a pretty irritating problem - we partially justifying our 
existence as a service by showing that the community is using us. So 
knowing that there are gaps in the stats data is a problem.

Best regards,
Anja



On 12/04/2014 02:20, Peter Dietz wrote:
> Hi Anja,
>
> One idea I have is that with solr, for performance reasons, we have an
> auto-commit process where UsageEvents don't write/commit/persist into
> SOLR until the commit gets triggered, so they live only in memory until
> triggered to write.
>
> ...so... If these periods had a higher than normal, or perhaps even
> normal occurrence of tomcat restarts, then perhaps pending documents are
> never written, thus lost, upon restart.
>
> Perhaps in the servlet container shutdown process, we could add
> something to have it signal for dspace/solr to write/save/flush/persist
> the documents before shutdown.
>
> Off the top of my head I don't recall how I've written to the elastic
> search API, but I'm assuming I never made these auto-commit / bulk /
> batch submit changes since I never encountered performance issues with
> elastic search. I'm guessing one UsageEvent equals one commit to Elastic
> Search, so no data loss on shutdown.
>
> This is just my guess of what could be happening. I suppose there could
> be other explanations too, such as corrupt solr index, but I would guess
> that would lose a greater amount of data. Another guess would be a
> server migration that didn't sync all data properly... An unguarded solr
> index that a mischievous user did a delete query... It's possibly
> possible that solr and elastic search dspace-stats could have slightly
> different robot rule processing (unlikely), so if your usage baseline
> was entirely robots, then GoogleBot taking a few days off from crawling
> you could cause a valley...
>
> Stats is tricky, part of me wishes I just leveraged Google analytics for
> everything, just to have one less system to manage. However I do like
> the flexibility when you build it yourself.
>
> On Apr 11, 2014 9:54 AM, "Anja Le Blanc"  > wrote:
>
> Hello All,
>
> (We are running on DSpace 1.8.2)
>
> I was looking at our stats data for the last year and a half and I
> noticed periodical drops in views/downloads which are inconsistent with
> the overall usage pattern. (I did not filter out bots for that
> exercise.) Numbers dropped for 1 to 5 days to below 10 and even to 0
> sometimes (from an average of about 5000 per day). I counted about 8
> such events since Jan 2013. (There are possibly more which don't stand
> out as much.) Our DSpace was always running and being monitored during
> that period.
>
> In our set-up we record stats in both Solr and ElasticSearch (at least
> we have done for the last half year). The data for ElasticSearch do not
> show drops for the days where Solr has data gaps. ElsaticSearch stats
> recording is triggered by the same DSpace events as Solr is.
>
> Unfortunately we have not kept log files for the periods with Solr data
> gaps.
>
> Has anyone else seen unexpected fluctuations in their stats?
> Anyone any idea of what could cause it. DSpace and Solr were running at
> the time since there are some data just not enough.
>
> To look at the data I use for views
> 
> http://localhost:8080/solr/statistics/select/?q=type+%3A+2+&version=2.2&start=0&rows=0&indent=on&facet=true&facet.range=time&f.time.facet.range.start=2013-01-01T00:00:00Z&f.time.facet.range.gap=%2B1DAY&f.time.facet.range.end=2014-04-11T00:00:00Z
>
>
> downloads
> 
> h

Re: [Dspace-tech] Solr stats data loss

[Peter Dietz]

Hi Anja,

One idea I have is that with solr, for performance reasons, we have an
auto-commit process where UsageEvents don't write/commit/persist into SOLR
until the commit gets triggered, so they live only in memory until
triggered to write.

...so... If these periods had a higher than normal, or perhaps even normal
occurrence of tomcat restarts, then perhaps pending documents are never
written, thus lost, upon restart.

Perhaps in the servlet container shutdown process, we could add something
to have it signal for dspace/solr to write/save/flush/persist the documents
before shutdown.

Off the top of my head I don't recall how I've written to the elastic
search API, but I'm assuming I never made these auto-commit / bulk / batch
submit changes since I never encountered performance issues with elastic
search. I'm guessing one UsageEvent equals one commit to Elastic Search, so
no data loss on shutdown.

This is just my guess of what could be happening. I suppose there could be
other explanations too, such as corrupt solr index, but I would guess that
would lose a greater amount of data. Another guess would be a server
migration that didn't sync all data properly... An unguarded solr index
that a mischievous user did a delete query... It's possibly possible that
solr and elastic search dspace-stats could have slightly different robot
rule processing (unlikely), so if your usage baseline was entirely robots,
then GoogleBot taking a few days off from crawling you could cause a
valley...

Stats is tricky, part of me wishes I just leveraged Google analytics for
everything, just to have one less system to manage. However I do like the
flexibility when you build it yourself.
On Apr 11, 2014 9:54 AM, "Anja Le Blanc" 
wrote:

> Hello All,
>
> (We are running on DSpace 1.8.2)
>
> I was looking at our stats data for the last year and a half and I
> noticed periodical drops in views/downloads which are inconsistent with
> the overall usage pattern. (I did not filter out bots for that
> exercise.) Numbers dropped for 1 to 5 days to below 10 and even to 0
> sometimes (from an average of about 5000 per day). I counted about 8
> such events since Jan 2013. (There are possibly more which don't stand
> out as much.) Our DSpace was always running and being monitored during
> that period.
>
> In our set-up we record stats in both Solr and ElasticSearch (at least
> we have done for the last half year). The data for ElasticSearch do not
> show drops for the days where Solr has data gaps. ElsaticSearch stats
> recording is triggered by the same DSpace events as Solr is.
>
> Unfortunately we have not kept log files for the periods with Solr data
> gaps.
>
> Has anyone else seen unexpected fluctuations in their stats?
> Anyone any idea of what could cause it. DSpace and Solr were running at
> the time since there are some data just not enough.
>
> To look at the data I use for views
>
> http://localhost:8080/solr/statistics/select/?q=type+%3A+2+&version=2.2&start=0&rows=0&indent=on&facet=true&facet.range=time&f.time.facet.range.start=2013-01-01T00:00:00Z&f.time.facet.range.gap=%2B1DAY&f.time.facet.range.end=2014-04-11T00:00:00Z
>
>
> downloads
>
> http://localhost:8080/solr/statistics/select/?q=type+%3A+0+&version=2.2&start=0&rows=0&indent=on&facet=true&facet.range=time&f.time.facet.range.start=2013-01-01T00:00:00Z&f.time.facet.range.gap=%2B1DAY&f.time.facet.range.end=2014-04-11T00:00:00Z
>
> Interestingly we can prove that there were more events.
>
> Any comments welcome :-)
>
> Best regards,
> Anja
>
>
> --
> Put Bad Developers to Shame
> Dominate Development with Jenkins Continuous Integration
> Continuously Automate Build, Test & Deployment
> Start a new project now. Try Jenkins in the cloud.
> http://p.sf.net/sfu/13600_Cloudbees
> ___
> DSpace-tech mailing list
> DSpace-tech@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspace-tech
> List Etiquette:
> https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
>
--
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment 
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

Re: [Dspace-tech] SOLR: stats-log-importer NullPointerException

[helix84]

While waiting for someone else to respond, you may want to check out
ElasticSearch stats [1], which use a similar procedure, but the second
step differs (stats-log-converter, stats-log-importer-elasticsearch).
I've been using them successfully.

Make sure to read my comments down on that page, especially use the
workaround script for importing in smaller batches because importing
many items is very slow. That should be fixed in 4.0.

[1] https://wiki.duraspace.org/display/DSDOC3x/Elastic+Search+Usage+Statistics


Regards,
~~helix84

Compulsory reading: DSpace Mailing List Etiquette
https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

--
LIMITED TIME SALE - Full Year of Microsoft Training For Just $49.99!
1,500+ hours of tutorials including VisualStudio 2012, Windows 8, SharePoint
2013, SQL 2012, MVC 4, more. BEST VALUE: New Multi-Library Power Pack includes
Mobile, Cloud, Java, and UX Design. Lowest price ever! Ends 9/20/13. 
http://pubads.g.doubleclick.net/gampad/clk?id=58041151&iu=/4140/ostg.clktrk
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

Re: [Dspace-tech] SOLR: stats-log-importer NullPointerException

[Michael White]

Hi helix84,

> I see nothing obviously wrong in what you're doing. The high-level procedure
> is correct.

OK, thanks - always good to know that I'm on the right track at least and not 
doing anything obviously stupid :-)

> The NullPointerException happens on this line [1], which indicates that
> metadataStorageInfo.keySet() is null. But I don't see where
> metadataStorageInfo is actually being set in the whole DSpace codebase. So I
> don't understand it myself.

OK, fair enough - I also had a quick peek at the code, but I'm not familiar 
enough with it to pick it apart ...

And just as a sanity check, have others successfully used the 
stats-log-importer in v3.1 to import ("old format") log files (i.e. is this a 
general problem, or am I just lucky)?

If anyone else has any thoughts or insights, they are, of course, most welcome!

Cheers,

Mike 

Michael White 
eLearning Liaison and Development (eLD)
Information Services
S8, Library
University of Stirling 
Stirling SCOTLAND 
FK9 4LA 
Email: michael.wh...@stir.ac.uk 
Tel: +44 (0) 1786 466877 
Fax: +44 (0) 1786 466880
http://www.stir.ac.uk/is/staff/about/teams/aldt/#eld


-- 
The University of Stirling has been ranked in the top 12 of UK universities for 
graduate employment*.
94% of our 2012 graduates were in work and/or further study within six months 
of graduation.
*The Telegraph
The University of Stirling is a charity registered in Scotland, number SC 
011159.

--
LIMITED TIME SALE - Full Year of Microsoft Training For Just $49.99!
1,500+ hours of tutorials including VisualStudio 2012, Windows 8, SharePoint
2013, SQL 2012, MVC 4, more. BEST VALUE: New Multi-Library Power Pack includes
Mobile, Cloud, Java, and UX Design. Lowest price ever! Ends 9/20/13. 
http://pubads.g.doubleclick.net/gampad/clk?id=58041151&iu=/4140/ostg.clktrk
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

Re: [Dspace-tech] SOLR: stats-log-importer NullPointerException

[helix84]

Hi Michael,

no solution, just some notes.

I see nothing obviously wrong in what you're doing. The high-level
procedure is correct.

The NullPointerException happens on this line [1], which indicates
that metadataStorageInfo.keySet() is null. But I don't see where
metadataStorageInfo is actually being set in the whole DSpace
codebase. So I don't understand it myself.


[1] 
https://github.com/DSpace/DSpace/blob/dspace-3.1/dspace-api/src/main/java/org/dspace/statistics/util/StatisticsImporter.java#L343


Regards,
~~helix84

Compulsory reading: DSpace Mailing List Etiquette
https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

--
LIMITED TIME SALE - Full Year of Microsoft Training For Just $49.99!
1,500+ hours of tutorials including VisualStudio 2012, Windows 8, SharePoint
2013, SQL 2012, MVC 4, more. BEST VALUE: New Multi-Library Power Pack includes
Mobile, Cloud, Java, and UX Design. Lowest price ever! Ends 9/20/13. 
http://pubads.g.doubleclick.net/gampad/clk?id=58041151&iu=/4140/ostg.clktrk
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

Re: [Dspace-tech] solr stats query

[Terry Brady]

Jose,

Try the following
https://localhost/solr/statistics/select?indent=on&fl=*,score&rows=10&q=type:0+AND+bundleName:ORIGINAL+AND+time:[NOW-6MONTHS+TO+NOW]

If you do not need faceted results, the query is a bit easier to construct.

Terry


On Thu, Aug 29, 2013 at 3:43 PM, Jose Blanco  wrote:

> From looking at your code and not really knowing what the URL is
> suppose to look like, I created this URL which gave me an error:
>
>
> http://dev.deepblue.lib.umich.edu/solr/statistics/select?bundleName:ORIGINAL&facet.date.start=NOW/MONTH/DAY-6MONTHS&facet.date.end=NOW&facet.date.gap=1MONTH+AND+type:0+bundleName:ORIGINAL
>
> I thought I was asking for bitstream stats for the last six months.
>
> On Thu, Aug 29, 2013 at 1:19 PM, Terry Brady  wrote:
> > Jose,
> >
> > The following code illustrates the options that Georgetown has provided
> for
> > filtering our statistics.  The code builds up a query string that might
> help
> > with your issue.
> >
> > The following file demonstrates how I constructed date range queries and
> > queries for original bitstream downloads.
> >
> >
> https://github.com/Georgetown-University-Libraries/DSpaceQCTools/blob/master/dspace-tools/web/solrFacets.php
> >
> > I was not satisfied with the existing bot detection.  This code shows
> some
> > additional filters that I put into place to filter out bots.
> >
> >
> https://github.com/Georgetown-University-Libraries/DSpaceQCTools/blob/master/dspace-tools/web/solrStats.php
> >
> > Terry
> >
> >
> >
> > On Thu, Aug 29, 2013 at 12:22 PM, Jose Blanco  wrote:
> >>
> >> I would like to create a solr query that would tell me how many
> >> bitstreams were downloaded from my repository in a particular month.
> >> I would want the query NOT to include the spider downloads.
> >>
> >> So say, how many bitstreams were downloaded from my repository in
> >> August, not including things downloaded by spiders.
> >>
> >> Working from this as an example, I don't see how to come up with the
> >> query.
> >>
> >> Top downloaded items by a specific user:
> >>
> >>
> >>
> >>
> http://localhost/solr/statistics/select?indent=on&version=2.2&start=0&rows=10&fl=*%2Cscore&qt=standard&wt=standard&explainOther=&hl.fl=&facet=true&facet.field=epersonid&q=type:0
> >>
> >> Thank you!
> >> Jose
> >>
> >>
> >>
> --
> >> Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
> >> Discover the easy way to master current and previous Microsoft
> >> technologies
> >> and advance your career. Get an incredible 1,500+ hours of step-by-step
> >> tutorial videos with LearnDevNow. Subscribe today and save!
> >>
> >>
> http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk
> >> ___
> >> DSpace-tech mailing list
> >> DSpace-tech@lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/dspace-tech
> >> List Etiquette:
> >> https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
> >
> >
> >
> >
> > --
> > Terry Brady
> > Applications Programmer Analyst
> > Lauinger Information Technology
> > 202-687-7053
> >
> >
> --
> > Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
> > Discover the easy way to master current and previous Microsoft
> technologies
> > and advance your career. Get an incredible 1,500+ hours of step-by-step
> > tutorial videos with LearnDevNow. Subscribe today and save!
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk
> > ___
> > DSpace-tech mailing list
> > DSpace-tech@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/dspace-tech
> > List Etiquette:
> > https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
>



-- 
Terry Brady
Applications Programmer Analyst
Lauinger Information Technology
202-687-7053
--
Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
Discover the easy way to master current and previous Microsoft technologies
and advance your career. Get an incredible 1,500+ hours of step-by-step
tutorial videos with LearnDevNow. Subscribe today and save!
http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

Re: [Dspace-tech] solr stats query

[Jose Blanco]

>From looking at your code and not really knowing what the URL is
suppose to look like, I created this URL which gave me an error:

http://dev.deepblue.lib.umich.edu/solr/statistics/select?bundleName:ORIGINAL&facet.date.start=NOW/MONTH/DAY-6MONTHS&facet.date.end=NOW&facet.date.gap=1MONTH+AND+type:0+bundleName:ORIGINAL

I thought I was asking for bitstream stats for the last six months.

On Thu, Aug 29, 2013 at 1:19 PM, Terry Brady  wrote:
> Jose,
>
> The following code illustrates the options that Georgetown has provided for
> filtering our statistics.  The code builds up a query string that might help
> with your issue.
>
> The following file demonstrates how I constructed date range queries and
> queries for original bitstream downloads.
>
> https://github.com/Georgetown-University-Libraries/DSpaceQCTools/blob/master/dspace-tools/web/solrFacets.php
>
> I was not satisfied with the existing bot detection.  This code shows some
> additional filters that I put into place to filter out bots.
>
> https://github.com/Georgetown-University-Libraries/DSpaceQCTools/blob/master/dspace-tools/web/solrStats.php
>
> Terry
>
>
>
> On Thu, Aug 29, 2013 at 12:22 PM, Jose Blanco  wrote:
>>
>> I would like to create a solr query that would tell me how many
>> bitstreams were downloaded from my repository in a particular month.
>> I would want the query NOT to include the spider downloads.
>>
>> So say, how many bitstreams were downloaded from my repository in
>> August, not including things downloaded by spiders.
>>
>> Working from this as an example, I don't see how to come up with the
>> query.
>>
>> Top downloaded items by a specific user:
>>
>>
>>
>> http://localhost/solr/statistics/select?indent=on&version=2.2&start=0&rows=10&fl=*%2Cscore&qt=standard&wt=standard&explainOther=&hl.fl=&facet=true&facet.field=epersonid&q=type:0
>>
>> Thank you!
>> Jose
>>
>>
>> --
>> Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
>> Discover the easy way to master current and previous Microsoft
>> technologies
>> and advance your career. Get an incredible 1,500+ hours of step-by-step
>> tutorial videos with LearnDevNow. Subscribe today and save!
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk
>> ___
>> DSpace-tech mailing list
>> DSpace-tech@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/dspace-tech
>> List Etiquette:
>> https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
>
>
>
>
> --
> Terry Brady
> Applications Programmer Analyst
> Lauinger Information Technology
> 202-687-7053
>
> --
> Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
> Discover the easy way to master current and previous Microsoft technologies
> and advance your career. Get an incredible 1,500+ hours of step-by-step
> tutorial videos with LearnDevNow. Subscribe today and save!
> http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk
> ___
> DSpace-tech mailing list
> DSpace-tech@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspace-tech
> List Etiquette:
> https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

--
Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
Discover the easy way to master current and previous Microsoft technologies
and advance your career. Get an incredible 1,500+ hours of step-by-step
tutorial videos with LearnDevNow. Subscribe today and save!
http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

Re: [Dspace-tech] solr stats query

[Terry Brady]

Jose,

The following code illustrates the options that Georgetown has provided for
filtering our statistics.  The code builds up a query string that might
help with your issue.

The following file demonstrates how I constructed date range queries and
queries for original bitstream downloads.

https://github.com/Georgetown-University-Libraries/DSpaceQCTools/blob/master/dspace-tools/web/solrFacets.php

I was not satisfied with the existing bot detection.  This code shows some
additional filters that I put into place to filter out bots.

https://github.com/Georgetown-University-Libraries/DSpaceQCTools/blob/master/dspace-tools/web/solrStats.php

Terry



On Thu, Aug 29, 2013 at 12:22 PM, Jose Blanco  wrote:

> I would like to create a solr query that would tell me how many
> bitstreams were downloaded from my repository in a particular month.
> I would want the query NOT to include the spider downloads.
>
> So say, how many bitstreams were downloaded from my repository in
> August, not including things downloaded by spiders.
>
> Working from this as an example, I don't see how to come up with the query.
>
> Top downloaded items by a specific user:
>
>
>
> http://localhost/solr/statistics/select?indent=on&version=2.2&start=0&rows=10&fl=*%2Cscore&qt=standard&wt=standard&explainOther=&hl.fl=&facet=true&facet.field=epersonid&q=type:0
>
> Thank you!
> Jose
>
>
> --
> Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
> Discover the easy way to master current and previous Microsoft technologies
> and advance your career. Get an incredible 1,500+ hours of step-by-step
> tutorial videos with LearnDevNow. Subscribe today and save!
> http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk
> ___
> DSpace-tech mailing list
> DSpace-tech@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspace-tech
> List Etiquette:
> https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
>



-- 
Terry Brady
Applications Programmer Analyst
Lauinger Information Technology
202-687-7053
--
Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
Discover the easy way to master current and previous Microsoft technologies
and advance your career. Get an incredible 1,500+ hours of step-by-step
tutorial videos with LearnDevNow. Subscribe today and save!
http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

Re: [Dspace-tech] Solr Stats: Countries and IPs

[Sean Carte]

On 15 November 2012 10:36, helix84  wrote:
>> Could my GeoLiteCity.dat have become corrupted somehow? Will it work
>> if I swap it out with a more recent one from another installation?
>
> Sure, you can do that. Or just get the latest version from here and unzip it:
>
> http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz

OK. Done that. It is a much smaller file:

-rw-r--r-- 1 tomcat6 tomcat6 18205777 2012-11-15 11:03 GeoLiteCity.dat
-rw-r--r-- 1 tomcat6 tomcat6 45701644 2012-03-22 13:45 GeoLiteCity.dat.new

I'll keep an eye on it.

Thanks for the input, helix84.

Sean
-- 
Sean Carte
DUT E-Repository Administrator
+27 72 898 8775
+27 31 373 5761
fax: +27 86 674 1254
http://ir.dut.ac.za

--
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech

Re: [Dspace-tech] Solr Stats: Countries and IPs

[helix84]

On Thu, Nov 15, 2012 at 9:01 AM, Sean Carte  wrote:
> Thanks, helix84, but I *hope* private IP addresses aren't listed in
> there. I had a look at one of the files: GeoIPCountryCSV and that
> doesn't contain private ranges as far as I can tell.

Obviously, that's a wrong thing to do. But I cannot immagine where
else that info would come from. If it's from there, it would be best
to report it to them.

> Could my GeoLiteCity.dat have become corrupted somehow? Will it work
> if I swap it out with a more recent one from another installation?

Sure, you can do that. Or just get the latest version from here and unzip it:

http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz


Regards,
~~helix84

Compulsory reading: DSpace Mailing List Etiquette
https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

--
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech

Re: [Dspace-tech] Solr Stats: Countries and IPs

[Sean Carte]

On 14 November 2012 13:56, helix84  wrote:
> Hi Sean,
>
> the only place I can think of where it may come from is the GeoIP
> database ([dspace]/config/GeoLiteCity.dat). Try looking here if you
> can find it:
>
> http://www.maxmind.com/download/geoip/database/
>
>
> Regards,
> ~~helix84

Thanks, helix84, but I *hope* private IP addresses aren't listed in
there. I had a look at one of the files: GeoIPCountryCSV and that
doesn't contain private ranges as far as I can tell.

Could my GeoLiteCity.dat have become corrupted somehow? Will it work
if I swap it out with a more recent one from another installation?

Sean
-- 
Sean Carte
DUT E-Repository Administrator
+27 72 898 8775
+27 31 373 5761
fax: +27 86 674 1254
http://ir.dut.ac.za

--
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech

Re: [Dspace-tech] Solr Stats: Countries and IPs

[helix84]

Hi Sean,

the only place I can think of where it may come from is the GeoIP
database ([dspace]/config/GeoLiteCity.dat). Try looking here if you
can find it:

http://www.maxmind.com/download/geoip/database/


Regards,
~~helix84

Compulsory reading: DSpace Mailing List Etiquette
https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

--
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech

Re: [Dspace-tech] SOLR Stats("GeoLiteCity.dat")

[Lewatle Phaladi]

It is working!!

Thanks,
Lewatle 

-Original Message-
From: helix84 [mailto:heli...@centrum.sk] 
Sent: 31 July 2012 03:51 PM
To: Lewatle Phaladi
Cc: DSpace-tech@lists.sourceforge.net; durasp...@lists.lib.sun.ac.za
Subject: Re: [Dspace-tech] SOLR Stats("GeoLiteCity.dat")

GeoLiteCity.dat should be downloaded automatically by maven during the build 
process. If it's not, you can get it manually:
cd [dspace]/config/
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
gzip -d GeoLiteCity.dat.gz

Then restart Tomcat.

Regards,
~~helix84

This communication 
is intended for the addressee only. It is confidential. If you have received 
this communication in error, please notify us immediately and destroy the 
original message. You may not copy or disseminate this communication without 
the permission of the University. Only authorized signatories are competent to 
enter into agreements on behalf of the University and recipients are thus 
advised that the content of this message may not be legally binding on the 
University and may contain the personal views and opinions of the author, which 
are not necessarily the views and opinions of The University of the 
Witwatersrand, Johannesburg. All agreements between the University and 
outsiders are subject to South African Law unless the University agrees in 
writing to the contrary.
--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech

Re: [Dspace-tech] SOLR Stats("GeoLiteCity.dat")

[Lewatle Phaladi]

Thanks Helix.

-Original Message-
From: helix84 [mailto:heli...@centrum.sk] 
Sent: 31 July 2012 03:51 PM
To: Lewatle Phaladi
Cc: DSpace-tech@lists.sourceforge.net; durasp...@lists.lib.sun.ac.za
Subject: Re: [Dspace-tech] SOLR Stats("GeoLiteCity.dat")

GeoLiteCity.dat should be downloaded automatically by maven during the build 
process. If it's not, you can get it manually:
cd [dspace]/config/
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
gzip -d GeoLiteCity.dat.gz

Then restart Tomcat.

Regards,
~~helix84

This communication 
is intended for the addressee only. It is confidential. If you have received 
this communication in error, please notify us immediately and destroy the 
original message. You may not copy or disseminate this communication without 
the permission of the University. Only authorized signatories are competent to 
enter into agreements on behalf of the University and recipients are thus 
advised that the content of this message may not be legally binding on the 
University and may contain the personal views and opinions of the author, which 
are not necessarily the views and opinions of The University of the 
Witwatersrand, Johannesburg. All agreements between the University and 
outsiders are subject to South African Law unless the University agrees in 
writing to the contrary.
--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech

Re: [Dspace-tech] SOLR Stats("GeoLiteCity.dat")

[helix84]

GeoLiteCity.dat should be downloaded automatically by maven during the
build process. If it's not, you can get it manually:
cd [dspace]/config/
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
gzip -d GeoLiteCity.dat.gz

Then restart Tomcat.

Regards,
~~helix84

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech

Re: [Dspace-tech] SOLR Stats ERROR: Invalid Version or the data not in 'javabin' format

[Matthew McKinley]

Hello!
Sent out the below message on Friday and haven't yet gotten a response...but
I have found out some more information:

Apparently with SOLR/Solrj 1.4 the default Response parser switched from XML
to binary, meaning SOLR 1.4 will have trouble talking to SOLR 3.x and below.
Might this be what is causing the problem? How can I tell what version of
SOLR is in my dspace 1.7.2 instance?

Documentation says a "quick fix" might be to set the response parser as XML
(i.e.

server.setParser(new XMLResponseParser());)

within a CommonsHttpSolrServer java file, but I cannot find this file in the
dspace source. Any ideas?

Apologies for being a little lost in all this. Like many of you, I'm a
librarian/archivist type stumbling into the back end of a technical system.

Thanks in advance for any help,
*
Matthew McKinley
Digital Project Specialist, University of California,
Irvine
**about.me *



On Fri, Sep 16, 2011 at 3:04 PM, Matthew McKinley <
matthewjamesmckin...@gmail.com> wrote:

> Hello,
>
> Upgrading from 1.5.2 to 1.7.2 and the SOLR statistics are not working. I
> was able to convert the old DSpace logs, but when I try to run
> stats-log-importer, stats-util or access statistics from within the browser
> interface they all return with the following error message: "ERROR:
> java.lang.RuntimeException: Invalid version or the data in not in 'javabin'
> format". The full dspace.log output is below.
>
> Thanks for any help you can provide!
> Matthew
>
> Dspace 1.7.2
> Tomcat6
> Running on Linux 2.6
> Java 1.6
>
> 2011-09-16 12:15:45,416 ERROR org.dspace.statistics.SolrLogger @ Error
> executing query
> org.apache.solr.client.solrj.SolrServerException: Error executing query
> at
> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95)
> at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:118)
> at org.dspace.statistics.SolrLogger.(SolrLogger.java:79)
> at
> org.dspace.statistics.content.StatisticsDataVisits.queryFacetField(StatisticsDataVisits.java:640)
> at
> org.dspace.statistics.content.StatisticsDataVisits.createDataset(StatisticsDataVisits.java:247)
> at
> org.dspace.statistics.content.StatisticsDisplay.getDataset(StatisticsDisplay.java:88)
> at
> org.dspace.app.xmlui.aspect.statistics.StatisticsTransformer.addDisplayListing(StatisticsTransformer.java:350)
> at
> org.dspace.app.xmlui.aspect.statistics.StatisticsTransformer.renderHome(StatisticsTransformer.java:129)
> at
> org.dspace.app.xmlui.aspect.statistics.StatisticsTransformer.addBody(StatisticsTransformer.java:79)
> at
> org.dspace.app.xmlui.wing.AbstractWingTransformer.startElement(AbstractWingTransformer.java:223)
> at sun.reflect.GeneratedMethodAccessor85.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
> at $Proxy31.startElement(Unknown Source)
> at
> org.apache.cocoon.components.sax.XMLTeePipe.startElement(XMLTeePipe.java:87)
> at
> org.apache.cocoon.environment.internal.EnvironmentChanger.startElement(EnvironmentStack.java:140)
> at
> org.apache.cocoon.components.sax.XMLTeePipe.startElement(XMLTeePipe.java:87)
> at
> org.apache.cocoon.xml.AbstractXMLPipe.startElement(AbstractXMLPipe.java:94)
> at
> org.dspace.app.xmlui.wing.AbstractWingTransformer.startElement(AbstractWingTransformer.java:240)
> at sun.reflect.GeneratedMethodAccessor85.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
> at $Proxy28.startElement(Unknown Source)
> at
> org.apache.cocoon.environment.internal.EnvironmentChanger.startElement(EnvironmentStack.java:140)
> at
> org.apache.cocoon.components.sax.XMLTeePipe.startElement(XMLTeePipe.java:87)
> at
> org.apache.cocoon.xml.AbstractXMLPipe.startElement(AbstractXMLPipe.java:94)
> at
> org.dspace.app.xmlui.wing.AbstractWingTransformer.startElement(AbstractWingTransformer.java:240)
> at sun.reflect.GeneratedMethodAccessor85.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
> at $Proxy28.startElement(Unknown Source)
> at
> org.apache.cocoon.environment.internal.EnvironmentChanger.startElement(EnvironmentStack.java:140)
> at
> org.apache.cocoon.components.sax.XMLTeePipe.startElement(XMLTeePipe.java:87)
> at
> org

Re: [Dspace-tech] Solr Stats

[Sean Carte]

On 7 September 2011 15:44, Sean Carte  wrote:
>> I'm not 100% sure, but from what I recall, DSpace tries to determine the
>> geo data on the moment the usage event occurs. If it can't retrieve the geo
>> date (country, city, longitude, lattitude, ...), it will still store the
>> usage event, but without this data. However, it should keep the IP in there.
>
> That's really odd then: all the visits are resolved to specific countries.
> Does DSpace know what country and city I'm in and report all local IP
> addresses accordingly? I'll experiment to determine this one way or another.

Looking at a particular item, this is what statistics reports initially:

Mergers and managers : what’s needed for both to work?  661

Clicking View/Open using a browser on a local network:

2011-09-08 10:21:30,600 INFO
org.dspace.usage.LoggerUsageEventListener @
sean.ca...@gmail.com:session_id=1E77AD6CA31AE6D43C97879863EF13A6:ip_addr=10.4.36.158:view_bitstream:bitstream_id=2664

Clicking View/Open using a browser on a remote network:

2011-09-08 10:31:02,695 INFO
org.dspace.usage.LoggerUsageEventListener @
sean.ca...@gmail.com:session_id=073E16DB631648164AF2C876E61CEC68:ip_addr=196.21.58.14:view_bitstream:bitstream_id=2664
2011-09-08 10:31:03,293 ERROR org.dspace.statistics.SolrLogger @
Failed DNS Lookup for IP:196.21.58.14

Stats unchanged:
Mergers and managers : what’s needed for both to work?  661

Later from home and another network:

Mergers and managers : what’s needed for both to work?  663

2011-09-08 17:41:24,473 INFO
org.dspace.usage.LoggerUsageEventListener @
sean.ca...@gmail.com:session_id=2380AA8DE61C7FB43EA8081B5A227C1E:ip_addr=41.183.0.31:view_bitstream:bitstream_id=2664

Mergers and managers : what’s needed for both to work?  663

But it does look like my two earlier views of the document got added.



And today the statistics show another visit:

Mergers and managers : what’s needed for both to work?  664

Checking the log confirms that there were only my three downloads of the file:

root@ir:~# grep bitstream_id=2664 /dspace/log/dspace.log.2011-09-08
2011-09-08 10:21:30,600 INFO
org.dspace.usage.LoggerUsageEventListener @
sean.ca...@gmail.com:session_id=1E77AD6CA31AE6D43C97879863EF13A6:ip_addr=10.4.36.158:view_bitstream:bitstream_id=2664
2011-09-08 10:31:02,695 INFO
org.dspace.usage.LoggerUsageEventListener @
sean.ca...@gmail.com:session_id=073E16DB631648164AF2C876E61CEC68:ip_addr=196.21.58.14:view_bitstream:bitstream_id=2664
2011-09-08 17:41:24,473 INFO
org.dspace.usage.LoggerUsageEventListener @
sean.ca...@gmail.com:session_id=2380AA8DE61C7FB43EA8081B5A227C1E:ip_addr=41.183.0.31:view_bitstream:bitstream_id=2664

... and there's nothing in the 2011-09-09 log.

And all this leads me to conclude that DSpace does not differentiate
between local network access and remote. But also that I have to wait
a while for the data to be presented in the statistics page. There
must be some background process doing this?

The 'Top country views' remains unchanged. This must be because DSpace
was unable to resolve any of my IP addresses. Fair enough. That all
makes sense.

Sean
-- 
Sean Carte
esAL Library Systems Manager
+27 72 898 8775
+27 31 373 2490
fax: 0866741254
http://esal.dut.ac.za/

--
Why Cloud-Based Security and Archiving Make Sense
Osterman Research conducted this study that outlines how and why cloud
computing security and archiving is rapidly being adopted across the IT 
space for its ease of implementation, lower cost, and increased 
reliability. Learn more. http://www.accelacomm.com/jaw/sfnl/114/51425301/
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech

Re: [Dspace-tech] Solr Stats

[Sean Carte]

On 7 September 2011 15:31, Bram Luyten  wrote:

> with "looking at the statistics for several items" do you mean looking at
> what the DSpace web user interface presents, or the results that the SOLR
> throws back it you when you query it directly?
>

I meant the former: looking at what the DSpace Web UI presents.

>
> I'm not 100% sure, but from what I recall, DSpace tries to determine the
> geo data on the moment the usage event occurs. If it can't retrieve the geo
> date (country, city, longitude, lattitude, ...), it will still store the
> usage event, but without this data. However, it should keep the IP in there.
>

That's really odd then: all the visits are resolved to specific countries.
Does DSpace know what country and city I'm in and report all local IP
addresses accordingly? I'll experiment to determine this one way or another.

On a sidenote, differentiating your internal traffic vs your external
> repository traffic always seemed like a good reporting feature. If you could
> use this, please voice your support or contribute here:
> https://jira.duraspace.org/browse/DS-795
>

I wish I had the expertise to do more than add my support, which I've done.

Sean
-- 
Sean Carte
esAL Library Systems Manager
+27 72 898 8775
+27 31 373 2490
fax: 0866741254
http://esal.dut.ac.za/
--
Using storage to extend the benefits of virtualization and iSCSI
Virtualization increases hardware utilization and delivers a new level of
agility. Learn what those decisions are and how to modernize your storage 
and backup environments for virtualization.
http://www.accelacomm.com/jaw/sfnl/114/51434361/___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech

Re: [Dspace-tech] Solr Stats

[Bram Luyten]

Hi Sean,

with "looking at the statistics for several items" do you mean looking at
what the DSpace web user interface presents, or the results that the SOLR
throws back it you when you query it directly?

I'm not 100% sure, but from what I recall, DSpace tries to determine the geo
data on the moment the usage event occurs. If it can't retrieve the geo date
(country, city, longitude, lattitude, ...), it will still store the usage
event, but without this data. However, it should keep the IP in there.

On a sidenote, differentiating your internal traffic vs your external
repository traffic always seemed like a good reporting feature. If you could
use this, please voice your support or contribute here:
https://jira.duraspace.org/browse/DS-795

with kindest regards,

Bram Luyten

@mire

Esperantolaan 4 - 3001 Heverlee - Belgium
2888 Loker Avenue East, Suite 305 - Carlsbad, CA 92010 - USA

atmire.com - Institutional Repository Solutions


On Wed, Sep 7, 2011 at 3:04 PM, Sean Carte  wrote:

>
>
> On 7 September 2011 13:39, Bram Luyten  wrote:
>
>> that's definitely possible, the logging backend of the SOLR statistics
>> logs more than what is shown in the interface.
>> You can find a reference on everything that's logged, and some hints on
>> how to query SOLR directly here:
>> https://wiki.duraspace.org/display/DSDOCDEV/DSpace+Statistics
>>
>
> Thanks Bram, that got me started. But then I noticed something I should
> have seen ages ago: item-level statistics are available within DSpace. I
> always assumed that the 'Statistics/View Statistics' link would only work
> for communities or collections.
>
> But this leads me to a further question: I'd expect some, if not most, of
> our visits to come from our internal network (10.0.x.x), but looking at the
> statistics for several items shows the views all resolved to particular
> countries. Is solr discarding all the unresolvable IP addresses?
>
>
> Sean
> --
> Sean Carte
> esAL Library Systems Manager
> +27 72 898 8775
> +27 31 373 2490
> fax: 0866741254
> http://esal.dut.ac.za/
>
>
> --
> Using storage to extend the benefits of virtualization and iSCSI
> Virtualization increases hardware utilization and delivers a new level of
> agility. Learn what those decisions are and how to modernize your storage
> and backup environments for virtualization.
> http://www.accelacomm.com/jaw/sfnl/114/51434361/
> ___
> DSpace-tech mailing list
> DSpace-tech@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspace-tech
>
>
--
Using storage to extend the benefits of virtualization and iSCSI
Virtualization increases hardware utilization and delivers a new level of
agility. Learn what those decisions are and how to modernize your storage 
and backup environments for virtualization.
http://www.accelacomm.com/jaw/sfnl/114/51434361/___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech

Re: [Dspace-tech] Solr Stats

[Sean Carte]

On 7 September 2011 13:39, Bram Luyten  wrote:

> that's definitely possible, the logging backend of the SOLR statistics logs
> more than what is shown in the interface.
> You can find a reference on everything that's logged, and some hints on how
> to query SOLR directly here:
> https://wiki.duraspace.org/display/DSDOCDEV/DSpace+Statistics
>

Thanks Bram, that got me started. But then I noticed something I should have
seen ages ago: item-level statistics are available within DSpace. I always
assumed that the 'Statistics/View Statistics' link would only work for
communities or collections.

But this leads me to a further question: I'd expect some, if not most, of
our visits to come from our internal network (10.0.x.x), but looking at the
statistics for several items shows the views all resolved to particular
countries. Is solr discarding all the unresolvable IP addresses?

Sean
-- 
Sean Carte
esAL Library Systems Manager
+27 72 898 8775
+27 31 373 2490
fax: 0866741254
http://esal.dut.ac.za/
--
Using storage to extend the benefits of virtualization and iSCSI
Virtualization increases hardware utilization and delivers a new level of
agility. Learn what those decisions are and how to modernize your storage 
and backup environments for virtualization.
http://www.accelacomm.com/jaw/sfnl/114/51434361/___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech

Re: [Dspace-tech] Solr Stats

[Bram Luyten]

Hi Sean,

that's definitely possible, the logging backend of the SOLR statistics logs
more than what is shown in the interface.
You can find a reference on everything that's logged, and some hints on how
to query SOLR directly here:
https://wiki.duraspace.org/display/DSDOCDEV/DSpace+Statistics

with kindest regards,

Bram Luyten

@mire

Esperantolaan 4 - 3001 Heverlee - Belgium
2888 Loker Avenue East, Suite 305 - Carlsbad, CA 92010 - USA

atmire.com - Institutional Repository Solutions


On Wed, Sep 7, 2011 at 8:41 AM, Sean Carte  wrote:

> DSpace statistics gives the number of times an article has been
> viewed; is it possible to get more information about those views such
> as the IP address of the viewer?
>
> Sean
> --
> Sean Carte
> esAL Library Systems Manager
> +27 72 898 8775
> +27 31 373 2490
> fax: 0866741254
> http://esal.dut.ac.za/
>
>
> --
> Using storage to extend the benefits of virtualization and iSCSI
> Virtualization increases hardware utilization and delivers a new level of
> agility. Learn what those decisions are and how to modernize your storage
> and backup environments for virtualization.
> http://www.accelacomm.com/jaw/sfnl/114/51434361/
> ___
> DSpace-tech mailing list
> DSpace-tech@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspace-tech
>
--
Using storage to extend the benefits of virtualization and iSCSI
Virtualization increases hardware utilization and delivers a new level of
agility. Learn what those decisions are and how to modernize your storage 
and backup environments for virtualization.
http://www.accelacomm.com/jaw/sfnl/114/51434361/___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech

Re: [Dspace-tech] Solr stats refresh

[Germán Biozzoli]

Thank you very much Peter.
I'll look for this

Germán.

On Tue, Jan 4, 2011 at 1:11 PM, Peter Dietz  wrote:
> Hi Germán,
> The stats do update, however, for performance reasons, we've changed the
> default to not update in real-time, but to wait for an autoCommit interval.
>
> Here are the current defaults. If you want your stats to update sooner, you
> can lower the value for autocommit.
> 
> 
>     1 
>     90 
> 
> https://fisheye3.atlassian.com/browse/dspace/dspace/trunk/dspace/solr/statistics/conf/solrconfig.xml?r=HEAD#l229
>
>
> --
> Peter Dietz
> Systems Developer/Engineer
> Ohio State University Libraries
>
>
>
> On Tue, Jan 4, 2011 at 10:42 AM, Germán Biozzoli 
> wrote:
>>
>> Hi people
>>
>> I'm trying to get Dspace-Solr statistics to work in 1.70. It's seems
>> to be OK but it's not updated in real time, I mean that if anybody
>> visits an item or a collection, the number of accesses remain in the
>> old count. I've tried dspace index-update command, but it's seems to
>> update only Solr search cluster, isn't it? Also I've tried stats-util
>> but I can't see an option to update the index, only one to optimize,
>> should I use it as my cron task for update?
>>
>> Thanks a lot
>> German
>>
>>
>> --
>> Learn how Oracle Real Application Clusters (RAC) One Node allows customers
>> to consolidate database storage, standardize their database environment,
>> and,
>> should the need arise, upgrade to a full multi-node Oracle RAC database
>> without downtime or disruption
>> http://p.sf.net/sfu/oracle-sfdevnl
>> ___
>> DSpace-tech mailing list
>> DSpace-tech@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/dspace-tech
>
>

--
Learn how Oracle Real Application Clusters (RAC) One Node allows customers
to consolidate database storage, standardize their database environment, and, 
should the need arise, upgrade to a full multi-node Oracle RAC database 
without downtime or disruption
http://p.sf.net/sfu/oracle-sfdevnl
___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech

Re: [Dspace-tech] Solr stats refresh

[Peter Dietz]

Hi Germán,

The stats do update, however, for performance reasons, we've changed the
default to not update in real-time, but to wait for an autoCommit interval.

Here are the current defaults. If you want your stats to update sooner, you
can lower the value for autocommit.



1 
90 

https://fisheye3.atlassian.com/browse/dspace/dspace/trunk/dspace/solr/statistics/conf/solrconfig.xml?r=HEAD#l229



--
Peter Dietz
Systems Developer/Engineer
Ohio State University Libraries



On Tue, Jan 4, 2011 at 10:42 AM, Germán Biozzoli
wrote:

> Hi people
>
> I'm trying to get Dspace-Solr statistics to work in 1.70. It's seems
> to be OK but it's not updated in real time, I mean that if anybody
> visits an item or a collection, the number of accesses remain in the
> old count. I've tried dspace index-update command, but it's seems to
> update only Solr search cluster, isn't it? Also I've tried stats-util
> but I can't see an option to update the index, only one to optimize,
> should I use it as my cron task for update?
>
> Thanks a lot
> German
>
>
> --
> Learn how Oracle Real Application Clusters (RAC) One Node allows customers
> to consolidate database storage, standardize their database environment,
> and,
> should the need arise, upgrade to a full multi-node Oracle RAC database
> without downtime or disruption
> http://p.sf.net/sfu/oracle-sfdevnl
> ___
> DSpace-tech mailing list
> DSpace-tech@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspace-tech
>
--
Learn how Oracle Real Application Clusters (RAC) One Node allows customers
to consolidate database storage, standardize their database environment, and, 
should the need arise, upgrade to a full multi-node Oracle RAC database 
without downtime or disruption
http://p.sf.net/sfu/oracle-sfdevnl___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech

Re: [Dspace-tech] Solr stats granularity

[César Sabater]

Thank you Bram, the slide was very welpfull.

Cheers
César Sabater


2010/11/19 Bram Luyten 

> Dear César,
>
> the back end logs all page views and downloads. You are correct in your
> observation that "total visits per month" like shown for a community page
> here: http://demo.dspace.org/xmlui/handle/10673/1089/statistics represents
> only the pageviews of that specific community page.
>
> It's possible to formulate SOLR queries to aggregate data across
> collections/communities.
> Here's a slide presentation about the usage stats: http://bit.ly/bXCyFbthat 
> shows a few examples on how you can customize the views.
>
> with best regards,
>
> Bram Luyten
>
> @mire - http://www.atmire.com
>
> Technologielaan 9 - 3001 Heverlee - Belgium
> 533 2nd Street - Encinitas, CA 92024 - USA
>
> http://www.togather.eu - Before getting together, get t...@ther
>
>
> On Fri, Nov 19, 2010 at 1:51 PM, César Sabater wrote:
>
>> Hi all,
>>
>> I've seen that solr statistics has Fine-grained stats in dspace, it shows
>> community, collection, item and  bitstream views.
>>
>> Now I'm wondering if community views represent the sum of all
>> item/collection views (without counting items twice or more) placed on that
>> community, or they only represent the general view of the community. The
>> same question can be done for collections and its items.
>>
>> If the number of views are not accumulative botton-up, how can that be
>> done? anybody has done that before?
>>
>> Any help would be appreciated.
>> Cheers!
>> César
>>
>>
>> --
>> Beautiful is writing same markup. Internet Explorer 9 supports
>> standards for HTML5, CSS3, SVG 1.1,  ECMAScript5, and DOM L2 & L3.
>> Spend less time writing and  rewriting code and more time creating great
>> experiences on the web. Be a part of the beta today
>> http://p.sf.net/sfu/msIE9-sfdev2dev
>> ___
>> DSpace-tech mailing list
>> DSpace-tech@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/dspace-tech
>>
>>
>
--
Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1,  ECMAScript5, and DOM L2 & L3.
Spend less time writing and  rewriting code and more time creating great
experiences on the web. Be a part of the beta today
http://p.sf.net/sfu/msIE9-sfdev2dev___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech

Re: [Dspace-tech] Solr stats granularity

[Bram Luyten]

Dear César,

the back end logs all page views and downloads. You are correct in your
observation that "total visits per month" like shown for a community page
here: http://demo.dspace.org/xmlui/handle/10673/1089/statistics represents
only the pageviews of that specific community page.

It's possible to formulate SOLR queries to aggregate data across
collections/communities.
Here's a slide presentation about the usage stats: http://bit.ly/bXCyFb that
shows a few examples on how you can customize the views.

with best regards,

Bram Luyten

@mire - http://www.atmire.com

Technologielaan 9 - 3001 Heverlee - Belgium
533 2nd Street - Encinitas, CA 92024 - USA

http://www.togather.eu - Before getting together, get t...@ther


On Fri, Nov 19, 2010 at 1:51 PM, César Sabater  wrote:

> Hi all,
>
> I've seen that solr statistics has Fine-grained stats in dspace, it shows
> community, collection, item and  bitstream views.
>
> Now I'm wondering if community views represent the sum of all
> item/collection views (without counting items twice or more) placed on that
> community, or they only represent the general view of the community. The
> same question can be done for collections and its items.
>
> If the number of views are not accumulative botton-up, how can that be
> done? anybody has done that before?
>
> Any help would be appreciated.
> Cheers!
> César
>
>
> --
> Beautiful is writing same markup. Internet Explorer 9 supports
> standards for HTML5, CSS3, SVG 1.1,  ECMAScript5, and DOM L2 & L3.
> Spend less time writing and  rewriting code and more time creating great
> experiences on the web. Be a part of the beta today
> http://p.sf.net/sfu/msIE9-sfdev2dev
> ___
> DSpace-tech mailing list
> DSpace-tech@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspace-tech
>
>
--
Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1,  ECMAScript5, and DOM L2 & L3.
Spend less time writing and  rewriting code and more time creating great
experiences on the web. Be a part of the beta today
http://p.sf.net/sfu/msIE9-sfdev2dev___
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech