Re: [Dspace-tech] Fwd: Problem with Data Provider Validation
Hi Jodi / Keith, However this request returns error code=badArgument. http://dspace.udel.edu:8080/dspace-oai/request?verb=ListIdentifiers The 'badArgument' occurs because you need to specify which format you want the metadata in. So the following works: http://dspace.udel.edu:8080/dspace-oai/request?verb=ListIdentifiersmetadata Prefix=oai_dc However This still suffers from the problem of no records being returned when no dates are specified. As you have noticed, if you add a 'from' records are returned. (Not so if you just add a 'to'). What should be happening, is OAICat passes a 'null' for 'to' and 'from' to org.dspace.search.Harvest.java If it receives null values, it should not do any date filtering on the select query. Something must be going wrong here. To debug this, we need to see what queries are being run. Can you: 1) Upgrade your logging output to DEBUG (change 'INFO' to 'DEBUG' in [dspace]/conf/log4j.properties an then restart tomcat). 2) Watch you [dspace]/log/dspace.log file as you call the ListIdentifiers URL which is returning an empty result set. The debug output should show the output that OAICat is passing to Harvest.java, and should show the corresponding query that Harvest.java calls. If you can send these to the list, we can look into the problem further. Thanks, Stuart _ Datblygydd Cymwysiadau'r WeWeb Applications Developer Gwasanaethau Gwybodaeth Information Services Prifysgol Cymru Aberystwyth University of Wales Aberystwyth E-bost / E-mail: [EMAIL PROTECTED] Ffon / Tel: (01970) 622860 _ - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Fwd: Problem with Data Provider Validation
Not really. The badArgument is valid when you don't specify a metadataPrefix, and the noRecordsMatch error is valid when there are no records that match the date range specified. If you simply include a from timestamp, eg: http://dspace.udel.edu:8080/dspace-oai/request?verb=ListIdentifiersmetadataPrefix=oai_dcfrom=2001-01-01T00:00:00Z then it works. I don't think there is anything wrong with the harvesting code per se, it just looks like it is incorrectly interpreting a from date when you don't specify one. Although according to the oaicat.properties below, the earliestDatestamp is set to 2001-01-01T00:00:00Z - exactly what I'm passing in the above url, and which works! G On Tue, 2007-06-05 at 22:37 -0400, Jodi Schneider wrote: Forwarding this back out to the list for better answers to Keith's questions. http://sourceforge.net/tracker/index.php?func=detailaid=1688523group_id=19984atid=319984 mentions the TimeStamp function of *Oracle*. Is the bug also applicable to Postgres? http://dspace.udel.edu:8080/dspace-oai/request?verb=ListIdentifiersfrom=2007-04-01until=2007-05-15metadataPrefix=oai_dc works beautifully. However this request returns error code=badArgument. http://dspace.udel.edu:8080/dspace-oai/request?verb=ListIdentifiers Can anyone tell Keith how to fix this? Is upgrade to 1.4.2 the answer? Or...? -Jodi PS-Keith, what happens if you change granularity? i.e. in oaicat.properties change AbstractCatalog.granularity=-MM-DDThh:mm:ssZ to AbstractCatalog.granularity=-MM-DD -- Forwarded message -- From: Keith Jones [EMAIL PROTECTED] Date: Jun 5, 2007 3:41 PM Subject: Re: [Dspace-tech] Problem with Data Provider Validation To: Jodi Schneider [EMAIL PROTECTED] Hi Jodi, Thanks for the help. At my institution we are using Postgres, would this have any effect on the date granularity? Are you saying that the harvester will not work, unless you are at version 1.4.2? I've been able to get subsets if the listidentifiers working, atleast it will work when I put in a from date but will not work if no date is specified. I'm still getting the following error: Code=noRecordsMatch, Description=The combination of the values of the from, until, set, and metadataPrefix arguments results in an empty list. With the http request of: http://dspace.udel.edu:8080/dspace-oai/request, verb=ListIdentifiers, metadataPrefix=oai_dc But I can get results with the following http request http://dspace.udel.edu:8080/dspace-oai/request?verb=ListIdentifiersfrom=2007-04-01metadataPrefix=oai_dc Thanks Keith -- Forwarded message -- From: Jodi Schneider [EMAIL PROTECTED] Date: Jun 5, 2007 9:19 AM Subject: [Dspace-tech] Problem with Data Provider Validation To: Dspace Tech dspace-tech@lists.sourceforge.net Keith, It looks like your repository is still having this problem. I think you will need to upgrade to 1.4.2 or change timestamp granularity. It looks like 1.4.2 fixed a bug regarding OAI timestamps: The oracleTimeStampFunction did not correctly account for 20 character length timestamps (-MM-DDTHH:mm:ssZ), incorrectly treating them as '-MM-DDTHH:mm:ss.FFZ'. http://sourceforge.net/tracker/index.php?func=detailaid=1688523group_id=19984atid=319984 --- I compared the file you mention to my default oaicat.properties and it looks like one block is duplicated. I wouldn't expect this to harm anything. AbstractCatalog.oaiCatalogClassName=org.dspace.app.oai.DSpaceOAICatalog AbstractCatalog.recordFactoryClassName=org.dspace.app.oai.DSpaceRecordFactory AbstractCatalog.secondsToLive=3600 AbstractCatalog.granularity=-MM-DDThh:mm:ssZ Best luck, and apologies it took so long to figure this out! -Jodi Portable Citations for DSpace On 5/29/07, Keith Jones [EMAIL PROTECTED] wrote: I did not chnage anything in the oaicat.properties. Here is my properties file: ### # REMEMBER: ONLY UPDATE THE VERSION IN dspace/config/templates # AND THEN RUN dspace/bin/install-configs! # DO NOT EDIT THE 'LIVE' VERSION! ### # OAICat Configuration file - see OAICat documentation for details # Text surrounded by two '@' symbols is replaced with the corresponding # property from dspace.cfg. For example: # # http://dspace.udel.edu:8080/dspace # # would be replaced with the dspace.url property in dspace.cfg. # When /dspace/bin/install-configs is run, this file will be installed in the # location specified by the property: config.template.oaicat.properties
Re: [Dspace-tech] Fwd: Problem with Data Provider Validation
If there is no date range specified, all records with the specified metadataPrefix must be returned by protocol section 2.7.1: http://www.openarchives.org/OAI/openarchivesprotocol.html#SelectiveHarvestingandDatestamps Harvesting is restricted to the range specified by the from and untilarguments, extending back to the earliest datestamp if from is omitted, and forward to the most recent datestamp if until is omitted. -Jodi On 6/6/07, Graham Triggs [EMAIL PROTECTED] wrote: Not really. The badArgument is valid when you don't specify a metadataPrefix, and the noRecordsMatch error is valid when there are no records that match the date range specified. If you simply include a from timestamp, eg: http://dspace.udel.edu:8080/dspace-oai/request?verb=ListIdentifiersmetadataPrefix=oai_dcfrom=2001-01-01T00:00:00Z then it works. I don't think there is anything wrong with the harvesting code per se, it just looks like it is incorrectly interpreting a from date when you don't specify one. Although according to the oaicat.properties below, the earliestDatestamp is set to 2001-01-01T00:00:00Z - exactly what I'm passing in the above url, and which works! G On Tue, 2007-06-05 at 22:37 -0400, Jodi Schneider wrote: Forwarding this back out to the list for better answers to Keith's questions. http://sourceforge.net/tracker/index.php?func=detailaid=1688523group_id=19984atid=319984 mentions the TimeStamp function of *Oracle*. Is the bug also applicable to Postgres? http://dspace.udel.edu:8080/dspace-oai/request?verb=ListIdentifiersfrom=2007-04-01until=2007-05-15metadataPrefix=oai_dc works beautifully. However this request returns error code=badArgument. http://dspace.udel.edu:8080/dspace-oai/request?verb=ListIdentifiers Can anyone tell Keith how to fix this? Is upgrade to 1.4.2 the answer? Or...? -Jodi PS-Keith, what happens if you change granularity? i.e. in oaicat.properties change AbstractCatalog.granularity=-MM-DDThh:mm:ssZ to AbstractCatalog.granularity=-MM-DD -- Forwarded message -- From: Keith Jones [EMAIL PROTECTED] Date: Jun 5, 2007 3:41 PM Subject: Re: [Dspace-tech] Problem with Data Provider Validation To: Jodi Schneider [EMAIL PROTECTED] Hi Jodi, Thanks for the help. At my institution we are using Postgres, would this have any effect on the date granularity? Are you saying that the harvester will not work, unless you are at version 1.4.2? I've been able to get subsets if the listidentifiers working, atleast it will work when I put in a from date but will not work if no date is specified. I'm still getting the following error: Code=noRecordsMatch, Description=The combination of the values of the from, until, set, and metadataPrefix arguments results in an empty list. With the http request of: http://dspace.udel.edu:8080/dspace-oai/request, verb=ListIdentifiers, metadataPrefix=oai_dc But I can get results with the following http request http://dspace.udel.edu:8080/dspace-oai/request?verb=ListIdentifiersfrom=2007-04-01metadataPrefix=oai_dc Thanks Keith -- Forwarded message -- From: Jodi Schneider [EMAIL PROTECTED] Date: Jun 5, 2007 9:19 AM Subject: [Dspace-tech] Problem with Data Provider Validation To: Dspace Tech dspace-tech@lists.sourceforge.net Keith, It looks like your repository is still having this problem. I think you will need to upgrade to 1.4.2 or change timestamp granularity. It looks like 1.4.2 fixed a bug regarding OAI timestamps: The oracleTimeStampFunction did not correctly account for 20 character length timestamps (-MM-DDTHH:mm:ssZ), incorrectly treating them as '-MM-DDTHH:mm:ss.FFZ'. http://sourceforge.net/tracker/index.php?func=detailaid=1688523group_id=19984atid=319984 --- I compared the file you mention to my default oaicat.properties and it looks like one block is duplicated. I wouldn't expect this to harm anything. AbstractCatalog.oaiCatalogClassName=org.dspace.app.oai.DSpaceOAICatalog AbstractCatalog.recordFactoryClassName=org.dspace.app.oai.DSpaceRecordFactory AbstractCatalog.secondsToLive=3600 AbstractCatalog.granularity=-MM-DDThh:mm:ssZ Best luck, and apologies it took so long to figure this out! -Jodi Portable Citations for DSpace On 5/29/07, Keith Jones [EMAIL PROTECTED] wrote: I did not chnage anything in the oaicat.properties. Here is my properties file: ### # REMEMBER: ONLY UPDATE THE VERSION IN dspace/config/templates # AND THEN RUN dspace/bin/install-configs! # DO NOT EDIT THE 'LIVE' VERSION! ### # OAICat Configuration file - see OAICat documentation for details # Text surrounded by two '@' symbols is replaced with the corresponding # property from dspace.cfg. For example:
Re: [Dspace-tech] Fwd: Problem with Data Provider Validation
Thanks for the additional input. I turned on debug and here is what output I am getting in the log file. 2007-06-06 14:47:48,393 INFO org.dspace.app.oai.DSpaceOAICatalog @ anonymous:no _context:oai_request:verb=listIdentifiers,from=0001-01-01T00:00:00Z,until=-1 2-31T23:59:59Z,set=null,metadataPrefix=oai_dc 2007-06-06 14:47:49,008 DEBUG org.dspace.search.Harvest @ anonymous::harvest SQL :SELECT handle.handle, handle.resource_id, item.withdrawn, item.last_modified FR OM handle, item WHERE handle.resource_type_id=2 AND handle.resource_id=item.item _id AND item.last_modified = ? AND item.last_modified = ? ORDER BY handle.r esource_id 2007-06-06 14:47:49,009 DEBUG org.dspace.storage.rdbms.DatabaseManager @ Running query SELECT handle.handle, handle.resource_id, item.withdrawn, item.last_modi fied FROM handle, item WHERE handle.resource_type_id=2 AND handle.resource_id=it em.item_id AND item.last_modified = ? AND item.last_modified = ? ORDER BY h andle.resource_id 2007-06-06 14:47:49,782 INFO org.dspace.app.oai.DSpaceOAICatalog @ anonymous:no _context:oai_error:no_items_match On Wed, 6 Jun 2007, Stuart Lewis [sdl] wrote: Hi Jodi / Keith, However this request returns error code=badArgument. http://dspace.udel.edu:8080/dspace-oai/request?verb=ListIdentifiers The 'badArgument' occurs because you need to specify which format you want the metadata in. So the following works: http://dspace.udel.edu:8080/dspace-oai/request?verb=ListIdentifiersmetadata Prefix=oai_dc However This still suffers from the problem of no records being returned when no dates are specified. As you have noticed, if you add a 'from' records are returned. (Not so if you just add a 'to'). What should be happening, is OAICat passes a 'null' for 'to' and 'from' to org.dspace.search.Harvest.java If it receives null values, it should not do any date filtering on the select query. Something must be going wrong here. To debug this, we need to see what queries are being run. Can you: 1) Upgrade your logging output to DEBUG (change 'INFO' to 'DEBUG' in [dspace]/conf/log4j.properties an then restart tomcat). 2) Watch you [dspace]/log/dspace.log file as you call the ListIdentifiers URL which is returning an empty result set. The debug output should show the output that OAICat is passing to Harvest.java, and should show the corresponding query that Harvest.java calls. If you can send these to the list, we can look into the problem further. Thanks, Stuart _ Datblygydd Cymwysiadau'r WeWeb Applications Developer Gwasanaethau Gwybodaeth Information Services Prifysgol Cymru Aberystwyth University of Wales Aberystwyth E-bost / E-mail: [EMAIL PROTECTED] Ffon / Tel: (01970) 622860 _ - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Fwd: Problem with Data Provider Validation
Hi Graham, I in agreement that the code is working as specified. My problem occured when I tried to register my Dspace instance with the official Open Archives registry. The registry send a listIdentifiers request with no from date specified. As such my dspace instance has failed to pass the registration test. I'm trying to determine if this is a configuration problem, or could there be bad data that is effecting the data pull. On Wed, 6 Jun 2007, Graham Triggs wrote: Not really. The badArgument is valid when you don't specify a metadataPrefix, and the noRecordsMatch error is valid when there are no records that match the date range specified. If you simply include a from timestamp, eg: http://dspace.udel.edu:8080/dspace-oai/request?verb=ListIdentifiersmetadataPrefix=oai_dcfrom=2001-01-01T00:00:00Z then it works. I don't think there is anything wrong with the harvesting code per se, it just looks like it is incorrectly interpreting a from date when you don't specify one. Although according to the oaicat.properties below, the earliestDatestamp is set to 2001-01-01T00:00:00Z - exactly what I'm passing in the above url, and which works! G On Tue, 2007-06-05 at 22:37 -0400, Jodi Schneider wrote: Forwarding this back out to the list for better answers to Keith's questions. http://sourceforge.net/tracker/index.php?func=detailaid=1688523group_id=19984atid=319984 mentions the TimeStamp function of *Oracle*. Is the bug also applicable to Postgres? http://dspace.udel.edu:8080/dspace-oai/request?verb=ListIdentifiersfrom=2007-04-01until=2007-05-15metadataPrefix=oai_dc works beautifully. However this request returns error code=badArgument. http://dspace.udel.edu:8080/dspace-oai/request?verb=ListIdentifiers Can anyone tell Keith how to fix this? Is upgrade to 1.4.2 the answer? Or...? -Jodi PS-Keith, what happens if you change granularity? i.e. in oaicat.properties change AbstractCatalog.granularity=-MM-DDThh:mm:ssZ to AbstractCatalog.granularity=-MM-DD -- Forwarded message -- From: Keith Jones [EMAIL PROTECTED] Date: Jun 5, 2007 3:41 PM Subject: Re: [Dspace-tech] Problem with Data Provider Validation To: Jodi Schneider [EMAIL PROTECTED] Hi Jodi, Thanks for the help. At my institution we are using Postgres, would this have any effect on the date granularity? Are you saying that the harvester will not work, unless you are at version 1.4.2? I've been able to get subsets if the listidentifiers working, atleast it will work when I put in a from date but will not work if no date is specified. I'm still getting the following error: Code=noRecordsMatch, Description=The combination of the values of the from, until, set, and metadataPrefix arguments results in an empty list. With the http request of: http://dspace.udel.edu:8080/dspace-oai/request, verb=ListIdentifiers, metadataPrefix=oai_dc But I can get results with the following http request http://dspace.udel.edu:8080/dspace-oai/request?verb=ListIdentifiersfrom=2007-04-01metadataPrefix=oai_dc Thanks Keith -- Forwarded message -- From: Jodi Schneider [EMAIL PROTECTED] Date: Jun 5, 2007 9:19 AM Subject: [Dspace-tech] Problem with Data Provider Validation To: Dspace Tech dspace-tech@lists.sourceforge.net Keith, It looks like your repository is still having this problem. I think you will need to upgrade to 1.4.2 or change timestamp granularity. It looks like 1.4.2 fixed a bug regarding OAI timestamps: The oracleTimeStampFunction did not correctly account for 20 character length timestamps (-MM-DDTHH:mm:ssZ), incorrectly treating them as '-MM-DDTHH:mm:ss.FFZ'. http://sourceforge.net/tracker/index.php?func=detailaid=1688523group_id=19984atid=319984 --- I compared the file you mention to my default oaicat.properties and it looks like one block is duplicated. I wouldn't expect this to harm anything. AbstractCatalog.oaiCatalogClassName=org.dspace.app.oai.DSpaceOAICatalog AbstractCatalog.recordFactoryClassName=org.dspace.app.oai.DSpaceRecordFactory AbstractCatalog.secondsToLive=3600 AbstractCatalog.granularity=-MM-DDThh:mm:ssZ Best luck, and apologies it took so long to figure this out! -Jodi Portable Citations for DSpace On 5/29/07, Keith Jones [EMAIL PROTECTED] wrote: I did not chnage anything in the oaicat.properties. Here is my properties file: ### # REMEMBER: ONLY UPDATE THE VERSION IN dspace/config/templates # AND THEN RUN dspace/bin/install-configs! # DO NOT EDIT THE 'LIVE' VERSION! ### # OAICat Configuration file - see OAICat documentation for details # Text surrounded by two '@' symbols is replaced with the corresponding # property from dspace.cfg. For example: # # http://dspace.udel.edu:8080/dspace
Re: [Dspace-tech] Fwd: Problem with Data Provider Validation
On Wed, 6 Jun 2007, Jodi Schneider wrote: PS-Keith, what happens if you change granularity? i.e. in oaicat.properties change AbstractCatalog.granularity=-MM-DDThh:mm:ssZ to AbstractCatalog.granularity=-MM-DD Hi Jodi, I did make the change suggested above, but the result was the same. Thanks for all the help you have been providing. Am I correct in understanding that your Dspace instance is not exhibiting the same problem? Keith - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Fwd: Problem with Data Provider Validation
Hi Keith, Any chance you could install this patch, and try again: http://sourceforge.net/tracker/index.php?func=detailaid=1659868group_id=19 984atid=319984 The patch gives better database debugging, as it replaces the '?' in the log file with the actual values given. We can then see the actual queries being run, and you can try them by hand to see what is being returned. Thanks, Stuart _ Gwasanaethau Gwybodaeth Information Services Prifysgol Cymru Aberystwyth University of Wales Aberystwyth E-bost / E-mail: [EMAIL PROTECTED] Ffon / Tel: (01970) 622860 _ On 6/6/07 20:20, Keith Jones [EMAIL PROTECTED] wrote: Thanks for the additional input. I turned on debug and here is what output I am getting in the log file. 2007-06-06 14:47:48,393 INFO org.dspace.app.oai.DSpaceOAICatalog @ anonymous:no _context:oai_request:verb=listIdentifiers,from=0001-01-01T00:00:00Z,until= -1 2-31T23:59:59Z,set=null,metadataPrefix=oai_dc 2007-06-06 14:47:49,008 DEBUG org.dspace.search.Harvest @ anonymous::harvest SQL :SELECT handle.handle, handle.resource_id, item.withdrawn, item.last_modified FR OM handle, item WHERE handle.resource_type_id=2 AND handle.resource_id=item.item _id AND item.last_modified = ? AND item.last_modified = ? ORDER BY handle.r esource_id 2007-06-06 14:47:49,009 DEBUG org.dspace.storage.rdbms.DatabaseManager @ Running query SELECT handle.handle, handle.resource_id, item.withdrawn, item.last_modi fied FROM handle, item WHERE handle.resource_type_id=2 AND handle.resource_id=it em.item_id AND item.last_modified = ? AND item.last_modified = ? ORDER BY h andle.resource_id 2007-06-06 14:47:49,782 INFO org.dspace.app.oai.DSpaceOAICatalog @ anonymous:no _context:oai_error:no_items_match On Wed, 6 Jun 2007, Stuart Lewis [sdl] wrote: Hi Jodi / Keith, However this request returns error code=badArgument. http://dspace.udel.edu:8080/dspace-oai/request?verb=ListIdentifiers The 'badArgument' occurs because you need to specify which format you want the metadata in. So the following works: http://dspace.udel.edu:8080/dspace-oai/request?verb=ListIdentifiersmetadata Prefix=oai_dc However This still suffers from the problem of no records being returned when no dates are specified. As you have noticed, if you add a 'from' records are returned. (Not so if you just add a 'to'). What should be happening, is OAICat passes a 'null' for 'to' and 'from' to org.dspace.search.Harvest.java If it receives null values, it should not do any date filtering on the select query. Something must be going wrong here. To debug this, we need to see what queries are being run. Can you: 1) Upgrade your logging output to DEBUG (change 'INFO' to 'DEBUG' in [dspace]/conf/log4j.properties an then restart tomcat). 2) Watch you [dspace]/log/dspace.log file as you call the ListIdentifiers URL which is returning an empty result set. The debug output should show the output that OAICat is passing to Harvest.java, and should show the corresponding query that Harvest.java calls. If you can send these to the list, we can look into the problem further. Thanks, Stuart _ Datblygydd Cymwysiadau'r WeWeb Applications Developer Gwasanaethau Gwybodaeth Information Services Prifysgol Cymru Aberystwyth University of Wales Aberystwyth E-bost / E-mail: [EMAIL PROTECTED] Ffon / Tel: (01970) 622860 _ - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
