[dspace-tech] Anthropic’s ClaudeBot is aggressively scraping DSpace

[James Holobetz]

Has anyone in the DSpace community/tech seen a large spike in queries
for Anthropic’s ClaudeBot?

User Agent: compatible; "ClaudeBot/1.0; +claudebot\@anthropic.com"

Here is some other references lately:

https://www.reddit.com/r/singularity/comments/1cdm97j/anthropics_claudebot_is_aggressively_scraping_the/

We have been getting hit hard for a couple days now.

Best regards,

James

-- 
All messages to this mailing list should adhere to the Code of Conduct: 
https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
--- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/dspace-tech/CAAosP7VXJqUwxDs3q2wSBvE0Q9H6Ms%3DPd9Gvyo8G5KP0OqsQBQ%40mail.gmail.com.

Re: [dspace-tech] DSpace 6.4 - 7.6 migration failure

[James Holobetz]

Hi Steve,

I believe you may have to run:

./dspace database migrate ignored

as described in
https://wiki.lyrasis.org/display/DSDOC7x/Migrating+DSpace+to+a+new+server

James



On Tue, Nov 28, 2023 at 9:15 AM Steve Michaels <
steve.micha...@westernsem.edu> wrote:

> When I run "migrate" I'm receiving a relation "cwf_pooltask" does not
> exist ERROR.   We have been using the XML workflow, consequently DS 3431
> Add Policies for BasicWorkflow is still at version 5.7.2017.05.05.
>
> The full last error is:
> ---
> -- DS-4239 Migrate the workflow.xml to spring
> ---
> -- This script will rename the default workflow "default" name
> -- to the new "defaultWorkflow" identifier
> ---
>
> UPDATE cwf_pooltask SET workflow_id='defaultWorkflow' WHERE
> workflow_id='default'
>
> at
> org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.handleException(DefaultSqlScriptExecutor.java:275)
> at
> org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.executeStatement(DefaultSqlScriptExecutor.java:222)
> at
> org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.execute(DefaultSqlScriptExecutor.java:126)
> at
> org.flywaydb.core.internal.resolver.sql.SqlMigrationExecutor.executeOnce(SqlMigrationExecutor.java:69)
> at
> org.flywaydb.core.internal.resolver.sql.SqlMigrationExecutor.lambda$execute$0(SqlMigrationExecutor.java:58)
> at
> org.flywaydb.core.internal.database.DefaultExecutionStrategy.execute(DefaultExecutionStrategy.java:27)
> at
> org.flywaydb.core.internal.resolver.sql.SqlMigrationExecutor.execute(SqlMigrationExecutor.java:57)
> at
> org.flywaydb.core.internal.command.DbMigrate.doMigrateGroup(DbMigrate.java:377)
> ... 25 more
> Caused by: org.postgresql.util.PSQLException: ERROR: relation
> "cwf_pooltask" does not exist
>   Position: 684
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2713)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2401)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:368)
> at
> org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:498)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:415)
> at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:335)
> at
> org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:321)
> at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:297)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:292)
> at
> org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:193)
> at
> org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:193)
> at
> org.flywaydb.core.internal.jdbc.JdbcTemplate.executeStatement(JdbcTemplate.java:201)
> at
> org.flywaydb.core.internal.sqlscript.ParsedSqlStatement.execute(ParsedSqlStatement.java:95)
> at
> org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.executeStatement(DefaultSqlScriptExecutor.java:210)
> ... 31 more
>
> --
> All messages to this mailing list should adhere to the Code of Conduct:
> https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
> ---
> You received this message because you are subscribed to the Google Groups
> "DSpace Technical Support" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to dspace-tech+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/dspace-tech/91ab8d70-a24e-4bf1-aa5a-5585ce0bd40dn%40googlegroups.com
> 
> .
>

-- 
All messages to this mailing list should adhere to the Code of Conduct: 
https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
--- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/dspace-tech/CAAosP7UFw563b2U7RWwzLaPHm7%3DSqWG3izDhiRGuADxMdawa%3DQ%40mail.gmail.com.

[dspace-tech] Re: stats-util error

[James Holobetz]

After doing a fair amount of detective work, I have found through Solr
Admin queries and examining some of the csv files created by the
solr-export-statistics tool that there are a lot of -unmigrated entries
creatd by the solr-upgrade-statistics-6x tool ( If a UUID value cannot be
found for a legacy id, the legacy id will be converted to the form
"-unmigrated" where  is the legacy id.) These
"-unmigrated"  legacy
IDs are inconsistently in the id, owningComm, owningColl, and scopeId (so
far as I can tell right now). There is a lot of them (42 GB of csv files in
the export dump). How do I correct these values as they are breaking the
stats-util tool?

Thank you,

James Holobetz

On Fri, Oct 27, 2023 at 1:32 PM James Holobetz  wrote:

> Hello All,
>
> I am receiving this error when attempting to reindex the bitstreams using
> stats-util -b:
>
> holobetj ~ $ sudo -u dspace /opt/dspace/bin/dspace stats-util -b
> Exception: Invalid UUID string: 8240-unmigrated
> java.lang.IllegalArgumentException: Invalid UUID string: 8240-unmigrated
> at java.base/java.util.UUID.fromString1(UUID.java:280)
> at java.base/java.util.UUID.fromString(UUID.java:258)
> at
> org.dspace.content.BitstreamServiceImpl.findByIdOrLegacyId(BitstreamServiceImpl.java:452)
> at
> org.dspace.content.BitstreamServiceImpl.findByIdOrLegacyId(BitstreamServiceImpl.java:43)
> at
> org.dspace.statistics.SolrLoggerServiceImpl.reindexBitstreamHits(SolrLoggerServiceImpl.java:1484)
> at
> org.dspace.statistics.util.StatisticsClient.main(StatisticsClient.java:99)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:568)
> at
> org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:277)
> at
> org.dspace.app.launcher.ScriptLauncher.handleScript(ScriptLauncher.java:133)
> at
> org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:98)
>
> Any help would be greatly appreciated.
>
> James Holobetz
>
>
>
>

-- 
All messages to this mailing list should adhere to the Code of Conduct: 
https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
--- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/dspace-tech/CAAosP7X249xXHRvaPcKew9JNR%2B_DHYyhi1wRe_rw3Rz_28R31g%40mail.gmail.com.

[dspace-tech] stats-util error

[James Holobetz]

Hello All,

I am receiving this error when attempting to reindex the bitstreams using
stats-util -b:

holobetj ~ $ sudo -u dspace /opt/dspace/bin/dspace stats-util -b
Exception: Invalid UUID string: 8240-unmigrated
java.lang.IllegalArgumentException: Invalid UUID string: 8240-unmigrated
at java.base/java.util.UUID.fromString1(UUID.java:280)
at java.base/java.util.UUID.fromString(UUID.java:258)
at
org.dspace.content.BitstreamServiceImpl.findByIdOrLegacyId(BitstreamServiceImpl.java:452)
at
org.dspace.content.BitstreamServiceImpl.findByIdOrLegacyId(BitstreamServiceImpl.java:43)
at
org.dspace.statistics.SolrLoggerServiceImpl.reindexBitstreamHits(SolrLoggerServiceImpl.java:1484)
at
org.dspace.statistics.util.StatisticsClient.main(StatisticsClient.java:99)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at
org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:277)
at
org.dspace.app.launcher.ScriptLauncher.handleScript(ScriptLauncher.java:133)
at
org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:98)

Any help would be greatly appreciated.

James Holobetz

-- 
All messages to this mailing list should adhere to the Code of Conduct: 
https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
--- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/dspace-tech/CAAosP7X%3D0n2eN_X1H0aK_OvJAOQ6msePC3N02FEcHVAN4Lr4Fg%40mail.gmail.com.

Re: [dspace-tech] Re: solr error importing stats

[James Holobetz]

After evaluation I suspected the same thing. The big issue of the whole
matter is that DSpace 7.6, while it was exporting the statistics, was
adding escape characters (\) to the path character (again \) probably which
increased the "query" size.

I am just going to delete that record all together in our production system
so any further solr exports do not produce the same error when syncing our
development machine.

Thanks for your help Tim!

James

On Wed, Sep 13, 2023 at 10:25 AM DSpace Technical Support <
dspace-tech@googlegroups.com> wrote:

> Hi James,
>
> That long query looks suspiciously like a "directory traversal" attack
> that someone tried to (unsuccessfully) run against your system in the past
> using the search page.  For example:
> https://www.acunetix.com/websitesecurity/directory-traversal/  (Notice
> how the query you shared had "win.ini" which is a common directory
> traversal attack attempt)
>
> This sort of attack won't work on DSpace (so there's nothing to worry
> about). But it might have been logged in your statistics because it was
> attempted from the DSpace search page.
>
> Overall, my opinion is you may just want to *delete* this entry from your
> exported CSV.  It doesn't look like a valid statistical entry that you'd
> want to "count".  It looks like someone was attempting to attack your site
> (and failing to do so).
>
> Tim
>
> On Tuesday, September 12, 2023 at 4:56:51 PM UTC-5 James Holobetz wrote:
>
>> During our move from DSpace 6.x to DSpace 7.x we had to combine solr
>> shards and then use the UUIDfix tool to convert the old DSpace Object ID to
>> UUID. Anyways, I saved all the csv files for solr ingest and went looking
>> through them for clues about the "query" in question. The
>> solr-export-statistics dump from 6.3 looks different from the
>> solr-export-statistics dump from 7.6 for the query in question.
>>
>>
>>
>> On Tue, Sep 12, 2023 at 2:14 PM James Holobetz  wrote:
>>
>>> I have found the "query" string in question in the particular csv file
>>> that was dumped (solr-export-statistics) from our DSpace 7.6 production
>>> machine. I have attached the relevant files to help as to any clue what may
>>> be happening.
>>>
>>> Thank you,
>>>
>>> James
>>>
>>> On Tue, Sep 12, 2023 at 11:12 AM DSpace Technical Support <
>>> dspac...@googlegroups.com> wrote:
>>>
>>>> Hi James,
>>>>
>>>> I have to admit, I've never seen that error before.  My guess is
>>>> there's something odd/different (or incorrect) with the data that you are
>>>> trying to import.  But, I don't know what it could be.  That error mentions
>>>> the "query" field is the problematic one.  Have you looked at the data you
>>>> are trying to import to see why that "query" field is so long?  Maybe
>>>> something is incorrect in that import data, or maybe it's encoded
>>>> improperly and the script is stumbling over it?
>>>>
>>>> Tim
>>>>
>>>> On Monday, September 11, 2023 at 3:49:56 PM UTC-5 James Holobetz wrote:
>>>>
>>>>>
>>>>> (I sent it early be mistake)
>>>>>
>>>>>
>>>>> https://mail.google.com/mail/u/0/?tab=rm&ogbl#search/Exception+writing+document+id/FMfcgzGtvsbMhlrcPsHZXVSJjRcZSsvW
>>>>>
>>>>>
>>>>> https://stackoverflow.com/questions/37070593/how-to-deal-with-document-contains-at-least-one-immense-term-in-solr
>>>>>
>>>>>
>>>>> 1) What would cause this (on the production machine)?
>>>>>
>>>>> 2) How do I resolve this issue?
>>>>>
>>>>> Thank you
>>>>>
>>>>> On Mon, Sep 11, 2023 at 2:46 PM James Holobetz 
>>>>> wrote:
>>>>>
>>>>>> I am moving data from our production dspace 7.6 server to our
>>>>>> development dspace 7.6 server and I am repeatedly receiving this error:
>>>>>>
>>>>>> holobetj dspace $ dsp /opt/dspace/bin/dspace solr-import-statistics -c
>>>>>> No index name provided, defaulting to "statistics".
>>>>>> Exception: Error from server at http://localhost:8983/solr/statistics:
>>>>>> Exception writing document id 01072706-6b8a-420d-9bc0-cc637bce3df4 to the
>>>>>> index; possible analysis error: Document contains at least on

Re: [dspace-tech] Re: solr error importing stats

[James Holobetz]

During our move from DSpace 6.x to DSpace 7.x we had to combine solr shards
and then use the UUIDfix tool to convert the old DSpace Object ID to UUID.
Anyways, I saved all the csv files for solr ingest and went looking through
them for clues about the "query" in question. The solr-export-statistics
dump from 6.3 looks different from the solr-export-statistics dump from 7.6
for the query in question.



On Tue, Sep 12, 2023 at 2:14 PM James Holobetz  wrote:

> I have found the "query" string in question in the particular csv file
> that was dumped (solr-export-statistics) from our DSpace 7.6 production
> machine. I have attached the relevant files to help as to any clue what may
> be happening.
>
> Thank you,
>
> James
>
> On Tue, Sep 12, 2023 at 11:12 AM DSpace Technical Support <
> dspace-tech@googlegroups.com> wrote:
>
>> Hi James,
>>
>> I have to admit, I've never seen that error before.  My guess is there's
>> something odd/different (or incorrect) with the data that you are trying to
>> import.  But, I don't know what it could be.  That error mentions the
>> "query" field is the problematic one.  Have you looked at the data you are
>> trying to import to see why that "query" field is so long?  Maybe something
>> is incorrect in that import data, or maybe it's encoded improperly and the
>> script is stumbling over it?
>>
>> Tim
>>
>> On Monday, September 11, 2023 at 3:49:56 PM UTC-5 James Holobetz wrote:
>>
>>>
>>> (I sent it early be mistake)
>>>
>>>
>>> https://mail.google.com/mail/u/0/?tab=rm&ogbl#search/Exception+writing+document+id/FMfcgzGtvsbMhlrcPsHZXVSJjRcZSsvW
>>>
>>>
>>> https://stackoverflow.com/questions/37070593/how-to-deal-with-document-contains-at-least-one-immense-term-in-solr
>>>
>>>
>>> 1) What would cause this (on the production machine)?
>>>
>>> 2) How do I resolve this issue?
>>>
>>> Thank you
>>>
>>> On Mon, Sep 11, 2023 at 2:46 PM James Holobetz 
>>> wrote:
>>>
>>>> I am moving data from our production dspace 7.6 server to our
>>>> development dspace 7.6 server and I am repeatedly receiving this error:
>>>>
>>>> holobetj dspace $ dsp /opt/dspace/bin/dspace solr-import-statistics -c
>>>> No index name provided, defaulting to "statistics".
>>>> Exception: Error from server at http://localhost:8983/solr/statistics:
>>>> Exception writing document id 01072706-6b8a-420d-9bc0-cc637bce3df4 to the
>>>> index; possible analysis error: Document contains at least one immense term
>>>> in field="query" (whose UTF8 encoding is longer than the max length 32766),
>>>> all of which were skipped.  Please correct the analyzer to not produce such
>>>> terms.  The prefix of the first immense term is: '[117, 110, 101, 120, 105,
>>>> 115, 116, 105, 110, 103, 47, 46, 46, 47, 46, 46, 47, 46, 46, 47, 46, 46,
>>>> 47, 46, 46, 47, 46, 46, 47, 46]...', original message: bytes can be at most
>>>> 32766 in length; got 34396. Perhaps the document has an indexed string
>>>> field (solr.StrField) which is too large
>>>> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
>>>> Error from server at http://localhost:8983/solr/statistics: Exception
>>>> writing document id 01072706-6b8a-420d-9bc0-cc637bce3df4 to the index;
>>>> possible analysis error: Document contains at least one immense term in
>>>> field="query" (whose UTF8 encoding is longer than the max length 32766),
>>>> all of which were skipped.  Please correct the analyzer to not produce such
>>>> terms.  The prefix of the first immense term is: '[117, 110, 101, 120, 105,
>>>> 115, 116, 105, 110, 103, 47, 46, 46, 47, 46, 46, 47, 46, 46, 47, 46, 46,
>>>> 47, 46, 46, 47, 46, 46, 47, 46]...', original message: bytes can be at most
>>>> 32766 in length; got 34396. Perhaps the document has an indexed string
>>>> field (solr.StrField) which is too large
>>>>
>>>>
>>>> Looking in  the forums here I have seen the error very rarely:
>>>>
>>>> --
>> All messages to this mailing list should adhere to the Code of Conduct:
>> https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "DSpace Technical Support" group.
>> To unsubscribe from this

Re: [dspace-tech] Re: solr error importing stats

[James Holobetz]

I have found the "query" string in question in the particular csv file that
was dumped (solr-export-statistics) from our DSpace 7.6 production machine.
I have attached the relevant files to help as to any clue what may be
happening.

Thank you,

James

On Tue, Sep 12, 2023 at 11:12 AM DSpace Technical Support <
dspace-tech@googlegroups.com> wrote:

> Hi James,
>
> I have to admit, I've never seen that error before.  My guess is there's
> something odd/different (or incorrect) with the data that you are trying to
> import.  But, I don't know what it could be.  That error mentions the
> "query" field is the problematic one.  Have you looked at the data you are
> trying to import to see why that "query" field is so long?  Maybe something
> is incorrect in that import data, or maybe it's encoded improperly and the
> script is stumbling over it?
>
> Tim
>
> On Monday, September 11, 2023 at 3:49:56 PM UTC-5 James Holobetz wrote:
>
>>
>> (I sent it early be mistake)
>>
>>
>> https://mail.google.com/mail/u/0/?tab=rm&ogbl#search/Exception+writing+document+id/FMfcgzGtvsbMhlrcPsHZXVSJjRcZSsvW
>>
>>
>> https://stackoverflow.com/questions/37070593/how-to-deal-with-document-contains-at-least-one-immense-term-in-solr
>>
>>
>> 1) What would cause this (on the production machine)?
>>
>> 2) How do I resolve this issue?
>>
>> Thank you
>>
>> On Mon, Sep 11, 2023 at 2:46 PM James Holobetz  wrote:
>>
>>> I am moving data from our production dspace 7.6 server to our
>>> development dspace 7.6 server and I am repeatedly receiving this error:
>>>
>>> holobetj dspace $ dsp /opt/dspace/bin/dspace solr-import-statistics -c
>>> No index name provided, defaulting to "statistics".
>>> Exception: Error from server at http://localhost:8983/solr/statistics:
>>> Exception writing document id 01072706-6b8a-420d-9bc0-cc637bce3df4 to the
>>> index; possible analysis error: Document contains at least one immense term
>>> in field="query" (whose UTF8 encoding is longer than the max length 32766),
>>> all of which were skipped.  Please correct the analyzer to not produce such
>>> terms.  The prefix of the first immense term is: '[117, 110, 101, 120, 105,
>>> 115, 116, 105, 110, 103, 47, 46, 46, 47, 46, 46, 47, 46, 46, 47, 46, 46,
>>> 47, 46, 46, 47, 46, 46, 47, 46]...', original message: bytes can be at most
>>> 32766 in length; got 34396. Perhaps the document has an indexed string
>>> field (solr.StrField) which is too large
>>> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
>>> Error from server at http://localhost:8983/solr/statistics: Exception
>>> writing document id 01072706-6b8a-420d-9bc0-cc637bce3df4 to the index;
>>> possible analysis error: Document contains at least one immense term in
>>> field="query" (whose UTF8 encoding is longer than the max length 32766),
>>> all of which were skipped.  Please correct the analyzer to not produce such
>>> terms.  The prefix of the first immense term is: '[117, 110, 101, 120, 105,
>>> 115, 116, 105, 110, 103, 47, 46, 46, 47, 46, 46, 47, 46, 46, 47, 46, 46,
>>> 47, 46, 46, 47, 46, 46, 47, 46]...', original message: bytes can be at most
>>> 32766 in length; got 34396. Perhaps the document has an indexed string
>>> field (solr.StrField) which is too large
>>>
>>>
>>> Looking in  the forums here I have seen the error very rarely:
>>>
>>> --
> All messages to this mailing list should adhere to the Code of Conduct:
> https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
> ---
> You received this message because you are subscribed to the Google Groups
> "DSpace Technical Support" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to dspace-tech+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/dspace-tech/6810f574-69e6-4676-95ec-717b4ca22a72n%40googlegroups.com
> <https://groups.google.com/d/msgid/dspace-tech/6810f574-69e6-4676-95ec-717b4ca22a72n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
All messages to this mailing list should adhere to the Code of Conduct: 
https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
--- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To vie

[dspace-tech] Re: solr error importing stats

[James Holobetz]

(I sent it early be mistake)

https://mail.google.com/mail/u/0/?tab=rm&ogbl#search/Exception+writing+document+id/FMfcgzGtvsbMhlrcPsHZXVSJjRcZSsvW

https://stackoverflow.com/questions/37070593/how-to-deal-with-document-contains-at-least-one-immense-term-in-solr


1) What would cause this (on the production machine)?

2) How do I resolve this issue?

Thank you

On Mon, Sep 11, 2023 at 2:46 PM James Holobetz  wrote:

> I am moving data from our production dspace 7.6 server to our development
> dspace 7.6 server and I am repeatedly receiving this error:
>
> holobetj dspace $ dsp /opt/dspace/bin/dspace solr-import-statistics -c
> No index name provided, defaulting to "statistics".
> Exception: Error from server at http://localhost:8983/solr/statistics:
> Exception writing document id 01072706-6b8a-420d-9bc0-cc637bce3df4 to the
> index; possible analysis error: Document contains at least one immense term
> in field="query" (whose UTF8 encoding is longer than the max length 32766),
> all of which were skipped.  Please correct the analyzer to not produce such
> terms.  The prefix of the first immense term is: '[117, 110, 101, 120, 105,
> 115, 116, 105, 110, 103, 47, 46, 46, 47, 46, 46, 47, 46, 46, 47, 46, 46,
> 47, 46, 46, 47, 46, 46, 47, 46]...', original message: bytes can be at most
> 32766 in length; got 34396. Perhaps the document has an indexed string
> field (solr.StrField) which is too large
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> Error from server at http://localhost:8983/solr/statistics: Exception
> writing document id 01072706-6b8a-420d-9bc0-cc637bce3df4 to the index;
> possible analysis error: Document contains at least one immense term in
> field="query" (whose UTF8 encoding is longer than the max length 32766),
> all of which were skipped.  Please correct the analyzer to not produce such
> terms.  The prefix of the first immense term is: '[117, 110, 101, 120, 105,
> 115, 116, 105, 110, 103, 47, 46, 46, 47, 46, 46, 47, 46, 46, 47, 46, 46,
> 47, 46, 46, 47, 46, 46, 47, 46]...', original message: bytes can be at most
> 32766 in length; got 34396. Perhaps the document has an indexed string
> field (solr.StrField) which is too large
>
>
> Looking in  the forums here I have seen the error very rarely:
>
>

-- 
All messages to this mailing list should adhere to the Code of Conduct: 
https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
--- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/dspace-tech/CAAosP7WobVcztc2HfkBfYy7BkJz6O6mtbYLy7895mj2crmQJiQ%40mail.gmail.com.

[dspace-tech] solr error importing stats

[James Holobetz]

I am moving data from our production dspace 7.6 server to our development
dspace 7.6 server and I am repeatedly receiving this error:

holobetj dspace $ dsp /opt/dspace/bin/dspace solr-import-statistics -c
No index name provided, defaulting to "statistics".
Exception: Error from server at http://localhost:8983/solr/statistics:
Exception writing document id 01072706-6b8a-420d-9bc0-cc637bce3df4 to the
index; possible analysis error: Document contains at least one immense term
in field="query" (whose UTF8 encoding is longer than the max length 32766),
all of which were skipped.  Please correct the analyzer to not produce such
terms.  The prefix of the first immense term is: '[117, 110, 101, 120, 105,
115, 116, 105, 110, 103, 47, 46, 46, 47, 46, 46, 47, 46, 46, 47, 46, 46,
47, 46, 46, 47, 46, 46, 47, 46]...', original message: bytes can be at most
32766 in length; got 34396. Perhaps the document has an indexed string
field (solr.StrField) which is too large
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server at http://localhost:8983/solr/statistics: Exception writing
document id 01072706-6b8a-420d-9bc0-cc637bce3df4 to the index; possible
analysis error: Document contains at least one immense term in
field="query" (whose UTF8 encoding is longer than the max length 32766),
all of which were skipped.  Please correct the analyzer to not produce such
terms.  The prefix of the first immense term is: '[117, 110, 101, 120, 105,
115, 116, 105, 110, 103, 47, 46, 46, 47, 46, 46, 47, 46, 46, 47, 46, 46,
47, 46, 46, 47, 46, 46, 47, 46]...', original message: bytes can be at most
32766 in length; got 34396. Perhaps the document has an indexed string
field (solr.StrField) which is too large


Looking in  the forums here I have seen the error very rarely:

-- 
All messages to this mailing list should adhere to the Code of Conduct: 
https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
--- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/dspace-tech/CAAosP7WAoGGjXx_dCn%3DQzv1LQ%2Bxukv27RRR43WKLC40Ngp2fxg%40mail.gmail.com.

Re: [dspace-tech] workflow "approve" issues

[James Holobetz]

I appreciate your detailed feedback.

What you have detailed is exactly what methodology/process we are using as
well. I can enter it properly, can view it, claim it, and edit
it, return it to the pool, (I haven't tried to reject). My only issue is
that I can not "Approve" it properly after initial entry. The anomaly is
that I can "Approve" it after doing an "Edit." That is the only underlying
issue. Nothing else. Everything else works properly.

James

On Mon, Jul 31, 2023 at 4:41 PM Fitchett, Deborah <
deborah.fitch...@lincoln.ac.nz> wrote:

> You mention you’re using a multi-step process, so let me describe how our
> two-step process has been working for me in testing 7.4, in case that
> shines any light on your case:
>
>
>
>- The item is submitted.
>- It shows in the workflow task list as “Waiting for controller”
>- I claim it – it now shows as “Validation”
>- I can either edit, reject, or approve (ie workflow step 2 aka
>“editstep”)
>- If I approve it (without editing), it appears to disappear from the
>workflow. I need to refresh the screen to see it again in the workflow
>list, now showing as “Waiting for controller” again
>- I claim it again – it shows as “Validation” again – and I can either
>edit or approve (ie workflow step 3 aka “finaleditstep”)
>- If I approve it (without editing), it disappears from the workflow
>and is now published in the collection as expected.
>
>
>
> The status labels “Waiting for controller” and “Validation” are confusing
> in this kind of multi-step process because they actually have nothing to do
> with the steps, and there’s no way currently to add a label to show what
> step it’s at.  (See https://github.com/DSpace/dspace-angular/issues/1609
> for a description of a possible solution to this if someone’s interested in
> volunteering….)
>
>
>
> So possibly that’s part of what’s causing confusion for you?
>
>
>
> And/or you may be seeing a difference in behaviour between directly
> approving vs editing-then-approving because when you edit-then-approve, the
> process of returning from the item view to the workflow list reloads and
> updates that list. But if you approve direct from the workflow list, the
> list doesn’t fully update so the item seems to disappear from it and you
> need to refresh the page to see it.
>
>
>
> Deborah
>
>
>
> *From:* dspace-tech@googlegroups.com  *On
> Behalf Of *James Holobetz
> *Sent:* Tuesday, August 1, 2023 6:40 AM
> *To:* dspace-tech@googlegroups.com
> *Subject:* [dspace-tech] workflow "approve" issues
>
>
>
> You don't often get email from jholob...@gmail.com. Learn why this is
> important <https://aka.ms/LearnAboutSenderIdentification>
>
>
>
> *Caution:* This email originated from outside our organisation. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> Hi Everybody,
>
>
>
> I am experiencing weird behaviour when entering an item that has a
> multi-step process. Entering the item is fine and so is review and edit.
> When I try to "Approve" it for public consumption, the progress bar is
> green and it appears to remove it from the "Workflow tasks" area. But the
> item is not released and when you go back to the Workflow tasks area, the
> item is still there.The only time it successfully releases it is when,
> during the process, you edit something (even something as simple as remove
> and add back a period in the abstract). Nothing shows up in the logs at
> all. I have validated both the item-submission.xml and the
> submission-forms.xml. I am at a loss.
>
>
>
> For reference, we are using DSpace 7.6 and also, we ported our old
> input-forms.xml and item-submission.xml files and used the "./dspace
> submission-forms-migrate" script.
>
>
>
> Best regards,
>
>
>
> James Holobetz
>
> --
> All messages to this mailing list should adhere to the Code of Conduct:
> https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
> ---
> You received this message because you are subscribed to the Google Groups
> "DSpace Technical Support" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to dspace-tech+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/dspace-tech/CAAosP7WgOLej_%3DqYSZCUKrEQUz%2B_k-Eu%2Bb4%3DtFQVvqywMb3FDA%40mail.gmail.com
> <https://groups.google.com/d/msgid/dspace-tech/CAAosP7WgOLej_%3DqYSZCUKrEQUz%2B_k-Eu%2Bb4%3DtFQVvqywMb3FDA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> --

[dspace-tech] workflow "approve" issues

[James Holobetz]

Hi Everybody,

I am experiencing weird behaviour when entering an item that has a
multi-step process. Entering the item is fine and so is review and edit.
When I try to "Approve" it for public consumption, the progress bar is
green and it appears to remove it from the "Workflow tasks" area. But the
item is not released and when you go back to the Workflow tasks area, the
item is still there.The only time it successfully releases it is when,
during the process, you edit something (even something as simple as remove
and add back a period in the abstract). Nothing shows up in the logs at
all. I have validated both the item-submission.xml and the
submission-forms.xml. I am at a loss.

For reference, we are using DSpace 7.6 and also, we ported our old
input-forms.xml and item-submission.xml files and used the "./dspace
submission-forms-migrate" script.

Best regards,

James Holobetz

-- 
All messages to this mailing list should adhere to the Code of Conduct: 
https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
--- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/dspace-tech/CAAosP7WgOLej_%3DqYSZCUKrEQUz%2B_k-Eu%2Bb4%3DtFQVvqywMb3FDA%40mail.gmail.com.

Re: [dspace-tech] DSpace 7.5 Solr Statistics migration from 5.10 with sharding by year

[James Holobetz]

Hi Tomas,

I recently had this issue and I believe that I have found a solution, which
I will document in the next few days. The long and the short of it is that
DSpace 7 does not support solr shards. You have to create one large solr shard
(statistics) from the multiple shards. The biggest problem I found doing
this was that DSpace was only ingesting the current year statistics only.
The solution was to rename the *csv files that are dumped by solr-export-
statistics. For example: the csv files for the solr core "statistics-2012"
will look something like this -- statistics-2012_export_2013-12_5.csv. You
have to rename all the csv files to remove the -2012 in the filename to
look like this: statistics_export_2013-12_5.csv. I downloaded the zipped up
cores in csv form to my windows machine so I could use a bulk rename tool
to remove the year suffix in each core. I then uploaded them to my linux
box running DSpace and ingested each one using the solr-import-statistics tool.
This is a very time consuming task.

Hope this helps and I will try to document this in the next few days.

Best regards,

James Holobetz

On Fri, Mar 24, 2023 at 3:37 PM Tomas Hajek  wrote:

> Hello,
>I am working on migrating a DSpace 5.10 installation to a new server
> running DSpace 7.5.  I have the basic installation running on RHEL 8.7 with
> Tomcat 9.0.71, Solr 8.11.2, node.js 16.18.1, and pm2 5.2.2.
> I was able to import the database and assetstore and I set up the Solr
> cores (authority,oai,search,statistics) from the installation instructions.
>The Solr statistics from the 5.10 installation are sharded by year and
> I exported with the following:
>
> bin/dspace solr-export-statistics -i statistics-2015
> bin/dspace solr-export-statistics -i statistics-2016
> ...
> bin/dspace solr-export-statistics -i statistics-2022
>
> I have copied the exported files to the new 7.5 server
> into /opt/dspace/solr-export and am trying to import them but I get the
> following error (example when trying to import the 2015 statistics):
>
> /opt/dspace/bin/dspace solr-import-statistics -i statistics-2015
> Exception: Error from server at http://localhost:8983/solr/statistics-2015:
> Expected mime type application/octet-stream but got text/html. 
> 
> 
> Error 404 Not Found
> 
> HTTP ERROR 404 Not Found
> 
> URI:/solr/statistics-2015/admin/luke
> STATUS:404
> MESSAGE:Not Found
> SERVLET:default
> 
>
> 
> 
>
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> Error from server at http://localhost:8983/solr/statistics-2015: Expected
> mime type application/octet-stream but got text/html. 
> 
> 
> Error 404 Not Found
> 
> HTTP ERROR 404 Not Found
> 
> URI:/solr/statistics-2015/admin/luke
> STATUS:404
> MESSAGE:Not Found
> SERVLET:default
> 
>
> 
> 
>
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:635)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:214)
> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:231)
> at
> org.dspace.util.SolrImportExport.getMultiValuedFields(SolrImportExport.java:482)
> at org.dspace.util.SolrImportExport.importIndex(SolrImportExport.java:433)
> at org.dspace.util.SolrImportExport.main(SolrImportExport.java:148)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:568)
> at
> org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:277)
> at
> org.dspace.app.launcher.ScriptLauncher.handleScript(ScriptLauncher.java:133)
> at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:98)
>
> Presumably this is due to not having the sharded statistics-20## cores in
> Solr configured but I'm not sure at this point how to add and configure
> them so I can import the statistics.  I am not very familiar with Solr.
>
> Can anyone enlighten me on how I might do this or correct my steps or let
> me know what else to look at.
>
> Any assistance would be greatly appreciated.
> Thank you,
>  -Tomas
>
> --
> All messages to this mailing list should adhere to the Code of Conduct:
> https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
> ---
> You received this message because you are subscribed to the Google Groups
> "DSpa

[dspace-tech] GeoLite2 Database No Longer Publicly Available

[James Holobetz]

FYI

https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/

-- 
All messages to this mailing list should adhere to the DuraSpace Code of 
Conduct: https://duraspace.org/about/policies/code-of-conduct/
--- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/dspace-tech/CAAosP7WRob5t7G_x9VsSTMBXHeGZoTBmzOSwL%3DF9HnxXP6hycA%40mail.gmail.com.

[dspace-tech] Jpeg Thumbnails not being displayed

[James Holobetz]

Hi,

I am experiencing an issue with Dspace 6.3 on Suse Linux using Mirage 2.
With xmlui.theme.mirage.item-list.emphasis = file
the jpeg thumbnails that were generated by PDFBox for PDF's are displayed
as desired but the jpeg thumbnails generated JPEGFilter or ImageMagick for
jpeg's are not displayed in list view or full view but the the thumbnail is
displayed in item view.

I have tested this on our customized Dspace instance and on a vanilla
Dspace instance with the same result.

Any help is greatly appreciated.

Best regards,

James Holobetz

-- 
All messages to this mailing list should adhere to the DuraSpace Code of 
Conduct: https://duraspace.org/about/policies/code-of-conduct/
--- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/dspace-tech/CAAosP7VypR5TqJtaM6R7RhdUm2XmWOCvsTbYeF_F80BWU0Wi3A%40mail.gmail.com.

[dspace-tech] Mirage 2 Dspace 6.x use different version of Ruby

[James Holobetz]

Hello all,

I have encountered the dreaded build time error described in this link:
https://jira.duraspace.org/browse/DS-4115

TLDR - The build fails because it is expecting a version of Ruby >= 2.2.

We are using SLES 12 and the default Ruby is at version 2.1 (internally it
is probably higher; SUSE security is like BSD). My SysAdmin has installed
2.3 but has to keep the default 2.1 in place for retroactive purposes. How
do I set it so that the build process of Mirage 2 is pointed to Ruby 2.3
instead of the default?

Best regards and thanks in advance,

James Holobetz

-- 
All messages to this mailing list should adhere to the DuraSpace Code of 
Conduct: https://duraspace.org/about/policies/code-of-conduct/
--- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To post to this group, send email to dspace-tech@googlegroups.com.
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

[dspace-tech] Dspace 5.6 Workflow Issue

[James Holobetz]

I am experiencing the https://jira.duraspace.org/browse/DS-2920 problem and
was wondering if anyone had an idea how to fix the issue.

Is this the fix?

Marie-Helene Vézina
 added
a comment - 16/Dec/15 1:39 PM - edited

The workaround we found (at least for now) is to force DSpace to look into
the database instead of using the cache. Doing so doesn't throw an error.

So in file:

[Source]\dspace-api\src\main\java\org\dspace\workflow\WorkflowItem.java

We have added one more line:

public static WorkflowItem find(Context context, int id)
throws SQLException
{
// First check the cache
WorkflowItem fromCache = (WorkflowItem) context.fromCache(
WorkflowItem.class, id);

fromCache = null; // <= this forces Dspace not to use the cache

if (fromCache != null)
{   return fromCache;  }

(...)

I don't know if intervening with the cache is the right approach to solve
this problem but it seems to work for us.



James

-- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To post to this group, send email to dspace-tech@googlegroups.com.
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

Re: [dspace-tech] String index out of range: 3

[James Holobetz]

Thanks Tim, we suspected as much.

As always, we appreciate your help.

James


On Oct 31, 2016 3:27 PM, "Tim Donohue"  wrote:

Hi James,

As noted in the Tomcat ticket you linked to (https://bz.apache.org/bugzill
a/show_bug.cgi?id=58999), and the DS-3142 ticket, this is a bug in Tomcat
8.0.32.  I suspect this Tomcat bug is not just triggered in DSpace 6, but
also very likely triggered in any version of DSpace that uses XMLUI (JSPUI
seems to not trigger the issue in Tomcat).  Unfortunately, some OSes still
ship with Tomcat 8.0.32 by default. For example, Ubuntu 16.04 still
defaults to Tomcat 8.0.32, even though others are complaining about this
same bug, see https://bugs.launchpad.net/ubuntu/+source/tomcat8/+bug/1606331

So, long story short, I don't think there's anything wrong in your DSpace
site itself.  I suspect you will have to install a different version of
Tomcat to fix these errors.

- Tim
On 10/31/2016 3:07 PM, James Holobetz wrote:

Hello,

We are building a Dspace 5.6 server for production and we are running into
an issue when trying to ingest materials. We recieve ther error: String
index out of range: 3.

I did some research and this has been noted in https://jira.duraspace.org/
browse/DS-3142 and http://bz.apache.org/bugzilla/show_bug.cgi?id=58999 but
from the initial reports in DS-3142 this effect Dspace 6.x and not 5.x but
we are in fact using the noted tomcat version.


We are running (SUSE Linux Enterprise Server) SLES 12 SP1
(all packages are part of the SLES 12 SP1 system)

tomcat 8.0.32
maven 3.3.9
java 1.7.0_111 (OpenJDK)
ant 1.9.4
postgres 9.4.9



Can anybody provide any help/direction in this matter?


James Holobetz


-- 
You received this message because you are subscribed to the Google Groups
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to dspace-tech+unsubscr...@googlegroups.com.
To post to this group, send email to dspace-tech@googlegroups.com.
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.


-- 
Tim Donohue
Technical Lead for DSpace & DSpaceDirect
DuraSpace.org | DSpace.org | DSpaceDirect.org

-- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To post to this group, send email to dspace-tech@googlegroups.com.
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

Re: [dspace-tech] String index out of range: 3

[James Holobetz]

It should be noted that this error occurs with the default/unaltered Mirage
theme as well (using XMLUI). All themes are derivatives of Mirage and have
reverted back to Mirage to test as well.

James

On Mon, Oct 31, 2016 at 2:32 PM, James Holobetz  wrote:

> We are encountering this as we are using any create/edit command in the
> Context menu (UI).
>
> Here is the log output (just realized the sitemap error; any elaboration
> would be wonderful);
>
>
> 2016-10-31 14:28:07,620 ERROR cocoon.handled  - Sitemap: error calling
> function 'startCreateCommunity'
> at  - resource://aspects/Administrative/sitemap.xmap:
> 261:51
> at  - resource://aspects/
> Administrative/sitemap.xmap:257:53
> at  - resource://aspects/Administrative/sitemap.xmap:
> 252:43
> at  - resource://aspects/
> Administrative/sitemap.xmap:128:45
> at  - file:///opt/dspace/webapps/
> xmlui/aspects/aspects.xmap:89:72
> at  - file:///opt/dspace/webapps/
> xmlui/aspects/aspects.xmap:79:34
> at  - file:///opt/dspace/webapps/
> xmlui/aspects/aspects.xmap:78:36
> at  - file:///opt/dspace/webapps/
> xmlui/sitemap.xmap:488:100
> at  - file:///opt/dspace/webapps/
> xmlui/sitemap.xmap:487:49
> at  - resource://aspects/EPerson/
> sitemap.xmap:302:31
> at  - resource://aspects/EPerson/
> sitemap.xmap:107:38
> at  - resource://aspects/EPerson/sitemap.xmap:96:19
> at  - resource://aspects/Submission/
> sitemap.xmap:251:38
> at  -
> resource://aspects/Submission/sitemap.xmap:107:45
> at  - resource://aspects/Submission/
> sitemap.xmap:104:26
> at  - resource://aspects/Statistics/
> sitemap.xmap:292:36
> at  - resource://aspects/Statistics/
> sitemap.xmap:37:19
> at  - resource://aspects/Workflow/
> sitemap.xmap:139:38
> at  - resource://aspects/Workflow/sitemap.xmap:76:26
> at  - file:///opt/dspace/webapps/
> xmlui/aspects/aspects.xmap:85:34
> at  -
> file:///opt/dspace/webapps/xmlui/aspects/aspects.xmap:84:43
> at  - file:///opt/dspace/webapps/
> xmlui/aspects/aspects.xmap:83:22
> at  - file:///opt/dspace/webapps/
> xmlui/themes/oURspace/sitemap.xmap:179:46
> at  - file:///opt/dspace/webapps/
> xmlui/themes/oURspace/sitemap.xmap:101:44
> at  - file:///opt/dspace/webapps/
> xmlui/themes/oURspace/sitemap.xmap:97:59
> at  -
> file:///opt/dspace/webapps/xmlui/themes/oURspace/sitemap.xmap:82:51
> at  -
> file:///opt/dspace/webapps/xmlui/themes/oURspace/sitemap.xmap:78:51
> at  -
> file:///opt/dspace/webapps/xmlui/themes/oURspace/sitemap.xmap:70:51
> at  - file:///opt/dspace/webapps/
> xmlui/themes/oURspace/sitemap.xmap:171:67
> at  - file:///opt/dspace/webapps/
> xmlui/themes/oURspace/sitemap.xmap:167:37
> org.apache.cocoon.ProcessingException: Sitemap: error calling function
> 'startCreateCommunity'
> at  - resource://aspects/Administrative/sitemap.xmap:
> 261:51
> at  - resource://aspects/
> Administrative/sitemap.xmap:257:53
> at  - resource://aspects/Administrative/sitemap.xmap:
> 252:43
> at  - resource://aspects/
> Administrative/sitemap.xmap:128:45
> at  - file:///opt/dspace/webapps/
> xmlui/aspects/aspects.xmap:89:72
> at  - file:///opt/dspace/webapps/
> xmlui/aspects/aspects.xmap:79:34
> at  - file:///opt/dspace/webapps/
> xmlui/aspects/aspects.xmap:78:36
> at  - file:///opt/dspace/webapps/
> xmlui/sitemap.xmap:488:100
> at  - file:///opt/dspace/webapps/
> xmlui/sitemap.xmap:487:49
> at  - resource://aspects/EPerson/
> sitemap.xmap:302:31
> at  - resource://aspects/EPerson/
> sitemap.xmap:107:38
> at  - resource://aspects/EPerson/sitemap.xmap:96:19
> at  - resource://aspects/Submission/
> sitemap.xmap:251:38
> at  -
> resource://aspects/Submission/sitemap.xmap:107:45
> at  - resource://aspects/Submission/
> sitemap.xmap:104:26
> at  - resource://aspects/Statistics/
> sitemap.xmap:292:36
> at  - resource://aspects/Statistics/
> sitemap.xmap:37:19
> at  - resource://aspects/Workflow/
> sitemap.xmap:139:38
> at  - resource://aspects/Workflow/sitemap.xmap:76:26
> at  - file:///opt/dspace/webapps/
> xmlui/aspects/aspects.xmap:85:34
> at  -
> file:///opt/dspace/webapps/xmlui/aspects/aspects.xmap:84:43
> at  - file:///opt/dspace/webapps/
> xmlui/aspects/aspects.xmap:83:22
> at  - file:///opt/dspace/webapps/
> xmlui/themes/oURspace/sitemap.xmap:179:46
> at  - file:///opt/dspace/

Re: [dspace-tech] String index out of range: 3

[James Holobetz]

iptRuntime.java:1375)
at
org.mozilla.javascript.ScriptRuntime.getObjectProp(ScriptRuntime.java:1364)
at
org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:2965)
at
org.mozilla.javascript.Interpreter.interpret(Interpreter.java:2394)
at
org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:162)
at
org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:393)
at
org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:2834)
at
org.mozilla.javascript.InterpretedFunction.exec(InterpretedFunction.java:173)
at
org.apache.cocoon.components.flow.javascript.fom.FOM_JavaScriptInterpreter.setupContext(FOM_JavaScriptInterpreter.java:465)
at
org.apache.cocoon.components.flow.javascript.fom.FOM_JavaScriptInterpreter.callFunction(FOM_JavaScriptInterpreter.java:585)
at
org.apache.cocoon.components.treeprocessor.sitemap.CallFunctionNode.invoke(CallFunctionNode.java:109)
... 239 more
2016-10-31 14:28:07,625 INFO
 org.apache.cocoon.components.treeprocessor.sitemap.HandleErrorsNode  -
Processing handle-errors at  -
file:///opt/dspace/webapps/xmlui/themes/oURspace/sitemap.xmap:188:28
2016-10-31 14:28:07,642 INFO  cocoon.access  - 'admin/community' Processed
by Apache Cocoon in 35 milliseconds.

On Mon, Oct 31, 2016 at 2:14 PM, Terry Brady 
wrote:

> Are you encountering this issue while performing an item submission
> through the UI, or are you running a bulk ingest from the command line?
>
> Could you provide some additional details from your log files?
>
> Terry
>
> On Mon, Oct 31, 2016 at 1:07 PM, James Holobetz 
> wrote:
>
>> Hello,
>>
>> We are building a Dspace 5.6 server for production and we are running
>> into an issue when trying to ingest materials. We recieve ther error:
>> String index out of range: 3.
>>
>> I did some research and this has been noted in
>> https://jira.duraspace.org/browse/DS-3142 and http://bz.a
>> pache.org/bugzilla/show_bug.cgi?id=58999 but from the initial reports in
>> DS-3142 this effect Dspace 6.x and not 5.x but we are in fact using the
>> noted tomcat version.
>>
>>
>> We are running (SUSE Linux Enterprise Server) SLES 12 SP1
>> (all packages are part of the SLES 12 SP1 system)
>>
>> tomcat 8.0.32
>> maven 3.3.9
>> java 1.7.0_111 (OpenJDK)
>> ant 1.9.4
>> postgres 9.4.9
>>
>>
>>
>> Can anybody provide any help/direction in this matter?
>>
>>
>> James Holobetz
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "DSpace Technical Support" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to dspace-tech+unsubscr...@googlegroups.com.
>> To post to this group, send email to dspace-tech@googlegroups.com.
>> Visit this group at https://groups.google.com/group/dspace-tech.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Terry Brady
> Applications Programmer Analyst
> Georgetown University Library Information Technology
> http://georgetown-university-libraries.github.io/
> <https://www.library.georgetown.edu/lit/code>
> 425-298-5498 (Seattle, WA)
>

-- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To post to this group, send email to dspace-tech@googlegroups.com.
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

[dspace-tech] String index out of range: 3

[James Holobetz]

Hello,

We are building a Dspace 5.6 server for production and we are running into
an issue when trying to ingest materials. We recieve ther error: String
index out of range: 3.

I did some research and this has been noted in
https://jira.duraspace.org/browse/DS-3142 and
http://bz.apache.org/bugzilla/show_bug.cgi?id=58999 but from the initial
reports in DS-3142 this effect Dspace 6.x and not 5.x but we are in fact
using the noted tomcat version.


We are running (SUSE Linux Enterprise Server) SLES 12 SP1
(all packages are part of the SLES 12 SP1 system)

tomcat 8.0.32
maven 3.3.9
java 1.7.0_111 (OpenJDK)
ant 1.9.4
postgres 9.4.9



Can anybody provide any help/direction in this matter?


James Holobetz

-- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To post to this group, send email to dspace-tech@googlegroups.com.
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

[dspace-tech] Solr Issue

[James Holobetz]

: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:784)
... 10 more


Can anybody provide a clue as to why it is doing this now as it was fine
before and nothing has changed.

Thank you,


James Holobetz

-- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To post to this group, send email to dspace-tech@googlegroups.com.
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

[dspace-tech] Is this a valid warning -- Major DSpace security vulnerability affecting all XMLUI sites

[James Holobetz]

Is this from Tim Donohue?

*My supervisor received a direct email with the header: *

[URGENT] Major DSpace security vulnerability affecting all XMLUI sites


*The body of the message states:*

Hello,


The DSpace Committers team has been notified of a major security 
vulnerability affecting all sites that use the XMLUI (DSpace versions 
1.5.x, 1.6.x, 1.7.x, 1.8.x, 3.x, 4.x and 5.x). JSPUI sites are unaffected 
by this vulnerability.


If exploited, this vulnerability may allow an anonymous user to view any 
file on your filesystem that is accessible to the Tomcat user account. By 
default, this may include your dspace.cfg file (and all related DSpace 
configurations) as well as any operating system files that are readable by 
the Apache Tomcat user account.


The Committers team is working rapidly to patch this vulnerability in the 
DSpace XMLUI codebase and release security fixes for 3.x, 4.x and 5.x 
versions.


We expect 3.x, 4.x and 5.x security releases (and patches) to be made 
publicly available on Monday, March 21st. In the meantime, we have quick 
fixes available at the bottom of this email which will ensure your site is 
not vulnerable to these attacks. (Please pass them along to the individual 
who manages your DSpace.)


We ask that you please keep this information confidential and avoid posting 
questions to our public DSpace mailing lists. In order to protect DSpace 
sites, we will not be disclosing any further details of the vulnerability 
until the security releases/patches are available.


If you have any questions or concerns, we’ve set up a secur...@dspace.org 
email address which can be used to privately contact the DSpace Committers 
and DuraSpace staff.


The quick fix options can be found below. Please apply one of them to your 
site immediately. Because of the nature of this vulnerability, we also 
recommend changing any passwords (or secure information) that appear in 
your DSpace configuration files (e.g. database connection passwords).


Sincerely,


Tim Donohue (on behalf of the DSpace Committers)

Tech Lead for DSpace



Quick Fixes for XMLUI Vulnerability 


We HIGHLY RECOMMEND applying one of the following “quick fixes” to your 
production site immediately. These quick fixes are designed to block all 
known attack paths and may be left in place until you are able to upgrade 
to one of the forthcoming DSpace security releases. Because of the nature 
of this vulnerability, we also recommend changing any passwords (or secure 
information) that appear in your DSpace configuration files (e.g. database 
connection passwords).


We have three quick fixes available, based on your local DSpace setup. You 
only need to choose ONE.

   - 
   
   Apache Web Server Quick Fix (for sites that run an Apache Web Server in 
   front of Tomcat)
   - 
   
   NGINX Web Server Quick Fix (for sites that run an NGINX Web Server in 
   front of Tomcat)
   - 
   
   DSpace XMLUI Sitemap Quick Fix (for sites that simply run Tomcat or 
   another Java servlet container)
   

If you have any questions or concerns about these quick fixes, please email 
secur...@dspace.org. These quick fixes are all considered production-ready, 
and have already been applied to DSpace sites managed by various DSpace 
Committers.


Option #1: Apache Web Server Quick Fix 

For any sites that use Apache in front of Tomcat, you can block all 
affected URLs at the Apache level using "mod_rewrite". This does not 
actually fix the problem in DSpace, but it does block access to the 
vulnerable URLs (until you are able to upgrade). For example:


# Temporary block using Apache + mod_rewrite

# This redirects all vulnerable URLs to /error 

# (which doesn't exist and throws a 404 response)
RewriteEngine On
RewriteRule ^/+themes/.*:.*$ /error [R=permanent,L]


# If your DSpace XMLUI URL starts with "/xmlui", you should use this 
instead:
# RewriteRule ^/+xmlui/+themes/.*:.*$ /xmlui/error [R=permanent,L]

After putting these rules in place, you should be able to simply reload 
Apache to apply these changes to your site (e.g. sudo service apache2 
reload). Be sure to apply this fix for both HTTP and HTTPS URLs, if your 
site responds to both.


To verify the quick fix is working, visit a URL like: 
http://[dspace.url]/themes/Reference/test:url (Be sure to test both HTTP 
and HTTPS). The URL should be redirected to [dspace.url]/error/ and a Page 
Cannot be Found response returned. As long as this occurs, the quick fix 
was successful.
Option #2: NGINX Web Server Quick Fix 

For any sites that use NGINX in front of Tomcat, you can block all affected 
URLs from NGINX. This does not actually fix the problem in DSpace, but it 
does block access to the vulnerable URLs (until you are able to upgrade). 
Add this to your server or location directive:


rewrite ^/+themes/.*:.*$ /error permanent;


# If your DSpace XMLUI URL starts with “/xmlui”, you should use this 
instead:

# rewrite ^/+xmlui/+themes/.*:.*$ /xmlui/error permanent;

Then run n