Re: [Dspace-tech] Error creating indexes 'invalid multibyte character for locale'
Problem solved after upgrading to postgres to 8.4 and re-ran the indexer! Cheers, Osama Alkadi On 03/11/2010, at 1:57 PM, Steve Swinsburg wrote: > Thanks Stuart, that did the trick. We are now working through the invalid > records. > > For reference though, we had to enable debugging on the dspace packages as > well: > log4j.logger.org.dspace=DEBUG, A1 > > cheers, > Steve > > > On 01/11/2010, at 4:27 PM, Stuart Lewis wrote: > >> Hi Steve, >> >> See: >> >> - >> https://wiki.duraspace.org/display/DSPACE/TechnicalFaq#TechnicalFaq-SettingloggingleveluptoDEBUG >> >> If you set your logging to DEBUG, they will be *very* verbose, but should >> show you the item that is failing, and the SQL queries in particular that >> are causing the problem. >> >> Thanks, >> >> >> Stuart Lewis >> IT Innovations Analyst and Developer >> Te Tumu Herenga The University of Auckland Library >> Auckland Mail Centre, Private Bag 92019, Auckland 1142, New Zealand >> Ph: +64 (0)9 373 7599 x81928 >> >> >> >> On 1/11/2010, at 4:59 PM, Steve Swinsburg wrote: >> >>> Hi, >>> >>> We've created a new dump with -E UNICODE added to the dump command, >>> imported it into a clean UTF-8 database, upgraded via the 15-16 sql update >>> script which completes successfully, but as soon as we run the index-init >>> script again, the same issue occurs. It would be helpful if the exception >>> told us which item was causing the issue. >>> >>> We ran the checks and everything is UTF-8 so I have no idea why it's still >>> failing. >>> >>> We are now looking at different way of migrating our data: export and then >>> import the collections. This is going to take quite some time which is not >>> ideal, but seems to be the only way we can get our data migrated. >>> >>> Thanks for you help, >>> Steve >>> >>> >>> On 29/10/2010, at 2:05 PM, Kim Shepherd wrote: >>> Hi Steve, As well as Keith's suggestion of inspecting your locale, I'd try a redump of the database with -E UNICODE passed to 'pg_dump', otherwise the dumped tables will still be in their original encoding. http://www.postgresql.org/docs/8.4/static/app-pgdump.html Cheers, Kim On 28 October 2010 10:46, Steve Swinsburg wrote: Hi all, We've completed another restore but are still having the same issue. This time we created a new database with the UNICODE encoding as per the installation manual: createdb -U dspace -E UNICODE dspace We then took an export of the current production database: pg_dump dspace > dspace-prod.dump and imported it into the new database: psql -d dspace -f dspace-prod.dump We then upgraded the database to 1.6: psql -U dspace -f /dspace-1.6-2-src-release/dspace/etc/postgres/database_schema_15_16.sql dspace And ran the index-init script: [dspace]/bin/dspace index-init But still the same issue as before. >> dspace index-init -v >> Exception: org.postgresql.util.PSQLException: ERROR: invalid multibyte >> character for locale >> org.dspace.browse.BrowseException: org.postgresql.util.PSQLException: >> ERROR: invalid multibyte character for locale >> at >> org.dspace.browse.BrowseCreateDAOPostgres.insertDistinctRecord(BrowseCreateDAOPostgres.java:691) >> at >> org.dspace.browse.BrowseCreateDAOPostgres.getDistinctID(BrowseCreateDAOPostgres.java:566) >> at org.dspace.browse.IndexBrowse.indexItem(IndexBrowse.java:485) >> at org.dspace.browse.IndexBrowse.createIndex(IndexBrowse.java:1148) >> at org.dspace.browse.IndexBrowse.initBrowse(IndexBrowse.java:1098) >> at org.dspace.browse.IndexBrowse.main(IndexBrowse.java:749) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:212) Is there anything wrong with the above process? The next step we are going to attempt is checking the actual SQL dump for the bad characters but it is massive (1.5 Gb) thanks, Steve On 26/10/2010, at 10:10 AM, Steve Swinsburg wrote: > Hi Kim, > > We took a *copy* of the database and refreshed it back to dev, so it is > identical to the production one, then upgraded it in dev. So I'll need to > find out if the original database from production was created that way. > > Perhaps we should take a dump of the production database instead, and > then import that into a fresh UTF8 cre
Re: [Dspace-tech] Error creating indexes 'invalid multibyte character for locale'
Thanks Stuart, that did the trick. We are now working through the invalid records. For reference though, we had to enable debugging on the dspace packages as well: log4j.logger.org.dspace=DEBUG, A1 cheers, Steve On 01/11/2010, at 4:27 PM, Stuart Lewis wrote: > Hi Steve, > > See: > > - > https://wiki.duraspace.org/display/DSPACE/TechnicalFaq#TechnicalFaq-SettingloggingleveluptoDEBUG > > If you set your logging to DEBUG, they will be *very* verbose, but should > show you the item that is failing, and the SQL queries in particular that are > causing the problem. > > Thanks, > > > Stuart Lewis > IT Innovations Analyst and Developer > Te Tumu Herenga The University of Auckland Library > Auckland Mail Centre, Private Bag 92019, Auckland 1142, New Zealand > Ph: +64 (0)9 373 7599 x81928 > > > > On 1/11/2010, at 4:59 PM, Steve Swinsburg wrote: > >> Hi, >> >> We've created a new dump with -E UNICODE added to the dump command, imported >> it into a clean UTF-8 database, upgraded via the 15-16 sql update script >> which completes successfully, but as soon as we run the index-init script >> again, the same issue occurs. It would be helpful if the exception told us >> which item was causing the issue. >> >> We ran the checks and everything is UTF-8 so I have no idea why it's still >> failing. >> >> We are now looking at different way of migrating our data: export and then >> import the collections. This is going to take quite some time which is not >> ideal, but seems to be the only way we can get our data migrated. >> >> Thanks for you help, >> Steve >> >> >> On 29/10/2010, at 2:05 PM, Kim Shepherd wrote: >> >>> Hi Steve, >>> >>> As well as Keith's suggestion of inspecting your locale, I'd try a redump >>> of the database with -E UNICODE passed to 'pg_dump', otherwise the dumped >>> tables will still be in their original encoding. >>> >>> http://www.postgresql.org/docs/8.4/static/app-pgdump.html >>> >>> Cheers, >>> >>> Kim >>> >>> On 28 October 2010 10:46, Steve Swinsburg wrote: >>> Hi all, >>> >>> We've completed another restore but are still having the same issue. This >>> time we created a new database with the UNICODE encoding as per the >>> installation manual: >>> createdb -U dspace -E UNICODE dspace >>> We then took an export of the current production database: >>> pg_dump dspace > dspace-prod.dump >>> and imported it into the new database: >>> psql -d dspace -f dspace-prod.dump >>> >>> We then upgraded the database to 1.6: >>> >>> psql -U dspace -f >>> /dspace-1.6-2-src-release/dspace/etc/postgres/database_schema_15_16.sql >>> dspace >>> >>> >>> And ran the index-init script: >>> [dspace]/bin/dspace index-init >>> But still the same issue as before. >>> > dspace index-init -v > Exception: org.postgresql.util.PSQLException: ERROR: invalid multibyte > character for locale > org.dspace.browse.BrowseException: org.postgresql.util.PSQLException: > ERROR: invalid multibyte character for locale > at > org.dspace.browse.BrowseCreateDAOPostgres.insertDistinctRecord(BrowseCreateDAOPostgres.java:691) > at > org.dspace.browse.BrowseCreateDAOPostgres.getDistinctID(BrowseCreateDAOPostgres.java:566) > at org.dspace.browse.IndexBrowse.indexItem(IndexBrowse.java:485) > at org.dspace.browse.IndexBrowse.createIndex(IndexBrowse.java:1148) > at org.dspace.browse.IndexBrowse.initBrowse(IndexBrowse.java:1098) > at org.dspace.browse.IndexBrowse.main(IndexBrowse.java:749) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:212) >>> >>> >>> Is there anything wrong with the above process? The next step we are going >>> to attempt is checking the actual SQL dump for the bad characters but it is >>> massive (1.5 Gb) >>> >>> >>> thanks, >>> Steve >>> >>> >>> >>> >>> >>> >>> >>> >>> On 26/10/2010, at 10:10 AM, Steve Swinsburg wrote: >>> Hi Kim, We took a *copy* of the database and refreshed it back to dev, so it is identical to the production one, then upgraded it in dev. So I'll need to find out if the original database from production was created that way. Perhaps we should take a dump of the production database instead, and then import that into a fresh UTF8 created database? I think the issue is that the original database was never created as UTF8. Thanks, Steve On 26/10/2010, at 9:17 AM, Kim Shepherd wrote: > Hi Steve, > > Just to check, has your new/dev database been created with UTF8 character
Re: [Dspace-tech] Error creating indexes 'invalid multibyte character for locale'
Hi Steve, See: - https://wiki.duraspace.org/display/DSPACE/TechnicalFaq#TechnicalFaq-SettingloggingleveluptoDEBUG If you set your logging to DEBUG, they will be *very* verbose, but should show you the item that is failing, and the SQL queries in particular that are causing the problem. Thanks, Stuart Lewis IT Innovations Analyst and Developer Te Tumu Herenga The University of Auckland Library Auckland Mail Centre, Private Bag 92019, Auckland 1142, New Zealand Ph: +64 (0)9 373 7599 x81928 On 1/11/2010, at 4:59 PM, Steve Swinsburg wrote: > Hi, > > We've created a new dump with -E UNICODE added to the dump command, imported > it into a clean UTF-8 database, upgraded via the 15-16 sql update script > which completes successfully, but as soon as we run the index-init script > again, the same issue occurs. It would be helpful if the exception told us > which item was causing the issue. > > We ran the checks and everything is UTF-8 so I have no idea why it's still > failing. > > We are now looking at different way of migrating our data: export and then > import the collections. This is going to take quite some time which is not > ideal, but seems to be the only way we can get our data migrated. > > Thanks for you help, > Steve > > > On 29/10/2010, at 2:05 PM, Kim Shepherd wrote: > >> Hi Steve, >> >> As well as Keith's suggestion of inspecting your locale, I'd try a redump of >> the database with -E UNICODE passed to 'pg_dump', otherwise the dumped >> tables will still be in their original encoding. >> >> http://www.postgresql.org/docs/8.4/static/app-pgdump.html >> >> Cheers, >> >> Kim >> >> On 28 October 2010 10:46, Steve Swinsburg wrote: >> Hi all, >> >> We've completed another restore but are still having the same issue. This >> time we created a new database with the UNICODE encoding as per the >> installation manual: >> createdb -U dspace -E UNICODE dspace >> We then took an export of the current production database: >> pg_dump dspace > dspace-prod.dump >> and imported it into the new database: >> psql -d dspace -f dspace-prod.dump >> >> We then upgraded the database to 1.6: >> >> psql -U dspace -f >> /dspace-1.6-2-src-release/dspace/etc/postgres/database_schema_15_16.sql >> dspace >> >> >> And ran the index-init script: >> [dspace]/bin/dspace index-init >> But still the same issue as before. >> dspace index-init -v Exception: org.postgresql.util.PSQLException: ERROR: invalid multibyte character for locale org.dspace.browse.BrowseException: org.postgresql.util.PSQLException: ERROR: invalid multibyte character for locale at org.dspace.browse.BrowseCreateDAOPostgres.insertDistinctRecord(BrowseCreateDAOPostgres.java:691) at org.dspace.browse.BrowseCreateDAOPostgres.getDistinctID(BrowseCreateDAOPostgres.java:566) at org.dspace.browse.IndexBrowse.indexItem(IndexBrowse.java:485) at org.dspace.browse.IndexBrowse.createIndex(IndexBrowse.java:1148) at org.dspace.browse.IndexBrowse.initBrowse(IndexBrowse.java:1098) at org.dspace.browse.IndexBrowse.main(IndexBrowse.java:749) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:212) >> >> >> Is there anything wrong with the above process? The next step we are going >> to attempt is checking the actual SQL dump for the bad characters but it is >> massive (1.5 Gb) >> >> >> thanks, >> Steve >> >> >> >> >> >> >> >> >> On 26/10/2010, at 10:10 AM, Steve Swinsburg wrote: >> >>> Hi Kim, >>> >>> We took a *copy* of the database and refreshed it back to dev, so it is >>> identical to the production one, then upgraded it in dev. So I'll need to >>> find out if the original database from production was created that way. >>> >>> Perhaps we should take a dump of the production database instead, and then >>> import that into a fresh UTF8 created database? I think the issue is that >>> the original database was never created as UTF8. >>> >>> Thanks, >>> Steve >>> >>> >>> >>> >>> On 26/10/2010, at 9:17 AM, Kim Shepherd wrote: >>> Hi Steve, Just to check, has your new/dev database been created with UTF8 character encoding? (pass -E UNICODE or -E UTF8 to createdb) Cheers, Kim On 25 October 2010 14:57, Steve Swinsburg wrote: Hi all, We are having a problem upgrading an instance of dspace from 1.5.x-1.6.2. We built the new 1.6.2 code and took a snapshot of our old production database and migrated in back to our dev environment (along with the asse
Re: [Dspace-tech] Error creating indexes 'invalid multibyte character for locale'
Hi, We've created a new dump with -E UNICODE added to the dump command, imported it into a clean UTF-8 database, upgraded via the 15-16 sql update script which completes successfully, but as soon as we run the index-init script again, the same issue occurs. It would be helpful if the exception told us which item was causing the issue. We ran the checks and everything is UTF-8 so I have no idea why it's still failing. We are now looking at different way of migrating our data: export and then import the collections. This is going to take quite some time which is not ideal, but seems to be the only way we can get our data migrated. Thanks for you help, Steve On 29/10/2010, at 2:05 PM, Kim Shepherd wrote: > Hi Steve, > > As well as Keith's suggestion of inspecting your locale, I'd try a redump of > the database with -E UNICODE passed to 'pg_dump', otherwise the dumped tables > will still be in their original encoding. > > http://www.postgresql.org/docs/8.4/static/app-pgdump.html > > Cheers, > > Kim > > On 28 October 2010 10:46, Steve Swinsburg wrote: > Hi all, > > We've completed another restore but are still having the same issue. This > time we created a new database with the UNICODE encoding as per the > installation manual: > createdb -U dspace -E UNICODE dspace > We then took an export of the current production database: > pg_dump dspace > dspace-prod.dump > and imported it into the new database: > psql -d dspace -f dspace-prod.dump > > We then upgraded the database to 1.6: > > psql -U dspace -f > /dspace-1.6-2-src-release/dspace/etc/postgres/database_schema_15_16.sql dspace > > > And ran the index-init script: > [dspace]/bin/dspace index-init > But still the same issue as before. > >>> dspace index-init -v >>> Exception: org.postgresql.util.PSQLException: ERROR: invalid multibyte >>> character for locale >>> org.dspace.browse.BrowseException: org.postgresql.util.PSQLException: >>> ERROR: invalid multibyte character for locale >>>at >>> org.dspace.browse.BrowseCreateDAOPostgres.insertDistinctRecord(BrowseCreateDAOPostgres.java:691) >>>at >>> org.dspace.browse.BrowseCreateDAOPostgres.getDistinctID(BrowseCreateDAOPostgres.java:566) >>>at org.dspace.browse.IndexBrowse.indexItem(IndexBrowse.java:485) >>>at org.dspace.browse.IndexBrowse.createIndex(IndexBrowse.java:1148) >>>at org.dspace.browse.IndexBrowse.initBrowse(IndexBrowse.java:1098) >>>at org.dspace.browse.IndexBrowse.main(IndexBrowse.java:749) >>>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>>at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>at java.lang.reflect.Method.invoke(Method.java:597) >>>at >>> org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:212) > > > > Is there anything wrong with the above process? The next step we are going to > attempt is checking the actual SQL dump for the bad characters but it is > massive (1.5 Gb) > > > thanks, > Steve > > > > > > > > > On 26/10/2010, at 10:10 AM, Steve Swinsburg wrote: > >> Hi Kim, >> >> We took a *copy* of the database and refreshed it back to dev, so it is >> identical to the production one, then upgraded it in dev. So I'll need to >> find out if the original database from production was created that way. >> >> Perhaps we should take a dump of the production database instead, and then >> import that into a fresh UTF8 created database? I think the issue is that >> the original database was never created as UTF8. >> >> Thanks, >> Steve >> >> >> >> >> On 26/10/2010, at 9:17 AM, Kim Shepherd wrote: >> >>> Hi Steve, >>> >>> Just to check, has your new/dev database been created with UTF8 character >>> encoding? (pass -E UNICODE or -E UTF8 to createdb) >>> >>> Cheers, >>> >>> Kim >>> >>> On 25 October 2010 14:57, Steve Swinsburg wrote: >>> Hi all, >>> >>> We are having a problem upgrading an instance of dspace from 1.5.x-1.6.2. >>> We built the new 1.6.2 code and took a snapshot of our old production >>> database and migrated in back to our dev environment (along with the >>> assets). We then ran the db upgrade script. >>> >>> Now, when running the index-init script we get this: >>> >>> ./dspace index-init -v >>> Exception: org.postgresql.util.PSQLException: ERROR: invalid multibyte >>> character for locale >>> org.dspace.browse.BrowseException: org.postgresql.util.PSQLException: >>> ERROR: invalid multibyte character for locale >>>at >>> org.dspace.browse.BrowseCreateDAOPostgres.insertDistinctRecord(BrowseCreateDAOPostgres.java:691) >>>at >>> org.dspace.browse.BrowseCreateDAOPostgres.getDistinctID(BrowseCreateDAOPostgres.java:566) >>>at org.dspace.browse.IndexBrowse.indexItem(IndexBrowse.java:485) >>>at org.dspace.browse.IndexBrowse.createIn
Re: [Dspace-tech] Error creating indexes 'invalid multibyte character for locale'
Hi Steve, As well as Keith's suggestion of inspecting your locale, I'd try a redump of the database with -E UNICODE passed to 'pg_dump', otherwise the dumped tables will still be in their original encoding. http://www.postgresql.org/docs/8.4/static/app-pgdump.html Cheers, Kim On 28 October 2010 10:46, Steve Swinsburg wrote: > Hi all, > > We've completed another restore but are still having the same issue. This > time we created a new database with the UNICODE encoding as per the > installation manual: > > createdb -U dspace -E UNICODE dspace > > We then took an export of the current production database: > > pg_dump dspace > dspace-prod.dump > > and imported it into the new database: > > psql -d dspace -f dspace-prod.dump > > > We then upgraded the database to 1.6: > > psql -U dspace -f > /dspace-1.6-2-src-release/dspace/etc/postgres/database_schema_15_16.sql > dspace > > > And ran the index-init script: > > [dspace]/bin/dspace index-init > > But still the same issue as before. > > dspace index-init -v >> Exception: org.postgresql.util.PSQLException: ERROR: invalid multibyte >> character for locale >> org.dspace.browse.BrowseException: org.postgresql.util.PSQLException: >> ERROR: invalid multibyte character for locale >>at >> org.dspace.browse.BrowseCreateDAOPostgres.insertDistinctRecord(BrowseCreateDAOPostgres.java:691) >>at >> org.dspace.browse.BrowseCreateDAOPostgres.getDistinctID(BrowseCreateDAOPostgres.java:566) >>at org.dspace.browse.IndexBrowse.indexItem(IndexBrowse.java:485) >>at org.dspace.browse.IndexBrowse.createIndex(IndexBrowse.java:1148) >>at org.dspace.browse.IndexBrowse.initBrowse(IndexBrowse.java:1098) >>at org.dspace.browse.IndexBrowse.main(IndexBrowse.java:749) >>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>at java.lang.reflect.Method.invoke(Method.java:597) >>at >> org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:212) > > > > Is there anything wrong with the above process? The next step we are going > to attempt is checking the actual SQL dump for the bad characters but it is > massive (1.5 Gb) > > > thanks, > Steve > > > > > > > > > On 26/10/2010, at 10:10 AM, Steve Swinsburg wrote: > > Hi Kim, > > We took a *copy* of the database and refreshed it back to dev, so it is > identical to the production one, then upgraded it in dev. So I'll need to > find out if the original database from production was created that way. > > Perhaps we should take a dump of the production database instead, and then > import that into a fresh UTF8 created database? I think the issue is that > the original database was never created as UTF8. > > Thanks, > Steve > > > > > On 26/10/2010, at 9:17 AM, Kim Shepherd wrote: > > Hi Steve, > > Just to check, has your new/dev database been created with UTF8 character > encoding? (pass -E UNICODE or -E UTF8 to createdb) > > Cheers, > > Kim > > On 25 October 2010 14:57, Steve Swinsburg wrote: > >> Hi all, >> >> We are having a problem upgrading an instance of dspace from 1.5.x-1.6.2. >> We built the new 1.6.2 code and took a snapshot of our old production >> database and migrated in back to our dev environment (along with the >> assets). We then ran the db upgrade script. >> >> Now, when running the index-init script we get this: >> >> ./dspace index-init -v >> Exception: org.postgresql.util.PSQLException: ERROR: invalid multibyte >> character for locale >> org.dspace.browse.BrowseException: org.postgresql.util.PSQLException: >> ERROR: invalid multibyte character for locale >>at >> org.dspace.browse.BrowseCreateDAOPostgres.insertDistinctRecord(BrowseCreateDAOPostgres.java:691) >>at >> org.dspace.browse.BrowseCreateDAOPostgres.getDistinctID(BrowseCreateDAOPostgres.java:566) >>at org.dspace.browse.IndexBrowse.indexItem(IndexBrowse.java:485) >>at org.dspace.browse.IndexBrowse.createIndex(IndexBrowse.java:1148) >>at org.dspace.browse.IndexBrowse.initBrowse(IndexBrowse.java:1098) >>at org.dspace.browse.IndexBrowse.main(IndexBrowse.java:749) >>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>at java.lang.reflect.Method.invoke(Method.java:597) >>at >> org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:212) >> >> We are on Postgres: >> PostgreSQL 8.1.21 on x86_64-redhat-linux-gnu, compiled by GCC gcc (GCC) >> 4.1.2 20080704 (Red Hat 4.1.2-48) >> >> Any tips? >> >> thanks, >> Steve >> >> -- >> Nokia and AT&T pr
Re: [Dspace-tech] Error creating indexes 'invalid multibyte character for locale'
Steve - I used Google and found this: http://archives.postgresql.org/pgsql-bugs/2005-11/msg00269.php When I run the 'locale' command here, I get this: LC_CTYPE="en_US.UTF-8" The same thing when I check from psql: gkrtrain=> show lc_ctype gkrtrain-> ; lc_ctype - en_US.UTF-8 (1 row) I'm not sure if this is related to your problem, but it might be worth verifying that you get similar answers. On Oct 27, 2010, at 5:46 PM, Steve Swinsburg wrote: > Hi all, > > We've completed another restore but are still having the same issue. This > time we created a new database with the UNICODE encoding as per the > installation manual: > createdb -U dspace -E UNICODE dspace > We then took an export of the current production database: > pg_dump dspace > dspace-prod.dump > and imported it into the new database: > psql -d dspace -f dspace-prod.dump > > We then upgraded the database to 1.6: > > psql -U dspace -f > /dspace-1.6-2-src-release/dspace/etc/postgres/database_schema_15_16.sql dspace > > > And ran the index-init script: > [dspace]/bin/dspace index-init > But still the same issue as before. > >>> dspace index-init -v >>> Exception: org.postgresql.util.PSQLException: ERROR: invalid multibyte >>> character for locale >>> org.dspace.browse.BrowseException: org.postgresql.util.PSQLException: >>> ERROR: invalid multibyte character for locale >>>at >>> org.dspace.browse.BrowseCreateDAOPostgres.insertDistinctRecord(BrowseCreateDAOPostgres.java:691) >>>at >>> org.dspace.browse.BrowseCreateDAOPostgres.getDistinctID(BrowseCreateDAOPostgres.java:566) >>>at org.dspace.browse.IndexBrowse.indexItem(IndexBrowse.java:485) >>>at org.dspace.browse.IndexBrowse.createIndex(IndexBrowse.java:1148) >>>at org.dspace.browse.IndexBrowse.initBrowse(IndexBrowse.java:1098) >>>at org.dspace.browse.IndexBrowse.main(IndexBrowse.java:749) >>>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>>at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>at java.lang.reflect.Method.invoke(Method.java:597) >>>at >>> org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:212) > > > > Is there anything wrong with the above process? The next step we are going to > attempt is checking the actual SQL dump for the bad characters but it is > massive (1.5 Gb) > > > thanks, > Steve > > > > > > > > > On 26/10/2010, at 10:10 AM, Steve Swinsburg wrote: > >> Hi Kim, >> >> We took a *copy* of the database and refreshed it back to dev, so it is >> identical to the production one, then upgraded it in dev. So I'll need to >> find out if the original database from production was created that way. >> >> Perhaps we should take a dump of the production database instead, and then >> import that into a fresh UTF8 created database? I think the issue is that >> the original database was never created as UTF8. >> >> Thanks, >> Steve >> >> >> >> >> On 26/10/2010, at 9:17 AM, Kim Shepherd wrote: >> >>> Hi Steve, >>> >>> Just to check, has your new/dev database been created with UTF8 character >>> encoding? (pass -E UNICODE or -E UTF8 to createdb) >>> >>> Cheers, >>> >>> Kim >>> >>> On 25 October 2010 14:57, Steve Swinsburg wrote: >>> Hi all, >>> >>> We are having a problem upgrading an instance of dspace from 1.5.x-1.6.2. >>> We built the new 1.6.2 code and took a snapshot of our old production >>> database and migrated in back to our dev environment (along with the >>> assets). We then ran the db upgrade script. >>> >>> Now, when running the index-init script we get this: >>> >>> ./dspace index-init -v >>> Exception: org.postgresql.util.PSQLException: ERROR: invalid multibyte >>> character for locale >>> org.dspace.browse.BrowseException: org.postgresql.util.PSQLException: >>> ERROR: invalid multibyte character for locale >>>at >>> org.dspace.browse.BrowseCreateDAOPostgres.insertDistinctRecord(BrowseCreateDAOPostgres.java:691) >>>at >>> org.dspace.browse.BrowseCreateDAOPostgres.getDistinctID(BrowseCreateDAOPostgres.java:566) >>>at org.dspace.browse.IndexBrowse.indexItem(IndexBrowse.java:485) >>>at org.dspace.browse.IndexBrowse.createIndex(IndexBrowse.java:1148) >>>at org.dspace.browse.IndexBrowse.initBrowse(IndexBrowse.java:1098) >>>at org.dspace.browse.IndexBrowse.main(IndexBrowse.java:749) >>>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>>at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>at java.lang.reflect.Method.invoke(Method.java:597) >>>at >>> org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:212) >>> >>>
Re: [Dspace-tech] Error creating indexes 'invalid multibyte character for locale'
Hi all, We've completed another restore but are still having the same issue. This time we created a new database with the UNICODE encoding as per the installation manual: createdb -U dspace -E UNICODE dspace We then took an export of the current production database: pg_dump dspace > dspace-prod.dump and imported it into the new database: psql -d dspace -f dspace-prod.dump We then upgraded the database to 1.6: psql -U dspace -f /dspace-1.6-2-src-release/dspace/etc/postgres/database_schema_15_16.sql dspace And ran the index-init script: [dspace]/bin/dspace index-init But still the same issue as before. >> dspace index-init -v >> Exception: org.postgresql.util.PSQLException: ERROR: invalid multibyte >> character for locale >> org.dspace.browse.BrowseException: org.postgresql.util.PSQLException: ERROR: >> invalid multibyte character for locale >>at >> org.dspace.browse.BrowseCreateDAOPostgres.insertDistinctRecord(BrowseCreateDAOPostgres.java:691) >>at >> org.dspace.browse.BrowseCreateDAOPostgres.getDistinctID(BrowseCreateDAOPostgres.java:566) >>at org.dspace.browse.IndexBrowse.indexItem(IndexBrowse.java:485) >>at org.dspace.browse.IndexBrowse.createIndex(IndexBrowse.java:1148) >>at org.dspace.browse.IndexBrowse.initBrowse(IndexBrowse.java:1098) >>at org.dspace.browse.IndexBrowse.main(IndexBrowse.java:749) >>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>at java.lang.reflect.Method.invoke(Method.java:597) >>at >> org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:212) Is there anything wrong with the above process? The next step we are going to attempt is checking the actual SQL dump for the bad characters but it is massive (1.5 Gb) thanks, Steve On 26/10/2010, at 10:10 AM, Steve Swinsburg wrote: > Hi Kim, > > We took a *copy* of the database and refreshed it back to dev, so it is > identical to the production one, then upgraded it in dev. So I'll need to > find out if the original database from production was created that way. > > Perhaps we should take a dump of the production database instead, and then > import that into a fresh UTF8 created database? I think the issue is that the > original database was never created as UTF8. > > Thanks, > Steve > > > > > On 26/10/2010, at 9:17 AM, Kim Shepherd wrote: > >> Hi Steve, >> >> Just to check, has your new/dev database been created with UTF8 character >> encoding? (pass -E UNICODE or -E UTF8 to createdb) >> >> Cheers, >> >> Kim >> >> On 25 October 2010 14:57, Steve Swinsburg wrote: >> Hi all, >> >> We are having a problem upgrading an instance of dspace from 1.5.x-1.6.2. We >> built the new 1.6.2 code and took a snapshot of our old production database >> and migrated in back to our dev environment (along with the assets). We then >> ran the db upgrade script. >> >> Now, when running the index-init script we get this: >> >> ./dspace index-init -v >> Exception: org.postgresql.util.PSQLException: ERROR: invalid multibyte >> character for locale >> org.dspace.browse.BrowseException: org.postgresql.util.PSQLException: ERROR: >> invalid multibyte character for locale >>at >> org.dspace.browse.BrowseCreateDAOPostgres.insertDistinctRecord(BrowseCreateDAOPostgres.java:691) >>at >> org.dspace.browse.BrowseCreateDAOPostgres.getDistinctID(BrowseCreateDAOPostgres.java:566) >>at org.dspace.browse.IndexBrowse.indexItem(IndexBrowse.java:485) >>at org.dspace.browse.IndexBrowse.createIndex(IndexBrowse.java:1148) >>at org.dspace.browse.IndexBrowse.initBrowse(IndexBrowse.java:1098) >>at org.dspace.browse.IndexBrowse.main(IndexBrowse.java:749) >>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>at java.lang.reflect.Method.invoke(Method.java:597) >>at >> org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:212) >> >> We are on Postgres: >> PostgreSQL 8.1.21 on x86_64-redhat-linux-gnu, compiled by GCC gcc (GCC) >> 4.1.2 20080704 (Red Hat 4.1.2-48) >> >> Any tips? >> >> thanks, >> Steve >> -- >> Nokia and AT&T present the 2010 Calling All Innovators-North America contest >> Create new apps & games for the Nokia N8 for consumers in U.S. and Canada >> $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing >> Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store >> http://p.sf.net/sfu/nokia-dev2dev >> __
Re: [Dspace-tech] Error creating indexes 'invalid multibyte character for locale'
Hi Kim, We took a *copy* of the database and refreshed it back to dev, so it is identical to the production one, then upgraded it in dev. So I'll need to find out if the original database from production was created that way. Perhaps we should take a dump of the production database instead, and then import that into a fresh UTF8 created database? I think the issue is that the original database was never created as UTF8. Thanks, Steve On 26/10/2010, at 9:17 AM, Kim Shepherd wrote: > Hi Steve, > > Just to check, has your new/dev database been created with UTF8 character > encoding? (pass -E UNICODE or -E UTF8 to createdb) > > Cheers, > > Kim > > On 25 October 2010 14:57, Steve Swinsburg wrote: > Hi all, > > We are having a problem upgrading an instance of dspace from 1.5.x-1.6.2. We > built the new 1.6.2 code and took a snapshot of our old production database > and migrated in back to our dev environment (along with the assets). We then > ran the db upgrade script. > > Now, when running the index-init script we get this: > > ./dspace index-init -v > Exception: org.postgresql.util.PSQLException: ERROR: invalid multibyte > character for locale > org.dspace.browse.BrowseException: org.postgresql.util.PSQLException: ERROR: > invalid multibyte character for locale >at > org.dspace.browse.BrowseCreateDAOPostgres.insertDistinctRecord(BrowseCreateDAOPostgres.java:691) >at > org.dspace.browse.BrowseCreateDAOPostgres.getDistinctID(BrowseCreateDAOPostgres.java:566) >at org.dspace.browse.IndexBrowse.indexItem(IndexBrowse.java:485) >at org.dspace.browse.IndexBrowse.createIndex(IndexBrowse.java:1148) >at org.dspace.browse.IndexBrowse.initBrowse(IndexBrowse.java:1098) >at org.dspace.browse.IndexBrowse.main(IndexBrowse.java:749) >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >at java.lang.reflect.Method.invoke(Method.java:597) >at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:212) > > We are on Postgres: > PostgreSQL 8.1.21 on x86_64-redhat-linux-gnu, compiled by GCC gcc (GCC) 4.1.2 > 20080704 (Red Hat 4.1.2-48) > > Any tips? > > thanks, > Steve > -- > Nokia and AT&T present the 2010 Calling All Innovators-North America contest > Create new apps & games for the Nokia N8 for consumers in U.S. and Canada > $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing > Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store > http://p.sf.net/sfu/nokia-dev2dev > ___ > DSpace-tech mailing list > DSpace-tech@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/dspace-tech > -- Nokia and AT&T present the 2010 Calling All Innovators-North America contest Create new apps & games for the Nokia N8 for consumers in U.S. and Canada $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store http://p.sf.net/sfu/nokia-dev2dev___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Error creating indexes 'invalid multibyte character for locale'
Hi Steve, Just to check, has your new/dev database been created with UTF8 character encoding? (pass -E UNICODE or -E UTF8 to createdb) Cheers, Kim On 25 October 2010 14:57, Steve Swinsburg wrote: > Hi all, > > We are having a problem upgrading an instance of dspace from 1.5.x-1.6.2. > We built the new 1.6.2 code and took a snapshot of our old production > database and migrated in back to our dev environment (along with the > assets). We then ran the db upgrade script. > > Now, when running the index-init script we get this: > > ./dspace index-init -v > Exception: org.postgresql.util.PSQLException: ERROR: invalid multibyte > character for locale > org.dspace.browse.BrowseException: org.postgresql.util.PSQLException: > ERROR: invalid multibyte character for locale >at > org.dspace.browse.BrowseCreateDAOPostgres.insertDistinctRecord(BrowseCreateDAOPostgres.java:691) >at > org.dspace.browse.BrowseCreateDAOPostgres.getDistinctID(BrowseCreateDAOPostgres.java:566) >at org.dspace.browse.IndexBrowse.indexItem(IndexBrowse.java:485) >at org.dspace.browse.IndexBrowse.createIndex(IndexBrowse.java:1148) >at org.dspace.browse.IndexBrowse.initBrowse(IndexBrowse.java:1098) >at org.dspace.browse.IndexBrowse.main(IndexBrowse.java:749) >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >at java.lang.reflect.Method.invoke(Method.java:597) >at > org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:212) > > We are on Postgres: > PostgreSQL 8.1.21 on x86_64-redhat-linux-gnu, compiled by GCC gcc (GCC) > 4.1.2 20080704 (Red Hat 4.1.2-48) > > Any tips? > > thanks, > Steve > > -- > Nokia and AT&T present the 2010 Calling All Innovators-North America > contest > Create new apps & games for the Nokia N8 for consumers in U.S. and Canada > $10 million total in prizes - $4M cash, 500 devices, nearly $6M in > marketing > Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store > http://p.sf.net/sfu/nokia-dev2dev > ___ > DSpace-tech mailing list > DSpace-tech@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/dspace-tech > -- Nokia and AT&T present the 2010 Calling All Innovators-North America contest Create new apps & games for the Nokia N8 for consumers in U.S. and Canada $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store http://p.sf.net/sfu/nokia-dev2dev___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
[Dspace-tech] Error creating indexes 'invalid multibyte character for locale'
Hi all, We are having a problem upgrading an instance of dspace from 1.5.x-1.6.2. We built the new 1.6.2 code and took a snapshot of our old production database and migrated in back to our dev environment (along with the assets). We then ran the db upgrade script. Now, when running the index-init script we get this: ./dspace index-init -v Exception: org.postgresql.util.PSQLException: ERROR: invalid multibyte character for locale org.dspace.browse.BrowseException: org.postgresql.util.PSQLException: ERROR: invalid multibyte character for locale at org.dspace.browse.BrowseCreateDAOPostgres.insertDistinctRecord(BrowseCreateDAOPostgres.java:691) at org.dspace.browse.BrowseCreateDAOPostgres.getDistinctID(BrowseCreateDAOPostgres.java:566) at org.dspace.browse.IndexBrowse.indexItem(IndexBrowse.java:485) at org.dspace.browse.IndexBrowse.createIndex(IndexBrowse.java:1148) at org.dspace.browse.IndexBrowse.initBrowse(IndexBrowse.java:1098) at org.dspace.browse.IndexBrowse.main(IndexBrowse.java:749) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:212) We are on Postgres: PostgreSQL 8.1.21 on x86_64-redhat-linux-gnu, compiled by GCC gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-48) Any tips? thanks, Steve -- Nokia and AT&T present the 2010 Calling All Innovators-North America contest Create new apps & games for the Nokia N8 for consumers in U.S. and Canada $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store http://p.sf.net/sfu/nokia-dev2dev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
