Re: [firebird-support] FB 3 issues with String from FB 2.54

[fabia...@itbizolutions.com.au [firebird-support]]

Mark

Are you saying that the filter I placed on FB 2.5 with charset ASCII was 
correct from the theory, but there was a bug that made the filter be ignored by 
the FB engine? and the ugly characters got inside the DB because of the bug?

And for me to be able to move onto FB 3 I had to remove the ASCII filter 
(charset) because otherwise the restore onto 3.0 would fail, so now I have lots 
of records with unwanted characters and cannot put the filter back onto 3.0 
because if I do so I would end up having a table that does not allow such 
characters however the characters are already there, so for intance a user 
trying to change a dot, or a comma on one of those records would get a failed 
message as the DB cannot write back into the engine the same text that is 
alraedy there becuase of the offending character. Is that correct?  How do I 
solve the issue? At the moment I set charset to NONE and it allows garbage in 
and out with no issues, but I don't want the bad characters there..

Cheers,
Fabian

  - Original Message - 
  From: Mark Rotteveel m...@lawinegevaar.nl [firebird-support] 
  To: firebird-support@yahoogroups.com 
  Sent: Thursday, May 26, 2016 4:08 AM
  Subject: Re: [firebird-support] FB 3 issues with String from FB 2.54


  On 23-5-2016 03:45, fabia...@itbizolutions.com.au [firebird-support] wrote:
  > I have been trying to migrate from FB2.54 into FB 3 for a few weeks, and
  > after hitting a string related error for some time i have got to the
  > point where I do understand the issue, but I don't know how to solve it.
  > The issue is pretty simple, the FB 2.54 DB contains a few characters
  > that are not allowed into the FB 3 database, one example of a character
  > causing an error during the restore was "Mcgarrity’s" (note the ’) as it
  > appears to be outside the scope of the FB3 string domain, I have trying
  > creating a new FB3 DB with many different charsets but none works. The
  > other string causing issues is for example "΢ÈíÑźÚ", I have many
  > records with this type of strings because the DB contains raw emails
  > received by the system, stored into Varchars, and apparently some emails
  > contain very weird characters, all were handled by FB2.54 but FB3
  > rejects the records. I have been able to isolate all recrods with issues
  > using IBExpert's table data comparer function, as it created a script
  > with all recrods from all tables from FB2.54 and when running the script
  > against FB3.0 it singles out all the offending records.
  >
  > Can anyone advise what options I have available to force FB3.0 to accept
  > any stuff into string fields?

  In your other e-mail you indicate you solved this by changing the 
  character set from ASCII to NONE. The fact it worked before was a bug, 
  see http://tracker.firebirdsql.org/browse/CORE-3416. ASCII only supports 
  characters 0-127, characters outside that range are 'extended ascii', eg 
  one of the other singly by character sets like WIN1252 or ISO8859_1. The 
  characters shown (΢ÈíÑÅºÚ and ’) are all outside the ASCII range.

  The last (’) is particularly nasty, because it should have been a ' 
  (u+0027 Apostrophe, ascii 39), instead u+2019 Right single quotation 
  mark (character 146 in Windows-1252) was used.

  Given the context of e-mails either NONE or OCTETS is the only real 
  option, as e-mails can have multiple parts with each their own character 
  set, and can also have binary parts (although usually those are encoded 
  with something like base64).

  Mark
  -- 
  Mark Rotteveel


  
  Posted by: Mark Rotteveel <m...@lawinegevaar.nl>
  

  ++

  Visit http://www.firebirdsql.org and click the Documentation item
  on the main (top) menu.  Try FAQ and other links from the left-side menu 
there.

  Also search the knowledgebases at 
http://www.ibphoenix.com/resources/documents/ 

  ++
  

  Yahoo Groups Links

Re: [firebird-support] FB 3 issues with String from FB 2.54

[Mark Rotteveel m...@lawinegevaar.nl [firebird-support]]

On 23-5-2016 03:45, fabia...@itbizolutions.com.au [firebird-support] wrote:
> I have been trying to migrate from FB2.54 into FB 3 for a few weeks, and
> after hitting a string related error for some time i have got to the
> point where I do understand the issue, but I don't know how to solve it.
> The issue is pretty simple, the FB 2.54 DB contains a few characters
> that are not allowed into the FB 3 database, one example of a character
> causing an error during the restore was "Mcgarrity’s" (note the ’) as it
> appears to be outside the scope of the FB3 string domain, I have trying
> creating a new FB3 DB with many different charsets but none works. The
> other string causing issues is for example "΢ÈíÑźÚ", I have many
> records with this type of strings because the DB contains raw emails
> received by the system, stored into Varchars, and apparently some emails
> contain very weird characters, all were handled by FB2.54 but FB3
> rejects the records. I have been able to isolate all recrods with issues
> using IBExpert's table data comparer function, as it created a script
> with all recrods from all tables from FB2.54 and when running the script
> against FB3.0 it singles out all the offending records.
>
> Can anyone advise what options I have available to force FB3.0 to accept
> any stuff into string fields?

In your other e-mail you indicate you solved this by changing the 
character set from ASCII to NONE. The fact it worked before was a bug, 
see http://tracker.firebirdsql.org/browse/CORE-3416. ASCII only supports 
characters 0-127, characters outside that range are 'extended ascii', eg 
one of the other singly by character sets like WIN1252 or ISO8859_1. The 
characters shown (΢ÈíÑÅºÚ and ’) are all outside the ASCII range.

The last (’) is particularly nasty, because it should have been a ' 
(u+0027 Apostrophe, ascii 39), instead u+2019 Right single quotation 
mark (character 146 in Windows-1252) was used.

Given the context of e-mails either NONE or OCTETS is the only real 
option, as e-mails can have multiple parts with each their own character 
set, and can also have binary parts (although usually those are encoded 
with something like base64).

Mark
-- 
Mark Rotteveel

Re: [firebird-support] FB 3 issues with String from FB 2.54

[fabia...@itbizolutions.com.au [firebird-support]]

Hi Kjell 

 We had a FB 2.54 DB with a few varchars defined as charset ASCII, the 
intention was to stop users from entering unreadable characters, so the charset 
was there to attempt to filter garbage, at least that was the idea), anyways we 
converted all domains from charset ASCII to NONE, and then backup and restore 
onto FB 3.0, and it all worked fine. So I guess we were lucky, and the 
migration process was not too painfull. On the flip side, the performance we 
are getting on 3.0 is worst than on 2.54 it seems the SMP is not active as 
only one processor is doing all the job, and CPU affinity is set to zero, so we 
are not sure what's going on.
 

 Cheers,
 Fabian
 

 

Re: [firebird-support] FB 3 issues with String from FB 2.54

[Kjell Rilbe kjell.ri...@marknadsinformation.se [firebird-support]]

fabia...@itbizolutions.com.au [firebird-support] skrev:


I have been trying to migrate from FB2.54 into FB 3 for a few weeks, 
and after hitting a string related error for some time i have got to 
the point where I do understand the issue, but I don't know how to 
solve it. The issue is pretty simple, the FB 2.54 DB contains a few 
characters that are not allowed into the FB 3 database, one example of 
a character causing an error during the restore was "Mcgarrity’s" 
(note the ’) as it appears to be outside the scope of the FB3 string 
domain, I have trying creating a new FB3 DB with many different 
charsets but none works. The other string causing issues is for 
example "΢ÈíÑźÚ", I have many records with this type of strings 
because the DB contains raw emails received by the system, stored into 
Varchars, and apparently some emails contain very weird characters, 
all were handled by FB2.54 but FB3 rejects the records. I have been 
able to isolate all recrods with issues using IBExpert's table data 
comparer function, as it created a script with all recrods from all 
tables from FB2.54 and when running the script against FB3.0 it 
singles out all the offending records.



Can anyone advise what options I have available to force FB3.0 to 
accept any stuff into string fields?




The first thing that comes to mind is that ' needs to be duplicated 
inside string literals, so if your method of upgrading is backing up to 
insert statements, it might be the case that the backup file is 
incorrectly formatted in this reagard. I.e. to insert Mcgarrity's you 
would insert this string literal: 'Mcgarrity''s' (note the duplicated ').


Apart from that I would recommend UTF-8 for "everything", i.e. old table 
charset, connection charset when performing backup, backup file format, 
connection charset for importing the backup file into FB3, charset for 
all char/varchar columns in new database. Not sure all those "places" 
are even applicable.


Are you using gbak for backup from 2.5 and restore into 3.0, or how are 
you actually trying to perform the upgrade?


Regards,
Kjell
--

Marknadsinformation logotyp

Kjell Rilbe
Telefon: 08-761 06 55
Mobil: 0733-44 24 64

Marknadsinformation i Sverige AB
Ulvsundavägen 106C
168 67 Bromma
www.marknadsinformation.se 
08-514 905 90

Företagskontakt.se 
Personkontakt.se 

ODP: [firebird-support] FB 3 issues with String from FB 2.54

[liviuslivius liviusliv...@poczta.onet.pl [firebird-support]]

hi,
will be good to know which charsets did you tried. utf8, none, octets...
but remember to connet to db also with this charset settings
regards,Karol Bieniaszewski

 Oryginalna wiadomość 
Od: "fabia...@itbizolutions.com.au [firebird-support]" 
<firebird-support@yahoogroups.com> 
Data: 23.05.2016  03:45  (GMT+01:00) 
Do: firebird-support@yahoogroups.com 
Temat: [firebird-support] FB 3 issues with String from FB 2.54 


 



  



  
  
  Hi All
I have been trying to migrate from FB2.54 into FB 3 for a few weeks, and after 
hitting a string related error for some time i have got to the point where I do 
understand the issue, but I don't know how to solve it. The issue is pretty 
simple, the FB 2.54 DB contains a few characters that are not allowed into the 
FB 3 database, one example of a character causing an error during the restore 
was "Mcgarrity’s" (note the ’) as it appears to be outside the scope of the FB3 
string domain, I have trying creating a new FB3 DB with many different charsets 
but none works. The other string causing issues is for example "΢ÈíÑźÚ", I 
have many records with this type of strings because the DB contains raw emails 
received by the system, stored into Varchars, and apparently some emails 
contain very weird characters, all were handled by FB2.54 but FB3 rejects the 
records. I have been able to isolate all recrods with issues using IBExpert's 
table data comparer function, as it created a script with all recrods from all 
tables from FB2.54 and when running the script against FB3.0 it singles out all 
the offending records. 
Can anyone advise what options I have available to force FB3.0 to accept any 
stuff into string fields?
Thanks in advanced



 

[firebird-support] FB 3 issues with String from FB 2.54

[fabia...@itbizolutions.com.au [firebird-support]]

Hi All
 

 I have been trying to migrate from FB2.54 into FB 3 for a few weeks, and after 
hitting a string related error for some time i have got to the point where I do 
understand the issue, but I don't know how to solve it. The issue is pretty 
simple, the FB 2.54 DB contains a few characters that are not allowed into the 
FB 3 database, one example of a character causing an error during the restore 
was "Mcgarrity’s" (note the ’) as it appears to be outside the scope of the FB3 
string domain, I have trying creating a new FB3 DB with many different charsets 
but none works. The other string causing issues is for example "΢ÈíÑźÚ", I 
have many records with this type of strings because the DB contains raw emails 
received by the system, stored into Varchars, and apparently some emails 
contain very weird characters, all were handled by FB2.54 but FB3 rejects the 
records. I have been able to isolate all recrods with issues using IBExpert's 
table data comparer function, as it created a script with all recrods from all 
tables from FB2.54 and when running the script against FB3.0 it singles out all 
the offending records. 
 

 Can anyone advise what options I have available to force FB3.0 to accept any 
stuff into string fields?
 

 
 Thanks in advanced