[firebird-support] Charset and Font Character Set for IBX TIBDatabase component

[jonatan.laurit...@yahoo.dk [firebird-support]]

Hi!

IBExpert uses two settings for database charsets:
- Charset (apparently Database related) and
- Font Character Set (apparently Client related)

As I understand then Charset is the same setting as lc_ctype=UNICODE_FSS in 
TIBDatabase.Params parameters. But how can I indicate Font Character Set in my 
Delphi XE10 IBX program that uses TIBDatabase?

I have problem with legacy Firebird 1.5 character set NONE database. When my 
program is run on Baltic (default windows locale for non-unicode program) 
computer then the same character is reported as \u0101 (that is correct) but 
when my program is run no Russian computer then the same character is reported 
as \u0432 (incorrect). What happens here and how to configure TIBDatabase to 
work correctly on computers with any locale for non-unicode program?
 

RE: [firebird-support] Re: detect duplicate blobs, how to?

['Richard Damon' rich...@damon-family.org [firebird-support]]

Yes, it is a matter of scale, but it is also a matter of what you promise. If 
you are going to promise 100% reliable detection of duplicates, just using a 
hash (no matter how good) is a bug, because it ISN’T 100% reliable. It might be 
“Reliable Enough”, but don’t say it is 100%.

 

Please also review your math, as your argument actually proved its flaw. The 
number 2.71 x 10^18 ISN’T a chance of collision, or even a chance of any kind. 
It is the number of UUIDs (of 128 bits) that would need to be generated to have 
a 50% chance of collision. 

 

I would actually argue that the idea of using a very high quality hash, like 
SHA256 is taking the problem in the wrong direction. It is a somewhat expensive 
value to compute, and being 256 bits long (32 bytes) somewhat big to add, and 
index, for every file. I would be much more inclined to use a simple hash (like 
CRC) that gives maybe a 32 bit result, lookup the files that match (somewhat 
likely to have one) and do the bit by bit compare to see if it is a duplicate. 
That is probably faster than your method of computing the SHA256 and (possibly 
even if unlikely, incorrectly) assuming they are unique, the time saved in 
using a simpler hash, and quicker index, likely makes up for the time needed to 
compare the two files.

 

From: firebird-support@yahoogroups.com 
[mailto:firebird-support@yahoogroups.com] 
Sent: Thursday, February 9, 2017 6:40 PM
To: firebird-support@yahoogroups.com
Subject: RE: [firebird-support] Re: detect duplicate blobs, how to?

 

  

Richard, 

> Sean 
> Even SHA256 can’t eliminate all possibility of a duplicate. If you have files 
> of 
> more than 256 bits in them, by the pigeon hole principle, there WILL be 
> duplicates within the universe of all possible files. There HAS to be. The 
> probability is very low (but not zero) if you keep you set of files below 
> 2^128 
> members, but it is NOT 0. 

I agree that there is a possibility, but it is all about understanding scale. 

Consider, 2^128 is the same number space that is used to represent UUID and 
GUID values, and the changes of a collision is 2.71 x 10^18 
(https://en.wikipedia.org/wiki/Universally_unique_identifier ) 

Consider, the number of words ever spoken by human beings = 5 Exabytes (< 2^63) 
(http://highscalability.com/blog/2012/9/11/how-big-is-a-petabyte-exabyte-zettabyte-or-a-yottabyte.html),
 so what are the chances that more than 2^128 files have been created? 

Consider, the probability that a rogue asteroid crashes on Earth *within the 
next second*, obliterating civilization-as-we-know-it is about 10^-15. Which is 
45 **orders of magnitude** more probable than the SHA-256 collision. 
(http://stackoverflow.com/questions/4014090/is-it-safe-to-ignore-the-possibility-of-sha-collisions-in-practice)
 

IMO, the changes of a collision are for all practical purposes 0. 

But if you still think there is a chance, then use SHA512. 


Sean 

___

RE: [firebird-support] Re: detect duplicate blobs, how to?

['Leyne, Sean' s...@broadviewsoftware.com [firebird-support]]

Richard,

> Sean
> Even SHA256 can’t eliminate all possibility of a duplicate. If you have files 
> of
> more than 256 bits in them, by the pigeon hole principle, there WILL be
> duplicates within the universe of all possible files. There HAS to be. The
> probability is very low (but not zero) if you keep you set of files below 
> 2^128
> members, but it is NOT 0.

I agree that there is a possibility, but it is all about understanding scale.

Consider, 2^128 is the same number space that is used to represent UUID and 
GUID values, and the changes of a collision is 2.71 x 10^18 
(https://en.wikipedia.org/wiki/Universally_unique_identifier )

Consider, the number of words ever spoken by human beings = 5 Exabytes (< 2^63) 
(http://highscalability.com/blog/2012/9/11/how-big-is-a-petabyte-exabyte-zettabyte-or-a-yottabyte.html),
 so what are the chances that more than 2^128 files have been created?

Consider, the probability that a rogue asteroid crashes on Earth *within the 
next second*, obliterating civilization-as-we-know-it is about 10^-15.  Which 
is 45 **orders of magnitude** more probable than the SHA-256 collision. 
(http://stackoverflow.com/questions/4014090/is-it-safe-to-ignore-the-possibility-of-sha-collisions-in-practice)

IMO, the changes of a collision are for all practical purposes 0.

But if you still think there is a chance, then use SHA512.


Sean

RE: [firebird-support] Re: detect duplicate blobs, how to?

['Richard Damon' rich...@damon-family.org [firebird-support]]

 

 

From: firebird-support@yahoogroups.com 
[mailto:firebird-support@yahoogroups.com] 
Sent: Thursday, February 9, 2017 11:33 AM
To: firebird-support@yahoogroups.com
Subject: RE: [firebird-support] Re: detect duplicate blobs, how to?

 

  



> You are aware of course that you can't use any hashing function on its own to 
> detect duplicates? - the best you can do is detect *probable* duplicates, 

Actually, if you choose the right hash function you can detect duplicates. 

If you create a UDF based on/using SHA256, the result would be unique (with a 
2^256 certainty) -- there is no known collision of a SHA256 hash 
(https://en.wikipedia.org/wiki/Hash_function_security_summary). 


Sean 

Even SHA256 can’t eliminate all possibility of a duplicate. If you have files 
of more than 256 bits in them, by the pigeon hole principle, there WILL be 
duplicates within the universe of all possible files. There HAS to be. The 
probability is very low (but not zero) if you keep you set of files below 2^128 
members, but it is NOT 0. The key property of a has like SHA256 is that given a 
hash value, you cannot create a file (other than by brute force) that will 
yield that hash value. When using a hash, you need to decide if the chance of a 
false positive on match. With a good large hash, that probability gets very 
small, so maybe you can assume it is perfect. I would likely still compare the 
files to be sure, since you will likely only occur that cost if you do have a 
duplicate.   ,_._,___

RE: [firebird-support] Re: detect duplicate blobs, how to?

['Leyne, Sean' s...@broadviewsoftware.com [firebird-support]]

> You are aware of course that you can't use any hashing function on its own to
> detect duplicates? - the best you can do is detect *probable* duplicates,

Actually, if you choose the right hash function you can detect duplicates.

If you create a UDF based on/using SHA256, the result would be unique  (with a 
2^256 certainty) -- there is no known collision of a SHA256 hash 
(https://en.wikipedia.org/wiki/Hash_function_security_summary).


Sean

RE: [firebird-support] Is this a bug of Firebird?

['Leyne, Sean' s...@broadviewsoftware.com [firebird-support]]

Doug,

> Delphi's latest compiler provides a hint in these cases:
> 
> var
>   x: Int32;
> begin
>   try
>     x := 0;
>     x := 1;
>     writeln(x);
> end.
> 
> [dcc32 Hint] Project1.dpr(14): H2077 Value assigned to 'x' never used
> (Line 14 is the one assigning zero.)

In hindsight my post should have read more like:

- No compiler would treat the condition as an error.  A warning, Yes.  But not 
an error.

- There are plenty of DB tools (IBExpert for one) which perform their own 
"syntax checking" and report warnings (though I would need to test if IBE 
reports a warning for this case).

- AFAIK, the FB engine only reports errors.  So, a new set of functions/methods 
would need to be developed to report and handle warnings.


Sean

[firebird-support] Re: detect duplicate blobs, how to?

[Dmitry Yemanov dim...@users.sourceforge.net [firebird-support]]

09.02.2017 17:39, hamacker wrote:

> Well, the man page says fully supports text BLOBs, I assume that binary not.

For blobs, it works with raw binary data.


Dmitry

Re: [firebird-support] Is this a bug of Firebird?

[hamacker sirhamac...@gmail.com [firebird-support]]

Some times,

I need to create a query where there is a 'where (0=0)' .
it´s not logic for anyone.
But to me, it´s significant because I need to certify that WHERE clausule
always exists when populate my query with dinamic filters controlled by my
application.
In your example, I can espaculate that your intention is not a bug, but
there is a reason for that or your value is innocuous.
innocuous value is not the same as bug, sometimes is not intentional but
happens when generated by automatic selected fields or filters in
application.
The RDBMS does not know the context situation to raise an exception for
ambiguals or innocuous values.



2017-02-08 13:55 GMT-02:00 'Walter R. Ojeda Valiente'
sistemas2000profesio...@gmail.com [firebird-support] <
firebird-support@yahoogroups.com>:

>
>
> CREATE PROCEDURE TEST
> RETURNS(
>   lcName VARCHAR(40))
> AS
> BEGIN
>FOR SELECT
>   ALU_NOMBRE,
>   ALU_APELLD
>FROM
>   ALUMNOS
>INTO
>   :lcName,
>   :lcName
>DO
>   SUSPEND;
>
> END;
>
> In the above stored procedure there are 2 columns in the FOR SELECT but
> just 1 variable name.
>
> It seems a bug for me. Or there are some reasons for allowing to do such
> thing?
>
> Of course, the fault is of the programmer, but it can be detected at
> compilation time.
>
> Firebird version: 2.5.5
>
> Greetings.
>
> Walter.
>
> 
>

Re: [firebird-support] Re: detect duplicate blobs, how to?

[Tim Ward t...@telensa.com [firebird-support]]

You are aware of course that you can't use any hashing function on its 
own to detect duplicates? - the best you can do is detect *probable* 
duplicates, filtering out ones that are *definitely* different, in order 
to *reduce*, not eliminate, the amount of work you end up doing 
comparing byte by byte.


On 09/02/2017 14:39, hamacker sirhamac...@gmail.com [firebird-support] 
wrote:
Well, the man page says fully supports text BLOBs, I assume that 
binary not.
If HASH() behavior will operate ^Z and others control chars in Blob 
Image, then now I need to think how to use this function to kill 
duplications.


Any function like CHECKSUM, CRC32,...?

2017-02-09 10:36 GMT-02:00 Dmitry Yemanov dim...@users.sourceforge.net 
 [firebird-support] 
>:


09.02.2017 14:48, hamacker wrote:
>
> There is a HASH function but only for string.

It works for blobs as well.

Dmitry






--
Tim Ward

Re: [firebird-support] Re: detect duplicate blobs, how to?

[hamacker sirhamac...@gmail.com [firebird-support]]

Well, the man page says fully supports text BLOBs, I assume that binary not.
If HASH() behavior will operate ^Z and others control chars in Blob Image,
then now I need to think how to use this function to kill duplications.

Any function like CHECKSUM, CRC32,...?

2017-02-09 10:36 GMT-02:00 Dmitry Yemanov dim...@users.sourceforge.net
[firebird-support] :

>
>
> 09.02.2017 14:48, hamacker wrote:
> >
> > There is a HASH function but only for string.
>
> It works for blobs as well.
>
> Dmitry
>
> 
>

Re: [firebird-support] Is this a bug of Firebird?

[Tim Ward t...@telensa.com [firebird-support]]

Interesting. (I haven't used Delphi for decades, certainly not this 
century.)


On 09/02/2017 13:17, DougC d...@moosemail.net [firebird-support] wrote:

Delphi's latest compiler provides a hint in these cases:

var
  x: Int32;
begin
  try
x := 0;
x := 1;
writeln(x);
end.

[dcc32 Hint] Project1.dpr(14): H2077 Value assigned to 'x' never used
(Line 14 is the one assigning zero.)

 On Thu, 09 Feb 2017 04:30:06 -0500 *Tim Ward t...@telensa.com 
[firebird-support] * wrote 


It' the equivalent in a conventional programming language of saying:

x = a;
x = b;

where the compiler is expected to know that neither a not the
first assignment have any side effects other than the assignment
(and where the expression b doesn't depend on the value of x)(and
where x isn't volatile, ect ect).

A compiler *could* detect and warn about such things (ie it's not
forbidden by the laws of mathematics) but I don't think I know of
any that do. And as there are good reasons for deliberately
wanting to do the above it could only be a warning, not an error.






--
Tim Ward

Re: [firebird-support] Is this a bug of Firebird?

[DougC d...@moosemail.net [firebird-support]]

Delphi's latest compiler provides a hint in these cases:



var

  x: Int32;

begin

  try

x := 0;

x := 1;

writeln(x);

end.



[dcc32 Hint] Project1.dpr(14): H2077 Value assigned to 'x' never used

(Line 14 is the one assigning zero.)



 On Thu, 09 Feb 2017 04:30:06 -0500 Tim Ward t...@telensa.com 
[firebird-support]  wrote 




  It' the equivalent in a conventional programming language of saying:

 

 x = a;

 x = b;

 

 where the compiler is expected to know that neither a not the first assignment 
have any side effects other than the assignment (and where the expression b 
doesn't depend on the value of x)(and where x isn't volatile, ect ect).

 

 A compiler *could* detect and warn about such things (ie it's not forbidden by 
the laws of mathematics) but I don't think I know of any that do. And as there 
are good reasons for deliberately wanting to do the above it could only be a 
warning, not an error. 

Re: [firebird-support] Is this a bug of Firebird?

[Marsupilami79 marsupilam...@gmx.de [firebird-support]]

Hmm - Delphi and Free Pascal do exactly that - they emit a warning, that
the first assigned value never gets used.

Am 09.02.2017 um 10:30 schrieb Tim Ward t...@telensa.com [firebird-support]:
>  
>
> It' the equivalent in a conventional programming language of saying:
>
> x = a;
> x = b;
>
> where the compiler is expected to know that neither a not the first
> assignment have any side effects other than the assignment (and where
> the expression b doesn't depend on the value of x)(and where x isn't
> volatile, ect ect).
>
> A compiler *could* detect and warn about such things (ie it's not
> forbidden by the laws of mathematics) but I don't think I know of any
> that do. And as there are good reasons for deliberately wanting to do
> the above it could only be a warning, not an error.
>
> On 08/02/2017 23:36, 'Walter R. Ojeda Valiente'
> sistemas2000profesio...@gmail.com [firebird-support] wrote:
>>  
>> The error is of the programmer, I agree with you, but to repeat the
>> name of a variable without the Firebird showing any message of error
>> is, at least for me, a bug.
>>
>> To have 2 or more variables with the same name after the INTO clause
>> is useless. The compiler can be smart enough to detect such thing.
>>
>> Or not?
>>
>> Greetings.
>>
>> Walter.
>>
>> On Wed, Feb 8, 2017 at 2:36 PM, 'Leyne, Sean'
>> s...@broadviewsoftware.com [firebird-support]
>> > > wrote:
>>
>>  
>>
>>
>>
>> > Yes, but I can not know the value of the column X.ALU_NOMBRE
>> >
>> > And the idea, of course, is know that value, that's why it
>> appears in the FOR
>> > SELECT. If not, I can do nothing with X.ALU_NOMBRE
>>
>> You are asking for the system to evaluate the *intent* of logic.
>>
>> That is completely outside the purview of any application
>> environment that I know.
>>
>> The only thing that a system can check/enforce is the correctness
>> of the code, not to check whether the developer has 2 brain cells.
>>
>>
>> Sean
>>
>>
>
>
> -- 
> Tim Ward
> 

[firebird-support] Re: detect duplicate blobs, how to?

[Dmitry Yemanov dim...@users.sourceforge.net [firebird-support]]

09.02.2017 14:48, hamacker wrote:
>
> There is a HASH function but only for string.

It works for blobs as well.


Dmitry

[firebird-support] detect duplicate blobs, how to?

[hamacker sirhamac...@gmail.com [firebird-support]]

Hi All,

I am using FB3 and I have a table where each record has a picture and some
picture are duplicated.
I think to separate pictures into a new table.
But I need a method (server side is better) to not duplicate pictures again.

First, I need to know if exists function as CHECKSUM (magical number or
else) to determinate a number to each picture. If exists, maybe will be
easy to deploy.

Looking for new functions of FB3 in release notes, I dont find anything.
There is a HASH function but only for string.

Any suggestions?

Re: [firebird-support] Is this a bug of Firebird?

[Tim Ward t...@telensa.com [firebird-support]]

It' the equivalent in a conventional programming language of saying:

x = a;
x = b;

where the compiler is expected to know that neither a not the first 
assignment have any side effects other than the assignment (and where 
the expression b doesn't depend on the value of x)(and where x isn't 
volatile, ect ect).


A compiler *could* detect and warn about such things (ie it's not 
forbidden by the laws of mathematics) but I don't think I know of any 
that do. And as there are good reasons for deliberately wanting to do 
the above it could only be a warning, not an error.


On 08/02/2017 23:36, 'Walter R. Ojeda Valiente' 
sistemas2000profesio...@gmail.com [firebird-support] wrote:
The error is of the programmer, I agree with you, but to repeat the 
name of a variable without the Firebird showing any message of error 
is, at least for me, a bug.


To have 2 or more variables with the same name after the INTO clause 
is useless. The compiler can be smart enough to detect such thing.


Or not?

Greetings.

Walter.

On Wed, Feb 8, 2017 at 2:36 PM, 'Leyne, Sean' 
s...@broadviewsoftware.com [firebird-support] 
> wrote:




> Yes, but I can not know the value of the column X.ALU_NOMBRE
>
> And the idea, of course, is know that value, that's why it
appears in the FOR
> SELECT. If not, I can do nothing with X.ALU_NOMBRE

You are asking for the system to evaluate the *intent* of logic.

That is completely outside the purview of any application
environment that I know.

The only thing that a system can check/enforce is the correctness
of the code, not to check whether the developer has 2 brain cells.


Sean






--
Tim Ward