Re: [fpc-devel] RTL and Unicode filenames operations.

[Michael Van Canneyt]

On Tue, 23 Mar 2010, Graeme Geldenhuys wrote:


Marco van de Voort het geskryf:


For the same reason why the Unix rtl is not a Windows emulation. We are
writing a multiplatform compiler and rtl, not an emulation.


If only that was true... I can highlight a few places where Windows
emulation is used rather that pure multi-platform implementation.  :-/


Most cases are out of Delphi/TP compatibility reasons. 
We cannot throw that overboard.


But nothing stops people from developing an alternate RTL, free of
windows-isms, and preferably 100% OOP :-)

Michael.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Graeme Geldenhuys]

Marco van de Voort het geskryf:
> 
> For the same reason why the Unix rtl is not a Windows emulation. We are
> writing a multiplatform compiler and rtl, not an emulation.

If only that was true... I can highlight a few places where Windows
emulation is used rather that pure multi-platform implementation.  :-/



Regards,
  - Graeme -

-- 
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://opensoft.homeip.net/fpgui/

___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[dmitry boyarintsev]

On Sun, Mar 21, 2010 at 7:53 PM, Jonas Maebe  wrote:
> On Mac it's utf-8, not utf-16.

At least, not as "sequence of bytes", see test1.pas of the ufiles package.

thanks,
dmitry
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Jonas Maebe]

On 21 Mar 2010, at 17:29, Martin Schreiber wrote:

> On Windows and Mac utf-16 is then native filename encoding

On Mac it's utf-8, not utf-16.


Jonas
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Marco van de Voort]

In our previous episode, Martin Schreiber said:
> > Yes, it does so and i see no problems here. The package also converts
> > UTF-16 to ansi encoding for Win9x.
> > Converting string from UTF16 to UTF8 (or any other encoding) is not
> > much time penalty comparing to the time of the file operation itself.
> 
> Agreed. The MSEgui file utils do the same with good results.

> If I understand right, Marco insists that on Linux utf-8 should be used for
> file operations because it is the "native" file name encoding on Linux. 

On Linux and FreeBSD yes, everything is UTF-8. Then using UTF-16 types here
and there should not be necessary. Even if it is not a performance killer.

> don't know how that can be achieved BTW.  On Windows and Mac utf-16 is
> then native filename encoding so I don't understand why the API can't be
> standardised to utf-16.

For the same reason why the Unix rtl is not a Windows emulation. We are
writing a multiplatform compiler and rtl, not an emulation.

The Delphi stuff causes already enough windowsisms. Let's not add one.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Martin Schreiber]

On Sunday 21 March 2010 16:17:49 dmitry boyarintsev wrote:
> On Tue, Mar 21, 2010 at 2:01 PM, Marco van de Voort
>
>  wrote:
> > I'll reiterate my opinion that first a decision about what the working
> > stringtype of the RTL will be. IMHO there is no decent solution till
> > there is a real utf-8 type (read cpnewstr)
>
> The whole reason for the package, is not to wait for cpnewstr being
> implemented. That's the point that package's interface uses UnicodeString,
> rather than ambiguous "String" type (that's currently system dependent).
>
> > Note that the recent update does not tackle my main gripe at all, the
> > main units interface is still UTF-16 on Unix, it just translates it to
> > UTF-8 in the backend.
>
> Yes, it does so and i see no problems here. The package also converts
> UTF-16 to ansi encoding for Win9x.
> Converting string from UTF16 to UTF8 (or any other encoding) is not
> much time penalty comparing to the time of the file operation itself.

Agreed. The MSEgui file utils do the same with good results. If I understand 
right, Marco insists that on Linux utf-8 should be used for file operations 
because it is the "native" file name encoding on Linux. There are still Linux 
installations and partitions with other encodings than utf-8 and on Linux, 
filenames are an array of bytes without any encoding AFAIK, so invalid utf-8 
sequences are possible and must be filtered and converted to valid sequences 
in both utf-8 or utf-16.
I don't know how that can be achieved BTW.
On Windows and Mac utf-16 is then native filename encoding so I don't 
understand why the API can't be standardised to utf-16.

Martin
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Marco van de Voort]

In our previous episode, dmitry boyarintsev said:
> On Tue, Mar 21, 2010 at 2:01 PM, Marco van de Voort
>  wrote:
> > I'll reiterate my opinion that first a decision about what the working 
> > stringtype of the RTL will be.
> > IMHO there is no decent solution till there is a real utf-8 type (read 
> > cpnewstr)
> 
> The whole reason for the package, is not to wait for cpnewstr being 
> implemented.
> That's the point that package's interface uses UnicodeString, rather
> than ambiguous "String" type (that's currently system dependent).

Yes. Which is no problem for the package. But it is a reason to not include
it. 
 
> > Note that the recent update does not tackle my main gripe at all, the main 
> > units interface is still UTF-16 on Unix,
> > it just translates it to UTF-8 in the backend.
> Yes, it does so and i see no problems here. The package also converts
> UTF-16 to ansi encoding for Win9x.
> Converting string from UTF16 to UTF8 (or any other encoding) is not
> much time penalty comparing to the time of the file operation itself.

Sure. But that still doesn't mean UTF-16 is the ideal choice.

> Btw, the current RTL implementation also forces a user to convert a
> "string" type to utf8, to work correctly on Unixes for unicode names.

Currently there is no unicode support. The point is rather that I don't want
to commit an halve one, and be stuck with it (and all kinds of packages
"just" using it) forever.

___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[dmitry boyarintsev]

On Tue, Mar 21, 2010 at 2:01 PM, Marco van de Voort
 wrote:
> I'll reiterate my opinion that first a decision about what the working 
> stringtype of the RTL will be.
> IMHO there is no decent solution till there is a real utf-8 type (read 
> cpnewstr)

The whole reason for the package, is not to wait for cpnewstr being implemented.
That's the point that package's interface uses UnicodeString, rather
than ambiguous "String" type (that's currently system dependent).

> Note that the recent update does not tackle my main gripe at all, the main 
> units interface is still UTF-16 on Unix,
> it just translates it to UTF-8 in the backend.
Yes, it does so and i see no problems here. The package also converts
UTF-16 to ansi encoding for Win9x.
Converting string from UTF16 to UTF8 (or any other encoding) is not
much time penalty comparing to the time of the file operation itself.
Btw, the current RTL implementation also forces a user to convert a
"string" type to utf8, to work correctly on Unixes for unicode names.

thanks,
dmitry
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[dmitry boyarintsev]

Any hints about the package?
http://bugs.freepascal.org/view.php?id=15795

thanks,
dmitry
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Marco van de Voort]

In our previous episode, dmitry boyarintsev said:
> 
> The patch has been updated: http://bugs.freepascal.org/view.php?id=15795
> (is it possible to allow the reporter to delete its files in FPC bug tracker?)
> 
> The library is now RTL "encoding" independent. Actually it's
> configurable, so, if RTL string encoding changes, only a 2-3 lines in
> the library should be changed.

I don't know, but I can delete anything but the most recent if you wish?
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[dmitry boyarintsev]

On Tue, Mar 9, 2010 at 1:49 PM, Marco van de Voort  wrote:
>> The library is now RTL "encoding" independent. Actually it's
>> configurable, so, if RTL string encoding changes, only a 2-3 lines in
>> the library should be changed.
>
> I don't know, but I can delete anything but the most recent if you wish?

yes, please. just keep: ufiles-03032010-clean.zip

afaik, in Lazarus it's allowed to delete attached files for the reporter.

thanks,
dmitry
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[dmitry boyarintsev]

Hello,

The patch has been updated: http://bugs.freepascal.org/view.php?id=15795
(is it possible to allow the reporter to delete its files in FPC bug tracker?)

The library is now RTL "encoding" independent. Actually it's
configurable, so, if RTL string encoding changes, only a 2-3 lines in
the library should be changed.

On Wed, Feb 17, 2010 at 1:10 PM, Mattias Gaertner
 wrote:
>> The implementation is not only about RTL encoding, but system
>> environment as well, since it depends upon Windows *W function to be
>> available. But who cares about 9x these days?! :)
>
> Notes:
> - Comparison is wrong under OS X, see lcl FileUtil.pas
> - missing a note that files with broken UTF8 chars can not be read
>  under Linux. Maybe add a note where the system encoding file routines
>  are.
> - ExtractRelativePath works Delphi compatible, but always case
>  insensitive, so not useful under Linux.

All 3 issues should be fixed by now.

thanks,
dmitry
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Florian Klaempfl]

Felipe Monteiro de Carvalho schrieb:
> 2010/2/17 Flávio Etrusco :
>> I've read somewhere that Windows ANSI functions support utf-8?
>> (despite the name)
> 
> lol!!!
> 
> You read from someone deeply missinformed =D

Switch the console to a font supporting unicode, execute chcp 65001 and
you can output utf-8.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Michael Schnell]

On 02/17/2010 06:52 PM, Martin Schreiber wrote:
> Aren't filenames on Linux an array of bytes without any encoding by 
> definition?
>   
Right. But how to visualize them ?

-Michael
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Felipe Monteiro de Carvalho]

2010/2/17 Flávio Etrusco :
> I've read somewhere that Windows ANSI functions support utf-8?
> (despite the name)

lol!!!

You read from someone deeply missinformed =D

-- 
Felipe Monteiro de Carvalho
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Flávio Etrusco]

On Wed, Feb 17, 2010 at 4:10 AM, dmitry boyarintsev
 wrote:
> Reported: http://bugs.freepascal.org/view.php?id=15795
>
> It's up to FPC team to accept or reject the package.
>
> thanks,
> dmitry

I've read somewhere that Windows ANSI functions support utf-8?
(despite the name)

-Flávio
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Marco van de Voort]

In our previous episode, Martin Schreiber said:
> On Wednesday 17 February 2010 09:15:53 Marco van de Voort wrote:
> >
> > Reject as far as I'm concerned. I'd like to see the RTL being in the system
> > encoding. (IOW UTF-8 on at least FreeBSD/Linux) using an abstracted
> > RTLString.
> >
> Aren't filenames on Linux an array of bytes without any encoding by 
> definition?

I never saw that. The encoding is sometimes a bit opague yes. It will be a
practical detail which exact 1-byte encoding to choose for rtlstring
(cp_utf8, rawbytestring or default). Whatever works best. 

> Are you sure there are no Linux running with other system 
> encoding than utf-8 today?

No. Does it matter?

> How are imported filesystems with other encodings (for example Samba
> shares) handled?

Don't see a relevance here.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Martin Schreiber]

On Wednesday 17 February 2010 09:15:53 Marco van de Voort wrote:

>
> Reject as far as I'm concerned. I'd like to see the RTL being in the system
> encoding. (IOW UTF-8 on at least FreeBSD/Linux) using an abstracted
> RTLString.
>
Aren't filenames on Linux an array of bytes without any encoding by 
definition? Are you sure there are no Linux running with other system 
encoding than utf-8 today? How are imported filesystems with other encodings 
(for example Samba shares) handled?

Martin
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Marco van de Voort]

In our previous episode, dmitry boyarintsev said:
> > RTLString.
> >
> > This sets a precedent.
> 
> Could you please resolve the issue with "won't be accepted" (or no
> changes required)?

It's not my decision alone. The whole problem with this unicode string is
that there is no definitive roadmap, and the utf-8 ansistring is not ready.
 
> The implementation is not only about RTL encoding, but system
> environment as well, since it depends upon Windows *W function to be
> available.

True, but I'm not going to commit anything Windows only till the plans for
the system environment are clear. On all relevant platforms.

> But who cares about 9x these days?! :)

I don't. But that is not even a problem;

FPC has prepared long for this by having and UNICODE define in most
windows headers. (afaik Peter already did when he translated windows unit
originally), but specially the older bits are not test and probably full of
bugs.

In theory, we could have a win32u target that sets UNICODE and switch the
default of most functions to unicode, like D2009+. It is probably going to
happen that way anyway, sooner or later.

___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Marco van de Voort]

In our previous episode, Mattias Gaertner said:
> > 
> > Could you please resolve the issue with "won't be accepted" (or no
> > changes required)?
> > 
> > The implementation is not only about RTL encoding, but system
> > environment as well, since it depends upon Windows *W function to be
> > available. But who cares about 9x these days?! :)
> 
> Notes:
> - Comparison is wrong under OS X, see lcl FileUtil.pas
> - missing a note that files with broken UTF8 chars can not be read
>   under Linux. Maybe add a note where the system encoding file routines
>   are.
> - ExtractRelativePath works Delphi compatible, but always case
>   insensitive, so not useful under Linux.

On the implementation front I had some doubts about charinset. It doesn't
seem to be locale aware and only treat the bottom 128 chars. Is this
compatible?
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[dmitry boyarintsev]

Thanks Mattias, i'll do proper changes to file name comparison functions.

thanks,
dmitry
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Mattias Gaertner]

On Wed, 17 Feb 2010 12:34:35 +0300
dmitry boyarintsev  wrote:

> On Wed, Feb 17, 2010 at 11:15 AM, Marco van de Voort  wrote:
> > Reject as far as I'm concerned. I'd like to see the RTL being in the system
> > encoding. (IOW UTF-8 on at least FreeBSD/Linux) using an abstracted
> > RTLString.
> >
> > This sets a precedent.
> 
> Could you please resolve the issue with "won't be accepted" (or no
> changes required)?
> 
> The implementation is not only about RTL encoding, but system
> environment as well, since it depends upon Windows *W function to be
> available. But who cares about 9x these days?! :)

Notes:
- Comparison is wrong under OS X, see lcl FileUtil.pas
- missing a note that files with broken UTF8 chars can not be read
  under Linux. Maybe add a note where the system encoding file routines
  are.
- ExtractRelativePath works Delphi compatible, but always case
  insensitive, so not useful under Linux.


Mattias
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[dmitry boyarintsev]

On Wed, Feb 17, 2010 at 11:15 AM, Marco van de Voort  wrote:
> Reject as far as I'm concerned. I'd like to see the RTL being in the system
> encoding. (IOW UTF-8 on at least FreeBSD/Linux) using an abstracted
> RTLString.
>
> This sets a precedent.

Could you please resolve the issue with "won't be accepted" (or no
changes required)?

The implementation is not only about RTL encoding, but system
environment as well, since it depends upon Windows *W function to be
available. But who cares about 9x these days?! :)

thanks,
dmitry
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Vincent Snijders]

dmitry boyarintsev schreef:

On Wed, Feb 17, 2010 at 11:15 AM, Marco van de Voort  wrote:

Reject as far as I'm concerned. I'd like to see the RTL being in the system
encoding. (IOW UTF-8 on at least FreeBSD/Linux) using an abstracted
RTLString.


Vincent, is there a place in CCR for the run-time package?


Yes.

Vincent
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Michael Schnell]

No comment on the idea of converting dynamically typed string constants
at the first touch in realtime ?

-Michael
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[dmitry boyarintsev]

On Wed, Feb 17, 2010 at 11:15 AM, Marco van de Voort  wrote:
> Reject as far as I'm concerned. I'd like to see the RTL being in the system
> encoding. (IOW UTF-8 on at least FreeBSD/Linux) using an abstracted
> RTLString.

Vincent, is there a place in CCR for the run-time package?

thanks,
dmitry
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Marco van de Voort]

In our previous episode, Vincent Snijders said:
> dmitry boyarintsev schreef:
> > Reported: http://bugs.freepascal.org/view.php?id=15795
> > 
> > It's up to FPC team to accept or reject the package.
> 
> Thanks, I will monitor the issue to see the outcome.

Reject as far as I'm concerned. I'd like to see the RTL being in the system
encoding. (IOW UTF-8 on at least FreeBSD/Linux) using an abstracted
RTLString.

This sets a precedent.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Vincent Snijders]

dmitry boyarintsev schreef:

Reported: http://bugs.freepascal.org/view.php?id=15795

It's up to FPC team to accept or reject the package.


Thanks, I will monitor the issue to see the outcome.

Vincent
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[dmitry boyarintsev]

Reported: http://bugs.freepascal.org/view.php?id=15795

It's up to FPC team to accept or reject the package.

thanks,
dmitry
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Vincent Snijders]

Marco van de Voort schreef:


That is not my call, but I won't commit it. Because all this stuff must be
maintained when real solutions come later.

if Lazarus needs a quick fix, they should keep it in own SVN.



Or use the units from msegui or use the units from tiopf or use the 
units from (any other group that needed a quick fix). I just do not like 
that everybody needs to implement this on their own.


I would vote to add them in 2.5.1 and you can remove them if you have 
implemented these things in the RTL.


Vincent
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Marco van de Voort]

In our previous episode, dmitry boyarintsev said:
> On Mon, Feb 1, 2010 at 3:20 PM, Marco van de Voort  wrote:
> > More importantly, the stuff in the FCL depends on the RTL, so stuffing in
> > the FCL is useless since it would need primitives in the RTL to operate on.
> 
> Does it mean, if i provide a FCL package with:
> 
> unit
>   UnicodeFileUtils;
> 
> function FindFirst(const Name: UnicodeString)
> ...
> 
> function IncludeTrailngPathDelimiter(const path: UnicodeString)
> 
> 
> type
>   TFileStreamUnicode = class(THandleStream)
> constructor Create(const AFileName: UnicodeString)
> 
> etc...
> 
> it would be rejected?

That is not my call, but I won't commit it. Because all this stuff must be
maintained when real solutions come later.

if Lazarus needs a quick fix, they should keep it in own SVN.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[dmitry boyarintsev]

On Mon, Feb 1, 2010 at 3:20 PM, Marco van de Voort  wrote:
> More importantly, the stuff in the FCL depends on the RTL, so stuffing in
> the FCL is useless since it would need primitives in the RTL to operate on.

Does it mean, if i provide a FCL package with:

unit
  UnicodeFileUtils;

function FindFirst(const Name: UnicodeString)
...

function IncludeTrailngPathDelimiter(const path: UnicodeString)


type
  TFileStreamUnicode = class(THandleStream)
constructor Create(const AFileName: UnicodeString)

etc...

it would be rejected?

thanks,
dmitry
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Marco van de Voort]

In our previous episode, dmitry boyarintsev said:
> > in fpGUI project (fpg_utils.pas unit), having functions like
> > fpgFileExists(), fpgFindFirst(), fpgFindNext() etc. Use them instead of the
> > standard RTL functions, and they will try and handle Unicode file system
> > access correctly on each platform.
> 
> Can Unicode file operations become a FCL package?

No, since this is part of the RTL, and needs to stay that way out of delphi
compat.

More importantly, the stuff in the FCL depends on the RTL, so stuffing in
the FCL is useless since it would need primitives in the RTL to operate on.

___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[dmitry boyarintsev]

On Mon, Feb 1, 2010 at 11:09 AM, Graeme Geldenhuys
 wrote:
> Alternatively you can introduce helper RTL functions. We have done similar
> in fpGUI project (fpg_utils.pas unit), having functions like
> fpgFileExists(), fpgFindFirst(), fpgFindNext() etc. Use them instead of the
> standard RTL functions, and they will try and handle Unicode file system
> access correctly on each platform.

Can Unicode file operations become a FCL package?
So it would be RTL implementation independent.  As well as always
available for FPC user, without need of external GUI packages (i.e.
fpGUI, LCL or mse)

thanks,
dmitry
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Michael Schnell]

On 02/01/2010 12:26 PM, Marco van de Voort wrote:
> Daniel touches the main problem. The string type is compiletime, while the
> standard encoding is runtime.
>   
I do see this (that is why I  said "If possible", even if I have no idea
about what impact this has.

Maybe it could be possible to find a way to invent a "one-time
conversion" for this class of strings: The first time they are touched
they get converted and the next time they are already in place correctly.

-Michael
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Marco van de Voort]

In our previous episode, Michael Schnell said:
> 
> I feel that the runtime performance overhead necessary for doing the RTL
> with dynamically encoded strings is not too big, provided the RTL itself
> does not force automatic conversions (e.g. by inappropriately encoded
> string constants) .

Daniel touches the main problem. The string type is compiletime, while the
standard encoding is runtime.


___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Michael Schnell]

On 02/01/2010 10:53 AM, Marco van de Voort wrote:
> It makes sense to take the
> encoding closest to the system, to at least allow minimal conversion
> scenarios.
>   

Of course !

If the RTL is done with dynamically encoded strings you would need to
take care that anything string value that neither comes from the GUI
framework (and thus is encoded in a way the RTL designer might know but
should ignore as largely as possible to produce as portable code as
possible) nor comes from the user code (the RTL designers can't know
which encoding they use) is handled appropriately. I suppose most of
this is string constants. If possible, those should be allocated with
the system encoding at compile time.

I feel that the runtime performance overhead necessary for doing the RTL
with dynamically encoded strings is not too big, provided the RTL itself
does not force automatic conversions (e.g. by inappropriately encoded
string constants) .

-Michael
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Daniël Mantione]

Op Mon, 1 Feb 2010, schreef Marco van de Voort:


In our previous episode, Dani?l Mantione said:

Some commercial unices are also using utf-16 afaik, since they haven't
switched their TTY to UTF-8 yet. To be investigated.

I lean towards RTLString=utf8string; and RTLString=unicodestring on the
respective platforms, and then express the RTL in RTLString.


No, RTLString=ansistring on Linux and most Unix platforms. Unix is
encoding agnostic so if you want to use the native string type you must
simple use old fashioned ansistring.


This means crossplatform unicode support goes out of the window. I don't
like that.


It is a consequence if using the native string type. Otherwiser you can 
just as well use UTF-16 everywhere.


Daniël___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Marco van de Voort]

In our previous episode, Michael Schnell said:
> Right, But exactly this seems to ask for dynamically encoded strings
> allowing to do the user code as agnostic of the system encoding as
> possible while automatic conversions are made whenever necessary, but
> avoided whenever possible.

Both string types will probably be fully supported on unicode supporting
platforms. The point is what the RTL uses. It makes sense to take the
encoding closest to the system, to at least allow minimal conversion
scenarios.

> (Which as we all know can't be provided in an absolutely perfect way: in
> some cases the "agnostic" user code will not work as expected and in some
> other cases performance might be reduced due to multiple automatic
> conversions).

Yes. Note that what I say goes for the RTL, since obviously, in the core
library, speed and reducing dependancies is the most important. 

I don't mind if some XML lib or so is is always either UTF-16 or UTF-8
(depending on pedigree).
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Michael Schnell]

On 02/01/2010 10:27 AM, Marco van de Voort wrote:
> We are not Codegear, for which everything but Windows is an afterthought.
>   

Right, But exactly this seems to ask for dynamically encoded strings
allowing to do the user code as agnostic of the system encoding as
possible while automatic conversions are made whenever necessary, but
avoided whenever possible. (Which as we all know can't be provided in an
absolutely perfect way: in some cases the "agnostic" user code will not
work as expected and in some other cases performance might be reduced
due to multiple automatic conversions).

-Michael
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Marco van de Voort]

In our previous episode, Dani?l Mantione said:
> > Some commercial unices are also using utf-16 afaik, since they haven't
> > switched their TTY to UTF-8 yet. To be investigated.
> >
> > I lean towards RTLString=utf8string; and RTLString=unicodestring on the
> > respective platforms, and then express the RTL in RTLString.
> 
> No, RTLString=ansistring on Linux and most Unix platforms. Unix is 
> encoding agnostic so if you want to use the native string type you must 
> simple use old fashioned ansistring.

This means crossplatform unicode support goes out of the window. I don't
like that.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Daniël Mantione]

Op Mon, 1 Feb 2010, schreef Marco van de Voort:


In our previous episode, Michael Schnell said:

D2009/d2010 are still windows only.


Do you think there is any general problem in using D2009 strings with
dynamic encoding with Unix ? On the first view I don't see one.


In general I prefer to use each platform using its native encoding.

Some commercial unices are also using utf-16 afaik, since they haven't
switched their TTY to UTF-8 yet. To be investigated.

I lean towards RTLString=utf8string; and RTLString=unicodestring on the
respective platforms, and then express the RTL in RTLString.


No, RTLString=ansistring on Linux and most Unix platforms. Unix is 
encoding agnostic so if you want to use the native string type you must 
simple use old fashioned ansistring.


Daniël___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Marco van de Voort]

In our previous episode, Michael Schnell said:
> > D2009/d2010 are still windows only.
> >   
> Do you think there is any general problem in using D2009 strings with
> dynamic encoding with Unix ? On the first view I don't see one.

In general I prefer to use each platform using its native encoding. 

Some commercial unices are also using utf-16 afaik, since they haven't
switched their TTY to UTF-8 yet. To be investigated.

I lean towards RTLString=utf8string; and RTLString=unicodestring on the
respective platforms, and then express the RTL in RTLString.

Since both types can be passed to another for the most, aside for a few VAR
functions this is transparent, and leaves the decision to the user what his
main string type is, and how many conversions he wants to allow or avoid.

Moreover, it allows for passing UTF-8 strings to system functions on
freebsd/linux in the base system (which could be benificial for the smaller
utils and the compiler itself), since then just passing it on does not
require cwstring

We are not Codegear, for which everything but Windows is an afterthought.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Michael Schnell]

On 02/01/2010 09:46 AM, Marco van de Voort wrote:
> D2009/d2010 are still windows only.
>   
Do you think there is any general problem in using D2009 strings with
dynamic encoding with Unix ? On the first view I don't see one.

-Michael
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Marco van de Voort]

In our previous episode, dmitry boyarintsev said:
> Now, i see, the technical solution is not easy.
> 
> > The point is more or less to come up with some logical system, not just
> > random additions when something is missing.
> 
> FPC will be forced to catch-up with D2009, anyway.

Yes, but D2009/d2010 are still windows only.

This is probably going to change with Codegear's crosscompiling plans
(expected in +/- august/september). Otoh they'll probably keep everything
UTF-16, since that is what they are used to.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[dmitry boyarintsev]

On Mon, Feb 1, 2010 at 11:31 AM, Marco van de Voort  wrote:
> Which requires libc, and *nix supports UTF-8 directly.

Now, i see, the technical solution is not easy.

> The point is more or less to come up with some logical system, not just
> random additions when something is missing.

FPC will be forced to catch-up with D2009, anyway.

I guess Win32 Unicode problem will be fixed on LCL level.

thanks,
dmitry
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Marco van de Voort]

In our previous episode, dmitry boyarintsev said:
> > What would you do on Unix? Any solution must be platform-wide.
> 
> I'm not talking about removing Ansi functions. I'm talking about the
> possibility of adding Unicode function equivalents.

> For Unix unicode functions would simply call Ansi functions,
> converting Unicode to AnsiString.

Which requires libc, and *nix supports UTF-8 directly.

The point is more or less to come up with some logical system, not just
random additions when something is missing.

And IMHO it is still to early, first the unicodestring support must be
finished, and then a course has to be laid out which of the thousands of
strings operations to convert.

___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[dmitry boyarintsev]

On Mon, Feb 1, 2010 at 11:01 AM, Marco van de Voort  wrote:
> What would you do on Unix? Any solution must be platform-wide.

I'm not talking about removing Ansi functions. I'm talking about the
possibility of adding Unicode function equivalents.
For Unix unicode functions would simply call Ansi functions,
converting Unicode to AnsiString.

thanks,
dmitry
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Graeme Geldenhuys]

dmitry boyarintsev wrote:
> 
> Would it be possible to fix the problem in RTL? by adding
> UnicodeString (WideString) version of file operation functions?

Until the cpstrnew branch in FPC is complete,  I guess the answer is No.

Alternatively you can introduce helper RTL functions. We have done similar
in fpGUI project (fpg_utils.pas unit), having functions like
fpgFileExists(), fpgFindFirst(), fpgFindNext() etc. Use them instead of the
standard RTL functions, and they will try and handle Unicode file system
access correctly on each platform.


Regards,
  - Graeme -

-- 
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://opensoft.homeip.net/fpgui/

___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

Re: [fpc-devel] RTL and Unicode filenames operations.

[Marco van de Voort]

In our previous episode, dmitry boyarintsev said:
> Any hints on these bugs?
> http://bugs.freepascal.org/view.php?id=12923
> http://bugs.freepascal.org/view.php?id=15642 // patch proposed to
> workaround RTL limitation in LCL
> 
> This is mostly concern of Windows OS. Moders Unix doesn't have the
> problem, usually having UTF8 as their's Ansi codepage.
> 
> Would it be possible to fix the problem in RTL? by adding
> UnicodeString (WideString) version of file operation functions?

What would you do on Unix? Any solution must be platform-wide.

First the UnicodeString (which is not Widestring) should be finished, and
some global decisions how to implement unicode system wide should be taken.

> Ant other "simplier" way than switching compiler itself to Unicode or
> other special string type?

Temporarily have some helper functions in the LCL. FPC doesn't support
unicode for all operations atm, while Lazarus does.
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel

[fpc-devel] RTL and Unicode filenames operations.

[dmitry boyarintsev]

Hello, FPC developers

Any hints on these bugs?
http://bugs.freepascal.org/view.php?id=12923
http://bugs.freepascal.org/view.php?id=15642 // patch proposed to
workaround RTL limitation in LCL

This is mostly concern of Windows OS. Moders Unix doesn't have the
problem, usually having UTF8 as their's Ansi codepage.

Would it be possible to fix the problem in RTL? by adding
UnicodeString (WideString) version of file operation functions?
Ant other "simplier" way than switching compiler itself to Unicode or
other special string type?

thanks,
dmitry
___
fpc-devel maillist  -  fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel