Re: [sqlite] BUG: The sqlite3 shell does not call setlocale

[Jean-Denis Muys]

On 12/14/09 16:56 , "Jay A. Kreibich"  wrote:

> On Mon, Dec 14, 2009 at 10:41:04AM +0100, Jean-Denis Muys scratched on the
> wall:
>> 
>> So Jay, what you are saying is that SQLite doesn't have to provide bug-free
>> features when those features are optional?
> 
[snip]
>   So, yes, I'm saying, and consider it acceptable, that the SQLite team
>   does not have to provide bug-free features when those features are
>   optional-- at least if they state the fact that the features are
>   optional and unsupported.  That is, in effect, what "unsupported"
>   means.
>   

Ok, I think I get your position better, and I agree with it, about
unsupported features.

So the question is not so much whether readline support is optional, than
whether it's supported or not. (I don't know).

Regarding Alexei's request, would their be any downside to the addition of
the single line he submitted?

If not, then, rather than a bug fix, it could be considered an improvement.

What I'm getting at is that I've seen arguments that the requested addition
might in some (many?, most?) cases not be necessary, but not that it would
be harmful. So would it?

If no harm is expected, beside some changes to an optional, unsupported
feature, then perhaps it should be considered. The fact that the shell is
out of the main library should possibly make it easier to improve without
going through an extensive process.

Jean-Denis

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] BUG: The sqlite3 shell does not call setlocale

[Nicolas Williams]

On Sat, Dec 12, 2009 at 12:39:23PM -0800, Roger Binns wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> Alexey Pechnikov wrote:
> > On unix the shell must do this initialisation:
> > 
> > setlocale(LC_ALL, "");
> 
> Why?  Yes I know what the call does, but what desirable effect does it have
> on shell input and output?  Pretty much all formatted output is done using
> %s.  The only float output is for timing commands.  No input is done using
> scanf.  Except for the disabled by default iotrace, all fopens are in binary
> mode.

If there's any perror()/strerror() calls, or if you're printing
floats/doubles for user presentation then you definitely want to
setlocale().  There are probably other reasons to call setlocale().

I see only strerror() calls in the proxy lock paths, so that's out.  And
SQLite3 doesn't distinguish between user presentation and other
contexts.  So not calling setlocale(LC_ALL, "") seems possibly
appropriate, but it seems better to either not depend on it (so that if
called by an the application, or an extension, nothing bad happens) or
to force the use of the C locale (but a library must not do that!).

Since an application using libsqlite3 can always call setlocale(), it
seems to me that the sqlite3 shell should as well.

Nico
-- 
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] BUG: The sqlite3 shell does not call setlocale

[Jay A. Kreibich]

On Mon, Dec 14, 2009 at 10:41:04AM +0100, Jean-Denis Muys scratched on the wall:
> 
> On 12/13/09 16:34 , "Jay A. Kreibich"  wrote:
> 
> > On Sun, Dec 13, 2009 at 01:47:43PM +0300, Alexey Pechnikov scratched on the
> > wall:
> 
> >> On Sunday 13 December 2009 02:09:48 Roger Binns wrote:
> 
> >> The libsqlite is locale-independent but sqlite3 shell linked with
> >> readline and it's locale-dependent.
> > 
> >   That's a user choice.  It does not link with readline by default.
> > 
> 
> The rest of this topic is way over my head, but I don't get this argument.
> Basically it seems to me it goes like that:
> 
> Alexei: SQLite has a bug when use with optional feature X
> Jay: that's a user choice. Feature X is not on by default.
> 
> So Jay, what you are saying is that SQLite doesn't have to provide bug-free
> features when those features are optional?

  First of all, I personally don't consider this a "bug."  It isn't
  like the shell crashes, or corrupts data.  The issue is that, when
  utilizing an external library, some presets are not configured the
  way the original poster desires.
  
  From my viewpoint, if this is really truly "incorrect" or not is an
  issue that is still open for debate.  This gets into the question of
  the intent and purpose of the application at hand.  As others have
  pointed out, the sqlite3 command line shell is designed to be fairly
  light-weight and minimal.  I'm not sure I'd go as far as saying it is
  really just for debugging, but it is definitely not an sqlplus, or some
  other massive "database administrator" shell environment that is
  commonly seen with large RDBMS servers.
  
  From that stance, the readline support is optional, not on by default,
  and if you really don't like the way it works your best bet is to just
  turn it off.  Complaining about how readline support is completely broken
  is most likely to simply get the whole feature marked "unsupported"--
  or simply ripped out all together.  No more bugs; problem solved.
  
  But their high bar for supported features puts the SQLite developers
  in a very difficult situation for anything a bit off the beaten path.
  If a customer (and they do have paying customers, and they do get to
  call some of the shots) wants an odd feature added to the shell or
  one of the utility interfaces, there is the question of, "is this
  officially supported?"   And very often the answer is, "no."

  And I think that is reasonable.  The SQLite team is small, yet they
  provide a great product that is constantly being updated.  They take
  testing of the core database engine very very seriously, and go to
  great lengths to make sure the core library does what it is supposed
  to do.  I'd rather they spend time working on the core than adding
  cute options to the shell.  If I want to mess around with the shell,
  I can do that.  I'm not as adventurous to jump into the core library.
  There are also other GUI tools and other products that allow me to
  interact with the database library.  If I really don't like the
  shell, I can use one of those or write my own.

  Also, consider this paragraph from the compile-options for SQLite:

 The SQLITE_OMIT_* compile-time options are usually untested and
 are almost certainly untested in combination. Any or all of these
 options may be removed from the code in future releases and
 without warning. For any particular release, some of these options
 may cause compile-time or run-time failures, particularly when
 used in combination with other options.

  The team has been nice enough to try to provide these features.
  They're very useful to some customers.  But the team also knows they
  can't test and support all these features as well as they should to
  officially call them supported and correct.  So they just admit--
  these may work, they may not.  If that makes you uncomfortable, don't
  use them.  If you need them, test them and possibly fix them yourself--
  which is the exact same position you would be in if you had to develop
  the feature yourself.

  So, yes, I'm saying, and consider it acceptable, that the SQLite team
  does not have to provide bug-free features when those features are
  optional-- at least if they state the fact that the features are
  optional and unsupported.  That is, in effect, what "unsupported"
  means.
  
  Or "experimental."  Virtual tables have been around some four years
  and are still "experimental," despite the fact FTS3 (a rather
  important feature) depends on them.  I suspect the biggest reason
  virtual tables are still marked experimental is that, being an
  external interface, they're very very difficult to test.



  The only place I think there is room for an argument is what,
  exactly, is the status of the shell.c file and the sqlite3 tool.  It
  is clearly considered an important part of the SQLite distribution,
  at least by customers, but it is not part of the core library and is
  not 

Re: [sqlite] BUG: The sqlite3 shell does not call setlocale

[Jean-Denis Muys]

On 12/13/09 16:34 , "Jay A. Kreibich"  wrote:

> On Sun, Dec 13, 2009 at 01:47:43PM +0300, Alexey Pechnikov scratched on the
> wall:

>> On Sunday 13 December 2009 02:09:48 Roger Binns wrote:

>> The libsqlite is locale-independent but sqlite3 shell linked with
>> readline and it's locale-dependent.
> 
>   That's a user choice.  It does not link with readline by default.
> 

The rest of this topic is way over my head, but I don't get this argument.
Basically it seems to me it goes like that:

Alexei: SQLite has a bug when use with optional feature X
Jay: that's a user choice. Feature X is not on by default.

So Jay, what you are saying is that SQLite doesn't have to provide bug-free
features when those features are optional?

Did I read you correctly?

Jean-Denis

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] BUG: The sqlite3 shell does not call setlocale

[Roger Binns]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Alexey Pechnikov wrote:
> The locale of CSV file may be defined in virtualtable arguments like to
> create virtual table test using VirtualText 
> ('/srv/projects/work/billing/export/ats.csv','utf8','','.',',',',');
> 
> But _default_ may be system locale. How to do filesystem access in a 
> virtualtable without locale 
> information? 

The best I can see is to call setlocale yourself, saving the prior locale
that it returns, then using localeconv/nl_langinfo to get the info you need
and then calling setlocale again with the original locale returned by your
setlocale call.

This will be fine for single threaded programs but for multi-threaded
programs will have temporarily changed the locale for a short while.  To be
100% correct in that case I'd fork and then let the child process gather
locale information to return to the parent over a pipe.  (A lot of work for
something so simple!)

On Linux you can also parse files in /usr/share/i18n/locales.  Heck you
could probably just compile the info into your code.

Roger
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkslM/kACgkQmOOfHg372QT8ZgCfTLb/1zCLQZORm6vYcrAklrWm
/wIAn1FmC3mmKaYMBsCmL6Oq+GSbRw2d
=hUCe
-END PGP SIGNATURE-
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] BUG: The sqlite3 shell does not call setlocale

[Jay A. Kreibich]

On Sun, Dec 13, 2009 at 01:47:43PM +0300, Alexey Pechnikov scratched on the 
wall:
> Hello!
> 
> On Sunday 13 December 2009 02:09:48 Roger Binns wrote:
> > Alexey Pechnikov wrote:
> > > This is standart for all i18n applications.
> > 
> > The SQLite shell is not an i18n application, and this is deliberate.  It is
> > a developer tool.  That is why for example it always uses a dot for a
> > decimal point and not a comma even if that is what the locale does.  The
> > output is always the same wherever it is used.  (Same thing applies for 
> > input.)
> 
> The libsqlite is locale-independent but sqlite3 shell linked with 
> readline and it's locale-dependent. 

  That's a user choice.  It does not link with readline by default.

  (Well, the UNIX autoconf version does, but the raw source has it off
  by default.)

   -j

-- 
Jay A. Kreibich < J A Y  @  K R E I B I.C H >

"Our opponent is an alien starship packed with atomic bombs.  We have
 a protractor."   "I'll go home and see if I can scrounge up a ruler
 and a piece of string."  --from Anathem by Neal Stephenson
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] BUG: The sqlite3 shell does not call setlocale

[Alexey Pechnikov]

Hello!

On Sunday 13 December 2009 02:09:48 Roger Binns wrote:
> Alexey Pechnikov wrote:
> > This is standart for all i18n applications.
> 
> The SQLite shell is not an i18n application, and this is deliberate.  It is
> a developer tool.  That is why for example it always uses a dot for a
> decimal point and not a comma even if that is what the locale does.  The
> output is always the same wherever it is used.  (Same thing applies for 
> input.)

The libsqlite is locale-independent but sqlite3 shell linked with readline and 
it's locale-dependent. 

> > The SQLite virtualtables can
> > perform access to filesystem, read/write scv files and other - how they can 
> > to
> > determine the current locale?
> 
> That is indeed trickier.  But even your example is hard. If I am sitting in
> Canada and get a German CSV file, which locale applies?  

The locale of CSV file may be defined in virtualtable arguments like to
create virtual table test using VirtualText 
('/srv/projects/work/billing/export/ats.csv','utf8','','.',',',',');

But _default_ may be system locale. How to do filesystem access in a 
virtualtable without locale 
information? 

Best regards, Alexey Pechnikov.
http://pechnikov.tel/
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] BUG: The sqlite3 shell does not call setlocale

[Roger Binns]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Alexey Pechnikov wrote:
> This is standart for all i18n applications. 

The SQLite shell is not an i18n application, and this is deliberate.  It is
a developer tool.  That is why for example it always uses a dot for a
decimal point and not a comma even if that is what the locale does.  The
output is always the same wherever it is used.  (Same thing applies for input.)

> The SQLite virtualtables can
> perform access to filesystem, read/write scv files and other - how they can 
> to 
> determine the current locale?

That is indeed trickier.  But even your example is hard. If I am sitting in
Canada and get a German CSV file, which locale applies?  Virtual tables are
analogous to libraries and so cannot always rely on the hosting process.
Additionally many processes these days have multiple libraries (and virtual
tables if appropriate) so requiring things setup just for one is not
helpful.  For example if the process is serving web requests it may want the
locale to match the user not the server, and potentially change on each request.

In your particular scenario I'd add some sort of explicit parameter for the
virtual table that allows specifying the locale, with perhaps an empty
string/null meaning look at the environment variables or code page as a
fallback.

Roger
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkskIrwACgkQmOOfHg372QSS/gCeI6KPWCJzK54VFL4JN804ryiK
YwMAoMJcgHqAXxbLreuxgLurT2diFjcw
=Kq1K
-END PGP SIGNATURE-
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] BUG: The sqlite3 shell does not call setlocale

[Alexey Pechnikov]

Hello!

On Saturday 12 December 2009 23:39:23 Roger Binns wrote:
> > setlocale(LC_ALL, "");
> 
> Why?  Yes I know what the call does, but what desirable effect does it have
> on shell input and output?  Pretty much all formatted output is done using
> %s.  The only float output is for timing commands.  No input is done using
> scanf.  Except for the disabled by default iotrace, all fopens are in binary
> mode.

This is standart for all i18n applications. The SQLite virtualtables can 
perform access to filesystem, read/write scv files and other - how they can to 
determine the current locale?

Best regards, Alexey Pechnikov.
http://pechnikov.tel/
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] BUG: The sqlite3 shell does not call setlocale

[Roger Binns]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Alexey Pechnikov wrote:
> On unix the shell must do this initialisation:
> 
> setlocale(LC_ALL, "");

Why?  Yes I know what the call does, but what desirable effect does it have
on shell input and output?  Pretty much all formatted output is done using
%s.  The only float output is for timing commands.  No input is done using
scanf.  Except for the disabled by default iotrace, all fopens are in binary
mode.

Roger

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAksj/3YACgkQmOOfHg372QQ6sACdHJfH6+fhNnJDyg8eMcApliQs
t6MAoNlEtawQpNjM9T/9DjnWD41j6imc
=MwgT
-END PGP SIGNATURE-
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users