Well sure, but at least my approach would execute in less than
geological time.
----- Original Message -----
From: "Pavel Ivanov" <paiva...@gmail.com>
To: "Jim Showalter" <j...@jimandlisa.com>; "General Discussion of
SQLite Database" <sqlite-users@sqlite.org>
Sent: Tuesday, September 22, 2009 10:37 AM
Subject: Re: [sqlite] cygwin and sqlite
> Wow!!!! Jim, you're asking to rewrite the whole code of cygwin. You
> want all sorts of programs (like sed, awk, expr etc) to suddenly
> become a part of one program and the code of bash to be rewritten
> from
> scratch so that it never forks itself into new process but still
> implements all features that it has (I don't even sure if it's
> possible). Sorry, but that's not what cygwin is about. It's about
> taking sources of all programs from Unix and compiling them without
> changes (or with minimal changes) so that they work on Windows and
> so
> that they use all Unix principles (like standard structure of file
> system directories e.g.).
> With your requirements you'll need some other project. But I don't
> know any that supports bash scripts from Unix as is, without
> changes...
>
> Pavel
>
> On Tue, Sep 22, 2009 at 1:01 PM, Jim Showalter <j...@jimandlisa.com>
> wrote:
>> The point I was trying to make is that the majority of commands
>> cygwin
>> supports don't have to do any forking. They could just be method
>> calls. Pipes would be parameters going in and return values coming
>> out, all running in a single process. If fork() is specifically
>> called
>> in a script, then the slower approach would have to be used, but
>> most
>> of the time it would not.
>>
>> ----- Original Message -----
>> From: "Pavel Ivanov" <paiva...@gmail.com>
>> To: "General Discussion of SQLite Database"
>> <sqlite-users@sqlite.org>
>> Sent: Tuesday, September 22, 2009 6:58 AM
>> Subject: Re: [sqlite] cygwin and sqlite
>>
>>
>>>> But after all these years I wonder why they don't fix the fork
>>>> problem?
>>>> MacOS runs on Intel processors. Windows runs on Intel processors.
>>>> Surely
>>>> they could learn how it *should* be done by studying things like
>>>> the
>>>> Open Source Java code?
>>>
>>> You seem to forget the basics. It's not the processor who makes
>>> fork()
>>> possible, it's OS. Unix kernel implemented fork() and Unix kernel
>>> implemented process management in the way that makes
>>> implementation
>>> of
>>> fork() very quick and easy. Windows didn't implement fork() and it
>>> implemented process management in the way making fork()
>>> impossible.
>>> I'd say it's a superior achievement on cygwin side that they were
>>> able
>>> to implement fork() somehow at all. Just a simple fact: you
>>> execute
>>> some code that uses memory in some way then you call fork() on
>>> Unix
>>> and you already have 2 absolutely different processes that can
>>> access
>>> the same data in memory. On windows there's no way to start a new
>>> process so that it can access the same data as first process
>>> unless
>>> you thought about that beforehand, placed all your data into
>>> shared
>>> memory (which is a lot harder to work with, btw) and made another
>>> process to read the same shared memory. Also there's no way to
>>> start
>>> new process on Windows so that it executes the same code as first
>>> process from the point where second process was started...
>>> So I'd better not complained but tried to understand the roots of
>>> the problem...
>>>
>>> And if you want the comparable speed of scripts on both platforms
>>> I
>>> suggest you to look into perl (or python, whatever you prefer).
>>> The
>>> both have native implementations and their speed should be
>>> comparable
>>> to each other. That said of course if you don't start a lot of
>>> processes from scripts and don't try to run command line
>>> utilities...
>>>
>>> Pavel
>>>
>>> On Mon, Sep 21, 2009 at 10:29 PM, John <jhy...@earthlink.net>
>>> wrote:
>>>> Pavel Ivanov wrote:
>>>>>> MacBook Mac OS X 10.5.8
>>>>>> 2 GHz Intel Core Duo
>>>>>> 1 GB memory:
>>>>>> 17 minutes 46 seconds.
>>>>>>
>>>>>> IBM ThinkPad
>>>>>> Windows XP (latest patches)
>>>>>> 1.70 GHz, 512 MB memory:
>>>>>> 6 hours 25 minutes 57 seconds
>>>>>
>>>>> Windows is very slow in starting new processes if compared to
>>>>> any
>>>>> Unix
>>>>> system (especially if compared Windows + 512 MB and Unix + 1
>>>>> GB).
>>>>> In
>>>>> cygwin starting new processes even slower because for some
>>>>> reason
>>>>> emulating fork() involves starting 2 processes one of which dies
>>>>> immediately. And bash scripts use processes a lot especially
>>>>> with
>>>>> Unix
>>>>> paradigm when for each small action you start new program (like
>>>>> sed,
>>>>> awk, test, true and whole lot of others). Thus bash scripts on
>>>>> cygwin
>>>>> will be slow unavoidably.
>>>>> But I'm digressing. This is subject for some other mailing list.
>>>>> :)
>>>>
>>>> I'm blacklisted apparently on the cygwin mailing list for when a
>>>> couple of years ago I complained rather unflatteringly about how
>>>> slow
>>>> it was/is when I was writing a simple expenses program (that
>>>> works
>>>> in
>>>> seconds on my Mac). I forgot about that. A badge of honor in my
>>>> opinion.
>>>>
>>>> But after all these years I wonder why they don't fix the fork
>>>> problem?
>>>> MacOS runs on Intel processors. Windows runs on Intel processors.
>>>> Surely
>>>> they could learn how it *should* be done by studying things like
>>>> the
>>>> Open Source Java code?
>>>>
>>>> It looks like I won't be able to distribute my stopgap cygwin
>>>> code
>>>> on
>>>> Windows. I need to start speed reading my Java/Swing books I
>>>> guess
>>>> in
>>>> my quest for my program to write once, run anywhere.
>>>>
>>>>> Pavel
>>>>>
>>>>> On Fri, Sep 18, 2009 at 3:26 AM, John <jhy...@earthlink.net>
>>>>> wrote:
>>>>>> Pavel Ivanov wrote:
>>>>>>>> At least I think that is what you suggest, and think it just
>>>>>>>> may work! But I could be wrong!
>>>>>>> Yes, that's exactly what I suggest.
>>>>>>>
>>>>>>> Pavel
>>>>>> It worked! Fortunately I had already parameterized SQLITE3 as a
>>>>>> preference variable so I could have the same scripts run easily
>>>>>> on Mac
>>>>>> OS and Windows. There are dozens of sqlite3 calls throughout
>>>>>> the
>>>>>> scripts.
>>>>>>
>>>>>> My whole set of scripts that process raw data and load the
>>>>>> database by
>>>>>> reading text files seem to work.
>>>>>>
>>>>>> cygwin is as slow as I recall, however. I was writing expense
>>>>>> scripts a
>>>>>> few years ago and abandoned it for MacOS Unix. I moved 100% to
>>>>>> Mac OS.
>>>>>> (except for this project which I want to work on Mac, linux,
>>>>>> and
>>>>>> Windows; my next goal is recoding it in Java with its Swing
>>>>>> GUI,
>>>>>> but I'm
>>>>>> just learning Java and Swing, but I'm on my way...).
>>>>>>
>>>>>> Observed elapsed times on my two notebook computers for the
>>>>>> same
>>>>>> scripts
>>>>>> to load the database (using sqlite3 calls and lots of sed and
>>>>>> awk
>>>>>> processing of thousands of lines of input data):
>>>>>>
>>>>>> MacBook Mac OS X 10.5.8
>>>>>> 2 GHz Intel Core Duo
>>>>>> 1 GB memory:
>>>>>> 17 minutes 46 seconds.
>>>>>>
>>>>>> IBM ThinkPad
>>>>>> Windows XP (latest patches)
>>>>>> 1.70 GHz, 512 MB memory:
>>>>>> 6 hours 25 minutes 57 seconds
>>>>>>
>>>>>> Fortunately, sqlite .dump and restoring from the resultant sql
>>>>>> will be
>>>>>> able to be used for most of the heavy lifting when I'm done.
>>>>>> Changes to
>>>>>> the data will come in small increments over time from then on.
>>>>>> My
>>>>>> dumpit
>>>>>> and restoreit scripts each take only seconds on both platforms
>>>>>> for the
>>>>>> full set of current data.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>>> On Thu, Sep 17, 2009 at 1:18 PM, John <jhy...@earthlink.net>
>>>>>>> wrote:
>>>>>>>> Pavel Ivanov wrote:
>>>>>>>>>> I'd rather avoid building sqlite3 under cygwin. I would
>>>>>>>>>> like
>>>>>>>>>> to keep as much as possible in native code, compromising
>>>>>>>>>> only
>>>>>>>>>> on cygwin to run my scripts.
>>>>>>>>> And this is root of your problem. Using mix of cygwin-native
>>>>>>>>> applications with windows-native applications will always
>>>>>>>>> have
>>>>>>>>> such
>>>>>>>>> problem.
>>>>>>>>>
>>>>>>>>>> When installing cygwin, you it offers you the choice to
>>>>>>>>>> switch
>>>>>>>>>> to default text file type to DOS (\r\n). Should I try that?
>>>>>>>>> Don't do that. This mode of operation is not supported much
>>>>>>>>> and not
>>>>>>>>> recommended by cygwin developers and it reportedly will
>>>>>>>>> significantly
>>>>>>>>> slow down cygwin's operation.
>>>>>>>>>
>>>>>>>>>> So I guess my question here is, do any sqlite users here
>>>>>>>>>> have experience fixing this on Windows for Unix cygwin
>>>>>>>>>> script calls?
>>>>>>>>> The major suggestion here: write some "windows native code
>>>>>>>>> launcher"
>>>>>>>>> that will be used for running all non-cygwin applications
>>>>>>>>> (this can be
>>>>>>>>> just function in the script). It will do nothing on unix
>>>>>>>>> platforms
>>>>>>>>> (select your own preferred way of distinguishing it) and it
>>>>>>>>> will
>>>>>>>>> always strip off '\r' from output of running application on
>>>>>>>>> windows
>>>>>>>>> (you can use sed for that). And there's nothing else you can
>>>>>>>>> do about
>>>>>>>>> it.
>>>>>>>> This sounds like a great idea. I can have all sqlite3.exe
>>>>>>>> calls
>>>>>>>> "intercepted" by another script call like:
>>>>>>>>
>>>>>>>> NumPar=`WINDOWSCALL Program Arguments`
>>>>>>>>
>>>>>>>> WINDOWSCALL is the launcher that calls Program sqlite3.exe
>>>>>>>> with its arguments and strips off any trailing \r's
>>>>>>>> and returns that string to the caller through stdout,
>>>>>>>> as to NumPar here. WINDOWSCALL can do nothing on Unix/MacOS,
>>>>>>>> and fix the string on Windows.
>>>>>>>>
>>>>>>>> At least I think that is what you suggest, and think it just
>>>>>>>> may work! But I could be wrong!
>>>>>>>>
>>>>>>>> Thanks! I'll try coding it.
>>>>>>>>
>>>>>>>>> Pavel
>>>>>>>>>
>>>>>>>>> On Thu, Sep 17, 2009 at 12:26 PM, John
>>>>>>>>> <jhy...@earthlink.net>
>>>>>>>>> wrote:
>>>>>>>>>> I am writing some Unix scripts on Mac OS X that use
>>>>>>>>>> sqlite3. Since the program could be useful to those
>>>>>>>>>> on Windows, I figured I'd see if they worked under
>>>>>>>>>> cygwin.
>>>>>>>>>>
>>>>>>>>>> A lot of it works, but calling sqlite3.exe from
>>>>>>>>>> cygwin and returning a string with the value
>>>>>>>>>> returned from the database seems to attach a
>>>>>>>>>> "\r" that expr doesn't remove.
>>>>>>>>>>
>>>>>>>>>> That is:
>>>>>>>>>>
>>>>>>>>>> NumPar=`sqlite3.exe ${DATABASE} "SELECT NumPar FROM
>>>>>>>>>> citations
>>>>>>>>>> WHERE
>>>>>>>>>> X='Key' ;"`
>>>>>>>>>>
>>>>>>>>>> NumPar comes back as: "12\r"
>>>>>>>>>>
>>>>>>>>>> and
>>>>>>>>>>
>>>>>>>>>> NumPar=`expr ${NumPar}`
>>>>>>>>>>
>>>>>>>>>> doesn't convert it to integer, as the subsequent test fails
>>>>>>>>>> because
>>>>>>>>>> of NumPar being non-integer (it isn't complaining about N,
>>>>>>>>>> that
>>>>>>>>>> is integer in the code):
>>>>>>>>>>
>>>>>>>>>> if [ ${N} -le ${NumPar} ]
>>>>>>>>>> ...
>>>>>>>>>>
>>>>>>>>>> I can fix this case by:
>>>>>>>>>>
>>>>>>>>>> NumPar=`printf '%s' "${NumPar}" | sed 's/[^0-9]//g'`
>>>>>>>>>>
>>>>>>>>>> but then other scripts fail later, presumably because of
>>>>>>>>>> strings with \r on them. (I suppose I can use sed to
>>>>>>>>>> always remove \r's on every one of these calls, but
>>>>>>>>>> that seems pretty kludgy, especially since "clean"
>>>>>>>>>> Mac OS X handles all this "properly" without that.
>>>>>>>>>> I'm hoping to find an elegant solution.
>>>>>>>>>>
>>>>>>>>>> I'd rather avoid building sqlite3 under cygwin. I would
>>>>>>>>>> like
>>>>>>>>>> to keep as much as possible in native code, compromising
>>>>>>>>>> only
>>>>>>>>>> on cygwin to run my scripts.
>>>>>>>>>>
>>>>>>>>>> When installing cygwin, you it offers you the choice to
>>>>>>>>>> switch
>>>>>>>>>> to default text file type to DOS (\r\n). Should I try that?
>>>>>>>>>> My pretty serious objection to that would be that any users
>>>>>>>>>> already using cygwin with the "correct" default settings
>>>>>>>>>> would
>>>>>>>>>> not be able to use the scripts anyway.
>>>>>>>>>>
>>>>>>>>>> So I guess my question here is, do any sqlite users here
>>>>>>>>>> have experience fixing this on Windows for Unix cygwin
>>>>>>>>>> script calls?
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>> _______________________________________________
>>>>>>>>>> sqlite-users mailing list
>>>>>>>>>> sqlite-users@sqlite.org
>>>>>>>>>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> sqlite-users mailing list
>>>>>>>>> sqlite-users@sqlite.org
>>>>>>>>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> sqlite-users mailing list
>>>>>>>> sqlite-users@sqlite.org
>>>>>>>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> sqlite-users mailing list
>>>>>>> sqlite-users@sqlite.org
>>>>>>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>>>>>>
>>>>>> _______________________________________________
>>>>>> sqlite-users mailing list
>>>>>> sqlite-users@sqlite.org
>>>>>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>>>>>
>>>>> _______________________________________________
>>>>> sqlite-users mailing list
>>>>> sqlite-users@sqlite.org
>>>>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>>>>
>>>> _______________________________________________
>>>> sqlite-users mailing list
>>>> sqlite-users@sqlite.org
>>>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>>>
>>> _______________________________________________
>>> sqlite-users mailing list
>>> sqlite-users@sqlite.org
>>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>
>> _______________________________________________
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
