How to detect beginning and end of output from a SQL Statement piped to a subprocess:
Initialize the subprocess with .header off .separator <whatever> .mode list And then delimit your queries - SELECT '<uniqueID> START'; - <your statment> - SELECT '<uniqueID> END'; Discarding everything not between the expected START and END lines -----Urspr?ngliche Nachricht----- Von: sqlite-users-bounces at mailinglists.sqlite.org [mailto:sqlite-users-bounces at mailinglists.sqlite.org] Im Auftrag von Roger Binns Gesendet: Samstag, 16. J?nner 2016 19:36 An: SQLite mailing list Betreff: Re: [sqlite] Using sqlite3.exe as a subprocess -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 15/01/16 13:16, Matthew Allen wrote: > Yeah in hindsight it wasn't the best. I was trying to do: > > while still running: p.communicate etc > > Poll is not the right method to call. It especially is the wrong method in your scenario since you want to dynamically read and write from the subprocess. You'll want stdin/out/err all to be subprocess.PIPE. You'll possibly get lucky with the SQLite shell, but it is a *very* good idea to do the reading of their stdout and writing to their stdin in different threads. The reason is that many of these tools have an loop that looks like this: repeat: - write prompt to stdout - read a command from stdin - write output to stdout, errors to stderr If the output is bigger than the pipe buffer size, then the "write output" step blocks until there is space in the pipe, and will do so before reading the next command. Remember that you don't know when the output is done - in theory you could try to detect the prompt and hope that something similar is not in the data, but that is brittle. If you look at the implementation of the communicate method, you'll see it addresses this issue by using multiple threads (~one per pipe of interest). Since you are using Python 2, another issue you need to be aware of is that the subprocess module is buggy when your python code is multi-threaded. This issue affects non-Windows systems (eg Linux and Mac) and you can end up with the wrong child process being accessed! It isn't theoretical - happened to me two weeks ago and was a pain to realise what had happened and debug. The good news is Google have a fix: https://github.com/google/python-subprocess32 But finally I'd recommend you just avoid all this subprocess stuff. SQLite will quite happily take commands from a script. Consequently you can create a script file like the following and have SQLite execute it (command line parameter): .bail on .open c:\path\to\your\db .output c:\...\tmp\dump.sql .dump .exit You can use the tempfile module to get a temporary filename. The script approach avoids many text vs binary, quoting, encoding, testing and other issues. Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlaajaQACgkQmOOfHg372QQ68QCeJuZzafMJMAju3F7twPbyToAM XOcAnic84HeEtyTIg/u0r7gbRHt/x/+o =Xyid -----END PGP SIGNATURE----- _______________________________________________ sqlite-users mailing list sqlite-users at mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ___________________________________________ Gunter Hick Software Engineer Scientific Games International GmbH FN 157284 a, HG Wien Klitschgasse 2-4, A-1130 Vienna, Austria Tel: +43 1 80100 0 E-Mail: hick at scigames.at This communication (including any attachments) is intended for the use of the intended recipient(s) only and may contain information that is confidential, privileged or legally protected. Any unauthorized use or dissemination of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender by return e-mail message and delete all copies of the original communication. Thank you for your cooperation.
