[Orgmode] Re: [babel] Sh problem when echo'ing input data with ' inside
Sébastien Vauban wxhgmqzgw...@spammotel.com writes: I asked the question on the Cygwin mailing list, and got an answer from Tobias Schlottke, telling me to use the 'plain old' syntax (=`...`= instead of =$(...)=): On further investigation, this is a bug in bash3, fixed in bash4 (hence why I didn't see it on Linux). Bash4 is now available as experimental in the latest version of Cygwin. It's a bit tricky to install since you will also need the experimental version of libreadline7 (package version 6.1) and the setup kind of works against you: I've not managed to install both of them in one go and you will need to be careful with updates since I believe setup will always try to reverse these to the non-experimental versions unless you allow them globally (which I don't want). In any case, the problem with the '$( )' construct is indeed fixed and the next Cygwin release will hopefully have this new bash as standard. Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ Factory and User Sound Singles for Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds ___ Emacs-orgmode mailing list Please use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
[Orgmode] Re: [babel] Sh problem when echo'ing input data with ' inside
Hi Achim, Achim Gratz wrote: Sébastien Vauban wxhgmqzgw...@spammotel.com writes: I asked the question on the Cygwin mailing list, and got an answer from Tobias Schlottke, telling me to use the 'plain old' syntax (=`...`= instead of =$(...)=): On further investigation, this is a bug in bash3, fixed in bash4 (hence why I didn't see it on Linux). Bash4 is now available as experimental in the latest version of Cygwin. It's a bit tricky to install since you will also need the experimental version of libreadline7 (package version 6.1) and the setup kind of works against you: I've not managed to install both of them in one go and you will need to be careful with updates since I believe setup will always try to reverse these to the non-experimental versions unless you allow them globally (which I don't want). In any case, the problem with the '$( )' construct is indeed fixed and the next Cygwin release will hopefully have this new bash as standard. Thanks a lot for the follow-up, and for the above details about the problem! Best regards, Seb -- Sébastien Vauban ___ Emacs-orgmode mailing list Please use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
[Orgmode] Re: [babel] Sh problem when echo'ing input data with ' inside
Hi Eric and Achim, Sébastien Vauban wrote: Achim Gratz wrote: The script produced by babel should actually look like this: #+begin_src sh data=$(cat 'BABEL_STRING' Num.ro du compte :;979-9500975-24;Compte Maxi Date valeur;R.f.rence de l'op.ration;Description;Montant de l'op.ration;Devise;Date d'op.ration;Compte de contrepartie;Nom de la contrepartie :;Communication 1 :;Communication 2 : 04-06-2009;A9F04NT01WK300TG;Virem. internet;420,00;EUR;04-06-2009;799-5900947-23;PAYEE ONE; ; 24-02-2009;A9B24NT012K4018Z;Virem. internet;-54,93;EUR;24-02-2009;799-9974005-30;ME;Eigen rekening; 18-05-2008;A8E19NT000S604QI;Virem. internet;-1.000,00;EUR;19-05-2008;799-8068445-18;PAYEE TWO; ; BABEL_STRING ) echo $data #+end_src That is no indentation and the here-tag should be in quotes (otherwise the here-script is subject to shell expansion). The above, from which I even removed the accentuated characters (from French), still does not work... Now, I can imagine it is due to Cygwin, not to Emacs/Org/Babel. Though, that's a problem in general for the use of sh blocks from Org... under Windows, then. - Has anyone any experience with this? - Can some Bash user confirm what I see? - Can some Windows user confirm this with other Unix-emulation shells (MinGW and the like)? I asked the question on the Cygwin mailing list, and got an answer from Tobias Schlottke, telling me to use the 'plain old' syntax (=`...`= instead of =$(...)=): #+begin_src sh data=`cat EOF ... EOF ` echo $data #+end_src and... it indeed works! Can we move to that syntax? Best regards, Seb -- Sébastien Vauban ___ Emacs-orgmode mailing list Please use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] Re: [babel] Sh problem when echo'ing input data with ' inside
I asked the question on the Cygwin mailing list, and got an answer from Tobias Schlottke, telling me to use the 'plain old' syntax (=`...`= instead of =$(...)=): #+begin_src sh data=`cat EOF ... EOF ` echo $data #+end_src and... it indeed works! Can we move to that syntax? I just made this customizable, after a git pull the following elisp code will use that syntax on your system. #+begin_src emacs-lisp (setq org-babel-sh-var-quote-fmt `cat 'BABEL_TABLE'\n%s\nBABEL_TABLE\n`) #+end_src Best -- Eric ___ Emacs-orgmode mailing list Please use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
[Orgmode] Re: [babel] Sh problem when echo'ing input data with ' inside
Hi Eric, Eric Schulte wrote: I asked the question on the Cygwin mailing list, and got an answer from Tobias Schlottke, telling me to use the 'plain old' syntax (=`...`= instead of =$(...)=): #+begin_src sh data=`cat EOF ... EOF ` echo $data #+end_src and... it indeed works! Can we move to that syntax? I just made this customizable, after a git pull the following elisp code will use that syntax on your system. #+begin_src emacs-lisp (setq org-babel-sh-var-quote-fmt `cat 'BABEL_TABLE'\n%s\nBABEL_TABLE\n`) #+end_src That's, indeed, better than what I had in mind, as it now is customizable... ... and, of course, that does the trick! Thanks a lot for your prompt fix. BTW, I would, though, advice to make this the default, as it should be fully functional under any Linux, and then would be directly usable by any user on Windows/Cygwin, without having to wonder why sh code blocks don't work the way they should. Best regards, Seb -- Sébastien Vauban ___ Emacs-orgmode mailing list Please use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
[Orgmode] Re: [babel] Sh problem when echo'ing input data with ' inside
Sébastien Vauban wxhgmqzgw...@spammotel.com writes: I asked the question on the Cygwin mailing list, and got an answer from Tobias Schlottke, telling me to use the 'plain old' syntax (=`...`= instead of =$(...)=): I can confirm. This is really bad, since now you'd have to escape backticks in the data coming from emacs. Do you have a link to the discussion, provided there is a web-mirror of the mailing list someplace? I'm a bit out of my waters here since mostly I'm using tcsh, but it appears that dash on cygwin (not ash, even though dash is just a symlink to ash) does the right thing. It is also vastly smaller than bash, so you might just use it for babel scripting. Can we move to that syntax? For the reason above this would not be safe without more changes to babel and backticks are actively discouraged to use in new scripts by just about any shell introduction/ FAQ for years now. Eric has implemented a workaround and the problem deserves to be documented, though. Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ SD adaptation for Waldorf microQ V2.22R2: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada ___ Emacs-orgmode mailing list Please use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
[Orgmode] Re: [babel] Sh problem when echo'ing input data with ' inside
Hi Eric, Eric Schulte wrote: I tried to work on a CSV2Ledger, accepting the particular (European) CSV format of my bank: * Europeanize the CSV file Here, we have to make several manipulations on the input file [...]. My goal is to make that as a suite of bash commands (=echo | cmd 1 | cmd 2=), tangled in a script file. However, I can't do any work on the input file, the very basic echo command already giving an error: #+begin_src stderr sh: line 3: unexpected EOF while looking for matching `'' sh: line 10: syntax error: unexpected end of file #+end_src In this sample, this is due to the apostrophe in the header line (Montant de l'opération). I can't reproduce this problem, on my system, I get the following correct behavior #+results: excel-from-bank #+begin_example Num.ro du compte :;979-9500975-24;Compte Maxi Date valeur;R.f.rence de l'op.ration;Description;Montant de l'op.ration;Devise;Date d'op.ration;Compte de contrepartie;Nom de la contrepartie :;Communication 1 :;Communication 2 : 04-06-2009;A9F04NT01WK300TG;Virem. internet;420,00;EUR;04-06-2009;799-5900947-23;PAYEE ONE; ; 24-02-2009;A9B24NT012K4018Z;Virem. internet;-54,93;EUR;24-02-2009;799-9974005-30;ME;Eigen rekening; 18-05-2008;A8E19NT000S604QI;Virem. internet;-1.000,00;EUR;19-05-2008;799-8068445-18;PAYEE TWO; ; #+end_example #+begin_src sh :var data=excel-from-bank :results output echo $data #+end_src #+results: : Num.ro du compte :;979-9500975-24;Compte Maxi : Date valeur;R.f.rence de l'op.ration;Description;Montant de l'op.ration;Devise;Date d'op.ration;Compte de contrepartie;Nom de la contrepartie :;Communication 1 :;Communication 2 : : 04-06-2009;A9F04NT01WK300TG;Virem. internet;420,00;EUR;04-06-2009;799-5900947-23;PAYEE ONE; ; : 24-02-2009;A9B24NT012K4018Z;Virem. internet;-54,93;EUR;24-02-2009;799-9974005-30;ME;Eigen rekening; : 18-05-2008;A8E19NT000S604QI;Virem. internet;-1.000,00;EUR;19-05-2008;799-8068445-18;PAYEE TWO; ; and the sh code block expands to the following (with C-c C-v v) which is not affected by commas... I've no trouble with commas, except that it's not an American formatting for amounts. Though, I does well have problems, as showed, with the apostrophe... Previewing does work for me as well: #+begin_src sh data=$(cat BABEL_STRING Num.ro du compte :;979-9500975-24;Compte Maxi Date valeur;R.f.rence de l'op.ration;Description;Montant de l'op.ration;Devise;Date d'op.ration;Compte de contrepartie;Nom de la contrepartie :;Communication 1 :;Communication 2 : 04-06-2009;A9F04NT01WK300TG;Virem. internet;420,00;EUR;04-06-2009;799-5900947-23;PAYEE ONE; ; 24-02-2009;A9B24NT012K4018Z;Virem. internet;-54,93;EUR;24-02-2009;799-9974005-30;ME;Eigen rekening; 18-05-2008;A8E19NT000S604QI;Virem. internet;-1.000,00;EUR;19-05-2008;799-8068445-18;PAYEE TWO; ; BABEL_STRING ) echo $data #+end_src Evaluation does not. Are you on a windows machine? I'm not sure what else could be different about the shell environment on our machines. Yes, it is on Windows ;-) Though, with Cygwin bash support... which should behave as on any Linux... note: some of the characters in your email weren't printable when I tried to send this with gnus, so they've been replaced with .s These are French accentuated letters. Any idea on how to: - circumvent such problem? - or, eventually, do this better? No idea on any workaround, then? ;-( Best regards, Seb -- Sébastien Vauban ___ Emacs-orgmode mailing list Please use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] Re: [babel] Sh problem when echo'ing input data with ' inside
and the sh code block expands to the following (with C-c C-v v) which is not affected by commas... I've no trouble with commas, except that it's not an American formatting for amounts. Though, I does well have problems, as showed, with the apostrophe... Sorry, I mis-typed, as far as I can tell the apostrophe works on my machine without causing any issues. Previewing does work for me as well: #+begin_src sh data=$(cat BABEL_STRING Num.ro du compte :;979-9500975-24;Compte Maxi Date valeur;R.f.rence de l'op.ration;Description;Montant de l'op.ration;Devise;Date d'op.ration;Compte de contrepartie;Nom de la contrepartie :;Communication 1 :;Communication 2 : 04-06-2009;A9F04NT01WK300TG;Virem. internet;420,00;EUR;04-06-2009;799-5900947-23;PAYEE ONE; ; 24-02-2009;A9B24NT012K4018Z;Virem. internet;-54,93;EUR;24-02-2009;799-9974005-30;ME;Eigen rekening; 18-05-2008;A8E19NT000S604QI;Virem. internet;-1.000,00;EUR;19-05-2008;799-8068445-18;PAYEE TWO; ; BABEL_STRING ) echo $data #+end_src if you execute the previewed code block as an independent shell script on your machine does it throw errors or does it echo correctly? Evaluation does not. Are you on a windows machine? I'm not sure what else could be different about the shell environment on our machines. Yes, it is on Windows ;-) Though, with Cygwin bash support... which should behave as on any Linux... note: some of the characters in your email weren't printable when I tried to send this with gnus, so they've been replaced with .s These are French accentuated letters. Any idea on how to: - circumvent such problem? - or, eventually, do this better? No idea on any workaround, then? ;-( I can't reproduce the problem locally, and I believe that this is due to differences in our shell environments rather than Emacs-specific issues, so it is hard for me to diagnose or propose a solution. Sorry I can't be of more help -- Eric Best regards, Seb ___ Emacs-orgmode mailing list Please use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
[Orgmode] Re: [babel] Sh problem when echo'ing input data with ' inside
Sébastien Vauban wxhgmqzgw...@spammotel.com writes: However, I can't do any work on the input file, the very basic echo command already giving an error: This is because quoting the arguments to echo does not work when your input contains unescaped quote characters. I've no idea how exactly babel tries to get it's arguments to the command, but it looks like it simply copies text and executes the resulting script? If it could open a pipe and send the data through, no quoting would be necessary since the shell never sees the data. I'm not sure if that's possible because I can't find specific documentation on language sh. [Note: orgmode online manual links to http://orgmode.org/worg/org-contrib/babel/languages/ which is broken.] #+begin_src sh :var data=excel-from-bank :results output echo $data #+end_src So, you're using a un*xoid? This is your lucky day, since in your case you should be able to use a here script: #+begin_src sh :var data=excel-from-bank :results output echo EOF | cmd1 | cmd2 $data EOF #+end_src HTH, Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ SD adaptations for KORG EX-800 and Poly-800MkII V0.9: http://Synth.Stromeko.net/Downloads.html#KorgSDada ___ Emacs-orgmode mailing list Please use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] Re: [babel] Sh problem when echo'ing input data with ' inside
Achim Gratz strom...@nexgo.de writes: Sébastien Vauban wxhgmqzgw...@spammotel.com writes: However, I can't do any work on the input file, the very basic echo command already giving an error: This is because quoting the arguments to echo does not work when your input contains unescaped quote characters. I've no idea how exactly babel tries to get it's arguments to the command, but it looks like it simply copies text and executes the resulting script? If it could open a pipe and send the data through, no quoting would be necessary since the shell never sees the data. I'm not sure if that's possible because I can't find specific documentation on language sh. [Note: orgmode online manual links to http://orgmode.org/worg/org-contrib/babel/languages/ which is broken.] yes, the Worg documentation is in dire need of maintenance, see below for discussion of how arguments are passed in the sh blocks #+begin_src sh :var data=excel-from-bank :results output echo $data #+end_src So, you're using a un*xoid? This is your lucky day, since in your case you should be able to use a here script: #+begin_src sh :var data=excel-from-bank :results output echo EOF | cmd1 | cmd2 $data EOF #+end_src here scripts are how Babel sends data through to a shell script, as far as I know they should have no issues with quotes (or any other character) specifically, the following code block #+results: something #+begin_example something and something else with apostrophes ' and quotes this should all work... #+end_example #+begin_src sh :var data=something echo $data #+end_src Expands to #+begin_src sh data=$(cat BABEL_STRING something and something else with apostrophes ' and quotes this should all work... BABEL_STRING ) echo $data #+end_src Cheers -- Eric ___ Emacs-orgmode mailing list Please use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
[Orgmode] Re: [babel] Sh problem when echo'ing input data with ' inside
Achim Gratz strom...@nexgo.de writes: #+begin_src sh :var data=excel-from-bank :results output echo EOF | cmd1 | cmd2 $data EOF #+end_src Sent too soon: use cat, not echo. Actually that is redundant, too: #+begin_src sh :var data=excel-from-bank :results output cmd1 EOF | cmd2 $data EOF #+end_src Or even simpler (but that might be a bashism...): #+begin_src sh :var data=excel-from-bank :results output cmd1 $data | cmd2 #+end_src And I missed the preview in Erics post, which gave me the clue I was looking for; the script produced by babel should actually look like this: #+begin_src sh data=$(cat 'BABEL_STRING' Num.ro du compte :;979-9500975-24;Compte Maxi Date valeur;R.f.rence de l'op.ration;Description;Montant de l'op.ration;Devise;Date d'op.ration;Compte de contrepartie;Nom de la contrepartie :;Communication 1 :;Communication 2 : 04-06-2009;A9F04NT01WK300TG;Virem. internet;420,00;EUR;04-06-2009;799-5900947-23;PAYEE ONE; ; 24-02-2009;A9B24NT012K4018Z;Virem. internet;-54,93;EUR;24-02-2009;799-9974005-30;ME;Eigen rekening; 18-05-2008;A8E19NT000S604QI;Virem. internet;-1.000,00;EUR;19-05-2008;799-8068445-18;PAYEE TWO; ; BABEL_STRING ) echo $data #+end_src That is no indentation and the here-tag should be in quotes (otherwise the here-script is subject to shell expansion). It's probably possible to do a contrived redirection and not use a variable to store the data, but I'm afraid I won't grok this today. Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ Wavetables for the Terratec KOMPLEXER: http://Synth.Stromeko.net/Downloads.html#KomplexerWaves ___ Emacs-orgmode mailing list Please use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
[Orgmode] Re: [babel] Sh problem when echo'ing input data with ' inside
Hi Achim and Eric, Achim Gratz wrote: The script produced by babel should actually look like this: #+begin_src sh data=$(cat 'BABEL_STRING' Num.ro du compte :;979-9500975-24;Compte Maxi Date valeur;R.f.rence de l'op.ration;Description;Montant de l'op.ration;Devise;Date d'op.ration;Compte de contrepartie;Nom de la contrepartie :;Communication 1 :;Communication 2 : 04-06-2009;A9F04NT01WK300TG;Virem. internet;420,00;EUR;04-06-2009;799-5900947-23;PAYEE ONE; ; 24-02-2009;A9B24NT012K4018Z;Virem. internet;-54,93;EUR;24-02-2009;799-9974005-30;ME;Eigen rekening; 18-05-2008;A8E19NT000S604QI;Virem. internet;-1.000,00;EUR;19-05-2008;799-8068445-18;PAYEE TWO; ; BABEL_STRING ) echo $data #+end_src That is no indentation and the here-tag should be in quotes (otherwise the here-script is subject to shell expansion). The above, from which I even removed the accentuated characters (from French), still does not work... Now, I can imagine it is due to Cygwin, not to Emacs/Org/Babel. Though, that's a problem in general for the use of sh blocks from Org... under Windows, then. - Has anyone any experience with this? - Can some Bash user confirm what I see? - Can some Windows user confirm this with other Unix-emulation shells (MinGW and the like)? It's probably possible to do a contrived redirection and not use a variable to store the data, but I'm afraid I won't grok this today. Or maybe using a real file as temporary storage? Overkill, or simply needed? Best regards, Seb -- Sébastien Vauban ___ Emacs-orgmode mailing list Please use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
[Orgmode] Re: [babel] Sh problem when echo'ing input data with ' inside
Sébastien Vauban wxhgmqzgw...@spammotel.com writes: The above, from which I even removed the accentuated characters (from French), still does not work... That's strange. Now, I can imagine it is due to Cygwin, not to Emacs/Org/Babel. Though, that's a problem in general for the use of sh blocks from Org... under Windows, then. I can try tomorrow, I won't start the Windows box today... Or maybe using a real file as temporary storage? Overkill, or simply needed? Temporary files should work from emacs, but on Windows it is somewhat troublesome to reliably get rid of them when they're no longer needed. Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ Factory and User Sound Singles for Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds ___ Emacs-orgmode mailing list Please use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
[Orgmode] Re: [babel] Sh problem when echo'ing input data with ' inside
Eric, Eric Schulte wrote: if you execute the previewed code block as an independent shell script on your machine does it throw errors or does it echo correctly? On this one: the behavior seen from Org Babel is the same as the one seen when saving the previewed script as an executable file and launching it from Bash. No diff. Best regards, Seb -- Sébastien Vauban ___ Emacs-orgmode mailing list Please use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
[Orgmode] Re: [babel] Sh problem when echo'ing input data with ' inside
Hi Achim, Achim Gratz wrote: Sébastien Vauban wxhgmqzgw...@spammotel.com writes: The above, from which I even removed the accentuated characters (from French), still does not work... That's strange. Now, I can imagine it is due to Cygwin, not to Emacs/Org/Babel. Though, that's a problem in general for the use of sh blocks from Org... under Windows, then. I can try tomorrow, I won't start the Windows box today... Thanks for trying. Best regards, Seb -- Sébastien Vauban ___ Emacs-orgmode mailing list Please use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
