Re: [O] org-mode and python pandas
Using the tabulate python module it is possible to have the following inline workaround: 8888888 #+BEGIN_SRC python :results raw import pandas as pd import numpy as np from tabulate import tabulate df = pd.DataFrame(np.random.random((4,3)), columns=['A','B','C']) print(foo) return(tabulate(df, headers=keys, tablefmt=orgtbl)) #+END_SRC #+RESULTS: | |A |B |C | |---+--+--+--| | 0 | 0.754799 | 0.492722 | 0.144595 | | 1 | 0.198475 | 0.417721 | 0.083459 | | 2 | 0.645011 | 0.444857 | 0.278874 | | 3 | 0.314883 | 0.7521 | 0.789418 | 8888888 However, this is not optimal, mainly because it pollutes the code block. Note, that the :results is set to `raw`. A brief discussion with Dov, yielded the following better workaround: 8888 888 (setq org-babel-python-wrapper-method def main(): %s res = main() if 'pandas.core.frame.DataFrame' in str(type(res)): from tabulate import tabulate out = tabulate(res, headers='keys', tablefmt='orgtbl') else: out = str(res) open('%s', 'w').write(out)) 8888 888 This allows a nice output of pandas.DataFrame (again when using :results raw). Unfortunately, this wrapper has no influence in the case when :session is used. To that end, it is possible to hack 8888 888 (defun org-babel-python-evaluate-session (session body optional result-type result-params) Pass BODY to the Python process in SESSION. If RESULT-TYPE equals 'output then return standard output as a string. If RESULT-TYPE equals 'value then return the value of the last statement in BODY, as elisp. (let* ((send-wait (lambda () (comint-send-input nil t) (sleep-for 0 5))) (dump-last-value (lambda (tmp-file pp) (mapc (lambda (statement) (insert statement) (funcall send-wait)) (if pp (list import pprint (format open('%s', 'w').write(pprint.pformat(_)) (org-babel-process-file-name tmp-file 'noquote))) (list (format _org_tmp = _; if 'pandas.core.frame.DataFrame' in str(type(_org_tmp)): from tabulate import tabulate _org_out = tabulate(_org_tmp, headers='keys', tablefmt='orgtbl') else: _org_out = _org_tmp open('%s', 'w').write(str(_org_out)) (org-babel-process-file-name tmp-file 'noquote))) (input-body (lambda (body) (mapc (lambda (line) (insert line) (funcall send-wait)) (split-string body [\r\n])) (funcall send-wait))) (results (case result-type (output (mapconcat #'org-babel-trim (butlast (org-babel-comint-with-output (session org-babel-python-eoe-indicator t body) (funcall input-body body) (funcall send-wait) (funcall send-wait) (insert org-babel-python-eoe-indicator) (funcall send-wait)) 2) \n)) (value (let ((tmp-file (org-babel-temp-file python-))) (org-babel-comint-with-output (session org-babel-python-eoe-indicator nil body) (let ((comint-process-echoes nil)) (funcall input-body body) (funcall dump-last-value tmp-file (member pp result-params)) (funcall send-wait) (funcall send-wait) (insert org-babel-python-eoe-indicator) (funcall send-wait))) (org-babel-eval-read-file tmp-file)) (unless (string= (substring org-babel-python-eoe-indicator 1 -1) results) (org-babel-result-cond result-params results (org-babel-python-table-or-string results) 8888 888 This works, but I would be surprised if this hack meets org-mode's standards. Nevertheless, maybe someone would find it useful. What do you think? How can it be improved? Best, Dror
Re: [O] org-mode and python pandas
I returned to this issue recently and tried to get the ob-python to output a table with a header, but didn't manage. Here is the code: #+BEGIN_SRC python :colnames true return [['','A','B','C'], [0,0.628365,0.424279,0.619791], [1,0.799666,0.527572,0.132928]] #+END_SRC #+RESULTS: | |A |B |C | | 0 | 0.628365 | 0.424279 | 0.619791 | | 1 | 0.799666 | 0.527572 | 0.132928 | What I want is: | |A |B |C | |---+--+--+--| | 0 | 0.628365 | 0.424279 | 0.619791 | | 1 | 0.799666 | 0.527572 | 0.132928 | Is there any way to do that besides using the :results raw option? Thanks! Dov On Wed, Jul 3, 2013 at 5:09 PM, Eric Schulte schulte.e...@gmail.com wrote: Dov Grobgeld dov.grobg...@gmail.com writes: Thanks for the answers, but there is still something missing in order to get it to work. Part of it seems to be connected to the python parsing. E.g. the following translation of Eric's sh example doesn't output correctly with python: #+BEGIN_SRC python :results output print ,A,B,C 0,0.628365,0.424279,0.619791 1,0.799666,0.527572,0.132928 2,0.837255,0.138906,0.408233 3,0.388080,0.146212,0.575346 #+END_SRC #+RESULTS: : ,A,B,C : 0,0.628365,0.424279,0.619791 : 1,0.799666,0.527572,0.132928 : 2,0.837255,0.138906,0.408233 : 3,0.388080,0.146212,0.575346 #+BEGIN_SRC python :results table return ,A,B,C 0,0.628365,0.424279,0.619791 1,0.799666,0.527572,0.132928 2,0.837255,0.138906,0.408233 3,0.388080,0.146212,0.575346 #+END_SRC #+RESULTS: | ,A,B,C\n\n0,0.628365,0.424279,0.619791\n\n1,0.799666,0.527572,0.132928\n\n2,0.837255,0.138906,0.408233\n\n3,0.388080,0.146212,0.575346 | It seems that the only way to get a table from python is by outputting a two dimensional python structure: #+BEGIN_SRC python return [[0,0.628365,0.424279,0.619791], [1,0.799666,0.527572,0.132928]] #+END_SRC #+RESULTS: | 0 | 0.628365 | 0.424279 | 0.619791 | | 1 | 0.799666 | 0.527572 | 0.132928 | This seems quite limiting In most cases this is what one wants when returning data from python code. The following elisp defined a panda code block, which is just like python, only it assumes that the results will be these sort of human readable strings instead of python code. ;; -*- emacs-lisp -*- (defun org-babel-execute:panda (body params) (let ((results (org-babel-execute:python body (org-babel-merge-params '((:results . scalar)) params (org-babel-result-cond (cdr (assoc :result-params params)) results (let ((tmp-file (org-babel-temp-file sh-))) (with-temp-file tmp-file (insert results)) (org-babel-import-elisp-from-file tmp-file) With the above evaluated the following works #+BEGIN_SRC panda return ,A,B,C 0,0.628365,0.424279,0.619791 1,0.799666,0.527572,0.132928 2,0.837255,0.138906,0.408233 3,0.388080,0.146212,0.575346 #+END_SRC #+RESULTS: | |A |B |C | | 0 | 0.628365 | 0.424279 | 0.619791 | | 1 | 0.799666 | 0.527572 | 0.132928 | | 2 | 0.837255 | 0.138906 | 0.408233 | | 3 | 0.38808 | 0.146212 | 0.575346 | Another related question is if there is any support for header tables? I.e. instead of this: | |A |B |C | | 0 | 0.827817 | 0.664009 | 0.089161 | | 1 | 0.170031 | 0.729214 | 0.110918 | | 2 | 0.575918 | 0.863924 | 0.757536 | | 3 | 0.682722 | 0.774445 | 0.992041 | I want this: | |A |B |C | |---+--+--+--| | 0 | 0.827817 | 0.664009 | 0.089161 | | 1 | 0.170031 | 0.729214 | 0.110918 | | 2 | 0.575918 | 0.863924 | 0.757536 | | 3 | 0.682722 | 0.774445 | 0.992041 | I guess that if I start playing around with the python ob module, it should be possible to get this working? See the :colnames header argument in the manual. Best, Regards, Dov On Mon, Jul 1, 2013 at 8:04 PM, Rasmus ras...@gmx.us wrote: Achim Gratz strom...@nexgo.de writes: 2. Add to pandas the option of globally influencing the text formatting so that it outputs something more parsable by org-mode. This sounds promising, if pandas support csv output that will be correctly parsed by Org-mode. The package already has CSV export, so one could use that. I don't know if you could echo the result directly to the output, all examples revolve around putting the CSV into a file. For Org, TSV output would be more natural. Something like: from pandas import DataFrame from numpy.random import rand from sys import stdout df = DataFrame(rand(10,3), columns = list('abc')) df df.to_csv(stdout, sep=\t, header = True, cols=(1,2)) I was completely unable to get ob-python working this morning, so I haven't tested it. I'm
Re: [O] org-mode and python pandas
Thanks for the answers, but there is still something missing in order to get it to work. Part of it seems to be connected to the python parsing. E.g. the following translation of Eric's sh example doesn't output correctly with python: #+BEGIN_SRC python :results output print ,A,B,C 0,0.628365,0.424279,0.619791 1,0.799666,0.527572,0.132928 2,0.837255,0.138906,0.408233 3,0.388080,0.146212,0.575346 #+END_SRC #+RESULTS: : ,A,B,C : 0,0.628365,0.424279,0.619791 : 1,0.799666,0.527572,0.132928 : 2,0.837255,0.138906,0.408233 : 3,0.388080,0.146212,0.575346 #+BEGIN_SRC python :results table return ,A,B,C 0,0.628365,0.424279,0.619791 1,0.799666,0.527572,0.132928 2,0.837255,0.138906,0.408233 3,0.388080,0.146212,0.575346 #+END_SRC #+RESULTS: | ,A,B,C\n\n0,0.628365,0.424279,0.619791\n\n1,0.799666,0.527572,0.132928\n\n2,0.837255,0.138906,0.408233\n\n3,0.388080,0.146212,0.575346 | It seems that the only way to get a table from python is by outputting a two dimensional python structure: #+BEGIN_SRC python return [[0,0.628365,0.424279,0.619791], [1,0.799666,0.527572,0.132928]] #+END_SRC #+RESULTS: | 0 | 0.628365 | 0.424279 | 0.619791 | | 1 | 0.799666 | 0.527572 | 0.132928 | This seems quite limiting Another related question is if there is any support for header tables? I.e. instead of this: | |A |B |C | | 0 | 0.827817 | 0.664009 | 0.089161 | | 1 | 0.170031 | 0.729214 | 0.110918 | | 2 | 0.575918 | 0.863924 | 0.757536 | | 3 | 0.682722 | 0.774445 | 0.992041 | I want this: | |A |B |C | |---+--+--+--| | 0 | 0.827817 | 0.664009 | 0.089161 | | 1 | 0.170031 | 0.729214 | 0.110918 | | 2 | 0.575918 | 0.863924 | 0.757536 | | 3 | 0.682722 | 0.774445 | 0.992041 | I guess that if I start playing around with the python ob module, it should be possible to get this working? Regards, Dov On Mon, Jul 1, 2013 at 8:04 PM, Rasmus ras...@gmx.us wrote: Achim Gratz strom...@nexgo.de writes: 2. Add to pandas the option of globally influencing the text formatting so that it outputs something more parsable by org-mode. This sounds promising, if pandas support csv output that will be correctly parsed by Org-mode. The package already has CSV export, so one could use that. I don't know if you could echo the result directly to the output, all examples revolve around putting the CSV into a file. For Org, TSV output would be more natural. Something like: from pandas import DataFrame from numpy.random import rand from sys import stdout df = DataFrame(rand(10,3), columns = list('abc')) df df.to_csv(stdout, sep=\t, header = True, cols=(1,2)) I was completely unable to get ob-python working this morning, so I haven't tested it. I'm using python3, build in python mode and elpy. In any case, the csv route might be better, as Pandas doesn't print the table if it's too big (try changing 10 to 1000 above). -- Powered by magic pixies!
Re: [O] org-mode and python pandas
Hi Dov, Another related question is if there is any support for header tables? I.e. instead of this: | |A |B |C | | 0 | 0.827817 | 0.664009 | 0.089161 | | 1 | 0.170031 | 0.729214 | 0.110918 | | 2 | 0.575918 | 0.863924 | 0.757536 | | 3 | 0.682722 | 0.774445 | 0.992041 | I want this: | |A |B |C | |---+--+--+--| | 0 | 0.827817 | 0.664009 | 0.089161 | | 1 | 0.170031 | 0.729214 | 0.110918 | | 2 | 0.575918 | 0.863924 | 0.757536 | | 3 | 0.682722 | 0.774445 | 0.992041 | This is an issue in R as well, e.g. often I might want to have |A| |-| |1| |2| |-| |3| where the last is a summary statistic. The last hline can be gotten as #+BEGIN_SRC emacs-lisp :var X=mytabel :exports results (let ((L (- (length X) 1))) (append (subseq X 0 L) (list 'hline) (subseq X L))) #+END_SRC Perhaps a better way exists? –Rasmus -- . . . It begins of course with The Internet. A Net of Peers.
Re: [O] org-mode and python pandas
Dov Grobgeld dov.grobg...@gmail.com writes: Thanks for the answers, but there is still something missing in order to get it to work. Part of it seems to be connected to the python parsing. E.g. the following translation of Eric's sh example doesn't output correctly with python: #+BEGIN_SRC python :results output print ,A,B,C 0,0.628365,0.424279,0.619791 1,0.799666,0.527572,0.132928 2,0.837255,0.138906,0.408233 3,0.388080,0.146212,0.575346 #+END_SRC #+RESULTS: : ,A,B,C : 0,0.628365,0.424279,0.619791 : 1,0.799666,0.527572,0.132928 : 2,0.837255,0.138906,0.408233 : 3,0.388080,0.146212,0.575346 #+BEGIN_SRC python :results table return ,A,B,C 0,0.628365,0.424279,0.619791 1,0.799666,0.527572,0.132928 2,0.837255,0.138906,0.408233 3,0.388080,0.146212,0.575346 #+END_SRC #+RESULTS: | ,A,B,C\n\n0,0.628365,0.424279,0.619791\n\n1,0.799666,0.527572,0.132928\n\n2,0.837255,0.138906,0.408233\n\n3,0.388080,0.146212,0.575346 | It seems that the only way to get a table from python is by outputting a two dimensional python structure: #+BEGIN_SRC python return [[0,0.628365,0.424279,0.619791], [1,0.799666,0.527572,0.132928]] #+END_SRC #+RESULTS: | 0 | 0.628365 | 0.424279 | 0.619791 | | 1 | 0.799666 | 0.527572 | 0.132928 | This seems quite limiting In most cases this is what one wants when returning data from python code. The following elisp defined a panda code block, which is just like python, only it assumes that the results will be these sort of human readable strings instead of python code. ;; -*- emacs-lisp -*- (defun org-babel-execute:panda (body params) (let ((results (org-babel-execute:python body (org-babel-merge-params '((:results . scalar)) params (org-babel-result-cond (cdr (assoc :result-params params)) results (let ((tmp-file (org-babel-temp-file sh-))) (with-temp-file tmp-file (insert results)) (org-babel-import-elisp-from-file tmp-file) With the above evaluated the following works #+BEGIN_SRC panda return ,A,B,C 0,0.628365,0.424279,0.619791 1,0.799666,0.527572,0.132928 2,0.837255,0.138906,0.408233 3,0.388080,0.146212,0.575346 #+END_SRC #+RESULTS: | |A |B |C | | 0 | 0.628365 | 0.424279 | 0.619791 | | 1 | 0.799666 | 0.527572 | 0.132928 | | 2 | 0.837255 | 0.138906 | 0.408233 | | 3 | 0.38808 | 0.146212 | 0.575346 | Another related question is if there is any support for header tables? I.e. instead of this: | |A |B |C | | 0 | 0.827817 | 0.664009 | 0.089161 | | 1 | 0.170031 | 0.729214 | 0.110918 | | 2 | 0.575918 | 0.863924 | 0.757536 | | 3 | 0.682722 | 0.774445 | 0.992041 | I want this: | |A |B |C | |---+--+--+--| | 0 | 0.827817 | 0.664009 | 0.089161 | | 1 | 0.170031 | 0.729214 | 0.110918 | | 2 | 0.575918 | 0.863924 | 0.757536 | | 3 | 0.682722 | 0.774445 | 0.992041 | I guess that if I start playing around with the python ob module, it should be possible to get this working? See the :colnames header argument in the manual. Best, Regards, Dov On Mon, Jul 1, 2013 at 8:04 PM, Rasmus ras...@gmx.us wrote: Achim Gratz strom...@nexgo.de writes: 2. Add to pandas the option of globally influencing the text formatting so that it outputs something more parsable by org-mode. This sounds promising, if pandas support csv output that will be correctly parsed by Org-mode. The package already has CSV export, so one could use that. I don't know if you could echo the result directly to the output, all examples revolve around putting the CSV into a file. For Org, TSV output would be more natural. Something like: from pandas import DataFrame from numpy.random import rand from sys import stdout df = DataFrame(rand(10,3), columns = list('abc')) df df.to_csv(stdout, sep=\t, header = True, cols=(1,2)) I was completely unable to get ob-python working this morning, so I haven't tested it. I'm using python3, build in python mode and elpy. In any case, the csv route might be better, as Pandas doesn't print the table if it's too big (try changing 10 to 1000 above). -- Powered by magic pixies! -- Eric Schulte http://cs.unm.edu/~eschulte
Re: [O] org-mode and python pandas
Eric Schulte writes: | |A |B |C | |---+--+--+--| | 0 | 0.827817 | 0.664009 | 0.089161 | | 1 | 0.170031 | 0.729214 | 0.110918 | | 2 | 0.575918 | 0.863924 | 0.757536 | | 3 | 0.682722 | 0.774445 | 0.992041 | What happens if you add :results table to your code block? Would that be sufficient? The problem is that he's trying to consume human-readable output. The alignment in this output is done with spaces, not tabs (as Org would expect). There are more problems considering that the numbers likely lose precision in this format (nota bene: there are other Babel languages that have that same problem). The default value should be to convert multi-line output to tables, the :results table option above will force this conversion in case it is currently not taking place due to the default header arguments in use. Since there are no tabs in there, it is impossible to skip empty cells like the first one (they'd have to be written as instead, as would be cells that should contain literal spaces). 2. Add to pandas the option of globally influencing the text formatting so that it outputs something more parsable by org-mode. This sounds promising, if pandas support csv output that will be correctly parsed by Org-mode. The package already has CSV export, so one could use that. I don't know if you could echo the result directly to the output, all examples revolve around putting the CSV into a file. For Org, TSV output would be more natural. Regards, Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ Waldorf MIDI Implementation additional documentation: http://Synth.Stromeko.net/Downloads.html#WaldorfDocs
Re: [O] org-mode and python pandas
Achim Gratz strom...@nexgo.de writes: 2. Add to pandas the option of globally influencing the text formatting so that it outputs something more parsable by org-mode. This sounds promising, if pandas support csv output that will be correctly parsed by Org-mode. The package already has CSV export, so one could use that. I don't know if you could echo the result directly to the output, all examples revolve around putting the CSV into a file. For Org, TSV output would be more natural. Something like: from pandas import DataFrame from numpy.random import rand from sys import stdout df = DataFrame(rand(10,3), columns = list('abc')) df df.to_csv(stdout, sep=\t, header = True, cols=(1,2)) I was completely unable to get ob-python working this morning, so I haven't tested it. I'm using python3, build in python mode and elpy. In any case, the csv route might be better, as Pandas doesn't print the table if it's too big (try changing 10 to 1000 above). -- Powered by magic pixies!
Re: [O] org-mode and python pandas
Dov Grobgeld dov.grobg...@gmail.com writes: Has anyone used org-mode with the python pandas package? Pandas is in a certain way an alternative to R, but with the (for me) familiar syntax of python. See: http://pandas.pydata.org/ Pandas is very much built to be used interactively, and it outputs its data in space separated tabular format. E.g. in ipython: In [1]: import pandas as pd In [2]: import numpy as np In [3]: pd.DataFrame(np.random.random((4,3)), columns=['A','B','C']) Out[3]: A B C 0 0.628365 0.424279 0.619791 1 0.799666 0.527572 0.132928 2 0.837255 0.138906 0.408233 3 0.388080 0.146212 0.575346 Unfortunately this doesn't output as nicely when used from org-mode: #+BEGIN_SRC python import pandas as pd import numpy as np return pd.DataFrame(np.random.random((4,3)), columns=list('ABC')) #+END_SRC #+RESULTS: : A B C : 0 0.827817 0.664009 0.089161 : 1 0.170031 0.729214 0.110918 : 2 0.575918 0.863924 0.757536 : 3 0.682722 0.774445 0.992041 while I would like to have: | |A |B |C | |---+--+--+--| | 0 | 0.827817 | 0.664009 | 0.089161 | | 1 | 0.170031 | 0.729214 | 0.110918 | | 2 | 0.575918 | 0.863924 | 0.757536 | | 3 | 0.682722 | 0.774445 | 0.992041 | What happens if you add :results table to your code block? Would that be sufficient? The question is how to get this? Here are a few ideas: 1. Write a general filter in the org-mode elisp than uses heuristics to recognize ascii aligned tables and change these to org-tables. The default value should be to convert multi-line output to tables, the :results table option above will force this conversion in case it is currently not taking place due to the default header arguments in use. 2. Add to pandas the option of globally influencing the text formatting so that it outputs something more parsable by org-mode. This sounds promising, if pandas support csv output that will be correctly parsed by Org-mode. 3. Create a special language pandas that recognize the ascii aligned tables and saves the need to import pandas and np? 4. And the obvious approach of writing a python function that writes a org-mode parsable table and always call it as part of the return. Which is the preferable approach? Any other ideas? I think a header-argument-based approach would be ideal, I'd look at the value of org-babel-default-header-args:python, and read the portion of the manual related to the results header arguments. I don't understand multi-line strings in python, but I get the following behavior from simple shell script blocks. #+begin_src sh cat EOF A B C 0 0.628365 0.424279 0.619791 1 0.799666 0.527572 0.132928 2 0.837255 0.138906 0.408233 3 0.388080 0.146212 0.575346 EOF #+end_src #+RESULTS: | A |B |C | | | 0 | 0.628365 | 0.424279 | 0.619791 | | 1 | 0.799666 | 0.527572 | 0.132928 | | 2 | 0.837255 | 0.138906 | 0.408233 | | 3 | 0.38808 | 0.146212 | 0.575346 | #+begin_src sh cat EOF ,A,B,C 0,0.628365,0.424279,0.619791 1,0.799666,0.527572,0.132928 2,0.837255,0.138906,0.408233 3,0.388080,0.146212,0.575346 EOF #+end_src #+RESULTS: | |A |B |C | | 0 | 0.628365 | 0.424279 | 0.619791 | | 1 | 0.799666 | 0.527572 | 0.132928 | | 2 | 0.837255 | 0.138906 | 0.408233 | | 3 | 0.38808 | 0.146212 | 0.575346 | Hope this helps, Regards, Dov -- Eric Schulte http://cs.unm.edu/~eschulte
[O] org-mode and python pandas
Has anyone used org-mode with the python pandas package? Pandas is in a certain way an alternative to R, but with the (for me) familiar syntax of python. See: http://pandas.pydata.org/ Pandas is very much built to be used interactively, and it outputs its data in space separated tabular format. E.g. in ipython: In [1]: import pandas as pd In [2]: import numpy as np In [3]: pd.DataFrame(np.random.random((4,3)), columns=['A','B','C']) Out[3]: A B C 0 0.628365 0.424279 0.619791 1 0.799666 0.527572 0.132928 2 0.837255 0.138906 0.408233 3 0.388080 0.146212 0.575346 Unfortunately this doesn't output as nicely when used from org-mode: #+BEGIN_SRC python import pandas as pd import numpy as np return pd.DataFrame(np.random.random((4,3)), columns=list('ABC')) #+END_SRC #+RESULTS: : A B C : 0 0.827817 0.664009 0.089161 : 1 0.170031 0.729214 0.110918 : 2 0.575918 0.863924 0.757536 : 3 0.682722 0.774445 0.992041 while I would like to have: | |A |B |C | |---+--+--+--| | 0 | 0.827817 | 0.664009 | 0.089161 | | 1 | 0.170031 | 0.729214 | 0.110918 | | 2 | 0.575918 | 0.863924 | 0.757536 | | 3 | 0.682722 | 0.774445 | 0.992041 | The question is how to get this? Here are a few ideas: 1. Write a general filter in the org-mode elisp than uses heuristics to recognize ascii aligned tables and change these to org-tables. 2. Add to pandas the option of globally influencing the text formatting so that it outputs something more parsable by org-mode. 3. Create a special language pandas that recognize the ascii aligned tables and saves the need to import pandas and np? 4. And the obvious approach of writing a python function that writes a org-mode parsable table and always call it as part of the return. Which is the preferable approach? Any other ideas? Regards, Dov