Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
Aaron Ecay aarone...@gmail.com writes: 2012ko azaroak 17an, Eric Schulte-ek idatzi zuen: Oh!, thanks for catching this, I just pushed up a fix. This is no longer a (complete) fix for the original problem, though. (A large part of) the slowdown comes from reading the results from a temp file and transforming them into an elispy format, which is handled by the backends. Your code only prevents them from being echoed to the minibuffer. You can see this by trying your original “seq” tests. You should see them taking the same (very long) amount of time. If you set debug-on-quit to t (before evaluating the block) and interrupt emacs with C-g when it hangs, you’ll see a backtrace that goes through ‘org-babel-execute:sh’ and then ‘org-babel-import-elisp-from-file’. The presence of the former in the call stack is why I claim that per-backend fixes are ultimately needed. The latter is where my patch addresses the problem in a temporary way. I apologize for rushing through your previous email and missing important portions of the content. I may be outvoted, but I find this approach too be overly complicated. Also, size may frequently not be the best proxy for the time which Emacs will take to ingest the results. I agree that my patch is a stopgap. But until per-backend fixes are available, this allows certain code to run that otherwise wouldn’t (at least not without hacks, such as putting NULL at the end of an R source block so that the “.Last.value” in the block is trivial). I'm attaching a new patch which should serve as a more permanent solution, however, given that it touches almost every backend I want to give the list a chance to try it out before committing it. This patch introduces a new macro (`org-babel-result-cond') which attempts to unify the results processing. Not only does this check for :results none and ignore all results in that case, but it also checks whether the results should be parsed as a scalar or a vector. Hopefully this will serve as useful refactoring and reduce the total amount of code as well as provide for more uniform results processing across languages. Please let me know if this looks like a good permanent solution. I see that I'm now getting 5 test failures with this patch applied locally, but I won't have time to look at this further until early next week at the earliest, so I'm sharing the existing patch now. Thanks, From a2878534832923e42aef49176b87d9993c04af71 Mon Sep 17 00:00:00 2001 From: Eric Schulte eric.schu...@gmx.com Date: Sun, 18 Nov 2012 18:02:09 -0700 Subject: [PATCH] org-babel-result-cond - unified handling of results * lisp/ob.el (org-babel-result-cond): This function should now be used by all language backends to handle the processing of raw code block output into scalar results, table results, or ignored results depending on the value of result-params. * lisp/ob-C.el (org-babel-C-execute): Use org-babel-result-cond. * lisp/ob-R.el (org-babel-R-evaluate-external-process): Use org-babel-result-cond. (org-babel-R-evaluate-session): Use org-babel-result-cond. * lisp/ob-awk.el (org-babel-execute:awk): Use org-babel-result-cond. * lisp/ob-clojure.el (org-babel-execute:clojure): Use org-babel-result-cond. * lisp/ob-emacs-lisp.el (org-babel-execute:emacs-lisp): Use org-babel-result-cond. * lisp/ob-fortran.el (org-babel-execute:fortran): Use org-babel-result-cond. * lisp/ob-io.el (org-babel-io-evaluate): Use org-babel-result-cond. * lisp/ob-java.el (org-babel-execute:java): Use org-babel-result-cond. * lisp/ob-lisp.el (org-babel-execute:lisp): Use org-babel-result-cond. * lisp/ob-maxima.el (org-babel-execute:maxima): Use org-babel-result-cond. * lisp/ob-picolisp.el (org-babel-execute:picolisp): Use org-babel-result-cond. * lisp/ob-python.el (org-babel-python-evaluate-external-process): Use org-babel-result-cond. (org-babel-python-evaluate-session): Use org-babel-result-cond. * lisp/ob-scala.el (org-babel-scala-evaluate): Use org-babel-result-cond. * lisp/ob-sh.el (org-babel-sh-evaluate): Use org-babel-result-cond. * lisp/ob-shen.el (org-babel-execute:shen): Use org-babel-result-cond. * lisp/ob-sql.el (org-babel-execute:sql): Use org-babel-result-cond. * lisp/ob-sqlite.el (org-babel-execute:sqlite): Use org-babel-result-cond. --- lisp/ob-C.el | 10 +- lisp/ob-R.el | 18 -- lisp/ob-awk.el| 6 ++ lisp/ob-clojure.el| 5 ++--- lisp/ob-emacs-lisp.el | 9 - lisp/ob-fortran.el| 10 +- lisp/ob-io.el | 4 ++-- lisp/ob-java.el | 8 lisp/ob-lisp.el | 4 ++-- lisp/ob-maxima.el | 6 ++ lisp/ob-picolisp.el | 9 ++--- lisp/ob-python.el | 14 -- lisp/ob-scala.el | 4 ++-- lisp/ob-sh.el | 6 ++ lisp/ob-shen.el | 5 ++--- lisp/ob-sql.el| 8 ++-- lisp/ob-sqlite.el | 32 +++- lisp/ob.el| 14
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
Eric Schulte schulte.e...@gmail.com writes: Aaron Ecay aarone...@gmail.com writes: 2012ko azaroak 17an, Eric Schulte-ek idatzi zuen: Oh!, thanks for catching this, I just pushed up a fix. This is no longer a (complete) fix for the original problem, though. (A large part of) the slowdown comes from reading the results from a temp file and transforming them into an elispy format, which is handled by the backends. Your code only prevents them from being echoed to the minibuffer. You can see this by trying your original “seq” tests. You should see them taking the same (very long) amount of time. If you set debug-on-quit to t (before evaluating the block) and interrupt emacs with C-g when it hangs, you’ll see a backtrace that goes through ‘org-babel-execute:sh’ and then ‘org-babel-import-elisp-from-file’. The presence of the former in the call stack is why I claim that per-backend fixes are ultimately needed. The latter is where my patch addresses the problem in a temporary way. I apologize for rushing through your previous email and missing important portions of the content. I may be outvoted, but I find this approach too be overly complicated. Also, size may frequently not be the best proxy for the time which Emacs will take to ingest the results. I agree that my patch is a stopgap. But until per-backend fixes are available, this allows certain code to run that otherwise wouldn’t (at least not without hacks, such as putting NULL at the end of an R source block so that the “.Last.value” in the block is trivial). I'm attaching a new patch which should serve as a more permanent solution, however, given that it touches almost every backend I want to give the list a chance to try it out before committing it. This patch introduces a new macro (`org-babel-result-cond') which attempts to unify the results processing. Not only does this check for :results none and ignore all results in that case, but it also checks whether the results should be parsed as a scalar or a vector. Hopefully this will serve as useful refactoring and reduce the total amount of code as well as provide for more uniform results processing across languages. Please let me know if this looks like a good permanent solution. I see that I'm now getting 5 test failures with this patch applied locally, but I won't have time to look at this further until early next week at the earliest, so I'm sharing the existing patch now. Thanks, Alright, I've now changed the above patch so that as many tests are passing locally as were passing without the patch. Given that, I've applied this patch to the main git repository. Everything which worked previously should still be working, only now the :results none options should have the desired effect. Cheers, -- Eric Schulte http://cs.unm.edu/~eschulte
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
Achim Gratz strom...@nexgo.de writes: Andreas Leha writes: I would suggest simply none for the new really-silent. Yes please, especially since I tried :results none two days ago to achieve that effect and wondered why it wouldn't work. :-) Great, since none seems like the obvious choice I just committed a version of this patch which places this behavior behind :results none. At some point documentation should be added to the relevant portion of the manual, but at least we now have this for those reading the mailing list and for those who simply try none with this behavior in mind. Cheers, -- Eric Schulte http://cs.unm.edu/~eschulte
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
2012ko azaroak 16an, Eric Schulte-ek idatzi zuen: The attached patch adds a really-silent results header argument. To see the impact compare the running time of the following two code blocks. Unfortunately, the attached patch doesn’t work correctly. This can be seen by replacing the “seq” command in your examples with a command that has side effects – notify-send, aplay, etc. In the :results none” case, the command is not run at all. That’s because the “(funcall cmd body params)” call at l. 574 of ob.el (patched version) has been put in a branch that is only run if :results != none. That funcall is responsible for actually evaluating the code block. (The indentation of the patch as applied isn’t correct – the two branches of the if on l. 565 are indented at the same depth as each other, and as the if. So it’s possible that the problem is due to a paren in the wrong place. But I cannot see a way to make this approach work.) The code generating the slowdown is in backend-specific files. So, for example, a new branch needs to be added to the case statements in org-babel-R-evaluate(-session), to detect the :results none case and not read the result. I suspect that each backend will need this treatment (unless some of them share code). In the meantime, attached to this email is a patch that implements a size check on reading results. If the results file is over 10kb, it asks the user whether to proceed. You can test this by evaluating: #+begin_src sh :results silent seq 1 #+end_src 10kb of results actually doesn’t result in a very serious hang (~5sec on my system). But I chose the value as more of a sanity check – odds are (?) that very few people want to see 10k of results in the (mini)buffer. The value could be made customizable. I also chose the polarity of the y-or-n-p so that picking the default (yes) option does the sensible thing of not hanging emacs, although it thus does discard data. I’m not sure which is the worse problem. From 1053f3acfc21f24fc994ae85adff6779838b0ce7 Mon Sep 17 00:00:00 2001 From: Aaron Ecay aarone...@gmail.com Date: Sat, 17 Nov 2012 19:26:43 -0500 Subject: [PATCH] lisp/ob.el: add a size check to `org-babel-import-elisp-from-file' Reading large results can cause emacs to hang for a long time. Ask the user whether to proceed in such cases. Signed-off-by: Aaron Ecay aarone...@gmail.com --- lisp/ob.el | 42 -- 1 file changed, 24 insertions(+), 18 deletions(-) diff --git a/lisp/ob.el b/lisp/ob.el index bf4b455..b2385e9 100644 --- a/lisp/ob.el +++ b/lisp/ob.el @@ -2462,24 +2462,30 @@ appropriate. (defun org-babel-import-elisp-from-file (file-name optional separator) Read the results located at FILE-NAME into an elisp table. If the table is trivial, then return it as a scalar. - (let (result) -(save-window-excursion - (with-temp-buffer - (condition-case err - (progn - (org-table-import file-name separator) - (delete-file file-name) - (setq result (mapcar (lambda (row) - (mapcar #'org-babel-string-read row)) - (org-table-to-lisp - (error (message Error reading results: %s err) nil))) - (if (null (cdr result)) ;; if result is trivial vector, then scalarize it - (if (consp (car result)) - (if (null (cdr (car result))) - (caar result) - result) - (car result)) - result + (let* ((file-size (nth 7 (file-attributes file-name))) + (can-load (or ( file-size (* 10 1024)) ; 10kb + (not (y-or-n-p (concat Displaying the block's large + results may hang emacs; skip + reading them?)) +(when can-load + (let (result) +(save-window-excursion + (with-temp-buffer +(condition-case err +(progn + (org-table-import file-name separator) + (delete-file file-name) + (setq result (mapcar (lambda (row) + (mapcar #'org-babel-string-read row)) + (org-table-to-lisp + (error (message Error reading results: %s err) nil))) + (if (null (cdr result)) ;; if result is trivial vector, then scalarize it + (if (consp (car result)) + (if (null (cdr (car result))) + (caar result) +result) +(car result)) +result)) (defun org-babel-string-read (cell) Strip nested \s from around strings. -- 1.8.0 -- Aaron Ecay
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
Aaron Ecay aarone...@gmail.com writes: 2012ko azaroak 16an, Eric Schulte-ek idatzi zuen: The attached patch adds a really-silent results header argument. To see the impact compare the running time of the following two code blocks. Unfortunately, the attached patch doesn’t work correctly. This can be seen by replacing the “seq” command in your examples with a command that has side effects – notify-send, aplay, etc. In the :results none” case, the command is not run at all. That’s because the “(funcall cmd body params)” call at l. 574 of ob.el (patched version) has been put in a branch that is only run if :results != none. That funcall is responsible for actually evaluating the code block. Oh!, thanks for catching this, I just pushed up a fix. (The indentation of the patch as applied isn’t correct – the two branches of the if on l. 565 are indented at the same depth as each other, and as the if. So it’s possible that the problem is due to a paren in the wrong place. But I cannot see a way to make this approach work.) Yes, sometimes I find this approach to be preferable to make the diffs more readable and to avoid over-indentation in very large functions. The code generating the slowdown is in backend-specific files. So, for example, a new branch needs to be added to the case statements in org-babel-R-evaluate(-session), to detect the :results none case and not read the result. I suspect that each backend will need this treatment (unless some of them share code). In the meantime, attached to this email is a patch that implements a size check on reading results. If the results file is over 10kb, it asks the user whether to proceed. You can test this by evaluating: #+begin_src sh :results silent seq 1 #+end_src 10kb of results actually doesn’t result in a very serious hang (~5sec on my system). But I chose the value as more of a sanity check – odds are (?) that very few people want to see 10k of results in the (mini)buffer. The value could be made customizable. I also chose the polarity of the y-or-n-p so that picking the default (yes) option does the sensible thing of not hanging emacs, although it thus does discard data. I’m not sure which is the worse problem. I may be outvoted, but I find this approach too be overly complicated. Also, size may frequently not be the best proxy for the time which Emacs will take to ingest the results. Best, From 1053f3acfc21f24fc994ae85adff6779838b0ce7 Mon Sep 17 00:00:00 2001 From: Aaron Ecay aarone...@gmail.com Date: Sat, 17 Nov 2012 19:26:43 -0500 Subject: [PATCH] lisp/ob.el: add a size check to `org-babel-import-elisp-from-file' Reading large results can cause emacs to hang for a long time. Ask the user whether to proceed in such cases. Signed-off-by: Aaron Ecay aarone...@gmail.com --- lisp/ob.el | 42 -- 1 file changed, 24 insertions(+), 18 deletions(-) diff --git a/lisp/ob.el b/lisp/ob.el index bf4b455..b2385e9 100644 --- a/lisp/ob.el +++ b/lisp/ob.el @@ -2462,24 +2462,30 @@ appropriate. (defun org-babel-import-elisp-from-file (file-name optional separator) Read the results located at FILE-NAME into an elisp table. If the table is trivial, then return it as a scalar. - (let (result) -(save-window-excursion - (with-temp-buffer - (condition-case err - (progn - (org-table-import file-name separator) - (delete-file file-name) - (setq result (mapcar (lambda (row) - (mapcar #'org-babel-string-read row)) -(org-table-to-lisp - (error (message Error reading results: %s err) nil))) - (if (null (cdr result)) ;; if result is trivial vector, then scalarize it - (if (consp (car result)) - (if (null (cdr (car result))) - (caar result) - result) - (car result)) - result + (let* ((file-size (nth 7 (file-attributes file-name))) + (can-load (or ( file-size (* 10 1024)) ; 10kb + (not (y-or-n-p (concat Displaying the block's large + results may hang emacs; skip + reading them?)) +(when can-load + (let (result) +(save-window-excursion + (with-temp-buffer +(condition-case err +(progn + (org-table-import file-name separator) + (delete-file file-name) + (setq result (mapcar (lambda (row) + (mapcar #'org-babel-string-read row)) + (org-table-to-lisp + (error (message Error reading results: %s err) nil))) + (if (null (cdr result)) ;; if result is trivial vector, then scalarize it +
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
2012ko azaroak 17an, Eric Schulte-ek idatzi zuen: Oh!, thanks for catching this, I just pushed up a fix. This is no longer a (complete) fix for the original problem, though. (A large part of) the slowdown comes from reading the results from a temp file and transforming them into an elispy format, which is handled by the backends. Your code only prevents them from being echoed to the minibuffer. You can see this by trying your original “seq” tests. You should see them taking the same (very long) amount of time. If you set debug-on-quit to t (before evaluating the block) and interrupt emacs with C-g when it hangs, you’ll see a backtrace that goes through ‘org-babel-execute:sh’ and then ‘org-babel-import-elisp-from-file’. The presence of the former in the call stack is why I claim that per-backend fixes are ultimately needed. The latter is where my patch addresses the problem in a temporary way. I may be outvoted, but I find this approach too be overly complicated. Also, size may frequently not be the best proxy for the time which Emacs will take to ingest the results. I agree that my patch is a stopgap. But until per-backend fixes are available, this allows certain code to run that otherwise wouldn’t (at least not without hacks, such as putting NULL at the end of an R source block so that the “.Last.value” in the block is trivial). -- Aaron Ecay
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
Andreas Leha andreas.l...@med.uni-goettingen.de writes: t...@tsdye.com (Thomas S. Dye) writes: Aloha Aaron, Welcome to Org-mode. Aaron Ecay aarone...@gmail.com writes: 2012ko azaroak 13an, John Hendy-ek idatzi zuen: [...] Crazy. I really wondered if it had something to do with trying to spit out the results into the minibuffer. Why is that behavior included? “:results silent” just means “silent except for the minibuffer”; IDK why. [...] Just for the record: I would also love to see the really-silent results. Yes, the existing :results silent option still echos the results to the minibuffer. It was originally added in the case where one does not want to change the Org-mode buffer (but would still like to see code block output). The attached patch adds a really-silent results header argument. To see the impact compare the running time of the following two code blocks. #+begin_src sh :results really-silent seq 1 #+end_src #+begin_src sh :results silent seq 1 #+end_src Before such a patch is applied it would need corresponding documentation, and the really-silent moniker may need to be reconsidered in favor of something more informative (the main purpose of this header argument is not silence but is rather avoiding all post processing), and ideally shorter as well. These new header arguments are of no value if they are not easily discover-able and easily remember-able to help others avoid the same issue that launched this thread. Cheers, From 8ae5b59acdaaa271c797756bfe7f9479e1f9a277 Mon Sep 17 00:00:00 2001 From: Eric Schulte eric.schu...@gmx.com Date: Fri, 16 Nov 2012 08:40:46 -0700 Subject: [PATCH] really-silent header avoids result post-process This patch introduces a new header argument which hide all code block results from Emacs so that they are not processed into Emacs-lisp objects; a process which may take a great deal of time for large objects. * lisp/ob.el (org-babel-common-header-args-w-values): Adding the new really-silent header argument. (org-babel-execute-src-block): Don't do *any* result processing if the really-silent header argument has been specified. (org-babel-sha1-hash): Remove the really-silent header argument from header arg lists when calculating cache sums. (org-babel-insert-result): Flesh out some documentation. --- lisp/ob.el | 15 +++ 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/lisp/ob.el b/lisp/ob.el index 69a5c58..167fe1a 100644 --- a/lisp/ob.el +++ b/lisp/ob.el @@ -421,7 +421,7 @@ then run `org-babel-pop-to-session'. (padline . ((yes no))) (results . ((file list vector table scalar verbatim) (raw html latex org code pp drawer) - (replace silent append prepend) + (replace silent really-silent append prepend) (output value))) (rownames . ((no yes))) (sep . :any) @@ -562,6 +562,8 @@ block. (message executing %s code block%s... (capitalize lang) (if (nth 4 info) (format (%s) (nth 4 info)) )) + (if (member really-silent result-params) + (message result silenced) (setq result ((lambda (result) (if (and (eq (cdr (assoc :result-type params)) 'value) @@ -581,7 +583,8 @@ block. (org-babel-insert-result result result-params info new-hash indent lang) (run-hooks 'org-babel-after-execute-hook) - result)) + result + ))) (setq call-process-region 'org-babel-call-process-region-original)) (defun org-babel-expand-body:generic (body params optional var-lines) @@ -1028,7 +1031,8 @@ the current subtree. (sort (copy-sequence (nth 2 info)) (lambda (a b) (string (car a) (car b) (let* ((rm (lambda (lst) - (dolist (p '(replace silent append prepend)) + (dolist (p '(replace silent really-silent + append prepend)) (setq lst (remove p lst))) lst)) (norm (lambda (arg) @@ -1851,7 +1855,10 @@ RESULT-PARAMS can take the following values: replace - (default option) insert results after the source block replacing any previously inserted results -silent -- no results are inserted +silent -- no results are inserted into the Org-mode buffer but + the results are echoed to the minibuffer and are + ingested by Emacs (a potentially time consuming + process) file the results are interpreted as a file path, and are inserted into the buffer using the Org-mode file syntax -- 1.8.0 -- Eric Schulte http://cs.unm.edu/~eschulte
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
Hi Eric, Eric Schulte schulte.e...@gmail.com writes: Andreas Leha andreas.l...@med.uni-goettingen.de writes: t...@tsdye.com (Thomas S. Dye) writes: Aloha Aaron, Welcome to Org-mode. Aaron Ecay aarone...@gmail.com writes: 2012ko azaroak 13an, John Hendy-ek idatzi zuen: [...] Crazy. I really wondered if it had something to do with trying to spit out the results into the minibuffer. Why is that behavior included? “:results silent” just means “silent except for the minibuffer”; IDK why. [...] Just for the record: I would also love to see the really-silent results. Yes, the existing :results silent option still echos the results to the minibuffer. It was originally added in the case where one does not want to change the Org-mode buffer (but would still like to see code block output). The attached patch adds a really-silent results header argument. To see the impact compare the running time of the following two code blocks. #+begin_src sh :results really-silent seq 1 #+end_src #+begin_src sh :results silent seq 1 #+end_src Before such a patch is applied it would need corresponding documentation, and the really-silent moniker may need to be reconsidered in favor of something more informative (the main purpose of this header argument is not silence but is rather avoiding all post processing), and ideally shorter as well. These new header arguments are of no value if they are not easily discover-able and easily remember-able to help others avoid the same issue that launched this thread. Cheers, this is simply great! Thank a lot. I just tested and like it a lot! With respect to the name: I think choosing a new name is hard because it really should be silent. What is now silent would better be named minibuffer or sth. similar. But since changing that would break quite some (of my) code, I would suggest simply none for the new really-silent. Just my 2ct. Thanks again. I'll start using it right now. Regards, Andreas
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
On Fri, Nov 16, 2012 at 11:47 AM, Andreas Leha andreas.l...@med.uni-goettingen.de wrote: Hi Eric, Eric Schulte schulte.e...@gmail.com writes: Andreas Leha andreas.l...@med.uni-goettingen.de writes: t...@tsdye.com (Thomas S. Dye) writes: Aloha Aaron, Welcome to Org-mode. Aaron Ecay aarone...@gmail.com writes: 2012ko azaroak 13an, John Hendy-ek idatzi zuen: [...] Crazy. I really wondered if it had something to do with trying to spit out the results into the minibuffer. Why is that behavior included? “:results silent” just means “silent except for the minibuffer”; IDK why. [...] Just for the record: I would also love to see the really-silent results. Yes, the existing :results silent option still echos the results to the minibuffer. It was originally added in the case where one does not want to change the Org-mode buffer (but would still like to see code block output). The attached patch adds a really-silent results header argument. To see the impact compare the running time of the following two code blocks. #+begin_src sh :results really-silent seq 1 #+end_src #+begin_src sh :results silent seq 1 #+end_src Before such a patch is applied it would need corresponding documentation, and the really-silent moniker may need to be reconsidered in favor of something more informative (the main purpose of this header argument is not silence but is rather avoiding all post processing), and ideally shorter as well. These new header arguments are of no value if they are not easily discover-able and easily remember-able to help others avoid the same issue that launched this thread. Cheers, this is simply great! Thank a lot. I just tested and like it a lot! I'll have to give this a try. With respect to the name: I think choosing a new name is hard because it really should be silent. What is now silent would better be named minibuffer or sth. similar. But since changing that would break quite some (of my) code, I would suggest simply none for the new really-silent. I like the none idea. Or another approach would be a variable/in-buffer setting that defines the meaning of silent? Something like =org-babel-echo-results-in-minibuffer= or something like that? Though perhaps folks would like to mix and match silent and none within the same document. Thanks for creating the patch, Eric. I'll try shortly. Best regards, John Just my 2ct. Thanks again. I'll start using it right now. Regards, Andreas
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
Andreas Leha writes: I would suggest simply none for the new really-silent. Yes please, especially since I tried :results none two days ago to achieve that effect and wondered why it wouldn't work. :-) Regards, Achim. -- +[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+ SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
Aloha Aaron, Welcome to Org-mode. Aaron Ecay aarone...@gmail.com writes: 2012ko azaroak 13an, John Hendy-ek idatzi zuen: [...] Crazy. I really wondered if it had something to do with trying to spit out the results into the minibuffer. Why is that behavior included? “:results silent” just means “silent except for the minibuffer”; IDK why. I think silent has to do with the Org buffer that contains the source code block. Non-silent blocks do something in the buffer. This was, for me, one of the amazing things about babel when it was being developed. One could speak any language and have sensible results written back to a plain text file. Are you sure it's only on file import? I was generating some large output when I first experienced this. Yes, that’s what I meant (and didn’t explain well). Org writes R’s output to a file, and then imports it into Emacs (to print in the minibuffer). But it also does an expensive transformation before printing it – whence the slowdown. I have to change to the R buffer to see if there were any errors. I agree that this is sub-optimal. I seem to remember that ob-R.el is unmaintained (or was at some point in the past). Once there is consensus on what a fix should look like, I can attempt to produce a patch implementing it. And I’ll take a crack at error reporting in the minibuffer too. Yes, ob-R.el is currently without a maintainer. My sense and experience is that Eric S. is happy to entertain patches to the various ob language files. My experience is also that active maintainers welcome patches--I don't think there is a barrier to patch proposals. I can help with documentation if your patch results in some substantive changes to R support in babel. Please don't hesitate to ask if need be. All the best, Tom -- Thomas S. Dye http://www.tsdye.com
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
t...@tsdye.com (Thomas S. Dye) writes: Aloha Aaron, Welcome to Org-mode. Aaron Ecay aarone...@gmail.com writes: 2012ko azaroak 13an, John Hendy-ek idatzi zuen: [...] Crazy. I really wondered if it had something to do with trying to spit out the results into the minibuffer. Why is that behavior included? “:results silent” just means “silent except for the minibuffer”; IDK why. [...] Just for the record: I would also love to see the really-silent results. Regards, Andreas
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
I just ran into this problem. It is caused by the calls to ‘org-babel-import-elisp-from-file’ on lines 310 and 343 of ‘ob-R.el’ (for non-session and session code blocks respectively). I determined this by setting ‘debug-on-quit’ to t in Emacs, then hitting C-g when cpu usage spiked, and examining the backtrace. It seems like Emacs tries to read in the last value of the R command, to print it in the minibuffer. It gets bogged down doing so. As a stopgap, putting NULL as the last line of the source block fixes the problem (the value “returned” by the block is just NULL). I believe setting “:results output” (as opposed to “value”) would also make the bug disappear, since the output will be very small. This is likely what made cberry’s test differ from John’s. I’m not sure what the right strategy would be for a fix. Two ideas would be: 1) adding a sanity check to o-b-import-elisp-from-file, which asks the user whether to really import very large files (or just ignores such files) 2) adding a new :results handling header arg “really-silent” (probably with a better name...) that discards the result entirely, not even printing it to the minibuffer. -- Aaron Ecay
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
On Tue, Nov 13, 2012 at 9:27 PM, Aaron Ecay aarone...@gmail.com wrote: I just ran into this problem. It is caused by the calls to ‘org-babel-import-elisp-from-file’ on lines 310 and 343 of ‘ob-R.el’ (for non-session and session code blocks respectively). I determined this by setting ‘debug-on-quit’ to t in Emacs, then hitting C-g when cpu usage spiked, and examining the backtrace. Yes! I'm not crazy! I'm not very familiar with the debug options in emacs and had a heck of a time setting it up right the last time. Thanks so much for checking this out. It seems like Emacs tries to read in the last value of the R command, to print it in the minibuffer. It gets bogged down doing so. As a stopgap, putting NULL as the last line of the source block fixes the problem (the value “returned” by the block is just NULL). I believe setting “:results output” (as opposed to “value”) would also make the bug disappear, since the output will be very small. This is likely what made cberry’s test differ from John’s. Crazy. I really wondered if it had something to do with trying to spit out the results into the minibuffer. Why is that behavior included? I’m not sure what the right strategy would be for a fix. Two ideas would be: 1) adding a sanity check to o-b-import-elisp-from-file, which asks the user whether to really import very large files (or just ignores such files) Are you sure it's only on file import? I was generating some large output when I first experienced this. It was only to avoid running that code that I wrote it to file once and decided to just read it once per session afterward (thinking reading would be faster than generating the input on demand). I was generating a grid of interactions with something akin to: #+begin_src R dat - expand.grid(var1 = seq(0, 4, by = 200), var2 = seq(0, 4, by = 200)) #+end_src I can check this tomorrow if you'd like, but I remember it lagging as it tried to print all of that into the minibuffer. 2) adding a new :results handling header arg “really-silent” (probably with a better name...) that discards the result entirely, not even printing it to the minibuffer. I like this, in addition to changing the minibuffer output as it is. It would be fantastic to have the minibuffer say more than it does already (something akin to Block execution complete). I have to change to the R buffer to see if there were any errors. I like how the LaTeX exporter tells you if PDF export failed. It'd be nice to see something like (in addition to no results in the minibuffer), Block execution complete but with errors. Thanks again, and I'm glad you were able to track this down. John -- Aaron Ecay
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
2012ko azaroak 13an, John Hendy-ek idatzi zuen: [...] Crazy. I really wondered if it had something to do with trying to spit out the results into the minibuffer. Why is that behavior included? “:results silent” just means “silent except for the minibuffer”; IDK why. Are you sure it's only on file import? I was generating some large output when I first experienced this. Yes, that’s what I meant (and didn’t explain well). Org writes R’s output to a file, and then imports it into Emacs (to print in the minibuffer). But it also does an expensive transformation before printing it – whence the slowdown. I have to change to the R buffer to see if there were any errors. I agree that this is sub-optimal. I seem to remember that ob-R.el is unmaintained (or was at some point in the past). Once there is consensus on what a fix should look like, I can attempt to produce a patch implementing it. And I’ll take a crack at error reporting in the minibuffer too. -- Aaron Ecay
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
On Wed, Oct 31, 2012 at 5:53 PM, Nick Dokos nicholas.do...@hp.com wrote: John Hendy jw.he...@gmail.com wrote: On Wed, Oct 31, 2012 at 3:12 PM, cbe...@tajo.ucsd.edu wrote: John Hendy jw.he...@gmail.com writes: On Wed, Oct 31, 2012 at 11:41 AM, span dir=ltrmailto: cbe...@tajo.ucsd.edu/span wrote: John Hendy mailto:jw.he...@gmail.com writes: I edited the subject to be more concise/clear.I let orgmode chug away on reading in some ~10-30mb csv files for nearly 30min. [rest deleted] You need an ECM.I did my best to provide one, other than the file, which I offered to provide if others requested that I upload it somewhere. Since you have done so, so have I: - https://docs.google.com/open?id=0BzQupOSnvw08WHdabHh5VVczRGM Let me know if that doesn#39;t work. I put it on Google docs and sometimes have issues with the sharing settings... Not an ECM in my book, but ... What else would you like? I provided: - the config - the data - how to [attempt to] reproduce - the org-mode text Smaller set of data I'd guess :-) But it does not seem to be the size of the data that matters. On my 4 year old MacBook: , | | #+PROPERTY: session *R* | | #+name: bigcsv | #+begin_src R | bigcsv - Sys.glob(~/Downloads/*.csv) | #+end_src | | #+RESULTS: bigcsv | : /Users/cberry/Downloads/test-file.csv | | #+name: readbig | #+begin_src R :results output | system.time( | tmp - read.csv(bigcsv) | ) | | #+end_src | | #+RESULTS: readbig | :user system elapsed | : 5.679 0.306 6.002 | ` About the same as running from ESS. Not sure what to say. Looking for ways to troubleshoot or confirm. Since you can't confirm, any suggestions on where I should look for my issue? I can't explain it! All I know is that org chugs and chugs and the direct execution in ESS session is lightning fast. A few things to try in no particular order: This was extremely helpful. Thanks for the suggestions. Here's my attempt at an ECM, though I'm going to keep using the big file since that's what's actually doing it an I've already uploaded it :) - Using emacs config here: http://pastebin.com/raw.php?i=iTbRtCE9 - Using this org-mode file: #+begin_src org * headline #+begin_src R :session r :results silent # file here: https://docs.google.com/uc?export=downloadconfirm=no_antivirusid=0BzQupOSnvw08WHdabHh5VVczRGM data - read.csv(path/to/file.csv) #+end_src #+end_src org - Execute block with C-c C-c after downloading and changing path o run top (or whatever equivalent is available on your OS) and see whether the CPU (or one of the CPUs) gets pegged at 100% utilization and stays there. If yes, that's an indication of an infinite loop somewhere. - quit any other instances of emacs/R - start `top` in terminal - execute block - Use '' '' to sort back and forth between cpu and ram Observations - R is at 80-100% cpu for about 5sec - Then emacs shifts to fairly constant ~100% cpu usage - After about a minute, the minibuffer expands to ~1/3 of the window height and fills with the csv data - Finished after ~5min total time - So, R took about 5sec, emacs took another 5min to finish o run vmstat (or equivalent) and see if any of the counters are out of whack. That requires some experience though. I'll skip for now; no experience with that. o use elp-instrument-package to instrument org and run the test, getting a profile. I'm not sure whether the results will be useful, since you are going to interrupt the test when you run out of patience, but it cannot hurt and it might tell you something useful. o run your ECM on a different computer/OS/emacs installation. Being able to compare things side by side is often very useful. o Halve your file and run the test on each half (but that's probably not the problem given Chuck's results). o Reinstall org from scratch - you might have some corruption in one of the compiled files that's causing it to go into an infinite loop. - `cd ~/.elisp` - `sudo rm -r org.git` - `git clone http://git://orgmode.org/org-mode.git org.git` - cd org.git make clean make make doc - Quit previous emacs instance; reopen - Remove (require 'org-install) per prompt; restart again - Repeat `top` experiment Results: - Didn't even see R flash on the screen this time; emacs just jumped to 100% - After 1min 10sec, the minibuffer filled with data - At that point I quit, as I think it will be a repeat of the above o Turn on debug-on-quit, start your test, wait a bit and then interrupt it. Check the backtrace. Do it again and check whether the backtrace looks the same. That's often an indication of an infinite loop (inferring an infinite loop from a two element sample
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
John Hendy jw.he...@gmail.com wrote: o run top (or whatever equivalent is available on your OS) and see whether the CPU (or one of the CPUs) gets pegged at 100% utilization and stays there. If yes, that's an indication of an infinite loop somewhere. - quit any other instances of emacs/R - start `top` in terminal - execute block - Use '' '' to sort back and forth between cpu and ram Observations - R is at 80-100% cpu for about 5sec - Then emacs shifts to fairly constant ~100% cpu usage - After about a minute, the minibuffer expands to ~1/3 of the window height and fills with the csv data - Finished after ~5min total time - So, R took about 5sec, emacs took another 5min to finish So not an infinite loop. That's progress ;-) Perhaps emacs is thrashing? If you are on linux, use swapon -s or (perhaps better) iostat, or (perhaps even better, at least if you are on the Gnome 2.x desktop[fn:1]), run the system monitor applet, click Properties, enable all the monitored resources (cpu, mem, net, swap, load, disk) and watch it while you are running the test: in particular, check memory and swap. If you are swapping even a little bit, that's enough to cause severe performacne problems[fn:2], which can be cured by using less memory (so stop any memory hogs from running) or by adding memory. Top can also show you memory and swap so maybe that's the quickest way to check. Nick Footnotes: [fn:1] You can pobably do this in other desktops but I have no experience with them, so no guidance to give. [fn:2] My old laptop with 1GB of memory would swap whenever I ran mairix to index my mail: it would take 30 minutes or so to finish. My new(er) laptop has 4GB and finishes in 30 seconds: memory usage peaks at about 2.5 GB.
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
On Thu, Nov 1, 2012 at 10:38 AM, Nick Dokos nicholas.do...@hp.com wrote: John Hendy jw.he...@gmail.com wrote: o run top (or whatever equivalent is available on your OS) and see whether the CPU (or one of the CPUs) gets pegged at 100% utilization and stays there. If yes, that's an indication of an infinite loop somewhere. - quit any other instances of emacs/R - start `top` in terminal - execute block - Use '' '' to sort back and forth between cpu and ram Observations - R is at 80-100% cpu for about 5sec - Then emacs shifts to fairly constant ~100% cpu usage - After about a minute, the minibuffer expands to ~1/3 of the window height and fills with the csv data - Finished after ~5min total time - So, R took about 5sec, emacs took another 5min to finish So not an infinite loop. That's progress ;-) Perhaps emacs is thrashing? If you are on linux, use swapon -s or (perhaps better) iostat, or (perhaps even better, at least if you are on the Gnome 2.x desktop[fn:1]), run the system monitor applet, click Properties, enable all the monitored resources (cpu, mem, net, swap, load, disk) and watch it while you are running the test: in particular, check memory and swap. If you are swapping even a little bit, that's enough to cause severe performacne problems[fn:2], which can be cured by using less memory (so stop any memory hogs from running) or by adding memory. Top can also show you memory and swap so maybe that's the quickest way to check. It's not swap... :) $ free total used free sharedbuffers cached Mem: 397886424271321551732 0 122072 509140 -/+ buffers/cache:17959202182944 Swap:0 0 0 Best regards, John Nick Footnotes: [fn:1] You can pobably do this in other desktops but I have no experience with them, so no guidance to give. [fn:2] My old laptop with 1GB of memory would swap whenever I ran mairix to index my mail: it would take 30 minutes or so to finish. My new(er) laptop has 4GB and finishes in 30 seconds: memory usage peaks at about 2.5 GB.
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
On Thu, Nov 1, 2012 at 10:38 AM, Nick Dokos nicholas.do...@hp.com wrote: John Hendy jw.he...@gmail.com wrote: o run top (or whatever equivalent is available on your OS) and see whether the CPU (or one of the CPUs) gets pegged at 100% utilization and stays there. If yes, that's an indication of an infinite loop somewhere. - quit any other instances of emacs/R - start `top` in terminal - execute block - Use '' '' to sort back and forth between cpu and ram Observations - R is at 80-100% cpu for about 5sec - Then emacs shifts to fairly constant ~100% cpu usage - After about a minute, the minibuffer expands to ~1/3 of the window height and fills with the csv data - Finished after ~5min total time - So, R took about 5sec, emacs took another 5min to finish So not an infinite loop. That's progress ;-) Perhaps emacs is thrashing? If you are on linux, use swapon -s or (perhaps better) iostat, or (perhaps even better, at least if you are on the Gnome 2.x desktop[fn:1]), run the system monitor applet, click Properties, enable all the monitored resources (cpu, mem, net, swap, load, disk) and watch it while you are running the test: in particular, check memory and swap. If you are swapping even a little bit, that's enough to cause severe performacne problems[fn:2], which can be cured by using less memory (so stop any memory hogs from running) or by adding memory. Oh, and I'm conscious of R using RAM, so I tried to run this stuff without browsers open, no other R sessions, etc. Prior to running R and without a browser, I'm typically around 10-15% RAM utilized... I have 4gb, so that should be enough to read this, especially per Chuck's results! This is also an i7 work computer. It should fly (and in the ESS session itself, it does). John Top can also show you memory and swap so maybe that's the quickest way to check. Nick Footnotes: [fn:1] You can pobably do this in other desktops but I have no experience with them, so no guidance to give. [fn:2] My old laptop with 1GB of memory would swap whenever I ran mairix to index my mail: it would take 30 minutes or so to finish. My new(er) laptop has 4GB and finishes in 30 seconds: memory usage peaks at about 2.5 GB.
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
John Hendy jw.he...@gmail.com wrote: On Thu, Nov 1, 2012 at 10:38 AM, Nick Dokos nicholas.do...@hp.com wrote: John Hendy jw.he...@gmail.com wrote: o run top (or whatever equivalent is available on your OS) and see whether the CPU (or one of the CPUs) gets pegged at 100% utilization and stays there. If yes, that's an indication of an infinite loop somewhere. - quit any other instances of emacs/R - start `top` in terminal - execute block - Use '' '' to sort back and forth between cpu and ram Observations - R is at 80-100% cpu for about 5sec - Then emacs shifts to fairly constant ~100% cpu usage - After about a minute, the minibuffer expands to ~1/3 of the window height and fills with the csv data - Finished after ~5min total time - So, R took about 5sec, emacs took another 5min to finish So not an infinite loop. That's progress ;-) Perhaps emacs is thrashing? If you are on linux, use swapon -s or (perhaps better) iostat, or (perhaps even better, at least if you are on the Gnome 2.x desktop[fn:1]), run the system monitor applet, click Properties, enable all the monitored resources (cpu, mem, net, swap, load, disk) and watch it while you are running the test: in particular, check memory and swap. If you are swapping even a little bit, that's enough to cause severe performacne problems[fn:2], which can be cured by using less memory (so stop any memory hogs from running) or by adding memory. Oh, and I'm conscious of R using RAM, so I tried to run this stuff without browsers open, no other R sessions, etc. Prior to running R and without a browser, I'm typically around 10-15% RAM utilized... I have 4gb, so that should be enough to read this, especially per Chuck's results! This is also an i7 work computer. It should fly (and in the ESS session itself, it does). Check /var/log/syslog (or thereabouts) to see if the kernel is seeing errors with some blocks on the disk and keeps retrying. Assuming SMART is enabled, run smartctl -a as well and look for problems. Nick
[O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
I edited the subject to be more concise/clear. I let orgmode chug away on reading in some ~10-30mb csv files for nearly 30min. I finally gave up with Ctrl + g. Out of curiosity, I copied my babel block code (four simple lines in the form of `var - read.csv(./file.csv, header = T, sep = ,)` into the ESS session already started by my first block execution, and it finished in less than 10 seconds. Something seems seriously wrong with however org + babel is executing. I'm suspicious of it printing the output to the minibuffer. I have about 200^2 - 400^2 lines in the csv files. Was shocked that I'd been waiting for org this whole time and could have just run the code in the ESS R session directly in a minuscule fraction of the time. I could post a reproducible csv file if others would like to try this? I tried the following to make sure it wasn't my setup: 1) from terminal: `emacs -Q --load=~/Desktop/basic-config`, where basic-config contains: ;; set load paths ;; set load dirs and global config options (add-to-list 'load-path ~/.elisp/site-lisp/) (add-to-list 'load-path ~/.elisp/org.git/contrib/lisp/) (add-to-list 'load-path ~/.elisp/org.git/lisp/) (add-to-list 'load-path ~/.elisp/site-lisp/ess/lisp/) ;; setup org (require 'org-install) ;; setup babel languages (require 'ess-site) (org-babel-do-load-languages 'org-babel-load-languages '((R . t))) 2) Created a simple org file: -- * headline #+begin_src R :session r :results silent data - read.csv(/media/vault/misc/benefit-plan-analysis/2012/contour_table.csv) #+end_src -- 3) Run =C-c C-c YES RET= on block 4) Change to R session and enter the same line manually. The results are unbelievably different. I didn't let the org session finish as once it took longer than 30sec I figured I'd get the same results as with my actual file in this example. Changing to the R session and it was done in less than 5sec with a ~70mb csv. Any suggestions on troubleshooting this? Best regards, John On Tue, Oct 30, 2012 at 2:46 AM, John Hendy jw.he...@gmail.com wrote: I'm working with a large dataset. I try to minimize overhead by doing some bigger data rearranging and calculations in a single block that I execute once per session and then switch :eval to no. This block is set with :results silent, but I know with this option the results are still printed to the minibuffer. Usually this isn't a problem, but I've had some hangs with emacs and started wondering if this could be due to Orgmode regurgitating the results. Could this be the case? If so, could Org be made to stop this behavior? If I accidentally type the name of a data frame in the R session buffer, even when I use C-c C-c to stop the print out, it still takes quite a while to catch up and stop itself. It made me wonder if trying to print all of those results is increasing my compilation times unnecessarily. Thanks for any thoughts, John
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
John Hendy jw.he...@gmail.com writes: I edited the subject to be more concise/clear.I let orgmode chug away on reading in some ~10-30mb csv files for nearly 30min. [rest deleted] You need an ECM. I cannot reproduce your issue. This runs in the same amount of time, whether I execute the src block or run the code from ESS: #+begin_src R :session write.csv(diag(10)[rep(1:10,10),],file=abc.csv) #+end_src #+begin_src R :session :results output system.time(read.csv(abc.csv)) #+end_src abc.csv is 25MB If the behavior you describe only happens with some files, I suggest you try to bisect them to find the issue. HTH, Chuck
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
On Wed, Oct 31, 2012 at 11:41 AM, cbe...@tajo.ucsd.edu wrote: John Hendy jw.he...@gmail.com writes: I edited the subject to be more concise/clear.I let orgmode chug away on reading in some ~10-30mb csv files for nearly 30min. [rest deleted] You need an ECM. I did my best to provide one, other than the file, which I offered to provide if others requested that I upload it somewhere. Since you have done so, so have I: - https://docs.google.com/open?id=0BzQupOSnvw08WHdabHh5VVczRGM Let me know if that doesn't work. I put it on Google docs and sometimes have issues with the sharing settings... You will find the rest needed for a complete ECM of my setup in the original email: - minimal config I used with `emacs -Q --load=file` - org file with minimal babel R block Let me know if you can reproduce with the linked file. I'm open to suggestions on how to bisect a file, though it's a column of strings and several columns of numbers, I'm not exactly sure what to look for. Best regards, John I cannot reproduce your issue. This runs in the same amount of time, whether I execute the src block or run the code from ESS: #+begin_src R :session write.csv(diag(10)[rep(1:10,10),],file=abc.csv) #+end_src #+begin_src R :session :results output system.time(read.csv(abc.csv)) #+end_src abc.csv is 25MB If the behavior you describe only happens with some files, I suggest you try to bisect them to find the issue. HTH, Chuck
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
John Hendy jw.he...@gmail.com writes: On Wed, Oct 31, 2012 at 11:41 AM, span dir=ltrmailto:cbe...@tajo.ucsd.edu/span wrote: John Hendy mailto:jw.he...@gmail.com writes: I edited the subject to be more concise/clear.I let orgmode chug away on reading in some ~10-30mb csv files for nearly 30min. [rest deleted] You need an ECM.I did my best to provide one, other than the file, which I offered to provide if others requested that I upload it somewhere. Since you have done so, so have I: - https://docs.google.com/open?id=0BzQupOSnvw08WHdabHh5VVczRGM Let me know if that doesn#39;t work. I put it on Google docs and sometimes have issues with the sharing settings... Not an ECM in my book, but ... On my 4 year old MacBook: , | | #+PROPERTY: session *R* | | #+name: bigcsv | #+begin_src R | bigcsv - Sys.glob(~/Downloads/*.csv) | #+end_src | | #+RESULTS: bigcsv | : /Users/cberry/Downloads/test-file.csv | | #+name: readbig | #+begin_src R :results output | system.time( | tmp - read.csv(bigcsv) | ) | | #+end_src | | #+RESULTS: readbig | :user system elapsed | : 5.679 0.306 6.002 | ` About the same as running from ESS. FYI, dim(tmp) [1] 964805 9 Chuck [ rest deleted ]
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
On Wed, Oct 31, 2012 at 3:12 PM, cbe...@tajo.ucsd.edu wrote: John Hendy jw.he...@gmail.com writes: On Wed, Oct 31, 2012 at 11:41 AM, span dir=ltrmailto: cbe...@tajo.ucsd.edu/span wrote: John Hendy mailto:jw.he...@gmail.com writes: I edited the subject to be more concise/clear.I let orgmode chug away on reading in some ~10-30mb csv files for nearly 30min. [rest deleted] You need an ECM.I did my best to provide one, other than the file, which I offered to provide if others requested that I upload it somewhere. Since you have done so, so have I: - https://docs.google.com/open?id=0BzQupOSnvw08WHdabHh5VVczRGM Let me know if that doesn#39;t work. I put it on Google docs and sometimes have issues with the sharing settings... Not an ECM in my book, but ... What else would you like? I provided: - the config - the data - how to [attempt to] reproduce - the org-mode text On my 4 year old MacBook: , | | #+PROPERTY: session *R* | | #+name: bigcsv | #+begin_src R | bigcsv - Sys.glob(~/Downloads/*.csv) | #+end_src | | #+RESULTS: bigcsv | : /Users/cberry/Downloads/test-file.csv | | #+name: readbig | #+begin_src R :results output | system.time( | tmp - read.csv(bigcsv) | ) | | #+end_src | | #+RESULTS: readbig | :user system elapsed | : 5.679 0.306 6.002 | ` About the same as running from ESS. Not sure what to say. Looking for ways to troubleshoot or confirm. Since you can't confirm, any suggestions on where I should look for my issue? I can't explain it! All I know is that org chugs and chugs and the direct execution in ESS session is lightning fast. FYI, dim(tmp) [1] 964805 9 Was that just to show the size, or is something wrong with my file? Chuck Thanks for attempting to replicate! John [ rest deleted ]
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
Hi John, Have you tried wrapping your R read in system.time()? If you are right about :results silent eating up lots of time, then this should fix the problem. system.time yields just a bit of output, so shouldn't slow things down if writing out the data is indeed the problem as you suspect. hth, Tom John Hendy jw.he...@gmail.com writes: On Wed, Oct 31, 2012 at 3:12 PM, cbe...@tajo.ucsd.edu wrote: John Hendy jw.he...@gmail.com writes: On Wed, Oct 31, 2012 at 11:41 AM, span dir=ltrmailto: cbe...@tajo.ucsd.edu/span wrote: John Hendy mailto:jw.he...@gmail.com writes: I edited the subject to be more concise/clear.I let orgmode chug away on reading in some ~10-30mb csv files for nearly 30min. [rest deleted] You need an ECM.I did my best to provide one, other than the file, which I offered to provide if others requested that I upload it somewhere. Since you have done so, so have I: - https://docs.google.com/open?id=0BzQupOSnvw08WHdabHh5VVczRGM Let me know if that doesn#39;t work. I put it on Google docs and sometimes have issues with the sharing settings... Not an ECM in my book, but ... What else would you like? I provided: - the config - the data - how to [attempt to] reproduce - the org-mode text On my 4 year old MacBook: , | | #+PROPERTY: session *R* | | #+name: bigcsv | #+begin_src R | bigcsv - Sys.glob(~/Downloads/*.csv) | #+end_src | | #+RESULTS: bigcsv | : /Users/cberry/Downloads/test-file.csv | | #+name: readbig | #+begin_src R :results output | system.time( | tmp - read.csv(bigcsv) | ) | | #+end_src | | #+RESULTS: readbig | :user system elapsed | : 5.679 0.306 6.002 | ` About the same as running from ESS. Not sure what to say. Looking for ways to troubleshoot or confirm. Since you can't confirm, any suggestions on where I should look for my issue? I can't explain it! All I know is that org chugs and chugs and the direct execution in ESS session is lightning fast. FYI, dim(tmp) [1] 964805 9 Was that just to show the size, or is something wrong with my file? Chuck Thanks for attempting to replicate! John [ rest deleted ] On Wed, Oct 31, 2012 at 3:12 PM, cbe...@tajo.ucsd.edu wrote: John Hendy jw.he...@gmail.com writes: On Wed, Oct 31, 2012 at 11:41 AM, span dir=ltrmailto:cbe...@tajo.ucsd.edu/span wrote: John Hendy mailto:jw.he...@gmail.com writes: I edited the subject to be more concise/clear.I let orgmode chug away on reading in some ~10-30mb csv files for nearly 30min. [rest deleted] You need an ECM.I did my best to provide one, other than the file, which I offered to provide if others requested that I upload it somewhere. Since you have done so, so have I: - https://docs.google.com/open?id=0BzQupOSnvw08WHdabHh5VVczRGM Let me know if that doesn#39;t work. I put it on Google docs and sometimes have issues with the sharing settings... Not an ECM in my book, but ... What else would you like? I provided: - the config - the data - how to [attempt to] reproduce - the org-mode text On my 4 year old MacBook: , | | #+PROPERTY: session *R* | | #+name: bigcsv | #+begin_src R | bigcsv - Sys.glob(~/Downloads/*.csv) | #+end_src | | #+RESULTS: bigcsv | : /Users/cberry/Downloads/test-file.csv | | #+name: readbig | #+begin_src R :results output | system.time( | tmp - read.csv(bigcsv) | ) | | #+end_src | | #+RESULTS: readbig | : user system elapsed | : 5.679 0.306 6.002 | ` About the same as running from ESS. Not sure what to say. Looking for ways to troubleshoot or confirm. Since you can't confirm, any suggestions on where I should look for my issue? I can't explain it! All I know is that org chugs and chugs and the direct execution in ESS session is lightning fast. FYI, dim(tmp) [1] 964805 9 Was that just to show the size, or is something wrong with my file? Chuck Thanks for attempting to replicate! John [ rest deleted ] -- Thomas S. Dye http://www.tsdye.com
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
John Hendy jw.he...@gmail.com wrote: On Wed, Oct 31, 2012 at 3:12 PM, cbe...@tajo.ucsd.edu wrote: John Hendy jw.he...@gmail.com writes: On Wed, Oct 31, 2012 at 11:41 AM, span dir=ltrmailto:cbe...@tajo.ucsd.edu/span wrote: John Hendy mailto:jw.he...@gmail.com writes: I edited the subject to be more concise/clear.I let orgmode chug away on reading in some ~10-30mb csv files for nearly 30min. [rest deleted] You need an ECM.I did my best to provide one, other than the file, which I offered to provide if others requested that I upload it somewhere. Since you have done so, so have I: - https://docs.google.com/open?id=0BzQupOSnvw08WHdabHh5VVczRGM Let me know if that doesn#39;t work. I put it on Google docs and sometimes have issues with the sharing settings... Not an ECM in my book, but ... What else would you like? I provided: - the config - the data - how to [attempt to] reproduce - the org-mode text Smaller set of data I'd guess :-) But it does not seem to be the size of the data that matters. On my 4 year old MacBook: , | | #+PROPERTY: session *R* | | #+name: bigcsv | #+begin_src R | bigcsv - Sys.glob(~/Downloads/*.csv) | #+end_src | | #+RESULTS: bigcsv | : /Users/cberry/Downloads/test-file.csv | | #+name: readbig | #+begin_src R :results output | system.time( | tmp - read.csv(bigcsv) | ) | | #+end_src | | #+RESULTS: readbig | : user system elapsed | : 5.679 0.306 6.002 | ` About the same as running from ESS. Not sure what to say. Looking for ways to troubleshoot or confirm. Since you can't confirm, any suggestions on where I should look for my issue? I can't explain it! All I know is that org chugs and chugs and the direct execution in ESS session is lightning fast. A few things to try in no particular order: o run top (or whatever equivalent is available on your OS) and see whether the CPU (or one of the CPUs) gets pegged at 100% utilization and stays there. If yes, that's an indication of an infinite loop somewhere. o run vmstat (or equivalent) and see if any of the counters are out of whack. That requires some experience though. o use elp-instrument-package to instrument org and run the test, getting a profile. I'm not sure whether the results will be useful, since you are going to interrupt the test when you run out of patience, but it cannot hurt and it might tell you something useful. o run your ECM on a different computer/OS/emacs installation. Being able to compare things side by side is often very useful. o Halve your file and run the test on each half (but that's probably not the problem given Chuck's results). o Reinstall org from scratch - you might have some corruption in one of the compiled files that's causing it to go into an infinite loop. o Turn on debug-on-quit, start your test, wait a bit and then interrupt it. Check the backtrace. Do it again and check whether the backtrace looks the same. That's often an indication of an infinite loop (inferring an infinite loop from a two element sample is statistically suspect of course, but surprisingly effective nevertheless). The point here is that the infinite loop is in emacs and the backtrace tells you something about the parties involved. These are obviously not independent and the results of one experiment will have to guide you in what you try next. Good luck, Nick
Re: [O] Bug? R: Org babel block execution *drastically* slower than in ESS session directly
Thomas S. Dye t...@tsdye.com wrote: Hi John, Have you tried wrapping your R read in system.time()? If you are right about :results silent eating up lots of time, then this should fix the problem. system.time yields just a bit of output, so shouldn't slow things down if writing out the data is indeed the problem as you suspect. I missed this in the thread so I thought I'd add one more thing to try: o run iostat (or equivalent) to check how your disks are doing. on linux, iostat is part of a package usually called sysstat. Nick John Hendy jw.he...@gmail.com writes: On Wed, Oct 31, 2012 at 3:12 PM, cbe...@tajo.ucsd.edu wrote: John Hendy jw.he...@gmail.com writes: On Wed, Oct 31, 2012 at 11:41 AM, span dir=ltrmailto: cbe...@tajo.ucsd.edu/span wrote: John Hendy mailto:jw.he...@gmail.com writes: I edited the subject to be more concise/clear.I let orgmode chug away on reading in some ~10-30mb csv files for nearly 30min. [rest deleted] You need an ECM.I did my best to provide one, other than the file, which I offered to provide if others requested that I upload it somewhere. Since you have done so, so have I: - https://docs.google.com/open?id=0BzQupOSnvw08WHdabHh5VVczRGM Let me know if that doesn#39;t work. I put it on Google docs and sometimes have issues with the sharing settings... Not an ECM in my book, but ... What else would you like? I provided: - the config - the data - how to [attempt to] reproduce - the org-mode text On my 4 year old MacBook: , | | #+PROPERTY: session *R* | | #+name: bigcsv | #+begin_src R | bigcsv - Sys.glob(~/Downloads/*.csv) | #+end_src | | #+RESULTS: bigcsv | : /Users/cberry/Downloads/test-file.csv | | #+name: readbig | #+begin_src R :results output | system.time( | tmp - read.csv(bigcsv) | ) | | #+end_src | | #+RESULTS: readbig | :user system elapsed | : 5.679 0.306 6.002 | ` About the same as running from ESS. Not sure what to say. Looking for ways to troubleshoot or confirm. Since you can't confirm, any suggestions on where I should look for my issue? I can't explain it! All I know is that org chugs and chugs and the direct execution in ESS session is lightning fast. FYI, dim(tmp) [1] 964805 9 Was that just to show the size, or is something wrong with my file? Chuck Thanks for attempting to replicate! John [ rest deleted ] On Wed, Oct 31, 2012 at 3:12 PM, cbe...@tajo.ucsd.edu wrote: John Hendy jw.he...@gmail.com writes: On Wed, Oct 31, 2012 at 11:41 AM, span dir=ltrmailto:cbe...@tajo.ucsd.edu/span wrote: John Hendy mailto:jw.he...@gmail.com writes: I edited the subject to be more concise/clear.I let orgmode chug away on reading in some ~10-30mb csv files for nearly 30min. [rest deleted] You need an ECM.I did my best to provide one, other than the file, which I offered to provide if others requested that I upload it somewhere. Since you have done so, so have I: - https://docs.google.com/open?id=0BzQupOSnvw08WHdabHh5VVczRGM Let me know if that doesn#39;t work. I put it on Google docs and sometimes have issues with the sharing settings... Not an ECM in my book, but ... What else would you like? I provided: - the config - the data - how to [attempt to] reproduce - the org-mode text On my 4 year old MacBook: , | | #+PROPERTY: session *R* | | #+name: bigcsv | #+begin_src R | bigcsv - Sys.glob(~/Downloads/*.csv) | #+end_src | | #+RESULTS: bigcsv | : /Users/cberry/Downloads/test-file.csv | | #+name: readbig | #+begin_src R :results output | system.time( | tmp - read.csv(bigcsv) | ) | | #+end_src | | #+RESULTS: readbig | : user system elapsed | : 5.679 0.306 6.002 | ` About the same as running from ESS. Not sure what to say. Looking for ways to troubleshoot or confirm. Since you can't confirm, any suggestions on where I should look for my issue? I can't explain it! All I know is that org chugs and chugs and the direct execution in ESS session is lightning fast. FYI, dim(tmp) [1] 964805 9 Was that just to show the size, or is something wrong with my file? Chuck Thanks for attempting to replicate! John [ rest deleted ] -- Thomas S. Dye http://www.tsdye.com
