[O] Bug?/performance issue in org-icalendar-export-current-agenda

2014-08-26 Thread Richard Lawrence 
Hi all,

I pulled from master recently (for the first time since the spring), and
since the update, I have noticed a performance/lockup issue.  The
problem occurs when exporting an agenda view to iCalendar format.

My Emacs version is 23.4.1 and my Org version is 8.3beta (I am
currently on commit 767895...)
 
I have traced the problem down to a call to
`org-icalendar-export-current-agenda', though I don't know if the
problem is in this function or in another function called by it.
Somewhere during the execution of this function, Emacs goes to 100% CPU
usage (on one core) and memory usage starts growing until I kill the
process.
 
Here's what my setup looks like.  I have just one entry in
`org-agenda-custom-commands' that defines the files field.  It looks
like:

 (Z Export to iCalendar file tags +event|+appointment nil
  (~/Documents/website/public_html/lib/attachments/calendar.ics)

I normally only call this command via a cron job that exports my agenda
files to .ics, which until recently was working fine.  But I have
noticed I can reproduce the lockup interactively by building an agenda
view with this command, and then calling
`org-icalendar-export-current-agenda' on it (which is also what happens,
a few steps down the call chain, during the cron job).

I have run some tests by manually setting org-agenda-files, then
building this agenda and eval'ing

(org-icalendar-export-current-agenda /tmp/agenda.ics) 

in the agenda buffer.  The results:

1) Using just a simple three-entry test file, the export completes quickly
   and I don't see the lockup.  
2) Using an agenda file which contains plenty of real data
   (about 1000 lines) but no headlines tagged event or appointment
   (i.e., a file for which this agenda view is empty), the call to
   `org-icalendar-export-current-agenda' takes about 5 seconds to
   complete, even though there are no entries to export.  During those
   5 seconds, the Emacs process goes to 100% CPU usage.
3) Using a different agenda file (of about 4000 lines) which contains
   about 75 matching headlines, the call to
   `org-icalendar-export-current-agenda' does not complete within
   several minutes, during which time I see 100% CPU usage and rapid
   memory usage growth.  

So it looks to me like maybe some part of the export process uses
exponentially more time and memory, depending on the input, since a
minimal test case does not produce the problem.

What's the next step for debugging this?  Please let me know if I can
provide more information.  

Thanks! 

-- 
Best,
Richard

Re: [O] Bug?/performance issue in org-icalendar-export-current-agenda

2014-08-26 Thread OSiUX 
El mar, 26 ago 2014, Richard Lawrence decía:

 Hi all,
 
 I pulled from master recently (for the first time since the spring), and
 since the update, I have noticed a performance/lockup issue.  The
 problem occurs when exporting an agenda view to iCalendar format.
 
 My Emacs version is 23.4.1 and my Org version is 8.3beta (I am
 currently on commit 767895...)
  
 I have traced the problem down to a call to
 `org-icalendar-export-current-agenda', though I don't know if the
 problem is in this function or in another function called by it.
 Somewhere during the execution of this function, Emacs goes to 100% CPU
 usage (on one core) and memory usage starts growing until I kill the
 process.
  
 Here's what my setup looks like.  I have just one entry in
 `org-agenda-custom-commands' that defines the files field.  It looks
 like:
 
  (Z Export to iCalendar file tags +event|+appointment nil
   (~/Documents/website/public_html/lib/attachments/calendar.ics)
 
 I normally only call this command via a cron job that exports my agenda
 files to .ics, which until recently was working fine.  But I have
 noticed I can reproduce the lockup interactively by building an agenda
 view with this command, and then calling
 `org-icalendar-export-current-agenda' on it (which is also what happens,
 a few steps down the call chain, during the cron job).
 
 I have run some tests by manually setting org-agenda-files, then
 building this agenda and eval'ing
 
 (org-icalendar-export-current-agenda /tmp/agenda.ics) 
 
 in the agenda buffer.  The results:
 
 1) Using just a simple three-entry test file, the export completes quickly
and I don't see the lockup.  
 2) Using an agenda file which contains plenty of real data
(about 1000 lines) but no headlines tagged event or appointment
(i.e., a file for which this agenda view is empty), the call to
`org-icalendar-export-current-agenda' takes about 5 seconds to
complete, even though there are no entries to export.  During those
5 seconds, the Emacs process goes to 100% CPU usage.
 3) Using a different agenda file (of about 4000 lines) which contains
about 75 matching headlines, the call to
`org-icalendar-export-current-agenda' does not complete within
several minutes, during which time I see 100% CPU usage and rapid
memory usage growth.  
 
 So it looks to me like maybe some part of the export process uses
 exponentially more time and memory, depending on the input, since a
 minimal test case does not produce the problem.

 What's the next step for debugging this?  Please let me know if I can
 provide more information.  
 
 Thanks! 
 
 -- 
 Best,
 Richard

I use a bash script: =agenda-ascii= [0] to
reduce the list of file to export, first =grep=
current =year-month= pattern in the *.org
files:

  #+BEGIN_EXAMPLE
ORG_FILES=$($GREP -l $YEAR-$MONTH *.org)
echo $ORG_FILES

if [ $EXPORT_ICS = TRUE ]
then
for AGENDA in $(echo $ORG_FILES)
do
echo EXPORT $AGENDA ...
ARCHIVO=$DIR/$AGENDA
$EMACS --batch -q --directory=${DIR} --visit=${ARCHIVO} --eval 
'(org-export-icalendar-this-file)' /dev/null 21
done
fi
  #+END_EXAMPLE

Actually the process, cosume 42%-97% CPU, for
526 appoinments, takes 0m 30.396s

  Org-mode version 8.3beta (release_8.3beta-264-g7a5f3f
  GNU Emacs 24.3.1 (x86_64-pc-linux-gnu) of 2014-06-06 on barber, modified by 
Debian


[0] https://github.com/osiris/org-bash-utils/blob/master/agenda-ascii

-- 

::

  Osiris Alejandro Gomez (OSiUX) os...@osiux.com.ar
  DC44 95D2 0D5D D544 FC1A F00F B308 A671 9237 D36C
  http://www.osiux.com.ar http://www.altermundi.net


signature.asc
Description: Digital signature

Re: [O] Bug?/performance issue in org-icalendar-export-current-agenda

2014-08-26 Thread Nicolas Goaziou 
Hello,

Richard Lawrence richard.lawre...@berkeley.edu writes:

 I pulled from master recently (for the first time since the spring), and
 since the update, I have noticed a performance/lockup issue.  The
 problem occurs when exporting an agenda view to iCalendar format.

 My Emacs version is 23.4.1 and my Org version is 8.3beta (I am
 currently on commit 767895...)
  
 I have traced the problem down to a call to
 `org-icalendar-export-current-agenda', though I don't know if the
 problem is in this function or in another function called by it.
 Somewhere during the execution of this function, Emacs goes to 100% CPU
 usage (on one core) and memory usage starts growing until I kill the
 process.

Interestingly, I introduced a patch that should speed up this function
yesterday. You may want to update Org (the patch landed after 767895)
and try again.

If it is still suspiciously slow, please use profiler (`profiler-start')
and report the results here.


Regards,

-- 
Nicolas Goaziou

Re: [O] Bug?/performance issue in org-icalendar-export-current-agenda

2014-08-26 Thread Richard Lawrence 
Nicolas Goaziou m...@nicolasgoaziou.fr writes:

 Interestingly, I introduced a patch that should speed up this function
 yesterday. You may want to update Org (the patch landed after 767895)
 and try again.

Indeed!  I just pulled again and the problem is gone.  The export runs
much more reasonably now, completing in a few seconds even with all my
agenda files in the mix.  Thanks!

Best,
Richard