Re: [O] Opening (for the first time) a 10-line Org doc takes 4 seconds

[Achim Gratz]

Bernt Hansen writes:
 For the record here is the output on my Emacs:

 GNU Emacs 23.2.1 (i486-pc-linux-gnu, GTK+ Version 2.20.1) of 2013-01-08
 on murphy, modified by Debian

 Current git head:  release_7.9.3d-826-gbe0d87d

I can't check Emacs 23.2 right now, but Emacs 23.4 comes with Org
version 6.33x installed.  Just looking at what loaddefs.el has defined
as autoloads from org.el:

;;;### (autoloads (org-customize org-reload org-require-autoloaded-modules
;;  org-submit-bug-report org-cycle-agenda-files org-iswitchb
;;  org-map-entries org-open-link-from-string org-open-at-point-global
;;  org-insert-link-global org-store-link org-run-like-in-org-mode
;;  turn-on-orgstruct++ turn-on-orgstruct orgstruct-mode org-global-cycle
;;  org-mode) org org/org.el (20712 11830))
;;; Generated autoloads from org/org.el

Now compare that with the same section generated for a recent Git version:

;;;### (autoloads (org-customize org-reload org-submit-bug-report
;;  org-cycle-agenda-files org-switchb org-open-link-from-string
;;  org-open-at-point-global org-insert-link-global org-store-link
;;  org-run-like-in-org-mode turn-on-orgstruct++ turn-on-orgstruct
;;  orgstruct-mode org-global-cycle org-cycle org-mode 
org-clock-persistence-insinuate
;;  turn-on-orgtbl org-version org-babel-do-load-languages) org
;;  org.el (20726 54443))
;;; Generated autoloads from org.el

So yes, org-mode is autoloaded, but some other stuff that should be
isn't.  Things get more interesting with the autoloads that are defined
in the earlier version, but not anymore in the newer.  In the case of
org-iswitchb the autoload still works since it is defaliased to
org-switchb in the same file.  If it were removed, you'd get an error
when trying to use it, the same if it was moved to another file (since
the autoload definition says to look in org it will not look in
org-whatever-i-have-been-moved-to.  Also if the file in question has
been renamed (org-exp-blocks exists in 6.33x, but not in a current
version), then those autoloads will also not work.

Having Emacs pull in the current autoload definitions will ensure that
all current autoloads are pointing to the correct file, so if you are
using only those everything is fine.  It still leaves some definitions
for functions that may not exist anymore, but you would not be able to
use them anyway, so while not squeaky-clean, it will work in practise.
There is one problem that all this still doesn't solve: if a previously
autoloaded function has no current autoload definition and the file it
used to reside in has been renamed, then Emacs will happily descend into
the load-path to load the function along with that old file.  This is
why Emacs itself should provide a way to de-activate a built-in package
when a newer version has been installed so that these stale autoload
definitions aren't present after initialization (it is probably possible
to traverse load-history to remove superseded autoload definitions, but
there is no ready-made function to do this AFAIK).


Regards,
Achim.
-- 
+[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables

Re: [O] Opening (for the first time) a 10-line Org doc takes 4 seconds

[Bernt Hansen]

Bastien b...@altern.org writes:

 Achim Gratz strom...@nexgo.de writes:

 Bastien writes:
 What earlier version of Emacs does not behave like the current Emacs 24
 trunk with respect to this issue,

 Emacs 22 doesn't have any autoloads for Org at all, so nothing works.

 Emacs 23 has some autoload definitions from its built-in Org version(s),
 but there have been many changes to function/library name pairs in later
 versions of Org, so I'd generally expect this to be broken, but maybe
 not overtly if org gets loaded first (but org-version is not present as
 an autoloaded function for example).

 Emacs 24.x should be mostly clean, but there have been some additional
 autoloads in Babel that would be missing.

 All this I know -- I'm a bit stubborn but I read and learn :)

 Let me restate my question.

 Let's say that someone uses Emacs 23.2 with Org from Git.  This user
 has this line in her ~/.emacs.el before any other Org configuration:

   (add-to-list 'load-path ~/git/org-mode/lisp/)

 My assumption is that C-h f org-mode RET will show that org-mode is
 an autoloaded function in ~/git/org-mode/lisp/ -- as it does for
 me (using Emacs 24).

 Is that so?  (I can't test Org with Emacs 23.2 right now.)

For the record here is the output on my Emacs:

GNU Emacs 23.2.1 (i486-pc-linux-gnu, GTK+ Version 2.20.1) of 2013-01-08
on murphy, modified by Debian

Current git head:  release_7.9.3d-826-gbe0d87d

,[ minimal.emacs ]
| (add-to-list 'load-path (expand-file-name ~/git/org-mode/lisp))
| (add-to-list 'auto-mode-alist '(\\.\\(org\\|org_archive\\|txt\\)$ . 
org-mode))
| (require 'org)
| 
| (global-set-key \C-cl 'org-store-link)
| (global-set-key \C-ca 'org-agenda)
| (global-set-key \C-cb 'org-iswitchb)
`

,[ minimal-emacs ]
| #!/bin/sh
| TESTEL=
| TESTFILE=/tmp/test.el
| if test -e $TESTFILE
| then
|   TESTEL=-l /tmp/test.el
| fi
| emacs -q -l ~/minimal.emacs $TESTEL $1
`

,[ C-h f org-mode RET ]
| org-mode is an interactive compiled Lisp function in `org.el'.
| 
| (org-mode)
| 
| Outline-based notes management and organizer, alias
| Carsten's outline-mode for keeping track of everything.
| 
| Org-mode develops organizational tasks around a NOTES file which
| contains information about projects as plain text.  Org-mode is
| implemented on top of outline-mode, which is ideal to keep the content
| of large files well structured.  It supports ToDo items, deadlines and
| time stamps, which magically appear in the diary listing of the Emacs
| calendar.  Tables are easily created with a built-in table editor.
| Plain text URL-like links connect to websites, emails (VM), Usenet
| messages (Gnus), BBDB entries, and any files related to the project.
| For printing and sharing of notes, an Org-mode file (or a part of it)
| can be exported as a structured ASCII or HTML file.
| 
| The following commands are available:
| 
| key binding
| --- ---
| 
| C-a   org-beginning-of-line
| C-c   Prefix Command
| C-e   org-end-of-line
| TAB   org-cycle
| C-j   org-return-indent
| C-k   org-kill-line
| RET   org-return
| C-y   org-yank
| ESC   Prefix Command
| | org-force-self-insert
| C-#   org-table-rotate-recalc-marks
| C-'   org-cycle-agenda-files
| C-,   org-cycle-agenda-files
| C-S-downorg-shiftcontroldown
| C-S-leftorg-shiftcontrolleft
| C-S-return  org-insert-todo-heading-respect-content
| C-S-right   org-shiftcontrolright
| C-S-up  org-shiftcontrolup
| C-returnorg-insert-heading-respect-content
| C-tab   org-force-cycle-archived
| M-S-downorg-shiftmetadown
| M-S-leftorg-shiftmetaleft
| M-S-return  org-insert-todo-heading
| M-S-right   org-shiftmetaright
| M-S-up  org-shiftmetaup
| M-down  org-metadown
| M-left  org-metaleft
| M-returnorg-meta-return
| M-right org-metaright
| M-uporg-metaup
| S-down  org-shiftdown
| S-iso-lefttab   org-shifttab
| S-left  org-shiftleft
| S-returnorg-table-copy-down
| S-right org-shiftright
| S-tab   org-shifttab
| S-uporg-shiftup
| backtab org-shifttab
| remap   Prefix Command
| tab org-cycle
| 
| M-TAB pcomplete
| M-RET org-insert-heading
| C-M-t org-transpose-element
| M-a   org-backward-sentence
| M-e   org-forward-sentence
| M-h   org-mark-element
| M-{   org-backward-element
| M-}   org-forward-element
| 
| C-c C-a   org-attach
| C-c C-b   org-backward-heading-same-level
| C-c C-c   org-ctrl-c-ctrl-c
| C-c C-d   org-deadline
| C-c C-e   org-export
| C-c C-f   org-forward-heading-same-level
| C-c C-j   org-goto
| C-c C-k   org-kill-note-or-show-branches
| C-c C-l   org-insert-link
| C-c RET   

Re: [O] Opening (for the first time) a 10-line Org doc takes 4 seconds

[Bastien]

Hi Achim,

Achim Gratz strom...@nexgo.de writes:

 You don't need this, there is (load org-loaddefs.el t t) in org.el.

 Well, yes he does when he is using Org from Git.

I use Org from Git and I don't have 

  (load org-loaddefs.el t t)

in my .emacs.el.  I simply have:

  (add-to-list 'load-path ~/install/git/org-mode/lisp/)

before any Org configuration and *all* autoloads are correct.

My understanding is that users who don't use make install
will have to do the same and add the correct load-path, which
will in turn produce the correct autoloads.  Am I right?

-- 
 Bastien

Re: [O] Opening (for the first time) a 10-line Org doc takes 4 seconds

[Achim Gratz]

Bastien writes:
 I use Org from Git and I don't have 

   (load org-loaddefs.el t t)

 in my .emacs.el.  I simply have:

   (add-to-list 'load-path ~/install/git/org-mode/lisp/)

 before any Org configuration and *all* autoloads are correct.

Only with a recent Emacs 24 and it really only works for code paths that
load org.el early enough.  I cannot stress often enough that the
autoload definitions are not absolute and _this_ is the part that makes
things really tricky.  They will follow changes in load-path, but only
if the library name stays the same.  Again, the example to try is this:

Move an autoloaded function from one file to another in the Git
directory.  Re-generate the autoload file.  Re-start Emacs, try to call
that function and see Emacs try to load the _old_ file (because that's
the autoload definition it has at that point).  Re-start Emacs again,
load org-loaddefs with the proper load-path in place and see that it
loads the new file.

 My understanding is that users who don't use make install
 will have to do the same and add the correct load-path, which
 will in turn produce the correct autoloads.  Am I right?

I'm not sure I understand you correctly.  All of the following
conditions must hold for this to work:

- the load-path is set up to point to the Git work-tree first and no
  functions in the org namespace have been called before doing so

- the org-loaddefs.el in the Git worktree is up-to-date

- none of the invoked functions have their library-name changed in the
  Git worktree

- all code-paths must trigger loading of org (and thus org-loaddefs)
  before invoking further autoloads


Regards,
Achim.
-- 
+[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: [O] Opening (for the first time) a 10-line Org doc takes 4 seconds

[Bastien]

Achim Gratz strom...@nexgo.de writes:

 Bastien writes:
 I use Org from Git and I don't have 

   (load org-loaddefs.el t t)

 in my .emacs.el.  I simply have:

   (add-to-list 'load-path ~/install/git/org-mode/lisp/)

 before any Org configuration and *all* autoloads are correct.

 Only with a recent Emacs 24 and it really only works for code paths that
 load org.el early enough.

What earlier version of Emacs does not behave like the current Emacs 24
trunk with respect to this issue, and more importantly, *why* is it so?

Thanks,

-- 
 Bastien

Re: [O] Opening (for the first time) a 10-line Org doc takes 4 seconds

[Bastien]

Achim Gratz strom...@nexgo.de writes:

 I'm not sure I understand you correctly.  All of the following
 conditions must hold for this to work:

 - the load-path is set up to point to the Git work-tree first and no
   functions in the org namespace have been called before doing so

 - the org-loaddefs.el in the Git worktree is up-to-date

 - none of the invoked functions have their library-name changed in the
   Git worktree

Okay.  This are normal conditions to me, and we can document this if
necessary.

 - all code-paths must trigger loading of org (and thus org-loaddefs)
   before invoking further autoloads

I'm not sure what you mean by code-paths.

You mean that no non-org library should call an org-* function before 
the load-path for org-mode is set correctly?

-- 
 Bastien

Re: [O] Opening (for the first time) a 10-line Org doc takes 4 seconds

[Achim Gratz]

Bastien writes:
 What earlier version of Emacs does not behave like the current Emacs 24
 trunk with respect to this issue,

Emacs 22 doesn't have any autoloads for Org at all, so nothing works.

Emacs 23 has some autoload definitions from its built-in Org version(s),
but there have been many changes to function/library name pairs in later
versions of Org, so I'd generally expect this to be broken, but maybe
not overtly if org gets loaded first (but org-version is not present as
an autoloaded function for example).

Emacs 24.x should be mostly clean, but there have been some additional
autoloads in Babel that would be missing.

 and more importantly, *why* is it so?

Uh… iff any autoload definitions exist, they may or may not be
associated with the correct library name.  They also may or may not have
the correct doc string or number of parameters.  You have to load these
definitions from the correct files, they don't get picked up
automatically.  The files that get loaded at startup are defined in
prefix/emacs-version/lisp/loadup.el (this is either loaddefs.el or
ldefs-boot.el, depending on whether the bootstrap or the normal Emacs is
started).


Regards,
Achim.
-- 
+[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+

DIY Stuff:
http://Synth.Stromeko.net/DIY.html

Re: [O] Opening (for the first time) a 10-line Org doc takes 4 seconds

[Bastien]

Achim Gratz strom...@nexgo.de writes:

 Bastien writes:
 What earlier version of Emacs does not behave like the current Emacs 24
 trunk with respect to this issue,

 Emacs 22 doesn't have any autoloads for Org at all, so nothing works.

 Emacs 23 has some autoload definitions from its built-in Org version(s),
 but there have been many changes to function/library name pairs in later
 versions of Org, so I'd generally expect this to be broken, but maybe
 not overtly if org gets loaded first (but org-version is not present as
 an autoloaded function for example).

 Emacs 24.x should be mostly clean, but there have been some additional
 autoloads in Babel that would be missing.

All this I know -- I'm a bit stubborn but I read and learn :)

Let me restate my question.

Let's say that someone uses Emacs 23.2 with Org from Git.  This user
has this line in her ~/.emacs.el before any other Org configuration:

  (add-to-list 'load-path ~/git/org-mode/lisp/)

My assumption is that C-h f org-mode RET will show that org-mode is
an autoloaded function in ~/git/org-mode/lisp/ -- as it does for
me (using Emacs 24).

Is that so?  (I can't test Org with Emacs 23.2 right now.)

-- 
 Bastien

Re: [O] Opening (for the first time) a 10-line Org doc takes 4 seconds

[Bastien]

Hi Sébastien,

Sebastien Vauban
wxhgmqzgwmuf-genee64ty+gs+fvcfc7...@public.gmane.org writes:

 OK. And `org.el' is loaded[1] because of the major-mode association to the
 `.org' files, then, right?

Right.

 [1] BTW, why not a `require' instead?

About (load org-loaddefs.el t t) ?

Because it is auto-generated.

-- 
 Bastien

Re: [O] Opening (for the first time) a 10-line Org doc takes 4 seconds

[Sebastien Vauban]

Hi Bastien,

Bastien wrote:
 Sebastien Vauban writes:

 (if (locate-library org-loaddefs)
 (require 'org-loaddefs)
^

 You don't need this, there is (load org-loaddefs.el t t) in org.el.

OK. And `org.el' is loaded[1] because of the major-mode association to the
`.org' files, then, right?

Anyway, that does not have impacts here.

 Are your org elisp files compiled?

Nope. Never.

 What results do you have without the advice on `require'?

I've now ran 5 tests with each config:

| without advice | with advice |
|+-|
|   3.78 |3.95 |
|   3.73 |3.86 |
|   3.76 |3.83 |
|   3.80 |3.83 |
|   3.81 |3.94 |
|+-|
|   3.78 |3.88 |
#+TBLFM: @7$1..@7$2=vmean(@I..@-I);%.2f

That is less than 3% time penalty for the logs done during the execution of
all the `require'.

Best regards,
  Seb

[1] BTW, why not a `require' instead?

-- 
Sebastien Vauban

Re: [O] Opening (for the first time) a 10-line Org doc takes 4 seconds

[Achim Gratz]

Bastien writes:
 (if (locate-library org-loaddefs)
 (require 'org-loaddefs)
^

 You don't need this, there is (load org-loaddefs.el t t) in org.el.

Well, yes he does when he is using Org from Git.  It's a wash when he's
using builtin Org from Emacs and potentially wrong when he's using Org
from ELPA, depending on if he does it before or after
package-initialize.

You've expressed before that you want to know why this is so.  As a
reminder, Org from Git currently has all its autoload definitions in
org-loaddefs.  Lets look at this for all possible Emacs versions:

XEmacs and Emacs 22 don't come with a builtin Org version.  You have no
autoloads including none for the entry point for the major mode until
you load org-loaddefs (or org).

Emacs 23, 24.1 and 24.2 comes with older versions of Org that don't even
have org-loaddefs.  All autoloads for the builtin version are in
loaddefs from Emacs and they are already loaded when the user
initialization stuff is run.  Some things in those autoloads happen to
point to the correct libraries, but only by chance.  Generally one has
to assume that the autoloads that are present have the wrong docstring,
point to the wrong file and present an outdated API.  To fix this, one
must load org-loaddefs (or org).

Emacs 24.3 will come with a newer version of Org, but still older than
the one from Git.  Only part of the autoloads are present in loaddefs,
the rest would be pulled in from the bundled org-loaddefs when org is
loaded.  That is the reason why one does not need to do that when only
using the builtin Org version.  Again however, these builtin autoload
definitions from loaddefs must be presumed outdated for Org from Git, so
it is still necessary to load org-loaddefs (or org) in the initalization
code to make sure the correct first-level autoloads take precedence even
when it is assured that all autoload code-paths also trigger loading of
org (I don't think that this can be easily guaranteed and I know of no
tool that would allow one to analyse).

Last but not least, if one uses Org from ELPA, then it must be arranged
that org-loaddefs gets loaded only after package-initialize, otherwise
you'll end up with the correct first-level autoloads, but outdated
second-level autoloads (that get replaced with the current ones when org
gets loaded).


Regards,
Achim.
-- 
+[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]+

DIY Stuff:
http://Synth.Stromeko.net/DIY.html

[O] Opening (for the first time) a 10-line Org doc takes 4 seconds

[Sebastien Vauban]

Hello,

Looking at how to improve my Org config, regarding speed, I notice that it
takes 4.22 s to open a simple 10-line Org file *for the first time* (when Org
wasn't loaded yet). The reason is that it requires a big amount of packages,
those requiring themselves many others, and so on.

Do you find such a time normal, or too much?

How to redo the test:

emacs -q -l .emacs-minimal.el

where `.emacs-minimal.el' is:

--8---cut here---start-8---
(message Loading Minimal Emacs...)

(defconst em/emacs-load-time-start (float-time))

;; change the pathnames appropriately!
(add-to-list 'load-path (expand-file-name ~/src/org-mode/lisp))
(add-to-list 'load-path (expand-file-name ~/src/org-mode/contrib/lisp))

(add-to-list 'auto-mode-alist '(\\.\\(org\\|org_archive\\|txt\\)$ . org-mode))
(if (locate-library org-loaddefs)
(require 'org-loaddefs)
  (require 'org-install))

(defadvice require (around require-around)
  Leave a trace of packages being loaded.
  (let* ((feature (ad-get-arg 0))
 (require-depth (or (and (boundp 'require-depth) require-depth)
0))
 (prefix (concat (make-string (* 2 require-depth) ? ) )))
(cond ((featurep feature)
   (message %s%s - %s... Done ;; already loaded
prefix
(if load-file-name
(file-name-base load-file-name)
  nil?)
feature)
   (setq ad-return-value feature))
  (t
   (let ((lvn/time-start))
 (message %s%s - %s...
  prefix
  (if load-file-name
  (file-name-base load-file-name)
nil?)
  feature)
 (setq lvn/time-start (float-time))
 (let ((require-depth (1+ require-depth)))
   ad-do-it))

(ad-activate 'require)

(find-file TODO.org)

(message Loading Minimal Emacs... Done (in %.2f s)
 (- (float-time) em/emacs-load-time-start))
--8---cut here---end---8---

and `TODO.org' is some simple Org file such as:

--8---cut here---start-8---
#+TITLE: Simple Org file
#+LANGUAGE:  en-us

* Emacs

** TODO Improve performance

Check out how to improve overall perf by putting chunks of code inside
(eval-after-load) forms.
--8---cut here---end---8---

That results in all the following packages to be loaded:

--8---cut here---start-8---
Loading Minimal Emacs...
.emacs-minimal - help-fns... Done [2 times]
org - cl...
  cl - cl-lib... Done
  cl - macroexp... Done
org - gnus-sum...
  gnus-sum - gnus...
gnus - wid-edit...
gnus - mm-util...
  mm-util - mail-prsvr...
  mm-util - timer... Done
gnus - nnheader...
  nnheader - mail-utils...
  nnheader - mm-util... Done
  nnheader - gnus-util...
gnus-util - time-date... Done
gnus - gnus-util... Done
gnus - nnheader... Done
gnus - gnus-ems...
  gnus-sum - gnus-group...
gnus-group - gnus... Done
gnus-group - gnus-start...
  gnus-start - gnus... Done
  gnus-start - gnus-win...
gnus-win - gnus... Done
gnus-win - gnus-util... Done
  gnus-start - gnus-int...
gnus-int - gnus... Done
gnus-int - message...
  message - mailheader...
  message - gmm-utils...
  message - mail-utils... Done
  message - mailabbrev...
  message - mail-parse...
mail-parse - mail-prsvr... Done
mail-parse - ietf-drums...
  ietf-drums - mm-util... Done
mail-parse - rfc2231...
  rfc2231 - ietf-drums... Done
  rfc2231 - rfc2047...
rfc2047 - mm-util... Done
rfc2047 - ietf-drums... Done
rfc2047 - mail-prsvr... Done
rfc2047 - rfc2045...
  rfc2045 - ietf-drums... Done
mail-parse - rfc2047... Done
mail-parse - rfc2045... Done
  message - mml...
mml - mm-util... Done
mml - mm-bodies...
  mm-bodies - mm-util... Done
  mm-bodies - rfc2047... Done
  mm-bodies - mm-encode...
mm-encode - mail-parse... Done
mml - mm-encode... Done
mml - mm-decode...
  mm-decode - mail-parse... Done
  mm-decode - mm-bodies... Done
mml - mml-sec...
  message - rfc822...
  message - format-spec...
  cl-macs - cl-lib... Done
  cl-macs - macroexp... Done
  cl-macs - gv...
gv - macroexp... Done
  message - idna...
gnus-int - gnus-range...
  gnus-start - gnus-spec...
gnus-spec - gnus... Done
  gnus-start - gnus-range... Done
  gnus-start - gnus-util... Done

Re: [O] Opening (for the first time) a 10-line Org doc takes 4 seconds

[Bastien]

Hi Sébastien,

Sebastien Vauban
wxhgmqzgwmuf-genee64ty+gs+fvcfc7...@public.gmane.org writes:

 (if (locate-library org-loaddefs)
 (require 'org-loaddefs)
   ^

You don't need this, there is (load org-loaddefs.el t t) in org.el.

Are your org elisp files compiled?

What results do you have without the advice on `require'?

-- 
 Bastien