[O] bug#10125: 24.0.91; package.el (org): Macros in tar packages & order of byte compilation
Glenn Morris writes: > Glenn Morris wrote: > >> fresh Emacs instance. There's no reason the "package manager" could not >> spawn a separate Emacs in batch-mode as a sub-job to do the compilation. > > Very lightly tested version: This uses the Emacs executable on the exec path, which might not be the correct one. I'm wary of making such a fundamental change to package.el without first exploring possible package-specific work-arounds. Could problematic packages use an eval-when-compile form to forcibly load the correct version of the required file?
[O] bug#10125: 24.0.91; package.el (org): Macros in tar packages & order of byte compilation
Glenn Morris wrote: > the package has defined. Though you would have to trust the package not > to do anything nasty. Of course, you already have to trust it since byte-compiling can run arbitrary code.
[O] bug#10125: 24.0.91; package.el (org): Macros in tar packages & order of byte compilation
Stefan Monnier wrote: > E.g. we could add to bytecomp.el the ability to force `require' to > reload a package if it's not already loaded from the file that > locate-library returns. That will probably work fine most of the time, but what if a package is restructed so that the feature names are different? Or a feature is removed? Simply starting a fresh Emacs seems fine to me. Though there is the issue of should it be a `-Q' one or not. BTW, I was also thinking that rather than simply byte-recompiling, package.el ought to check for a Makefile in the package directory, and if it finds one, call make to build the package according to the rules the package has defined. Though you would have to trust the package not to do anything nasty.
[O] bug#10125: 24.0.91; package.el (org): Macros in tar packages & order of byte compilation
Glenn Morris wrote: >> This uses the Emacs executable on the exec path, which might not be the >> correct one. > > Yes; I wondered if (car command-line-args) was a reliable way to find > the actual name of the running Emacs binary? Turns out I was looking for invocation-directory and invocation-name.
[O] bug#10125: 24.0.91; package.el (org): Macros in tar packages & order of byte compilation
Eli Zaretskii wrote: > I think package.el should test with featurep whether a version of a > package is already loaded, and refuse to load it into a running > session, or at least display a warning to that effect, suggesting to > restart Emacs. Yes, that might be better.
[O] bug#10125: 24.0.91; package.el (org): Macros in tar packages & order of byte compilation
Chong Yidong wrote: > This uses the Emacs executable on the exec path, which might not be the > correct one. Yes; I wondered if (car command-line-args) was a reliable way to find the actual name of the running Emacs binary?
[O] bug#10125: 24.0.91; package.el (org): Macros in tar packages & order of byte compilation
>>> fresh Emacs instance. There's no reason the "package manager" could not >>> spawn a separate Emacs in batch-mode as a sub-job to do the compilation. >> Very lightly tested version: > This uses the Emacs executable on the exec path, which might not be the > correct one. > I'm wary of making such a fundamental change to package.el without first > exploring possible package-specific work-arounds. Yes, this should not a candidate for 24.1. > Could problematic packages use an eval-when-compile form to forcibly > load the correct version of the required file? I think it's a sufficiently general problem that we might want to provide some special support for a good solution (tho again, I doubt there's something simple&safe enough to be a candidate for 24.1). E.g. we could add to bytecomp.el the ability to force `require' to reload a package if it's not already loaded from the file that locate-library returns. Stefan
[O] bug#10125: 24.0.91; package.el (org): Macros in tar packages & order of byte compilation
On Fri, Nov 25, 2011 at 04:31, Jambunathan K wrote: > > Stelian > >> I am sorry to be asking a stupid question, but then, wouldn't restart >> Emacs fix the issue and have the new compiled org files loaded? In my >> case, that didn't seem to happen either (even though load-library >> showed org-compat to be from ELPA). > > locate-library doesn't show what is *already* loaded. It will only shows > what *will* be loaded. > > The most important thing during package compilation is this: Make sure > there is no running instance of Org in any form. If this condition is > not satisfied then to-be-installed files will be compiled with old > definitions of macro. This is not what we want. We want the new files to > be compiled with new macro definitions. > > Do things work for you when you start a *minimal* Emacs and *then* do > M-x list-packages -> install? You don't have to apply Glenn's patch to > get the desired behaviour (I am assuming here that you are probably not > that comfortable working with patches). > I will try with a minimal Emacs first. You're right, I don't know my way around patches that well :-). I am on OSX and I suppose I could use something like homebrew for including the patch and building. > Jambunathan K. > -- >
[O] bug#10125: 24.0.91; package.el (org): Macros in tar packages & order of byte compilation
Glenn Glenn Morris writes: > Glenn Morris wrote: > >> fresh Emacs instance. There's no reason the "package manager" could not >> spawn a separate Emacs in batch-mode as a sub-job to do the >> compilation. I will let Stelian or any aggrieved future parties to test the patch. [...context switch...] I am wondering whether a knob to "not compile" installed packages is a good option in general. Can the autoloads be still generated even if the files themselves are not compiled? Jambunathan K.
[O] bug#10125: 24.0.91; package.el (org): Macros in tar packages & order of byte compilation
> From: Glenn Morris > Cc: Eli Zaretskii , 10...@debbugs.gnu.org, > stelian.ia...@gmail.com > Date: Thu, 24 Nov 2011 14:09:38 -0500 > > I guess nobody ever expected the package manager to be used to load a > different version of something that was already in Emacs. Indeed. Because doing this is asking for trouble: Emacs loads packages in a certain order to avoid several subtle problems; doing that in a different order defeats that. I think package.el should test with featurep whether a version of a package is already loaded, and refuse to load it into a running session, or at least display a warning to that effect, suggesting to restart Emacs.
[O] bug#10125: 24.0.91; package.el (org): Macros in tar packages & order of byte compilation
Stelian > I am sorry to be asking a stupid question, but then, wouldn't restart > Emacs fix the issue and have the new compiled org files loaded? In my > case, that didn't seem to happen either (even though load-library > showed org-compat to be from ELPA). locate-library doesn't show what is *already* loaded. It will only shows what *will* be loaded. The most important thing during package compilation is this: Make sure there is no running instance of Org in any form. If this condition is not satisfied then to-be-installed files will be compiled with old definitions of macro. This is not what we want. We want the new files to be compiled with new macro definitions. Do things work for you when you start a *minimal* Emacs and *then* do M-x list-packages -> install? You don't have to apply Glenn's patch to get the desired behaviour (I am assuming here that you are probably not that comfortable working with patches). Jambunathan K. --
[O] bug#10125: 24.0.91; package.el (org): Macros in tar packages & order of byte compilation
Stelian Iancu wrote: > I am sorry to be asking a stupid question, but then, wouldn't restart > Emacs fix the issue and have the new compiled org files loaded? No, because the files get compiled with a mix of old and new code loaded, so the compiled files are probably messed up. Restarting Emacs would not help with that. (You'll definitely need to restart Emacs if you had one version of Org loaded and want to switch to another.)
[O] bug#10125: 24.0.91; package.el (org): Macros in tar packages & order of byte compilation
On Thu, Nov 24, 2011 at 20:09, Glenn Morris wrote: > Jambunathan K wrote: > >> When compiling with package manager, the compilation happens from within >> a running Emacs session and very likely the "old" Org files are already >> loaded in to the runtime "inadvertently" by the user either by looking >> at the org agenda for the day or may be by just viewing an Org file or >> by the plain old (require 'org-whatever) out of habit in .emacs. > > There's your problem. The only way to reliably compile, especially > something where an old version might already be loaded, is to use a > fresh Emacs instance. There's no reason the "package manager" could not > spawn a separate Emacs in batch-mode as a sub-job to do the compilation. > > cc-mode tries to have some voodoo to get around this, but please, please > don't go down that road. > > I guess nobody ever expected the package manager to be used to load a > different version of something that was already in Emacs. > I am sorry to be asking a stupid question, but then, wouldn't restart Emacs fix the issue and have the new compiled org files loaded? In my case, that didn't seem to happen either (even though load-library showed org-compat to be from ELPA).
[O] bug#10125: 24.0.91; package.el (org): Macros in tar packages & order of byte compilation
Glenn Morris wrote: > fresh Emacs instance. There's no reason the "package manager" could not > spawn a separate Emacs in batch-mode as a sub-job to do the compilation. Very lightly tested version: *** lisp/emacs-lisp/package.el 2011-11-20 03:48:53 + --- lisp/emacs-lisp/package.el 2011-11-24 19:48:49 + *** *** 595,600 --- 595,612 (error "Package does not untar cleanly into directory %s/" dir (tar-untar-buffer)) + (defun package-compile (directory) + "Compile the Lisp files in DIRECTORY." + (with-current-buffer (get-buffer-create "*package-compile*") + (goto-char (point-max)) + (pop-to-buffer (current-buffer)) + (or (zerop (call-process "emacs" nil t t "--batch" "--eval" +(format + "(progn (setq load-path (cons \"%s\" load-path)) + (batch-byte-recompile-directory 0))" directory) +directory)) + (error "Compiling the package gave an error" + (defun package-unpack (name version) (let* ((dirname (concat (symbol-name name) "-" version)) (pkg-dir (expand-file-name dirname package-user-dir))) *** *** 603,610 (let* ((default-directory (file-name-as-directory package-user-dir))) (package-untar-buffer dirname) (package-generate-autoloads (symbol-name name) pkg-dir) ! (let ((load-path (cons pkg-dir load-path))) ! (byte-recompile-directory pkg-dir 0 t) (defun package--write-file-no-coding (file-name) (let ((buffer-file-coding-system 'no-conversion)) --- 615,621 (let* ((default-directory (file-name-as-directory package-user-dir))) (package-untar-buffer dirname) (package-generate-autoloads (symbol-name name) pkg-dir) ! (package-compile pkg-dir (defun package--write-file-no-coding (file-name) (let ((buffer-file-coding-system 'no-conversion)) *** *** 645,652 pkg-file nil nil nil 'excl)) (package-generate-autoloads file-name pkg-dir) ! (let ((load-path (cons pkg-dir load-path))) ! (byte-recompile-directory pkg-dir 0 t) (defmacro package--with-work-buffer (location file &rest body) "Run BODY in a buffer containing the contents of FILE at LOCATION. --- 656,662 pkg-file nil nil nil 'excl)) (package-generate-autoloads file-name pkg-dir) ! (package-compile pkg-dir (defmacro package--with-work-buffer (location file &rest body) "Run BODY in a buffer containing the contents of FILE at LOCATION.
[O] bug#10125: 24.0.91; package.el (org): Macros in tar packages & order of byte compilation
Jambunathan K wrote: > When compiling with package manager, the compilation happens from within > a running Emacs session and very likely the "old" Org files are already > loaded in to the runtime "inadvertently" by the user either by looking > at the org agenda for the day or may be by just viewing an Org file or > by the plain old (require 'org-whatever) out of habit in .emacs. There's your problem. The only way to reliably compile, especially something where an old version might already be loaded, is to use a fresh Emacs instance. There's no reason the "package manager" could not spawn a separate Emacs in batch-mode as a sub-job to do the compilation. cc-mode tries to have some voodoo to get around this, but please, please don't go down that road. I guess nobody ever expected the package manager to be used to load a different version of something that was already in Emacs.
Re: [O] bug#10125: 24.0.91; package.el (org): Macros in tar packages & order of byte compilation
> Org Mode files certainly have no dependency rules in lisp/Makefile.in. The Makefile - in devel repo of Orgmode - does define rules. Read on ... > So the question why the problem does not happen while compiling Org in > Emacs remains. I believe the way the files are compiled makes a substantial difference. When compiling with makefiles: The compilation happens with *minimal* emacs and in batch mode. --8<---cut here---start->8--- BATCH=$(EMACS) -batch -q -no-site-file -eval "(setq load-path (cons (expand-file-name \"./lisp/\") (cons \"$(lispdir)\" load-path)))" --8<---cut here---end--->8--- As can be seen above, any (require 'something) of macro files in the compiled elisp file has to be loaded from the development version itself. Furthermore there are dependencies like this in the Makefile: --8<---cut here---start->8--- lisp/org.elc: lisp/org-macs.el lisp/org-compat.el lisp/org-faces.el lisp/org-agenda.elc:lisp/org.el --8<---cut here---end--->8--- (I believe removing the dependencies from the Makefiles will still do the right thing because of the require directives in the compiled files will load the development version and not the system version) When compiling with package manager, the compilation happens from within a running Emacs session and very likely the "old" Org files are already loaded in to the runtime "inadvertently" by the user either by looking at the org agenda for the day or may be by just viewing an Org file or by the plain old (require 'org-whatever) out of habit in .emacs. While reporting macro issues, users never say whether they were already running Org when they were trying to fetch and compile a new Org. They think it is immaterial. I believe it matters If "old" org and hence "old" org-macs is already loaded in the environment when the package is installed, any subsequent (require 'something) will be essentially no-ops. (Can you confirm this?) What ideally should happen is that during package compilation, a require should *forcibly* load from the compiled package and not merely check for availability of a feature symbol. >> 2. While building from ELPA, the compilation order seems to be >> alphabetical. So the files get compiled bass ackwards. For example, >> org-macs.el gets compiled after org-agenda.el. >> >> In summary, there needs to be a way to specify the order in which files >> are compiled in a multifile tar. > > This was discussed several time, in the context of recompiling > multiple Lisp files while building Emacs, and the decision till now > was to ignore the issue. While at least in principle one could write > a Lisp program that would analyze the various `require' and `load' > calls (possibly as side effect of byte compilation, like GCC does), > and generate Makefile rules with correct prerequisites, this is a > non-trivial project. I have tried giving an explanation in the earlier paragraph. > One simple band-aid is to remove all the *.elc files before > byte-compiling after resync. This prolongs the compilation, but the > results are predictably correct. We are talking of "automatic" compilation by package manager. What you say applies to hand compilation via makefiles. >> A supplementary question to (2): >> >> During the package compilation, when encountering (require >> 'some-org-library-with-macros) does the library get loaded from *within* >> the tarball or from the *emacs core*. > > Whichever is found first along load-path, I think. See `openp'. What does package manager do? --
Re: [O] bug#10125: 24.0.91; package.el (org): Macros in tar packages & order of byte compilation
> From: Jambunathan K > Date: Thu, 24 Nov 2011 17:42:38 +0530 > Cc: emacs-orgmode@gnu.org, Stelian Iancu > > 1. While building via Makefile, there is an implicit dependency that is > *enforced* via make rules and files with macro definitions are compiled > ahead of their consumers. That's not true: there are no dependencies defined in lisp/Makefile.in in the Emacs source tree for Lisp files, with a very few exceptions. Org Mode files certainly have no dependency rules in lisp/Makefile.in. So the question why the problem does not happen while compiling Org in Emacs remains. > 2. While building from ELPA, the compilation order seems to be > alphabetical. So the files get compiled bass ackwards. For example, > org-macs.el gets compiled after org-agenda.el. > > In summary, there needs to be a way to specify the order in which files > are compiled in a multifile tar. This was discussed several time, in the context of recompiling multiple Lisp files while building Emacs, and the decision till now was to ignore the issue. While at least in principle one could write a Lisp program that would analyze the various `require' and `load' calls (possibly as side effect of byte compilation, like GCC does), and generate Makefile rules with correct prerequisites, this is a non-trivial project. One simple band-aid is to remove all the *.elc files before byte-compiling after resync. This prolongs the compilation, but the results are predictably correct. > A supplementary question to (2): > > During the package compilation, when encountering (require > 'some-org-library-with-macros) does the library get loaded from *within* > the tarball or from the *emacs core*. Whichever is found first along load-path, I think. See `openp'.
