Re: feature request - order only deps
On 4.2.2013 01:09, Sebastian Pipping wrote: On 04.02.2013 00:13, Matěj Týč wrote: On Ne, 2013-02-03 at 23:40 +0100, Sebastian Pipping wrote: To my understanding, it would have to be optional and off by default to not break other cases that are currently supported. Think of something like ... ... Dirs: dir1 dir2 dir%: mkdir $@ dir2/foo: dir1/bar touch $@ dir1/bar: | Dirs touch $@ .PHONY: Dirs If you make 'dir1/bar', then remove 'dir2' and decide to make 'dir2/foo', you appreciate that 'dir2' is remade because 'dirs' is an order-only dep of 'bar' and it is remade as soon as make realizes that 'dirs' is not complete because 'dir2' is missing. So although this can be considered as the Makefile bug, cases like this can exist, so the proposed change would break this behavior (I propose that the 'Dirs' order-only dep is ignored because dir1/bar is all right and dir2/foo says that it depends only on dir1/bar) If I am not mistaken, you are saying that: If 1. an order-only dependency is missing and 2. the target is not re-built (i.e. neither missing or older than normal prerequisites the order-only dependency should not be built. Thank you for your reply,and yes, that's it indeed. I'm starting to see why you want support for that. Thank you, this sounds good. I think that this is a nice idea, since 'make' doesn't play well with filters that can process batches of files at once, the server process is a good way to reduce overhead of processing those files one by one and you may end up needing a cache quite soon :-) Actually I might be wrong, but I think that if I have a program 'filter' capable of processing in01 in02 in03 ... files to out01 out02 out03 ... etc., it is not possible to tell this to 'make', so if out05 and out07 are needed at some point, 'make' would call 'filter in05 in07 --some --flags'. Is that right? Because if this was somehow possible, I would not need that process at all. I do not see the relation to the rest of this thread yet, but what you describe sounds like classic pattern rules: The only relation is that the fact that I wouldn't need a server process neither cache if 'n in' - 'n out' filter programs were supported like I explain in the comment below. out%: in% cp $ $@ Here, cp would be the filter. Are you referring to something else? The problem is that if I have like 50 'in' files and I want 50 'out' files, the 'cp' program would be launched 50 times, whereas the filter I have in mind is able to consume like 50 files and output 50 files as a result at once. I would say that this could be formulated as rule that has a dynamic number of targets depending on the number of dependencies (inputs). Best, Sebastian Best wishes, Matej ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: feature request - order only deps
On Ne, 2013-02-03 at 00:45 +0100, Sebastian Pipping wrote:
On 02.02.2013 18:38, Matěj Týč wrote:
How about something like this?
bar_deps = foo1 foo2
bar: $(bar_deps)
$(bar_deps):
$(MAKE) cache-foo
touch $@
%:
touch $@
I have also thought of that, but this can work well reliably only in the
case if there is only one make job. If I specify -j4, it may happen that
all jobs will attempt to load the same cache simultaneously, which I
have to avoid :-(
If that happens how about replacing
$(MAKE) cache-foo
by something like
mkdir .lock 2/dev/null || exit 0 ; \
$(MAKE) cache-foo ; \
ret=$$?; \
rmdir .lock exit $${ret}
The idea is:
- mkdir can only succeed once
- if $(MAKE) cache-foo fails
1. the whole should return non-zero
2. .lock is not left laying around
Thank you,
but just by looking at it, I think that if .lock exists, 0 is returned
and while one make job is busy loading the cache, the others act like
the cache was loaded.
Writing the makefile and trying it confirms this assumption, so I guess
that it has to behave like that.
I am beginning to think that the most elegant (from the user's point of
view) would be (probably optional) ignore of order-only dependencies if
the target exists and is more up-to-date than its ordinary dependencies.
How difficult could be writing a patch that would enable this?
Matej
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: feature request - order only deps
On 03.02.2013 23:20, Matěj Týč wrote:
If that happens how about replacing
$(MAKE) cache-foo
by something like
mkdir .lock 2/dev/null || exit 0 ; \
$(MAKE) cache-foo ; \
ret=$$?; \
rmdir .lock exit $${ret}
The idea is:
- mkdir can only succeed once
- if $(MAKE) cache-foo fails
1. the whole should return non-zero
2. .lock is not left laying around
Thank you,
but just by looking at it, I think that if .lock exists, 0 is returned
and while one make job is busy loading the cache, the others act like
the cache was loaded.
Writing the makefile and trying it confirms this assumption, so I guess
that it has to behave like that.
I see, so actually waiting is needed.
Have you tried modifying the call to do that? I'd bet it's possible :-)
I am beginning to think that the most elegant (from the user's point of
view) would be (probably optional) ignore of order-only dependencies if
the target exists and is more up-to-date than its ordinary dependencies.
To my understanding, it would have to be optional and off by default to
not break other cases that are currently supported. Think of something like
tmp:
mkdir tmp
tmp/foo.pdf: foo.tex | tmp
pdflatex -output-directory tmp $
In this scenario, the tmp directory needs to be created if missing.
How difficult could be writing a patch that would enable this?
No idea without having a closer look. To be honest, I am not yet
convinced that this feature would be of real use to other users or that
a new a global switch would be the best way to address this issue, if
really wanted.
Best,
Sebastian
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: feature request - order only deps
On Ne, 2013-02-03 at 23:40 +0100, Sebastian Pipping wrote:
On 03.02.2013 23:20, Matěj Týč wrote:
If that happens how about replacing
$(MAKE) cache-foo
by something like
mkdir .lock 2/dev/null || exit 0 ; \
$(MAKE) cache-foo ; \
ret=$$?; \
rmdir .lock exit $${ret}
The idea is:
- mkdir can only succeed once
- if $(MAKE) cache-foo fails
1. the whole should return non-zero
2. .lock is not left laying around
Thank you,
but just by looking at it, I think that if .lock exists, 0 is returned
and while one make job is busy loading the cache, the others act like
the cache was loaded.
Writing the makefile and trying it confirms this assumption, so I guess
that it has to behave like that.
I see, so actually waiting is needed.
Have you tried modifying the call to do that? I'd bet it's possible :-)
Well, an infinite loop checking every sec. whether the .lock directory
still exists might do the trick. However, additional check whether the
cache load was succesful would be needed and actually the directories
would have to have different names for diferent cache that could get
loaded.
I am beginning to think that the most elegant (from the user's point of
view) would be (probably optional) ignore of order-only dependencies if
the target exists and is more up-to-date than its ordinary dependencies.
To my understanding, it would have to be optional and off by default to
not break other cases that are currently supported. Think of something like
tmp:
mkdir tmp
tmp/foo.pdf: foo.tex | tmp
pdflatex -output-directory tmp $
In this scenario, the tmp directory needs to be created if missing.
Actually this would not be a problem, I would need to ignore the 'tmp'
target iff 'tmp/foo.pdf' existed and was more up-to-date than 'foo.tex'.
So in cases like that the behavior even wouldn't change at all.
I think that the problem could arise in cases like this:
Dirs: dir1 dir2
dir%:
mkdir $@
dir2/foo: dir1/bar
touch $@
dir1/bar: | Dirs
touch $@
.PHONY: Dirs
If you make 'dir1/bar', then remove 'dir2' and decide to make
'dir2/foo', you appreciate that 'dir2' is remade because 'dirs' is an
order-only dep of 'bar' and it is remade as soon as make realizes that
'dirs' is not complete because 'dir2' is missing.
So although this can be considered as the Makefile bug, cases like this
can exist, so the proposed change would break this behavior (I propose
that the 'Dirs' order-only dep is ignored because dir1/bar is all right
and dir2/foo says that it depends only on dir1/bar)
How difficult could be writing a patch that would enable this?
No idea without having a closer look. To be honest, I am not yet
convinced that this feature would be of real use to other users or that
a new a global switch would be the best way to address this issue, if
really wanted.
I think that this is a nice idea, since 'make' doesn't play well with
filters that can process batches of files at once, the server process
is a good way to reduce overhead of processing those files one by one
and you may end up needing a cache quite soon :-)
Actually I might be wrong, but I think that if I have a program 'filter'
capable of processing in01 in02 in03 ... files to out01 out02 out03 ...
etc., it is not possible to tell this to 'make', so if out05 and out07
are needed at some point, 'make' would call 'filter in05 in07 --some
--flags'. Is that right? Because if this was somehow possible, I would
not need that process at all.
Matej
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: feature request - order only deps
On 04.02.2013 00:13, Matěj Týč wrote: On Ne, 2013-02-03 at 23:40 +0100, Sebastian Pipping wrote: To my understanding, it would have to be optional and off by default to not break other cases that are currently supported. Think of something like tmp: mkdir tmp tmp/foo.pdf: foo.tex | tmp pdflatex -output-directory tmp $ In this scenario, the tmp directory needs to be created if missing. Actually this would not be a problem, I would need to ignore the 'tmp' target iff 'tmp/foo.pdf' existed and was more up-to-date than 'foo.tex'. So in cases like that the behavior even wouldn't change at all. I think that the problem could arise in cases like this: Dirs: dir1 dir2 dir%: mkdir $@ dir2/foo: dir1/bar touch $@ dir1/bar: | Dirs touch $@ .PHONY: Dirs If you make 'dir1/bar', then remove 'dir2' and decide to make 'dir2/foo', you appreciate that 'dir2' is remade because 'dirs' is an order-only dep of 'bar' and it is remade as soon as make realizes that 'dirs' is not complete because 'dir2' is missing. So although this can be considered as the Makefile bug, cases like this can exist, so the proposed change would break this behavior (I propose that the 'Dirs' order-only dep is ignored because dir1/bar is all right and dir2/foo says that it depends only on dir1/bar) If I am not mistaken, you are saying that: If 1. an order-only dependency is missing and 2. the target is not re-built (i.e. neither missing or older than normal prerequisites the order-only dependency should not be built. I'm starting to see why you want support for that. I think that this is a nice idea, since 'make' doesn't play well with filters that can process batches of files at once, the server process is a good way to reduce overhead of processing those files one by one and you may end up needing a cache quite soon :-) Actually I might be wrong, but I think that if I have a program 'filter' capable of processing in01 in02 in03 ... files to out01 out02 out03 ... etc., it is not possible to tell this to 'make', so if out05 and out07 are needed at some point, 'make' would call 'filter in05 in07 --some --flags'. Is that right? Because if this was somehow possible, I would not need that process at all. I do not see the relation to the rest of this thread yet, but what you describe sounds like classic pattern rules: out%: in% cp $ $@ Here, cp would be the filter. Are you referring to something else? Best, Sebastian ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: feature request - order only deps
On So, 2013-02-02 at 00:22 +0100, Sebastian Pipping wrote: On 01.02.2013 16:18, Matěj Týč wrote: ... Consider a server process that can execute commands and that can load (huge) data into cache to spped the execution up. Loading the data is a make task and a target file cache-foo is created to expose that the data has been succesfully loaded into the server cache. Then cache-foo is an order-only dependency of foo1, foo2 and foo3 targets that take advantage of it (rules to make them call the server process that uses the foo cache). Therefore those targets are not made before setting the cache up and also the cache is loaded once, not concurently even if like -j4 is used as a make argument. As you might guess, if I have a target 'bar' that depends on 'foo1' and 'foo2', then even if 'foo1' and 'foo2' exist, if 'cache-foo' is missing, it is remade and data are loaded into the cache without being of any use, which is quite annoying. In other words, the whole server might have to start, load the cache and then do nothing. This is what I understand to be our current Makefile: bar_deps = foo1 foo2 bar: $(bar_deps) $(bar_deps): | cache-foo %: touch $@ Yes, this is basically correct, great! Now you want that cache-foo is not built if all of $(bar_deps) exist. We can achieve that by only adding cache-foo as a dependency, if former is not the case. So let's replace the line $(bar_deps): | cache-foo by $(bar_deps): | $(if $(call any_file_missing,$(bar_deps)),cache-foo,) The first function here would be enough for me, BUT... (see below) BTW. it is a pity one can't write $(call file_missing,$@) using two custom functions: # $(call file_missing,file_1) # returns: # true (actually $(file_1)) if the file is missing # false (empty string) if the file exists define file_missing $(if $(wildcard $1),,$1) endef ... I hereby put that into the public domain. This session confirms it working: $ touch foo1 foo2 bar $ make make: `bar' is up to date.# cache-foo not built! $ rm foo1 $ make touch cache-foo # cache-foo built, since foo1 is missing touch foo1 touch bar I'm curious, if that helps. Thank you for your quick help, your example indeed works, but it has one weakness. Consider a case when eg. 'foo1' depends on 'baz' and suddenly 'baz' is updated, so 'foo1' should be updated, too. However, the old 'foo1' is still there, so the 'file_missing' function assumes that nothing has to be done = cache-foo is not needed, which is not true. Since I generate the makefile using some M4sugar macros, I don't mind having to write some extra stuff; however now the workaround path seems to complicate quite a lot... Thank you, Matej Best, Sebastian ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: feature request - order only deps
On 02.02.2013 16:19, Matěj Týč wrote: This is what I understand to be our current Makefile: bar_deps = foo1 foo2 bar: $(bar_deps) $(bar_deps): | cache-foo %: touch $@ [..] Thank you for your quick help, your example indeed works, but it has one weakness. Consider a case when eg. 'foo1' depends on 'baz' and suddenly 'baz' is updated, so 'foo1' should be updated, too. However, the old 'foo1' is still there, so the 'file_missing' function assumes that nothing has to be done = cache-foo is not needed, which is not true. Since I generate the makefile using some M4sugar macros, I don't mind having to write some extra stuff; however now the workaround path seems to complicate quite a lot... How about something like this? bar_deps = foo1 foo2 bar: $(bar_deps) $(bar_deps): $(MAKE) cache-foo touch $@ %: touch $@ Now cache-foo is remade iff at least one of $(bar_deps) needs a rebuild. Best, Sebastian ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: feature request - order only deps
On 02.02.2013 18:38, Matěj Týč wrote:
How about something like this?
bar_deps = foo1 foo2
bar: $(bar_deps)
$(bar_deps):
$(MAKE) cache-foo
touch $@
%:
touch $@
I have also thought of that, but this can work well reliably only in the
case if there is only one make job. If I specify -j4, it may happen that
all jobs will attempt to load the same cache simultaneously, which I
have to avoid :-(
If that happens how about replacing
$(MAKE) cache-foo
by something like
mkdir .lock 2/dev/null || exit 0 ; \
$(MAKE) cache-foo ; \
ret=$$?; \
rmdir .lock exit $${ret}
The idea is:
- mkdir can only succeed once
- if $(MAKE) cache-foo fails
1. the whole should return non-zero
2. .lock is not left laying around
If that works for you conceptually, you could abstract a little more and
turn it into something re-usable like
$(call synced_make,cache-foo)
Best,
Sebastian
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
feature request - order only deps
Hi, I have noticed that if we have a target that has prerequisities, if those prerequisities are missing and I want to make the target, then even if the target file exists, prerequisities are remade if possible and then, consequently, the target has to be remade, too, since its prerequisity is now more up-to-date. Which, of course, makes perfect sense. However, if a prerequisity is order-only, it is also going to be remade despite the target file existis already, but unlike the case above, the target won't be remade because it is allowed to be older than its order-only prerequisity. This is normally not an issue, since order-only prereqs are probably often cheap commands lke mkdir etc., but my case is different: Consider a server process that can execute commands and that can load (huge) data into cache to spped the execution up. Loading the data is a make task and a target file cache-foo is created to expose that the data has been succesfully loaded into the server cache. Then cache-foo is an order-only dependency of foo1, foo2 and foo3 targets that take advantage of it (rules to make them call the server process that uses the foo cache). Therefore those targets are not made before setting the cache up and also the cache is loaded once, not concurently even if like -j4 is used as a make argument. As you might guess, if I have a target 'bar' that depends on 'foo1' and 'foo2', then even if 'foo1' and 'foo2' exist, if 'cache-foo' is missing, it is remade and data are loaded into the cache without being of any use, which is quite annoying. In other words, the whole server might have to start, load the cache and then do nothing. Could you help me with this? There are two possible solutions I see here: - Change the make behavior or add an option to not to remake order-only deps if they are not relevant - Suggest a workaround for situations like this, so the cache is loaded only if it is needed and parallel make execution won't attempt to load the cache for each job separately. Best wishes, Matej ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: feature request - order only deps
On 01.02.2013 16:18, Matěj Týč wrote: Hi, I have noticed that if we have a target that has prerequisities, if those prerequisities are missing and I want to make the target, then even if the target file exists, prerequisities are remade if possible and then, consequently, the target has to be remade, too, since its prerequisity is now more up-to-date. Which, of course, makes perfect sense. However, if a prerequisity is order-only, it is also going to be remade despite the target file existis already, but unlike the case above, the target won't be remade because it is allowed to be older than its order-only prerequisity. This is normally not an issue, since order-only prereqs are probably often cheap commands lke mkdir etc., but my case is different: Consider a server process that can execute commands and that can load (huge) data into cache to spped the execution up. Loading the data is a make task and a target file cache-foo is created to expose that the data has been succesfully loaded into the server cache. Then cache-foo is an order-only dependency of foo1, foo2 and foo3 targets that take advantage of it (rules to make them call the server process that uses the foo cache). Therefore those targets are not made before setting the cache up and also the cache is loaded once, not concurently even if like -j4 is used as a make argument. As you might guess, if I have a target 'bar' that depends on 'foo1' and 'foo2', then even if 'foo1' and 'foo2' exist, if 'cache-foo' is missing, it is remade and data are loaded into the cache without being of any use, which is quite annoying. In other words, the whole server might have to start, load the cache and then do nothing. This is what I understand to be our current Makefile: bar_deps = foo1 foo2 bar: $(bar_deps) $(bar_deps): | cache-foo %: touch $@ Now you want that cache-foo is not built if all of $(bar_deps) exist. We can achieve that by only adding cache-foo as a dependency, if former is not the case. So let's replace the line $(bar_deps): | cache-foo by $(bar_deps): | $(if $(call any_file_missing,$(bar_deps)),cache-foo,) using two custom functions: # $(call file_missing,file_1) # returns: # true (actually $(file_1)) if the file is missing # false (empty string) if the file exists define file_missing $(if $(wildcard $1),,$1) endef # $(call any_file_missing,file_1 file_2 ..) # returns: # true (a non-empty string) if any file is missing # false (empty string) if all file exist define any_file_missing $(strip $(foreach filename,$(1),$(call file_missing,$(filename endef I hereby put that into the public domain. This session confirms it working: $ touch foo1 foo2 bar $ make make: `bar' is up to date.# cache-foo not built! $ rm foo1 $ make touch cache-foo # cache-foo built, since foo1 is missing touch foo1 touch bar I'm curious, if that helps. Best, Sebastian ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
