[Cocci] [PATCH v3] bundles/pycaml/: use .NOTPARALLEL on bundles
We need to disable parallel builds on the preparation of bundles. This fixes parallel compilation when using 32 threads on a 32-core system and on an 8 core system: The error: make[6]: Entering directory '/home/mcgrof/coccinelle/bundles/pycaml' /usr/bin/ocamlopt.opt -I chemoelectric-pycaml-8614105 -cclib -lpycaml_stubs -ccopt -I/usr/include/python2.7 -ccopt -I/usr/include/x86_64-linux-gnu/python2.7 -cclib -lpython2.7 -c chemoelectric-pycaml-8614105/pycaml.ml -o chemoelectric-pycaml-8614105/pycaml.cmx make[6]: *** No rule to make target 'chemoelectric-pycaml-8614105/pycaml.a', needed by 'all-opt-build-targets'. Stop. make[6]: *** Waiting for unfinished jobs make[6]: Leaving directory '/home/mcgrof/coccinelle/bundles/pycaml' ../Makefile.bundles:33: recipe for target 'all.opt' failed make[5]: *** [all.opt] Error 2 make[5]: Leaving directory '/home/mcgrof/coccinelle/bundles/pycaml' Makefile:186: recipe for target '/home/mcgrof/coccinelle/bundles/pycaml/.opt' failed make[4]: *** [/home/mcgrof/coccinelle/bundles/pycaml/.opt] Error 2 make[4]: Leaving directory '/home/mcgrof/coccinelle' Makefile:180: recipe for target 'subdirs.opt' failed make[3]: *** [subdirs.opt] Error 1 make[3]: Leaving directory '/home/mcgrof/coccinelle' Makefile:163: recipe for target 'opt-compil' failed make[2]: *** [opt-compil] Error 2 make[2]: Leaving directory '/home/mcgrof/coccinelle' Makefile:143: recipe for target 'all-release' failed make[1]: *** [all-release] Error 2 make[1]: Leaving directory '/home/mcgrof/coccinelle' Makefile:121: recipe for target 'all' failed make: *** [all] Error 2 Signed-off-by: Luis R. Rodriguez --- bundles/menhirLib/Makefile | 1 + bundles/parmap/Makefile| 1 + bundles/pcre/Makefile | 1 + bundles/pycaml/Makefile| 1 + 4 files changed, 4 insertions(+) diff --git a/bundles/menhirLib/Makefile b/bundles/menhirLib/Makefile index 23ac0d6..e1bc4d2 100644 --- a/bundles/menhirLib/Makefile +++ b/bundles/menhirLib/Makefile @@ -33,6 +33,7 @@ include ../Makefile.bundles depend: .depend @$(MAKE) all-build +.NOTPARALLEL: .prepare .prepare: $(MARKER) echo "let libdir = \"$(shell pwd\)\"" > $(SRC_DIR)/installation.ml echo "let ocamlfind = false" >> $(SRC_DIR)/installation.ml diff --git a/bundles/parmap/Makefile b/bundles/parmap/Makefile index 67c65ec..675a71d 100644 --- a/bundles/parmap/Makefile +++ b/bundles/parmap/Makefile @@ -17,6 +17,7 @@ OCAMLCCFLAGS+=-ccopt -D_GNU_SOURCE include ../Makefile.bundles +.NOTPARALLEL: .prepare .prepare: $(MARKER) cd $(SRC_DIR) && ./configure touch $@ diff --git a/bundles/pcre/Makefile b/bundles/pcre/Makefile index 97aa12e..4a07d2f 100644 --- a/bundles/pcre/Makefile +++ b/bundles/pcre/Makefile @@ -24,6 +24,7 @@ EXTRALNKFLAGS=$(PCRE_LIBS:%=-ldopt %) include ../Makefile.bundles +.NOTPARALLEL: .prepare .prepare: $(MARKER) if [ "$$OCAMLVERSION" "<" 4.02.0 ]; then \ cp $(SRC_DIR)/pcre_compat312.ml $(SRC_DIR)/pcre_compat.ml; \ diff --git a/bundles/pycaml/Makefile b/bundles/pycaml/Makefile index 5045c27..0ee7757 100644 --- a/bundles/pycaml/Makefile +++ b/bundles/pycaml/Makefile @@ -17,6 +17,7 @@ export PYMAJOR=$(shell echo ${PYTHON_VERSION} | sed -e 's/\..*//') include ../Makefile.bundles +.NOTPARALLEL: .prepare .prepare: $(MARKER) rm -f "$(SRC_DIR)/pycaml.mli" $(CC) -E -w -D PYMAJOR$(PYMAJOR) -xc pycaml.pp.ml >$(SRC_DIR)/pycaml.ml -- 2.7.0 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH v2 1/3] docs/demos: add a few ++ documentation and demos
This perhaps is not the best demo for use of ++ but it should suffice. This adds some basic documentation for it and a demo. Signed-off-by: Luis R. Rodriguez --- demos/plusplus1.c| 4 ++ demos/plusplus1.cocci| 17 demos/plusplus1.res | 5 +++ demos/plusplus2.c| 7 demos/plusplus2.cocci| 26 demos/plusplus2.res | 10 + docs/manual/cocci_syntax.tex | 95 7 files changed, 164 insertions(+) create mode 100644 demos/plusplus1.c create mode 100644 demos/plusplus1.cocci create mode 100644 demos/plusplus1.res create mode 100644 demos/plusplus2.c create mode 100644 demos/plusplus2.cocci create mode 100644 demos/plusplus2.res diff --git a/demos/plusplus1.c b/demos/plusplus1.c new file mode 100644 index ..2a28c8037cac --- /dev/null +++ b/demos/plusplus1.c @@ -0,0 +1,4 @@ +struct x { + int z; + char b; +}; diff --git a/demos/plusplus1.cocci b/demos/plusplus1.cocci new file mode 100644 index ..86e5950b2a93 --- /dev/null +++ b/demos/plusplus1.cocci @@ -0,0 +1,17 @@ +// This adds a float for one int variable found. +// Doing this however is limited to one int in the data structure, if +// more one int variable in the structure we would not know where to +// place the new one safely order-wise. See plusplus2.cocci for an +// example of dealing with this issue. + +@simpleplus@ +identifier x,v; +fresh identifier xx = v ## "_float"; +@@ + +struct x { ++ float xx; + ... + int v; + ... +} diff --git a/demos/plusplus1.res b/demos/plusplus1.res new file mode 100644 index ..ba5c5341639e --- /dev/null +++ b/demos/plusplus1.res @@ -0,0 +1,5 @@ +struct x { + float z_float; + int z; + char b; +}; diff --git a/demos/plusplus2.c b/demos/plusplus2.c new file mode 100644 index ..a03d26c6da14 --- /dev/null +++ b/demos/plusplus2.c @@ -0,0 +1,7 @@ +struct x { + int z; + int a; + char b; + int c; + int *d; +}; diff --git a/demos/plusplus2.cocci b/demos/plusplus2.cocci new file mode 100644 index ..bd61ac8f8771 --- /dev/null +++ b/demos/plusplus2.cocci @@ -0,0 +1,26 @@ +// This can support having a structure with more than two int variables, +// and adding a respective float for each. It can do this as order does +// not matter. +// +// This uses ++ to support the fact that the rule may be working +// with multiple variables that we need to modify and that order +// does not matter. +// +// If you don't use "++" you'll get "already tagged token" error since +// Coccinelle is concerned that the user has no way of specifying the order +// in which they should appear. By using "++" you are telling Coccinelle +// +// "I know that a lot of things can collect here, and I'm OK +//with that. I'm also OK with things getting added out of order. + +@plusplus@ +identifier x,v; +fresh identifier xx = v ## "_float"; +@@ + +struct x { +++ float xx; + ... + int v; + ... +} diff --git a/demos/plusplus2.res b/demos/plusplus2.res new file mode 100644 index ..4920fb7d7df9 --- /dev/null +++ b/demos/plusplus2.res @@ -0,0 +1,10 @@ +struct x { + float a_float; + float c_float; + float z_float; + int z; + int a; + char b; + int c; + int *d; +}; diff --git a/docs/manual/cocci_syntax.tex b/docs/manual/cocci_syntax.tex index e8d74a7d015f..49602ef1d732 100644 --- a/docs/manual/cocci_syntax.tex +++ b/docs/manual/cocci_syntax.tex @@ -851,6 +851,11 @@ rule should apply if rule XXX was never matched at all. \section{Transformation} +Coccinelle allows for transformations to enable modifying C code using +very precise grammar. + +\subsection{Basic transformations} + The transformation specification essentially has the form of C code, except that lines to remove are annotated with \verb+-+ in the first column, and lines to add are annotated with \verb-+-. A transformation specification @@ -1100,6 +1105,96 @@ write Some kinds of terms can only appear in + code. These include comments, ifdefs, and attributes (\texttt{\_\_attribute\_\_((...))}). +\subsection{Advanced transformations} + +You may run into the situation where grammar you specificy for +transformations might associate itself with a few consecutive tokens +in code, in such cases Coccinelle cannot gaurantee order when making +additions. If you are sure that order does not matter you can use the +optional double addition token \texttt{++} to annotate that Coccinelle +may add things out of order. This may be for intance rather safe in +certain situations when extending a data structure with more members, +based on existing members of the data structure. The following rule helps +to extend a data structure with a respective float for one present int. +This works well with the simple \texttt{+}. + +\begin{lstlisting}[language=Cocci] +@simp
[Cocci] [PATCH v2 2/3] docs/manual/cocci_syntax.tex: extend with python iteration
Signed-off-by: Luis R. Rodriguez --- docs/manual/cocci_syntax.tex | 51 ++-- 1 file changed, 40 insertions(+), 11 deletions(-) diff --git a/docs/manual/cocci_syntax.tex b/docs/manual/cocci_syntax.tex index 49602ef1d732..5adfc95b0ce1 100644 --- a/docs/manual/cocci_syntax.tex +++ b/docs/manual/cocci_syntax.tex @@ -1698,9 +1698,12 @@ of every token. \section{Iteration} It is possible to iterate Coccinelle, giving the subsequent iterations a -different set of virtual rules or virtual identifier bindings. And example -is found in {\tt demos/iteration.cocci}. The example shown there is as -follows: +different set of virtual rules or virtual identifier bindings. Coccinelle +currently supports iteration with both ocaml and python scripting. An +example with ocaml is fond in {\tt demos/iteration.cocci}, a python +exmaple is found in {\tt demos/python\_iteration.cocci}. + +The ocaml scripting iteration example starts as follows. \begin{quote} \begin{verbatim} @@ -1724,6 +1727,29 @@ with Not_found -> \end{verbatim} \end{quote} +The respective python scripting iteration example starts as follows: + +\begin{quote} +\begin{verbatim} +virtual after_start + +@initialize:python@ +@@ + +seen = set() + +def add_if_not_present (source, f, file): +if (f, file) not in seen: +seen.add((f, file)) +it = Iteration() +if file != None: +it.set_files([file]) +it.add_virtual_rule(after_start) +it.add_virtual_identifier(err_ptr_function, f) +it.register() +\end{verbatim} +\end{quote} + The virtual rule {\tt after\_start} is used to distinguish between the first iteration (in which it is not considered to have matched) and all others. This is done by not mentioning {\tt after\_start} in the command @@ -1734,11 +1760,12 @@ The main code for performing the iteration is found in the function {\tt {\tt register}. {\tt New iteration} creates a structure representing the new iteration. {\tt set\_files} sets the list of files to be considered on the new iteration. If this function is not called, the new iteration -treats the same files as the current iteration. {\tt add\_virtual\_rule A} -has the same effect as putting {\tt -D a} on the command line. Note that -the first letter of the rule name is capitalized, although this is not done -elsewhere. {\tt add\_virtual\_identifier X v} has the same effect as -putting {\tt -D x=v} on the command line. Note again the case change. +treats the same files as the current iteration. {\tt add\_virtual\_rule a} +has the same effect as putting {\tt -D a} on the command line. If +using ocaml scripting instead of python scripting the first letter of the rule +name is capitalized, although this is not done elsewhere. +{\tt add\_virtual\_identifier x v} has the same effect as putting {\tt -D x=v} +on the command line. Again, when using ocaml scripting there is a case change. {\tt extend\_virtual\_identifiers()} (not shown) preserves all virtual identifiers of the current iteration that are not overridden by calls to {\tt add\_virtual\_identifier}. Finally, the call to {\tt register} queues @@ -1749,9 +1776,11 @@ Modification is not allowed when using iteration. Thus, it is required to use the {\tt --no-show-diff}, unless the semantic patch contains {\tt *}s (a semantic match rather than a semantic patch). -The remainder of the code above uses a hash table to ensure that the same -information is not enqueued more than once. Coccinelle itself provides no -support for this. +When using python scripting a tuple is used throughout the rest of the +code to ensure that the same information is not enqueued more than once. +When using ocaml scripting a hash table is used for the same purposes. +Coccinelle itself provides no support for this and as such addressing +this with scripting is necessary. %%% Local Variables: %%% mode: LaTeX -- 2.8.2 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH v2 0/3] coccinelle: expand docs
This v2 modifies the ++ demo and documentation to something a bit more safe for its use, and while at it decided to extend the iteration documentation and document support for .cocciconfig. Luis R. Rodriguez (3): docs/demos: add a few ++ documentation and demos docs/manual/cocci_syntax.tex: extend with python iteration docs: document .cocciconfig demos/plusplus1.c| 4 + demos/plusplus1.cocci| 17 + demos/plusplus1.res | 5 ++ demos/plusplus2.c| 7 ++ demos/plusplus2.cocci| 26 +++ demos/plusplus2.res | 10 +++ docs/manual/cocci_syntax.tex | 169 --- docs/spatch.1.in | 32 +++- 8 files changed, 258 insertions(+), 12 deletions(-) create mode 100644 demos/plusplus1.c create mode 100644 demos/plusplus1.cocci create mode 100644 demos/plusplus1.res create mode 100644 demos/plusplus2.c create mode 100644 demos/plusplus2.cocci create mode 100644 demos/plusplus2.res -- 2.8.2 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH v2 3/3] docs: document .cocciconfig
Add documentation and example to manual and man page. Signed-off-by: Luis R. Rodriguez --- docs/manual/cocci_syntax.tex | 23 +++ docs/spatch.1.in | 32 +++- 2 files changed, 54 insertions(+), 1 deletion(-) diff --git a/docs/manual/cocci_syntax.tex b/docs/manual/cocci_syntax.tex index 5adfc95b0ce1..dc7aa17712af 100644 --- a/docs/manual/cocci_syntax.tex +++ b/docs/manual/cocci_syntax.tex @@ -1782,6 +1782,29 @@ When using ocaml scripting a hash table is used for the same purposes. Coccinelle itself provides no support for this and as such addressing this with scripting is necessary. +\section{.cocciconfig support} + +Coccinelle supports enabling custom options to be preferred when running +spatch. This is supported through the search of .cocciconfig files in each of +the following directories, later lines extend and may override earlier ones: + +\begin{itemize} + \item Your current user's home directory is processed first + \item Your directory from which spatch is called is processed next + \item The directory provided with the --dir option is processed last, if used +\end{itemize} + +Newlines, even with \, are not tolerated in attribute values. An example +follows: + +\begin{quote} +\begin{verbatim} +[spatch] + options = --jobs 4 + options = --show-trying +\end{verbatim} +\end{quote} + %%% Local Variables: %%% mode: LaTeX %%% TeX-master: "main_grammar" diff --git a/docs/spatch.1.in b/docs/spatch.1.in index dae38952e88c..121048b1ea0b 100644 --- a/docs/spatch.1.in +++ b/docs/spatch.1.in @@ -55,7 +55,8 @@ and at the \fBscripts/coccinelle\fP directory of the Linux Kernel source code. .SH OPTIONS -Here is a summary of the most commonly used options: +Here is a summary of the most commonly used options (also +see the "Configuration Mechanism" section below): .TP .B \-\-sp\-file \fI\fP @@ -566,6 +567,31 @@ It's for the other (internal) uses of the spatch program. \fB\-\-compare\-c\fR +.SH CONFIGURATION MECHANISM + +Coccinelle uses a simple text format to store customizations into +\.cocciconfig files, that can be per project, user, or target. The order +in which the configuration file is processed is as follows, the +later lines always extend and may override earlier ones: + + o Your current user's home directory is processed first + o Your directory from which spatch is called is processed next + o The directory provided with the --dir option is processed last, if used + +Such a configuration file may look like this: + +.if n \{\ +.RS 4 +.\} +.nf +[spatch] + options = --jobs 4 + options = --show-trying +.fi +.if n \{\ +.RE +.\} + .SH FILES .I @SHAREDIR@/standard.iso .RS @@ -574,6 +600,10 @@ This file contains the default set of isomorphisms. .I @SHAREDIR@/standard.h .RS This file contains the default set of macro hints. +.RE +.I .cocciconfig +.RS +This file contains the custom set of spatch options. .SH ENVIRONMENT .IP COCCINELLE_HOME -- 2.8.2 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] [PATCH v2 1/3] coccinelle: also catch kzfree() issues
On Mon, 20 Jun 2016, Michal Marek wrote: > On 2016-05-23 17:18, Julia Lawall wrote: > > > > > > On Mon, 23 May 2016, Yann Droneaud wrote: > > > >> Since commit 3ef0e5ba4673 ('slab: introduce kzfree()'), > >> kfree() is no more the only function to be considered: > >> kzfree() should be recognized too. > >> > >> In particular, kzfree() must not be called on memory > >> allocated through devm_*() functions. > >> > >> Cc: Johannes Weiner > >> Signed-off-by: Yann Droneaud > > > > Acked-by: Julia Lawall > > Hi Julia, > > does your ACK apply to the other two patches as well? Sorry, I seem to have missed the other two. I have reviewed them now, and the ack applies to all three. Thanks for checking on it. julia ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel
On Sat, Jun 18, 2016 at 07:51:55AM +0200, Julia Lawall wrote: > > > On Sat, 18 Jun 2016, Luis R. Rodriguez wrote: > > > On Fri, Jun 17, 2016 at 05:35:26PM +0200, Julia Lawall wrote: > > > On Fri, 17 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > On Fri, Jun 17, 2016 at 11:44:26AM +0200, Julia Lawall wrote: > > > > > I'm not sure that this is worth it. It adds a dependency on a tool > > > > > that > > > > > seems not to be well maintained. In terms of Coccinelle, I'm not sure > > > > > that it gives a big benefit. > > > > > > > > > > Attached is a graph showing the file selection time for Coccinelle > > > > > for a > > > > > selection of fairly complex semantic patches. Coccigrep is just a > > > > > line-by-line regexp search implemented in ocaml, gitgrep uses git > > > > > grep. > > > > > In most cases, glimpse is clearly faster. > > > > > > > > > > On the other hand, it seems that glimpse often selects more files. > > > > > Sometimes a few more, eg 16 vs 14, and sometimes quite a lot more, eg > > > > > 538 > > > > > vs 236. I suspect that this is because glimpse considers _ to be a > > > > > space, > > > > > and thus it can have many false positives. There are, however, a few > > > > > cases where glimpse also selects fewer files. > > > > > > > > > > The file processing time (ie parsing the file, searching for, matches > > > > > of > > > > > the semantic patch in the file, and performing the transformation) is > > > > > normally much higher than the file selection time. > > > > > > > > > > So it seems that git grep is currently a better option for the kernel. > > > > > > > > Great, thanks, consider this patch dropped, do we want the heuristics > > > > for the cache index in place though or should I drop that as well ? > > > > > > I assume you mean this patch: > > > > > > [PATCH v2 3/8] coccicheck: add indexing enhancement options > > > > > > I think it should be dropped. It adds complexity and git grep works > > > pretty well. > > > > Hmm but coccicheck does not make use of --git-grep even. > > > > > If people want to use something else, they can use SPARGS, > > > or a .cocciconfig file, eg: > > > > > > [spatch] > > > options = --use-glimpse > > > > Neat will these be used last and thus override anything? > > Good point. If it is in the home directory, it is overrided by > everything. So make coccicheck shouldn't have an option related to this > issue. Great. > > If so, what > > about just adding an upstream .cocciconfig with --use-gitgrep -- only > > issue then is what if a user wants to use idutils ? How do we let them > > override? > > If we have an upstream .cocciconfig with --use-gitgrep, then the user can > specify an SPARGS with --use-idutils and override. I take it you meant SPFLAGS. I just read the order rules, I'll past them for completeness: -- from coccinelle/read_options.ml: .cocciconfig files can be placed in the user's home directory, the directory from which spatch is called, and the directory provided with the --dir option. The .cocciconfig file in the user's home directory is processed first, the .cocciconfig file in the directory from which spatch is called is processed next, and the .cocciconfig file in the directory provided with the --dir option is processed last. In each case, the read options extend/override the previously read ones. In all cases, the user can extend/override the options found in the .cocciconfig files on the command line. --- So order is: 0. $HOME/.cocciconfig 1. $PWD/.cocciconfig 2. --dir .cocciconfig So indeed an upstream .cocciconfig would seem to work well. Drivers can also have their own .cocciconfig if they would need it, but I cannot see this being needed at this time though, but good to know and keep in mind. In the future this fact might be a bit more useful if we added support for instance of a rule namespace, then for instance if we know a tweak is only needed for one driver we might for instance have something like: [spatch rule=scripts/coccinelle/api/d_find_alias.cocci] options = --opt1 --opt2 ... But for now I think more than good with an upstream linux/.cocciconfig then and SPFLAGs. I will have to do just one small adjustment to SPFLAGS on coccicheck to ensure it does go at the end. I'll address that in the re-spin of this series. > If we are making an upstream .cocciconfig, I would put a timeout in it > too. In my experience, 120 (seconds) is fine. Maybe 200 to give a little > more margin. Again, this can be overridden on the command line. OK will use 200. Luis ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] [PATCH v3] Coccinelle: noderef: Add new rules and correct the old rule
On 2016-05-24 10:39, Julia Lawall wrote: > Acked-by: Julia Lawall > > On Tue, 24 May 2016, Vaishali Thakkar wrote: > >> Add new rules to detect the cases where sizeof is used in >> function calls as a argument. >> >> Also, for the patch mode third rule should behave same as >> second rule with arguments reversed. So, change that as well. >> >> Signed-off-by: Vaishali Thakkar Applied to kbuild.git#misc. Michal ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] [PATCH v2 1/3] coccinelle: also catch kzfree() issues
On 2016-05-23 17:18, Julia Lawall wrote: > > > On Mon, 23 May 2016, Yann Droneaud wrote: > >> Since commit 3ef0e5ba4673 ('slab: introduce kzfree()'), >> kfree() is no more the only function to be considered: >> kzfree() should be recognized too. >> >> In particular, kzfree() must not be called on memory >> allocated through devm_*() functions. >> >> Cc: Johannes Weiner >> Signed-off-by: Yann Droneaud > > Acked-by: Julia Lawall Hi Julia, does your ACK apply to the other two patches as well? Thanks, Michal ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci