Re: [Cocci] More support for SmPL data processing with databases?
> I think that you just want to disable the isomorphism. Put disable neg_if > in the initial @@ of your rule. >> >> Will a need evolve to switch isomorphisms completely off for safer >> (and efficient) analysis of original source code? Will the corresponding system configuration get any further development considerations? Regards, Markus ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] More support for SmPL data processing with databases?
On Mon, 13 Apr 2020, Markus Elfring wrote: > >> I would find it nice to explain the different software behaviour for the > >> mentioned > >> SmPL code variants > > > > Run spatch -parse-cocci and you will easily see. The if stays in the same > > place. When the isomorphism exchanges the branches, the condition gets > > negated, so in one case you get the position of the full condition and > > in the other case you get the position of the part of the condition under > > the negation. > > Do any of these transformations avoid the reporting of duplicate source > code positions? Scripts are run for each set of metavariable bindings. But there is no special treatment of positions. julia > > > >>> I think that you just want to disable the isomorphism. Put disable neg_if > >>> in the initial @@ of your rule. > > Will a need evolve to switch isomorphisms completely off for safer > (and efficient) analysis of original source code? > > Regards, > Markus > ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] More support for SmPL data processing with databases?
>> I would find it nice to explain the different software behaviour for the >> mentioned >> SmPL code variants > > Run spatch -parse-cocci and you will easily see. The if stays in the same > place. When the isomorphism exchanges the branches, the condition gets > negated, so in one case you get the position of the full condition and > in the other case you get the position of the part of the condition under > the negation. Do any of these transformations avoid the reporting of duplicate source code positions? >>> I think that you just want to disable the isomorphism. Put disable neg_if >>> in the initial @@ of your rule. Will a need evolve to switch isomorphisms completely off for safer (and efficient) analysis of original source code? Regards, Markus ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] More support for SmPL data processing with databases?
On Mon, 13 Apr 2020, Markus Elfring wrote:
> >>> Maybe you consider the following results to be a problem:
> >>>
> >>> kzalloc mcde [("mcde_probe", 2, 0, 198, 1, "m.c", 16, 1, 16, 3)]
> >>> kzalloc ! mcde [("mcde_probe", 2, 0, 198, 1, "m.c", 16, 1, 16, 3)]
> >>> platform_get_irq irq [("mcde_probe", 2, 0, 198, 1, "m.c", 97, 1, 97, 3)]
> >>> platform_get_irq ! irq [("mcde_probe", 2, 0, 198, 1, "m.c", 97, 1, 97, 3)]
> >>>
> >>> The positions are the same. The result is because of an isomorphism that
> >>> exchanges the branches of a conditional.
>
> I would find it nice to explain the different software behaviour for the
> mentioned
> SmPL code variants
Run spatch -parse-cocci and you will easily see. The if stays in the same
place. When the isomorphism exchanges the branches, the condition gets
negated, so in one case you get the position of the full condition and in
the other case you get the position of the part of the condition under the
negation.
julia
>
> A:
> if@p (
> ( <+... result ...+>
> & check
> ) )
>
>
> B:
> if (
> ( <+... result ...+>
> & check@p
> ) )
>
>
>
> >> Can the data which are provided because of isomorphism transformations
> >> be distinguished by an additional attribute?
> >
> > I think that you just want to disable the isomorphism. Put disable neg_if
> > in the initial @@ of your rule.
>
> Thanks for this suggestion.
>
> Source code analysis can be continued also with this configuration option.
>
>
> Now I become also more interested in improvements around parallel data
> processing.
> Which data structures would support aggregate computations for a growing code
> base?
>
> Regards,
> Markus
>
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] More support for SmPL data processing with databases?
>>> Maybe you consider the following results to be a problem:
>>>
>>> kzalloc mcde [("mcde_probe", 2, 0, 198, 1, "m.c", 16, 1, 16, 3)]
>>> kzalloc ! mcde [("mcde_probe", 2, 0, 198, 1, "m.c", 16, 1, 16, 3)]
>>> platform_get_irq irq [("mcde_probe", 2, 0, 198, 1, "m.c", 97, 1, 97, 3)]
>>> platform_get_irq ! irq [("mcde_probe", 2, 0, 198, 1, "m.c", 97, 1, 97, 3)]
>>>
>>> The positions are the same. The result is because of an isomorphism that
>>> exchanges the branches of a conditional.
I would find it nice to explain the different software behaviour for the
mentioned
SmPL code variants
A:
if@p (
( <+... result ...+>
& check
) )
B:
if (
( <+... result ...+>
& check@p
) )
>> Can the data which are provided because of isomorphism transformations
>> be distinguished by an additional attribute?
>
> I think that you just want to disable the isomorphism. Put disable neg_if
> in the initial @@ of your rule.
Thanks for this suggestion.
Source code analysis can be continued also with this configuration option.
Now I become also more interested in improvements around parallel data
processing.
Which data structures would support aggregate computations for a growing code
base?
Regards,
Markus
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] More support for SmPL data processing with databases?
On Mon, 13 Apr 2020, Markus Elfring wrote:
> > Maybe you consider the following results to be a problem:
> >
> > kzalloc mcde [("mcde_probe", 2, 0, 198, 1, "m.c", 16, 1, 16, 3)]
> > kzalloc ! mcde [("mcde_probe", 2, 0, 198, 1, "m.c", 16, 1, 16, 3)]
> > platform_get_irq irq [("mcde_probe", 2, 0, 198, 1, "m.c", 97, 1, 97, 3)]
> > platform_get_irq ! irq [("mcde_probe", 2, 0, 198, 1, "m.c", 97, 1, 97, 3)]
> >
> > The positions are the same. The result is because of an isomorphism that
> > exchanges the branches of a conditional.
>
> Can the data which are provided because of isomorphism transformations
> be distinguished by an additional attribute?
I think that you just want to disable the isomorphism. Put disable neg_if
in the initial @@ of your rule.
julia
>
>
> > I think that you could have figured this out by yourself. If you just
> > printed out the data instead of wrapping it in lots of database noise,
> > then you could have seen what was going on.
>
> I got the impression that there are further software development
> challenges to consider (also for safer counting of record sets).
>
> Regards,
> Markus
>
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] More support for SmPL data processing with databases?
> Maybe you consider the following results to be a problem:
>
> kzalloc mcde [("mcde_probe", 2, 0, 198, 1, "m.c", 16, 1, 16, 3)]
> kzalloc ! mcde [("mcde_probe", 2, 0, 198, 1, "m.c", 16, 1, 16, 3)]
> platform_get_irq irq [("mcde_probe", 2, 0, 198, 1, "m.c", 97, 1, 97, 3)]
> platform_get_irq ! irq [("mcde_probe", 2, 0, 198, 1, "m.c", 97, 1, 97, 3)]
>
> The positions are the same. The result is because of an isomorphism that
> exchanges the branches of a conditional.
Can the data which are provided because of isomorphism transformations
be distinguished by an additional attribute?
> I think that you could have figured this out by yourself. If you just
> printed out the data instead of wrapping it in lots of database noise,
> then you could have seen what was going on.
I got the impression that there are further software development
challenges to consider (also for safer counting of record sets).
Regards,
Markus
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] More support for SmPL data processing with databases?
> @script:ocaml@ > action << find.action; > check << find.check; > p << find.p; > @@ > Printf.printf "%s %s %s\n" action check (Dumper.dump p) > > On the code you provided whichis unfortunately much too long to include in > a message I found my code examples short enough. > and that you could have surely shortened to illustrate whatever > is the problem. Does such a feedback indicate that you are still struggling with the understanding for the presented software situation? > I get the same output from both semantic patches, except that the line > numbers are different for the position variable p, Does this information contain a contradiction? Do you observe any repeated data sets? > because you have put it in a different place. Yes. - Thus I would expect that different column numbers will be displayed while the line numbers should be (mostly?) identical. > I don't see any reason why making the script code call a database would > have any impact on this issue. The database can provide ACID properties for parallel data processing, can't it? I would like to perform aggregate computations finally. Would you become interested in counting record sets anyhow? > If you do get different results for the above semantic patches, > please make a small .c file that illustrates the problem. I presented questionable differences once more a moment ago. https://lore.kernel.org/cocci/7228efff-76bb-44c9-e2cd-ea19bbd49...@web.de/ https://systeme.lip6.fr/pipermail/cocci/2020-April/007142.html Regards, Markus ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] More support for SmPL data processing with databases?
I tried the following semantic patches: @find@ expression action, check, result; position p; statement is, es; @@ result = action(...); if ( ( <+... result ...+> & check@p ) ) is else es @script:ocaml@ action << find.action; check << find.check; p << find.p; @@ Printf.printf "%s %s %s\n" action check (Dumper.dump p) @find@ expression action, check, result; position p; statement is, es; @@ result = action(...); if@p ( ( <+... result ...+> & check ) ) is else es @script:ocaml@ action << find.action; check << find.check; p << find.p; @@ Printf.printf "%s %s %s\n" action check (Dumper.dump p) On the code you provided whichis unfortunately much too long to include in a message and that you could have surely shortened to illustrate whatever is the problem. I get the same output from both semantic patches, except that the line numbers are different for the position variable p, because you have put it in a different place. I don't see any reason why making the script code call a database would have any impact on this issue. If you do get different results for the above semantic patches, please make a small .c file that illustrates the problem. The .c file should be such that it illustrates the problem and no smaller file also illustrates the problem. julia ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] More support for SmPL data processing with databases?
>> How will the application of SmPL conjunctions evolve further?
>
> I really have no idea what you are talking about.
I suggest to take another look at affected implementation details.
> Make one single mail that contains all of the semantic patch variants
> that you want to have considered and explain what the problem is.
Would you find the following data processing approach easier to clarify?
I extracted a bit of source code from a known file for your convenience
as an attachment.
https://elixir.bootlin.com/linux/v5.6.3/source/drivers/gpu/drm/mcde/mcde_drv.c#L307
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/gpu/drm/mcde/mcde_drv.c?id=8f3d9f354286745c751374f5f1fcafee6b3f3136#n308
> These semantic patches should involve no use of databases.
A Python dictionary can also be used as a data storage technique
for the discussed system test.
@initialize:python@
@@
import sys
records = {}
class integrity_error:
pass
def store_positions(places, check_in, action_in):
"""Add source code positions to an internal table."""
for place in places:
key = (place.file, place.line, int(place.column) + 1)
if key in records:
sys.stderr.write("\n".join(["-> duplicate data",
"file:", key[0],
"function:", place.current_element,
"line:", str(place.line)]))
sys.stderr.write("\n")
raise integrity_error
else:
records[key] = (action_in, check_in, place.current_element)
@find@
expression action, check, result;
position p;
statement is, es;
@@
result = action(...);
if@p (
( <+... result ...+>
& check
) )
is
else
es
@script:python collection@
c << find.check;
action << find.action;
places << find.p;
@@
store_positions(places, c, action)
@finalize:python@
@@
if len(records) > 0:
delimiter = "|"
sys.stdout.write(delimiter.join(['action',
'check',
'"source file"',
'line',
'column'
]))
sys.stdout.write("\r\n")
for key, value in records.items():
sys.stdout.write(delimiter.join([value[0],
value[1],
key[0],
key[1],
str(key[2])
]))
sys.stdout.write("\r\n")
else:
sys.stderr.write("No result for this analysis!\n")
elfring@Sonne:~/Projekte/Coccinelle/janitor> spatch
list_condition_checks_after_function_calls7.cocci ../Probe/mcde_drv-excerpt1.c
…
-> duplicate data
file:
../Probe/mcde_drv-excerpt1.c
function:
mcde_probe
line:
16
Traceback …:
…, line 30, in store_positions
__main__.integrity_error: <__main__.integrity_error instance at 0x7fd3bff4d730>
…
Would you like compare the software behaviour any further with
the following rule variant for the semantic patch language?
@find@
expression action, check, result;
position p;
statement is, es;
@@
result = action(...);
if (
( <+... result ...+>
& check@p
) )
is
else
es
Regards,
Markus
// SPDX-License-Identifier: GPL-2.0
static int mcde_probe(struct platform_device *pdev)
{
struct device *dev = >dev;
struct drm_device *drm;
struct mcde *mcde;
struct component_match *match = NULL;
struct resource *res;
u32 pid;
u32 val;
int irq;
int ret;
int i;
mcde = kzalloc(sizeof(*mcde), GFP_KERNEL);
if (!mcde)
return -ENOMEM;
mcde->dev = dev;
ret = drm_dev_init(>drm, _drm_driver, dev);
if (ret) {
kfree(mcde);
return ret;
}
drm = >drm;
drm->dev_private = mcde;
platform_set_drvdata(pdev, drm);
/* Enable continuous updates: this is what Linux' framebuffer expects */
mcde->oneshot_mode = false;
drm->dev_private = mcde;
/* First obtain and turn on the main power */
mcde->epod = devm_regulator_get(dev, "epod");
if (IS_ERR(mcde->epod)) {
ret = PTR_ERR(mcde->epod);
dev_err(dev, "can't get EPOD regulator\n");
goto dev_unref;
}
ret = regulator_enable(mcde->epod);
if (ret) {
dev_err(dev, "can't enable EPOD regulator\n");
goto dev_unref;
}
mcde->vana = devm_regulator_get(dev, "vana");
if (IS_ERR(mcde->vana)) {
ret = PTR_ERR(mcde->vana);
dev_err(dev, "can't get VANA regulator\n");
goto regulator_epod_off;
}
ret = regulator_enable(mcde->vana);
if (ret) {
dev_err(dev, "can't enable VANA regulator\n");
goto regulator_epod_off;
}
/*
* The vendor code uses ESRAM (onchip RAM) and need to activate
* the v-esram34 regulator, but we don't use that yet
*/
/* Clock the silicon so we can access the registers */
mcde->mcde_clk = devm_clk_get(dev, "mcde");
if (IS_ERR(mcde->mcde_clk)) {
dev_err(dev, "unable to get MCDE main clock\n");
ret = PTR_ERR(mcde->mcde_clk);
goto regulator_off;
Re: [Cocci] More support for SmPL data processing with databases?
>> Do I present another advanced use case for the application of known software >> tools? > > No. Such a view can be also interesting. > The faults in linux paper (work done 10 years ago) uses a database as well. Thanks for this background information. How will the software evolve further in affected areas? Regards, Markus ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] More support for SmPL data processing with databases?
> Now you have the two scripts, which is good, but you haven't included the > test data that causes the problem. Please make one mail that includes > everything. I find that I provided relevant data already. https://systeme.lip6.fr/pipermail/cocci/2020-April/007135.html https://lore.kernel.org/cocci/ab6d402e-3c12-25ab-162a-f200c382d...@web.de/ Regards, Markus ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] More support for SmPL data processing with databases?
>>> The database itself doesn't bring any value. >> >> Another unique constraint violation was noticed by using such technology. > > It's an accident. This might be. - I got into the development mood to publish patches for a few source files of Linux modules according to another search approach. I picked one of these files up for the discussed test. > You could have noticed the same thing by just printing the results > in the python script. This can be also possible in principle. I find the programming convenience different. >> I am curious when you are going to take related functionality better into >> account. > > Never. Interesting … > It has nothing to do with Coccinelle. Do I present another advanced use case for the application of known software tools? > When you make a bug report, you should show the minimal code that shows the > problem. I hope that variations can occasionally help also to achieve a better understanding. Regards, Markus ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] More support for SmPL data processing with databases?
>> I observe that the following SmPL code variant can result also in >> a significant difference. >> >> @find@ >> expression action, check, result; >> position p; >> statement is, es; >> @@ >> result = action(...); >> if ( >> ( <+... result ...+> >> & check@p >> ) ) >> is >> else >> es >> >> >> How will the application of SmPL conjunctions evolve further? > > I really have no idea what you are talking about. I suggest to take another look for the software behaviour. > that contains all of the semantic patch variants that you want to have > These semantic patches should involve no use of databases. Please compare details with the following SmPL code example. @find@ expression action, check, result; position p; statement is, es; @@ result = action(...); if@p ( ( <+... result ...+> & check ) ) is else es Under which circumstances will the Coccinelle software provide duplicate data for their processing by (Python) script rules? > The database itself doesn't bring any value. Another unique constraint violation was noticed by using such technology. I am curious when you are going to take related functionality better into account. Regards, Markus ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] More support for SmPL data processing with databases?
> I hope that it can become easier to clarify where unexpected duplicate keys > would occur as in my test approach. There is the possibility to apply information according to a clarification request on a topic like “Checking the handling of unique keys/indexes” from 2014-12-20. https://groups.google.com/d/msg/sqlalchemy/klmUwiirIQw/g6NLiVAGGAIJ elfring@Sonne:~/Projekte/Linux/next-patched> spatch ~/Projekte/Coccinelle/janitor/list_condition_checks_after_function_calls2c.cocci drivers/gpu/drm/mcde/mcde_drv.c … Using SQLAlchemy version: 1.3.15 … %% function: mcde_probe line: 322 %% function: mcde_probe line: 403 action|check|"source file"|line|column … How should the error reporting be clarified around such source code places? https://elixir.bootlin.com/linux/v5.6.3/source/drivers/gpu/drm/mcde/mcde_drv.c#L307 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/gpu/drm/mcde/mcde_drv.c?id=8f3d9f354286745c751374f5f1fcafee6b3f3136#n308 I observe that the following SmPL code variant can result also in a significant difference. @find@ expression action, check, result; position p; statement is, es; @@ result = action(...); if ( ( <+... result ...+> & check@p ) ) is else es How will the application of SmPL conjunctions evolve further? Regards, Markus ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] More support for SmPL data processing with databases?
>> I hope that it can become easier to clarify where unexpected duplicate keys >> would occur as in my test approach. > > Use --debug and just print out the infomation rather than putting is in > your database. We have got different views around this logging approach. > With the database you add complexity for nothing. I disagree to this view. Database software usually provides some helpful functionality, doesn't it? The mentioned programming interface quickly pointed an unique constraint violation out. Now I am looking again for possible solutions. Regards, Markus ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] More support for SmPL data processing with databases?
> I think that your issue about something matching or not has nothing to do > with the database code, and you could easily remove it for the purposes of > reporting a concern with Coccinelle. Software evolution can be continued also together with your constructive feedback. I adapted another SmPL script according to a recent information. https://lore.kernel.org/cocci/17a4592b-92bd-e4c9-8481-7d46616cb...@web.de/T/#m4f59721d2011ac4ded602f2e63e0c63f22a15fa9 https://systeme.lip6.fr/pipermail/cocci/2020-April/007133.html It can become more interesting to increase also the application of SmPL conjunctions like the following. @find@ expression action, check, result; position p; statement is, es; @@ result = action(...); if@p ( ( <+... result ...+> & check ) ) is else es Now I stumble on corresponding software development challenges. elfring@Sonne:~/Projekte/Linux/next-patched> spatch ~/Projekte/Coccinelle/janitor/list_condition_checks_after_function_calls2.cocci drivers/gpu/drm/mcde/mcde_drv.c … Using SQLAlchemy version: 1.3.15 … sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: pairs.function, pairs.source_file, pairs.line, pairs.column [SQL: INSERT INTO pairs (function, source_file, line, "column", action, "check") VALUES (?, ?, ?, ?, ?, ?)] … I hope that it can become easier to clarify where unexpected duplicate keys would occur as in my test approach. Regards, Markus ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] More support for SmPL data processing with databases?
> I think that your issue about something matching or not has nothing to do > with the database code, Such a view can be partly appropriate. > and you could easily remove it I hope that the understanding of the presented SmPL code example could also be sufficient in the way that the data processing for the SmPL rules “find1” and “check1” will need another bit of clarification. > for the purposes of reporting a concern with Coccinelle. Another code variant can eventually be clarified a bit later. Regards, Markus ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] More support for SmPL data processing with databases?
On Sun, 12 Apr 2020, Markus Elfring wrote: > >> I hope that another clarification can be achieved also for the software > >> behaviour of the following source code analysis approach. > > > > I don't run code that involves databases. > > Can the situation evolve in a way so that this programming interface > will become better supported together with programming languages > and development tools you are more familiar with (so far)? I think that your issue about something matching or not has nothing to do with the database code, and you could easily remove it for the purposes of reporting a concern with Coccinelle. julia > > Regards, > Markus > ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] More support for SmPL data processing with databases?
>> I hope that another clarification can be achieved also for the software >> behaviour of the following source code analysis approach. > > I don't run code that involves databases. Can the situation evolve in a way so that this programming interface will become better supported together with programming languages and development tools you are more familiar with (so far)? Regards, Markus ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
