Re: [Cocci] Data exchange over network interfaces by SmPL scripts
> Unfortunately, I observed during a few runs on my test system > that the displayed record sets can vary. Thus I guess that this approach > (which works together with Python multi-threading functionality) will need > further software adjustments. I stumbled on general software development challenges from inter-process communication over TCP connections. This programming interface supports reliable data transmissions. But the POSIX API does not directly support so far to determine how many of the sent data are still on the way for delivery to the receiving process. * Operating systems can provide additional functions for this purpose. I find then that Linux APIs could be improved for more efficient analysis of network connections. * Network protocols influence also corresponding data processing approaches. + Customised network communication is not needed if you can depend on system functionality by databases. + If you would occasionally like to experiment with related services, the application of the technology “Common Object Request Broker Architecture” can be another interesting design option. Example: http://omniorb.sourceforge.net/ Regards, Markus ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] Data exchange over network interfaces by SmPL scripts
> Unfortunately, I observed during a few runs on my test system > that the displayed record sets can vary. Thus I guess that this approach > (which works together with Python multi-threading functionality) will need > further software adjustments. I am curious how the clarification of such software behaviour will evolve further also with the help from additional information around the topic “Checking network input processing by Python for a multi-threaded server”. https://mail.python.org/pipermail/python-list/2019-April/740645.html Regards, Markus ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] Data exchange over network interfaces by SmPL scripts
> connecting … > File "", line 35, in store_statements > socket.gaierror: [Errno -2] Name or service not known > Error in Python script, line 56, file … It seems that the attached adjusted data processing approach can produce an usable analysis result. elfring@Sonne:~/Projekte/Coccinelle/janitor> time /usr/bin/python3 list_duplicate_statement_pairs_from_if_branches-server4.py statement1|statement2|"function name"|"source file"|incidence dprintk ( "%s: readreg error (reg == 0x%02x, ret == %i)\n" , __func__ , reg , ret ) ;|return - 1 ;|stv0297_readreg|/home/elfring/Projekte/Linux/next-patched/drivers/media/dvb-frontends/stv0297.c|3 dprintk ( "%s: readreg error (reg == 0x%02x, ret == %i)\n" , __func__ , reg1 , ret ) ;|return - 1 ;|stv0297_readregs|/home/elfring/Projekte/Linux/next-patched/drivers/media/dvb-frontends/stv0297.c|3 real0m1,044s user0m0,389s sys 0m0,055s Unfortunately, I observed during a few runs on my test system that the displayed record sets can vary. Thus I guess that this approach (which works together with Python multi-threading functionality) will need further software adjustments. Would you like to add any advices here? Regards, Markus import threading, socket, socketserver, struct, subprocess inputs = [] def receive_data(s, n): d = b'' while len(d) < n: p = s.recv(n - len(d)) if not p: return None d += p return d def receive_message(s): expect = receive_data(s, 4) if not expect: return None return receive_data(s, struct.unpack(">I", expect)[0]) class threaded_TCP_request_handler(socketserver.BaseRequestHandler): def handle(self): data = receive_message(self.request) if data: inputs.append(data.decode()) class threaded_TCP_server(socketserver.ThreadingMixIn, socketserver.TCPServer): pass if __name__ == "__main__": server = threaded_TCP_server(("localhost", 1234), threaded_TCP_request_handler) with server: ip, port = server.server_address server_thread = threading.Thread(target = server.serve_forever) server_thread.daemon = True server_thread.start() cp = subprocess.run(["/usr/local/bin/spatch", "--timeout", "9", "--python", "/usr/bin/python3", "-D", "server_id=" + str(ip), "-D", "server_port=" + str(port), "/home/elfring/Projekte/Coccinelle/janitor/list_duplicate_statement_pairs_from_if_branches-client3.cocci", "/home/elfring/Projekte/Linux/next-patched/drivers/media/dvb-frontends/stv0297.c"], capture_output = True, text = True) server.shutdown() import sys if cp.returncode: sys.stderr.write("%s\n===\nexit code: %d" % (cp.stderr, cp.returncode)) else: if len(inputs) > 0: def report(): mapping = {} def insert(x): """Add data to an internal table.""" key = x["name"], x["file"], x["line"], x["column"] if key in mapping: sys.stderr.write("""A duplicate key was passed. function: %s file: %s line: %s column: %d """ % key) raise RuntimeError else: mapping[key] = x["s1"], x["s2"] def data_import(): import json for k in inputs: for v in json.loads(k): insert(v) data_import() from collections import Counter counts = Counter() for k, v in mapping.items(): counts[(v[0], v[1], k[0], k[1])] += 1 delimiter = "|" duplicates = {} for k, v in counts.items(): if v > 1: duplicates[k] = v if len(duplicates.keys()) > 0: sys.stdout.write(delimiter.join(["statement1", "statement2", '"function name"', '"source file"', "incidence"]) + "\r\n") for k, v in duplicates.items(): sys.stdout.write(delimiter.join([k[0], k[1], k[2], k[3], str(v)]) + "\r\n") else: sys.stderr.write("Duplicate statements were not determined" " from the following records.\n"
Re: [Cocci] Data exchange over network interfaces by SmPL scripts
> File "", line 35, in store_statements > socket.gaierror: [Errno -2] Name or service not known > Error in Python script, line 56, file … It seems that the attached adjusted data processing approach can produce an usable analysis result. elfring@Sonne:~/Projekte/Coccinelle/janitor> time /usr/bin/python3 list_duplicate_statement_pairs_from_if_branches-server4.py statement1|statement2|"function name"|"source file"|incidence dprintk ( "%s: readreg error (reg == 0x%02x, ret == %i)\n" , __func__ , reg , ret ) ;|return - 1 ;|stv0297_readreg|/home/elfring/Projekte/Linux/next-patched/drivers/media/dvb-frontends/stv0297.c|3 dprintk ( "%s: readreg error (reg == 0x%02x, ret == %i)\n" , __func__ , reg1 , ret ) ;|return - 1 ;|stv0297_readregs|/home/elfring/Projekte/Linux/next-patched/drivers/media/dvb-frontends/stv0297.c|3 real0m1,044s user0m0,389s sys 0m0,055s Unfortunately, I observed during a few runs on my test system that the displayed record sets can vary. Thus I guess that this approach (which works together with Python multi-threading functionality) will need further software adjustments. Would you like to add any advices here? Regards, Markus ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] Data exchange over network interfaces by SmPL scripts
>> File "", line 34, in store_statements >> AttributeError: __exit__ >> Error in Python script, line 55, file … > > I have no idea. It looks like a python problem. Partly, yes (of course). > If you want help, you will have to construct a script > that exhibits the error with print statements only. This suggestion will probably not work (because additional test output will not trigger the shown error response alone.) I see solution challenges like the following after the addition of another data type conversion for the network service port variable. elfring@Sonne:~/Projekte/Coccinelle/janitor> /usr/local/bin/spatch --python /usr/bin/python3 -D server_id=127.0.0.1 -D server_port=1234 list_duplicate_statement_pairs_from_if_branches-client2.cocci ~/Projekte/Linux/next-patched/drivers/media/dvb-frontends/stv0297.c … Using Python version: 3.7.2 (default, Dec 30 2018, 16:18:15) [GCC] … connecting Traceback (most recent call last): File "", line 4, in File "", line 35, in store_statements socket.gaierror: [Errno -2] Name or service not known Error in Python script, line 56, file … How should the system configuration details be improved for a simple test connection? Regards, Markus ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] Data exchange over network interfaces by SmPL scripts
> Is there still a need to perform parallelisation for the mentioned software > components by other approaches? The multi-processing support by the Coccinelle software triggers some development challenges. If data should be shared between started (background) processes, an external system need to be selected for the desired storage service. Thus I would like to send these data over network interfaces by the attached script for the evolving semantic patch language. I stumble on the following error message. elfring@Sonne:~/Projekte/Coccinelle/janitor> /usr/local/bin/spatch -D server_id=localhost -D server_port=1234 list_duplicate_statement_pairs_from_if_branches-client2.cocci ~/Projekte/Linux/next-patched/drivers/media/dvb-frontends/stv0297.c … Using Python version: 2.7.15 (default, May 21 2018, 17:53:03) [GCC] … Traceback (most recent call last): File "", line 4, in File "", line 34, in store_statements AttributeError: __exit__ Error in Python script, line 55, file … I would appreciate if the shown data processing approach can work with version ranges of involved recent software components. So I am looking for additional solution ideas. Regards, Markus @initialize:python@ server_id << virtual.server_id; server_port << virtual.server_port; @@ import json, socket, struct, sys sys.stderr.write("Using Python version:\n%s\n" % (sys.version)) if server_id == False: server_id = "localhost" if server_port == False: server_port = 1234 def store_statements(fun, source, s1, s2): """Send data for the service.""" records = [] for place in source: records.append('{"name":%s,"file":%s,"line":%s,"column":%s,"s1":%s,"s2":%s}' % (json.dumps(fun), json.dumps(place.file), json.dumps(place.line), json.dumps(int(place.column) + 1), json.dumps(s1), json.dumps(s2))) with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as so: so.connect((server_id, server_port)) result = "[\n" result += ",\n".join(records) result += "]" b = bytes(result) p = struct.pack(">I", len(b)) p += b so.sendall(p) @searching exists@ identifier work; statement s1, s2; position pos; type T; @@ T work(...) { ... when any if (...) { ... when any s1@pos s2 } ... when any } @script:python collection@ fun << searching.work; s1 << searching.s1; s2 << searching.s2; place << searching.pos; @@ store_statements(fun, place, s1, s2) ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci