Re: [Twisted-Python] Changing procmon.ProcessMonitor.processes
OK I opened a ticket with a plan, after discussing with Mark Williams. 1. Make a custom class that implements the Sequence ABC and pretends to be sized the old size. 2. Internally all access will be moved to attributes 3. All sequence methods will be marked as Deprecated. We'll kill them in a year. 4. processes will become _processes, with a Deprecated .processes accessor, again for backwards compatibility. I am technically changing the class of a thing from a tuple to a different class, but I don't think that this can be in general relied on. The API docs do not mention the attribute, much less promise the concrete class of its contents. Objections? Ticket: https://twistedmatrix.com/trac/ticket/9287 On Wed, Sep 20, 2017 at 1:34 AM ex vito wrote: > On 2017-09-19, at 15:49, Moshe Zadka wrote: > > On Tue, Sep 19, 2017 at 3:51 AM ex vito wrote: > > > >> Other than that, again, per that section's rules, not being a commiter >> myself, I'm in no position to approve such a change. I wonder, however, how >> "urgent" such a final change is to you and why a deprecation cycle does not >> fit your purpose (even though, admittedly, it may represent more effort). >> > > I think that between the low likelihood that someone went crawling over > the attributes manually, the RoI of having a deprecation cycle with some > intermediate solution that later needs to be cleaned up, and the fact that > this would be a clean break (i.e., "AttributeError") rather than some > obscure error, I am at least interested in opinions about going the > exception route. > > ProcMon is non-trivial to product ionize, and I'm not aware of anyone even > using it in production, other than me, let alone crawling over its internal > state. > > > Agreed. For completeness, the code I work with does not make use of > ProcMon and I don't recall having ever used it. > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] verilog simulation calling out to twisted
Hi, I've been writing Verilog simulations for years, and tinkering with Python for years. In the last couple years I have been calling out to Python from Verilog simulations, which has been awesome. Just this week I tried twisted out for the first time to write a simple custom TCP server. It was a great experience and even though I still know next to nothing about twisted, I think I'm in love :-) Now I have a crazy idea. I'd like that Python code that my Verilog calls out to to use twisted. OK, actually I've already done that, my Verilog calls out to my that TCP server that I wrote. I had to spawn a thread and run the server in that thread so that the Verilog could continue to do stuff in parallel with the server. It's working great. The real crazy thing I want is for the Verilog to call out to a twisted TCP client as well. Again, I need the client and server to not block the Verilog. My first attempt was to run client and server in two separate threads. I start the server thread, then I start the client thread and only in the client thread do I call reactor.run(). It seems to not be working. Before I try to figure out why, I thought I might ask here if it even should work. Here's a different way to explain it, in case that helps: main thread is verilog which spawns server_thread def server_thread(): reactor.listenTCP(ServerFactory()) main thread spawns client_thread def client_thread() reactor.connectTCP(ClientFactory()) reactor.run() I read the page on threads in twisted and I'm guessing Would it be better to have the main thread just spawn one thread that does this? def client_server_thread(): reactor.listenTCP(...) reactor.connectTCP(...) reactor.run() Hmm, as I finish typing this all out, I'm realizing that surely someone has written at TCP proxy (essentially what this verilog simulation is doing) in twisted before. A quick internet search tells me yes...and of course they are not using threads. The way the whole verilog mess is written right now it would be much easier to use separate threads for client and server. Is that possible? If not I'll have to find a way to justify re-architecting the verilog mess. Thanks, Bryan ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Truly distributed tests with Twisted
Hi All, I have been using DistTrialRunner with great success, but now I would like to run tests both in parallel and distributed across a set of testing nodes. Can anyone recommend existing solutions for this? I tried extending DistTrialRunner and using a generator to feed it some tests, but it has some limitations that made this non-trivial. Thanks! -Todd ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Need help reviewing improvement to inlineCallbacks
Pierre Tardy, the main developer of Buildbot, submitted this patch to add full stack traces to inlineCallbacks: https://github.com/twisted/twisted/pull/814 I think the patch looks good, but can someone with more knowledge of inlineCallbacks review it? Right now, if you have a stack of inlineCallbacks, andan exception is thrown, some of the stacktrace is lost. By adding full stack traces to inlineCallbacks, this greatly improves the debuggability of software like Buildbot, which makes heavy use of inlineCallbacks. -- Craig ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] How to correctly run Sqlite with Twisted?
Le 2017-09-20 10:24, ex vito a écrit : On 2017-09-19, at 21:59, Goffi wrote: [SNIP] I would subscribe to Moshe's and Jean-Paul's suggestions of using higher-level tools to handle database interactions. [SNIP] A final thought, from the realm of intuition: if I were targeting heavily concurrent access to a database (subjective, yes), I'd consider replacing SQLite with some other engine. SQLite is a great tool for many purposes, but concurrent access is not its strong point (see https://www.sqlite.org/lockingv3.html). Thanks for all your answers! I'll check every options before taking a decision, but I'm quite seduced by Alchimia, the SqlAlchemy wrapper. I'm a bit worrying because it has not seen any commit since 11 months (I have had bad experiences with unmaintained libraries). This said, the wrapper is thin, I can probably maintain it myself if needed. As a bonus, it would allow me to propose other storage backends than Sqlite. For the use case, it's used mainly for cache and various data for an XMPP client which can handle several users at the same time, and only one backend is accessing the database. There is already an automatic update mechanism that I would have to adapt, but it may simplify everything at the end. Also I need to check if Full Text Search and Json data type are easily usable, but it seems than plain SQL request are still possible at worst. Thanks again for your answers, it's really helpful! Goffi ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Changing procmon.ProcessMonitor.processes
On 2017-09-19, at 15:49, Moshe Zadka wrote: > On Tue, Sep 19, 2017 at 3:51 AM ex vito wrote: > > Other than that, again, per that section's rules, not being a commiter > myself, I'm in no position to approve such a change. I wonder, however, how > "urgent" such a final change is to you and why a deprecation cycle does not > fit your purpose (even though, admittedly, it may represent more effort). > > I think that between the low likelihood that someone went crawling over the > attributes manually, the RoI of having a deprecation cycle with some > intermediate solution that later needs to be cleaned up, and the fact that > this would be a clean break (i.e., "AttributeError") rather than some obscure > error, I am at least interested in opinions about going the exception route. > > ProcMon is non-trivial to product ionize, and I'm not aware of anyone even > using it in production, other than me, let alone crawling over its internal > state. Agreed. For completeness, the code I work with does not make use of ProcMon and I don't recall having ever used it. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] How to correctly run Sqlite with Twisted?
On 2017-09-19, at 21:59, Goffi wrote:
> I'm using Sqlite3 module through Twisted's enterpirse.adbapi, I create the
> ConnectionPool instance like this:
>
> self.dbpool = ConnectionPool("sqlite3", db_filename,
> check_same_thread=False)
>
> You can see the code at
> https://repos.goffi.org/sat/file/tip/src/memory/sqlite.py
>
> Sometime, the writing is failing with following exception:
>
> Failure instance: Traceback: : database is
> locked
>
> So I wonder if the database is correctly used, did anybody experienced
> something similar with Twisted and Sqlite ?
>
> Should I just augment timeout as advised at https://stackoverflow.com/a/
> 8618328? Looks like more a workaround than a clean solution.
>
> Python 2 documentation doesn't talk about check_same_thread argument, but
> Python 3 at https://docs.python.org/3.5/library/sqlite3.html#sqlite3.connect
> says that writing operation should be serialized by the user (I thought it
> was
> the default as said in https://sqlite.org/threadsafe.html), how should I
> achieve that?
>
> Also PRAGMA are not working (specially "PRAGMA foreign_keys = ON"), I guess
> because of multi-threading, what is the good way to activate foreign_keys for
> all following request?
I would subscribe to Moshe's and Jean-Paul's suggestions of using higher-level
tools to handle database interactions. If, however, you'd rather continue with
the adbapi approach, here are a few tips that may help your diagnostic, given
that behavior you are observing seems to result from
multi-threaded/multi-process interactions:
- Confirm you have no other SQLite client at play, locking your data somehow.
- Correct serialization of independent SQL queries/transactions may be a
complex topic.
- But you can try forcing the ConnectionPool thread count to 1 via the
cp_min/cp_max init args.
- For your PRAGMA statements you should probably use the cp_openfun
ConnectionPool init arg.
- Using the undocumented "check_same_thread" on Python 2 seems pointless.
(not sure it is, though; I did not look at the underlying Python 2 stdlib
code)
Lastly:
I have experienced issues with SQLite3 + Python on one occasion in the past:
depending on the platform, SQLite itself is/is-not distributed with Python,
IIRC; for diagnostics, confirm the exact version of the sqlite .SO / .DLL in
use and, ideally, its compile time options. I recall my troubles were between
CentOS 5 vs. 6, due to the specific SQLite versions included in each
distribution.
A final thought, from the realm of intuition: if I were targeting heavily
concurrent access to a database (subjective, yes), I'd consider replacing
SQLite with some other engine. SQLite is a great tool for many purposes, but
concurrent access is not its strong point (see
https://www.sqlite.org/lockingv3.html).
Cheers,
--
exvito
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
