Re: [Twisted-Python] ANN: Eliot 0.9, the logging system with causality - now with journald support

[Glyph Lefkowitz]

> On Oct 8, 2015, at 5:47 PM, exar...@twistedmatrix.com wrote:
> 
> On 8 Oct, 11:38 pm, gl...@twistedmatrix.com wrote:
>>> On Oct 8, 2015, at 6:04 AM, Itamar Turner-Trauring  
>>> wrote:
>>> 
>>> Eliot 0.9 is out, with a new utility for pretty-printing log messages and 
>>> native journald support 
>>> . You can now route 
>>> Eliot logs to journald and when an error occurs easily find all logged 
>>> actions that led up to that particular error, as opposed to most logging 
>>> systems where this would involve reading all the logs and figuring out 
>>> which ones apply and which to ignore.
>>> 
>>> Most logging systems can tell you what happened; Eliot tells you why it 
>>> happened:
>> 
>> Cool, this looks awesome Itamar, thanks for letting us know!
>> 
>> Does Eliot make use of the new logging system at all?  If not, is there 
>> anything that might make Eliot's Twisted backend easier to maintain if we 
>> added it?  Just kind of curious about adoption of those new APIs in the 
>> world of logging...
> 
> As a user of Eliot and Twisted, an unpleasantness I commonly encounter is 
> that the logs written by trial when running tests for code that uses Eliot 
> for logging are a mish-mash of text messages and mangled Eliot logs.  The 
> Eliot bits often get in the way of me reading the text part and the text 
> parts and mangling ensure I can't use any Eliot tools to interpret the Eliot 
> bits.
> 
> My understanding is that there aren't any existing hooks in Twisted to 
> customize trial's logging which would allow this to be improved.
> 
> There is some related discussion (maybe more oriented towards a particular 
> solution than a full description of the problem, though) on a ticket soon to 
> be a couple years old:
> 
> https://twistedmatrix.com/trac/ticket/6939 
> 

I actually ran into a variant of this exact same issue recently; thank you for 
directing my attention to this ticket.

I agree that the log output should be customizable, but I'm curious: if the 
logs were just serialized as JSON by default everywhere, would that make it 
easier to make the Eliot tooling work well?

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] ANN: Eliot 0.9, the logging system with causality - now with journald support

[exarkun]

On 8 Oct, 11:38 pm, gl...@twistedmatrix.com wrote:
On Oct 8, 2015, at 6:04 AM, Itamar Turner-Trauring 
 wrote:


Eliot 0.9 is out, with a new utility for pretty-printing log messages 
and native journald support 
. You can now 
route Eliot logs to journald and when an error occurs easily find all 
logged actions that led up to that particular error, as opposed to 
most logging systems where this would involve reading all the logs and 
figuring out which ones apply and which to ignore.


Most logging systems can tell you what happened; Eliot tells you why 
it happened:


Cool, this looks awesome Itamar, thanks for letting us know!

Does Eliot make use of the new logging system at all?  If not, is there 
anything that might make Eliot's Twisted backend easier to maintain if 
we added it?  Just kind of curious about adoption of those new APIs in 
the world of logging...


As a user of Eliot and Twisted, an unpleasantness I commonly encounter 
is that the logs written by trial when running tests for code that uses 
Eliot for logging are a mish-mash of text messages and mangled Eliot 
logs.  The Eliot bits often get in the way of me reading the text part 
and the text parts and mangling ensure I can't use any Eliot tools to 
interpret the Eliot bits.


My understanding is that there aren't any existing hooks in Twisted to 
customize trial's logging which would allow this to be improved.


There is some related discussion (maybe more oriented towards a 
particular solution than a full description of the problem, though) on a 
ticket soon to be a couple years old:


 https://twistedmatrix.com/trac/ticket/6939

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] ANN: Eliot 0.9, the logging system with causality - now with journald support

[Glyph Lefkowitz]

> On Oct 8, 2015, at 6:04 AM, Itamar Turner-Trauring  
> wrote:
> 
> Eliot 0.9 is out, with a new utility for pretty-printing log messages and 
> native journald support 
> . You can now route 
> Eliot logs to journald and when an error occurs easily find all logged 
> actions that led up to that particular error, as opposed to most logging 
> systems where this would involve reading all the logs and figuring out which 
> ones apply and which to ignore.
>  
> Most logging systems can tell you what happened; Eliot tells you why it 
> happened:

Cool, this looks awesome Itamar, thanks for letting us know!

Does Eliot make use of the new logging system at all?  If not, is there 
anything that might make Eliot's Twisted backend easier to maintain if we added 
it?  Just kind of curious about adoption of those new APIs in the world of 
logging...

-glyph___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] txmongo vs. pymongo + callInThread

[Jonathan Vanasco]

On Oct 8, 2015, at 6:19 AM, bret curtis wrote:

> We're not sure about what OP has done, but if there is ever anything lacking 
> in TxMongo... please open an issue/ticket with us over at:
> https://github.com/twisted/txmongo

Thanks for the reply!   I actually did open a few lowball tickets on TxMongo 
yesterday -- the package doesn't specify the minimum required versions for 
mongo or twisted.  

I did some tests yesterday on PyMongo, and it seems like using it in 
deferToThread is safe.  I also ran some simulations on staging under a heavy 
load; no exceptions, everything passes, data is stored correctly.   The current 
pymongo "client" appears to be a threadsafe connection pool, and I can just 
grab a connection from it when needed in a thread.  Woo Hoo!

If the next version of this tool stays in Python, we'll definitely need to use 
TxMongo.  In the current design, we're already deferred to a thread and would 
have to restructure too much to use TxMongo.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] ANN: Eliot 0.9, the logging system with causality - now with journald support

[Itamar Turner-Trauring]

 
Eliot 0.9 is out, with a new utility for pretty-printing log messages
and native journald support [1]. You can now route Eliot logs to
journald and when an error occurs easily find all logged actions that
led up to that particular error, as opposed to most logging systems
where this would involve reading all the logs and figuring out which
ones apply and which to ignore. 

Most logging systems can tell you what happened; Eliot tells you _why_
it happened:

$ python linkcheck.py | eliot-tree
4c42a789-76f5-4f0b-b154-3dd0e3041445
+-- check_links@1/started
`-- urls: [u'http://google.com', u'http://nosuchurl']
+-- download@2,1/started
`-- url: http://google.com
+-- download@2,2/succeeded
+-- download@3,1/started
`-- url: http://nosuchurl
+-- download@3,2/failed
|-- exception: requests.exceptions.ConnectionError
|-- reason: ('Conn aborted', gaierror(-2, 'Name unknown'))
+-- check_links@4/failed
|-- exception: exceptions.ValueError
|-- reason: ('Conn aborted.', gaierror(-2, 'Name unknown'))

And here's the code that generated these logs (eliot-tree [2] was used
to render the output):

import sys
from eliot import start_action, to_file
import requests
to_file(sys.stdout)

def check_links(urls):
with start_action(action_type="check_links", urls=urls):
for url in urls:
try:
with start_action(action_type="download", url=url):
response = requests.get(url)
response.raise_for_status()
except Exception as e:
raise ValueError(str(e))

check_links(["http://google.com";], ["http://nosuchurl";])

Interested? Read more at https://eliot.readthedocs.org/.

Eliot is released under the Apache License 2 by ClusterHQ [3], the
Container Data People. We're hiring! [4]
 

Links:
--
[1] http://eliot.readthedocs.org/en/0.9.0/journald.html
[2] https://warehouse.python.org/project/eliot-tree/
[3] https://clusterhq.com
[4] https://clusterhq.com/careers/
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] txmongo vs. pymongo + callInThread

[bret curtis]

We're not sure about what OP has done, but if there is ever anything
lacking in TxMongo... please open an issue/ticket with us over at:
https://github.com/twisted/txmongo

Cheers,
Bret

On Wed, Oct 7, 2015 at 8:18 PM, Jonathan Vanasco 
wrote:

>
> On Dec 21, 2014, at 4:05 PM, Clayton Daley wrote:
>
> > Are there any major disadvantages of using pymongo with callInThread
> instead of txmongo?  I'd like to take advantage of some newer features in
> pymongo (unfortunately not available in txmongo) and it's certainly easier
> to maintain feature parity using callInThread.
>
> I know thread is a bit old, but I'm wondering if the OP ended up using
> callInThread or not , and if there were any issues.
>
> I started working on a txmongo solution to offload some storage, but then
> realized I was already in a deferred thread (to handle blocking operations)
> and i probably should be using pymongo.
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python