Re: [Twisted-Python] Twisted as a House for Sale

[Tom Prince]

Glyph  writes:
> I'm pretty sure we can scare up some hosting resources for things if
> you want to have some kind of demo persistent Twisted services
> running.  twistedmatrix.com is no longer buckling under the pressure
> of its users - why, I just ssh'd in, and the load is less than 1.0!
> By our historical standards that's practically idle! ;-) And we have
> some other machines that we have not had the system administration
> bandwidth to make effective use of.

And we have access to various clouds, to deploy things on.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Advice Request: Under what circumstances should I use AMP's Command Response field?

[Laurens Van Houtven]

Hi Burak,

I don't have time to write a more complete response right now, but you
might want to get in touch with Pete Fein (@wearpants). He was trying to do
this, but with the added restriction that it had to work on WinXP Tomato's
opinion of IE6.

cheers
lvh
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] SQL ORM for Twisted & PostgreSQL?

[Matt Haggard]

FWIW, the separation of generating SQL and executing SQL was my intent
in designing this: https://github.com/iffy/norm  Currently, it *only*
has asynchronous SQL execution, but it wouldn't be hard to add
synchronous execution.

Also, there's no subclassing of modeled classes.

Matt Haggard

On Wed, Aug 21, 2013 at 12:35 PM, Glyph  wrote:
>
> On Aug 21, 2013, at 11:05 AM, Jonathan Vanasco 
> wrote:
>
> Having a great ORM for twisted is wonderful , but its way less exciting and
> attractive if it's only for twisted.
>
>
> Doing this is simple, although probably not easy: you just need to convince
> the SQLAlchemy folks to separate the process of generating SQL and executing
> SQL, and expose hooks for event-driven frameworks (which is an
> ever-expanding circle now, encompassing Twisted, Tornado, Pulsar, and Tulip)
> to schedule SQL execution with an event-driven backend rather than assuming
> it can block.
>
> When I faced this problem, I wanted to use SQLAlchemy rather than write my
> own ORM, but I was unable to due the lack of a public API for generating
> cross-database SQL.
>
> -glyph
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] SQL ORM for Twisted & PostgreSQL?

[Mike Winter (miwinter)]

This reminded me of something I read years ago from Doug Schmidt on a pattern 
half sync half async:

http://www.cs.wustl.edu/~schmidt/PDF/PLoP-95.pdf

I feel it is relevant but the picture may be upside down in relation to writing 
twisted wrappers which may want to wrap a blocking API.


On Aug 21, 2013, at 11:05 AM, Jonathan Vanasco 
mailto:twisted-pyt...@2xlp.com>> wrote:

Having a great ORM for twisted is wonderful , but its way less exciting and 
attractive if it's only for twisted.

Doing this is simple, although probably not easy: you just need to convince the 
SQLAlchemy folks to separate the process of generating SQL and executing SQL, 
and expose hooks for event-driven frameworks (which is an ever-expanding circle 
now, encompassing Twisted, Tornado, Pulsar, and Tulip) to schedule SQL 
execution with an event-driven backend rather than assuming it can block.

When I faced this problem, I wanted to use SQLAlchemy rather than write my own 
ORM, but I was unable to due the lack of a public API for generating 
cross-database SQL.

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] SQL ORM for Twisted & PostgreSQL?

[Glyph]

On Aug 21, 2013, at 11:05 AM, Jonathan Vanasco  wrote:

> Having a great ORM for twisted is wonderful , but its way less exciting and 
> attractive if it's only for twisted.

Doing this is simple, although probably not easy: you just need to convince the 
SQLAlchemy folks to separate the process of generating SQL and executing SQL, 
and expose hooks for event-driven frameworks (which is an ever-expanding circle 
now, encompassing Twisted, Tornado, Pulsar, and Tulip) to schedule SQL 
execution with an event-driven backend rather than assuming it can block.

When I faced this problem, I wanted to use SQLAlchemy rather than write my own 
ORM, but I was unable to due the lack of a public API for generating 
cross-database SQL.

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted as a House for Sale

[Glyph]

On Aug 21, 2013, at 10:02 AM, Matt Haggard  wrote:

> I've got the start of a demo multi-protocol chat server, if anyone's
> interested in taking a look.  (Is this the right place for this
> discussion?  Would IRC be better?)

This is a fine place for discussion.  If you prefer IRC, by all means use it, 
but mailing lists create a more persistent artifact that is easier to reference 
later.

> Run it on your own machine by doing this:
> # make a virtualenv if you want
> git clone -b demo https://github.com/iffy/twistedftw.git
> cd twistedftw
> pip install -r demo_requirements.txt
> twistd -n txftw
> # go to http://127.0.0.1:8400

Cool.

> I'm thinking I'll add an SSH and SCP element, too.  So far, it seems
> kind of boring.  Maybe a game of some kind would be a better demo?  I
> appreciate any feedback.

I think that as long as you make it easy to hack it up, it's OK to be a bit 
boring at first; the real thrill comes from editing the code and watching it 
change.

If you really want to make it exciting though, there's always 

 if you're feeling adventurous ;-).

-g


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] SQL ORM for Twisted & PostgreSQL?

[Burak Nehbit]

I am currently using SQLAlchemy with Twisted with deferToThread and it works 
rather well, have you tried it? So long as you create a new session for each 
thread you spawn (which you should also do without Twisted) it works without 
any modification required.

Here's an example of using SQLAlchemy with Twisted— no guarantees on it's the 
best way to do things, but it works.

def checkIfNodeExists(nodeId):
def threadFunction():
s = Session()
r =  s.query(Node).filter(Node.NodeId == nodeId).count()
if r is not 0:
return True
else:
return False

return threads.deferToThread(threadFunction)



On Aug 21, 2013, at 9:05 PM, Jonathan Vanasco  wrote:

> 
> It would be really beneficial if this were something that fully works with 
> twisted, but is not dependent on it.
> 
> For example, I have a "Project" that mostly uses SqlAlchemy.  It started out 
> in Pylons, new development is on Pyramid and there are additional tasks in 
> Celery + some more in Twisted.  Aside from a few manual db tasks in Twisted, 
> they all share and re-use a common "Model" package.  
> 
> Having a great ORM for twisted is wonderful , but its way less exciting and 
> attractive if it's only for twisted.
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python



smime.p7s
Description: S/MIME cryptographic signature
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Advice Request: Under what circumstances should I use AMP's Command Response field?

[Jeffrey Ollie]

Burak, what you're trying to do sounds a little like the Freenet
Project. https://freenetproject.org/  If you can't use their code
directly you can certainly get some inspiration or learn how they
solved similar problems.

-- 
Jeff Ollie

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Advice Request: Under what circumstances should I use AMP's Command Response field?

[Burak Nehbit]

(Only the latter part of this post is related to Twisted. I apologise in 
advance for the former, and you can skip to that half directly to read about 
the gist)

Hi Laurens,

> I think you're doing fine. Distributed systems are just kind of hard :-)

This is encouraging to hear!

> It sounds like your fundamentally building an eventually consistent 
> distributed database. We have a few of those already: it might be 
> significantly less work to just use one of them. I suppose it depends why 
> you're trying to make it distributed. 

I have done a superficial research, so far it points to implementing my own 
solution. Or rather, what I am producing is not a full–fledged eventually 
consistent database at all, it's a loosely knit bunches of people. There is no 
'database' in the classical sense.

> Is this about reliability in the face of e.g. hardware failure, or is this 
> about being able to disseminate data when someone tries to stop you from 
> doing so? Are you trying to protect against byzantine failures too?

It's about the right to free speech, so I would say the latter. I'm from Turkey 
and currently living in the United States, so while I am protected under First 
Amendment, my compatriots are not. This is significant: As you might have 
heard, we had a wave of peaceful protests two months ago against the last green 
space being demolished in the heart of Istanbul, the Gezi Park, to turn the 
space into a shopping mall. The riot police attacked unarmed, stationary people 
with tear gas canisters, which, led to a series of much larger protests (a few 
million large at its height) and an international outcry. Thousands arrested, 
five dead, state apparatus pressing charges for everyone from high schoolers to 
suburban moms. This country is not a banana republic and it's what made these 
so shocking: Turkey is the 15th largest economy in the world, larger than South 
Korea in terms of GDP, in the process of joining the European Union etc. 

The reason I'm telling this is what happened afterwards; after the govt. had to 
stop police violence after the EU / US pressure, they silently began a 
witch–hunt of Twitter, Facebook users and bloggers they deemed to be 'promoting 
armed insurgency against the state', and there are currently around 20-25 
people currently under detention, waiting for trial. Of course, Twitter and 
Facebook does not give Turkish government any IP addresses: They just look at 
people's profiles, and just grab the most likely person having the same name. 
Our very broken legal system allows for up to five years (!) of contempt of 
court without pressing any charges, something which the Islamist government 
really loves to use against its people. 

I am building a tool that allows people to express their opinions without 
necessarily revealing their identities. It's called Aether, a distributed 
network that allows its users—all users are anonymous and unregistered by 
default— to exercise their right to free speech without being endangered by 
state violence. Everything within it is public, and everything posted on Aether 
is in public domain. (And please excuse the holier–than–thou sounding 
copywriting on the webpage—this was my thesis project and it was one of the 
requirements.)

The backend process of this application runs entirely on Twisted. The business 
rules are simple. Consider Alice the local node, Bob the remote node, and Carol 
another remote node. When Bob connects to Alice and gets a list of the posts 
Alice has, it will request the posts he does not have. The posts Alice has 
publicly available are the posts Alice has either a) created, or b) upvoted. If 
Bob, at some point, also likes the post he has gotten from Alice, Bob will also 
start to publicly distribute that post. At that point, it is impossible to 
determine whether Alice or Bob has created this post—Alice might as well have 
gotten the post from a third party which Bob is not aware of. The post, being 
distributed from two nodes, now has have a higher chance of being found by 
Carol. If you extrapolate it to a thousand people who all have upvoted the 
post, it becomes rather impossible to determine the origin. The act of 'liking' 
something is the exact same thing as sharing something, as is 'creating' 
something—there is absolutely no difference, and every node only has the IP 
address of the last ring of the chain. The nodes simply count how many times 
they encounter the same post digest to determine the amount of upvotes the post 
has gotten, and they use it to determine the lineup of posts the app shows to 
its user. Other than that, there is no global database, there is no global 
state, no people in the entire network is aware of all rest—I just strive to 
distribute the maximum amount of popular data to maximum amount of people 
possible. The client application pieces together all this information into a 
coherent whole of topics, subjects and posts. There is also distribution of 
user addres

Re: [Twisted-Python] SQL ORM for Twisted & PostgreSQL?

[Jonathan Vanasco]

It would be really beneficial if this were something that fully works with 
twisted, but is not dependent on it.

For example, I have a "Project" that mostly uses SqlAlchemy.  It started out in 
Pylons, new development is on Pyramid and there are additional tasks in Celery 
+ some more in Twisted.  Aside from a few manual db tasks in Twisted, they all 
share and re-use a common "Model" package.  

Having a great ORM for twisted is wonderful , but its way less exciting and 
attractive if it's only for twisted.
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted as a House for Sale

[Matt Haggard]

I've got the start of a demo multi-protocol chat server, if anyone's
interested in taking a look.  (Is this the right place for this
discussion?  Would IRC be better?)

Run it on your own machine by doing this:
# make a virtualenv if you want
git clone -b demo https://github.com/iffy/twistedftw.git
cd twistedftw
pip install -r demo_requirements.txt
twistd -n txftw
# go to http://127.0.0.1:8400

I'm thinking I'll add an SSH and SCP element, too.  So far, it seems
kind of boring.  Maybe a game of some kind would be a better demo?  I
appreciate any feedback.

Matt Haggard

On Tue, Aug 20, 2013 at 11:09 AM, Zooko O'Whielacronx  wrote:
> I loved the story! Thanks.
>
> I think the idea of a separate site for the newcomer audience has a
> lot of promise. Also that site might have different (primary)
> maintainers than the twistedmatrix.com site. I guess maybe one of the
> values of the "front door site" is what it excludes. All the
> documentation which is either incomplete or past its prime could be
> kept off of the front door site.
>
> http://iffy.github.io/twistedftw/ looks very promising!
>
> Regards,
>
> Zooko
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Advice Request: Under what circumstances should I use AMP's Command Response field?

[Laurens Van Houtven]

Hi Burak,


I think you're doing fine. Distributed systems are just kind of hard :-)

It sounds like your fundamentally building an eventually consistent
distributed database. We have a few of those already: it might be
significantly less work to just use one of them. I suppose it depends why
you're trying to make it distributed. Is this about reliability in the face
of e.g. hardware failure, or is this about being able to disseminate data
when someone tries to stop you from doing so? Are you trying to protect
against byzantine failures too?

That said, you might want to consider how you communicate posts. Six months
worth of posts is a lot. Even with ten posts per day, you'd end up with
~10*30*6 = 1800 hash values. The digest size of BLAKE2 is variable, but if
you're using 512 bit digests, that's 64 bytes, or 112.5 kibibytes for the
whole thing. That's probably more than you want to send in a single message.

While you can't rely on synchronized clocks (in the wall-clock time sense)
in a distributed system, you *can* rely on timestamps of your immutable
messages. You could send only message id's in the preceding time window,
for example. You can use hash chains to guarantee that the boards share the
same history.

cheers
lvh
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python