Re: [pylons-discuss] Re: SQLAlchemy 2.0 support

[Zsolt Ero]

I just use "PYTHONWARNINGS=default", afaik that's all I need to do right?
It shows me all the Pyramid and waitress warnings.



On 28. Nov 2022 at 19:53:27, Jonathan Vanasco  wrote:

> On Sunday, November 27, 2022 at 1:23:21 PM UTC-5 zsol...@gmail.com wrote:
>
>> Great to know! About the warnings, I'm on 2.0 and it works, so either
>> some of those RemovedIn20Warning are not removed or none of them are left.
>>
>
> The warnings are still there, you most likely have fully compatible code.
> Congrats!
>
> One small thing to look out for: SqlAlchemy does some extra stuff now to
> make sure warnings are visible.  I don't think zope.sqlalchemy or many
> other projects do this.  TLDR: Python started to hide "warnings" around
> 2010, and you need to opt-in to seeing them. This caused many issues with
> SqlAlchemy, because the "postgres" driver name had been deprecated for 10+
> years but people were still using it because they missed the "warnings".
>
> I mean I rewrote my queries to 2.0 style, but I've read that 1.x style
>> queries will continue to work, they are just removed from the documentation
>> now.
>>
>
> Yes.  AFAIK, there is no planned deprecation for it.  Mike re-wrote the
> tutorial for 2.0 and we had a few people help out in older docs to remove
> all the 1.x query instructions.  IIRC, they should still be around in the
> FAQ and some various docs for legacy troubleshooting, but there is a
> decision for all new development to happen in the 2.0 API style and to help
> people migrate to the new syntax.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "pylons-discuss" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/pylons-discuss/sDMJlpQQedM/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> pylons-discuss+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/0a5f033b-4bc5-45c7-88c1-f6c8c56a600cn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAKw-smCA8Wxu0Q9FgLVERXZNFvtHdfUehE1Jb5SB6xKKTfrEig%40mail.gmail.com.

Re: [pylons-discuss] Re: SQLAlchemy 2.0 support

[Zsolt Ero]

Great to know! About the warnings, I'm on 2.0 and it works, so either some
of those RemovedIn20Warning are not removed or none of them are left.

I mean I rewrote my queries to 2.0 style, but I've read that 1.x style
queries will continue to work, they are just removed from the documentation
now.

For cookiecutter I recommend changing to using DeclarativeBase and Mapped
+ mapped_column in the model, once 2.0 is released.

Zsolt





On 27. Nov 2022 at 19:02:33, Jonathan Vanasco  wrote:

> There was one potential incompatibility with transaction, but the
> zope.sqlalchemy team addressed it already ( see
> https://github.com/zopefoundation/zope.sqlalchemy/issues/60 ).  There are
> deprecation warnings that are still unhandled ( see
> https://github.com/zopefoundation/zope.sqlalchemy/pull/64 ) but
> everything else should be fine.
>
> Most of the 2.0 sqlalchemy changes are in the engine and query systems,
> and pretty isolated from the support the cookiecutter and zope.sqlalchemy
> provide.
> On Thursday, November 24, 2022 at 1:38:01 PM UTC-5 zsol...@gmail.com
> wrote:
>
>> Hi,
>>
>> I'm starting a new project from cookiecutter and I wanted to use
>> SQLAlchemy 2.0 (pre-release currently).
>>
>> So far I changed meta.py to
>>
>> class Base(DeclarativeBase):
>> metadata = my_metadata
>>
>> as well as changing the model to the new Mapped / mapped_column syntax.
>>
>> My question is that is there anything in pyramid_tm, transaction or
>> zope.sqlalchemy packages which needs to be changed, or this was all I
>> needed?
>>
>> SQLAlchemy changed a *lot* in 2.0, that's why I think the whole
>> transaction related part might be affected.
>>
>> If it's the case then I'll just go back to 1.4 and wait a year or so till
>> it matures. What do you think?
>>
>> Zsolt
>>
>>
>>
>>
>> --
> You received this message because you are subscribed to a topic in the
> Google Groups "pylons-discuss" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/pylons-discuss/sDMJlpQQedM/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> pylons-discuss+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/bb45eb53-c263-4d35-af74-60691ca010d5n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAKw-smB15UqAWVmdnUQAkeiO_CR1eLYYgHkWi7Naeb61TAvAJw%40mail.gmail.com.

[pylons-discuss] Re: webtest upload file in body

[Zsolt Ero]

 I think I figured this one out, I can pass the bytes into params= and
it'll work. Strange naming though.



On 24. Jun 2021 at 20:56:50, Zsolt Ero  wrote:

> Hi,
>
> How can I upload a file in the body in webtest.post? requests allows me
> to do this in data=, curl in --data-binary.
>
> Webob reads this in body_file_seekable. How can I do this in webtest.post?
>
> Regards,
> Zsolt
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAKw-smBAwXr4oaE5R3%2B6us2TAD1UYrq2gt71OAc%3DjN-5YUhQQA%40mail.gmail.com.

[pylons-discuss] webtest upload file in body

[Zsolt Ero]

Hi,

How can I upload a file in the body in webtest.post? requests allows me to
do this in data=, curl in --data-binary.

Webob reads this in body_file_seekable. How can I do this in webtest.post?

Regards,
Zsolt

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAKw-smDWxxsPCWFRnYoxTeaKF8gED%3D8Oq8dtB5P%3DPD6-dtuXKQ%40mail.gmail.com.

[pylons-discuss] modifying database in webtest based testing

[Zsolt Ero]

Hi,

I have my webtest based testing set up like the following.

@pytest.fixture(scope="session")
def app():
testing_config = {
'sqlalchemy.url': 'postgresql+psycopg2://x@/x',
'redis.sessions.secret': 'x',  # random string
'redis.sessions.db': 1,
'tm.annotate_user': 'no',
}

return main({"testing": True}, **testing_config)


@pytest.fixture
def testapp(app):
return TestApp(app)

main() is a config.make_wsgi_app().

My questions is, how can I have direct database connection - normally a
request.dbsession in the app - in the testing fixtures? I'd like to run
some DB queries in the setup/teardown process.

I can access testapp.app or "app" from the fixtures, that works. What I
don't know is how can I get a request object from there.

In command line scripts I use

request_dummy = Request.blank('/', base_url=base_url)
env = bootstrap(config_uri, request=request_dummy)
request = env['request']

Do I need to do something similar here?

Zsolt

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAKw-smAqGQjoX1h%2B_rWHjh6Pvi95Vb5RVQ5E%3Djf91B8AW6b1Yg%40mail.gmail.com.

Re: [pylons-discuss] minimal auth/security policy implementation

[Zsolt Ero]

 Hi Theron,

Thanks for your reply. It looks indeed simpler. How much more minimal can I
make it? I definitely want to "circumvent" the whole security system, I'm
perfectly happy with using my new require_admin=True like options.

I just want CSRF to work and it seems to be dependent on RootFactory being
defined, which I don't understand.

Zsolt






On 24. May 2021 at 20:31:45, Theron Luhn  wrote:

> You may have better luck with the Pyramid 2.0 security system.  It’s much
> simpler for cases like yours where you don’t need ACL.  For example, your
> implementation might look like:
>
> class CustomSecurityPolicy:
>   def identity(self, request):
> return request.user
>
>   def authenticated_userid(self, request):
> return request.user.id if request.user else None
>
>   def permits(self, request, context, permission):
> if permission == ‘user’ and request.user:
>   return Allowed(‘User is signed in.’)
> elif permission == ‘admin’ and request.user and request.user.id == 1:
>   return Allowed(‘Admin user is signed in.’)
> else:
>   return Denied(‘Access is not allowed.’)
>
>   def remember(request, userid, **kw):
> …  # Same as before
>
>   def forget(request, **kw):
> …
>
> That’s all.  No ACL or root factory, just
> identity()/authenticated_userid() returning the current user and permits()
> giving a thumbs up or down if access should be allowed.  Docs:
> https://docs.pylonsproject.org/projects/pyramid/en/latest/narr/security.html
>
>
> View derivers would certainly work.  After all, the security system itself
> is implemented with a view deriver.  But personally I would avoid
> circumventing the entire security system like that.
>
> — Theron
>
>
>
> On May 22, 2021, at 2:16 PM, zsol...@gmail.com 
> wrote:
>
> Hi,
>
> I've been using the following auth policies for years, it's been working
> fine:
>
> authn_policy = CustomSessionAuthenticationPolicy()
> authz_policy = ACLAuthorizationPolicy()
>
> config = Configurator(
> settings=settings,
> root_factory=RootFactory,
> authentication_policy=authn_policy,
> authorization_policy=authz_policy,
> )
>
>
> class RootFactory(object):
> __acl__ = [
> (Allow, Authenticated, 'user'),
> (Allow, 'g:admin', 'admin'),
> (Allow, 'g:superadmin', ALL_PERMISSIONS),
> ]
>
> def __init__(self, request):
> pass
>
>
>
> class CustomSessionAuthenticationPolicy(SessionAuthenticationPolicy):
> def authenticated_userid(self, request):
> return request.user.id
>
> def effective_principals(self, request):
> principals = [Everyone]
> if request.user:
> principals += [Authenticated]
>
> if request.user.id == 1:
> principals += ['g:superadmin', 'g:admin']
>
> return principals
>
> ---
>
> I'm trying to migrate off from this, as I simply don't understand what is
> happening behind and I prefer a much simpler view deriver based approach.
>
> Basically, with a couple of view derivers I could solve all my problems in
> a few hours, and it also allows me much more flexibility. For example for
> some views now I can do auth based on API tokens, while most of the views
> are using session based auth.
>
> My questions is, how can I make the auth/security policies as simple as
> possible? All I need is working CSRF,  remember and forget.
>
> I'm on 1.10 but I'm happy to migrate to 2.0 if that allows a simplified
> approach.
>
> So far I was able to get it down to this:
>
> config = Configurator(
> settings=settings,
> root_factory=RootFactory,
> authentication_policy=SessionAuthenticationPolicy(),
> )
>
> class RootFactory(object):
> __acl__ = [
> (Allow, Authenticated, 'user'),
> ]
>
> def __init__(self, request):
> pass
>
> Session is via pyramid_session_redis.
>
> Thanks,
> Zsolt
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pylons-discuss+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/60c5a72f-c847-46a9-8e5f-3ed2521f55a1n%40googlegroups.com
> 
> .
>
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "pylons-discuss" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/pylons-discuss/7BKhj0G-mbg/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> pylons-discuss+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/19F53725-D9C4-4D09-950A-CD92C46CBDCF%40luhn.com
> 

Re: [pylons-discuss] changing session cookie max_age?

[Zsolt Ero]

 Just for reference I'd like to post what worked for me. Thanks for the
detailed help.

Finally I've settled on the following values:
```
redis.sessions.secret = xxx
redis.sessions.cookie_max_age = 31536   # 10 years, basically forever
redis.sessions.timeout = 1800
redis.sessions.cookie_secure = True
redis.sessions.cookie_httponly = True
redis.sessions.cookie_samesite = lax
```

login:
```
headers = remember(request, user.id)

redis_timeout = 3600 * 24 * 365  # one year in Redis
request.session.adjust_timeout_for_session(redis_timeout)

return HTTPFound(location=next, headers=headers)
```

I've thought about it and analyzed it and come up with the solution that
this will work well for my usecase. I've never experienced any problem with
the previous version of the library with similar values, which have created
way more sessions then this one, as this only creates a session when it's
actually needed on a login/registration page, leaving home page, etc.
session-less.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAKw-smCUE%3DwgPfvFLpR9%2B21r_2gey27hHYopxOK43LYzHor76w%40mail.gmail.com.

[pylons-discuss] Non-multithread compatible modules and waitress

[Zsolt Ero]

I had a few questions in
https://github.com/Pylons/waitress/issues/253, which were not really
an issue and I'd like to ask them here instead.

My question mostly is that if I'm using modules which are
non-multithread compatible in pserve / waitress, how can I make sure
they work correctly? Shouldn't I reset / reinitialize them at some
point of the startup process?

For example, Google client Python libraries, which are not thread
safe, as they are based on httplib2.
https://developers.google.com/api-client-library/python/guide/thread_safety

Zsolt

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAKw-smCwEmXb-qyh%3D0hfSHTjqt%3DOMV_BzSaAaSn3FKGYYbOkhQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] How to detect WebOb NoVars case

[Zsolt Ero]

There is a bug in WTForms 2.2+ which related to how WebOb handle 
request.POST in GET requests. 
https://github.com/wtforms/wtforms/issues/460

We are trying to find a WebOb specific solution with the maintainer, to 
handle the NoVars case. I just wanted to ask, what approach would you 
recommend for detecting this case, without importing WebOb? 

My idea would be to use type(...).__name__ == 'NoVars'. Is this a good 
solution?


-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/0b3081e8-93e9-4d31-9af7-960d0b3da626%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Questions about DB engine setup

[Zsolt Ero]

Thanks for the answers.

I've been using gunicorn with sync workers and it seems clear to me.
If I understand correctly, up-till-some-point, everything in the
script is shared across the processes, and these receive identical ids
in Python. Everything which happens at import time is definitely in
this "shared" memory space. Client libraries which cannot be shared,
need to be regenerated / disposed / taken care of like with
engine.dispose().

The fact that database engines get different ids in each worker means
to me that I'm totally safe under gunicorn sync with SQLAlchemy.

With waitress on the other hand, I'm surprised. I mean if waitress is
multi-threaded, then how come it doesn't require every single module
and sub-dependency to be thread-safe? I've only been using waitress
for local development via pserve, but I see it's a popular choice for
a production server as well. How can it work in a multi-threaded mode,
if you have no idea about every single library's every dependency you
use? Or you actually check out all your libraries?

Now about SQL, why do I need any kind of transaction implementation? I
mean isn't

with engine.connect() as conn:
  conn.execute(stmt)

 a transaction? I'd just need to explicitly pass conn to every
function which need DB access, but I don't see why is it important to
do anything more than this, even with write queries. (I don't know how
would I have DB access in view derivers with this pattern though).

Zsolt

On Wed, 1 May 2019 at 19:46, Jonathan Vanasco  wrote:
>
> > As I understand each worker needs it's own engine instance, don't they? I 
> > think the gunicorn behaviour is good, but I'm puzzled by the 
> > pserve/Waitress behaviour. Is this by design?
>
> Just to expand on Michael's comment on there usually only being one engine 
> but multiple connections...
>
> Worker's don't need their own engine instances, but they need their own 
> connection pools.  gunicorn (and uwsgi) implement a forked process model, and 
> the database connections are not safe to move across the process boundaries 
> due to some things inherent to Python and/or filesystems .  if you connect to 
> the database during setup in this model, you need to call `engine.dispose()` 
> when setup is complete or after the fork, or else weird and bad things will 
> eventually happen.  see 
> https://docs.sqlalchemy.org/en/13/core/connections.html#engine-disposal .   
> waitress is multithreaded and the pool is supposed to be safe across process 
> boundaries.
>
> You should be able to just do an Engine on the request, but if you do ensure 
> you do the `dispose()` or set up an event to ensure process safety.  it's all 
> the pooling docs https://docs.sqlalchemy.org/en/13/core/pooling.html
> you'll also have to implement your own transaction management, which is 
> honestly not fun.
>
> if you are sure you're only doing SELECTS... you could have a dedicated 
> engine/connection string for a read-only database user and have that bypass 
> the transaction system entirely.  i use that pattern a lot, but you need to 
> make sure you enforce it as read-only at the database credentials level or 
> you have the potential for a lot of issues bubbling up.
>
>
> --
> You received this message because you are subscribed to a topic in the Google 
> Groups "pylons-discuss" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/pylons-discuss/8Rndsn6oLyg/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to 
> pylons-discuss+unsubscr...@googlegroups.com.
> To post to this group, send email to pylons-discuss@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/pylons-discuss/0e53f127-d115-4724-95aa-5cfd14f23223%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAKw-smDgNYOx9oOxePp7vWAxVC67KzPgfmmayA5uKGcoBYKfNA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Questions about DB engine setup

[Zsolt Ero]

Thanks a lot for all the answers. So far I've only implemented select
queries, using dbsession.execute. I'll read about the "changed" state
handling.

About the additional options: pool_pre_ping seems to work well and
doesn't seem to have any side effects.

pool_reset_on_return on the other hand is interesting, as it did make
my development queries 3x faster. (I did 50 curl queries and it was
600 ms with rollback vs 200 ms with none for a "select 1" endpoint,
between EU <> US).

My only problem was that it left the connection in "idle in
transaction" state, on which we have a 5 minute timer on the server,
so the connection get's disconnected. That breaks SQLAlchemy. Because
of this, I've commented it out (i.e. left it on "rollback").

I think this would only work if I didn't use transactions, but just
direct queries.

Wouldn't it be possible to simply add the engine to request and then
do something like with request.engine.connect() as conn: ?

// I know this is only affecting dev setup, and the overhead of a
transaction + rollback is almost nothing within the same datacenter,
but still, I'm curious about how much of this session + transaction
mechanism is needed for select-only queries.

Zsolt


On Mon, 29 Apr 2019 at 18:24, Jonathan Vanasco  wrote:
>
>
>
> On Saturday, April 27, 2019 at 11:43:33 PM UTC-4, Mike Orr wrote:
>
>>
>> 'session.execute()' executes SQL like 'engine.execute()' does, so you
>> can get the advantages of a request transaction without using ORM
>> queries.
>
>
> in addition to Mike's comments - make sure to read the zope.sqlalchemy docs 
> on `mark_changed` or starting the session on a 'changed' state
>
> https://pypi.org/project/zope.sqlalchemy/
>
>
> if you don't do that, sqlalchemy won't know you did something and will 
> rollback instead of commit.
>
> --
> You received this message because you are subscribed to a topic in the Google 
> Groups "pylons-discuss" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/pylons-discuss/8Rndsn6oLyg/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to 
> pylons-discuss+unsubscr...@googlegroups.com.
> To post to this group, send email to pylons-discuss@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/pylons-discuss/6015d75c-f277-42cb-880c-9d30632fdcb5%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAKw-smD4XG3k-3U0sJ8Pq9Oxx3bObG0uB1wfhPrN97M%2BMRC3ig%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Questions about DB engine setup

[Zsolt Ero]

I'm trying to set up an API which would use SQLAlchemy Core (not ORM) + 
PostgreSQL. The server is a Google managed PostgreSQL instance, on external 
IP.

I have a couple of questions. Since I needed to manually add SSL 
certificates as connect_args to create_engine + some additional arguments, 
I'm using create_engine(). My questions are related to this

1. Does this look ok?

import zope.sqlalchemy
from populus_lib.config import in_worker, pg_certs, pg_url
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker

def get_session_factory(engine):
factory = sessionmaker()
factory.configure(bind=engine)
return factory

def get_tm_session(session_factory, transaction_manager):
dbsession = session_factory()
zope.sqlalchemy.register(dbsession, transaction_manager=
transaction_manager)
return dbsession

def includeme(config):
settings = config.get_settings()
settings['tm.manager_hook'] = 'pyramid_tm.explicit_manager'

config.include('pyramid_tm')

engine = create_engine(
pg_url,
connect_args=pg_certs,
pool_pre_ping=True,
pool_reset_on_return='rollback' if in_worker else None,  # 
in_worker means production
)

session_factory = get_session_factory(engine)
config.registry['dbsession_factory'] = session_factory

config.add_request_method(
lambda r: get_tm_session(session_factory, r.tm),
'dbsession',
reify=True,
)


2. Since I'm not using ORM, but core only, do I need from sqlalchemy.orm 
import sessionmaker?

3. Is pool_pre_ping supported with Pyramid's way of session/transaction 
handling? I want to be sure that external server disconnects/reconnects are 
handled automatically and I think using pool_pre_ping is the best for this.

4. Isn't pool_reset_on_return conflicting pyramid_tm / session handling? I 
only need to use this in development, since the SQL server is in US and I'm 
in Europe and without this settings SQLAlchemy has a huge overhead on each 
query, like 300 ms.

5, Finally, what's puzzling me is that if I create a view like this:
def ping(request):
print(id(request.dbsession.connection().engine))
sleep(60)

And I run this via curl from two concurrent terminal windows, I get equal 
ids in pserve / Waitress, while I get different ids with gunicorn defaults 
(which I believe is multiprocessing).

As I understand each worker needs it's own engine instance, don't they? I 
think the gunicorn behaviour is good, but I'm puzzled by the 
pserve/Waitress behaviour. Is this by design?

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/e6dc1571-5b50-4b77-be1e-62c9c5964f4d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] processing after responding to client

[Zsolt Ero]

So I guess the simplest possible solution is Celery + Redis (since I'm
already using Redis for sessions), I guess?

On Thu, 10 Jan 2019 at 13:55, Thierry Florac  wrote:
>
> Hi,
>
> After-commit hooks don't work in async mode!
> They allow to declare callbacks which will be called after transaction 
> commit, but server is waiting for these callbacks to be finished (in 
> synchronous mode) before sending response to client.
> If you need async mode, you have (at least) to start another thread which 
> will handle your asynchronous tasks.
>
> Regards,
> Thierry
>
>
> Le jeu. 10 janv. 2019 à 13:37, Zsolt Ero  a écrit :
>>
>> Actually, it doesn't result in async-like behaviour, the client is
>> still waiting for the function inside the commit hook. I guess I don't
>> understand the concept then.
>>
>> On Thu, 10 Jan 2019 at 13:15, Zsolt Ero  wrote:
>> >
>> > I'm just now implementing this. Are you saying that basically, by using 
>> > addAfterCommitHook, I can replace put relatively fast (say, <10 seconds), 
>> > functions into async-like behaviour, without requiring any external queue 
>> > service?
>> >
>> > old:
>> >
>> > # send email was an async function, via a decorator (using Huey)
>> > send_email(
>> > to_email=user.email,
>> > subject='account activation',
>> > text=email_text,
>> > )
>> >
>> >
>> >
>> >
>> >
>> > new:
>> >
>> > request.tm.get().addAfterCommitHook(send_email_hook, kws=dict(
>> > to_email=user.email,
>> > subject='account activation',
>> > text=email_text,
>> > ))
>> >
>> >
>> > + a helper function:
>> >
>> >
>> > def send_email_hook(success, **kwargs):
>> > if success:
>> > send_email(**kwargs)
>> >
>> > I have two questions regarding this:
>> > 1. When does success == False?
>> > 2. What does the warning on transaction's doc page mean?
>> >
>> >> Hook functions which do raise or propagate exceptions will leave the 
>> >> application in an undefined state.
>> >
>> >
>> > Practically speaking, should I put the send_email function into a 
>> > try-except block?
>> >
>> > --
>> > You received this message because you are subscribed to a topic in the 
>> > Google Groups "pylons-discuss" group.
>> > To unsubscribe from this topic, visit 
>> > https://groups.google.com/d/topic/pylons-discuss/tf5kKsnZRTc/unsubscribe.
>> > To unsubscribe from this group and all its topics, send an email to 
>> > pylons-discuss+unsubscr...@googlegroups.com.
>> > To post to this group, send email to pylons-discuss@googlegroups.com.
>> > To view this discussion on the web visit 
>> > https://groups.google.com/d/msgid/pylons-discuss/6831e434-6e85-4c41-9797-82c71ad7f2b3%40googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "pylons-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to pylons-discuss+unsubscr...@googlegroups.com.
>> To post to this group, send email to pylons-discuss@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/pylons-discuss/CAKw-smAukCYyjhg7wp2PoP4cUCJ0JuchNqYG0LLHpVGGL%2B84mA%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> http://www.imagesdusport.com -- http://pyams.readthedocs.io
>
> --
> You received this message because you are subscribed to a topic in the Google 
> Groups "pylons-discuss" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/pylons-discuss/tf5kKsnZRTc/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to 
> pylons-discuss+unsubscr...@googlegroups.com.
> To post to this group, send email to pylons-discuss@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/pylons-discuss/CAPX_VWCt_SfPDmYsVSK%2Bwt74j5%2BSFj8vb40LrN9PRBPY2%2Bb7Gg%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAKw-smCPd0j-sCuHzMa0b-sXNdXrRzB%2BWs2JHzsx5Tu6w02c7A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] processing after responding to client

[Zsolt Ero]

Actually, it doesn't result in async-like behaviour, the client is
still waiting for the function inside the commit hook. I guess I don't
understand the concept then.

On Thu, 10 Jan 2019 at 13:15, Zsolt Ero  wrote:
>
> I'm just now implementing this. Are you saying that basically, by using 
> addAfterCommitHook, I can replace put relatively fast (say, <10 seconds), 
> functions into async-like behaviour, without requiring any external queue 
> service?
>
> old:
>
> # send email was an async function, via a decorator (using Huey)
> send_email(
> to_email=user.email,
> subject='account activation',
> text=email_text,
> )
>
>
>
>
>
> new:
>
> request.tm.get().addAfterCommitHook(send_email_hook, kws=dict(
> to_email=user.email,
> subject='account activation',
> text=email_text,
> ))
>
>
> + a helper function:
>
>
> def send_email_hook(success, **kwargs):
> if success:
> send_email(**kwargs)
>
> I have two questions regarding this:
> 1. When does success == False?
> 2. What does the warning on transaction's doc page mean?
>
>> Hook functions which do raise or propagate exceptions will leave the 
>> application in an undefined state.
>
>
> Practically speaking, should I put the send_email function into a try-except 
> block?
>
> --
> You received this message because you are subscribed to a topic in the Google 
> Groups "pylons-discuss" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/pylons-discuss/tf5kKsnZRTc/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to 
> pylons-discuss+unsubscr...@googlegroups.com.
> To post to this group, send email to pylons-discuss@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/pylons-discuss/6831e434-6e85-4c41-9797-82c71ad7f2b3%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAKw-smAukCYyjhg7wp2PoP4cUCJ0JuchNqYG0LLHpVGGL%2B84mA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] processing after responding to client

[Zsolt Ero]

I'm just now implementing this. Are you saying that basically, by using 
addAfterCommitHook, I can replace put relatively fast (say, <10 seconds), 
functions into async-like behaviour, without requiring any external queue 
service?

old:

# send email was an async function, via a decorator (using Huey)
send_email(
to_email=user.email,
subject='account activation',
text=email_text,
)





new: 

request.tm.get().addAfterCommitHook(send_email_hook, kws=dict(
to_email=user.email,
subject='account activation',
text=email_text,
))


+ a helper function:


def send_email_hook(success, **kwargs):
if success:
send_email(**kwargs)

I have two questions regarding this:
1. When does success == False?
2. What does the warning on transaction's doc page mean?

Hook functions which do raise or propagate exceptions will leave the 
> application in an undefined state. 


Practically speaking, should I put the send_email function into a 
try-except block?

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/6831e434-6e85-4c41-9797-82c71ad7f2b3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] processing after responding to client

[Zsolt Ero]

On a fairly classic Pyramid app (sync, gunicorn, SQLAlchemy / Postgresql, 
like the starter template), I am storing analytics-like events in a table 
for some of my views.

Like this skeleton:

@view_config(route_name='test', request_method='POST', renderer='json')
def test(request):
# body of the view
map_id = request.json_body.get('map_id')
data = {...}

event_data = {'action': 'viewed'}
event = MapEvent(user=request.user, action=event_data, map_id=map_id)
request.dbsession.add(event)

return data



My problem is that while 99% of the views make read-only requests to the 
database, and thus are very fast, the analytics event is a writing and can 
be slow occasionally. 

Would it be possible to somehow send the request to the client and still 
keep processing the view? Like a send() + end() method or something 
similar, without returning?

// Adding to a tasks queue is not an option as it'd be an overkill and an 
overcomplicated solution, which wouldn't work for hundreds of thousands of 
events per day. 


-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/c44e1af6-8c30-478e-9baf-d7fd8c93e0b5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: API file upload

[Zsolt Ero]

OK, I'm getting there, althought I'm still confused a bit. In WebOb
docs I found request.body_file, request.body_file_raw,
request.body_file_seekable.

In multipart's request.POST, I'm doing:

file_obj.seek(0, 2)
file_size = file_obj.tell()
file_obj.seek(0)

Should I be using seekable or raw for this?

Zsolt


On 17 April 2018 at 15:10, Zsolt Ero <zsolt@gmail.com> wrote:
> I've realised the following:
> 1. If I don't specify Content-Type, curl defaults to x-www-form-urlencoded
> 2. What I thought is the binary file's contents as a string is
> actually not working reliably. On an XML upload of a single file I get
> thousands of items and request.POST.items() looks like:
> [' 2. The binary string I can use is actually request.body. Still, is
> there any potential problems with handling this as string and not as a
> file object?
>
> Zsolt
>
> On 17 April 2018 at 14:24, Zsolt Ero <zsolt@gmail.com> wrote:
>> Hi,
>>
>> I'm trying to implement an API to a website which didn't have an API
>> yet. It's purpose will be to allow file uploads from 3rd party native
>> apps.
>>
>> I'd like to implement the API like Dropbox v2 API, just as a good
>> reference for API design.
>>
>> It's upload endpoint has the following specs:
>> https://www.dropbox.com/developers/documentation/http/documentation#files-upload
>>
>> And the following cURL example:
>>
>> curl -X POST https://content.dropboxapi.com/2/files/upload \
>> --header "Authorization: Bearer " \
>> --header "Dropbox-API-Arg: {\"path\":
>> \"/Homework/math/Matrices.txt\",\"mode\": \"add\",\"autorename\":
>> true,\"mute\": false}" \
>> --header "Content-Type: application/octet-stream" \
>> --data-binary @local_file.txt
>>
>> Now my problem is that I've implemented most parts, but if the request
>> has --header "Content-Type: application/octet-stream" then WebOb
>> doesn't allow using request.POST. It says:
>>
>> Not an HTML form submission (Content-Type: application/octet-stream)
>>
>> If I remove that header, I can use request.POST.keys()[0] to read the
>> contents of the file as a string.
>>
>> My question is:
>> 1. What am I doing wrong that the Content-Type is not supported?
>> 2. Is there any downside of having an up-to-100 MB file as a string?
>> Wouldn't the HTML multipart-form-data's file solution use less memory?
>> Can I make WebOb handle this kind of uploads like it does multipart
>> ones?
>>
>> Zsolt

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAKw-smBHD9Dg45R1UCu47NTQzYR1H1xdq8_RFyG4E0EKb0zsDw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: API file upload

[Zsolt Ero]

I've realised the following:
1. If I don't specify Content-Type, curl defaults to x-www-form-urlencoded
2. What I thought is the binary file's contents as a string is
actually not working reliably. On an XML upload of a single file I get
thousands of items and request.POST.items() looks like:
[' wrote:
> Hi,
>
> I'm trying to implement an API to a website which didn't have an API
> yet. It's purpose will be to allow file uploads from 3rd party native
> apps.
>
> I'd like to implement the API like Dropbox v2 API, just as a good
> reference for API design.
>
> It's upload endpoint has the following specs:
> https://www.dropbox.com/developers/documentation/http/documentation#files-upload
>
> And the following cURL example:
>
> curl -X POST https://content.dropboxapi.com/2/files/upload \
> --header "Authorization: Bearer " \
> --header "Dropbox-API-Arg: {\"path\":
> \"/Homework/math/Matrices.txt\",\"mode\": \"add\",\"autorename\":
> true,\"mute\": false}" \
> --header "Content-Type: application/octet-stream" \
> --data-binary @local_file.txt
>
> Now my problem is that I've implemented most parts, but if the request
> has --header "Content-Type: application/octet-stream" then WebOb
> doesn't allow using request.POST. It says:
>
> Not an HTML form submission (Content-Type: application/octet-stream)
>
> If I remove that header, I can use request.POST.keys()[0] to read the
> contents of the file as a string.
>
> My question is:
> 1. What am I doing wrong that the Content-Type is not supported?
> 2. Is there any downside of having an up-to-100 MB file as a string?
> Wouldn't the HTML multipart-form-data's file solution use less memory?
> Can I make WebOb handle this kind of uploads like it does multipart
> ones?
>
> Zsolt

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAKw-smA8HwKK5o3RNhQPfwAzo-ATgE1zrzj6TrwsLOzZjcka-A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] API file upload

[Zsolt Ero]

Hi,

I'm trying to implement an API to a website which didn't have an API
yet. It's purpose will be to allow file uploads from 3rd party native
apps.

I'd like to implement the API like Dropbox v2 API, just as a good
reference for API design.

It's upload endpoint has the following specs:
https://www.dropbox.com/developers/documentation/http/documentation#files-upload

And the following cURL example:

curl -X POST https://content.dropboxapi.com/2/files/upload \
--header "Authorization: Bearer " \
--header "Dropbox-API-Arg: {\"path\":
\"/Homework/math/Matrices.txt\",\"mode\": \"add\",\"autorename\":
true,\"mute\": false}" \
--header "Content-Type: application/octet-stream" \
--data-binary @local_file.txt

Now my problem is that I've implemented most parts, but if the request
has --header "Content-Type: application/octet-stream" then WebOb
doesn't allow using request.POST. It says:

Not an HTML form submission (Content-Type: application/octet-stream)

If I remove that header, I can use request.POST.keys()[0] to read the
contents of the file as a string.

My question is:
1. What am I doing wrong that the Content-Type is not supported?
2. Is there any downside of having an up-to-100 MB file as a string?
Wouldn't the HTML multipart-form-data's file solution use less memory?
Can I make WebOb handle this kind of uploads like it does multipart
ones?

Zsolt

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAKw-smAFJd%2BMUcWmTn%3DMV9Mk5NQ1qHQGiMnHj4tVnEyhFJK6Gg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Pyramid + SQLAlchemy + PostgreSQL: idle connections

[Zsolt Ero]

Thanks for the detailed explanation. Yes I was talking about console scripts.

1. First, I had my console scripts based on initializedb.py (with no
explicit manager in models/__init__.py)
https://github.com/Pylons/pyramid-cookiecutter-alchemy/blob/latest/%7B%7Bcookiecutter.repo_name%7D%7D/%7B%7Bcookiecutter.repo_name%7D%7D/scripts/initializedb.py

2. Then I added explicit manager to models/__init__.py which broke my
console scripts running in pshell, so I reverted to implicit.
https://github.com/Pylons/pyramid/issues/3083

3. I migrated my console scripts to use to a request based config,
using this pattern:

setup_logging(config_uri)
settings = get_appsettings(config_uri)
request_dummy = Request.blank('/', base_url=base_url)
env = bootstrap(config_uri, request=request_dummy)
request = env['request']

do_things() # using "with transaction.manager:"

env['closer']()

4. Now, using this new console script structure allowed me to replace
transaction.manager with request.tm and enable explicit manager.

Does my new console script make any sense? Maybe it'd make sense to
modernize initializedb.py like this, as most new users would probably
go through the same path as I did.

Zsolt




On 2 April 2018 at 23:18, Michael Merickel <mmeri...@gmail.com> wrote:
> You almost never want to use "with request.tm". This cannot be nested with
> another call to request.tm.begin() and pyramid_tm already does
> request.tm.begin() for you. You can basically think of pyramid_tm as doing a
> "with request.tm" in a tween around all of your views except for some
> exception views in very rare circumstances (read the pyramid_tm docs for
> more about that).
>
> The explicit vs implicit transaction managers is this: If you call
> tm.begin() twice in a row with the implicit transaction manager then it will
> be basically like calling tm.begin(), tm.abort() tm.begin(). The abort is
> implicit. With the explicit manager if you call tm.begin() twice in a row
> then the second call will raise an exception. It requires an explicit call
> to abort and an explicit call to begin(). This is probably what you expect,
> whereas the implicit workflow is the default bw-compat behavior that is very
> likely not what you expect if you're coming from a more traditional RDBMS
> background.
>
> You'll notice that in normal operation you never call tm.begin() or
> tm.abort()... pyramid_tm does this for you. The explicit manager weeds out
> issues where you might use the database before or after pyramid_tm is
> controlling the lifecycle whereas the implicit manager would silently hide
> some bugs where you call tm.begin() twice without realizing it - thus
> causing some objects to be detached from the first session because of the
> tm.abort() that was called alongside the second tm.begin()... Sorry I can't
> think of a more succinct way of saying it other than "just use the explicit
> manager".
>
> The only time you really need to think about transaction stuff is when
> pyramid_tm is not active which is basically at config-time and in console
> scripts.
>
> - Michael
>
> On Mon, Apr 2, 2018 at 2:54 PM, Zsolt Ero <zsolt@gmail.com> wrote:
>>
>> Thanks. So the key point for me is to just use engine.dispose() (I
>> don't want to dig into gunicorn preload, this seems much cleaner).
>>
>> About explicit manager: in practice it's as simple as
>> 1. enabling it in models/__init__.py
>> 2. using "with request.tm" everywhere instead of "with
>> transaction.manager", right?
>>
>> > This is more likely to be an issue in web requests where your entire
>> > request lifecycle is not actually protected by a transaction.
>>
>> About your final sentence, I'm not sure I understand it. If I'm always
>> using request.dbsession, by definition I'm protected, to a request's
>> lifecycle am I not?
>>
>> Zsolt
>>
>> On 2 April 2018 at 21:47, Michael Merickel <mmeri...@gmail.com> wrote:
>> > The forking issue is likely because you're using a connection pool and
>> > so
>> > once a connection is opened at config-time, even though the session is
>> > properly closed the connection is just returned to the pool. The pool
>> > here
>> > is shared across the fork which is bad. The basic solution here is to
>> > add
>> > some sort of pre-fork hook that closes out every existing connection in
>> > the
>> > pool. I personally have not used forking wsgi servers in production and
>> > thus
>> > have not needed to deal with this problem but I'm pretty sure my advice
>> > hits
>> > at the core issue you're experiencing. Calling dispose() is one way to
>> 

Re: [pylons-discuss] Pyramid + SQLAlchemy + PostgreSQL: idle connections

[Zsolt Ero]

Thanks. So the key point for me is to just use engine.dispose() (I
don't want to dig into gunicorn preload, this seems much cleaner).

About explicit manager: in practice it's as simple as
1. enabling it in models/__init__.py
2. using "with request.tm" everywhere instead of "with
transaction.manager", right?

> This is more likely to be an issue in web requests where your entire request 
> lifecycle is not actually protected by a transaction.

About your final sentence, I'm not sure I understand it. If I'm always
using request.dbsession, by definition I'm protected, to a request's
lifecycle am I not?

Zsolt

On 2 April 2018 at 21:47, Michael Merickel <mmeri...@gmail.com> wrote:
> The forking issue is likely because you're using a connection pool and so
> once a connection is opened at config-time, even though the session is
> properly closed the connection is just returned to the pool. The pool here
> is shared across the fork which is bad. The basic solution here is to add
> some sort of pre-fork hook that closes out every existing connection in the
> pool. I personally have not used forking wsgi servers in production and thus
> have not needed to deal with this problem but I'm pretty sure my advice hits
> at the core issue you're experiencing. Calling dispose() is one way to make
> sure that the underlying connection is closed which is why it's working for
> you. If I recall, gunicorn has some way to fork before config is called such
> that each subprocess is loaded independently which can help with this as
> well... If memory serves it was something like turning off the preloading
> feature.
>
> The explicit manager is about weeding out situations where you use a
> connection after it's closed [1]. Basically without it you could use
> something joined to the "tm" after the "with" block is done and you might
> not see an error. This is more likely to be an issue in web requests where
> your entire request lifecycle is not actually protected by a transaction.
>
> [1]
> https://docs.pylonsproject.org/projects/pyramid-tm/en/latest/#custom-transaction-managers
>
> On Mon, Apr 2, 2018 at 2:21 PM, Zsolt Ero <zsolt@gmail.com> wrote:
>>
>> Michael, I've updated the code to your recommendation.
>>
>> https://github.com/hyperknot/pyramid_connections_bug
>>
>> It still requires the explicit engine.dispose() line, otherwise it
>> does bring the connection to the forked processes.
>>
>> This is with and without the explicit manager. What does the explicit
>> manager protect us from?
>>
>> Zsolt
>>
>>
>> On 2 April 2018 at 20:11, Michael Merickel <mmeri...@gmail.com> wrote:
>> > Using the pyramid-cookiecutter-alchemy setup you can access config data
>> > at
>> > config time using a pattern like this:
>> >
>> > from transaction import TransactionManager
>> >
>> > from myapp.models import get_tm_session
>> >
>> > def main(global_config, **settings):
>> > config = Configurator(settings=settings)
>> > config.include('myapp.models')
>> >
>> > tm = TransactionManager(explicit=True)
>> > with tm:
>> > dbsession = get_tm_session(config.registry['dbsession_factory'],
>> > tm)
>> > ...  # do queries and stuff
>> >
>> > return config.make_wsgi_app()
>> >
>> > This will properly handle the lifecycle of a session for you, same as
>> > when
>> > serving a request. Be sure if you keep any objects around that you
>> > expunge
>> > them from the session and re-attach/merge them into any session where
>> > you
>> > use them.. ORM objects are only valid on the dbsession they were loaded
>> > from.
>> >
>> > - Michael
>> >
>> >
>> > On Mon, Apr 2, 2018 at 11:59 AM, Zsolt Ero <zsolt@gmail.com> wrote:
>> >>
>> >> OK, I agree with that. Still, storing config values in the database is
>> >> a common pattern for medium to large web apps, so it at least makes
>> >> sense to have some kind of resource about how to do it. I hope that if
>> >> nothing else at least this thread will be useful for someone in the
>> >> future.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "pylons-discuss" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> >> an
>> >> email to pylons-discuss+unsubscr...@googlegroups.com.

Re: [pylons-discuss] Pyramid + SQLAlchemy + PostgreSQL: idle connections

[Zsolt Ero]

Michael, I've updated the code to your recommendation.

https://github.com/hyperknot/pyramid_connections_bug

It still requires the explicit engine.dispose() line, otherwise it
does bring the connection to the forked processes.

This is with and without the explicit manager. What does the explicit
manager protect us from?

Zsolt


On 2 April 2018 at 20:11, Michael Merickel <mmeri...@gmail.com> wrote:
> Using the pyramid-cookiecutter-alchemy setup you can access config data at
> config time using a pattern like this:
>
> from transaction import TransactionManager
>
> from myapp.models import get_tm_session
>
> def main(global_config, **settings):
> config = Configurator(settings=settings)
> config.include('myapp.models')
>
> tm = TransactionManager(explicit=True)
> with tm:
> dbsession = get_tm_session(config.registry['dbsession_factory'], tm)
> ...  # do queries and stuff
>
> return config.make_wsgi_app()
>
> This will properly handle the lifecycle of a session for you, same as when
> serving a request. Be sure if you keep any objects around that you expunge
> them from the session and re-attach/merge them into any session where you
> use them.. ORM objects are only valid on the dbsession they were loaded
> from.
>
> - Michael
>
>
> On Mon, Apr 2, 2018 at 11:59 AM, Zsolt Ero <zsolt@gmail.com> wrote:
>>
>> OK, I agree with that. Still, storing config values in the database is
>> a common pattern for medium to large web apps, so it at least makes
>> sense to have some kind of resource about how to do it. I hope that if
>> nothing else at least this thread will be useful for someone in the
>> future.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "pylons-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to pylons-discuss+unsubscr...@googlegroups.com.
>> To post to this group, send email to pylons-discuss@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/pylons-discuss/CAKw-smBuH5hKGpNgs-6dYkNiAU%3DGGUUOF7aLHAzHe0AE25T7qg%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "pylons-discuss" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/pylons-discuss/_MJflNUcjdg/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> pylons-discuss+unsubscr...@googlegroups.com.
> To post to this group, send email to pylons-discuss@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/CAKdhhwGoPbgM7gDWBLxdanT%3DhcJQR8sT01E2xwWFxw1zxpCD1A%40mail.gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAKw-smBMmRqu2ZTPSp%3Djh%2BY0RriEwvxgdYQnwq--k4abeUkd-g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Pyramid + SQLAlchemy + PostgreSQL: idle connections

[Zsolt Ero]

OK, I agree with that. Still, storing config values in the database is
a common pattern for medium to large web apps, so it at least makes
sense to have some kind of resource about how to do it. I hope that if
nothing else at least this thread will be useful for someone in the future.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAKw-smBuH5hKGpNgs-6dYkNiAU%3DGGUUOF7aLHAzHe0AE25T7qg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Pyramid + SQLAlchemy + PostgreSQL: idle connections

[Zsolt Ero]

Wouldn't the second / "foolproof" way described here be a good choice
for the default Pyramid implementation? Just as a safety measure.

http://docs.sqlalchemy.org/en/latest/core/pooling.html#using-connection-pools-with-multiprocessing

On 2 April 2018 at 18:32, Jonathan Vanasco  wrote:
>
>
> On Monday, April 2, 2018 at 12:07:14 PM UTC-4, Bert JW Regeer wrote:
>>
>>
>> This is only required if you are not using pyramid_tm. If you are using
>> pyramid_tm which is what the sqlalchemy cookie cutter does, you do NOT need
>> to add this.
>
>
> Thanks, Bert.  I stand corrected.  pyarmid_tm has a finish() that eventually
> calls a session.close() in Zope.SqlAlchemy.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "pylons-discuss" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/pylons-discuss/_MJflNUcjdg/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> pylons-discuss+unsubscr...@googlegroups.com.
> To post to this group, send email to pylons-discuss@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/e1f81475-8b31-44ca-b0cf-af3448a3b131%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAKw-smAQ5QWad0eBA60MTDNS9EBxCrxV%3D4wDQT%3DxvnERP%2BUOhw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Pyramid + SQLAlchemy + PostgreSQL: idle connections

[Zsolt Ero]

I've been looking into it a bit more.

I _think_ when a dbsession is deleted / garbage collected, it is
closed. And transaction.manager / pyramid_tm is making each session
garbage collected after each request, isn't it? Thus
add_finished_callback is implemented in SQLAlchemy side, at least this
is my idea.

Where is the forking happening in a normal Pyramid app? After the main
function finishes / "return config.make_wsgi_app()"?

So if I understand right, the engine in a gunicorn / process worker
Pyramid app is global / shared between all processes, right? And at
the time of forking, there should be no dbsessions being active and no
connections in the connection pool, right?

So the super safe code for init time db lookup is this?

# dbsession, siteconfig
dbsession = config.registry['dbsession_factory']()
siteconfig = get_siteconfig(dbsession)
dbsession.close()
dbsession.connection().engine.dispose()
del dbsession

Zsolt


On 2 April 2018 at 17:54, Jonathan Vanasco <jvana...@gmail.com> wrote:
> On Sunday, April 1, 2018 at 8:18:39 PM UTC-4, Zsolt Ero wrote:
>>
>> Hi Jonathan,
>>
>> I'm not 100% sure I understand when are you talking about the
>> "standard" Pyramid way of the cookiecutter template's get_tm_session's
>> implementation and when are you talking about my special addition of
>> using the db to set up the app.
>
>
> I don't know why it was removed from the cookiecutter.
>
> It's been present in the cookbook and docs for years, see the cookbook:
>
> https://docs.pylonsproject.org/projects/pyramid-cookbook/en/latest/database/sqlalchemy.html
>
> I've opened a ticket on the cookiecutter to correct this.
>
>
>> I don't know how and when is a gunicorn process fork happening in the
>> standard Pyramid app, but I presume it's tested and proven by people
>> who understand the internals of SQLAlchemy connections. By this I mean
>> that I think about request.dbsession as something which is managed for
>> me. This results in 1 idle connection per gunicorn worker.
>
>
> It's not.  SqlAlchemy, Pyramid and Zope.SqlAlchemy (the transaction
> extension) do nothing to check or manage this.  Anything that is happening
> is pure coincidence and luck.
>
> If you connect to the database before the fork, you need to call
> `engine.dispose()`  If you don't connect to the database before the fork,
> you don't need to call `dispose`.
>
>
>> So far, my workaround which seems to work is the following:
>>
>> dbsession = config.registry['dbsession_factory']()
>> siteconfig = get_siteconfig(dbsession)
>> dbsession.close()
>> del dbsession
>>
>> (I'm deleting it to make sure that I cannot accidentally call it again).
>>
>> I tried calling dbsession.connection().engine.dispose() but it didn't
>> do anything.
>>
>> dbsession.connection().close() seems to behave the same as if I call
>> dbsession.close().
>
>
> I can try to spin up your test later.  `dispose` is really what you need to
> call in your situation.  `close` just 'ends' the session and returns the
> connection to the pool; `dispose` will close the actual db connection.
> (perhaps this is being done by the delete, i don't know).   it is important
> to close the connection and reset the pool before the fork, because you run
> the risk of different workers using the same connection.
>
>
>
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "pylons-discuss" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/pylons-discuss/_MJflNUcjdg/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> pylons-discuss+unsubscr...@googlegroups.com.
> To post to this group, send email to pylons-discuss@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/e5c2df72-f9b8-4de9-92e1-639e13e52391%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAKw-smBiwQsy6S%3D%2BptJ6SRd4ddBCzB4R-9a5e_2k03%3Du4hr_wA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Pyramid + SQLAlchemy + PostgreSQL: idle connections

[Zsolt Ero]

Hi Jonathan,

I'm not 100% sure I understand when are you talking about the
"standard" Pyramid way of the cookiecutter template's get_tm_session's
implementation and when are you talking about my special addition of
using the db to set up the app.

I don't know how and when is a gunicorn process fork happening in the
standard Pyramid app, but I presume it's tested and proven by people
who understand the internals of SQLAlchemy connections. By this I mean
that I think about request.dbsession as something which is managed for
me. This results in 1 idle connection per gunicorn worker.

So if I only want to solve my addition and not touch the standard
design of request.dbsession, what would I need to do?

So far, my workaround which seems to work is the following:

dbsession = config.registry['dbsession_factory']()
siteconfig = get_siteconfig(dbsession)
dbsession.close()
del dbsession

(I'm deleting it to make sure that I cannot accidentally call it again).

I tried calling dbsession.connection().engine.dispose() but it didn't
do anything.

dbsession.connection().close() seems to behave the same as if I call
dbsession.close().

Does this sound as a reasonable solution? It seems to fix my problem,
I just want to know if I'm doing anything wrong here.

Zsolt


On 2 April 2018 at 01:58, Jonathan Vanasco  wrote:
> Pyramid isn't opening the database connection, the SqlAlchemy pool is.
>
> On first glance, you're connecting to the database in your main(). This is
> guaranteed to create issues with SqlAlchemy's connection pool when you
> involve multiple workers/threads/processes.  In some situations the
> connection gets reused by multiple processes, in others it gets lost.
>
> The proper way to fix this is to call `engine.dispose()`
> [http://docs.sqlalchemy.org/en/latest/core/connections.html#engine-disposal]
> either immediately after you use it, or in a post-fork hook.
>
> I also don't see an add_finished_callback registered to call
> 'session.close() on every request (or at least the ones where you grab a
> dbsession), or the close() being called in any other way (like a tween).
>
> Between both of those traits, you're going to have a connection pool that
> constantly needs new connections (it has to wait for python's GC to clear
> out the connection), and might re-use existing connections unreliably.
>
> There could be other factors causing this too, but you should integrate a
> `dispose` after the app starts and add a utility to register a
> session.close() via add_finished_callback().  If that doesn't solve it,
> there are a lot of docs in SqlAlchemy on the connection pool specifics.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "pylons-discuss" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/pylons-discuss/_MJflNUcjdg/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> pylons-discuss+unsubscr...@googlegroups.com.
> To post to this group, send email to pylons-discuss@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/d21f0d7e-f20c-4760-9e03-9e057dde376c%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAKw-smAeZFzLUeAB00WWXd%3DJWi1c1eMkAXxkJtKYeKJBzOx%2BzQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Pyramid + SQLAlchemy + PostgreSQL: idle connections

[Zsolt Ero]

I created a minimal reproducible case, and in doing so I've figured
out what is happening, in a classical rubber duck debugging fashion.

https://github.com/hyperknot/pyramid_connections_bug

Nevertheless, it's still a very interesting problem, as I don't think
Pyramid/pyramid_tm should ever silently open a 2nd database
connection. DB connections are quite a scare resource on the server
side, postgresql guys always recommend me to have no more than
CPU_cores * 2 + 1, using pgBouncer if I need more, etc.

What was happening here is that

1. I needed a dbsession to set up config variables before a request is
available.

2. I was using it to setup the app. After the values have been read
from the db, I had to explicitly rollback() it, but close() also
worked. Which one is recommended in this case?

3. I used this dbsession (thinking it's the same as request.dbsession) in
config.add_request_method:
https://github.com/hyperknot/pyramid_connections_bug/blob/master/pyramid_connections_bug/__init__.py#L27
was using a different dbsession.

4. This one ended up silently creating a new connection, which was
stuck in "idle in transaction" forever.

pyramid_tm.explicit_manager didn't help, but actually I don't know
what is it doing.

What is the recommended way to use a dbsession to read config values
and then just get rid of it? I mean close the connection and delete
the variable, to make sure that I cannot use it like I did? Also, what
is the point of explicit_manager and how can I benefit from including
it?

Zsolt


On 1 April 2018 at 09:53, Zsolt Ero <zsolt@gmail.com> wrote:
> Hi,
>
> I'm running Pyramid + SQLAlchemy + PostgreSQL in a "classic" sync stack
> (like the cookiecutter one, pyramid_tm, etc.), simple gunicorn default
> (process) workers.
>
> I have a few problems (via pg_top or ps aux):
>
> 1. I'm getting about 2x the number of PostgreSQL connections as the number
> of workers. Is this normal?
> 2. About half of the connections are in "idle in transaction" state. They
> shouldn't be in transaction, right?
>
> So for --workers=1 I get one idle and one idle in transaction connection.
>
> 3. PGSQL 9.6+ supports idle_in_transaction_session_timeout
>
> https://www.postgresql.org/docs/9.6/static/runtime-config-client.html#GUC-IDLE-IN-TRANSACTION-SESSION-TIMEOUT
>
> Is this a good idea to use this to clean those idle in transaction
> connections?
>
> Should I use something else to close the normal, idle connections as well?
> Is this a good idea? Would Pyramid reconnect if needed?
>
> Zsolt
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "pylons-discuss" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/pylons-discuss/_MJflNUcjdg/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> pylons-discuss+unsubscr...@googlegroups.com.
> To post to this group, send email to pylons-discuss@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/8cc52db3-3f5a-43b9-b4e1-4b1b1a6149e8%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAKw-smD0mVBzU_UOXJ2F%2BpfoRAkBWVczxHHAe9fF448rdLQL1w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Pyramid + SQLAlchemy + PostgreSQL: idle connections

[Zsolt Ero]

Hi,

I'm running Pyramid + SQLAlchemy + PostgreSQL in a "classic" sync stack 
(like the cookiecutter one, pyramid_tm, etc.), simple gunicorn default 
(process) workers.

I have a few problems (via pg_top or ps aux):

1. I'm getting about 2x the number of PostgreSQL connections as the number 
of workers. Is this normal?
2. About half of the connections are in "idle in transaction" state. They 
shouldn't be in transaction, right?

So for --workers=1 I get one idle and one idle in transaction connection. 

3. PGSQL 9.6+ supports idle_in_transaction_session_timeout

https://www.postgresql.org/docs/9.6/static/runtime-config-client.html#GUC-IDLE-IN-TRANSACTION-SESSION-TIMEOUT

Is this a good idea to use this to clean those idle in transaction 
connections?

Should I use something else to close the normal, idle connections as well? 
Is this a good idea? Would Pyramid reconnect if needed?

Zsolt

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/8cc52db3-3f5a-43b9-b4e1-4b1b1a6149e8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] gevent/sqlalchemy/psycopg2/gunicorn setup?

[Zsolt Ero]

Ok, I suspected it but I'm really puzzled then why does Mike Bayer 
recommends it? Is concurrency under Python really such an impossible task?

With asyncio / twisted I pretty much cannot use any of my existing code and 
any of the common libraries. They all need their own tx/aio version. No 
more import antigravity. SQLAlchemy ORM is one thing which doesn't exist 
under those worlds.

Also, the second part is the programming style. If I were to write using 
that async style, I'd just write in Javascript, at least in the Node.js 
ecosystem every library is designed to work in an async environment. So you 
don't have to use ORM's with 47 Github stars 
(https://github.com/fantix/gino) but one with 10.000 
(https://github.com/sequelize/sequelize).

So how do you solve slow HTTP endpoints with Pyramid? Just start a lot of 
workers in a server with lots of RAM?







On Thursday, 31 August 2017 16:34:36 UTC+2, Mikko Ohtamaa wrote:
>
> Hi,
>
>
> On 31 August 2017 at 17:19, Zsolt Ero <zsol...@gmail.com > 
> wrote:
>
>> After reading zzzeek's great blog post: 
>> http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ 
>> and SO answer: https://stackoverflow.com/a/16503103/518169 I would like 
>> to use gevent / sqlalchemy / psycopg2 / gunicorn in a new application.
>>
>
> I had a project using SQLAlchemy, Pyramid and gevent 
> (gevent.monkey.patch_all()).
>
> Before engaging to this interesting experience, I suggest you prebook a 
> bed in an asylum. That is the level of problems you need to debug with 
> Python interpreter, stdlib, web server threading, etc. All of those had 
> subtle but hard to debug threading issues that took days and days to debug 
> when you no longer can trust that lower levels of your stack (web server, 
> database connections, etc.) correctly behaving under asyncio. It's 
> especially fun if the problems only appear under a production load.
>
> It was very happy moment when I could finally pip uninstall gevent and 
> move back to well proven threading model.
>
> -- 
> Mikko Ohtamaa
> http://opensourcehacker.com
> http://twitter.com/moo9000
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/18b21377-c974-4fa5-991d-814c7b66ae4d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] gevent/sqlalchemy/psycopg2/gunicorn setup?

[Zsolt Ero]

After reading zzzeek's great blog 
post: http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ 
and SO answer: https://stackoverflow.com/a/16503103/518169 I would like to 
use gevent / sqlalchemy / psycopg2 / gunicorn in a new application.

All I've found for Pyramid is the old TicTacToe project, as well as this 5 
year old Flask project, 
https://github.com/kljensen/async-flask-sqlalchemy-example, but I'm afraid 
Pyramid's transaction handling might need to be taken care of.

Can you point me to any project or snippet which shows how to set it up? I 
am looking for the required changes I need to make to 
pyramid-cookiecutter-alchemy. 

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/b0d56c8c-693b-4951-9bb9-11397a289e80%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Re: pyramid and react working together?

[Zsolt Ero]

Yes, it works perfectly. The only interesting bit is integrating 
create-react-app's watch mode with Pyramid, or any classic backend for that 
matter, if you use server side routing.

Not very complicated, but to set up watch mode, you need to:
1. Eject the app
2. Make it look for a different port compared Pyramid. For example 3000 to 
Webpack and 6543 to Pyramid or your favourite ports. By default it's hard 
coded to use the window.location's port, so change that to 3000 manually.
3. I use the following Jinja2 snippet to eject the live script in devel 
mode and just use the built js file in production.


  {% if request.webpack %}
http://localhost:3000/static/js/bundle.js";>
  {% else %}

  {% endif %}




-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/d5f731b1-24e2-492a-8ef9-653cc8783290%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] uWSGI logging issues with Pyramid

[Zsolt Ero]

I was thinking of that as well, but it implies using the same ini file,
which I'd like to keep separate.


On 2017. Jan 20., Fri at 17:56, Jonathan Vanasco  wrote:

> I think the command you want in uwsgi is `ini-paste-logged` instead of
> `paste`
>
>
> http://uwsgi-docs.readthedocs.io/en/latest/Options.html?highlight=paste-logger#ini-paste-logged
>
>
>
>
>
>
>
>
>
>
> --
>
>
> You received this message because you are subscribed to a topic in the
> Google Groups "pylons-discuss" group.
>
>
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/pylons-discuss/CO5z2MD3iA0/unsubscribe.
>
>
> To unsubscribe from this group and all its topics, send an email to
> pylons-discuss+unsubscr...@googlegroups.com.
>
>
> To post to this group, send email to pylons-discuss@googlegroups.com.
>
>
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/d3e7d2b8-84a1-436c-aa5f-6b245afda5f7%40googlegroups.com
> 
> .
>
>
> For more options, visit https://groups.google.com/d/optout.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAKw-smDuUigFE37DvXCD%2BM0GPWt6MaFqKB4Wx7QTrDWYC-uYTA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] uWSGI logging issues with Pyramid

[Zsolt Ero]

Thanks! So I added "paste-logger = true" and then I got a warning saying:

ImportError: No module named script.util.logging_config

Which I found out on SO can be fixed by installing pastescript. So the
combination of paste-logger = true + pastescript in requirements seems
to have fixed it!

On 19 January 2017 at 18:42, Bert JW Regeer <xiste...@0x58.com> wrote:
> You’ll need something like:
>
> paste-logger = %p
>
> As well for uWSGI to automatically pick up your logger configuration from
> .ini file:
> http://uwsgi-docs.readthedocs.io/en/latest/Options.html?highlight=paste-logger#paste-logger
>
> uWSGI doesn’t automatically pick up your fileConfig logging otherwise.
>
> (Basically unless you explicitly call fileConfig for your .ini right now
> uWSGI doesn’t do anything with your logging handlers/information)
>
>
> On Jan 19, 2017, at 10:17, Zsolt Ero <zsolt@gmail.com> wrote:
>
> I'm just simply calling uwsgi uwsgi.ini and and it has the logto line in
> there.
>
> On 19 January 2017 at 18:16, Bert JW Regeer <xiste...@0x58.com> wrote:
>
> Are you setting up logging explicitly?
>
> uWSGI doesn’t automatically set up Python logging when you provide it a
> paster ini file, how are you starting uWSGI?
>
> Bert
>
> On Jan 19, 2017, at 08:08, Zsolt Ero <zsolt@gmail.com> wrote:
>
> I believe this is uWSGI specific, and might even be a bug in uWSGI, but
> since many people are using uWSGI with Pyramid here, I'd be interested to
> know how did you solve it.
>
>
> I've migrated a project from Gunicorn to uWSGI. My problem is that my
> previously set-up file-based logging is all bypassed / redirected now with
> custom format discarded.
>
>
> Previously it wrote to pyramid.log (and any other specific logs I had). Now,
> after converting to uWSGI:
>
> pyramid.log file is created but empty
> every log is redirected to uwsgi.log, with all custom formatters discarded.
>
> How can I make uWSGI not overwrite the whole, carefully set-up Python
> logging system, and only output it's own log to uwsgi.log?
>
>
> my uwsgi.ini:
>
> [uwsgi]
> paste = config:/home/app/web/app_web/production.ini
> http-socket = :5000
> uid = app
> gid = app
>
> master = true
> processes = 16
> enable-threads = true
> harakiri = 60
> harakiri-verbose = true
> single-interpreter = true
>
> die-on-term = true
> vacuum = true
>
> disable-logging = true
> logto2 = /shared/logs/CURRENT/app/uwsgi.log
>
> stats = /home/app/uwsgi_stats.socket
>
>
> My logging setup is as follows:
>
> [loggers]
> keys = root, app_web, sqlalchemy
>
> [handlers]
> keys = filelog_pyramid
>
> [formatters]
> keys = generic
>
> [logger_root]
> level = WARN
> handlers = filelog_pyramid
>
> [logger_app_web]
> level = WARN
> handlers =
> qualname = app_web
>
> [logger_sqlalchemy]
> level = WARN
> handlers =
> qualname = sqlalchemy.engine
>
> [handler_filelog_pyramid]
> class = FileHandler
> args = ('/shared/logs/CURRENT/app/pyramid.log','a')
> level = NOTSET
> formatter = generic
>
> [formatter_generic]
> format = %(asctime)s %(levelname)-5.5s [%(name)s] %(message)s
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pylons-discuss+unsubscr...@googlegroups.com.
> To post to this group, send email to pylons-discuss@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/6c07825a-34f5-4542-b4c2-bf7759eda509%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "pylons-discuss" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/pylons-discuss/CO5z2MD3iA0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> pylons-discuss+unsubscr...@googlegroups.com.
> To post to this group, send email to pylons-discuss@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/8DBCF3A1-E4AF-44A2-AD98-0E7A5381DC16%400x58.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pylons-discuss+unsubscr...@googlegroups.com.
> To post to this group, send email to pylons-discuss@g

Re: [pylons-discuss] uWSGI logging issues with Pyramid

[Zsolt Ero]

I'm just simply calling uwsgi uwsgi.ini and and it has the logto line in there.

On 19 January 2017 at 18:16, Bert JW Regeer <xiste...@0x58.com> wrote:
> Are you setting up logging explicitly?
>
> uWSGI doesn’t automatically set up Python logging when you provide it a
> paster ini file, how are you starting uWSGI?
>
> Bert
>
> On Jan 19, 2017, at 08:08, Zsolt Ero <zsolt@gmail.com> wrote:
>
> I believe this is uWSGI specific, and might even be a bug in uWSGI, but
> since many people are using uWSGI with Pyramid here, I'd be interested to
> know how did you solve it.
>
>
> I've migrated a project from Gunicorn to uWSGI. My problem is that my
> previously set-up file-based logging is all bypassed / redirected now with
> custom format discarded.
>
>
> Previously it wrote to pyramid.log (and any other specific logs I had). Now,
> after converting to uWSGI:
>
> pyramid.log file is created but empty
> every log is redirected to uwsgi.log, with all custom formatters discarded.
>
> How can I make uWSGI not overwrite the whole, carefully set-up Python
> logging system, and only output it's own log to uwsgi.log?
>
>
> my uwsgi.ini:
>
> [uwsgi]
> paste = config:/home/app/web/app_web/production.ini
> http-socket = :5000
> uid = app
> gid = app
>
> master = true
> processes = 16
> enable-threads = true
> harakiri = 60
> harakiri-verbose = true
> single-interpreter = true
>
> die-on-term = true
> vacuum = true
>
> disable-logging = true
> logto2 = /shared/logs/CURRENT/app/uwsgi.log
>
> stats = /home/app/uwsgi_stats.socket
>
>
> My logging setup is as follows:
>
> [loggers]
> keys = root, app_web, sqlalchemy
>
> [handlers]
> keys = filelog_pyramid
>
> [formatters]
> keys = generic
>
> [logger_root]
> level = WARN
> handlers = filelog_pyramid
>
> [logger_app_web]
> level = WARN
> handlers =
> qualname = app_web
>
> [logger_sqlalchemy]
> level = WARN
> handlers =
> qualname = sqlalchemy.engine
>
> [handler_filelog_pyramid]
> class = FileHandler
> args = ('/shared/logs/CURRENT/app/pyramid.log','a')
> level = NOTSET
> formatter = generic
>
> [formatter_generic]
> format = %(asctime)s %(levelname)-5.5s [%(name)s] %(message)s
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pylons-discuss+unsubscr...@googlegroups.com.
> To post to this group, send email to pylons-discuss@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/6c07825a-34f5-4542-b4c2-bf7759eda509%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "pylons-discuss" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/pylons-discuss/CO5z2MD3iA0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> pylons-discuss+unsubscr...@googlegroups.com.
> To post to this group, send email to pylons-discuss@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/8DBCF3A1-E4AF-44A2-AD98-0E7A5381DC16%400x58.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAKw-smCoGqReO7N7mBcbZH4yC2bft0LtW%3DPZdTd9rsWj7O-M4Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] uWSGI logging issues with Pyramid

[Zsolt Ero]

I believe this is uWSGI specific, and might even be a bug in uWSGI, but 
since many people are using uWSGI with Pyramid here, I'd be interested to 
know how did you solve it.


I've migrated a project from Gunicorn to uWSGI. My problem is that my 
previously set-up file-based logging is all bypassed / redirected now with 
custom format discarded.


Previously it wrote to pyramid.log (and any other specific logs I had). 
Now, after converting to uWSGI:

   - pyramid.log file is created but empty
   - every log is redirected to uwsgi.log, with all custom formatters 
   discarded.

How can I make uWSGI not overwrite the whole, carefully set-up Python 
logging system, and only output it's own log to uwsgi.log?


my uwsgi.ini:

[uwsgi]
paste = config:/home/app/web/app_web/production.ini
http-socket = :5000
uid = app
gid = app

master = true
processes = 16
enable-threads = true
harakiri = 60
harakiri-verbose = true
single-interpreter = true

die-on-term = true
vacuum = true

disable-logging = true
logto2 = /shared/logs/CURRENT/app/uwsgi.log

stats = /home/app/uwsgi_stats.socket


My logging setup is as follows:

[loggers]
keys = root, app_web, sqlalchemy

[handlers]
keys = filelog_pyramid

[formatters]
keys = generic

[logger_root]
level = WARN
handlers = filelog_pyramid

[logger_app_web]
level = WARN
handlers =
qualname = app_web

[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine

[handler_filelog_pyramid]
class = FileHandler
args = ('/shared/logs/CURRENT/app/pyramid.log','a')
level = NOTSET
formatter = generic

[formatter_generic]
format = %(asctime)s %(levelname)-5.5s [%(name)s] %(message)s

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/6c07825a-34f5-4542-b4c2-bf7759eda509%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Checking if a route is allowed

[Zsolt Ero]

Thanks

On Fri, 11 Nov 2016 at 06:25 Nejc Zupan

<
mailto:Nejc Zupan <nejc.zu...@gmail.com>
> wrote:

a, pre, code, a:link, body { word-wrap: break-word !important; }

Just a bit of logistics help, should you need it:

* an affordable and reliable way of getting to Ljubljana from Budapest is the 
van sharing service
http://GoOpti.com
. Been using them for years. 

* there is a discussion on the sprint’s mailinglist about sharing an AirBNB 
place: 
https://groups.google.com/forum/#!topic/dragonsprint/4Dew-8y6qMk

Cheers,

z.

On 10 Nov 2016, at 23:53, Zsolt Ero <
mailto:zsolt@gmail.com
> wrote:

Thanks a lot! I'm thinking about it, since I'm quite close (in Budapest).

On Thu, 10 Nov 2016 at 23:48 Mikko Ohtamaa

 

<> wrote:

This might or might not work, but looks complicated enough for me not to know 
if there is a possible bug in it, that I'll just stick with has_permission and 
duplicated values in templates.

Fair. There is a Pyramid sprint coming in December:

 

https://dragonsprint.com/
 

 

If you want to get this resolved and participate locally/remotely we might find 
tutors help to build route

_execution_permitted(request, route_name)

Thanks!

Mikko

 

--

 

You received this message because you are subscribed to a topic in the Google 
Groups "pylons-discuss" group.

To unsubscribe from this topic, visit

 

https://groups.google.com/d/topic/pylons-discuss/4FlvcEXKon4/unsubscribe
.

To unsubscribe from this group and all its topics, send an email to

 

mailto:pylons-discuss+unsubscr...@googlegroups.com
.

To post to this group, send email to

 

mailto:pylons-discuss@googlegroups.com
.

To view this discussion on the web visit

 

https://groups.google.com/d/msgid/pylons-discuss/CAK8RCUuq%2BQXczJWm4eBjEDphym5x6g%2BR7s3v3A-Uc-uPjyA44A%40mail.gmail.com?utm_medium=email_source=footer
.

For more options, visit

 

https://groups.google.com/d/optout
.

--

 

You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email 
to

 

mailto:pylons-discuss+unsubscr...@googlegroups.com
.

To post to this group, send email to

 

mailto:pylons-discuss@googlegroups.com
.

To view this discussion on the web visit

 

https://groups.google.com/d/msgid/pylons-discuss/5824f9e68e02ab0d2412%40polymail.io?utm_medium=email_source=footer
.

For more options, visit

 

https://groups.google.com/d/optout
.

--

You received this message because you are subscribed to a topic in the Google 
Groups "pylons-discuss" group.

To unsubscribe from this topic, visit
https://groups.google.com/d/topic/pylons-discuss/4FlvcEXKon4/unsubscribe
.

To unsubscribe from this group and all its topics, send an email to
mailto:pylons-discuss+unsubscr...@googlegroups.com
.

To post to this group, send email to
mailto:pylons-discuss@googlegroups.com
.

To view this discussion on the web visit
https://groups.google.com/d/msgid/pylons-discuss/33BD1C53-1461-409E-B260-E00B31FCB168%40gmail.com?utm_medium=email_source=footer
.

For more options, visit
https://groups.google.com/d/optout
.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/582603e4109d0aa6a2a4%40polymail.io.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Checking if a route is allowed

[Zsolt Ero]

Thanks a lot! I'm thinking about it, since I'm quite close (in Budapest).

On Thu, 10 Nov 2016 at 23:48 Mikko Ohtamaa

<
mailto:Mikko Ohtamaa 
> wrote:

a, pre, code, a:link, body { word-wrap: break-word !important; }

This might or might not work, but looks complicated enough for me not to know 
if there is a possible bug in it, that I'll just stick with has_permission and 
duplicated values in templates.

Fair. There is a Pyramid sprint coming in December:
https://dragonsprint.com/
  If you want to get this resolved and participate locally/remotely we might 
find tutors help to build route

_execution_permitted(request, route_name)

Thanks!

Mikko

 

--

You received this message because you are subscribed to a topic in the Google 
Groups "pylons-discuss" group.

To unsubscribe from this topic, visit
https://groups.google.com/d/topic/pylons-discuss/4FlvcEXKon4/unsubscribe
.

To unsubscribe from this group and all its topics, send an email to
mailto:pylons-discuss+unsubscr...@googlegroups.com
.

To post to this group, send email to
mailto:pylons-discuss@googlegroups.com
.

To view this discussion on the web visit
https://groups.google.com/d/msgid/pylons-discuss/CAK8RCUuq%2BQXczJWm4eBjEDphym5x6g%2BR7s3v3A-Uc-uPjyA44A%40mail.gmail.com?utm_medium=email_source=footer
.

For more options, visit
https://groups.google.com/d/optout
.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/5824f9e68e02ab0d2412%40polymail.io.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] Checking if a route is allowed

[Zsolt Ero]

Thanks. So this is how my site is setup:

I have a RootFactory:
class RootFactory(object):
__acl__ = [
(Allow, Authenticated, 'user'),
(Allow, 'g:admin', 'admin'),
(Allow, 'g:superadmin', 'ALL_PERMISSIONS'),
]


def __init__(self, request):
pass


used in config:


config = Configurator(
settings=settings,
root_factory=RootFactory,
authentication_policy=authn_policy,
authorization_policy=authz_policy,
session_factory=session_factory)

And my views are defined like this:
@view_config(route_name='admin_db_list', renderer='admin/db_list.jinja2', 
permission='superadmin')
def db_list(request): ...


So in this situation, my context is request.root (or request.context), is 
this right?

If I try view_execution_permitted(request.root, request, name='admin_db_list'), 
I get an "TypeError: No registered view satisfies the constraints."

Do I understand correctly that the name should be a @view_config name _and_ 
this means using traversal, so I should just forget about using it?

=> So in conclusion, I can only use request.has_permission and duplicate 
the permission values in template as well?













On Thursday, 10 November 2016 22:50:37 UTC+1, Mikko Ohtamaa wrote:
>
> And to elaborate the following:
>
> I simply check for the permission I know the target has using 
> request.has_permission():
>
>
> https://websauna.org/docs/narrative/user/permissions.html?highlight=permissions#checking-permissions-in-templates
>
> - Define a Root object
>
> - In this root you have a dynamic __acl__() property that gives logged in 
> users permissions based on their user id or group id
>
> - In your view you have @view_config(permission="my_permission")
>
> Example of setting a custom root:
>
>
> https://websauna.org/docs/_modules/websauna/system.html#Initializer.configure_root
>
> Some examples of dynamic __acl__
>
>
> https://github.com/websauna/websauna.blog/blob/master/websauna/blog/views.py#L45
>
>
> https://websauna.org/docs/narrative/crud/standalone.html?highlight=contract#creating-crud-resources
>
> -M
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/b5dd965d-4d4b-48a1-b6c5-fe60eae13c57%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] Checking if a route is allowed

[Zsolt Ero]

Hi, I'd like to do the simplest possible thing in Pyramid / URL Dispatch, 
yet it seems almost impossibly hard to do it.

So it's the super common case of having a menu, and I'd only like to insert 
menu items which are available for the current user.

I'm looking for a function to fit in this usage:

{% macro nav_item(request, route_name, text) -%}
  {% if request.view_execution_permitted(route_name) %}

  {{ text }}

  {% endif %}
{%- endmacro %}


My problems are the following:

1. view_execution_permitted doesn't work like this, unlike other security 
functions, for example request.has_permission(). Why?

2. Going the hard way and making a custom wrapper around 
view_execution_permitted, 
and adding it to request via add_request_method, I'm still stuck in how to 
use view_execution_permitted, for the following reason:

2.1. It needs a context. What is a context? I never had to use any context 
in URL Dispatch with SessionAuthenticationPolicy, and it really isn't 
explained on the website. I tried Googling view_execution_permitted and 
grepping for code in Github, but I couldn't find anything, except a Github 
issue ticket saying: "view_execution_permitted does not work with URL 
dispatch", which didn't help me.
https://github.com/Pylons/pyramid/issues/673

2.2. It needs a name. Is this route_name? I hope so? 

Maybe I don't even need view_execution_permitted. I just want a simple 
request.is_allowed_route(route_name). How can I do it?

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/a5285237-b1a5-44e5-bd5e-3fd0e4b11c44%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] How to fix request.client_addr

[Zsolt Ero]

Thanks, that's a great howto, unfortunately, I'd need to compile nginx
from sources to get that module.



On 29 April 2016 at 03:03, Jonathan Vanasco  wrote:
> You can do this in nginx.
>
> Cloudflare publishes a list of trusted ips; the nginx set_real_ip module
> will only apply the real-ip header to those matching ips.
>
> https://support.cloudflare.com/hc/en-us/articles/200170706-How-do-I-restore-original-visitor-IP-with-Nginx-
>
> you could do it in python, but it would be easier to just enable the nginx
> module
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "pylons-discuss" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/pylons-discuss/3dh4lS1PVog/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> pylons-discuss+unsubscr...@googlegroups.com.
> To post to this group, send email to pylons-discuss@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/3976a70a-3e60-41f1-a94f-3afb6048637f%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAKw-smBdMnVJfKJdyKdeBM%2BiYy%2B8GOJk6Dv4_YqyWYqesGDD3Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [pylons-discuss] How to fix request.client_addr

[Zsolt Ero]

Hi, can you guide me how is it best to do it? I'm using nginx -> gunicorn 
-> pyramid, where should I be modifying the WSGI environment?



On Thursday, 28 April 2016 17:57:13 UTC+2, Bert JW Regeer wrote:
>
> Set up the WSGI environment appropriately, replacing the X-Forwarded-For 
> with the value of Cf-Connecting-Ip. 
>
> Bert 
>
> > On Apr 23, 2016, at 17:09, Zsolt Ero <zsol...@gmail.com > 
> wrote: 
> > 
> > I got a weird bug, in which request.client_addr was reported as 
> 192.168.76.75:52411, which broke a function which expected it to be a 
> standard IP address. In the documentation I've read that this could be 
> anything, so I guess this isn't a surprise. 
> > 
> > I am using CloudFlare -> nginx -> gunicorn -> pyramid setup. These were 
> the headers of that request: 
> > 
> > X-Forwarded-For 192.168.76.75:52411,67.133.63.210, 108.162.246.204 
> > Cf-Connecting-Ip 67.133.63.210 
> > 
> > My question is that since I'm using CloudFlare and I know that 
> Cf-Connecting-Ip is a reliable, trustable source of IP address, unlike 
> X-Forwarded-For. How should I modify my request.client_addr to use this 
> value whenever it is present? 
> > 
> > Zsolt 
> > 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "pylons-discuss" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to pylons-discus...@googlegroups.com . 
> > To post to this group, send email to pylons-...@googlegroups.com 
> . 
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/pylons-discuss/e643ca5d-763e-45bd-b077-d09b448b2fb8%40googlegroups.com.
>  
>
> > For more options, visit https://groups.google.com/d/optout. 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/bfc47ce5-91bb-434a-8056-c018623d6200%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pylons-discuss] How to fix request.client_addr

[Zsolt Ero]

I got a weird bug, in which request.client_addr was reported as 
192.168.76.75:52411, which broke a function which expected it to be a 
standard IP address. In the documentation I've read that this could be 
anything, so I guess this isn't a surprise.

I am using CloudFlare -> nginx -> gunicorn -> pyramid setup. These were the 
headers of that request:

X-Forwarded-For 192.168.76.75:52411,67.133.63.210, 108.162.246.204
Cf-Connecting-Ip 67.133.63.210

My question is that since I'm using CloudFlare and I know that 
Cf-Connecting-Ip is a reliable, trustable source of IP address, unlike 
X-Forwarded-For. How should I modify my request.client_addr to use this 
value whenever it is present?

Zsolt

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/e643ca5d-763e-45bd-b077-d09b448b2fb8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.