Re: [google-appengine] Re: The Performance of the SDK development server has become unbearable

[Andrius A]

I would like to add to that as well: developing backends in local
environment is almost impossible because it is really slow running and
there is no documentation how to do that using SDK?

On 6 December 2011 20:19, PK  wrote:

> Very happy to report that "1.6.1 prerelease" has addressed the performance
> regressions reported here.  Many thanks to everybody who starred the issues
> and to Robert and the rest of the Google team who followed up.
>
> PK
> http://www.gae123.com
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/zbXjoh77m9YJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: The Performance of the SDK development server has become unbearable

[PK]

Very happy to report that "1.6.1 prerelease" has addressed the performance 
regressions reported here.  Many thanks to everybody who starred the issues 
and to Robert and the rest of the Google team who followed up.

PK
http://www.gae123.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/zbXjoh77m9YJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: The Performance of the SDK development server has become unbearable

[Vivek Puri]

I got the proxy working on my end and it has definitely helped. Local
pages are loading in around 20 seconds, which were earlier taking 60+
seconds.



On Nov 30, 2:20 pm, "Ikai Lan (Google)"  wrote:
> Vivek,
>
> I get you. There are definitely parts of the SDK and dev appserver that we
> can improve, and this is one of them.
>
> --
> Ikai Lan
> Developer Programs Engineer, Google App Engine
> plus.ikailan.com | twitter.com/ikai
>
>
>
>
>
>
>
> On Wed, Nov 30, 2011 at 10:49 AM, Vivek Puri  wrote:
> > Ikai,
>
> > Is it possible to bundle this into the sdk itself. All this is way too
> > much custom config to deal with. I already loose my email sending
> > capabilities from localhost(needed for testing purposes) every time i
> > upgrade to new version of sdk.
>
> > On Nov 29, 1:56 pm, "Ikai Lan (Google)"  wrote:
> > > Desmond, thanks for that! I think nginx is a bit lighter weight and
> > easier
> > > to configure:
>
> > >https://help.ubuntu.com/community/Nginx/ReverseProxy
>
> > > nginx is a bit easier to configure than Apache, especially if you're just
> > > using it for development. Here's a sample reverse proxy config:
>
> > > server {        listen   80;        server_name  foo.bar.no
> > > foo.bar.yes foo.bar.ok;        access_log  /var/log/nginx/access.log;
> > >       location / {                proxy_pass      http://172.27.0.2/;
> > >               include         /etc/nginx/proxy.conf;        }}
>
> > > --
> > > Ikai Lan
> > > Developer Programs Engineer, Google App Engine
> > > plus.ikailan.com | twitter.com/ikai
>
> > > On Fri, Nov 25, 2011 at 5:36 PM, Desmond Brand  > >wrote:
>
> > > > I wrote more detailed instructions for how to set this up:
>
> > > >http://desmondbrand.com/blog/2011/11/15/speed-up-your-app-engine-dev-.
> > ..
>
> > > > On 23 November 2011 18:43, Calvin  wrote:
>
> > > >> On Wednesday, November 23, 2011 4:38:48 PM UTC-8, PK wrote:
>
> > > >>> 3. Finally, this is not new and was already worked around in my
> > > >>> environment, but if you have a realistic application you need to
> > proxy the
> > > >>> serving of static content (images/javascript/css) if you want to get
> > any
> > > >>> work done. I use apache for that and a mini framework I have built. I
> > > >>> describe the issue more here<
> >http://www.gae123.com/articles/gaet/fir-tim-exp.html>.
> > > >>> It might also help in the long term, if you star issues 343<
> >http://code.google.com/p/googleappengine/issues/detail?id=343>,
> > > >>> 6005 and 4720.
>
> > > >>> Do you have more details on how you do this?  Do you just do a
> > redirect
> > > >> to the Apache server when a static asset is requested from
> > dev_appserver?
>
> > > >> --
> > > >> You received this message because you are subscribed to the Google
> > Groups
> > > >> "Google App Engine" group.
> > > >> To view this discussion on the web visit
> > > >>https://groups.google.com/d/msg/google-appengine/-/M5yP6ScsCC4J.
>
> > > >> To post to this group, send email to
> > google-appengine@googlegroups.com.
> > > >> To unsubscribe from this group, send email to
> > > >> google-appengine+unsubscr...@googlegroups.com.
> > > >> For more options, visit this group at
> > > >>http://groups.google.com/group/google-appengine?hl=en.
>
> > > >  --
> > > > You received this message because you are subscribed to the Google
> > Groups
> > > > "Google App Engine" group.
> > > > To post to this group, send email to google-appengine@googlegroups.com
> > .
> > > > To unsubscribe from this group, send email to
> > > > google-appengine+unsubscr...@googlegroups.com.
> > > > For more options, visit this group at
> > > >http://groups.google.com/group/google-appengine?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine+unsubscr...@googlegroups.com.
> > For more options, visit this group at
> >http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: The Performance of the SDK development server has become unbearable

[Robert Schuppenies]

On Wed, Nov 30, 2011 at 11:33 AM, PK  wrote:

> Hi Ikai,
>
> Thanks for acknowledging this.
>
> Good software gets better but great software also does nor regress first.
> As painful as is to configure static resource serving, it is by now a known
> evil and I have automated it. Hunting down regressions is unexpected and
> totally unproductive. Getting back to where this thread started I would
> like to hear that you will also/first add some benchmarks in your internal
> SDK testing process so that huge performance regressions like the ones
> described here will be less likely to happen again.
>

You're right PK and we will look into this.

cheers,
robert

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: The Performance of the SDK development server has become unbearable

[PK]

Hi Ikai,

Thanks for acknowledging this.

Good software gets better but great software also does nor regress first. As 
painful as is to configure static resource serving, it is by now a known evil 
and I have automated it. Hunting down regressions is unexpected and totally 
unproductive. Getting back to where this thread started I would like to hear 
that you will also/first add some benchmarks in your internal SDK testing 
process so that huge performance regressions like the ones described here will 
be less likely to happen again.

Thanks for your attention,
PK

On Nov 30, 2011, at 11:20 AM, Ikai Lan (Google) wrote:

> Vivek,
> 
> I get you. There are definitely parts of the SDK and dev appserver that we 
> can improve, and this is one of them. 
> 
> --
> Ikai Lan 
> Developer Programs Engineer, Google App Engine
> plus.ikailan.com | twitter.com/ikai

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: The Performance of the SDK development server has become unbearable

[Ikai Lan (Google)]

Vivek,

I get you. There are definitely parts of the SDK and dev appserver that we
can improve, and this is one of them.

--
Ikai Lan
Developer Programs Engineer, Google App Engine
plus.ikailan.com | twitter.com/ikai



On Wed, Nov 30, 2011 at 10:49 AM, Vivek Puri  wrote:

> Ikai,
>
> Is it possible to bundle this into the sdk itself. All this is way too
> much custom config to deal with. I already loose my email sending
> capabilities from localhost(needed for testing purposes) every time i
> upgrade to new version of sdk.
>
>
> On Nov 29, 1:56 pm, "Ikai Lan (Google)"  wrote:
> > Desmond, thanks for that! I think nginx is a bit lighter weight and
> easier
> > to configure:
> >
> > https://help.ubuntu.com/community/Nginx/ReverseProxy
> >
> > nginx is a bit easier to configure than Apache, especially if you're just
> > using it for development. Here's a sample reverse proxy config:
> >
> > server {listen   80;server_name  foo.bar.no
> > foo.bar.yes foo.bar.ok;access_log  /var/log/nginx/access.log;
> >   location / {proxy_pass  http://172.27.0.2/;
> >   include /etc/nginx/proxy.conf;}}
> >
> > --
> > Ikai Lan
> > Developer Programs Engineer, Google App Engine
> > plus.ikailan.com | twitter.com/ikai
> >
> > On Fri, Nov 25, 2011 at 5:36 PM, Desmond Brand  >wrote:
> >
> >
> >
> >
> >
> >
> >
> > > I wrote more detailed instructions for how to set this up:
> >
> > >http://desmondbrand.com/blog/2011/11/15/speed-up-your-app-engine-dev-.
> ..
> >
> > > On 23 November 2011 18:43, Calvin  wrote:
> >
> > >> On Wednesday, November 23, 2011 4:38:48 PM UTC-8, PK wrote:
> >
> > >>> 3. Finally, this is not new and was already worked around in my
> > >>> environment, but if you have a realistic application you need to
> proxy the
> > >>> serving of static content (images/javascript/css) if you want to get
> any
> > >>> work done. I use apache for that and a mini framework I have built. I
> > >>> describe the issue more here<
> http://www.gae123.com/articles/gaet/fir-tim-exp.html>.
> > >>> It might also help in the long term, if you star issues 343<
> http://code.google.com/p/googleappengine/issues/detail?id=343>,
> > >>> 6005 and 4720.
> >
> > >>> Do you have more details on how you do this?  Do you just do a
> redirect
> > >> to the Apache server when a static asset is requested from
> dev_appserver?
> >
> > >> --
> > >> You received this message because you are subscribed to the Google
> Groups
> > >> "Google App Engine" group.
> > >> To view this discussion on the web visit
> > >>https://groups.google.com/d/msg/google-appengine/-/M5yP6ScsCC4J.
> >
> > >> To post to this group, send email to
> google-appengine@googlegroups.com.
> > >> To unsubscribe from this group, send email to
> > >> google-appengine+unsubscr...@googlegroups.com.
> > >> For more options, visit this group at
> > >>http://groups.google.com/group/google-appengine?hl=en.
> >
> > >  --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Google App Engine" group.
> > > To post to this group, send email to google-appengine@googlegroups.com
> .
> > > To unsubscribe from this group, send email to
> > > google-appengine+unsubscr...@googlegroups.com.
> > > For more options, visit this group at
> > >http://groups.google.com/group/google-appengine?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: The Performance of the SDK development server has become unbearable

[Vivek Puri]

Ikai,

Is it possible to bundle this into the sdk itself. All this is way too
much custom config to deal with. I already loose my email sending
capabilities from localhost(needed for testing purposes) every time i
upgrade to new version of sdk.


On Nov 29, 1:56 pm, "Ikai Lan (Google)"  wrote:
> Desmond, thanks for that! I think nginx is a bit lighter weight and easier
> to configure:
>
> https://help.ubuntu.com/community/Nginx/ReverseProxy
>
> nginx is a bit easier to configure than Apache, especially if you're just
> using it for development. Here's a sample reverse proxy config:
>
> server {        listen   80;        server_name  foo.bar.no
> foo.bar.yes foo.bar.ok;        access_log  /var/log/nginx/access.log;
>       location / {                proxy_pass      http://172.27.0.2/;
>               include         /etc/nginx/proxy.conf;        }}
>
> --
> Ikai Lan
> Developer Programs Engineer, Google App Engine
> plus.ikailan.com | twitter.com/ikai
>
> On Fri, Nov 25, 2011 at 5:36 PM, Desmond Brand wrote:
>
>
>
>
>
>
>
> > I wrote more detailed instructions for how to set this up:
>
> >http://desmondbrand.com/blog/2011/11/15/speed-up-your-app-engine-dev-...
>
> > On 23 November 2011 18:43, Calvin  wrote:
>
> >> On Wednesday, November 23, 2011 4:38:48 PM UTC-8, PK wrote:
>
> >>> 3. Finally, this is not new and was already worked around in my
> >>> environment, but if you have a realistic application you need to proxy the
> >>> serving of static content (images/javascript/css) if you want to get any
> >>> work done. I use apache for that and a mini framework I have built. I
> >>> describe the issue more 
> >>> here.
> >>> It might also help in the long term, if you star issues 
> >>> 343,
> >>> 6005 and 4720.
>
> >>> Do you have more details on how you do this?  Do you just do a redirect
> >> to the Apache server when a static asset is requested from dev_appserver?
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Google App Engine" group.
> >> To view this discussion on the web visit
> >>https://groups.google.com/d/msg/google-appengine/-/M5yP6ScsCC4J.
>
> >> To post to this group, send email to google-appengine@googlegroups.com.
> >> To unsubscribe from this group, send email to
> >> google-appengine+unsubscr...@googlegroups.com.
> >> For more options, visit this group at
> >>http://groups.google.com/group/google-appengine?hl=en.
>
> >  --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine+unsubscr...@googlegroups.com.
> > For more options, visit this group at
> >http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: The Performance of the SDK development server has become unbearable

[Ikai Lan (Google)]

Desmond, thanks for that! I think nginx is a bit lighter weight and easier
to configure:

https://help.ubuntu.com/community/Nginx/ReverseProxy

nginx is a bit easier to configure than Apache, especially if you're just
using it for development. Here's a sample reverse proxy config:


server {listen   80;server_name  foo.bar.no
foo.bar.yes foo.bar.ok;access_log  /var/log/nginx/access.log;
  location / {proxy_pass  http://172.27.0.2/;
  include /etc/nginx/proxy.conf;}}


--
Ikai Lan
Developer Programs Engineer, Google App Engine
plus.ikailan.com | twitter.com/ikai



On Fri, Nov 25, 2011 at 5:36 PM, Desmond Brand wrote:

> I wrote more detailed instructions for how to set this up:
>
> http://desmondbrand.com/blog/2011/11/15/speed-up-your-app-engine-dev-server-with-an-apache-reverse-proxy/
>
>
> On 23 November 2011 18:43, Calvin  wrote:
>
>> On Wednesday, November 23, 2011 4:38:48 PM UTC-8, PK wrote:
>>>
>>> 3. Finally, this is not new and was already worked around in my
>>> environment, but if you have a realistic application you need to proxy the
>>> serving of static content (images/javascript/css) if you want to get any
>>> work done. I use apache for that and a mini framework I have built. I
>>> describe the issue more 
>>> here.
>>> It might also help in the long term, if you star issues 
>>> 343,
>>> 6005 and 4720.
>>>
>>> Do you have more details on how you do this?  Do you just do a redirect
>> to the Apache server when a static asset is requested from dev_appserver?
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msg/google-appengine/-/M5yP6ScsCC4J.
>>
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: The Performance of the SDK development server has become unbearable

[Desmond Brand]

I wrote more detailed instructions for how to set this up:
http://desmondbrand.com/blog/2011/11/15/speed-up-your-app-engine-dev-server-with-an-apache-reverse-proxy/

On 23 November 2011 18:43, Calvin  wrote:

> On Wednesday, November 23, 2011 4:38:48 PM UTC-8, PK wrote:
>>
>> 3. Finally, this is not new and was already worked around in my
>> environment, but if you have a realistic application you need to proxy the
>> serving of static content (images/javascript/css) if you want to get any
>> work done. I use apache for that and a mini framework I have built. I
>> describe the issue more 
>> here.
>> It might also help in the long term, if you star issues 
>> 343,
>> 6005 and 4720.
>>
>> Do you have more details on how you do this?  Do you just do a redirect
> to the Apache server when a static asset is requested from dev_appserver?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/M5yP6ScsCC4J.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: The Performance of the SDK development server has become unbearable

[Andrin von Rechenberg]

+1 on speeding up  dev_appserver.py

On Thu, Nov 24, 2011 at 4:49 AM, Mickey  wrote:

> Big thanks PK. That patch for Issue 6335 made a HUGE difference in the
> performance of the dev server.  I wasn't even aware that logging was
> identified as an issue in SDK 1.6.0 until this post.  Everything else I
> read seemed to point to the SetupIndexes problem, and that wasn't effecting
> me.
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/LRi97gavywIJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: The Performance of the SDK development server has become unbearable

[Mickey]

Big thanks PK. That patch for Issue 6335 made a HUGE difference in the 
performance of the dev server.  I wasn't even aware that logging was 
identified as an issue in SDK 1.6.0 until this post.  Everything else I 
read seemed to point to the SetupIndexes problem, and that wasn't effecting 
me.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/LRi97gavywIJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: The Performance of the SDK development server has become unbearable

[Tim Hoffman]

Hi PK

Not sure what you call a lot of data.  I have successfully imported 500,000 
entities into the sqlite backend.
The trick I found was to specify single thread during the bulk upload.  I 
found with multiple threads the dev server
would ultimately lock up or the bulk upload would fall over with lots of 
errors.

Rgds

Tim

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/r2Eu2awiOPUJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: The Performance of the SDK development server has become unbearable

[Calvin]

On Wednesday, November 23, 2011 4:38:48 PM UTC-8, PK wrote:
>
> 3. Finally, this is not new and was already worked around in my 
> environment, but if you have a realistic application you need to proxy the 
> serving of static content (images/javascript/css) if you want to get any 
> work done. I use apache for that and a mini framework I have built. I 
> describe the issue more 
> here. 
> It might also help in the long term, if you star issues 
> 343, 
> 6005 and 4720. 
>
> Do you have more details on how you do this?  Do you just do a redirect to 
the Apache server when a static asset is requested from dev_appserver?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/M5yP6ScsCC4J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: The Performance of the SDK development server has become unbearable

[PK]

I want to close the loop on this thread with some positive notes. For the 
past few days, I have been working with a super fast dev environment thanks 
to the workarounds I describe below, follow the links for the kudos. In 
addition, to how the server feels during the development cycle, the 
application regression tests now run three times faster. 

Here is a summary of workarounds for 1.6.0. Please do star the issues 
mentioned if you want to see them fixed by the Google team in a future 
release. Patches are at the end.

1. Comment out the SetupIndexes line in dev_appserver.py (see/star issue 
6005  for 
the reason)
2. Turn off all dev server attempts to write logs into the datastore, this 
seems to be an upcoming feature that currently just slows down development 
(see/star issue 
6355for this) 
3. Finally, this is not new and was already worked around in my 
environment, but if you have a realistic application you need to proxy the 
serving of static content (images/javascript/css) if you want to get any 
work done. I use apache for that and a mini framework I have built. I 
describe the issue more 
here. 
It might also help in the long term, if you star issues 
343, 
6005 and 4720. 

One more piece of good news for me, sqllite started working with my 
application. I do not remember when was the last time I tried it and would 
not pass my test cases and/or hang once in a while; now it mostly works. 
Unfortunately, when I tried to upload a lot of data it locked pretty 
quickly, so it is not as great as I thought but these are steps in the 
right direction.

I want to acknowledge Robert S. help from Google who followed up on this 
thread and updated the bugs in the database.

PATCHES:
*1. **+++ google_appengine_1.6.0/google/appengine/tools/dev_appserver.py  *

*@@ -2628,7 +2629,7 @@*
-dev_appserver_index.SetupIndexes(config.application, root_path)
+#dev_appserver_index.SetupIndexes(config.application, root_path)

*2. +++ google_appengine_1.6.0/google/appengine/tools/dev_appserver.py  *  
@@ -1435,11 +1435,12 @@

 sys.stderr = LoggingStream()

-logservice._global_buffer = logservice.LogsBuffer()
+logservice.AUTOFLUSH_ENABLED = False
+#logservice._global_buffer = logservice.LogsBuffer()

-app_log_handler = app_logging.AppLogsHandler(
-logservice.logs_buffer().stream())
-logging.getLogger().addHandler(app_log_handler)
+#app_log_handler = app_logging.AppLogsHandler(
+#logservice.logs_buffer().stream())
+#logging.getLogger().addHandler(app_log_handler)

 os.environ.clear()
 os.environ.update(env)
@@ -1514,9 +1515,9 @@
 sys.stdout = old_stdout


-logservice_stub._flush_logs_buffer()
+#logservice_stub._flush_logs_buffer()
 sys.stderr = old_stderr
-logging.getLogger().removeHandler(app_log_handler)
+#logging.getLogger().removeHandler(app_log_handler)

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/dTTOjkgp1tsJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: The Performance of the SDK development server has become unbearable

[PK]

Hi Robert,

I Updated issue 6355 with appstats data, let me know if this is what you 
want to see.

Thanks,
PK

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/gXic3t67u7UJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: The Performance of the SDK development server has become unbearable

[PK]

Robert:

1. The static content issue I know about is an issue on all platforms and 
has primarily to do with the single threaded nature of the dev server. See 
here about what I mean: http://www.gae123.com/articles/gaet/fir-tim-exp.html

2. Issue 6005 is definitely part of the what I see, I updated the bug, 
thanks for pointing me to that direction

3. However, the issue of the dev server being extremely slow on certain 
queries that appears to be new in 1.6.0 is a separate issue. I filed issue 
6355 to track this.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/xwo_zPqOU-sJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: The Performance of the SDK development server has become unbearable

[Mickey]

Robert: I posted the following comment to 
http://code.google.com/p/googleappengine/issues/detail?id=5134, but it may 
be more useful here...

I didn't notice this until after 1.6.0 came out. It seems as if the 
following two lines of code is a lot slower than before.

The first runs fast (1/10th of a second): a query that looks for child 
Books of the_user entity.
The second runs much slower (4 seconds): a query that looks for Books in 
which the_user is in the Book's ReferenceProperty

DEBUG2011-11-17 05:44:02,299 main.py:483] 1
some_books = Book.all().ancestor(the_user).filter('good =', True).fetch(30)
DEBUG2011-11-17 05:44:02,336 main.py:485] 2
other_book = the_user.books.filter('good =', False).get()
DEBUG2011-11-17 05:44:06,661 main.py:487] 3

plus it seems like writing to the datastore takes a lot longer than 
before.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/1RxDMDUOyK8J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: The Performance of the SDK development server has become unbearable

[Matt Jibson]

I saw equivalent behavior with the SDK on Windows and OpenBSD when
serving images from the datastore. I had a page with multiple
images.get_serving_url (with and without resizing) uses, and those
increased page load times by some seconds per image in the SDK.

On Thu, Nov 17, 2011 at 9:37 AM, Robert Schuppenies  wrote:
> Thanks PK. Do you happen to have a appstats screenshot for things like
> that?
> From what I read so far, it seems the speed issues are either related to
> serving static content or datastore, so that's already some valuable
> insight.
> For datastore, 'vladdrac' already did some sleuthing and put his results
> in http://code.google.com/p/googleappengine/issues/detail?id=6005. Do you
> think what you are seeing could be related?
> For the slow serving of static content, is that an issue for only Windows or
> are people seeing this on other platforms, too? If so, could you elaborate
> and maybe provide a reproducible test scenario?
>
> On Wed, Nov 16, 2011 at 7:51 PM, PK  wrote:
>>
>> But, what is even more annoying and new with 1.6.0 is that very often the
>> datastore decides to go out for lunch in the middle of executing a query.
>> Look for example at this one: about a minute to run a really simple query
>> that returns one entity, in a datastore that I just created and maybe it has
>> 200 entities of about 15 kinds. The Domain kind has two entities!!!
>>
>> INFO 2011-11-17 03:39:47,371 query.py:260] SWQ: Executing 'SELECT *
>> FROM Domain WHERE dom_names = :1' ['ls']
>> INFO 2011-11-17 03:40:38,751 query.py:264] SWQ: 1
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msg/google-appengine/-/nE4DRRHUdC8J.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: The Performance of the SDK development server has become unbearable

[Robert Schuppenies]

Thanks PK. Do you happen to have a appstats screenshot for things like
that?

>From what I read so far, it seems the speed issues are either related to
serving static content or datastore, so that's already some valuable
insight.

For datastore, 'vladdrac' already did some sleuthing and put his results in
http://code.google.com/p/googleappengine/issues/detail?id=6005. Do you
think what you are seeing could be related?

For the slow serving of static content, is that an issue for only Windows
or are people seeing this on other platforms, too? If so, could you
elaborate and maybe provide a reproducible test scenario?


On Wed, Nov 16, 2011 at 7:51 PM, PK  wrote:

> But, what is even more annoying and new with 1.6.0 is that very often the
> datastore decides to go out for lunch in the middle of executing a query.
> Look for example at this one: about a minute to run a really simple query
> that returns one entity, in a datastore that I just created and maybe it
> has 200 entities of about 15 kinds. The Domain kind has two entities!!!
>
> INFO 2011-11-17 *03:39:47*,371 query.py:260] SWQ: Executing 'SELECT
> *  FROM Domain WHERE dom_names = :1' ['ls']
> INFO 2011-11-17 *03:40:38*,751 query.py:264] SWQ: 1
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/nE4DRRHUdC8J.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: The Performance of the SDK development server has become unbearable

[PK]

But, what is even more annoying and new with 1.6.0 is that very often the 
datastore decides to go out for lunch in the middle of executing a query. 
Look for example at this one: about a minute to run a really simple query 
that returns one entity, in a datastore that I just created and maybe it 
has 200 entities of about 15 kinds. The Domain kind has two entities!!!

INFO 2011-11-17 *03:39:47*,371 query.py:260] SWQ: Executing 'SELECT *  
FROM Domain WHERE dom_names = :1' ['ls']
INFO 2011-11-17 *03:40:38*,751 query.py:264] SWQ: 1

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/nE4DRRHUdC8J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: The Performance of the SDK development server has become unbearable

[PK]

Hi Robert,

I am glad to see somebody from Google showing interest in the problem. 
Please take a look at the benchmarks I reported in 
thisthread
 on Sep 6 and just updated with results for 1.6.0

This is a test case from a test suite of a big app. I cannot cut and paste 
but is totally reproducible on both my mac and ubuntu systems. Same code 
base, I just change the GAE SDK version from 1.5.2 to a later one, run the 
tests and get 100%+ performance degradation!!!

Having said that, I am in the Mt View area and open to a number of options 
to work with Google to get to the bottom of this:

1. come in the Mt. View campus with the code
2. show you through a gotomeeting/webex session
3. run the tests with instrumentation code you provide and send you the 
results

Let me know

Thanks,
PK

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/vQj5FXMLAd8J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: The Performance of the SDK development server has become unbearable

[King]

Hi Robert,

I haven't been able to figure out exactly what is the issue, so that others 
could reproduce it. It seems like it's happening for all my requests, so 
that's not helping me narrow it down at the moment.  I'll try and track 
down which code is the slowest, and see if I can get enough information to 
open a ticket.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/hDGWipXvkRYJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: The Performance of the SDK development server has become unbearable

[Wilson MacGyver]

is it across the board or just gae-python? I haven't noticed any problem
with gae-java or gae-go.




-- 
Omnem crede diem tibi diluxisse supremum.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: The Performance of the SDK development server has become unbearable

[Robert Schuppenies]

Is it possible to reproduce the issues you are seeing, for example with a
code snippet or an app? If so could you file a ticket, attach the
snippet/app and describe how we can reproduce it?

FWIW, I personally haven't experienced any performance degradation, but
that only indicates I am doing something differently from you.

cheers,
robert

On Wed, Nov 16, 2011 at 2:20 PM, King  wrote:

> Has there been any response from Google about this yet?  Perhaps in
> another thread or the chatrooms? I have noticed a huge drop in performance
> with the python 1.6.0 dev server.
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/5eQIDomPdwwJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: The Performance of the SDK development server has become unbearable

[King]

Has there been any response from Google about this yet?  Perhaps in another 
thread or the chatrooms? I have noticed a huge drop in performance with the 
python 1.6.0 dev server.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/5eQIDomPdwwJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: The Performance of the SDK development server has become unbearable

[Andrius A]

+10 totally agree. big pain and impossible to develop using backends +
queues + channels, it takes all my day, night and weekends!

On 16 November 2011 22:12, PK  wrote:

> *Rekby*: Using --sql_lite does not work for me because to this day there
> is no parity between the --sql_lite version and the production datastore.
> Others and I have filed issues about it. Others have reported that sqllite
> helps with large datasets. It used to be that the SDK datastore would get
> very slow as the dataset was getting larger, but with 1.6.0 it has become
> slow and unpredictable/cranky even with tiny test datasets.
>
> *Desmond*: Proxying static content is a useful and necessary technique
> for any real world app developed with the GAE SDK. I have been developing a
> real-world app with GAE for four years and I could have not survived
> without it I first wrote about this 
> here
> back on July 10th 2010. I do miss the days when this was my only major
> performance issue.
>
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/AEabqH7w2M4J.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: The Performance of the SDK development server has become unbearable

[PK]

*Rekby*: Using --sql_lite does not work for me because to this day there is 
no parity between the --sql_lite version and the production datastore. 
Others and I have filed issues about it. Others have reported that sqllite 
helps with large datasets. It used to be that the SDK datastore would get 
very slow as the dataset was getting larger, but with 1.6.0 it has become 
slow and unpredictable/cranky even with tiny test datasets.

*Desmond*: Proxying static content is a useful and necessary technique for 
any real world app developed with the GAE SDK. I have been developing a 
real-world app with GAE for four years and I could have not survived 
without it I first wrote about this 
here  
back on July 10th 2010. I do miss the days when this was my only major 
performance issue.


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/AEabqH7w2M4J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: The Performance of the SDK development server has become unbearable

[Desmond Brand]

Most of the slowness I experience comes from allowing dev_appserver.py to 
serve static assets. It's possible to get a big speed boost by placing a 
reverse proxy in from of the dev server. Serve static files directly from 
the proxy, and pass through dynamic requests to the dev server as usual. I 
was able to drop my pageload time from 35 s to 2 s by doing this. 
Instructions are on my blog: 
http://desmondbrand.com/blog/2011/11/15/speed-up-your-app-engine-dev-server-with-an-apache-reverse-proxy/

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/LyXVqjsL0t8J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: The Performance of the SDK development server has become unbearable

[Timofey Koolin]

I have big startup time dev server.

It will be fast when I drop datastore file and use  --use_sqlite parameter.

2011/11/16 Rohan Chandiramani 

> +1
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/xqLXe-vABTIJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>



-- 
Blog: www.rekby.ru

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: The Performance of the SDK development server has become unbearable

[Rohan Chandiramani]

+1

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/xqLXe-vABTIJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.