Re: [google-appengine] URLFetch is timing out excessively

[Nischal Shetty]

Twitter has fixed the connectivity issues.

On 26 October 2011 20:07, Nischal Shetty nischalshett...@gmail.com wrote:

 Yeah, you are right about that. I had faced it a few months ago following
 which I moved that part to AWS. The other API calls though in excess are
 below those variable limits.

 I got a reply by twitter on their discussion forum that they are looking
 into the issue. Hoping they find the cause soon.


 On 26 October 2011 20:01, Brandon Wirtz drak...@digerat.com wrote:

 Signed calls are still subject to a variable rate limit based on Twitters
 given load. They throttle up and down based on available capacity.

 ** **

 ** **

 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *Nischal Shetty
 *Sent:* Wednesday, October 26, 2011 2:47 PM

 *To:* google-appengine@googlegroups.com
 *Subject:* Re: [google-appengine] URLFetch is timing out excessively

 ** **

 It works from a non google environment. We have a service running on AWS
 which is working smooth. It's not a whitelist problem because these are
 signed calls which do not account for rate limiting based on IP.

 ** **

 Someone from google has replied saying the problem is not on their end. I
 have raised an issue on the twitter discussion forum (stupid of me to not
 have done that simultaneously). 

 ** **

 ** **

 On 26 October 2011 18:33, Brandon Wirtz drak...@digerat.com wrote:

 So you think that Fetches to only Twitter and FB are broken?   Did you try
 from a non-Goog Environment?  There are Fail Whales all over today.   ***
 *

  

 And if you are running “Hundreds of API calls a minute” you are likely
 hitting the API IP Limits since Twitter won’t grant a white list for Google
 IP’s.

  

 You should be using a proxy to access twitter through a single IP what is
 on the Twitter API white List.

  

  

 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *nischalshetty
 *Sent:* Wednesday, October 26, 2011 12:11 PM
 *To:* google-appengine@googlegroups.com
 *Subject:* Re: [google-appengine] URLFetch is timing out excessively

  

 I agree twitter times out but this is not because of them. I'm talking
 about hundreds of timeout every minute.

  

 We have things in place to retry timeouts, but I'm talkin about timeouts
 even after 3 retries with a timeout of 20 second each. 

  

  

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/YJ2bm0bM_l8J.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



 

 ** **

 --
 -Nischal

 twitter: NischalShetty http://twitter.com/nischalshetty

 facebook: Nischal http://facebook.com/nischal

 ** **

 [image: Description: Image removed by sender.]http://www.justunfollow.com/
   [image: Description: Image removed by sender.] http://www.buffr.com/*
 ***

 ** **

 ** **

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




 --
 -Nischal
 twitter: NischalShetty http://twitter.com/nischalshetty
 facebook: Nischal http://facebook.com/nischal

 http://www.justunfollow.com http://www.buffr.com





-- 
-Nischal
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com http://www.buffr.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine

Re: [google-appengine] Re: URLFetch is timing out excessively

[Nischal Shetty]

Done. I too have raised a production issue -
http://code.google.com/p/googleappengine/issues/detail?id=6188

On 26 October 2011 16:25, Vivek Puri v...@vivekpuri.com wrote:

 Nischal, please star -
 http://code.google.com/p/googleappengine/issues/detail?id=6189

 On Oct 26, 3:20 pm, Vivek Puri v...@vivekpuri.com wrote:
  +1 . Requests to Facebook are also timing out.
 
  On Oct 26, 3:10 pm, nischalshetty nischalshett...@gmail.com wrote:
 
 
 
 
 
 
 
   I agree twitter times out but this is not because of them. I'm talking
   about hundreds of timeout every minute.
 
   We have things in place to retry timeouts, but I'm talkin about
 timeouts
   even after 3 retries with a timeout of 20 second each.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com http://www.buffr.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] URLFetch is timing out excessively

[Nischal Shetty]

It works from a non google environment. We have a service running on AWS
which is working smooth. It's not a whitelist problem because these are
signed calls which do not account for rate limiting based on IP.

Someone from google has replied saying the problem is not on their end. I
have raised an issue on the twitter discussion forum (stupid of me to not
have done that simultaneously).



On 26 October 2011 18:33, Brandon Wirtz drak...@digerat.com wrote:

 So you think that Fetches to only Twitter and FB are broken?   Did you try
 from a non-Goog Environment?  There are Fail Whales all over today.   

 ** **

 And if you are running “Hundreds of API calls a minute” you are likely
 hitting the API IP Limits since Twitter won’t grant a white list for Google
 IP’s.

 ** **

 You should be using a proxy to access twitter through a single IP what is
 on the Twitter API white List.

 ** **

 ** **

 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *nischalshetty
 *Sent:* Wednesday, October 26, 2011 12:11 PM
 *To:* google-appengine@googlegroups.com
 *Subject:* Re: [google-appengine] URLFetch is timing out excessively

 ** **

 I agree twitter times out but this is not because of them. I'm talking
 about hundreds of timeout every minute.

 ** **

 We have things in place to retry timeouts, but I'm talkin about timeouts
 even after 3 retries with a timeout of 20 second each. 

 ** **

 ** **

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/YJ2bm0bM_l8J.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com http://www.buffr.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: URLFetch is timing out excessively

[Nischal Shetty]

I saw the issue that you've reported on twitter. I've subscribed to that.
The issue that I reported on twitter has not been approved yet (probably
because ur issue is already visible) though @episod from twitter commented
that they are looking into the connectivity issue.

On 26 October 2011 18:56, Vivek Puri v...@vivekpuri.com wrote:

 Nischal,

 Send me link to the issue you report on Twitter. Thank you



 On Oct 26, 5:46 pm, Nischal Shetty nischalshett...@gmail.com wrote:
  It works from a non google environment. We have a service running on AWS
  which is working smooth. It's not a whitelist problem because these are
  signed calls which do not account for rate limiting based on IP.
 
  Someone from google has replied saying the problem is not on their end. I
  have raised an issue on the twitter discussion forum (stupid of me to not
  have done that simultaneously).
 
  On 26 October 2011 18:33, Brandon Wirtz drak...@digerat.com wrote:
 
 
 
 
 
 
 
 
 
   So you think that Fetches to only Twitter and FB are broken?   Did you
 try
   from a non-Goog Environment?  There are Fail Whales all over today.
 
 
   ** **
 
   And if you are running “Hundreds of API calls a minute” you are likely
   hitting the API IP Limits since Twitter won’t grant a white list for
 Google
   IP’s.
 
   ** **
 
   You should be using a proxy to access twitter through a single IP what
 is
   on the Twitter API white List.
 
   ** **
 
   ** **
 
   *From:* google-appengine@googlegroups.com [mailto:
   google-appengine@googlegroups.com] *On Behalf Of *nischalshetty
   *Sent:* Wednesday, October 26, 2011 12:11 PM
   *To:* google-appengine@googlegroups.com
   *Subject:* Re: [google-appengine] URLFetch is timing out
 excessively
 
   ** **
 
   I agree twitter times out but this is not because of them. I'm talking
   about hundreds of timeout every minute.
 
   ** **
 
   We have things in place to retry timeouts, but I'm talkin about
 timeouts
   even after 3 retries with a timeout of 20 second each. 
 
   ** **
 
   ** **
 
   --
   You received this message because you are subscribed to the Google
 Groups
   Google App Engine group.
   To view this discussion on the web visit
  https://groups.google.com/d/msg/google-appengine/-/YJ2bm0bM_l8J.
   To post to this group, send email to google-appengine@googlegroups.com
 .
   To unsubscribe from this group, send email to
   google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
   --
   You received this message because you are subscribed to the Google
 Groups
   Google App Engine group.
   To post to this group, send email to google-appengine@googlegroups.com
 .
   To unsubscribe from this group, send email to
   google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
  --
  -Nischal
  twitter: NischalShetty http://twitter.com/nischalshetty
  facebook: Nischal http://facebook.com/nischal
 
  http://www.justunfollow.com http://www.buffr.com

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com http://www.buffr.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] URLFetch is timing out excessively

[Nischal Shetty]

Yeah, you are right about that. I had faced it a few months ago following
which I moved that part to AWS. The other API calls though in excess are
below those variable limits.

I got a reply by twitter on their discussion forum that they are looking
into the issue. Hoping they find the cause soon.

On 26 October 2011 20:01, Brandon Wirtz drak...@digerat.com wrote:

 Signed calls are still subject to a variable rate limit based on Twitters
 given load. They throttle up and down based on available capacity.

 ** **

 ** **

 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *Nischal Shetty
 *Sent:* Wednesday, October 26, 2011 2:47 PM

 *To:* google-appengine@googlegroups.com
 *Subject:* Re: [google-appengine] URLFetch is timing out excessively

 ** **

 It works from a non google environment. We have a service running on AWS
 which is working smooth. It's not a whitelist problem because these are
 signed calls which do not account for rate limiting based on IP.

 ** **

 Someone from google has replied saying the problem is not on their end. I
 have raised an issue on the twitter discussion forum (stupid of me to not
 have done that simultaneously). 

 ** **

 ** **

 On 26 October 2011 18:33, Brandon Wirtz drak...@digerat.com wrote:

 So you think that Fetches to only Twitter and FB are broken?   Did you try
 from a non-Goog Environment?  There are Fail Whales all over today.   

  

 And if you are running “Hundreds of API calls a minute” you are likely
 hitting the API IP Limits since Twitter won’t grant a white list for Google
 IP’s.

  

 You should be using a proxy to access twitter through a single IP what is
 on the Twitter API white List.

  

  

 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *nischalshetty
 *Sent:* Wednesday, October 26, 2011 12:11 PM
 *To:* google-appengine@googlegroups.com
 *Subject:* Re: [google-appengine] URLFetch is timing out excessively

  

 I agree twitter times out but this is not because of them. I'm talking
 about hundreds of timeout every minute.

  

 We have things in place to retry timeouts, but I'm talkin about timeouts
 even after 3 retries with a timeout of 20 second each. 

  

  

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/YJ2bm0bM_l8J.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



 

 ** **

 --
 -Nischal

 twitter: NischalShetty http://twitter.com/nischalshetty

 facebook: Nischal http://facebook.com/nischal

 ** **

 [image: Description: Image removed by sender.]http://www.justunfollow.com/
   [image: Description: Image removed by sender.] http://www.buffr.com/**
 **

 ** **

 ** **

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com http://www.buffr.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

image001.jpgimage002.jpg

Re: [appengine-java] URLFetch Http Method DELETE returns http response 400

[Nischal Shetty]

Anything on this? I'm sorta stuck :(


On 2 September 2011 00:32, Nischal nischalshett...@gmail.com wrote:

 This is the code :

 URLFetchService fetchService = URLFetchServiceFactory.getURLFetchService();
 URL posturl = new URL(http://www.example.com/comment?token=xxx;);
 HTTPResponse response = fetchService.fetch(new HTTPRequest(posturl,
 HTTPMethod.DELETE));
 int responseCode = response.getResponseCode();

 What am I doing wrong here? This always returns a http response code 400. I
 am trying to use the http Delete method.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine for Java group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine-java/-/ybSlx3wJpdkJ.
 To post to this group, send email to
 google-appengine-java@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine-java+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine-java?hl=en.




-- 
-Nischal
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com http://www.buffr.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

Re: [google-appengine] URLFetch causing timeout exceptions but does not appear to be reaching limit (1.5.3 Java SDK)

[Nischal Shetty]

Can you paste an example url that you are using to fetch content?

On 29 August 2011 12:52, Deepak Singh deepaksingh...@gmail.com wrote:

 Can anyone suggest some workaround to resolve this / increase the time
 limit for all requests arising from JAX-WS client code.

 Thanks
 Deepak


 On Mon, Aug 29, 2011 at 1:38 AM, Deepak Singh deepaksingh...@gmail.comwrote:

 I too stuck with the same java.net.SocketTimeoutException: Timeout while
 fetching: exception.

 I am using JAX-WS client to connect to third party service at
 http://webservices.ticketvala.com/axis2/services/WSTicketvala?wsdl

 But i always get the exception even in less than 10 seconds.

 I need one solution to increase this timeout because in my application,
 some 3rd party servers may take too long time to respond. Otherwise if i am
 not able to increase this time, it will become a showstopper for me.

 I tried with TaskQueue but no luck.

 Thanks in advance

 On Fri, Aug 26, 2011 at 12:25 AM, Christopher cspeckrun@gmail.comwrote:

 Part of our application on appengine communicates with Amazon S3,
 mostly for checking the status of files, etc. (so mostly HEAD and GET
 requests, always small payload sizes). Lately (past few weeks) we've
 been getting a lot more SocketTimeout exceptions when we do fetches
 (async). We had the deadline set on the requests for 10 seconds,
 recently tried upping that to 30 until we noticed the documentation
 states 10 is the highest.

 What happens if you supply a number  10 for the deadline? Does it set
 to default (5), the actual max (10) or something else?

 Also, I find it difficult to believe that these basic requests could
 be taking as long as they are -- both Google and Amazon have blazing
 fast connections so why are HEAD/GET requests with extremely small
 (_maybe_ 1kb max) payloads causing timeouts? Some of the requests we
 send are delete or copy commands, which I could understand Amazon
 taking some time to execute before responding, but we're getting
 timeouts when asking Amazon to check existence of a file.

 Does anybody have any insights as to what we might be doing
 incorrectly to cause this to happen?

 Also, here's a sample from our logs, which the timing doesn't seem to
 indicate that this isn't even waiting a second before throwing an
 exception.

 - [25/Aug/2011:10:53:50 -0700] POST /mim/postUpload HTTP/1.1 500
 0.
  2011-08-25 13:53:50.338 /mim/postUpload
 java.util.concurrent.ExecutionException:
 java.net.SocketTimeoutException: Timeout while fetching:

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com http://www.buffr.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Problem with System time? Unable to use twitter oAuth to login to my app

[Nischal Shetty]

Thanks for the info! I will look into the logs soon to see if I can zero
this down to only a few instances.

On 24 August 2011 01:57, Robert Kluin robert.kl...@gmail.com wrote:

 There have been many discussions on this topic.  I think the general
 consensus is that it is pretty rare to find clocks that are out of
 sync by very much (seconds), but it is possible.  Probably if you were
 seeing this across your whole app it wasn't the issue.  If it was
 isolated to a subset of instances, perhaps.  The instance is indicated
 in the log entries, you could always download them and see if there
 was a correlation between errors and the instance.


 Robert






 On Tue, Aug 23, 2011 at 15:26, nischalshetty nischalshett...@gmail.com
 wrote:
  Haven't yet got a reply but the errors have slowed down. It's still
  appearing for a subset of our users though since I can see them in the
 logs
  at regular intervals. I saw similar complaints on the twitter mailing
 list
  but they seem to be suggesting users to check if their system clocks are
 out
  of sync.
  I remember reading that the system clocks on the appengine servers can be
  off by a few hours. Is there a way I can think of a possible solution?
 Would
  moving the login module of my app to a backend help? Will the backend
  clock be in sync or it can have the same problem of the time being off by
 a
  few hours?
  Any pointers to making sure we have an updated clock would be helpful.
 The
  clocks being off by a few minutes won't affect us but an hour or so I
 guess
  would make the oAuth tokens invalid.
 
  --
  You received this message because you are subscribed to the Google Groups
  Google App Engine group.
  To view this discussion on the web visit
  https://groups.google.com/d/msg/google-appengine/-/wPAdw5B4MbgJ.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com http://www.buffr.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Do I really need to switch to HR?

[Nischal Shetty]

Totally! I've been wanting to shift my app from MS to HRD but all the
troubles people are facing as seen on the mailing lists scares me.

@Greg hope it involves tools for apps on GAE/J as well :)

On 6 July 2011 20:01, Gregory D'alesandre gr...@google.com wrote:

 We are working on better tools for migrating to HRD (and they are currently
 being tested), I'll post once we have them widely available.

 Greg


 On Wed, Jul 6, 2011 at 5:52 PM, Waleed Abdulla wal...@ninua.com wrote:

 Please star this issue if you agree that Google should make the migration
 process easier rather than putting the burdon on the developers. After all,
 most developers signed up to GAE to avoid having to deal with issues like
 that.

 http://code.google.com/p/googleappengine/issues/detail?id=5250






 On Wed, Jul 6, 2011 at 8:39 AM, Robert Kluin robert.kl...@gmail.comwrote:

 Keys do contain the appid.

 One solution would be to adjust your code to catch the exception that
 gets thrown when accessing the old apps data, the recreate the key
 for the new appid.

 Robert



 On Wednesday, July 6, 2011, Joshua Smith joshuaesm...@charter.net
 wrote:
  Another thing that occurs to me is that anyplace I've used a key
 external to the application would have to be dealt with.  For example, one
 of our apps has an RSS feed that our managers use to keep track of whether
 customers have uploaded new information.  This looks like:
 
  http://myapp.com/rss?cust=DAFHP983RPFDSALFHDSKLFJHLSDKAFH
 
  (where the gobledegook is the datastore key)
 
  My managers would need to update all their bookmarks, since I believe
 that key has the name of the app pickled in it someplace, right?
 
  Is there any way to avoid having the keys change when I switch to HR?
 
  If not, is there a way that I could decorate database queries so that
 they quietly turn old keys into new ones?
 
  -Joshua
 
  On Jul 5, 2011, at 3:35 PM, Joshua Smith wrote:
 
  I have several apps and I've got this pesky to-do list item for all of
 them, to switch them over to the HR datastore.
 
  In preparation, I've read blog entries by people who have done this,
 and I'm still on the fence about whether it is worth the trouble.
 
  I get very few datastore timeouts in my apps, and periodic maintenance
 has never really been an issue.
 
  I've read about gotchas with eventual consistency, and it seems
 unlikely that I could convince myself that I'm safe from this for any
 reasonably complex app.  (I'm not concerned that I cannot reprogram the apps
 to fix the problem, but rather I'm concerned that I could not ensure there
 weren't going to be problems.  If it ain't broke...)
 
  I'm not clear on how I would go about getting google to grandfather my
 old 2000 emails quotas if I create a new -hr version of my app.
 
  I've also read that migrating uses TONS of CPU, and it is therefore
 going to cost me money to do.
 
  Really, the only reason TO do the switch is that I've read repeated
 admonitions from google that I *should* be using HR.
 
  What's the consensus here?  Is it worth the time, risk, and cost?
 
  -Joshua
 
  --
  You received this message because you are subscribed to the Google
 Groups Google App Engine group.
  To post to this group, send email to
 google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.
 
 
  --
  You received this message because you are subscribed to the Google
 Groups Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com
 .
  To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.
 
 

 --
 --
 Robert Kluin
 Ezox Systems, LLC

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 

Re: [google-appengine] Re: Sending email in batch

[Nischal Shetty]

I would suggest you do it in parallel. Create n tasks with x emails in each
of them. It would be faster that way!



On 26 June 2011 15:52, Branko Vukelic bg.bra...@gmail.com wrote:

 On Sun, Jun 26, 2011 at 9:41 PM, nischalshetty
 nischalshett...@gmail.com wrote:
  If you're using the appengine mail api be sure you want that or something
  along the lines of Amazons mail service because I remember reading a
 thread
  where one of the appengine devs said the email service is not meant for
  large amount of emails (clients would block your email and many others
 might
  take it to be spam)

 Yes I know about this. We will use SendGrid after Beta release, but
 for Alpha, we plan to use the GAE mail API simply to keep things
 rolling.

  How about querying the datastore and making use of a cursor. Pass the
 cursor
  value to the new tasks. Will that work for you?

 The mailing lists are generated dynamically. So once the list is
 compiled from the data in the datastore, I would not like to store it,
 and then repeatedly read it from the datastore each time a task is
 executed. What I do is:

 mailing_list = generate_list(from=some_data)
 task.add(url='/url/to/task',
 params={'mailing_list': mailing_list.join(',')})

 and in the task handler (when I want to do a batch of 50 emails in one
 task):

 mailing_list = params.POST.get('mailing_list').split(',')
 to_process = mailing_list[50:]
 the_rest = mailing_list[:50]

 or in case I want to go one by one:

 to_process = mailing_list.pop(0)
 the_rest = mailing_list

 enqueue the_rest, and process to_process. I haven't tested this yet,
 but I'm wondering if it's better to go one-by-one or many-by-many.

 --
 Branko Vukelić
 bra...@herdhound.com

 Lead Developer
 Herd Hound (tm) - Travel that doesn't bite
 www.herdhound.com

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Sending email in batch

[Nischal Shetty]

Also remember that in a batch you can add a maximum of 100 tasks.

On 26 June 2011 17:00, Branko Vukelic bg.bra...@gmail.com wrote:

 Ok, thanks for all the answers. I'll do it that way.

 On Sun, Jun 26, 2011 at 10:16 PM, Nischal Shetty
 nischalshett...@gmail.com wrote:
  I would suggest you do it in parallel. Create n tasks with x emails in
 each
  of them. It would be faster that way!
 
 
  On 26 June 2011 15:52, Branko Vukelic bg.bra...@gmail.com wrote:
 
  On Sun, Jun 26, 2011 at 9:41 PM, nischalshetty
  nischalshett...@gmail.com wrote:
   If you're using the appengine mail api be sure you want that or
   something
   along the lines of Amazons mail service because I remember reading a
   thread
   where one of the appengine devs said the email service is not meant
 for
   large amount of emails (clients would block your email and many others
   might
   take it to be spam)
 
  Yes I know about this. We will use SendGrid after Beta release, but
  for Alpha, we plan to use the GAE mail API simply to keep things
  rolling.
 
   How about querying the datastore and making use of a cursor. Pass the
   cursor
   value to the new tasks. Will that work for you?
 
  The mailing lists are generated dynamically. So once the list is
  compiled from the data in the datastore, I would not like to store it,
  and then repeatedly read it from the datastore each time a task is
  executed. What I do is:
 
  mailing_list = generate_list(from=some_data)
  task.add(url='/url/to/task',
  params={'mailing_list': mailing_list.join(',')})
 
  and in the task handler (when I want to do a batch of 50 emails in one
  task):
 
  mailing_list = params.POST.get('mailing_list').split(',')
  to_process = mailing_list[50:]
  the_rest = mailing_list[:50]
 
  or in case I want to go one by one:
 
  to_process = mailing_list.pop(0)
  the_rest = mailing_list
 
  enqueue the_rest, and process to_process. I haven't tested this yet,
  but I'm wondering if it's better to go one-by-one or many-by-many.
 
  --
  Branko Vukelić
  bra...@herdhound.com
 
  Lead Developer
  Herd Hound (tm) - Travel that doesn't bite
  www.herdhound.com
 
  --
  You received this message because you are subscribed to the Google
 Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
 
 
 
  --
  -Nischal
  twitter: NischalShetty
  facebook: Nischal
 
 
 
  --
  You received this message because you are subscribed to the Google Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 



 --
 Branko Vukelić
 bra...@herdhound.com

 Lead Developer
 Herd Hound (tm) - Travel that doesn't bite
 www.herdhound.com

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com http://www.buffr.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Not able to connect to PayPal, gives IO Exception!

[Nischal Shetty]

It's not the 30 second problem. This problem existed for a few hours. Paypal
servers were working fine with no lag. I could access them from my local
machine.



On 11 June 2011 16:52, Brandon Wirtz drak...@digerat.com wrote:

 Define “fine”  if you don’t get a response in 10 seconds it will fail on
 GAE.   I’ve run in to this a lot.  PayPal often takes up to 30 seconds to
 gen a response, and on GAE that means the user gets an error of some sort.





 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *nischalshetty
 *Sent:* Saturday, June 11, 2011 12:23 PM

 *To:* google-appengine@googlegroups.com
 *Subject:* [google-appengine] Re: Not able to connect to PayPal, gives IO
 Exception!



 So, this error went off after some time. I contacted the PayPal team and
 they said things were fine on their end. Can someone from the GAE team tell
 me what happened?

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/MXY9Q8FbCM4J.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com http://www.buffr.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Poor performance since the past 2-3 days

[Nischal Shetty]

My tasks are not running either. Or they run once a while and then stop/

AppID: justunfollow

On 26 May 2011 23:12, David s2kd...@gmail.com wrote:

 I am seeing the same problems on my app.  My request graph is
 extremely spiky and almost flat near the bottom in some parts.  It
 seemed to clear up a little near midnight PST last night, but the
 problems are definitely back.  My tasks are also backlogged around 25k
 tasks and not running.

 Please help.

 Thanks,
 David

 AppID: word-play

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
+91-9920240474
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com http://www.buffr.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Tasks are not running from the last one hour!

[Nischal Shetty]

@Robert It's back to normal now. Thanks for the link. Is that even for times
when Task Queue does not work for a few hours (because that happens often).
I thought the issue tracker was for reproducible bugs and feature request.
This seems more like some temporary glitch in the service.

On 27 May 2011 09:43, Robert Kluin robert.kl...@gmail.com wrote:

 I'd submit a production issue:

 http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue







 On Thu, May 26, 2011 at 15:12, nischalshetty nischalshett...@gmail.com
 wrote:
  Can someone please look into this. My tasks run intermittently. My users
  have been complaining :(
 
  --
  You received this message because you are subscribed to the Google Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
+91-9920240474
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com http://www.buffr.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Tasks are not running from the last one hour!

[Nischal Shetty]

Oh, then I'll definitely do that the next time my tasks stop. This wasn't
the first time and I'm guessing it wouldn't be the last. We shall know soon
;)

On 27 May 2011 10:49, Robert Kluin robert.kl...@gmail.com wrote:

 Hey Nischal,
   The production issues template in the issue tracker is for problems in
 the production environment, like serving issues and index problems.
  Personally, I'd say your tasks not executing for hours at a time is a
 production issue.  I guess if they close it and refer you back here we'll
 know I was wrong.  ;)



 Robert




 On Fri, May 27, 2011 at 01:14, Nischal Shetty 
 nischalshett...@gmail.comwrote:

 @Robert It's back to normal now. Thanks for the link. Is that even for
 times when Task Queue does not work for a few hours (because that happens
 often). I thought the issue tracker was for reproducible bugs and feature
 request. This seems more like some temporary glitch in the service.


 On 27 May 2011 09:43, Robert Kluin robert.kl...@gmail.com wrote:

 I'd submit a production issue:

 http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue







 On Thu, May 26, 2011 at 15:12, nischalshetty nischalshett...@gmail.com
 wrote:
  Can someone please look into this. My tasks run intermittently. My
 users
  have been complaining :(
 
  --
  You received this message because you are subscribed to the Google
 Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com
 .
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




 --
 -Nischal
 +91-9920240474
 twitter: NischalShetty http://twitter.com/nischalshetty
 facebook: Nischal http://facebook.com/nischal

 http://www.justunfollow.com http://www.buffr.com


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
+91-9920240474
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com http://www.buffr.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [appengine-java] Re: Getting class cast exception in BackendServersFilter after 1.5 sdk update

[Nischal Shetty]

@Don Happy to know you could find the problem. I'll stick to the previous
sdk till you guys get 1.5.1 out (the wait is gonna be harder considering all
the goodness packed into 1.5). Once again, thanks a lot for looking into
this.

On 17 May 2011 22:05, Don Schwarz schwa...@google.com wrote:

 I understand what's going on now.  No need to send me more war files, but
 my thanks to Nischal for a great test case.

 Struts2 wraps the request with StrutsRequestWrapper, and apparently
 modifies getAttribute() calls to check XWork's ValueStack if there is no
 attribute with the specified name.  XWork uses OGNL, and OGNL is trying to
 interpret the attribute name as an expression.

 We have some internal headers (e.g. X-AppEngine-BackendName) that we are
 also using as request attributes in some cases.  As far as I can tell, this
 is allowed by the servlet spec.  However, OGNL is interpreting the minus
 signs as subtraction and the unknown strings as 0 and returning a
 BigDecimal.  Weird.

 Since XWork is returning a non-null value for one of our backend-related
 attributes we think this is a request that should be redirect to a backend,
 so we're going through some new code in 1.5 that isn't relevant in your
 application.  However, we later fail because the attributes were of the
 wrong type.

 I will get this fixed in the next release, but I've been unable to find an
 easy workaround for you folks.  There appears to be no way to disable the
 use of OGNL in Struts2.  My best guess is that you can use the
 struts.valueStackFactory constant to register a custom ValueStackFactory
 that either does not use OGNL, or short-circuits this particular case.  If
 you're not feeling very adventurous, I recommend that you remain on the
 1.4.3 SDK for testing until 1.5.1 is released.

 On Mon, May 16, 2011 at 4:29 PM, Shashank sharma shash...@ebeedu.comwrote:

 @Don

 I am getting the same error. Also using struts 2.0 and also noticed
 this after upgrading to 1.5. Should i send you the war or has someone
 else already sent it to you?


 On May 17, 2:13 am, Don Schwarz schwa...@google.com wrote:
  This is very strange.  Can one of you send me your war directory?
 
  On Mon, May 16, 2011 at 12:14 PM, Nischal Shetty
  nischalshett...@gmail.comwrote:
 
 
 
 
 
 
 
   @Sean  Let's hope someone from the GAE team helps us. They've been
 super
   busy the entire last week. Hoping someone helps us out soon.
 
   If you come across a possible solution please do update this thread,
 I'll
   do the same.
 
   On 16 May 2011 00:11, svoeller sean.voel...@gmail.com wrote:
 
   I'm experiencing the same issue - stack trace is identical. Also
 using
   struts 2.0 and also noticed this after attempting to upgrade to 1.5.
 
   On May 13, 11:32 am, Nischal nischalshett...@gmail.com wrote:
GAE team, need your help here.
 
   --
   You received this message because you are subscribed to the Google
 Groups
   Google App Engine for Java group.
   To post to this group, send email to
   google-appengine-java@googlegroups.com.
   To unsubscribe from this group, send email to
   google-appengine-java+unsubscr...@googlegroups.com.
   For more options, visit this group at
  http://groups.google.com/group/google-appengine-java?hl=en.
 
   --
   -Nischal
   +91-9920240474
   twitter: NischalShetty http://twitter.com/nischalshetty
   facebook: Nischal http://facebook.com/nischal
 
   http://www.justunfollow.com http://www.buffr.com
 
--
   You received this message because you are subscribed to the Google
 Groups
   Google App Engine for Java group.
   To post to this group, send email to
   google-appengine-java@googlegroups.com.
   To unsubscribe from this group, send email to
   google-appengine-java+unsubscr...@googlegroups.com.
   For more options, visit this group at
  http://groups.google.com/group/google-appengine-java?hl=en.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine for Java group.
 To post to this group, send email to
 google-appengine-java@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine-java+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine-java?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine for Java group.
 To post to this group, send email to
 google-appengine-java@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine-java+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine-java?hl=en.




-- 
-Nischal
+91-9920240474
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com http://www.buffr.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine for Java group.
To post to this group, send email

Re: [appengine-java] Re: Getting class cast exception in BackendServersFilter after 1.5 sdk update

[Nischal Shetty]

@Sean  Let's hope someone from the GAE team helps us. They've been super
busy the entire last week. Hoping someone helps us out soon.

If you come across a possible solution please do update this thread, I'll do
the same.


On 16 May 2011 00:11, svoeller sean.voel...@gmail.com wrote:

 I'm experiencing the same issue - stack trace is identical. Also using
 struts 2.0 and also noticed this after attempting to upgrade to 1.5.

 On May 13, 11:32 am, Nischal nischalshett...@gmail.com wrote:
  GAE team, need your help here.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine for Java group.
 To post to this group, send email to
 google-appengine-java@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine-java+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine-java?hl=en.




-- 
-Nischal
+91-9920240474
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com http://www.buffr.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

Re: [appengine-java] Re: Backends.xml ERROR when deploying

[Nischal Shetty]

Here's how you GET using URLFetch :


URLFetchService fetchService =
URLFetchServiceFactory.getURLFetchService();
URL fetchURL = null;

HTTPResponse response = null;
String responseString = null;


try {

fetchURL = new URL(http://www.google.com;);
 response = fetchService.fetch(fetchURL);

int statusCode = response.getResponseCode();

if (statusCode == HttpStatus.SC_OK) {
 responseString = new String(response.getContent());
}
} catch (HttpException e) {
 } catch (IOException e) {
}



On 17 May 2011 04:53, Dennis Lo lo.den...@gmail.com wrote:

 Hi,

 I see.

 So a public backend would be:


 backends
 backend name=spider
 classB2/class
 instances5/instances
 options
 dynamictrue/dynamic
 publictrue/public
 /options
 /backend
 /backends

 Also, what is urlfetch?
 I haven't used these before.
 Do you have any blogs, tutorials with a great example which illustrates how
 I could use it to call a backend.

 Alternatively, if you don't have any blogs,tutorials etc, could you explain
 with some examples?



 On Tue, May 17, 2011 at 3:49 AM, JakeP jake.pier...@gmail.com wrote:

 Dennis,

 I'm still trying to figure out the best way to call the backend
 programmatically from within GAE. The simplest way I know of is with
 urlfetch.

 Your backend has to be marked public for you to be able to access it
 via 'spider' from outside of GAE - i.e. your browser.

 Thanks for the link.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine for Java group.
 To post to this group, send email to
 google-appengine-java@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine-java+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine-java?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine for Java group.
 To post to this group, send email to
 google-appengine-java@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine-java+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine-java?hl=en.




-- 
-Nischal
+91-9920240474
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com http://www.buffr.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

Re: [google-appengine] How to find IP address to add to blacklist

[Nischal Shetty]

I don't do that but I can still see the IPs. May be your lie works in my
case :P

On 14 May 2011 13:09, Brandon Wirtz drak...@digerat.com wrote:

 I lied, the stub code I start my projecta with has

logging.debug('User-Agent = %s, Referrer = %s',
  self.request.user_agent,
  self.request.referer)

 Which will give you this

 -Original Message-
 From: google-appengine@googlegroups.com
 [mailto:google-appengine@googlegroups.com] On Behalf Of Brandon Wirtz
 Sent: Friday, May 13, 2011 6:37 PM
 To: google-appengine@googlegroups.com
 Subject: RE: [google-appengine] How to find IP address to add to blacklist

 Logs should have the ip address as the first thing on the second line of
 each entry


 -Original Message-
 From: google-appengine@googlegroups.com
 [mailto:google-appengine@googlegroups.com] On Behalf Of Kwame
 Sent: Friday, May 13, 2011 6:22 PM
 To: Google App Engine
 Subject: [google-appengine] How to find IP address to add to blacklist

 Hi,
 I need help tracking down a possible hacker trying to access my GAE
 website.
 The person is sending multiple requests to phpMyAdmin/scripts/ setup.php I
 guess in an attempt to bring down my site!!! Fortunately they don't know I
 use GAE, and that GAE does not have this file.

 I really need to blacklist this persons URI, but the only information
 provided in Dashboard is the path listed above.

 How can I find out their URI?

 Thanks for your help.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
+91-9920240474
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com http://www.buffr.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] What do you want to see answered in Greg's pricing FAQ?

[Nischal Shetty]

You cannot just compare GAE with Rackspace simply because GAE is unmanaged,
we don't do a thing.

Though I would still maintain that the prices are indeed high.

On 12 May 2011 21:46, Peter Petrov onest...@gmail.com wrote:

 My single question for Greg's FAQ:

 1) What is the justification behind the extremely high price for additional
 instances?

 Here is a comparison between a GAE Frontend Python Instance and a small
 Rackspace Cloud Server:

 GAE Frontend Python Instance:
 -
 Monthly price: $57.60 ($36.00 if reserved)
 RAM: 128 MB memory cap
 CPU: 600 MHz limit
 Capabilities: Limited by the Python sandbox. No native code execution.
 Single-threaded right now, possible future multi-threading obstructed by the
 GIL.

 Rackspace Cloud Server 256 MB:
 -
 Monthly price: $10.80
 RAM: 256 MB fixed
 CPU: Guaranteed proportional minimum; Free CPU bursting (I'm using the full
 power of 4 cores 99% of the time)
 Capabilities: Full-featured Linux box, can do whatever you want on it.


 On Thu, May 12, 2011 at 10:14 AM, Kenneth kennet...@aladdinschools.comwrote:

 Greg mentioned he was putting together an FAQ so let's help him out!

 If you're going to answer this just put in your question into a single
 line, let's not try and answer them here or give opinions, there's plenty of
 other threads for that. I do understand that Google doesn't have answers to
 some these.

 Here's my list:

 1) What is the time granularity of the instance pricing?  ie if I have an
 instance up for 5 minutes, what am I charged, $0.08 / 60*5?
 2) Will I be able to tune the scheduler myself, ie set it to performance
 or low cost,  Will I be able to limit the min or max number of instances
 created (with the obvious impact on user experience)?
 3) Python concurrency, will this require any code changes, do you have any
 estimates based on your testing of the number of well behaved requests per
 second a single instance will be able to handle for a given framework?
 4) Database charges, when can you give us more details over what Max gave
 in the other thread, are you charging for deletes, what do you expect the
 ratio to be between the new pricing metric and the Datastore API calls
 metric we have today?
 5) Will you be charging differently for instances that use different
 amounts of memory, since this seems to be the cost that you're going after
 that isn't charged for in the current model.

 Thanks,
 Kenneth

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
+91-9920240474
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Is App Engine suddenly becoming more expensive???

[Nischal Shetty]

GAE now has Free and Enterprise plans. The free plan is useless for any
serious app and the enterprise plan will rip single developers like me and
everyone else who thinks they can sustain this through adsense and other
means (unless you do not want to make anything for yourself).

There should be a third plan in between the free and enterprise and that
plan exists, it is the current paid plan that we are all making use of where
we pay for the CPU Hours. Increase the price for that if you want!

I currently pay around $200-$300 per month to GAE. I can go a maximum of 2x
increase in price (I would still cry a couple of nights and curse whoever
thought of charging so much) but anything above that would simply make me
move on to some other platform.

I'm guessing there would be more like me. Enterprises will not spread the
word about GAE, most if not all will not participate in your google group
discussion to help others out, no one will tweet thank you GAE for making
my life easier. We are not Enterprise!

There are a lot of things that won't happen if GAE starts treating
individual devs (or small teams of 2-3 devs) as ENTERPRISE!

Bring out a third plan. Don't be Evil.

On 12 May 2011 21:19, Brandon Donnelson branflake2...@gmail.com wrote:

 I'm sorry, but what the FUD is going on in here :).

 Brandon

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
+91-9920240474
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com http://www.buffr.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Is App Engine suddenly becoming more expensive???

[Nischal Shetty]

I have anywhere from 80 - 150+ instances running at any point of time
(without the multi threaded thingy).

I have a question - Does it mean charges would be in 4 digits per month?

On 11 May 2011 19:58, Nitu Chiring nituchir...@gmail.com wrote:

 I am currently having Always on feature. so does that mean I shall end up
 paying 3 * 24 * 30*0.05 $s?
 Also my application uses email heavily to notify users.
 And the free quota been just reduced to 100 recipients .

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Is App Engine suddenly becoming more expensive???

[Nischal Shetty]

As far as I can tell, yes that is how it is. The second instance would spin
up if the first one is busy serving some other request.

They probably have a request pool. So requests would wait a little and if
none of the existing instances is ready to serve the request then a new
instance would spin up.

On 11 May 2011 20:54, Vinuth Madinur vinuth.madi...@gmail.com wrote:

 @Nischal
 I have anywhere from 80 - 150+ instances running at any point of time
 (without the multi threaded thingy).

 Does this mean 2 simultaneous requests = 2 instances? That doesn't seem
 right. Am I missing something?



 On Wed, May 11, 2011 at 8:25 PM, Nischal Shetty nischalshett...@gmail.com
  wrote:

 I have anywhere from 80 - 150+ instances running at any point of time
 (without the multi threaded thingy).

 I have a question - Does it mean charges would be in 4 digits per month?

 On 11 May 2011 19:58, Nitu Chiring nituchir...@gmail.com wrote:

 I am currently having Always on feature. so does that mean I shall end
 up paying 3 * 24 * 30*0.05 $s?
 Also my application uses email heavily to notify users.
 And the free quota been just reduced to 100 recipients .

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




 --
 -Nischal
 twitter: NischalShetty http://twitter.com/nischalshetty
 facebook: Nischal http://facebook.com/nischal

 http://www.justunfollow.com


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
+91-9920240474
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com http://www.buffr.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Is App Engine suddenly becoming more expensive???

[Nischal Shetty]

We should all wait for the comparative billing to come out. Let's not jump
to conclusions. One thing is for sure, if the prices are insane, Google
wouldn't be so dumb to stick to it. They will obviously review it and change
it.

Hoping things turn positive for all of us.

On 12 May 2011 00:13, Gregory D'alesandre gr...@google.com wrote:

 On Wed, May 11, 2011 at 11:29 AM, Ugorji ugo...@gmail.com wrote:

 +1

 IMHO, This is the single biggest risk to Google with this move. The
 developers that adopted this did so out of trust, even when the platform was
 in preview mode and some runtimes were experimental. The price was right,
 and Google guaranteed that they would not pull the plug under us without
 giving us three year advance notice. With this, and the trust (mostly
 deserved), we went about happily enjoying all of this insane google
 engineering now at our fingertips.


 I can understand where you are coming from, but this is part of the reason
 we want to go out of preview, so that you can be certain App Engine will be
 around for a long time and things won't shift.  As long a product is
 preview, things can change at any point, we aren't changing prices
 arbitrarily, we are doing it so we can ensure the product is around for
 years to come and therefore we are able to leave preview.


 It feels like the rug was pulled out from under us, and the unknowns are
 killing us.


 I'm working on an FAQ that should help answer many of the questions we've
 heard.


 Google has always been very protective of their brand and their company's
 DNA (which is rooted in trust and engineering). Let's hope all this concern
 ends up being much ado about nothing.


 To be clear, prices will be higher, but I've seen people quoting number
 such as 70x higher which should not be the case.  Once we have the changes
 to the scheduler done as well as the comparative bills, you'll then be able
 to see how much it will actually cost.  We could've waited until that point
 to talk about this but we wanted to give as much advance notice as possible.

 Thanks again for the questions and comments, its getting hard to get on the
 wifi, so just another pre-apology if I'm unable to answer additional
 questions for a while...

 Greg




  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Tasks are not executing!

[Nischal Shetty]

Yeah, tasks have been behaving strange from the last couple of days. I hope
someone looks into this.

On 6 May 2011 19:23, Steven Cains ste...@cains.me wrote:

 Tasks have been slow here. Our default queue is supposed to run @ 60/
 m, it's slowed to 6/m for no reason.

 On May 5, 8:44 pm, David Mora dla.m...@gmail.com wrote:
  We are on master/slave: 18k per minutes as we speak
 
  On 5 May 2011 13:37, Stephen Johnson onepagewo...@gmail.com wrote:
 
 
 
   Sorry but that's what I'm seeing. Anyone else?
 
   On Thu, May 5, 2011 at 12:35 PM, Nischal Shetty 
 nischalshett...@gmail.com
wrote:
 
   I'm on Master/Slave. Don't tell me Task Queue's would also behave
   different ways for different data stores :(
 
   On 6 May 2011 01:01, Stephen Johnson onepagewo...@gmail.com wrote:
 
   Just to add some more insight hopefully. I'm still developing my
   application and have moved to HR datastore but kept my Master/Slave
 test
   running. Past couple of days Master/Slave has exhausted Task Queue of
   100,000 but HR is working fine. Typical Task Queue would probably be
 5000
   for the day. Already Master/Slave has exhausted quota. For those
 having task
   queue problems are you on HR or Master/Slave?
 
   On Thu, May 5, 2011 at 12:21 PM, David Mora dla.m...@gmail.com
 wrote:
 
   hey guys,
 
   We are not experiencing this problem and we execute like 15k+ task
 per
   minute probably.
 
   One tip tho, might worth to mention:
 
   The ones staying in the queue have 0 retry attempts? Retries uses a
 push
   back mechanism so if they fail they are kept on the queue if your
   configuration is not suitable to handle this case
 
   On 5 May 2011 13:12, nischalshetty nischalshett...@gmail.com
 wrote:
 
   Tasks are not executing again! Close to 30K pending tasks in my
 queue,
   help!
 
   --
   You received this message because you are subscribed to the Google
   Groups Google App Engine group.
   To post to this group, send email to
 google-appengine@googlegroups.com
   .
   To unsubscribe from this group, send email to
   google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
   --
  http://about.me/david.mora
 
   --
   You received this message because you are subscribed to the Google
   Groups Google App Engine group.
   To post to this group, send email to
 google-appengine@googlegroups.com.
   To unsubscribe from this group, send email to
   google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
   --
   You received this message because you are subscribed to the Google
 Groups
   Google App Engine group.
   To post to this group, send email to
 google-appengine@googlegroups.com.
   To unsubscribe from this group, send email to
   google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
   --
   -Nischal
   twitter: NischalShetty http://twitter.com/nischalshetty
   facebook: Nischal http://facebook.com/nischal
 
   --
You received this message because you are subscribed to the Google
   Groups Google App Engine group.
   To post to this group, send email to
 google-appengine@googlegroups.com.
   To unsubscribe from this group, send email to
   google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
--
   You received this message because you are subscribed to the Google
 Groups
   Google App Engine group.
   To post to this group, send email to google-appengine@googlegroups.com
 .
   To unsubscribe from this group, send email to
   google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
  --http://about.me/david.mora

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Tasks are not executing!

[Nischal Shetty]

@David It's a gae problem for sure. As I write I can see the queues have
started clearing again ( yay :) ) It's been happening a number of times
today. Hoping someone from the GAE team updates this thread when they get
time. I'm guessing they are all tied up fixing various issues.

On 6 May 2011 00:51, David Mora dla.m...@gmail.com wrote:

 hey guys,

 We are not experiencing this problem and we execute like 15k+ task per
 minute probably.

 One tip tho, might worth to mention:

 The ones staying in the queue have 0 retry attempts? Retries uses a push
 back mechanism so if they fail they are kept on the queue if your
 configuration is not suitable to handle this case


 On 5 May 2011 13:12, nischalshetty nischalshett...@gmail.com wrote:

 Tasks are not executing again! Close to 30K pending tasks in my queue,
 help!

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




 --
 http://about.me/david.mora

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Tasks are not executing!

[Nischal Shetty]

I'm on Master/Slave. Don't tell me Task Queue's would also behave different
ways for different data stores :(

On 6 May 2011 01:01, Stephen Johnson onepagewo...@gmail.com wrote:

 Just to add some more insight hopefully. I'm still developing my
 application and have moved to HR datastore but kept my Master/Slave test
 running. Past couple of days Master/Slave has exhausted Task Queue of
 100,000 but HR is working fine. Typical Task Queue would probably be 5000
 for the day. Already Master/Slave has exhausted quota. For those having task
 queue problems are you on HR or Master/Slave?

 On Thu, May 5, 2011 at 12:21 PM, David Mora dla.m...@gmail.com wrote:

 hey guys,

 We are not experiencing this problem and we execute like 15k+ task per
 minute probably.

 One tip tho, might worth to mention:

 The ones staying in the queue have 0 retry attempts? Retries uses a push
 back mechanism so if they fail they are kept on the queue if your
 configuration is not suitable to handle this case


 On 5 May 2011 13:12, nischalshetty nischalshett...@gmail.com wrote:

 Tasks are not executing again! Close to 30K pending tasks in my queue,
 help!

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




 --
 http://about.me/david.mora

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Appropriate way to save hundreds of thousands of ids per user

[Nischal Shetty]

It's a simple twitter app http://www.justunfollow.com

On 30 April 2011 20:13, Brandon Donnelson branflake2...@gmail.com wrote:

 Whats your app?

 Brandon

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Are we doing it right with the label implementaion?

[Nischal Shetty]

I started allocating negative numbers as ids to my entities(these would be
part of the key) and it seems to work.

The appengine returns results sorted in the ascending order of keys and
hence if I provide keys as -8, -9, -10 etc., I would get the results as -10,
-9, -8 and so on which suffices my use case :)



On 30 April 2011 22:53, Robert Kluin robert.kl...@gmail.com wrote:

 Hey Nischal,
   You could insert some records with known key names in various
 orders and a common value for a property to query on.  If the results
 come back out key-ordered, that would be a good indication that the
 method might work.  Also, I *think* there are some videos, groups
 posts, and references in the docs (user or code) mentioning entities
 in key order.



 Robert




 On Sat, Apr 23, 2011 at 23:59, Nischal Shetty nischalshett...@gmail.com
 wrote:
  @Robert Thank you so much! Any idea how I can confirm on the timestamp +
 key
  thingy? I always need it based on timestamp (descending). If that works
 then
  it would help me a lot!
  I have looked into exploding indexes. My query would always be equality
  checks over list properties exclusively. I'm hoping that would not be a
  problem?
  Example:
  select from post
  where listproperty := label1  listproperty := label2
  sortby time desc
  This is what we currently have. If the timestamp thingy is true then we
  would be able to remove the last line and that would be so damn cool :)
  On 24 April 2011 08:24, Robert Kluin robert.kl...@gmail.com wrote:
 
  You probably need to be aware of exploding indexes if you're indexing
  a list property.  It is just something you should consider and be
  aware of.
 
 
 http://code.google.com/appengine/docs/python/datastore/queries.html#Big_Entities_and_Exploding_Indexes
 
  If you're always planning to sort by timestamp, you could consider
  prepending the timestamp on the Post entity keys.  Your entities
  should then be returned to the app sorted by timestamp without
  explicitly specifying the sort order.  At least that is the case if I
  recall correctly.
 
 
  Robert
 
 
 
 
  On Sat, Apr 23, 2011 at 13:14, nischalshetty nischalshett...@gmail.com
 
  wrote:
   We're building a new web product.
   1. We have the concept of label in them.
   2. Each post gets one or more label.
   3. The posts have to be displayed sorted according to timestamp in
   descending order
   We're making use of ListString (ListProperty) to save these labels
 for
   each post. It's working well for us. We can now query over multiple
   labels
   and retrieve posts.
   However, we need the posts to be sorted on timestamp in descending
   order.
   Due to this our query would have the timestamp as the sort order along
   with
   the label names. Due to this, we need indexes. And the indexes are
 being
   built for combination of the labels and timestamp.
   So, if our query has 2 labels and the sort order then the index would
 be
   built for these. If we query for 3 labels and sort order we would have
 a
   new
   index and so on.
   I wanted to know if there is any other way such that we can avoid the
   indexes? Are indexes cool if you anticipate a LOT of data everyday? Is
   there
   a way to have the posts sorted based on timestamp in a different way?
  
  
   --
   You received this message because you are subscribed to the Google
   Groups
   Google App Engine group.
   To post to this group, send email to
 google-appengine@googlegroups.com.
   To unsubscribe from this group, send email to
   google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group at
   http://groups.google.com/group/google-appengine?hl=en.
  
 
  --
  You received this message because you are subscribed to the Google
 Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
 
 
 
 
 
 
 
 
  --
  You received this message because you are subscribed to the Google Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

-- 
You received this message because you are subscribed

Re: [google-appengine] Appropriate way to save hundreds of thousands of ids per user

[Nischal Shetty]

But my use case is a little different. It's not so much about displaying
data as much it is about performing operations on your follower and
following list to find deltas between the two.

I have these two use cases that need to be satisfied :

1. 80% of the time, I would need to see if a set of say 100 ids is present
among the user's follower ids that are saved in my datastore
2. 20% of the time, I would need to pull all the follower ids and
following ids and find the delta of the two (basically find the
intersection)

Relation index wouldn't help me with case 1, would it?

And case 2 would be fast if I saved the ids in a list but case1 fails.

Any thoughts on the best thing for this would help. At the moment, from all
the help that I've been getting from all the wonderful people on this
mailing list has made me think of this approach :

1. Save the hundreds of thousands of follower ids in sharded lists / array
(more inclined towards array due to less size and I can save lot more ids
than I can in lists so less sharding)
2. Have a bloom filter and use it to satisfy case 1 that is to search for a
set of 100 ids and see if they are present in the datastore

This seems good, however, bloom filters give false positives. I'm not very
particular about them if the error rate is under 10%, but I've not been able
to figure out if that's the case with regards to bloom filters when I have
users with 200,000+ ids.


On 29 April 2011 10:32, Brandon Wirtz drak...@digerat.com wrote:

 A really small font.





 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *Nick Johnson (Google)
 *Sent:* Thursday, April 28, 2011 9:36 PM

 *To:* google-appengine@googlegroups.com
 *Subject:* Re: [google-appengine] Appropriate way to save hundreds of
 thousands of ids per user



 The question you need to ask, then, is how you want to display your data.
 You'll notice twitter doesn't provide Stephen Fry with a page that shows all
 million followers all at once; presumably you won't try and do that either
 (since it'd be totally unmanageable). Structure your data so you can fetch
 it in similar units to the way it'll be displayed.



 For followers, that probably means either a relation entity, or the
 relation index pattern.



 -Nick Johnson

 On Thu, Apr 28, 2011 at 1:44 PM, Nischal Shetty nischalshett...@gmail.com
 wrote:

 I have users with more than half a million followers using my app. My own
 app account has 140k followers. I personally know a few of my app users who
 have more than 200,000 followers :)





 On 28 April 2011 08:44, Brandon Wirtz drak...@digerat.com wrote:

 Why not?  My two biggest projects have 180k and 90k friends.



 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *Nick Johnson (Google)
 *Sent:* Wednesday, April 27, 2011 7:40 PM


 *To:* google-appengine@googlegroups.com
 *Subject:* Re: [google-appengine] Appropriate way to save hundreds of
 thousands of ids per user



 Hi David,



 Can you elaborate on your exact use-case? You mentioned twitter friends,
 but I'm fairly sure no users have 200,000 friends on Twitter.



 -Nick Johnson

 On Mon, Apr 25, 2011 at 2:54 PM, David Parks davidpark...@yahoo.com
 wrote:

 I did indeed mean pulling back a result set of say 200,000 rows. If I’m
 following the conversation correctly then what you described was storing all
 IDs, querying that one field and de-serializing all IDs into an array that
 you can then search for the ID’s you need.



 I like that idea. But I certainly can’t tell you if the overhead of reading
 all values, and deserializing them will be better or worse than the overhead
 of scrolling through a large result set and loading the database with
 hundreds of millions of rows. Of all databases you could be using, googles
 big table is certainly well designed for large data sets.



 It seems that your proposed method makes great sense when you need the
 entire result set (or close to it) for one or more users. But when you only
 need 100 results of 150,000, then the deserialization process is going to
 constitute a measurable overhead. Also, I can’t say for sure how the google
 datastore will  perform when you commit hundreds of millions of rows to it.
 Of course, if small queries like are rare, then maybe it’s not so important
 to consider them.



 Anyway, I guess you could write, in perhaps a day or less, a very simple
 test case that populate the datastore with both scenarios and profile them.



 Doing the profiling work will probably give you some very useful insight
 and experience on how things will really perform in reality.



 I’d also suggest that you encapsulate this functionality so that you can
 easily replace one strategy with another without changing code unrelated to
 the data store (e.g. design your code using proper data access objects to
 keep this code separate from the rest of your code, and code to interfaces
 up front

Re: [google-appengine] Re: Appropriate way to save hundreds of thousands of ids per user

[Nischal Shetty]

@Brandon I wish he ends up using my app, that would probably make me rich :)
 Thanks for the input. I'm more for the breaking up into chunks approach,
let's see how that goes.

@Siegfried That video has been my guide to the appengine from quite some
time now. Though I dont remember about the list part. Will go through it
again and see what he has to say about it.




On 28 April 2011 21:16, Siegfried Hirsch siegfried.hir...@gmail.com wrote:

 Guesss it is helpful to view this video from Brett Slatkin about lists
 for a scalable application.
 He is just talking about the 5000 limit of lists and how to overcome
 this limitation.

 http://www.youtube.com/watch?v=AgaL6NGpkB8

 Siegfried

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Appropriate way to save hundreds of thousands of ids per user

[Nischal Shetty]

@Nick

1) the 1000 entities (rows) limit has been lifted long time ago.


I thought by lifting the limit it meant I could go ahead and fetech 1001-
2000 using a cursor. So I guess, it means pulling more than 1000 rows at a
time, stupid me :)

2) tasks are not limited by the 30s limit - can run for 10 minutes.


We provide URLs that would be called when the task executes. Those would
stop in  30s right? So, what exactly is this 10 minute limit, I haven't been
able to wrap my head around the 10 minute thingy.


On 26 April 2011 00:58, nickmilon nickmi...@gmail.com wrote:

 1) the 1000 entities (rows) limit has been lifted long time ago.
 2) tasks are not limited by the 30s limit - can run for 10 minutes.

 Happy coding ;-)
 Nick
 On Apr 25, 9:01 am, Nischal Shetty nischalshett...@gmail.com wrote:
  I will indeed try a few ways to do this. But pulling all rows
 individually
  would be an overkill because every query gives us 1000 rows at a time
 which
  means I would hit the 30s limit while I'm at it :(
 
  For searching the IDs that I have at hand, I would not need to
 deserialize
  the array of ids. I would be making use of Bloom Filter which I think
 would
  speed things up. I would need to deserialize all the ids occasionally for
  some rare computational purposes.
 
  So my use case would consist  80% search a bunch of IDs and 20%
 deserialize
  all the IDs.
 
  On 25 April 2011 10:24, David Parks davidpark...@yahoo.com wrote:
 
 
 
 
 
 
 
   I did indeed mean pulling back a result set of say 200,000 rows. If I’m
   following the conversation correctly then what you described was
 storing all
   IDs, querying that one field and de-serializing all IDs into an array
 that
   you can then search for the ID’s you need.
 
   I like that idea. But I certainly can’t tell you if the overhead of
 reading
   all values, and deserializing them will be better or worse than the
 overhead
   of scrolling through a large result set and loading the database with
   hundreds of millions of rows. Of all databases you could be using,
 googles
   big table is certainly well designed for large data sets.
 
   It seems that your proposed method makes great sense when you need the
   entire result set (or close to it) for one or more users. But when you
 only
   need 100 results of 150,000, then the deserialization process is going
 to
   constitute a measurable overhead. Also, I can’t say for sure how the
 google
   datastore will  perform when you commit hundreds of millions of rows to
 it.
   Of course, if small queries like are rare, then maybe it’s not so
 important
   to consider them.
 
   Anyway, I guess you could write, in perhaps a day or less, a very
 simple
   test case that populate the datastore with both scenarios and profile
 them.
 
   Doing the profiling work will probably give you some very useful
 insight
   and experience on how things will really perform in reality.
 
   I’d also suggest that you encapsulate this functionality so that you
 can
   easily replace one strategy with another without changing code
 unrelated to
   the data store (e.g. design your code using proper data access objects
 to
   keep this code separate from the rest of your code, and code to
 interfaces
   up front).
 
   *From:* google-appengine@googlegroups.com [mailto:
   google-appengine@googlegroups.com] *On Behalf Of *Nischal Shetty
   *Sent:* Monday, April 25, 2011 10:34 AM
 
   *To:* google-appengine@googlegroups.com
   *Subject:* Re: [google-appengine] Appropriate way to save hundreds of
   thousands of ids per user
 
   @David
 
   Querying the whole group would mean having 200,000 results for few of
 my
   users. Pulling all that and then searching, wouldn't that be
 inefficient? or
   are you talking about sharded ListProperty here?
 
   On 25 April 2011 05:41, David Parks davidpark...@yahoo.com wrote:
 
   That seems like a reasonable approach. But I think you should do both
   tests. 1) let google do the work and store a lot of records, 2) query
 the
   whole group and parse it into an array and search the array. It
 wouldn’t be
   too hard to created a simple test case that populates the data for
 whatever
   # of users you need to plan for and profile the lookup and storage
 speeds of
   both.
 
   I’d love to know your results if you do test both approaches.
 
   *From:* google-appengine@googlegroups.com [mailto:
   google-appengine@googlegroups.com] *On Behalf Of *Nischal Shetty
   *Sent:* Friday, April 22, 2011 3:10 PM
 
   *To:* google-appengine@googlegroups.com
 
   *Subject:* Re: [google-appengine] Appropriate way to save hundreds of
   thousands of ids per user
 
   @David
 
   Thanks for the input. Every reply gives me some more insight into how I
   achieve this. My use case is as below :
 
   1. At times I would need all the IDs at the same time in memory
 
   2. Most of the times I would need to check if a set of IDs as input by
 the
   user (say 100 IDs) are present in the datastore
 
   I've been

Re: [google-appengine] Appropriate way to save hundreds of thousands of ids per user

[Nischal Shetty]

I have users with more than half a million followers using my app. My own
app account has 140k followers. I personally know a few of my app users who
have more than 200,000 followers :)



On 28 April 2011 08:44, Brandon Wirtz drak...@digerat.com wrote:

 Why not?  My two biggest projects have 180k and 90k friends.



 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *Nick Johnson (Google)
 *Sent:* Wednesday, April 27, 2011 7:40 PM

 *To:* google-appengine@googlegroups.com
 *Subject:* Re: [google-appengine] Appropriate way to save hundreds of
 thousands of ids per user



 Hi David,



 Can you elaborate on your exact use-case? You mentioned twitter friends,
 but I'm fairly sure no users have 200,000 friends on Twitter.



 -Nick Johnson

 On Mon, Apr 25, 2011 at 2:54 PM, David Parks davidpark...@yahoo.com
 wrote:

 I did indeed mean pulling back a result set of say 200,000 rows. If I’m
 following the conversation correctly then what you described was storing all
 IDs, querying that one field and de-serializing all IDs into an array that
 you can then search for the ID’s you need.



 I like that idea. But I certainly can’t tell you if the overhead of reading
 all values, and deserializing them will be better or worse than the overhead
 of scrolling through a large result set and loading the database with
 hundreds of millions of rows. Of all databases you could be using, googles
 big table is certainly well designed for large data sets.



 It seems that your proposed method makes great sense when you need the
 entire result set (or close to it) for one or more users. But when you only
 need 100 results of 150,000, then the deserialization process is going to
 constitute a measurable overhead. Also, I can’t say for sure how the google
 datastore will  perform when you commit hundreds of millions of rows to it.
 Of course, if small queries like are rare, then maybe it’s not so important
 to consider them.



 Anyway, I guess you could write, in perhaps a day or less, a very simple
 test case that populate the datastore with both scenarios and profile them.



 Doing the profiling work will probably give you some very useful insight
 and experience on how things will really perform in reality.



 I’d also suggest that you encapsulate this functionality so that you can
 easily replace one strategy with another without changing code unrelated to
 the data store (e.g. design your code using proper data access objects to
 keep this code separate from the rest of your code, and code to interfaces
 up front).







 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *Nischal Shetty
 *Sent:* Monday, April 25, 2011 10:34 AM


 *To:* google-appengine@googlegroups.com
 *Subject:* Re: [google-appengine] Appropriate way to save hundreds of
 thousands of ids per user



 @David



 Querying the whole group would mean having 200,000 results for few of my
 users. Pulling all that and then searching, wouldn't that be inefficient? or
 are you talking about sharded ListProperty here?







 On 25 April 2011 05:41, David Parks davidpark...@yahoo.com wrote:

 That seems like a reasonable approach. But I think you should do both
 tests. 1) let google do the work and store a lot of records, 2) query the
 whole group and parse it into an array and search the array. It wouldn’t be
 too hard to created a simple test case that populates the data for whatever
 # of users you need to plan for and profile the lookup and storage speeds of
 both.



 I’d love to know your results if you do test both approaches.





 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *Nischal Shetty
 *Sent:* Friday, April 22, 2011 3:10 PM


 *To:* google-appengine@googlegroups.com

 *Subject:* Re: [google-appengine] Appropriate way to save hundreds of
 thousands of ids per user



 @David



 Thanks for the input. Every reply gives me some more insight into how I
 achieve this. My use case is as below :



 1. At times I would need all the IDs at the same time in memory

 2. Most of the times I would need to check if a set of IDs as input by the
 user (say 100 IDs) are present in the datastore



 I've been thinking of doing the following :



 1. Persisting all the IDs by putting them into an array (I will probably
 have shards where each array would hold 50k IDs)

 2. Implementing a bloom filter to search for the set of IDs if they exist
 in the datastore.





 On 22 April 2011 09:34, David Parks davidpark...@yahoo.com wrote:

 I don’t know your intended use of these ID’s, my thoughts here are limited
 to assumed use, feel free to ignore thoughts that are off base for your use
 case.



 If, when you query for the IDs you are looking for **all** the IDs, then
 just serialize them into one field and retrieve them as one record and
 de-serialize them in a way that doesn’t require they all fit

Re: [google-appengine] Appropriate way to save hundreds of thousands of ids per user

[Nischal Shetty]

I will indeed try a few ways to do this. But pulling all rows individually
would be an overkill because every query gives us 1000 rows at a time which
means I would hit the 30s limit while I'm at it :(

For searching the IDs that I have at hand, I would not need to deserialize
the array of ids. I would be making use of Bloom Filter which I think would
speed things up. I would need to deserialize all the ids occasionally for
some rare computational purposes.

So my use case would consist  80% search a bunch of IDs and 20% deserialize
all the IDs.


On 25 April 2011 10:24, David Parks davidpark...@yahoo.com wrote:

 I did indeed mean pulling back a result set of say 200,000 rows. If I’m
 following the conversation correctly then what you described was storing all
 IDs, querying that one field and de-serializing all IDs into an array that
 you can then search for the ID’s you need.



 I like that idea. But I certainly can’t tell you if the overhead of reading
 all values, and deserializing them will be better or worse than the overhead
 of scrolling through a large result set and loading the database with
 hundreds of millions of rows. Of all databases you could be using, googles
 big table is certainly well designed for large data sets.



 It seems that your proposed method makes great sense when you need the
 entire result set (or close to it) for one or more users. But when you only
 need 100 results of 150,000, then the deserialization process is going to
 constitute a measurable overhead. Also, I can’t say for sure how the google
 datastore will  perform when you commit hundreds of millions of rows to it.
 Of course, if small queries like are rare, then maybe it’s not so important
 to consider them.



 Anyway, I guess you could write, in perhaps a day or less, a very simple
 test case that populate the datastore with both scenarios and profile them.



 Doing the profiling work will probably give you some very useful insight
 and experience on how things will really perform in reality.



 I’d also suggest that you encapsulate this functionality so that you can
 easily replace one strategy with another without changing code unrelated to
 the data store (e.g. design your code using proper data access objects to
 keep this code separate from the rest of your code, and code to interfaces
 up front).







 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *Nischal Shetty
 *Sent:* Monday, April 25, 2011 10:34 AM

 *To:* google-appengine@googlegroups.com
 *Subject:* Re: [google-appengine] Appropriate way to save hundreds of
 thousands of ids per user



 @David



 Querying the whole group would mean having 200,000 results for few of my
 users. Pulling all that and then searching, wouldn't that be inefficient? or
 are you talking about sharded ListProperty here?







 On 25 April 2011 05:41, David Parks davidpark...@yahoo.com wrote:

 That seems like a reasonable approach. But I think you should do both
 tests. 1) let google do the work and store a lot of records, 2) query the
 whole group and parse it into an array and search the array. It wouldn’t be
 too hard to created a simple test case that populates the data for whatever
 # of users you need to plan for and profile the lookup and storage speeds of
 both.



 I’d love to know your results if you do test both approaches.





 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *Nischal Shetty
 *Sent:* Friday, April 22, 2011 3:10 PM


 *To:* google-appengine@googlegroups.com

 *Subject:* Re: [google-appengine] Appropriate way to save hundreds of
 thousands of ids per user



 @David



 Thanks for the input. Every reply gives me some more insight into how I
 achieve this. My use case is as below :



 1. At times I would need all the IDs at the same time in memory

 2. Most of the times I would need to check if a set of IDs as input by the
 user (say 100 IDs) are present in the datastore



 I've been thinking of doing the following :



 1. Persisting all the IDs by putting them into an array (I will probably
 have shards where each array would hold 50k IDs)

 2. Implementing a bloom filter to search for the set of IDs if they exist
 in the datastore.





 On 22 April 2011 09:34, David Parks davidpark...@yahoo.com wrote:

 I don’t know your intended use of these ID’s, my thoughts here are limited
 to assumed use, feel free to ignore thoughts that are off base for your use
 case.



 If, when you query for the IDs you are looking for **all** the IDs, then
 just serialize them into one field and retrieve them as one record and
 de-serialize them in a way that doesn’t require they all fit into memory at
 the same time (a tokenized CSV list is most straight forward example, but
 you can do more compact serializations).



 If you need to query for some subset of these IDs, then storing them in the
 datastore is indeed the way to go I suspect. You can

Re: [google-appengine] Appropriate way to save hundreds of thousands of ids per user

[Nischal Shetty]

@David

Querying the whole group would mean having 200,000 results for few of my
users. Pulling all that and then searching, wouldn't that be inefficient? or
are you talking about sharded ListProperty here?



On 25 April 2011 05:41, David Parks davidpark...@yahoo.com wrote:

 That seems like a reasonable approach. But I think you should do both
 tests. 1) let google do the work and store a lot of records, 2) query the
 whole group and parse it into an array and search the array. It wouldn’t be
 too hard to created a simple test case that populates the data for whatever
 # of users you need to plan for and profile the lookup and storage speeds of
 both.



 I’d love to know your results if you do test both approaches.





 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *Nischal Shetty
 *Sent:* Friday, April 22, 2011 3:10 PM

 *To:* google-appengine@googlegroups.com
 *Subject:* Re: [google-appengine] Appropriate way to save hundreds of
 thousands of ids per user



 @David



 Thanks for the input. Every reply gives me some more insight into how I
 achieve this. My use case is as below :



 1. At times I would need all the IDs at the same time in memory

 2. Most of the times I would need to check if a set of IDs as input by the
 user (say 100 IDs) are present in the datastore



 I've been thinking of doing the following :



 1. Persisting all the IDs by putting them into an array (I will probably
 have shards where each array would hold 50k IDs)

 2. Implementing a bloom filter to search for the set of IDs if they exist
 in the datastore.





 On 22 April 2011 09:34, David Parks davidpark...@yahoo.com wrote:

 I don’t know your intended use of these ID’s, my thoughts here are limited
 to assumed use, feel free to ignore thoughts that are off base for your use
 case.



 If, when you query for the IDs you are looking for **all** the IDs, then
 just serialize them into one field and retrieve them as one record and
 de-serialize them in a way that doesn’t require they all fit into memory at
 the same time (a tokenized CSV list is most straight forward example, but
 you can do more compact serializations).



 If you need to query for some subset of these IDs, then storing them in the
 datastore is indeed the way to go I suspect. You can batch many
 inserts/updates. You’ll have a large table, but that isn’t likely to be a
 problem with this data store, but do test it. If lookup times degrade with
 size you could consider partitioning your users into different groups
 (simple example: 1 group of users IDs that end in even #’s, another that
 ends in odd #’s), this can reduce the size of indexes and improve
 performance on some systems (I don’t have personal experience to tell you
 whether this is necessary in this system, but it’s a thought to consider).



 Again, I just offer this as food for thought. If you describe your intended
 access patterns it will probably help guide the discussion. Good luck.





 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *nischalshetty
 *Sent:* Tuesday, April 19, 2011 1:15 PM
 *To:* google-appengine@googlegroups.com
 *Subject:* [google-appengine] Appropriate way to save hundreds of
 thousands of ids per user



 Every user in my app would have thousands of ids corresponding to them. I
 would need to look up these ids often.

 Two things I could think of:

 1. Put them into Lists - (drawback is that lists have a maximum capacity of
 5000(hope I'm right here) and I have users who would need to save more than
 150,000 ids)
 2. Insert each id as a unique record in the datastore (too much of data? as
 it would be user * ids of all users). Can I batch put 5000 records at a
 time? Can I batch get at least 100 - 500 records at a time?

 Is there any other way to do this? I hope my question's clear. Your
 suggestions are greatly appreciated.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.
 --

 No virus found in this message.
 Checked by AVG - www.avg.com
 Version: 10.0.1209 / Virus Database: 1500/3582 - Release Date: 04/18/11

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




 --
 -Nischal

 +91-9920240474

 twitter: NischalShetty http://twitter.com/nischalshetty

 facebook: Nischal http://facebook.com/nischal

Re: [google-appengine] Are we doing it right with the label implementaion?

[Nischal Shetty]

@Robert Thank you so much! Any idea how I can confirm on the timestamp + key
thingy? I always need it based on timestamp (descending). If that works then
it would help me a lot!

I have looked into exploding indexes. My query would always be equality
checks over list properties exclusively. I'm hoping that would not be a
problem?

Example:
select from post
where listproperty := label1  listproperty := label2
sortby time desc

This is what we currently have. If the timestamp thingy is true then we
would be able to remove the last line and that would be so damn cool :)

On 24 April 2011 08:24, Robert Kluin robert.kl...@gmail.com wrote:

 You probably need to be aware of exploding indexes if you're indexing
 a list property.  It is just something you should consider and be
 aware of.

 http://code.google.com/appengine/docs/python/datastore/queries.html#Big_Entities_and_Exploding_Indexes

 If you're always planning to sort by timestamp, you could consider
 prepending the timestamp on the Post entity keys.  Your entities
 should then be returned to the app sorted by timestamp without
 explicitly specifying the sort order.  At least that is the case if I
 recall correctly.


 Robert




 On Sat, Apr 23, 2011 at 13:14, nischalshetty nischalshett...@gmail.com
 wrote:
  We're building a new web product.
  1. We have the concept of label in them.
  2. Each post gets one or more label.
  3. The posts have to be displayed sorted according to timestamp in
  descending order
  We're making use of ListString (ListProperty) to save these labels for
  each post. It's working well for us. We can now query over multiple
 labels
  and retrieve posts.
  However, we need the posts to be sorted on timestamp in descending order.
  Due to this our query would have the timestamp as the sort order along
 with
  the label names. Due to this, we need indexes. And the indexes are being
  built for combination of the labels and timestamp.
  So, if our query has 2 labels and the sort order then the index would be
  built for these. If we query for 3 labels and sort order we would have a
 new
  index and so on.
  I wanted to know if there is any other way such that we can avoid the
  indexes? Are indexes cool if you anticipate a LOT of data everyday? Is
 there
  a way to have the posts sorted based on timestamp in a different way?
 
 
  --
  You received this message because you are subscribed to the Google Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Appropriate way to save hundreds of thousands of ids per user

[Nischal Shetty]

@David

Thanks for the input. Every reply gives me some more insight into how I
achieve this. My use case is as below :

1. At times I would need all the IDs at the same time in memory
2. Most of the times I would need to check if a set of IDs as input by the
user (say 100 IDs) are present in the datastore

I've been thinking of doing the following :

1. Persisting all the IDs by putting them into an array (I will probably
have shards where each array would hold 50k IDs)
2. Implementing a bloom filter to search for the set of IDs if they exist in
the datastore.



On 22 April 2011 09:34, David Parks davidpark...@yahoo.com wrote:

 I don’t know your intended use of these ID’s, my thoughts here are limited
 to assumed use, feel free to ignore thoughts that are off base for your use
 case.



 If, when you query for the IDs you are looking for **all** the IDs, then
 just serialize them into one field and retrieve them as one record and
 de-serialize them in a way that doesn’t require they all fit into memory at
 the same time (a tokenized CSV list is most straight forward example, but
 you can do more compact serializations).



 If you need to query for some subset of these IDs, then storing them in the
 datastore is indeed the way to go I suspect. You can batch many
 inserts/updates. You’ll have a large table, but that isn’t likely to be a
 problem with this data store, but do test it. If lookup times degrade with
 size you could consider partitioning your users into different groups
 (simple example: 1 group of users IDs that end in even #’s, another that
 ends in odd #’s), this can reduce the size of indexes and improve
 performance on some systems (I don’t have personal experience to tell you
 whether this is necessary in this system, but it’s a thought to consider).



 Again, I just offer this as food for thought. If you describe your intended
 access patterns it will probably help guide the discussion. Good luck.





 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *nischalshetty
 *Sent:* Tuesday, April 19, 2011 1:15 PM
 *To:* google-appengine@googlegroups.com
 *Subject:* [google-appengine] Appropriate way to save hundreds of
 thousands of ids per user



 Every user in my app would have thousands of ids corresponding to them. I
 would need to look up these ids often.

 Two things I could think of:

 1. Put them into Lists - (drawback is that lists have a maximum capacity of
 5000(hope I'm right here) and I have users who would need to save more than
 150,000 ids)
 2. Insert each id as a unique record in the datastore (too much of data? as
 it would be user * ids of all users). Can I batch put 5000 records at a
 time? Can I batch get at least 100 - 500 records at a time?

 Is there any other way to do this? I hope my question's clear. Your
 suggestions are greatly appreciated.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.
 --

 No virus found in this message.
 Checked by AVG - www.avg.com
 Version: 10.0.1209 / Virus Database: 1500/3582 - Release Date: 04/18/11

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
+91-9920240474
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [appengine-java] Re: Which one is best for development...Spring Framework or Servlet Jsp....

[Nischal Shetty]

Sorry, haven't worked with GWT, hope someone helps you.

-N

On 19 April 2011 12:19, Ram ram...@gmail.com wrote:

 Hi,

  Thank you..Also I have another doubt, how to use GWT desiner in my
 Eclipse.In documents no continuity is therewhere i want to
 open GWT designer using eclipse.

 please help me..

 On Apr 19, 11:18 am, Nischal nischalshett...@gmail.com wrote:
  I'm using struts 2 and it seems good for development. However, there are
  other lighter frameworks that appengine users have spoken positively
 about.
  You should check out this thread -
 https://groups.google.com/forum/#!topic/google-appengine-java/7wkMDm6...

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine for Java group.
 To post to this group, send email to
 google-appengine-java@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine-java+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine-java?hl=en.




-- 
-Nischal
+91-9920240474
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

Re: [google-appengine] Being rate limited at twitter due to some rogue app on GAE, can u shift my app to different ip range?

[Nischal Shetty]

Haha, right now as a quick way to get my app back on it's feet I'm doing the
exact same thing!

-N

On 8 April 2011 12:13, Brandon Wirtz drak...@digerat.com wrote:

 I cheated horribly… My app calls a PHP file on another server and the PHP
 file does a Curl.  Was really easy to setup, because I just used the Twitter
 API PHP example, and added some security on the PHP so that only Appengine
 could hit it, and only if it new the rotating secret.



 -brandon



 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *Nischal Shetty
 *Sent:* Thursday, April 07, 2011 10:42 PM

 *To:* google-appengine@googlegroups.com
 *Subject:* Re: [google-appengine] Being rate limited at twitter due to
 some rogue app on GAE, can u shift my app to different ip range?



 Sigh :( Long day ahead for me. What proxy did you use? Have you documented
 your experience (of setting up a proxy) somewhere? Can you point me to
 something I can use?



 -N





 On 8 April 2011 11:08, Brandon Wirtz drak...@digerat.com wrote:

 Been there, argued with Ev about why this was worth having someone fix,
 resolved issue using a proxy. Down side is the proxy will likely cost you by
 the gig transferred and you have to pay the appengine price for traffic as
 well, so your costs go up, but it resolves the issue.  Plus in a world where
 IP’s are now in limited supply buying an IP somewhere is worth doing if you
 rely on API’s from others.







 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *Nischal Shetty
 *Sent:* Thursday, April 07, 2011 10:33 PM
 *To:* google-appengine@googlegroups.com
 *Subject:* Re: [google-appengine] Being rate limited at twitter due to
 some rogue app on GAE, can u shift my app to different ip range?



 Yes, they aren't whitelisting apps based on id and that sucks. I'm not
 using the search API, I'm making use of authenticated API calls against each
 user and it has been working well (my app makes thousands of reqs per hour).



 There actually is no rate limit on the calls I make (they are posts not get
 and twitter puts limits on gets). But, the twitter API team said that at
 times when there are excessive requests, they block these calls as well.



 I'm guessing there's some other app that is making a lot of post requests
 that are similar to mine. I don't know how to get over this. Right now I'm
 trying to move these calls over to some other server, been working on it.



 But, I love the appengine, I want to be here :( Is there no way you can
 make my app use some other ip range, that might help.



 -N



 On 8 April 2011 10:17, Nick Johnson (Google) nick.john...@google.com
 wrote:

 Hi Nischal,



 Are you using the search API, or one of the other APIs? The Search API
 doesn't require authentication, and so is limited by IP range. The most
 likely explanation is not a single badly behaving app - it's simply that
 there are a lot of App Engine apps that want to communicate with Twitter.
 This is a problem with any service that doesn't have dedicated IPs for each
 customer - and that's only going to get worse with the exhaustion of the
 IPv4 address space, unfortunately.



 We provide the App ID in the referer header so services like Twitter can
 distinguish requests from different apps for the purpose of ratelimiting and
 abuse detection, but Twitter haven't implemented anything to break apps out
 on this basis.



 -Nick Johnson



 On Fri, Apr 8, 2011 at 11:33 AM, nischalshetty nischalshett...@gmail.com
 wrote:

 Hi,

 My app has been facing limit issues with the twitter API and under
 regular circumstances this should not have happened. I've been in
 talks with the twitter API team from the last 36 hours and they
 checked things on their end. They say that the most probable cause is
 some app on the same ip range as my app which might be making a lot of
 requests causing my app to be rate limited in the process.

 I don't know what a quick solution to this is but I might have to move
 away from appengine because my app is unusable right now. My app has
 been running from the last 1 year on the appengine and has 200k+
 users. I need your help :(

 My app id is justunfollow . Is there any way you can move it to a
 different ip range?


 -Nischal

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




 --

 Nick Johnson, Developer Programs Engineer, App Engine

 --

 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine

Re: [google-appengine] Being rate limited at twitter due to some rogue app on GAE, can u shift my app to different ip range?

[Nischal Shetty]

Yes, they aren't whitelisting apps based on id and that sucks. I'm not using
the search API, I'm making use of authenticated API calls against each user
and it has been working well (my app makes thousands of reqs per hour).

There actually is no rate limit on the calls I make (they are posts not get
and twitter puts limits on gets). But, the twitter API team said that at
times when there are excessive requests, they block these calls as well.

I'm guessing there's some other app that is making a lot of post requests
that are similar to mine. I don't know how to get over this. Right now I'm
trying to move these calls over to some other server, been working on it.

But, I love the appengine, I want to be here :( Is there no way you can make
my app use some other ip range, that might help.

-N

On 8 April 2011 10:17, Nick Johnson (Google) nick.john...@google.comwrote:

 Hi Nischal,

 Are you using the search API, or one of the other APIs? The Search API
 doesn't require authentication, and so is limited by IP range. The most
 likely explanation is not a single badly behaving app - it's simply that
 there are a lot of App Engine apps that want to communicate with Twitter.
 This is a problem with any service that doesn't have dedicated IPs for each
 customer - and that's only going to get worse with the exhaustion of the
 IPv4 address space, unfortunately.

 We provide the App ID in the referer header so services like Twitter can
 distinguish requests from different apps for the purpose of ratelimiting and
 abuse detection, but Twitter haven't implemented anything to break apps out
 on this basis.


 -Nick Johnson

 On Fri, Apr 8, 2011 at 11:33 AM, nischalshetty 
 nischalshett...@gmail.comwrote:

 Hi,

 My app has been facing limit issues with the twitter API and under
 regular circumstances this should not have happened. I've been in
 talks with the twitter API team from the last 36 hours and they
 checked things on their end. They say that the most probable cause is
 some app on the same ip range as my app which might be making a lot of
 requests causing my app to be rate limited in the process.

 I don't know what a quick solution to this is but I might have to move
 away from appengine because my app is unusable right now. My app has
 been running from the last 1 year on the appengine and has 200k+
 users. I need your help :(

 My app id is justunfollow . Is there any way you can move it to a
 different ip range?


 -Nischal

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




 --
 Nick Johnson, Developer Programs Engineer, App Engine


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
+91-9920240474
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Being rate limited at twitter due to some rogue app on GAE, can u shift my app to different ip range?

[Nischal Shetty]

Sigh :( Long day ahead for me. What proxy did you use? Have you documented
your experience (of setting up a proxy) somewhere? Can you point me to
something I can use?

-N



On 8 April 2011 11:08, Brandon Wirtz drak...@digerat.com wrote:

 Been there, argued with Ev about why this was worth having someone fix,
 resolved issue using a proxy. Down side is the proxy will likely cost you by
 the gig transferred and you have to pay the appengine price for traffic as
 well, so your costs go up, but it resolves the issue.  Plus in a world where
 IP’s are now in limited supply buying an IP somewhere is worth doing if you
 rely on API’s from others.







 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *Nischal Shetty
 *Sent:* Thursday, April 07, 2011 10:33 PM
 *To:* google-appengine@googlegroups.com
 *Subject:* Re: [google-appengine] Being rate limited at twitter due to
 some rogue app on GAE, can u shift my app to different ip range?



 Yes, they aren't whitelisting apps based on id and that sucks. I'm not
 using the search API, I'm making use of authenticated API calls against each
 user and it has been working well (my app makes thousands of reqs per hour).



 There actually is no rate limit on the calls I make (they are posts not get
 and twitter puts limits on gets). But, the twitter API team said that at
 times when there are excessive requests, they block these calls as well.



 I'm guessing there's some other app that is making a lot of post requests
 that are similar to mine. I don't know how to get over this. Right now I'm
 trying to move these calls over to some other server, been working on it.



 But, I love the appengine, I want to be here :( Is there no way you can
 make my app use some other ip range, that might help.



 -N



 On 8 April 2011 10:17, Nick Johnson (Google) nick.john...@google.com
 wrote:

 Hi Nischal,



 Are you using the search API, or one of the other APIs? The Search API
 doesn't require authentication, and so is limited by IP range. The most
 likely explanation is not a single badly behaving app - it's simply that
 there are a lot of App Engine apps that want to communicate with Twitter.
 This is a problem with any service that doesn't have dedicated IPs for each
 customer - and that's only going to get worse with the exhaustion of the
 IPv4 address space, unfortunately.



 We provide the App ID in the referer header so services like Twitter can
 distinguish requests from different apps for the purpose of ratelimiting and
 abuse detection, but Twitter haven't implemented anything to break apps out
 on this basis.



 -Nick Johnson



 On Fri, Apr 8, 2011 at 11:33 AM, nischalshetty nischalshett...@gmail.com
 wrote:

 Hi,

 My app has been facing limit issues with the twitter API and under
 regular circumstances this should not have happened. I've been in
 talks with the twitter API team from the last 36 hours and they
 checked things on their end. They say that the most probable cause is
 some app on the same ip range as my app which might be making a lot of
 requests causing my app to be rate limited in the process.

 I don't know what a quick solution to this is but I might have to move
 away from appengine because my app is unusable right now. My app has
 been running from the last 1 year on the appengine and has 200k+
 users. I need your help :(

 My app id is justunfollow . Is there any way you can move it to a
 different ip range?


 -Nischal

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




 --

 Nick Johnson, Developer Programs Engineer, App Engine

 --

 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




 --
 -Nischal

 +91-9920240474

 twitter: NischalShetty http://twitter.com/nischalshetty

 facebook: Nischal http://facebook.com/nischal



 [image: Description: Image removed by sender.]http://www.justunfollow.com/





 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.

  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine

Re: [google-appengine] Many memcache hits are returning null

[Nischal Shetty]

Alright, thanks I'll keep that in mind :)

-N

On 31 March 2011 10:55, Nick Johnson (Google) nick.john...@google.comwrote:

 The limit depends on a number of factors, and isn't fixed. You shouldn't
 rely on your app having a specific amount of memcache space available -
 treat it like a cache, and expect that data may disappear from it without
 notice.

 -Nick Johnson


 On Thu, Mar 31, 2011 at 4:21 PM, Nischal Shetty nischalshett...@gmail.com
  wrote:

 @Nick

 Can you disclose what the limit is? And if there's a way it can be
 increased?

 -N


 On 31 March 2011 09:10, Nick Johnson (Google) nick.john...@google.comwrote:

 On Thu, Mar 31, 2011 at 1:34 PM, Nischal Shetty 
 nischalshett...@gmail.com wrote:

 @Nick

 I completely agree with you but I saw an elevated rate in the logs today
 which does not normally happen. Want to confirm if there's actually
 something like a memcache limit imposed on apps. I know there's a per 
 object
 size limit but is there a limit on the total data that can be cached by an
 app running on GAE?


 Yes.

 -Nick Johnson



 -N


 On 31 March 2011 07:44, Nick Johnson (Google) 
 nick.john...@google.comwrote:

 Hi,

 Memcache, as the name implies, is a cache. You can't rely on data
 existing in memcache, as it may be evicted at any time.

 -Nick Johnson


 On Thu, Mar 31, 2011 at 12:13 PM, nischalshetty 
 nischalshett...@gmail.com wrote:

 Calls to retrieve objects stored in the memcache are returning null.
 My app uses a lot of memcache and I had read some time back that there 
 is a
 certain limit on the amount of data that can be cached overall by an app.
 Can someone from the appengine team look into this?

 --
 You received this message because you are subscribed to the Google
 Groups Google App Engine group.
 To post to this group, send email to
 google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




 --
 Nick Johnson, Developer Programs Engineer, App Engine





 --
 -Nischal
 +91-9920240474
 twitter: NischalShetty http://twitter.com/nischalshetty
 facebook: Nischal http://facebook.com/nischal

 http://www.justunfollow.com





 --
 Nick Johnson, Developer Programs Engineer, App Engine


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




 --
 -Nischal
 +91-9920240474
 twitter: NischalShetty http://twitter.com/nischalshetty
 facebook: Nischal http://facebook.com/nischal

 http://www.justunfollow.com


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




 --
 Nick Johnson, Developer Programs Engineer, App Engine


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
+91-9920240474
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Many memcache hits are returning null

[Nischal Shetty]

@Nick

Can you disclose what the limit is? And if there's a way it can be
increased?

-N

On 31 March 2011 09:10, Nick Johnson (Google) nick.john...@google.comwrote:

 On Thu, Mar 31, 2011 at 1:34 PM, Nischal Shetty nischalshett...@gmail.com
  wrote:

 @Nick

 I completely agree with you but I saw an elevated rate in the logs today
 which does not normally happen. Want to confirm if there's actually
 something like a memcache limit imposed on apps. I know there's a per object
 size limit but is there a limit on the total data that can be cached by an
 app running on GAE?


 Yes.

 -Nick Johnson



 -N


 On 31 March 2011 07:44, Nick Johnson (Google) nick.john...@google.comwrote:

 Hi,

 Memcache, as the name implies, is a cache. You can't rely on data
 existing in memcache, as it may be evicted at any time.

 -Nick Johnson


 On Thu, Mar 31, 2011 at 12:13 PM, nischalshetty 
 nischalshett...@gmail.com wrote:

 Calls to retrieve objects stored in the memcache are returning null. My
 app uses a lot of memcache and I had read some time back that there is a
 certain limit on the amount of data that can be cached overall by an app.
 Can someone from the appengine team look into this?

 --
 You received this message because you are subscribed to the Google
 Groups Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




 --
 Nick Johnson, Developer Programs Engineer, App Engine





 --
 -Nischal
 +91-9920240474
 twitter: NischalShetty http://twitter.com/nischalshetty
 facebook: Nischal http://facebook.com/nischal

 http://www.justunfollow.com





 --
 Nick Johnson, Developer Programs Engineer, App Engine


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
+91-9920240474
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Request timeout to my domain name that points to app hosted on appengine

[Nischal Shetty]

@Nick

Clear on that. Thanks for the clarification. As far as tracert is concerned
there was some timeout error I guess.

-N

On 31 March 2011 09:09, Nick Johnson (Google) nick.john...@google.comwrote:

 Hi Nischal,

 On Thu, Mar 31, 2011 at 1:31 PM, Nischal Shetty nischalshett...@gmail.com
  wrote:

 @Nick

 Thanks for getting back. tracert just goes blank, I don't see the tracing
 route to ghs.l.google.com thingy.


 Really? It doesn't output _anything_ at all?



 That's the thing I wanted to confirm, if it's my registrar or google since
 I point my domains record to google?


 If your registrar's DNS server isn't returning DNS records when queried, it
 doesn't matter where it's pointed.

 -Nick Johnson



 -N


 On 31 March 2011 07:48, Nick Johnson (Google) nick.john...@google.comwrote:

 Hi,

 On Wed, Mar 30, 2011 at 5:15 PM, nischalshetty 
 nischalshett...@gmail.com wrote:

 I'm getting intermittent timeouts when hitting
 http://www.justunfollow.com It's been happening from yesterday and a
 few other users complained about the same.

 My domain registrar is Godaddy. When it's not accessible, doing a
 tracert www.justunfollow.com gives me an error. Please help.


 It sounds like your registrar's DNS servers are unreliable. What error do
 you get?

 -Nick


  --
 You received this message because you are subscribed to the Google
 Groups Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.







 --
 -Nischal
 +91-9920240474
  twitter: NischalShetty http://twitter.com/nischalshetty
 facebook: Nischal http://facebook.com/nischal

 http://www.justunfollow.com





 --
 Nick Johnson, Developer Programs Engineer, App Engine




 --
 Nick Johnson, Developer Programs Engineer, App Engine

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
+91-9920240474
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Please star this if you want datastore backups

[Nischal Shetty]

I'm guessing this wouldn't be a good idea for large datastores. I have
around 7 gigs of data. Downloading the data or copying it to another
datastore as part of a backup doesn't really seem like a good idea.

The feature that I have shared the link to in this thread, asks for
snapshots of the datastore to be saved at regular intervals. It would really
help everyone if Google did that.

-N

On 21 March 2011 02:24, nickmilon nickmi...@gmail.com wrote:

 @ctran
 I tried that with a dataset of around 190K items couple of times - No
 Errors No warnings
 but 
 as @Francois is writing I have also seen other people in the group
 that complain.

 On Mar 20, 9:06 pm, Francois Masurel f.masu...@gmail.com wrote:
  I have tried the Copy to another app option from datastore_admin python
  app with a rather small datastore.
 
  It was far from perfect and a few entities where not copied.  A few
 people
  on this forum had the same kind of problems.
 
  https://groups.google.com/group/google-appengine/browse_thread/thread...
 
  It's not a production ready way to backup/restore your data.
 
  François

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
-Nischal
+91-9920240474
twitter: NischalShetty http://twitter.com/nischalshetty
facebook: Nischal http://facebook.com/nischal

http://www.justunfollow.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.