[google-appengine] Re: Get request from logging for Google App Engine (flexible environment)

['Nick (Cloud Platform Support)' via Google App Engine]

Hey Lu,

We've taken a look at the issue, and engineering is now working on it, but 
of course you've seen that from the thread itself. Just wanted to say 
thanks again for making a report and linking it here. 

Cheers,

Nick
Cloud Platform Community Support

On Friday, June 24, 2016 at 4:53:56 PM UTC-4, Lu Rocky wrote:
>
> Hi Nick,
>
> Thanks for your suggestion, I have post the issue, and here is the link 
> for issue is:
> https://code.google.com/p/googleappengine/issues/detail?id=13087
>
> Sincerely,
> Rocky
>
> On Friday, June 24, 2016 at 2:21:07 PM UTC-5, Nick (Cloud Platform 
> Support) wrote:
>>
>> Hey Lu,
>>
>> Be sure to post the issue link here once you've created it. I'll be happy 
>> to help work on the feature request!
>>
>> Sincerely,
>>
>> Nick
>> Cloud Platform Community Support
>>
>> On Wednesday, June 22, 2016 at 5:21:10 PM UTC-4, Lu Rocky wrote:
>>>
>>> Hi I am immigrating an app from standard environment to flexible 
>>> environment. But I am facing a problem, in the standard environment I have 
>>> built a flask server using python, and whenever I receive a HTTP get, I 
>>> record the request ID for the request log by 
>>> "os.environ.get('REQUEST_LOG_ID')". This help me to check the request log 
>>> in the future easily. This method is also taught on Google's document with 
>>> topic "How requests are handled" (
>>> https://cloud.google.com/appengine/docs/python/how-requests-are-handled). 
>>> But now in the flexible environment, I find there is no "'REQUEST_LOG_ID" 
>>> in the result returned by "os.environ". And I can not find any instruction 
>>> on how to get request id in flexible environment (the request id is still 
>>> exist, I can see the request id in the request log). So is there any way to 
>>> get the request id?
>>>
>>> Thanks so much for any help!
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/5f969a09-2332-4e53-9d1d-3c66b0fec612%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: /_ah/background stops executing with Firebase

['Nick (Cloud Platform Support)' via Google App Engine]

Hey Olof,

This is interesting. Could you clarify whether the logs are associated with 
recent calls, or whether they are perhaps older background threads which 
are closing due to their age? It appears the logs output got a bit mangled 
as well during copy-paste, is there any chance a more faithful reproduction 
of the complete logs info, or a screenshot with all fields expanded, could 
be provided?

Cheers,

Nick
Cloud Platform Community Support

On Tuesday, June 28, 2016 at 5:24:02 AM UTC-4, Olof Gunnarsson wrote:
>
>
> I'm using Google Cloud Endpoints and Google App Engine with Firebase 
> (java). Firebase calls are done in a background thread "/_ah/background", 
> which works fine. However, after a couple of days the background thread 
> stops executing. Calls to GAE still work but calls to Firebase are not 
> executed. A restart of the instance makes everything work again.
>
> I get these in the log which contains log messages from my calls to 
> Firebase, and then suddenly I don't get anymore of these logs and the calls 
> to Firebase stops executing.
>
> Any ideas?
>
> 17:02:46.795GET0 B56,999.2 sUnknown/_ah/background
> 17:02:46.795GET0 B34,555.2 sUnknown/_ah/background
> 17:02:46.795GET0 B17,351 sUnknown/_ah/background
> 17:02:46.795GET0 B17,350.9 sUnknown/_ah/background
> 17:02:46.795GET0 B8,925.6 sUnknown/_ah/background
> 17:02:46.795GET0 B8,923.4 sUnknown/_ah/background
> 17:02:46.795GET0 B8,866 sUnknown/_ah/background
> 17:02:46.795GET0 B8,131.1 sUnknown/_ah/background
> 17:02:46.795GET0 B8,103.3 sUnknown/_ah/background
> 17:02:46.795GET0 B8,101.8 sUnknown/_ah/background
> 17:02:46.795GET0 B8,079.2 sUnknown/_ah/background
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/6c3694d9-bdf3-45cc-b95c-0acd7ed1317f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: All app engine projects timing out when deploying

['Nick (Cloud Platform Support)' via Google App Engine]

Here's a link to the Public Issue Tracker 
!

On Thursday, June 30, 2016 at 7:23:36 PM UTC-4, Nick (Cloud Platform 
Support) wrote:
>
> Hey Ken,
>
> This sounds like something that could be turned into a concrete issue 
> report in the Public Issue Tracker. This forum is for more general 
> discussion of the platform and services, while the Public Issue Tracker is 
> uniquely focused on taking issue reports and delving into them. When making 
> the report, be sure to provide as much information about the system and the 
> symptoms as possible. For this case, this might include the commands you 
> ran, the full output, and any details about the system which might be 
> helpful or relevant to the specific error seen. When you make the issue 
> report, be sure to post it here so that other users can find it if they 
> land on this thread via a google search, and also so that I can take a look 
> at it myself!
>
> Cheers,
>
> Nick
> Cloud Platform Community Support 
>
> On Wednesday, June 29, 2016 at 9:08:35 AM UTC-4, Ken Liu wrote:
>>
>> Getting errors on all my projects today, error 13, however, have tested 
>> code and all apps run locally, just not deploying properly
>>
>> Anyone else encountering this?
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/c4cb61d8-1881-4609-90ae-d17bdb160fd9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: All app engine projects timing out when deploying

['Nick (Cloud Platform Support)' via Google App Engine]

Hey Ken,

This sounds like something that could be turned into a concrete issue 
report in the Public Issue Tracker. This forum is for more general 
discussion of the platform and services, while the Public Issue Tracker is 
uniquely focused on taking issue reports and delving into them. When making 
the report, be sure to provide as much information about the system and the 
symptoms as possible. For this case, this might include the commands you 
ran, the full output, and any details about the system which might be 
helpful or relevant to the specific error seen. When you make the issue 
report, be sure to post it here so that other users can find it if they 
land on this thread via a google search, and also so that I can take a look 
at it myself!

Cheers,

Nick
Cloud Platform Community Support 

On Wednesday, June 29, 2016 at 9:08:35 AM UTC-4, Ken Liu wrote:
>
> Getting errors on all my projects today, error 13, however, have tested 
> code and all apps run locally, just not deploying properly
>
> Anyone else encountering this?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/4462a41c-39e7-4109-8f70-2ed93c44df5b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: How to store open connection to external MongoDB

['Nick (Cloud Platform Support)' via Google App Engine]

Hey Prasanga, 

I'm unsure what you meant by your last post. Do you think you could clarify 
in more precise language what behaviour you're referring to? Do you mean 
the fact that "Sockets may be reclaimed after 2 minutes of inactivity; any 
socket operation keeps the socket alive for a further 2 minutes.", as 
mentioned in the docs 

?

Reviewing this thread, I'd say that forcing a software implementation 
dependent on opening OS threads into the App Engine mold simply may not be 
the best use of dev cycles. It might be best to delegate the mongoDB 
connection functionality to a GCE instance which holds connection pools on 
proper OS threads. For modest loads, one such instance ought not to be much 
of a bottleneck because of the low-latency nature of simply forwarding 
requests and data. It would act basically like a proxy.

The other option is to connect to Mongo via REST, although this is more 
expensive, as it uses higher layers of the protocol stack, with HTTP 
headers, etc.

Anyways, let us know your thoughts Prasanga!

Cheers,

Nick
Cloud Platform Community Support 

On Wednesday, June 29, 2016 at 12:43:11 AM UTC-4, Prasanga Siripala wrote:
>
> I believe to keep the connection alive, SOCKET API requires a new request. 
> Therefore after the request is finished, you can't reuse connection.
>
> On Wednesday, November 4, 2015 at 6:39:17 PM UTC+11, Minie Takalova wrote:
>>
>> Hello
>>
>> I'am trying to build application on GAE which use MongoDB hosted at 
>> Mongolab provider.
>> Connection work, if I made new connection every request, but if I want to 
>> reuse once established connection, GAE freeze and die on timeout.
>> I believe that exist some simple way, how to do that such simple task.
>>
>>
>> -  This is woring, but open new connection every request: --
>> def get_database_connection():
>> client = pymongo.MongoClient(MONGO_URI)
>> DBCONN = client.get_default_database()
>> return DBCONN
>>
>> DBCON = get_database_connection()
>>  ... do something usefull with DBCONN ...
>>  ... not store connection anywhere 
>>  ... end request 
>>
>>
>>  This freeze GAE and die on timeout ---
>> def get_database_connection():
>> if "DBCONN" in globals(): # look for open connection in 
>> instance memory
>> return globals()["DBCONN"]
>> client = pymongo.MongoClient(MONGO_URI)
>> DBCONN = client.get_default_database()
>> globals()["DBCONN"] = DBCONN  # store connection to instance memory
>> return DBCONN
>>
>> DBCONN = get_database_connection()
>>  ... do something usefull with DBCONN ...
>>  ... end request  
>>
>> 
>>
>> Probably GAE module with backend thread can be uset to keep connection 
>> open in separate thread, but after many hours I can´t find working solution.
>> Hope somebody can experience with this usecase and can help me. Also link 
>> to well-tried solution will be helpfull. 
>>
>> Have a nice day.
>>
>>
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/7cca5ff6-e37b-47ef-b70d-e11d78580532%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Does Google App Engine (Flexible) Support Google Endpoints?

[Robert King]

Thanks Jon
Good to hear there are plans for investing in endpoints. I would say being 
able to easily build APIs is fairly fundamental and is a high priority. 
There is a lot of work that could be done to the api explorer, testing, and 
api clients. Having typescript definitions would also be really nice.
I like that you can swap out backends although I'm having to use 
go-endpoints which is not an official package and comes with no guarantees 
or support
I do run seperate modules which is nice - Although It does impact instance 
hours which generally puts you over the free quota. Perhaps 15 minutes to 
startup a go module is a bit harsh since go is very quick to startup.



On Friday, 1 July 2016 07:28:26 UTC+12, Jon Parrott wrote:
>
> Robert and everyone: we hear you. Please keep in mind that App Engine 
> Flexible is still in beta, and the Python compat runtime itself is in 
> alpha. We're still doing a lot of work on both and we're still collecting 
> user feedback. Things like this mailing list and the public issue track are 
> some of the ways we collect feedback, so while it may appear dismissive for 
> us to say "file an issue" - we do take those seriously.
>
> Regarding endpoints itself: I can't speak any definite plans, but I can 
> stay that we are investing in endpoints and that you should stay tuned.
>
> In the meantime, I would recommend looking into "hybrid" applications - 
> use app engine services (neé modules) to have one service running on 
> standard that works as your API frontend, and another service running on 
> flexible that can handle the heavy lifting. 
>
> On Wednesday, June 29, 2016 at 8:57:58 PM UTC-7, Robert King wrote:
>>
>> Sorry if this is harsh and correct me if i'm wrong.
>> I think google should take a stance on this. You can't just say "feel 
>> free to file a feature request". Google should decide if endpoints make 
>> sense or not and if not, apologies to existing developers using it & put a 
>> warning in the documentation for people new to app engine. I've been pretty 
>> disappointed with google apis, I suspect google doesn't use their own apis 
>> internally, and that their apis are just a quick and dirty wrapper around 
>> what they use internally (for example, how many full time engineers are 
>> there working on making google api clients nice?). For example look at the 
>> javascript client library that's not out of beta since 2012 
>> https://groups.google.com/forum/#!topic/google-api-javascript-client/sfxFLTI8h6Y
>> You would think they'd have a nice typescript version too for each api. 
>> Perhaps they're waiting on firebase integration and there is conflict with 
>> firebase auth etc, who knows?
>>
>>
>>
>> On Thursday, 30 June 2016 07:47:57 UTC+12, Nicholas (Google Cloud 
>> Support) wrote:
>>>
>>> You are correct that there is no documentation on Google Cloud Endpoints 
>>> within the App Engine flexible environment.  As posted by Zdenko, they are 
>>> not supported at this time.  Though it may work in some cases, there's 
>>> officially no support for this and it should not be relied upon.
>>>
>>> If you would like to see Endpoints implements into the flexible 
>>> environment, feel free to file a feature request on the App Engine 
>>> public issue  
>>> tracker.
>>>
>>> Hope this helps.
>>>
>>> On Tuesday, June 28, 2016 at 3:33:07 PM UTC-4, Bill Li wrote:

 Hi, I have used Google Endpoint on Google App Engine Standard 
 environment before, and I think it is easy to use. But I did not find any 
 documentation on how to deploy Endpoints on Google App Engine Flexible 
 Environment. Do anyone has any idea about whether we can use endpoints on 
 flexible environment?
 From my research, it seems that the Google App Engine SDK is only used 
 by Standard mode, and the endpoints need support from Google App Engine 
 SDK. So I am wondering whether this means the Flexible Environment does 
 not 
 support Google Endpoint?
 Thanks for any answer!

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/5228a6ca-6364-40b3-acd7-6f26b1bc5714%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Does Google App Engine (Flexible) Support Google Endpoints?

['Jon Parrott' via Google App Engine]

Robert and everyone: we hear you. Please keep in mind that App Engine 
Flexible is still in beta, and the Python compat runtime itself is in 
alpha. We're still doing a lot of work on both and we're still collecting 
user feedback. Things like this mailing list and the public issue track are 
some of the ways we collect feedback, so while it may appear dismissive for 
us to say "file an issue" - we do take those seriously.

Regarding endpoints itself: I can't speak any definite plans, but I can 
stay that we are investing in endpoints and that you should stay tuned.

In the meantime, I would recommend looking into "hybrid" applications - use 
app engine services (neé modules) to have one service running on standard 
that works as your API frontend, and another service running on flexible 
that can handle the heavy lifting. 

On Wednesday, June 29, 2016 at 8:57:58 PM UTC-7, Robert King wrote:
>
> Sorry if this is harsh and correct me if i'm wrong.
> I think google should take a stance on this. You can't just say "feel free 
> to file a feature request". Google should decide if endpoints make sense or 
> not and if not, apologies to existing developers using it & put a warning 
> in the documentation for people new to app engine. I've been pretty 
> disappointed with google apis, I suspect google doesn't use their own apis 
> internally, and that their apis are just a quick and dirty wrapper around 
> what they use internally (for example, how many full time engineers are 
> there working on making google api clients nice?). For example look at the 
> javascript client library that's not out of beta since 2012 
> https://groups.google.com/forum/#!topic/google-api-javascript-client/sfxFLTI8h6Y
> You would think they'd have a nice typescript version too for each api. 
> Perhaps they're waiting on firebase integration and there is conflict with 
> firebase auth etc, who knows?
>
>
>
> On Thursday, 30 June 2016 07:47:57 UTC+12, Nicholas (Google Cloud Support) 
> wrote:
>>
>> You are correct that there is no documentation on Google Cloud Endpoints 
>> within the App Engine flexible environment.  As posted by Zdenko, they are 
>> not supported at this time.  Though it may work in some cases, there's 
>> officially no support for this and it should not be relied upon.
>>
>> If you would like to see Endpoints implements into the flexible 
>> environment, feel free to file a feature request on the App Engine 
>> public issue  
>> tracker.
>>
>> Hope this helps.
>>
>> On Tuesday, June 28, 2016 at 3:33:07 PM UTC-4, Bill Li wrote:
>>>
>>> Hi, I have used Google Endpoint on Google App Engine Standard 
>>> environment before, and I think it is easy to use. But I did not find any 
>>> documentation on how to deploy Endpoints on Google App Engine Flexible 
>>> Environment. Do anyone has any idea about whether we can use endpoints on 
>>> flexible environment?
>>> From my research, it seems that the Google App Engine SDK is only used 
>>> by Standard mode, and the endpoints need support from Google App Engine 
>>> SDK. So I am wondering whether this means the Flexible Environment does not 
>>> support Google Endpoint?
>>> Thanks for any answer!
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/35fe00e2-8fd3-47a8-ada0-d7e2e90c01cd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] An error occured while connecting to the server

[정래진]

The question about the errors in the source code to scraping web pages. 
Code written in Python is running on the Google App Engine. url address can 
be properly connected, and also works fine scraping code. If it might be 
the cause of error please answer me.

Python + google app engine source























































*import sysreload(sys)sys.setdefaultencoding("utf-8")sys.path.insert(0, 
'libs')import webapp2from bs4 import BeautifulSoupimport mathimport 
urllib2class MainPage(webapp2.RequestHandler):def get(self, args1, args2):  
  url = "http://emart.ssg.com/category/list.ssg?dispCtgId=;url += 
str(args1)url += "="url += str(args2)print("log_print" + 
url);data = ""source_code = urllib2.urlopen(url).read()
plain_text = source_codesoup = BeautifulSoup(plain_text, 
"html.parser")for info_list in soup.find("tbody").find_all(class_="item 
w202"):title = info_list.find(class_="title").a["title"]if 
title is None:continueelse:data += title
data += "\t"price = info_list.find(class_="price")
if price is None:continueelse:data += 
price.strong.stringdata += "\t"img_url = 
info_list.find(class_="thm").a.img["src"]if img_url is None:
continueelse:data += "http:" + img_url
data += "\t"code = img_url.split("/")[7].split("_")[0]data 
+= code + "\n"self.response.write(data) app = 
webapp2.WSGIApplication([('/emart_product_v2/(\d+)_(\d+)', MainPage),], 
debug=True)*

error code

*Traceback (most recent call last): File 
"/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py",
 
line 267, in Handle result = handler(dict(self._environ), 
self._StartResponse) File 
"/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py",
 
line 1519, in __call__ response = self._internal_error(e) File 
"/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py",
 
line 1511, in __call__ rv = self.handle_exception(request, response, e) 
File 
"/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py",
 
line 1505, in __call__ rv = self.router.dispatch(request, response) File 
"/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py",
 
line 1253, in default_dispatcher return route.handler_adapter(request, 
response) File 
"/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py",
 
line 1077, in __call__ return handler.dispatch() File 
"/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py",
 
line 547, in dispatch return self.handle_exception(e, self.app.debug) File 
"/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py",
 
line 545, in dispatch return method(*args, **kwargs) File 
"/base/data/home/apps/s~inte-core1/v1.393867712548898881/emart_product.py", 
line 28, in get source_code = urllib2.urlopen(url).read() File 
"/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py", 
line 127, in urlopen return _opener.open(url, data, timeout) File 
"/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py", 
line 404, in open response = self._open(req, data) File 
"/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py", 
line 422, in _open '_open', req) File 
"/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py", 
line 382, in _call_chain result = func(*args) File 
"/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py", 
line 1214, in http_open return self.do_open(httplib.HTTPConnection, req) 
File 
"/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py", 
line 1187, in do_open r = h.getresponse(buffering=True) File 
"/base/data/home/runtimes/python27/python27_dist/lib/python2.7/gae_override/httplib.py",
 
line 536, in getresponse 'An error occured while connecting to the server: 
%s' % e) error: An error occured while connecting to the server: Connection 
closed unexpectedly by server at URL: 
http://emart.ssg.com/category/list.ssg?dispCtgId=0006511712=1*




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/d4e5e4f5-9f37-4445-a816-7094244531ff%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Gain the Skills Needed to Immediately and Effectively Participate in Big Data and Data Science

[Kenuma Vijay]

*Data Science * and big data analytics is an 'open' 
course that provides an introduction to big data  and 
the Data Analytics Lifecycle to address business challenges that leverage 
big data. it provides grounding in basic and advanced analytic methods and 
an introduction to big data analytics technology and tools, including 
Hadoop,Mahout,Apache Solr,Apache Storm,Splunk,Cassandra,Hbase and Apache 
Spark, Scala  etc.



Start learning Big Data and Data Science  from 
basics to advance levels here...

> *https://goo.gl/wubxSk*

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/78649858-406f-4862-ab60-a495332ddb90%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Gain the Skills Needed to Immediately and Effectively Participate in Big Data and Data Science

[Kenuma Vijay]

*Data Science * and big data analytics is an 'open' 
course that provides an introduction to big data  and 
the Data Analytics Lifecycle to address business challenges that leverage 
big data. it provides grounding in basic and advanced analytic methods and 
an introduction to big data analytics technology and tools, including 
Hadoop,Mahout,Apache Solr,Apache Storm,Splunk,Cassandra,Hbase and Apache 
Spark, Scala  etc.



Start learning Big Data and Data Science  from 
basics to advance levels here...

> *https://goo.gl/wubxSk*

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/e153a93a-d8da-4925-b783-256fdeb3dea4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: GAE Latency & Instance issues

[troberti]

Great to hear that it helps. Actually, if you are using F4s, I might try a 
slightly higher max_concurrent_requests , say 4. Again, test and compare to 
be sure.

Finally, to reduce costs, I would recommend to set max_idle_instances to 1. 
Keep min_idle_instances to what you need for your application. For us this 
reduces cost significantly without any apparent drawbacks.

On Thursday, June 30, 2016 at 11:44:34 AM UTC+2, Trevor wrote:
>
> Well, I have to say thank you very, very much. Thanks to your advice we 
> have our lowest latency in 3 years! Sub 300ms average.  As expected though, 
> we are now sitting on 21 billed f4 instances, which will potentially cost 
> us in the order of 3x our current ($30-40 -> $100+), but we will tweak that 
> from tomorrow onwards. Peak hour is about to hit so we are going to see if 
> the system can keep sub-300ms at the current "automatic" setting for 
> scaling. But yes, once again, thank you for solving in 5 minutes what I 
> have been working on doing for 2 weeks (my tears are from joy and sadness 
> all at once)
>
>
> 
>
>
> 
>
>
> On Thursday, June 30, 2016 at 6:03:23 PM UTC+9, troberti wrote:
>>
>> Right, you should definitely test and see what the results are. My first 
>> inclination was also to increase max_concurrent_requests, but because then 
>> all those requests have increased latency, the actual QPS per instance 
>> decreased! Lowering max_concurrent_requests decreased request latency, so 
>> each instance could process more requests/second.
>>
>> We use F1 instances, because we do not need the additional memory, and 
>> our requests perform mostly RPCs. In our testing, faster instance classes 
>> do process requests faster, but also cost significantly more.  F1s provide 
>> the best performance/cost ratio for us. This could be a Python thing, not 
>> sure. Again, you should really test and figure out what is the best for 
>> your application+runtime.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/beeb1e7a-669c-4900-b330-eec30aee9b2c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: GAE Latency & Instance issues

[Trevor]

Well, I have to say thank you very, very much. Thanks to your advice we 
have our lowest latency in 3 years! Sub 300ms average.  As expected though, 
we are now sitting on 21 billed f4 instances, which will potentially cost 
us in the order of 3x our current ($30-40 -> $100+), but we will tweak that 
from tomorrow onwards. Peak hour is about to hit so we are going to see if 
the system can keep sub-300ms at the current "automatic" setting for 
scaling. But yes, once again, thank you for solving in 5 minutes what I 
have been working on doing for 2 weeks (my tears are from joy and sadness 
all at once)






On Thursday, June 30, 2016 at 6:03:23 PM UTC+9, troberti wrote:
>
> Right, you should definitely test and see what the results are. My first 
> inclination was also to increase max_concurrent_requests, but because then 
> all those requests have increased latency, the actual QPS per instance 
> decreased! Lowering max_concurrent_requests decreased request latency, so 
> each instance could process more requests/second.
>
> We use F1 instances, because we do not need the additional memory, and our 
> requests perform mostly RPCs. In our testing, faster instance classes do 
> process requests faster, but also cost significantly more.  F1s provide the 
> best performance/cost ratio for us. This could be a Python thing, not sure. 
> Again, you should really test and figure out what is the best for your 
> application+runtime.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/286855e9-52ac-4ded-bed5-adfc3abebc8f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: GAE Latency & Instance issues

[troberti]

Right, you should definitely test and see what the results are. My first 
inclination was also to increase max_concurrent_requests, but because then 
all those requests have increased latency, the actual QPS per instance 
decreased! Lowering max_concurrent_requests decreased request latency, so 
each instance could process more requests/second.

We use F1 instances, because we do not need the additional memory, and our 
requests perform mostly RPCs. In our testing, faster instance classes do 
process requests faster, but also cost significantly more.  F1s provide the 
best performance/cost ratio for us. This could be a Python thing, not sure. 
Again, you should really test and figure out what is the best for your 
application+runtime.


On Thursday, June 30, 2016 at 10:09:56 AM UTC+2, Trevor wrote:
>
> Thanks for the advice! I hadn't considered setting it that low because we 
> are on F4 instances and I would really, *really* hope that they could 
> handle at least the default 8 concurrent requests. That being said, I will 
> throw the front-end on those settings this evening and monitor until noon 
> tomorrow. The only thing I worry about is the amount of instances 
> increasing rapidly and blowing out our monthly costs. We have 10 PV per sec 
> during low times, and up to 30/35 per sec during peak hours. That means to 
> maintain our current costs (3-5 billed f4 instances) we would need a 
> sub-300ms request-completion time during peak, if my muddled math is 
> correct. I suppose if there are less requests going to each instance, we 
> could drop down to a lower class, perhaps?
>
>
> 
>
>
> What instance class do you run, if I may ask?
>
> On Thursday, June 30, 2016 at 4:53:53 PM UTC+9, troberti wrote:
>>
>> I recommend trying to set max_concurrent_requests to 2. As you said, 
>> higher values only makes latency (much) worse. In our experience, a value 
>> of 2 gets the best latency results without an increase in cost.
>>
>> We still have some of those pauses mid-request in a trace, and I really 
>> would like to know where they come from (and get rid of them), but they 
>> seem much shorter with a lower max_concurrent_requests value.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/d43195ca-6d54-470f-bf2b-eef20274a2e0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Google App Engine multiple regions

[Miguel Vitorino]

Kaan,

I believe the ping time to appspot.com domains is not indicative of the 
actual smallest response time as it only pings the entry point of Google's 
global edge network. Once your request is inside Google internal network, 
it will then travel to your GAE instances and back. If you ping an app from 
London, you only get 10s of ms, yet if you perform a full simple HTTP 
request it will add the 100ms of cross atlantic RTT at the very least.


On Wednesday, June 22, 2016 at 9:38:22 AM UTC+1, Kaan Soral wrote:
>
> I think one way or another, there's going to be a delay, why do you need 
> to minimise the delay so much?
>
> Even if the instances were spread over the world, I'm guessing there would 
> be a smaller data access delay in most cases
>
> I'm way outside US, my users are closer to EU, yet my appengine location 
> is US, didn't have any issues up to now
>
> Out of curiosity, I pinged my domain and the appspot domain, the domain is 
> wrapped with Cloudflare
>
> Appspot-> 30ms
> Cloudflare-> 60ms
>
> Before this experiment I expected Cloudflare to decrease the international 
> ping, it didn't
>
> Then I wget'd the domain
>
> Appspot-> 0.1s
> Cloudflare-> 0.2s
>
> It obviously doesn't directly answer your question, yet it's an insight
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/f594f6fc-9e73-4ffe-b761-99adabc08b1d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: GAE Latency & Instance issues

[Trevor Chinn]

Thanks for the advice! I hadn't considered setting it that low because we 
are on F4 instances and I would really, *really* hope that they could 
handle at least the default 8 concurrent requests. That being said, I will 
throw the front-end on those settings this evening and monitor until noon 
tomorrow. The only thing I worry about is the amount of instances 
increasing rapidly and blowing out our monthly costs. We have 10 PV per sec 
during low times, and up to 30/35 per sec during peak hours. That means to 
maintain our current costs (3-5 billed f4 instances) we would need a 
sub-300ms request-completion time during peak, if my muddled math is 
correct. I suppose if there are less requests going to each instance, we 
could drop down to a lower class, perhaps?




What instance class do you run, if I may ask?

On Thursday, June 30, 2016 at 4:53:53 PM UTC+9, troberti wrote:
>
> I recommend trying to set max_concurrent_requests to 2. As you said, 
> higher values only makes latency (much) worse. In our experience, a value 
> of 2 gets the best latency results without an increase in cost.
>
> We still have some of those pauses mid-request in a trace, and I really 
> would like to know where they come from (and get rid of them), but they 
> seem much shorter with a lower max_concurrent_requests value.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/e22233db-bcfe-4a69-ba58-69b5b8cf0bce%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: GAE Latency & Instance issues

[troberti]

I recommend trying to set max_concurrent_requests to 2. As you said, higher 
values only makes latency (much) worse. In our experience, a value of 2 
gets the best latency results without an increase in cost.

We still have some of those pauses mid-request in a trace, and I really 
would like to know where they come from (and get rid of them), but they 
seem much shorter with a lower max_concurrent_requests value.


On Thursday, June 30, 2016 at 9:45:10 AM UTC+2, Trevor Chinn wrote:
>
> Hello ladies and gentlemen, I am here to hopefully draw on some collective 
> knowledge about App Engine and its intricacies. 
>
> For the last two weeks our (my company) site has been experiencing very 
> odd latency issues, and having now tried about 7 different methods of 
> solving it, we are left at exactly where we began: Rising costs with 
> performance that is much lower than previously. 
>
>
> 
>
>
> Essentially what happens is that say 50-60% of our requests are served 
> normally, however the remainder have these extremely long "pauses" in the 
> middle of the trace which is basically during the "processing" phase of the 
> backend handling (after the datastore & memcache data has been retrieved). 
> Here is an example of a single page that in the space of an hour had wildly 
> different loading times for users. The vast majority were the same thing, 
> grab 3 things from memcache and spit out the html retrieved from memcache. 
> That's it... 
>
>
> 
>
> And some individual traces to see what is happening
>
>
> 
>
>
> 
>
>
> 
>
>
> 
>
>
>
>
> So essentially the troubleshooting steps we took to figure out what was 
> going wrong. 
>
>- Checked all deployed code over the week preceding and following the 
>latency spike to ensure we hadn't let some truly horrendous, heavy code 
>slip through the review process. Everything deployed around that period 
> was 
>rather light, backend/cms based updates, hardly anything touching 
>customer-facing requests. 
>- Appstats, obviously. On the development server (and even unloaded 
>test versions on the production server) such behavior is not seen. Didn't 
>help. 
>- Reducing unnecessary requests (figure 1) - We noticed some of our 
>ajax-loaded content was creating 2-3 additional, separate requests per 
>user-page-load, and as such refactored the code to only call those things 
>when absolutely necessary, and eliminated one altogether. For the most 
>part, a page load now equals one request. This had no effect on the 
> latency 
>spikes
>- Created a separate system that meant that our backend task-based 
>processing was cut down by 90%, and thus the instance average load was 
>reduced significantly. This had the opposite effect and average latency 
>actually climbed, I suspect because of the extensive memcache use with 
>large chunks of data (tracking what things should be updated by the 
> backend 
>tasks)
>- Separated the front end and tasks-back-end into modules/services so 
>that frontend requests could have 100% instance attention. This had a 
> small 
>effect, but the spikes are still regularly happening (as seen in the above 
>traces). 
>- Played with max_idle_instances  - This had a wonderful effect of 
>*halving* our daily instance costs, with almost no effect on latency. 
>When this is set to automatic, we get charged for a huge amount of unused 
>instances, it actually borders on ludicrous (figure 2) 
>- Played with max_concurrent_requests (8->16->10) which only served to 
>make the latency issues worse. 
>- Hours and hours pouring over logs, traces, dashboard graphs. 
>
>
> * Figure 1 (Since the latency spike on June 5th, we have worked to reduce 
> meaningless requests through API calls or task queuing) *
>
>
> 
>
> *Figure 2 (14:40 is when the auto-scaling setting was deployed)*
>
>
> 

[google-appengine] GAE Latency & Instance issues

[Trevor Chinn]

Hello ladies and gentlemen, I am here to hopefully draw on some collective 
knowledge about App Engine and its intricacies. 

For the last two weeks our (my company) site has been experiencing very odd 
latency issues, and having now tried about 7 different methods of solving 
it, we are left at exactly where we began: Rising costs with performance 
that is much lower than previously. 




Essentially what happens is that say 50-60% of our requests are served 
normally, however the remainder have these extremely long "pauses" in the 
middle of the trace which is basically during the "processing" phase of the 
backend handling (after the datastore & memcache data has been retrieved). 
Here is an example of a single page that in the space of an hour had wildly 
different loading times for users. The vast majority were the same thing, 
grab 3 things from memcache and spit out the html retrieved from memcache. 
That's it... 



And some individual traces to see what is happening












So essentially the troubleshooting steps we took to figure out what was 
going wrong. 

   - Checked all deployed code over the week preceding and following the 
   latency spike to ensure we hadn't let some truly horrendous, heavy code 
   slip through the review process. Everything deployed around that period was 
   rather light, backend/cms based updates, hardly anything touching 
   customer-facing requests. 
   - Appstats, obviously. On the development server (and even unloaded test 
   versions on the production server) such behavior is not seen. Didn't help. 
   - Reducing unnecessary requests (figure 1) - We noticed some of our 
   ajax-loaded content was creating 2-3 additional, separate requests per 
   user-page-load, and as such refactored the code to only call those things 
   when absolutely necessary, and eliminated one altogether. For the most 
   part, a page load now equals one request. This had no effect on the latency 
   spikes
   - Created a separate system that meant that our backend task-based 
   processing was cut down by 90%, and thus the instance average load was 
   reduced significantly. This had the opposite effect and average latency 
   actually climbed, I suspect because of the extensive memcache use with 
   large chunks of data (tracking what things should be updated by the backend 
   tasks)
   - Separated the front end and tasks-back-end into modules/services so 
   that frontend requests could have 100% instance attention. This had a small 
   effect, but the spikes are still regularly happening (as seen in the above 
   traces). 
   - Played with max_idle_instances  - This had a wonderful effect of 
   *halving* our daily instance costs, with almost no effect on latency. 
   When this is set to automatic, we get charged for a huge amount of unused 
   instances, it actually borders on ludicrous (figure 2) 
   - Played with max_concurrent_requests (8->16->10) which only served to 
   make the latency issues worse. 
   - Hours and hours pouring over logs, traces, dashboard graphs. 


* Figure 1 (Since the latency spike on June 5th, we have worked to reduce 
meaningless requests through API calls or task queuing) *



*Figure 2 (14:40 is when the auto-scaling setting was deployed)*



What I have noticed is when the CPU cycles spike, *so does the latency*. So 
it would lead in the direction that our requests are starved for CPU time, 
however now that we have deployed the instance auto scaling (and are paying 
for an average of around 8 instances vs 4-5 previously), it has not 
improved the latency, which confuses me considerably. 

If it were all requests that had slowed down, our code would clearly need 
optimization. If the rise in latency coincided with a change in our 
frontend processing, it would make sense, but there were only very light 
backend changes deployed within +/- 2 days of the first latency spike 

Re: [google-appengine] Flexible enviorment server 503/502

[Stefano Ciccarelli]

I have a lot of 502 errors everytime a VM starts or stops


Il giorno mer 29 giu 2016 alle ore 22:10 Felipe Augusto Meirelles <
fel...@innova.re> ha scritto:

> I'm getting a lot of badgateway/server error on GAE flexible env on the
> past few days. Is there any know issue?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/426d4195-6a5d-4288-b977-5ef536243d97%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
-- 
*Stefano Ciccarelli*
GAE Application Division
/ Director
stefano.ciccare...@mmbsoftware.it

*M.M.B. s.r.l.*
via Granarolo, 177/7 - 48018 Faenza (RA) - Italy
tel. +39.0546.637711 - fax +39.0546.46077
www.mmbsoftware.it - i...@mmbsoftware.it

Le informazioni contenute in questa comunicazione sono riservate e
destinate esclusivamente alla/e persona/e o all'ente sopra indicati. E'
vietato ai soggetti diversi dai destinatari qualsiasi uso, copia,
diffusione di quanto in esso contenuto sia ai sensi dell'art. 616 c.p., sia
ai sensi del DL n. 196/03. Se questa comunicazione Vi e' pervenuta per
errore, Vi preghiamo di rispondere a questa e-mail e successivamente
cancellarla dal Vostro sistema.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAArRkeWteEKoCGCASkBLqK-2nP8tRSsDxiOxhzGtMw3SfJfN7Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.