[google-appengine] Re: Using pubsub to trigger subsequent cloud build with filtering

[Tom Harris]

To be 100% explicit, I am adding the following into the Filters section of 
my Cloud Build Trigger, within the "Filter Editor" tab, as the first tab 
doesn't let me select anything useful from the build pub/sub message. This 
is right from the documentation.

*filter: build.status == Build.Status.SUCCESS*

And when I try to save my trigger, I receive this error at the bottom of 
the page:

*Failed to update trigger: generic::invalid_argument: ERROR: :1:7: 
Syntax error: mismatched input ':' expecting  | filter: build.status 
== Build.Status.SUCCESS | ..^*

I also tried removing "*filter: *" as it seems redundant to have that, but 
that also fails with an error:

*Failed to update trigger: generic::invalid_argument: ERROR: :1:1: 
undeclared reference to 'build' (in container '') | build.status == 
Build.Status.SUCCESS | ^ ERROR: :1:17: undeclared reference to 
'Build' (in container '') | build.status == Build.Status.SUCCESS | 
^*

Nothing I have tried seems to be valid in this and the documentation link 
right there in the trigger editor UI is exactly the documentation giving 
the examples.
[image: cloud-build-trigger-filters-error.png]

I've been working at this for a few days, and I'm out of ideas for making 
this function as documented. Please, some assistance is needed.

Thanks.

On Thursday, June 2, 2022 at 5:52:19 PM UTC-7 Tom Harris wrote:

>
> I have one cloud build trigger in one of our projects that deploys an App 
> Engine. I have also created a pubsub topic named 'cloud-builds' in this 
> same project, and I see that build event messages are being sent to that 
> topic during and at the end of builds.
>
> I am attempting to use this topic in another project with a pubsub cloud 
> build trigger. This works, but has no filtering, and as described on this 
> page, its firing the build way too often:
>
>
> https://cloud.google.com/build/docs/automating-builds/create-pubsub-triggers#risks_associated_with_an_unfiltered_trigger
>
> Now, all I need to determine how to do is add the filtering so only 
> successful builds with the right trigger ID are used. The documentation (
> https://cloud.google.com/build/docs/automating-builds/create-pubsub-triggers#using_cel_to_filter_build_events)
>  
> says to add something like this to my cloud build trigger filter:
>
> filter: build.build_trigger_id == "redacted" && build.status == 
> Build.Status.SUCCESS
>
> However, I am never able to save my trigger, as whatever I type into the 
> filter fails syntax validation. All the examples given on that page fail. I 
> have not been able to find any examples on the net, either. I have tried 
> subsets of these, exactly as described in the documentation, and they all 
> fail with syntax errors.
>
> How is this supposed to be entered into the cloud build trigger filter? We 
> cannot have every one of the pubsub messages firing a build, as I saw at 
> least 15 messages delivered when there was no filter, triggering 15 builds 
> in quick succession, most of which failed due to app engine deployment 
> concurrency.
>
> Please, need some assistance. Also, better documentation around this area 
> would be good.
>
> Many thanks,
> Tom Harris
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/e4e20d14-8b5a-4922-93a7-06025f370e30n%40googlegroups.com.

[google-appengine] Using pubsub to trigger subsequent cloud build with filtering

[Tom Harris]

I have one cloud build trigger in one of our projects that deploys an App 
Engine. I have also created a pubsub topic named 'cloud-builds' in this 
same project, and I see that build event messages are being sent to that 
topic during and at the end of builds.

I am attempting to use this topic in another project with a pubsub cloud 
build trigger. This works, but has no filtering, and as described on this 
page, its firing the build way too often:

https://cloud.google.com/build/docs/automating-builds/create-pubsub-triggers#risks_associated_with_an_unfiltered_trigger

Now, all I need to determine how to do is add the filtering so only 
successful builds with the right trigger ID are used. The documentation 
(https://cloud.google.com/build/docs/automating-builds/create-pubsub-triggers#using_cel_to_filter_build_events)
 
says to add something like this to my cloud build trigger filter:

filter: build.build_trigger_id == "redacted" && build.status == 
Build.Status.SUCCESS

However, I am never able to save my trigger, as whatever I type into the 
filter fails syntax validation. All the examples given on that page fail. I 
have not been able to find any examples on the net, either. I have tried 
subsets of these, exactly as described in the documentation, and they all 
fail with syntax errors.

How is this supposed to be entered into the cloud build trigger filter? We 
cannot have every one of the pubsub messages firing a build, as I saw at 
least 15 messages delivered when there was no filter, triggering 15 builds 
in quick succession, most of which failed due to app engine deployment 
concurrency.

Please, need some assistance. Also, better documentation around this area 
would be good.

Many thanks,
Tom Harris


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/3d4b74b4-30c4-437b-b1c9-f55de84ade7dn%40googlegroups.com.

[google-appengine] Re: GAE standard and flex: support PHP 8

[Tom Sykes]

In fact, if I deploy with a runtime that really doesn't exist, I get the 
following error which lists php81 as an option (but it doesn't work)

```
ERROR: (gcloud.app.deploy) INVALID_ARGUMENT: Invalid runtime 'php80' 
specified. Accepted runtimes are: [php, php55, python27, java, java7, 
java8, dotnet3, go111, go112, go113, go114, go115, go116, java11, java17, 
nodejs10, nodejs12,
 nodejs14, nodejs16, php72, php73, php74, php81, python37, python38, 
python39, python310, ruby25, ruby26, ruby27, ruby30, gcf preprod, gcf prod, 
gcf staging, java8 canary, php55 canary]
```

On Thursday, 27 January 2022 at 14:49:53 UTC Tom Sykes wrote:

> Spotted this today while trying to investigate a separate issue
> https://console.cloud.google.com/gcr/images/gae-runtimes/global/php81
>
> I've tried deploying a test using php81 as the runtime, but it doesn't 
> work.
>
> Does this mean php 8.1 support is coming? Does anyone have a timeframe for 
> public availability?
>
> On Thursday, 13 January 2022 at 14:27:19 UTC llu...@google.com wrote:
>
>> I think your point makes sense and I agree that warning people who use 
>> deprecated software could be useful. 
>>
>> I looked if there were any feature request with that or a similar idea 
>> but I wasn't able to find any.
>> I'd like to encourage you to create a feature request 
>> <https://cloud.google.com/support/docs/issue-trackers#feature_requests> 
>> explaining 
>> your idea of warning people who use deprecated versions of software.
>>
>> On Tuesday 11 January 2022 at 23:09:59 UTC+1 fkra...@gmail.com wrote:
>>
>>> Yeah that makes sense. But given that PHP 7.2 is still not even 
>>> deprecated (at least based on documentation) and PHP 7.4 soon reaching the 
>>> official deprecation from the PHP team without an alternative on App Engine 
>>> is concerning. Like obviously if anyone wanna use PHP 7.2 thats there 
>>> business but I feel like Google could at least pro-active warn about using 
>>> PHP 7.2 instead PHP 7.4 and hopefully provide soon PHP 8.0 and 8.1 support 
>>> so people can use a PHP version that is actively maintained.
>>>
>>> On Tuesday, January 11, 2022 at 2:14:32 a.m. UTC-8 llu...@google.com 
>>> wrote:
>>>
>>>> Hi, 
>>>>
>>>> I understand your concerns about PHP 7.2 being available in GAE even 
>>>> though it's out of security support. 
>>>> However, please note that according to the terms of service 
>>>> <https://cloud.google.com/terms#:~:text=(d)%C2%A0Discontinuation%20of%20Services.%20Google%20will%20notify%20Customer%20at%20least%2012%20months%20before%20discontinuing%20any%20Service%20(or%20associated%20material%20functionality)%20unless%20Google%20replaces%20such%20discontinued%20Service%20or%20functionality%20with%20a%20materially%20similar%20Service%20or%20functionality.>
>>>>  Google 
>>>> has to notify 12 months before discontinuing any Service. 
>>>>
>>>> On Tuesday 11 January 2022 at 09:22:20 UTC+1 fkra...@gmail.com wrote:
>>>>
>>>>> I was wondering as well if this can be somehow expedited. PHP 7.4 is 
>>>>> out of active support since a month (28 Nov 2021) and will be out of 
>>>>> security support in ten months (28 Nov 2022). It is also a bit scary that 
>>>>> PHP 7.2 is still offered which is out of security support since more than 
>>>>> half a year!
>>>>>
>>>>> On Thursday, December 30, 2021 at 9:36:45 a.m. UTC-8 cool...@gmail.com 
>>>>> wrote:
>>>>>
>>>>>> Thank you for the update. 
>>>>>> Regards.
>>>>>> On Monday, December 27, 2021 at 7:53:35 AM UTC-5 
>>>>>> bergall...@google.com wrote:
>>>>>>
>>>>>>> Hey there,
>>>>>>> There are no public updates at this time - the team is currently 
>>>>>>> still working on this.
>>>>>>>
>>>>>>> As for PHP 8.1, since it is a minor version I don't believe that a 
>>>>>>> separate tracker is required for it. If it is, said tracker will likely 
>>>>>>> be 
>>>>>>> created once this first tracker is set as `Closed` (more specifically I 
>>>>>>> believe the correct status would be either `Fixed` or `Fixed 
>>>>>>> (Verified)`)
>>>>>>>
>>>>>>> Kind regards,
>>>>>>>
>>>>>>> On Wednesday, December 22, 202

[google-appengine] Re: GAE standard and flex: support PHP 8

[Tom Sykes]

Spotted this today while trying to investigate a separate issue
https://console.cloud.google.com/gcr/images/gae-runtimes/global/php81

I've tried deploying a test using php81 as the runtime, but it doesn't work.

Does this mean php 8.1 support is coming? Does anyone have a timeframe for 
public availability?

On Thursday, 13 January 2022 at 14:27:19 UTC llu...@google.com wrote:

> I think your point makes sense and I agree that warning people who use 
> deprecated software could be useful. 
>
> I looked if there were any feature request with that or a similar idea but 
> I wasn't able to find any.
> I'd like to encourage you to create a feature request 
>  
> explaining 
> your idea of warning people who use deprecated versions of software.
>
> On Tuesday 11 January 2022 at 23:09:59 UTC+1 fkra...@gmail.com wrote:
>
>> Yeah that makes sense. But given that PHP 7.2 is still not even 
>> deprecated (at least based on documentation) and PHP 7.4 soon reaching the 
>> official deprecation from the PHP team without an alternative on App Engine 
>> is concerning. Like obviously if anyone wanna use PHP 7.2 thats there 
>> business but I feel like Google could at least pro-active warn about using 
>> PHP 7.2 instead PHP 7.4 and hopefully provide soon PHP 8.0 and 8.1 support 
>> so people can use a PHP version that is actively maintained.
>>
>> On Tuesday, January 11, 2022 at 2:14:32 a.m. UTC-8 llu...@google.com 
>> wrote:
>>
>>> Hi, 
>>>
>>> I understand your concerns about PHP 7.2 being available in GAE even 
>>> though it's out of security support. 
>>> However, please note that according to the terms of service 
>>> 
>>>  Google 
>>> has to notify 12 months before discontinuing any Service. 
>>>
>>> On Tuesday 11 January 2022 at 09:22:20 UTC+1 fkra...@gmail.com wrote:
>>>
 I was wondering as well if this can be somehow expedited. PHP 7.4 is 
 out of active support since a month (28 Nov 2021) and will be out of 
 security support in ten months (28 Nov 2022). It is also a bit scary that 
 PHP 7.2 is still offered which is out of security support since more than 
 half a year!

 On Thursday, December 30, 2021 at 9:36:45 a.m. UTC-8 cool...@gmail.com 
 wrote:

> Thank you for the update. 
> Regards.
> On Monday, December 27, 2021 at 7:53:35 AM UTC-5 bergall...@google.com 
> wrote:
>
>> Hey there,
>> There are no public updates at this time - the team is currently 
>> still working on this.
>>
>> As for PHP 8.1, since it is a minor version I don't believe that a 
>> separate tracker is required for it. If it is, said tracker will likely 
>> be 
>> created once this first tracker is set as `Closed` (more specifically I 
>> believe the correct status would be either `Fixed` or `Fixed (Verified)`)
>>
>> Kind regards,
>>
>> On Wednesday, December 22, 2021 at 10:00:13 AM UTC+1 
>> cool...@gmail.com wrote:
>>
>>> Dear Kevin,
>>>  Any updates on this? The public tracker shows no 
>>> updates and it has been over 12 months since you first created these? 
>>> In 
>>> the meanwhile, PHP has released 8.1. Do we need to create separate 
>>> trackers 
>>> for PHP 8.1 too?
>>>
>>> On Tuesday, October 13, 2020 at 6:01:06 PM UTC-4 Kevin Hirsh (Google 
>>> Cloud Platform Support) wrote:
>>>
 Hello,

 Although I do understand that this can be frustrating news, I have 
 gone ahead and submitted Public Feature Requests for PHP 8's possible 
 integration in the future. Please note that there is no guarantee of 
 implementation for Feature Requests nor is there a ETA.

 Please follow along the Public Tracker for the Standard 
  and Flexible 
  runtimes as all 
 updates from the App Engine specialist team will be found there.

 On Tuesday, October 13, 2020 at 4:58:57 PM UTC-4 Kevin Hirsh wrote:

> Hello,
>
> As of now, the Cloud Platform support team cannot comment on the 
> roadmap of GCP products and other such related issues.
>
> We apologize for the inconvenience!
>
> On Monday, October 12, 2020 at 9:53:26 AM UTC-4 
> advanced.develo...@gmail.com wrote:
>
>> Is GAE standard and flex: support PHP 8 implementation in the 
>> current roadma

[google-appengine] Re: When will we get App Engine Flex on Java 11?

[Tom Harris]

I cannot seem to find any Google-approved docker images with Jetty 9 with 
Java 11, they are all Java 8, so a solution using a custom runtime is 
impossible. The release not link provided doesn't mention Java 11 going 
years back.

Please add support for Java 11 for App Engine Flex. We will continue to run 
this App Engine Flex instance in Java 8 until this essential feature is 
added.

We can we make a formal request for this support - is there an ticketing 
system that is open to users external to Google that we could use to make 
the request?

Thanks,
Tom Harris

On Wednesday, January 12, 2022 at 12:39:09 AM UTC-8 Sima (Cloud Platform 
Support) wrote:

> Hello , 
>
> Note that, all updates about App Engine Flex will be reflected in the 
> respective release note 
> <https://cloud.google.com/appengine/docs/flexible/java/release-notes>. 
> You can use a custom runtime in the App Engine flexible environment 
> <https://cloud.google.com/appengine/docs/flexible/custom-runtimes> to use 
> an alternative implementation of Java 11 or any other languages.
>
> On Monday, January 10, 2022 at 4:44:52 PM UTC-5 t...@terratrue.com wrote:
>
>> Hi all:
>>
>> We have a set of microservices written in Java that run on App Engine 
>> standard, and one that runs in App Engine Flex. Last year, we upgraded all 
>> the App Engine standard services to run spring boot with Java 11, and all 
>> are now running fine in production. However, our one App Engine Flex 
>> service is still stuck on Java 8, waiting for Java 11 support for Flex. 
>> This service currently runs with Jetty 9 and is using web sockets and 
>> google cloud pub/sub.
>>
>> Is support for the App Engine Flex platform with Java 11 planned? If so, 
>> when? And if not, what is the suggested alternative? We cannot keep running 
>> things on Java 8 forever, as it has end-of-life for a while now.
>>
>> Thanks,
>> Tom Harris
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/592ac46a-43df-4781-ba53-95579ea820b3n%40googlegroups.com.

[google-appengine] When will we get App Engine Flex on Java 11?

[Tom Harris]

Hi all:

We have a set of microservices written in Java that run on App Engine 
standard, and one that runs in App Engine Flex. Last year, we upgraded all 
the App Engine standard services to run spring boot with Java 11, and all 
are now running fine in production. However, our one App Engine Flex 
service is still stuck on Java 8, waiting for Java 11 support for Flex. 
This service currently runs with Jetty 9 and is using web sockets and 
google cloud pub/sub.

Is support for the App Engine Flex platform with Java 11 planned? If so, 
when? And if not, what is the suggested alternative? We cannot keep running 
things on Java 8 forever, as it has end-of-life for a while now.

Thanks,
Tom Harris

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/b517c603-29e9-4ec5-bb80-6de6007447d9n%40googlegroups.com.

Re: [google-appengine] App Engine 11 with Cloud SQL mysql "No suitable driver"

[Tom Harris]

No, we are using pure app.yaml, no more appengine-web.xml in our modules.

On Mon, Oct 11, 2021 at 5:24 PM 'Ludovic Champenois' via Google App Engine <
google-appengine@googlegroups.com> wrote:

> Hi,
> Thanks for the update and the solution,
> We are looking and trying to understand the delta between the 2 runtimes...
> Just to be sure, you are using the new Beta Java11 runtime with bundled
> APIs (so using appengine-web.xml), right?
> Ludo
>
> On Monday, October 11, 2021 at 1:04:24 PM UTC-7 t...@terratrue.com wrote:
>
>> I posted this a week ago, and I solved this issue - I just had to add a
>> `Class.forName("com.mysql.cj.jdbc.Driver")` call, which was not needed in
>> App Engine 8 with Java 8 when running in the cloud, but apparently it is
>> now required. Without this, the database connection pool just
>> cannot initialize, and all you get is a stack trace which ultimately ends
>> with "No suitable driver".
>>
>> Thanks,
>> Tom Harris
>>
>>
>> On Mon, Oct 11, 2021 at 6:45 AM Tom Harris  wrote:
>>
>>> Hi all, running into a very thorny issue that I have spent considerable
>>> time on now, without a working solution.
>>>
>>> We have a set of Java 8 AppEngine-based microservices that I am
>>> upgrading to Java 11 and Spring Boot. The services are using MySQL via
>>> Cloud SQL with Google's socket factory. Locally, we use the MySQL driver
>>> directly and the services run as expected.
>>>
>>> However when I try to deploy out to a cloud environment, when the code
>>> tries to instantiate the Hikari connection pool using Cloud SQL, I receive
>>> the dreaded "No suitable driver" error from the JDBC DriverManager.
>>>
>>> Our project includes these two dependencies:
>>> 
>>>   com.google.cloud.sql
>>>   mysql-socket-factory-connector-j-8
>>>   1.3.3
>>> 
>>> 
>>>   mysql
>>>   mysql-connector-java
>>>   8.0.26
>>> 
>>>
>>> Here is the code that is failing:
>>> HikariConfig config = new HikariConfig();
>>> config.addDataSourceProperty("useSSL", "false");
>>> config.addDataSourceProperty("useUnicode", "true");
>>> config.addDataSourceProperty("characterEncoding", "UTF-8");
>>> config.setRegisterMbeans(true);
>>>
>>> config.setJdbcUrl("jdbc:mysql:///dbname");
>>> config.setUsername(dbUser);
>>> config.setPassword(dbPassword);
>>> config.addDataSourceProperty("socketFactory",
>>> "com.google.cloud.sql.mysql.SocketFactory");
>>> config.addDataSourceProperty("cloudSqlInstance", instanceConnectionName);
>>>
>>> return new HikariDataSource(config);
>>>
>>> And every time I try to fire up the service after deploying it to the
>>> cloud, I am seeing the error:
>>>
>>> Caused by: java.lang.RuntimeException: Failed to get driver instance for
>>> jdbcUrl=jdbc:mysql:///dbname  at
>>> com.zaxxer.hikari.util.DriverDataSource.(DriverDataSource.java:114)
>>> at com.zaxxer.hikari.pool.PoolBase.initializeDataSource(PoolBase.java:331)
>>> at com.zaxxer.hikari.pool.PoolBase.(PoolBase.java:114) at
>>> com.zaxxer.hikari.pool.HikariPool.(HikariPool.java:108) at
>>> com.zaxxer.hikari.HikariDataSource.(HikariDataSource.java:81) at
>>> com.terratrue.db.DataSources.initDataSourceForCloudEnvironmentForUser(DataSources.java:276)
>>> Caused by: java.sql.SQLException: No suitable driver at
>>> java.sql/java.sql.DriverManager.getDriver(DriverManager.java:298) at
>>> com.zaxxer.hikari.util.DriverDataSource.(DriverDataSource.java:106)
>>> ... 53 common frames omitted"
>>>
>>> I have searched the internet for anything that would help resolve this
>>> issue, but I can't seem to find a relevant solution. The jar files for both
>>> libs are found in the app-staging folder, so I know they are getting
>>> deployed with the service.
>>>
>>> Does anyone have suggestions as to what to do next?
>>>
>>> Thanks,
>>> Tom Harris
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Google App Engine" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to google-appengi...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.

Re: [google-appengine] App Engine 11 with Cloud SQL mysql "No suitable driver"

[Tom Harris]

I posted this a week ago, and I solved this issue - I just had to add a
`Class.forName("com.mysql.cj.jdbc.Driver")` call, which was not needed in
App Engine 8 with Java 8 when running in the cloud, but apparently it is
now required. Without this, the database connection pool just
cannot initialize, and all you get is a stack trace which ultimately ends
with "No suitable driver".

Thanks,
Tom Harris


On Mon, Oct 11, 2021 at 6:45 AM Tom Harris  wrote:

> Hi all, running into a very thorny issue that I have spent considerable
> time on now, without a working solution.
>
> We have a set of Java 8 AppEngine-based microservices that I am upgrading
> to Java 11 and Spring Boot. The services are using MySQL via Cloud SQL with
> Google's socket factory. Locally, we use the MySQL driver directly and the
> services run as expected.
>
> However when I try to deploy out to a cloud environment, when the code
> tries to instantiate the Hikari connection pool using Cloud SQL, I receive
> the dreaded "No suitable driver" error from the JDBC DriverManager.
>
> Our project includes these two dependencies:
> 
>   com.google.cloud.sql
>   mysql-socket-factory-connector-j-8
>   1.3.3
> 
> 
>   mysql
>   mysql-connector-java
>   8.0.26
> 
>
> Here is the code that is failing:
> HikariConfig config = new HikariConfig();
> config.addDataSourceProperty("useSSL", "false");
> config.addDataSourceProperty("useUnicode", "true");
> config.addDataSourceProperty("characterEncoding", "UTF-8");
> config.setRegisterMbeans(true);
>
> config.setJdbcUrl("jdbc:mysql:///dbname");
> config.setUsername(dbUser);
> config.setPassword(dbPassword);
> config.addDataSourceProperty("socketFactory",
> "com.google.cloud.sql.mysql.SocketFactory");
> config.addDataSourceProperty("cloudSqlInstance", instanceConnectionName);
>
> return new HikariDataSource(config);
>
> And every time I try to fire up the service after deploying it to the
> cloud, I am seeing the error:
>
> Caused by: java.lang.RuntimeException: Failed to get driver instance for
> jdbcUrl=jdbc:mysql:///dbname  at
> com.zaxxer.hikari.util.DriverDataSource.(DriverDataSource.java:114)
> at com.zaxxer.hikari.pool.PoolBase.initializeDataSource(PoolBase.java:331)
> at com.zaxxer.hikari.pool.PoolBase.(PoolBase.java:114) at
> com.zaxxer.hikari.pool.HikariPool.(HikariPool.java:108) at
> com.zaxxer.hikari.HikariDataSource.(HikariDataSource.java:81) at
> com.terratrue.db.DataSources.initDataSourceForCloudEnvironmentForUser(DataSources.java:276)
> Caused by: java.sql.SQLException: No suitable driver at
> java.sql/java.sql.DriverManager.getDriver(DriverManager.java:298) at
> com.zaxxer.hikari.util.DriverDataSource.(DriverDataSource.java:106)
> ... 53 common frames omitted"
>
> I have searched the internet for anything that would help resolve this
> issue, but I can't seem to find a relevant solution. The jar files for both
> libs are found in the app-staging folder, so I know they are getting
> deployed with the service.
>
> Does anyone have suggestions as to what to do next?
>
> Thanks,
> Tom Harris
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/792e5183-dea5-458f-8c44-abf185d5f8ebn%40googlegroups.com
> <https://groups.google.com/d/msgid/google-appengine/792e5183-dea5-458f-8c44-abf185d5f8ebn%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAB9U9Q8aoaCaKy4wnH%3DJU7mCfZu27jBkZmdSNgitDD2HxYD-7g%40mail.gmail.com.

[google-appengine] App Engine 11 with Cloud SQL mysql "No suitable driver"

[Tom Harris]

Hi all, running into a very thorny issue that I have spent considerable 
time on now, without a working solution.

We have a set of Java 8 AppEngine-based microservices that I am upgrading 
to Java 11 and Spring Boot. The services are using MySQL via Cloud SQL with 
Google's socket factory. Locally, we use the MySQL driver directly and the 
services run as expected.

However when I try to deploy out to a cloud environment, when the code 
tries to instantiate the Hikari connection pool using Cloud SQL, I receive 
the dreaded "No suitable driver" error from the JDBC DriverManager.

Our project includes these two dependencies:

  com.google.cloud.sql
  mysql-socket-factory-connector-j-8
  1.3.3


  mysql
  mysql-connector-java
  8.0.26


Here is the code that is failing:
HikariConfig config = new HikariConfig();
config.addDataSourceProperty("useSSL", "false");
config.addDataSourceProperty("useUnicode", "true");
config.addDataSourceProperty("characterEncoding", "UTF-8");
config.setRegisterMbeans(true);

config.setJdbcUrl("jdbc:mysql:///dbname");
config.setUsername(dbUser);
config.setPassword(dbPassword);
config.addDataSourceProperty("socketFactory", 
"com.google.cloud.sql.mysql.SocketFactory");
config.addDataSourceProperty("cloudSqlInstance", instanceConnectionName);

return new HikariDataSource(config);

And every time I try to fire up the service after deploying it to the 
cloud, I am seeing the error:

Caused by: java.lang.RuntimeException: Failed to get driver instance for 
jdbcUrl=jdbc:mysql:///dbname  at 
com.zaxxer.hikari.util.DriverDataSource.(DriverDataSource.java:114) 
at com.zaxxer.hikari.pool.PoolBase.initializeDataSource(PoolBase.java:331) 
at com.zaxxer.hikari.pool.PoolBase.(PoolBase.java:114) at 
com.zaxxer.hikari.pool.HikariPool.(HikariPool.java:108) at 
com.zaxxer.hikari.HikariDataSource.(HikariDataSource.java:81) at 
com.terratrue.db.DataSources.initDataSourceForCloudEnvironmentForUser(DataSources.java:276)
Caused by: java.sql.SQLException: No suitable driver at 
java.sql/java.sql.DriverManager.getDriver(DriverManager.java:298) at 
com.zaxxer.hikari.util.DriverDataSource.(DriverDataSource.java:106) 
... 53 common frames omitted"

I have searched the internet for anything that would help resolve this 
issue, but I can't seem to find a relevant solution. The jar files for both 
libs are found in the app-staging folder, so I know they are getting 
deployed with the service.

Does anyone have suggestions as to what to do next?

Thanks,
Tom Harris

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/792e5183-dea5-458f-8c44-abf185d5f8ebn%40googlegroups.com.

[google-appengine] Variable substitution in App Engine 11 app.yaml

[Tom Harris]

We've got several App Engine projects that I am migrating from App Engine 8
to App Engine 11 on Spring Boot. A number of them use a
vpc-access-connector in their appengine-web.xml file, which we inject using
resource filtering in the maven-war-plugin. The new way to configure the
app engine is to use a `src/main/appengine/app.yaml` file and a jar instead
of a war, as the artifact however I have been unable to determine how to do
variable substitution into the app.yaml file.

I need to inject different values based on the maven profile being run. We
currently define vpc access connector values in maven properties sections
within profiles aligned with our spring profiles for each of our deployment
environments, like test, ea and prod.

What is the solution here?

Is there some way to use different app.yaml files based on which profile is
being invoked? (This is not the best solution, as it would mean most of the
file content would be static copies, though the common stuff could be moved
to an included yaml.)

Or is there some syntax I can't locate online that will allow me to set an
environment variable or maven property and refer to it from within the
`src/main/appengine/app.yaml` file?

Thanks for any assistance!
Tom Harris

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAB9U9Q9KS9_DSc%2BXY6V30eGY0JPfPLaj63nn4U%2BJ1OAP5h6%2BTA%40mail.gmail.com.

Re: [google-appengine] Re: Determining if running local on App Engine Standard with Java 11

[Tom Harris]

I have found this works for my case, as null is being returned when running
locally in spring boot via maven.

 null = ServiceOptions.getAppEngineAppId()

In any non-static context, using the spring profile is the way to go.

Thanks,
Tom Harris
Tech Lead
TerraTrue, Inc.

On Wed, Apr 7, 2021 at 11:42 AM 'Elliott (Cloud Platform Support)' via
Google App Engine  wrote:

> Hello Tom,
>
> Thank you for your question regarding “SystemProperty.environment.value()”
> not being available for Java 11. Since upgrading to java 11, the value
> output of this code is always just null.
>
> For the community, this code is used to know if you are running the code
> locally or out in the cloud. You may see the details here
> <https://cloud.google.com/appengine/docs/standard/java/javadoc/com/google/appengine/api/utils/SystemProperty.Environment.Value>
> .
>
> You would like to know the equivalent code for App Engine 11 because there
> doesn't seem to be any documentation for this feature at all.
>
> I was able to verify your claim in public documentation. The information I
> researched is very old and does not appear to be included in the Java 11
> Runtime as you need it.
>
> So this is an opportunity to request this functionality but to add value
> to this feature request, how would you implement it?
>
> We may use your ideas to get this done but please note that there is no
> guarantee that it will be implemented but we will take a look and consider
> your request seriously.
>
>
> On Wednesday, April 7, 2021 at 10:45:20 AM UTC-4 t...@terratrue.com wrote:
>
>> Hi all:
>>
>> I am in the process of upgrading all our App Engine 8 services to 11, but
>> I am running into an issue. We use the `SystemProperty.environment.value()`
>> to determine if we are running locally on one of our developer machines or
>> out in the cloud, however since upgrading to java 11, the value is always
>> just null.
>>
>> What is the equivalent code for App Engine 11? There doesn't seem to be
>> any documentation for this at all.
>>
>> Thanks,
>> Tom Harris
>> Tech Lead
>> TerraTrue, Inc.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/3a358b5b-dbef-4f53-9685-f94ec2cb9033n%40googlegroups.com
> <https://groups.google.com/d/msgid/google-appengine/3a358b5b-dbef-4f53-9685-f94ec2cb9033n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAB9U9Q8%3DhCk-CsErn-CQC%2BJMJw3Zv5n5nt_kUPU6KkQ49YWP_g%40mail.gmail.com.

[google-appengine] Determining if running local on App Engine Standard with Java 11

[Tom Harris]

Hi all:

I am in the process of upgrading all our App Engine 8 services to 11, but I 
am running into an issue. We use the `SystemProperty.environment.value()` 
to determine if we are running locally on one of our developer machines or 
out in the cloud, however since upgrading to java 11, the value is always 
just null.

What is the equivalent code for App Engine 11? There doesn't seem to be any 
documentation for this at all.

Thanks,
Tom Harris
Tech Lead
TerraTrue, Inc.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/0a3581c7-4244-4144-9051-e359280a5aefn%40googlegroups.com.

[google-appengine] Is app-engine-go still a thing?

[Tom Mitchell]

I hope this is a good place to ask this question. I am trying to deploy a 
Golang service I wrote for App Engine. I am running Cloud SDK 322.0.

I can't seem to install app-engine-sdk because it does not look like it is 
even a thing any more. I would like to understand this. Here is what I am 
seeing:

➜  ~ gcloud components list

Your current Cloud SDK version is: 322.0.0
The latest available version is: 322.0.0

┌──┐
│  
Components  │
├───┬┬──┬──┤
│ Status│  Name  
│ID│   Size   │
├───┼┼──┼──┤
│ Not Installed │ Cloud Datalab Command Line Tool│ 
datalab  │  < 1 MiB │
│ Not Installed │ Cloud Datastore Emulator   │ 
cloud-datastore-emulator │ 18.4 MiB │
│ Not Installed │ Cloud Firestore Emulator   │ 
cloud-firestore-emulator │ 41.5 MiB │
│ Not Installed │ Cloud Pub/Sub Emulator │ 
pubsub-emulator  │ 56.4 MiB │
│ Not Installed │ gcloud Alpha Commands  │ 
alpha│  < 1 MiB │
│ Not Installed │ gcloud Beta Commands   │ 
beta │  < 1 MiB │
│ Not Installed │ gcloud app Java Extensions │ 
app-engine-java  │ 59.6 MiB │
│ Not Installed │ gcloud app PHP Extensions  │ 
app-engine-php   │ 21.9 MiB │
│ Not Installed │ gcloud app Python Extensions   │ 
app-engine-python│  6.1 MiB │
│ Not Installed │ gcloud app Python Extensions (Extra Libraries) │ 
app-engine-python-extras │ 27.1 MiB │
│ Not Installed │ pkg│ 
pkg  │  │
│ Installed │ BigQuery Command Line Tool │ 
bq   │  < 1 MiB │
│ Installed │ Cloud SDK Core Libraries   │ 
core │ 16.1 MiB │
│ Installed │ Cloud Storage Command Line Tool│ 
gsutil   │  3.5 MiB │
│ Installed │ kubectl│ 
kubectl  │  < 1 MiB │
└───┴┴──┴──┘
To install or remove components at your current SDK version [322.0.0], run:
  $ gcloud components install COMPONENT_ID
  $ gcloud components remove COMPONENT_ID

To update your SDK installation to the latest version [322.0.0], run:
  $ gcloud components update

Looks like no offering for go any more??

And to be sure, I tried to install it:

➜  ~ gcloud components install app-engine-golang
ERROR: (gcloud.components.install) The following components are unknown 
[app-engine-golang].
➜  ~

Other than this, the sdk and command seem to work fine.

Is golang even supported any more??

Thanks in advance.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/592aca92-e6c5-45b8-885c-320637f3b111n%40googlegroups.com.

[google-appengine] Re: How to proceed multiple streamingRecognize request in Google Speech-to-Text API?

[Tom Chiang]

Umm... So it this the only way to archive the goal by implementing like 
this?
eval("var " + myString + " = new speech.SpeechClient();");



Fabio Santos於 2020年2月21日星期五 UTC+8上午1時58分36秒寫道：
>
> Wow, this is pretty recent, I'm also on the same boat right. Everything 
> working flawlessly except for this, I might have to create a JS Object like 
> {[userKey]: new speech.SpeechClient} as to add all users as they need to 
> talk.
>
> On Thursday, February 20, 2020 at 2:53:50 AM UTC, Tom Chiang wrote:
>>
>>
>> I have built a web based Speech-to-Text application with Google 
>> Speech-to-Text API.
>>
>> The application flow:
>>
>>1. Get audio stream from browser 
>>2. Stream audio data to a speech recognition service and get real 
>>time results as our user talks
>>
>> [image: 0_MJ6ydPicL3CrJ6zg.png]
>>
>> I have referenced the sample code 
>> <https://github.com/googleapis/nodejs-speech> in my application, the 
>> streamingRecognize request works perfectly with a signal client call, 
>> and my server can receive the interim-transcription result from Google 
>> Speech API. However, when there are more then one client call 
>> streamingRecognize simultaneously, Google Speech API will get stuck. So, 
>> may I know how to handle more than one streamingRecognize request 
>> concurrently? Do I need to create another client to process the second 
>> streamingRecognize request? Thanks.
>>
>> Back-end server specifications:
>>
>>- NodeJS
>>- Socket.io
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/7aa577ef-79b1-4967-9f29-8bf1d7468aad%40googlegroups.com.

[google-appengine] How to proceed multiple streamingRecognize request in Google Speech-to-Text API?

[Tom Chiang]

I have built a web based Speech-to-Text application with Google 
Speech-to-Text API.

The application flow:

   1. Get audio stream from browser 
   2. Stream audio data to a speech recognition service and get real time 
   results as our user talks

[image: 0_MJ6ydPicL3CrJ6zg.png]

I have referenced the sample code 
 in my application, the 
streamingRecognize request works perfectly with a signal client call, and 
my server can receive the interim-transcription result from Google Speech 
API. However, when there are more then one client call streamingRecognize 
simultaneously, 
Google Speech API will get stuck. So, may I know how to handle more than 
one streamingRecognize request concurrently? Do I need to create another 
client to process the second streamingRecognize request? Thanks.
   
Back-end server specifications:

   - NodeJS
   - Socket.io

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/052726a1-0668-4443-9f4e-4c1ddc5c0b1d%40googlegroups.com.

[google-appengine] Using app engine for a SAAS with custom domains and SSL

[tom berwick]

Hi,

I'm currently in the process of putting together a SAAS offering that will 
give users a unique url e.g app.com/username

I want the ability to allow them to use their own custom domain. I 
obviously want to use SSL/https for this but also require as little work as 
possible on the users part. Possibly only updating some DNS records such as 
a cname etc.

For the certificates I planned to use LetsEncrypt and by the looks of 
things I will need to use a separate server to issue the certificates and 
then use the REST api to send them to gcloud. However one issue I will run 
into is that I won't be able to verify the custom domains of the other 
users. I could, in theory create each user their own subdomain e.g 
username.app.com for them to cname to if this makes things easier? Or 
should I actually use A records and point them to my app engine instance 
that way?

Is there a way to do this with App Engine? I'm not all up to date with 
containers etc. and App Engine is perfect for me in terms of the scaling 
factor without having to worry about infrastructure. But I'm wondering if 
I'm barking up the wrong tree in trying to achieve what I want.

I think I'll have to use DNS validation from a LetsEncrypt side as well as 
HTTP validation won't work as I won't be able to create the required 
challenge files on the app engine server either for each domain.

Has anyone done anything similar?

Otherwise my only other idea is having a single server responsible for 
creating the certs which can template the virtual hosts, but even then I 
don't think the cname redirects will work for all the custom domains on App 
engine as I won't be able to verify them.

I would love to know how things like app engine and firebase actually 
achieve this.

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/e48c29dd-f751-497b-8181-99b73f110dc5%40googlegroups.com.

[google-appengine] Where to find my repo?

[Tom Williams]

This is probably really obvious to a lot of you, but I can't figure out 
where to find the address for my app's repo.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/b594cf60-ecb3-43f8-bfa8-ae4f4dc717e0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] my website is down

[Alpintom Tom]

Hi there,

My website now is down. I do not know what happened, I can not connect via 
FTP as well. Could you please help me about that?
My website: www.alpinpon.com

Thank you

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/947613d0-bd65-4b8d-8ce4-3338923f697a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Autoscaling with Flex

[Tom Stuart]

Hi Yanik, 

Do you have any examples of bumping up the number of instances using cron 
jobs?

Thanks
Tom

On Thursday, October 12, 2017 at 5:03:55 PM UTC+1, Yannick (Cloud Platform 
Support) wrote:
>
> Hello Patrick,
>
> As can be seen on the high-level feature comparison chart 
> <https://cloud.google.com/appengine/docs/the-appengine-environments#comparing_high-level_features>,
>  
> App Engine Flexible is indeed supposed to be slower than Standard. That is 
> because you are booting up an entire VM as opposed to a simple container.
>
> You can certainly keep a number of instances up and ready to take requests 
> using cron jobs but there will always come a time where your application 
> needs to scale up due to receiving too much new traffic and where cold 
> starts will occur. The scheduler attempts to reduce the incidences of cold 
> starts by scaling up the number of instances quickly to handle traffic 
> spikes. And yes, in the end a Go application would perform faster than a 
> Java application.
>
> On Thursday, October 12, 2017 at 8:31:47 AM UTC-4, Patrick Jackson wrote:
>>
>> Hi Folks,
>>
>> In regards to autoscaling with the flex environment, what sort of spin up 
>> time can be expected?  In other words, once the scheduler decides to create 
>> new instances, how long before that instance is ready for traffic?  I am 
>> working with Java 8 with a small - medium sized application, but insights 
>> from other runtimes are welcome.
>>
>> We've tried GAE Standard with Java 8, however have been completely 
>> dissatisfied with the autoscaling in the Java environment.  Start up on new 
>> instances take 9+ seconds, and there does not appear to be a way to keep 
>> user facing requests from hitting a cold start.  I've tested with resident 
>> instances and cron jobs for "always on", but just does not appear to be 
>> possible.  Plenty of other antedotes in this group to assure me that 
>> preventing a user facing request from hitting a cold start (or heavy 
>> latency due to startup not complete) is simply not possible on GAE 
>> Standard.  If I'd know this up front, I would have choosen Go, or python.  
>> If Google would be more up front about this situation it would help many 
>> devs and the GAE as a whole.  
>>
>> So investigation is now if the flex environment can serve traffic without 
>> a user facing high latency due to an instance starting up.  Also curious in 
>> the time it takes to start instances.  One speaker at Cloud Next said a 
>> couple of minutes, which is much higher than GAE standard.
>>
>> Patrick Jackson
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/e5ed1e49-3d37-423e-a419-1142eb042119%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] App Engine Flexible VM - Minimum instances

[Tom Stuart]

As per the docs, each instance restarts weekly for security updates.

Does that therefore mean in order to achieve 100% uptime, we must maintain 
2 instances running at all times?
Or does App Engine work this out, and start a new instance up before 
shutting down the other instance?



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/24fb91c0-de75-4939-93d0-1d5f3d0cda4d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Flexible VM - Weekly reboot

[Tom Stuart]

What is the minimum number of instances required for App Engine Flexible to 
maintain 100% uptime?


As per the documentation:

All flexible instances are restarted on a weekly basis. During restarts, 
critical, backwards-compatible updates are automatically rolled out to the 
underlying operating system. Your application's image will remain the same 
across restarts.

Does this mean that the minimum number of instances is 2?
Or would I be able to maintain 100% uptime using just 1 instance and Google 
App Engine starts a new instance to deal (and I pay for 2 instances for a 
short period of time)?


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/5fdbde58-69b5-49d0-b7ee-7a34b8d9e64f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Google App Engine is slow to deploy, hangs on "Updating service [someproject]..."

[Tom Jowett]

Thanks DeWitt!

I just signed up using that form for my app - is this still slated for 
release this month?

Cheers!
Tom

On Tuesday, December 12, 2017 at 7:37:43 AM UTC-10, DeWitt Clinton wrote:
>
> Hi all,
>
> I’m on the App Engine team and I agree, flexible deployments are currently 
> too slow. The observations in this thread generally match our own: 
> Deploying a new version of an app takes around six minutes median (50th 
> percentile), with tail spikes several times that (99th percentile). 
> Language choice or app size isn’t a factor, as these setup costs are 
> independent of the runtime or code being deployed. 
>
> For many apps, where production versions are pushed via long-running, 
> asynchronous CI/CD pipelines, the exact timing may not matter so much. But 
> for everyone who depends on app deployments for development purposes, this 
> is too much.
>
> For those cases, we are testing an improved deployment flow that will let 
> you update an existing version in-place in around 150 seconds median. We 
> hope to release the fast in-place updates for everyone next month, but if 
> you’d like to help us test it out a little sooner please tell me your 
> project id here 
> <https://docs.google.com/forms/d/e/1FAIpQLSd_IpiUKjWVAmRLVjKBEkYoYSiLtYg6tm3GKV2gRjNEnp-UrA/viewform>
>  
> and I’ll whitelist it.
>
> If you miss this signup, don’t worry, we plan to turn faster in-place 
> updates on for all projects by default in the new year.
>
> Thanks!
>
> -DeWitt
>
>
> On Tuesday, November 21, 2017 at 12:46:32 PM UTC-8, Gunar Cassiano Gessner 
> wrote:
>>
>> Speaking from experience here.
>> Just move on.
>> Google App Engine (esp. Flexible Engine) is not worthy of your time.
>>
>> Take care,
>> @gunar
>>
>> On Tue, Nov 21, 2017 at 1:24 AM, Vijay M  wrote:
>>
>>> I have been struggling with the same issue. I tried deploying a very 
>>> simple app and waited for several minutes before the process timed out. I 
>>> even ran into the same timeout issue with Google's own samples from  
>>>
>>> https://github.com/GoogleCloudPlatform/python-docs-samples/
>>> appengine/flexible/hello_world
>>>
>>> Coming from AWS, it has been very frustrating working with GCP. One more 
>>> day of this and I am going back to AWS. 
>>>
>>> On Thursday, September 29, 2016 at 12:48:26 PM UTC-7, Kevin Lau wrote:
>>>>
>>>>
>>>> Is there a faster way to deploy an app? Google App Engine is slow to 
>>>> deploy, hangs on "Updating service [someproject]..." I am using a flexible 
>>>> environment with PHP.
>>>>
>>> -- 
>>> You received this message because you are subscribed to a topic in the 
>>> Google Groups "Google App Engine" group.
>>> To unsubscribe from this topic, visit 
>>> https://groups.google.com/d/topic/google-appengine/hZMEkmmObDU/unsubscribe
>>> .
>>> To unsubscribe from this group and all its topics, send an email to 
>>> google-appengi...@googlegroups.com.
>>> To post to this group, send email to google-a...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/google-appengine.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/google-appengine/0c5f7fed-3551-4426-88e4-1383b4151084%40googlegroups.com
>>>  
>>> <https://groups.google.com/d/msgid/google-appengine/0c5f7fed-3551-4426-88e4-1383b4151084%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/51f43aa1-1cfc-447d-8093-bdfac4768ee1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Custom runtime - using search API / AppEngine services

[Tom Walder]

I guess you could deploy a PHP/Python/Standard runtime app to just expose a 
RESTish API to the Search service.

I've done this before.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/7f700818-c47c-40c4-b880-d40aa4ec0973%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] PHP Support for Datastore REST API v1

[Tom Walder]

Just released, version 3 of PHP-GDS @GoogleCloud #Datastore Library for 
#PHP and #AppEngine Support for the new v1 REST API. 

https://github.com/tomwalder/php-gds

Disclaimer: I am the author

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/7e2395c2-3e8c-4054-8681-b87e08fc7623%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Google App Engine multiple regions

[Tom Walder]

You cannot load balance between US and EU using "conventional" load 
balancing.

There might be some thoroughly complex way to do it with localised DNS 
responses.

However, it would be 2 distinct App Engine projects.

To be honest if cross-region load balancing is a need, App Engine is 
probably not the right solution. Look at Compute engine instead.

Tom




On Tuesday, 21 June 2016 06:36:26 UTC+1, Yon Dev wrote:
>
> Hello Nick,
>
> Thank you for your answer.
>
> I might be misunderstanding something so please correct me if I am wrong.
>
> I can only upload 1 app engine instance per project. So if I want to 
> deploy to both the US and EU I will have to create 2 different projects. 
> And if I were to create multiple projects, how should I handle my custom 
> domain?
>
> To have a single domain I believe I have to use a load balancer, but I 
> thought the HTTP load balancer was only available when using Compute Engine 
> and not for App Engine?
> And if I were to have to have 2 separate projects I don't think it's 
> possible to do load balancing between them, right?
>
> Sincerely,
> Yon
>
> Den tisdag 21 juni 2016 kl. 03:16:11 UTC+9 skrev Nick (Cloud Platform 
> Support):
>>
>> Hey Yon,
>>
>> App Engine instances are scaled in zones within the region the 
>> application is created in 
>> <https://cloud.google.com/appengine/docs/python/console/#create>. If 
>> you'd like to follow a pattern which others have made use of, you can 
>> deploy another project in the EU to handle traffic there. As you point out, 
>> an HTTP Load Balancer is a very useful in this situation, and you can use 
>> Container 
>> Engine <https://cloud.google.com/container-engine/> to deploy Docker 
>> images of your applications, using Custom Runtimes 
>> <https://cloud.google.com/appengine/docs/flexible/custom-runtimes/>.
>>
>> I hope this has helped answer your question. Let me know if you have any 
>> further questions, and I'll be happy to assist.
>>
>> Sincerely,
>>
>> Nick
>> Cloud Platform Community Support
>>
>> On Monday, June 20, 2016 at 5:13:12 AM UTC-4, Yon Dev wrote:
>>>
>>> I am evaluating whether GAE would be a good fit for an upcoming project.
>>>
>>> I was wondering how scaling across regions work with GAE? If I set my 
>>> project location to US and then create an GAE instance and most access to 
>>> the GAE is from EU will it automatically scale out to Europe? 
>>> Or how does it work?
>>> If it was a on Compute Engine I believe I could have created multiple 
>>> instances and then made a HTTP Load Balancer that would route the user to 
>>> the closest instance based on proximity.
>>>
>>> https://cloud.google.com/about/locations/
>>> Reading the link above it says that it scales automatically within or 
>>> accross multi-regional locations.
>>> What does that mean? Do I have to enable automatic scaling across 
>>> regions? If so, how do I do that?
>>> And secondly, if it does handle automatic scaling across regions, the 
>>> choice one makes for App engine location when creating a new project, is 
>>> that irrelevant for a Google App Engine instance?
>>>
>>> I tried asking on Stackoverflow but without answer so I am turning to 
>>> you in hope for an answer.
>>>
>>> http://stackoverflow.com/questions/37849433/google-app-engine-multiple-regions
>>>
>>> Thank you in advance,
>>> Yon
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/ef9ceaa1-91e8-4202-9d23-fb7a6f184314%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Custom Domains (and SSL)

[tom saffell]

Hi Ryan, 

I am trying follow these instructions, but when I click on the 'here' after 
'step 1' it shows me an empty screen, like this:
  http://imgur.com/ZKhv46j

Is this a bug? What can I do to get around it?
The url is takes me to is:
  
https://console.cloud.google.com/appengine/settings/domains/add?project=[my-app-id]

thanks

tom

 

On Wednesday, September 30, 2015 at 6:24:44 AM UTC-7, Ryan (Cloud Platform 
Support) wrote:
>
> Glad to hear your good to go!
>
> On Tuesday, September 29, 2015 at 10:24:49 AM UTC-4, Jay wrote:
>>
>> I'm not sure if this question is best here or Stackoverflow. 
>> I am looking to use the feature recently moved to the google cloud 
>> console to add an SSL certificate. We are already using a custom domain.
>> Here is the new thing I ran into. When I log in and look at a particular 
>> project, on which we already have a custom domain setup, I see nothing. It 
>> looks like we have not set anything up. A colleague of mine logs in and 
>> does the same thing, on the same project, and he sees a custom domain 
>> already configured. This is before we get to the stage of setting up the 
>> SSL.
>>
>> So we are seeing two different things. This makes me nervous. It also 
>> means that I can't, apparently, setup the SSL, he would have to do that. 
>> Surely, the configuration here is not tied to the user in some way. I must 
>> be missing something.
>>
>> Thanks for any insight or suggestions.
>>
>> -- Jay
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/ae099400-429e-4c0c-9b38-2e8e6241b0a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Handling user logins with AppEngine PHP and Datastore?

[Tom Walder]

Hi there.

You might want to consider something like one of these
https://github.com/ircmaxell/password_compat
https://github.com/hautelook/phpass

Which do a reasonable job of one way password encryption and then validation.

You'll likely need sessions too - and you tweak the session hash if you want to 
with these options
http://php.net/manual/en/session.configuration.php

Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/cf5674a0-0e7c-45eb-a899-f6e6e94b79f0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Geospatial/geolocation queries from GAE (or other Cloud products)?

[Tom Walder]

Hi Alex,

Geocoding options are many.

I use the Royal Mail PAF data (commercial). Also, there are options like

http://postcodes.io/

Which I have not used, but looks pretty interesting.

There is no restriction on the volume of data you put into the Search API - 
although you will start to pay if you index or store over the free quota.

https://cloud.google.com/appengine/docs/quotas?hl=en#search

Best of luck!

On Wednesday, 6 January 2016 13:54:11 UTC, Alex Kerr wrote:
>
> Thanks Tom, I much appreciate these libraries which I already planned to 
> use, but wasn't quite sure how to layer my requirements on top of them - I 
> think the primary issue is how to have many records of people that I can 
> manipulate easily (e.g. in DataStore, Search API is a bad match for this I 
> think?) and also work out the nearest other people to them based on a human 
> readable address (or postcode at least). Do you recommend or use any 
> particular methods for Geocoding? As mentioned by barryhunter above use of 
> the Maps APIs requires display of, and results put on, a Google Map, which 
> is not necessarily what I want. These guys look like a potential 
> alternative tho: http://geocoder.opencagedata.com/
>
> Thanks again,
> Alex
>
> On Tuesday, January 5, 2016 at 7:26:29 PM UTC, Tom Walder wrote:
>>
>> Hi Alex,
>>
>> You can do all of this on App Engine, with PHP using the native Search 
>> API.
>>
>> I usually keep my transactional data in Datastore and then the "searchy" 
>> stuff, including Geo search in the Search API (when I'm, not using 
>> Elasticsearch).
>>
>> You will need one or more of these open source libraries (which I wrote 
>> [disclaimer)) - They are using the native Google Protocol Buffers API, same 
>> as the Python/Java/Go runtimes, so they are super quick.
>>
>> For Datastore
>> https://github.com/tomwalder/php-gds
>>
>> For Search API - there are specific examples and helper methods for Geo.
>> https://github.com/tomwalder/php-appengine-search
>> http://pub-search.appspot.com/
>>
>> Some more reading
>>
>> https://gae-php-tips.appspot.com/2015/12/19/making-cloud-datastore-easy-for-php-on-app-engine/
>>
>> Enjoy!
>>
>> Tom
>>
>>
>>
>>
>>
>> On Monday, 4 January 2016 20:50:44 UTC, Alex Kerr wrote:
>>>
>>> Hi all,
>>> Wondering how best to implement some basic geolocation stuff using 
>>> Google Cloud products (and open source if appropriate)
>>>
>>> I need to search user records I'm storing in Datastore, which include 
>>> the user's address (in ordinary human readable text format) to see which 
>>> other users are within a certain distance from them. Seems I can't do a 
>>> geospatial search directly on Datastore records but found something that 
>>> suggests a clever little mashup with Google's Search API: 
>>> http://ralphbarbagallo.com/2013/05/14/app-engine-geospatial-datastore-search-a-new-way/
>>>
>>> Questions:
>>> 1.) Is there any better (/more standard) way than what's suggested at 
>>> that link? (I know Cloud SQL allows geospatial searches but ideally I want 
>>> to use Datastore if possible to store my data in)
>>> 2.) Geospatial matches (at least using above techniques) need latitude 
>>> and longitude - are there any Google APIs or open source PHP stuff I could 
>>> use to convert addresses into lat/lon?
>>>
>>> I'm running on PHP AppEngine by the way and need a solution I can 
>>> implement very quickly and easily if at all possible!
>>>
>>> Many thanks for any info/suggestions!
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/390e949e-9296-49a7-911d-a2db0d21f79c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Geospatial/geolocation queries from GAE (or other Cloud products)?

[Tom Walder]

Hi Alex,

You can do all of this on App Engine, with PHP using the native Search API.

I usually keep my transactional data in Datastore and then the "searchy" 
stuff, including Geo search in the Search API (when I'm, not using 
Elasticsearch).

You will need one or more of these open source libraries (which I wrote 
[disclaimer)) - They are using the native Google Protocol Buffers API, same 
as the Python/Java/Go runtimes, so they are super quick.

For Datastore
https://github.com/tomwalder/php-gds

For Search API - there are specific examples and helper methods for Geo.
https://github.com/tomwalder/php-appengine-search
http://pub-search.appspot.com/

Some more reading
https://gae-php-tips.appspot.com/2015/12/19/making-cloud-datastore-easy-for-php-on-app-engine/

Enjoy!

Tom





On Monday, 4 January 2016 20:50:44 UTC, Alex Kerr wrote:
>
> Hi all,
> Wondering how best to implement some basic geolocation stuff using Google 
> Cloud products (and open source if appropriate)
>
> I need to search user records I'm storing in Datastore, which include the 
> user's address (in ordinary human readable text format) to see which other 
> users are within a certain distance from them. Seems I can't do a 
> geospatial search directly on Datastore records but found something that 
> suggests a clever little mashup with Google's Search API: 
> http://ralphbarbagallo.com/2013/05/14/app-engine-geospatial-datastore-search-a-new-way/
>
> Questions:
> 1.) Is there any better (/more standard) way than what's suggested at that 
> link? (I know Cloud SQL allows geospatial searches but ideally I want to 
> use Datastore if possible to store my data in)
> 2.) Geospatial matches (at least using above techniques) need latitude and 
> longitude - are there any Google APIs or open source PHP stuff I could use 
> to convert addresses into lat/lon?
>
> I'm running on PHP AppEngine by the way and need a solution I can 
> implement very quickly and easily if at all possible!
>
> Many thanks for any info/suggestions!
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/9e809448-cbef-4df1-a6e9-15d3927d6729%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Labelled versions added to appengine-deploy (PHP)

[Tom Walder]

For anyone using our environment and deployment manager tool, we've release 
v1.1 with a couple of key changes

1. Support for labelled versions - make it easier to recognise version 
purpose (e.g. "version3-feature")
2. Change to command line parameters (- and -- syntax)

https://github.com/Venditan/appengine-deploy

Cheers,

Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/7f4f3461-abd7-4602-b7d3-c3e1937ef5cb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Announcing SSL for GAE Custom Domains in the Developers Console

[tom saffell]

Thank you for making the change - it's much appreciated!

I tried to add our naked domain. It instructed me to add some type A DNS 
records, with host "@".
I tried adding the records on name.com (our registrar and DNS provider) and 
got an error "The Record Host field is invalid."

Anyone have this working with name.com?

thanks

tom


On Tuesday, September 15, 2015 at 5:10:30 PM UTC-7, Lorne Kligerman wrote:
>
> At long last, we are happy to announce that your App Engine custom domains 
> and SSL security can now be configured within the Google Developers 
> Console 
> <https://console.developers.google.com/project/_/appengine/settings/certificates>.
>  
>  
>
> For many historical reasons, this functionality was split between the 
> Developers Console and the Google Apps Admin console.  With this change, 
> the Apps Admin console is no longer needed which will make your setup much 
> more simple and straightforward.
>
> Rest assured that all existing domains and certificates will continue to 
> serve as normal.
>
> If you have an existing App Engine application serving over a custom 
> domain and SSL certificate, you will have to verify ownership of your 
> domain in the Developers Console here 
> <https://console.developers.google.com/project/_/appengine/settings/domains/add>
> .  Once verified, you will see all of your existing certificates and the 
> domain to which they belong.
>
> For those just getting started, simply follow the instructions on the Custom 
> Domains and SSL Certificates tabs.  For more details you can check out 
> the full documentation here 
> <http://cloud.google.com/appengine/docs/using-custom-domains-and-ssl>.
>
>
> This is just the beginning of a larger project that is under way to help 
> you quickly get your app online, brand it properly with your own domain, 
> make it as secure as possible, and manage the setup with ease among your 
> organization, however large or small.
>
> Cheers and thanks for your patience,
> The Cloud Custom Domains SSL Team.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/33c360f6-316c-4063-abec-30fbad5b91f7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Multi-Environment & Deployment Manager for App Engine / PHP

[Tom Walder]

 

Just open sourced this today... Hope someone finds it useful.


https://github.com/Venditan/appengine-deploy


*What is this? Is it different from appcfg.py?*


This allows you to *deploy the same code/application to multiple target 
environments* (local, multiple different App Engine projects).


Critically, it also allows you to *manage environment variables distinctly 
for each deployment target*.


This tool uses appcfg.py to actually push the deployments out, but it 
builds a dynamic command, overriding the target application id and 
environment variables at deploy time.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/fa04d658-2b39-4a05-b118-7fbe35680e94%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Native access to Search API from PHP on App Engine - Get it here!

[Tom Walder]

Added support for auto-complete Edge NGram tokenizers, Match Scorers, Geo 
distance queries, Unit tests

Still in alpha.


On Thursday, 27 August 2015 22:42:24 UTC+1, Tom Walder wrote:
>
> Hi all,
>
> For anyone that's interested, I've published an alpha version of a library 
> for accessing the App Engine Search API.
>
> At the time of writing, this is only available from Python/Java/Go with 
> the SDK's provided by Google.
>
> So, here it is for those of us using PHP.
>
> https://github.com/tomwalder/php-appengine-search
>
> It uses the low-level Protocol Buffer access - just like the 
> Python/Java/Go implementations.
>
> Enjoy - feedback appreciated!!
>
> Tom
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/aada2ae2-d265-4faa-ac44-56221ba16c62%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Native access to Search API from PHP on App Engine - Get it here!

[Tom Walder]

Hi all,

For anyone that's interested, I've published an alpha version of a library 
for accessing the App Engine Search API.

At the time of writing, this is only available from Python/Java/Go with the 
SDK's provided by Google.

So, here it is for those of us using PHP.

https://github.com/tomwalder/php-appengine-search

It uses the low-level Protocol Buffer access - just like the Python/Java/Go 
implementations.

Enjoy - feedback appreciated!!

Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/c10e72b8-1bee-4fe2-8b59-ddccc632fb0f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Understanding the value proposition of App Engine for data processing

['Tom Kaitchuck' via Google App Engine]

I think you may be more interested in Cloud Dataflow:
https://cloud.google.com/dataflow/what-is-google-cloud-dataflow

On Tue, Aug 11, 2015 at 11:30 AM, Duo Zmo  wrote:

> I'm just digging into map reduce on Google App Engine, and my early
> results are discouraging. I had in mind that I'd process about 10GB of data
> for an analysis I want to do, and I didn't even think that'd be that big a
> deal (given all the talk about petabyte-scale storage and such), but it's
> currently looking impossible.
>
> I did a simple word count mapreduce on some Gutenberg books (63MB zipped,
> 166MB unzipped), once using Google's Python mapreduce example (
> https://cloud.google.com/appengine/docs/python/dataprocessing/) and once
> using the dumb-as-rocks standalone Python scripts posted at the top of
> Michael Noll's Hadoop tutorial (
> http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/
> ).
>
> Experimental results:
>
> Simple Python: 1 minute 22 seconds
> GAE dev server: 2 hours 17 minutes 12 seconds
>
> Given the staggering difference in run time, even if computation in the
> cloud were free, I'd still opt to compute locally unless my hand were
> forced somehow (e.g. input files that didn't fit on my disk). Of course,
> the computation is not free, which means you're not only enduring all that
> overhead, but paying for it too.
>
> I did try running this same test "in production," i.e. on Google Cloud's
> infrastructure. At first it failed, because just getting the job started
> exceeded the 128MB memory limit for the free tier. I turned on billing,
> bumped up the instance class to F4, and let it go. It chewed through the
> free tier quickly, then about USD$8 of instance time before one of the
> shuffle-merge shards seemed to enter an infinite loop (ran for 2 hours, no
> errors in logs). I aborted and gave up at that point.
>
> Everything I hear about cloud computing makes it sound like the gleaming,
> glossy future, but these results makes it seem expensive and slow. $8+ to
> do a mapreduce across 60MB of data just doesn't seem like a good deal to
> me. At that rate, there's no way I can afford to process my 10GB dataset on
> App Engine. I understand that with the pipeline model you get fault
> tolerance and status reports and basic job management, but none of that is
> worth the expense or a 100x performance hit.
>
> I think there's two possible problems going on here:
>
> 1) I made a technical mistake in my experiment and my results are invalid
> 2) I'm not understanding the benefits / value proposition of App Engine
>
> Are my results consistent with what others would expect? Do either or both
> of my candidate explanations ring true? What else am I not considering? I
> think this is more a discussion topic than a discrete ask-and-answer, which
> is why I'm posting here instead of Stack Overflow.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/467aec55-e518-40b8-81b4-d62fb54a3dcb%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAN4PiZERboinWnN65N16b0CA2O%3Dx%2BUbhdB%3DDL52s7VB-oeVhww%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] PHP Library for Cloud Datastore 2.0 Production Release

[Tom Walder]

Great - I'd be really interested to find out how you get on with running it 
on AppScale.

On Wednesday, 5 August 2015 08:44:51 UTC+1, Anthony Shapley wrote:
>
> Hi Tom,
>
> Thanks for coming back to me - thats really awesome, I'll give it a test 
> and let you know. Thanks so much for sharing this I've just started 
> rewriting my entire code base into Python after writing everything in PHP 
> (using your original library).
>
> This means I can potentially just carry on as is!
>
> Ant
>
> On Wednesday, 5 August 2015 08:24:24 UTC+1, Tom Walder wrote:
>>
>> Yes, I believe you could run this on AppScale.
>>
>> Whilst I have not done so myself, my understanding is that AppScale 
>> exposes the same Protocol Buffer APIs that AppEngine does.
>>
>>
>>
>> On Wednesday, August 5, 2015, Anthony Shapley  wrote:
>>
>>> Hi Tom,
>>>
>>> This sounds excellent - just so I am nor misreading, this no longer 
>>> requires the Google API, so could this be used on AppScale for example?
>>>
>>> Ant
>>>
>>> On Tuesday, 4 August 2015 22:30:55 UTC+1, Tom Walder wrote:
>>>>
>>>> https://github.com/tomwalder/php-gds
>>>>
>>>> Enjoy!
>>>>
>>>> New features in 2.0 include 
>>>>
>>>>- *Faster* - use of Google Protocol Buffer allows faster, low-level 
>>>>access to Datastore
>>>>- *Easier to use* - sensible defaults and auto-detection for 
>>>>AppEngine environments
>>>>- *Less dependencies* - no need for the Google PHP API Client, 
>>>>unless running remote or from non-AppEngine environments
>>>>- *Local development* - Using the Protocol Buffers allows us to 
>>>>access the development server Datastore
>>>>- *Local GQL support* - By default, the local development server 
>>>>does not support GQL. I've included a basic GQL parser that makes this 
>>>> work.
>>>>- *Data Migrations* - leverage multiple Gateways to ship data 
>>>>between local and live Datastore
>>>>- *Contention Exceptions* - standardised Exception for handling 
>>>>Datastore transaction contention`
>>>>- *Unit tests*
>>>>- Optional drop-in JSON API Gateway for remote or non-AppEngine 
>>>>environments (this was the only Gateway in 1.x)
>>>>
>>>> -- 
>>> You received this message because you are subscribed to a topic in the 
>>> Google Groups "Google App Engine" group.
>>> To unsubscribe from this topic, visit 
>>> https://groups.google.com/d/topic/google-appengine/VUBSVMW4V84/unsubscribe
>>> .
>>> To unsubscribe from this group and all its topics, send an email to 
>>> google-appengine+unsubscr...@googlegroups.com.
>>> To post to this group, send email to google-appengine@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/google-appengine.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/google-appengine/34ae4d07-9e5a-499b-8f0f-439e54dac3fd%40googlegroups.com
>>>  
>>> <https://groups.google.com/d/msgid/google-appengine/34ae4d07-9e5a-499b-8f0f-439e54dac3fd%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>> -- 
>> *Tom Walder*, CTO*Call*: 0845 521 0444 / *Web*: www.docnet.nu 
>> <http://www.docnet.nu/?utm_source=email_signature&utm_medium=email&utm_campaign=email_signature>This
>>  
>> message is private and confidential. If you have received this message in 
>> error, please notify us and remove it from your system. Venditan Limited 
>> t/a Docnet is a company registered in England and Wales. Registered number: 
>> 9604502. Registered office: Speakers House, 39 Deansgate, Manchester, M3 2BA
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/c9ab3324-b45b-4ea3-a33f-e810386cf999%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] PHP Library for Cloud Datastore 2.0 Production Release

[Tom Walder]

Yes, I believe you could run this on AppScale.

Whilst I have not done so myself, my understanding is that AppScale exposes
the same Protocol Buffer APIs that AppEngine does.



On Wednesday, August 5, 2015, Anthony Shapley  wrote:

> Hi Tom,
>
> This sounds excellent - just so I am nor misreading, this no longer
> requires the Google API, so could this be used on AppScale for example?
>
> Ant
>
> On Tuesday, 4 August 2015 22:30:55 UTC+1, Tom Walder wrote:
>>
>> https://github.com/tomwalder/php-gds
>>
>> Enjoy!
>>
>> New features in 2.0 include
>>
>>- *Faster* - use of Google Protocol Buffer allows faster, low-level
>>access to Datastore
>>- *Easier to use* - sensible defaults and auto-detection for
>>AppEngine environments
>>- *Less dependencies* - no need for the Google PHP API Client, unless
>>running remote or from non-AppEngine environments
>>- *Local development* - Using the Protocol Buffers allows us to
>>access the development server Datastore
>>- *Local GQL support* - By default, the local development server does
>>not support GQL. I've included a basic GQL parser that makes this work.
>>- *Data Migrations* - leverage multiple Gateways to ship data between
>>local and live Datastore
>>- *Contention Exceptions* - standardised Exception for handling
>>Datastore transaction contention`
>>- *Unit tests*
>>- Optional drop-in JSON API Gateway for remote or non-AppEngine
>>environments (this was the only Gateway in 1.x)
>>
>> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Google App Engine" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/google-appengine/VUBSVMW4V84/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> google-appengine+unsubscr...@googlegroups.com
> 
> .
> To post to this group, send email to google-appengine@googlegroups.com
> .
> Visit this group at http://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/34ae4d07-9e5a-499b-8f0f-439e54dac3fd%40googlegroups.com
> <https://groups.google.com/d/msgid/google-appengine/34ae4d07-9e5a-499b-8f0f-439e54dac3fd%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>


-- 
*Tom Walder*, CTO*Call*: 0845 521 0444 / *Web*: www.docnet.nu
<http://www.docnet.nu/?utm_source=email_signature&utm_medium=email&utm_campaign=email_signature>This
message is private and confidential. If you have received this message in
error, please notify us and remove it from your system. Venditan Limited
t/a Docnet is a company registered in England and Wales. Registered number:
9604502. Registered office: Speakers House, 39 Deansgate, Manchester, M3 2BA

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAO_O9QBCgnSFXpFoipWo8q7uJC87KM9DtG0DYDHcz9hyXSOGrg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] PHP Library for Cloud Datastore 2.0 Production Release

[Tom Walder]

https://github.com/tomwalder/php-gds

Enjoy!

New features in 2.0 include 

   - *Faster* - use of Google Protocol Buffer allows faster, low-level 
   access to Datastore
   - *Easier to use* - sensible defaults and auto-detection for AppEngine 
   environments
   - *Less dependencies* - no need for the Google PHP API Client, unless 
   running remote or from non-AppEngine environments
   - *Local development* - Using the Protocol Buffers allows us to access 
   the development server Datastore
   - *Local GQL support* - By default, the local development server does 
   not support GQL. I've included a basic GQL parser that makes this work.
   - *Data Migrations* - leverage multiple Gateways to ship data between 
   local and live Datastore
   - *Contention Exceptions* - standardised Exception for handling 
   Datastore transaction contention`
   - *Unit tests*
   - Optional drop-in JSON API Gateway for remote or non-AppEngine 
   environments (this was the only Gateway in 1.x)

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/cd36a07b-6cd9-434b-aed5-34cabe230b08%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] UrlFetch DeadlineExceededError

[Michael Tom]

Hey guys,

I'm trying to send a bunch of data through post requests using appengine rpc, 
however I keep getting DeadlineExceededError's after 5-7 seconds. The calls are 
launched within a deferred task, and then entire deferred task ends up taking 
upwards of a minute. I'm sending requests from multiple threads within the task.

I'm using this Send method, and passing a timeout of 60 seconds, but something 
isn't listening to this:

https://codesearch.corp.google.com/p#piper///depot/google3/apphosting/tools/appengine_rpc.py&rcl=98602889&l=337
 


And getting this in the logs: 

API call urlfetch.Fetch() took too long to respond and was cancelled.


Any help appreciated!

Cheers,

Michael Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/923d777f-63ca-4dd3-b92a-2884f326befb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] MapReduce not stopping MR controllers when completed - frontend charges increasing in bill

['Tom Kaitchuck' via Google App Engine]

I think this may be your problem:
https://github.com/GoogleCloudPlatform/appengine-mapreduce/issues/69

On Wed, Jul 15, 2015 at 9:03 AM, Camilo Silva 
wrote:

> So I've been working on Map Reduce Google library for quite a while on
> some Python App Engine projects. And to this day, I cannot comprehend why
> there are "mrcontrollers" alive doing callbacks everytime right after all
> processing is done (i.e., all shards terminated successfully). I always
> have to go back and purge the taskqueue so that unnecessary frontend calls
> are stopped -- this is an issue because the billing is affected due to
> this.
> Any feedback is welcomed.
> Thanks for your help.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/cc62f26e-8277-422e-968b-2dd5e662c17a%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAN4PiZF1otn%3DyhOk_Dfy5C85LBrEjwWdDpGJwa7PjP-OQ9dwcg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Quotas maxed, app nearly dormant. Looking for troubleshooting pointers! Thanks.

[Tom Mitchell]

Thanks to you both for responding. This must be what I was seeing because I 
didn't really change anything and things are more as I would expect.

Thanks again.

On Tuesday, July 14, 2015 at 11:33:14 AM UTC-4, Patrice (Cloud Platform 
Support) wrote:
>
>  Earlier today, a number of users observed incorrect usage statistics 
> being displayed by the App Engine dashboard in the Google Cloud Developer 
> Console.  Some resources, especially Frontend Instance Hours, were shown 
> with unrealistically elevated usage.
>
> The dashboard has been returned to normal operation and our engineers are 
> investigating to find and fix the root cause of the issue.  So far, the 
> results of our investigation indicate that the issue was purely cosmetic, 
> with no elevated billing or quota denials to any user application.  If your 
> application encountered quota denials because of this issue, if you later 
> observe elevated resource billing in your invoice for this month, or if you 
> are still observing unrealistic usage in the App Engine dashboard, please 
> notify Google Cloud Platform Support.
>
> On Monday, July 13, 2015 at 6:41:38 PM UTC-4, pdknsk wrote:
>>
>> If the app is not actually having quota denials, I wouldn't worry about 
>> it. Probably just a glitch in the dashboard. The true authoritative quota 
>> numbers will be in the usage report for today, posted tomorrow.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/82d26d8d-d26e-4a95-8bfb-c77e6f30f866%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Quotas maxed, app nearly dormant. Looking for troubleshooting pointers! Thanks.

[Tom Mitchell]

Hello,

I am just starting work on an application and I am getting strange quota 
#'s (my application has not been shut down yet though). I am not sure how 
to best represent the information for feedback here but I will try.

1. My application does not serve any advertised or public web pages, though 
there is one for testing.

2. The intent is that once per minute, two devices will upload an very 
small XML document for a GAE-hosted service to consume. Right now, all the 
app does is log that it received the data. No storage, presentation, or 
response. That seems to be all the activity taking place now, according to 
the logs. The service only returns an HTTP response. When I look at the 
logs for my app, that is exactly what I see: two incoming requests per 
minute. I scanned all the way back to last midnight PT and that's all there 
is.




3. I glanced at my quotas and they all seem OK, except three: requests, 
instance hours and outgoing bandwidth. Oddly, I have a second, nearly 
dormant app that shows almost the exact same numbers. All it does is make 
an outbound request every 15 minutes. A look at its logs shows that it is 
behaving as expected.




I have checked on my bandwidth usage from the two locations that are 
talking to this GAE app, and there is nothing like the activity I am seeing 
on the quotas page.


I can assume the issue is on my side, but I would like to know how to 
understand the issue so I can fix it.


Am I in the right place? Do I need to contrast the question differently?


Thanks in advance.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/32348490-0831-4720-862a-e7db3b80fe17%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Datastore Library for PHP in 2.0 beta (php-gds)

[Tom Walder]

Wow, thanks for the vote of confidence!

I certainly hope it helps some people out - Datstore has a lot to offer, and 
making it more accessible to PHP developers is what this library is all about!

Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/24ced923-38b7-4746-af62-1838ce0dbe5f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Datastore Library for PHP in 2.0 beta (php-gds)

[Tom Walder]

Hi there,

For anyone that is interested or wants to help with beta testing...

Version 2.0 of the PHP-GDS library for accessing Datastore from PHP on App 
Engine (or not) is in beta.

https://github.com/tomwalder/php-gds

Here's a copy of the "New in 2.0" section:


New features in 2.0 include 

   - *Faster* - use of Google Protocol Buffer allows faster, low-level 
   access to Datastore
   - *Easier to use* - sensible defaults and auto-detection for AppEngine 
   environments
   - *Less dependencies* - no need for the Google PHP API Client, unless 
   running remote or from non-AppEngine environments
   - *Local development* - Using the Protocol Buffers allows us to access 
   the development server Datastore
   - *Local GQL support* - By default, the local development server does 
   not support GQL. I've included a basic GQL parser that makes this work.
   - *Data Migrations* - leverage multiple Gateways to ship data between 
   local and live Datastore
   - *Unit tests*



Thanks in advance!

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/0cfb5211-f634-4376-9d03-c084b3704a82%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Saving an entity from a Reducer invoking my DAO (DatastoreService.put)

['Tom Kaitchuck' via Google App Engine]

Correct. You still have to deal with slice / shard retries. But it should
work fine.

On Wed, Jul 8, 2015 at 11:58 PM, Antonio Fornié Casarrubios <
antonio.for...@gmail.com> wrote:

> Thanks Tom. So it's that all? The only difference/disadvantage of this
> approach would be to have less performance?
>
>
> El miércoles, 8 de julio de 2015, 20:52:25 (UTC+2), Tom Kaitchuck escribió:
>>
>> The point of DatastoreMutationPool is to provide batching of updates to
>> increase throughput. It is fine to use something else.
>>
>> On Wed, Jul 8, 2015 at 7:34 AM, Antonio Fornié Casarrubios <
>> antonio...@gmail.com> wrote:
>>
>>>
>>> Hi all. I didn't find any answer or info for this:
>>>
>>> *Context*: Java Google App Engine project, using Datastore. For a
>>> certain kind, Sale, I save the entities in two ways:
>>>
>>>
>>>1. From a typical SaleDAO#save(saleEntity)
>>>2. During MapReduce, from my SaleReducer#emit(saleEntity)
>>>
>>>
>>> So I was investigating how to reuse some behavior as part of both cases,
>>> so I made something like:
>>>
>>> class SaleDAO {
>>> DatastoreService datastore
>>> public save(Entity entity) {
>>> // My common behavior that I want to reuse
>>> datastore.put(entity)
>>> // Some more common behavior that I want to reuse
>>> }
>>> }
>>>
>>>
>>>
>>> And then for the MapReduce part, I extended OutputWriter to call my DAO,
>>> instead of the DatastoreMutationPool#put(entity) method provided for that
>>> purpose.
>>>
>>>
>>> public class SaleOutputWriter extends OutputWriter {
>>> @Override
>>> public void write(Entity entity) {
>>> //pool.put(entity); - try to invoke the DAO instead
>>> saleDAO.save(sale)
>>> }
>>> }
>>>
>>>
>>>
>>>
>>>1. My surprise is that it works! Well, the MR jobs seem to go to a
>>>strange state, but the entities are correctly saved. Still I'm 99% sure
>>>that this must be a bad practice or bring some problems. Can somebody 
>>> tell
>>>me
>>>2. If that's not the case, why are we given a certain API for
>>>MapReduce and another one for normal persistence? Will be the same to
>>>invoke DatastoreMutationPool.put than DatastoreService.put (any
>>>implementation)?
>>>
>>>
>>> Honestly, this idea sounded crazy in my head and now that it seems to
>>> work I want to know more about it.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Google App Engine" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to google-appengi...@googlegroups.com.
>>> To post to this group, send email to google-a...@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/google-appengine.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/google-appengine/61276548-616e-4f94-9263-00bcfcb5e33d%40googlegroups.com
>>> <https://groups.google.com/d/msgid/google-appengine/61276548-616e-4f94-9263-00bcfcb5e33d%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/1cc85d13-ff75-4284-8e50-06b8a4eb17a5%40googlegroups.com
> <https://groups.google.com/d/msgid/google-appengine/1cc85d13-ff75-4284-8e50-06b8a4eb17a5%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAN4PiZH3Ex%2BMzkK1DJWX7qCNW2i74BVgoTVoPWbC%2Bdb3sxa4YA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Saving an entity from a Reducer invoking my DAO (DatastoreService.put)

['Tom Kaitchuck' via Google App Engine]

The point of DatastoreMutationPool is to provide batching of updates to
increase throughput. It is fine to use something else.

On Wed, Jul 8, 2015 at 7:34 AM, Antonio Fornié Casarrubios <
antonio.for...@gmail.com> wrote:

>
> Hi all. I didn't find any answer or info for this:
>
> *Context*: Java Google App Engine project, using Datastore. For a certain
> kind, Sale, I save the entities in two ways:
>
>
>1. From a typical SaleDAO#save(saleEntity)
>2. During MapReduce, from my SaleReducer#emit(saleEntity)
>
>
> So I was investigating how to reuse some behavior as part of both cases,
> so I made something like:
>
> class SaleDAO {
> DatastoreService datastore
> public save(Entity entity) {
> // My common behavior that I want to reuse
> datastore.put(entity)
> // Some more common behavior that I want to reuse
> }
> }
>
>
>
> And then for the MapReduce part, I extended OutputWriter to call my DAO,
> instead of the DatastoreMutationPool#put(entity) method provided for that
> purpose.
>
>
> public class SaleOutputWriter extends OutputWriter {
> @Override
> public void write(Entity entity) {
> //pool.put(entity); - try to invoke the DAO instead
> saleDAO.save(sale)
> }
> }
>
>
>
>
>1. My surprise is that it works! Well, the MR jobs seem to go to a
>strange state, but the entities are correctly saved. Still I'm 99% sure
>that this must be a bad practice or bring some problems. Can somebody tell
>me
>2. If that's not the case, why are we given a certain API for
>MapReduce and another one for normal persistence? Will be the same to
>invoke DatastoreMutationPool.put than DatastoreService.put (any
>implementation)?
>
>
> Honestly, this idea sounded crazy in my head and now that it seems to work
> I want to know more about it.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/61276548-616e-4f94-9263-00bcfcb5e33d%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAN4PiZHeGR%3Dt7NeY8PeG9eJmKe06UTbkm0XTRsPG8mBJe%3DNx9g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] What if GAE breaks? Backup server strategies?

[Tom Campbell]

Preparing to charge people for my Python/web2py-based online 
scheduling/membership app. Am concerned about GAE downtime. Incidents like 
this 

 
and this 

 concern 
me. What is the best way to be ready for GAE outages? Some hosts, for 
example, let you sign up for geographically distinct locations, which 
sounds great to me.

Prices are so low of course I'm quite willing to open multiple accounts, or 
allocate servers in different regions--whatever it takes. Let us also 
assume I am backing up data every few minutes to an hour.

What are best practices to prepare for a downed GAE server? 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/4312de5d-d8ec-4093-b12d-66193ce925b4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Site security & maintenance for a Python app on GAE: Easy?

[Tom Campbell]

>
> However, some aspects of security are up to you as a developer:

Yep, got that part--web2py happens to be super awesome at this stuff and
I'm also aware of these issues.

Does this help answer your question? Do you have any more specific
> questions about security?
>
It all sounds almost too good to be true, thanks.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CALXkWBR%3DDQP9Rp-4hMM43VbTFg1mAvniZ%2BX_g4iMcPA%2Bz_9L6w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Suddenly App Engine not Serving Anymore: Error code 123

[Tom Campbell]

I couldn't really afford this kind of outage when my app is deployed. is 
there a away around this, for example, having both a US and  EUR -based 
servers?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/64814580-cf27-4ef5-b61a-1db52f431eb7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Site security & maintenance for a Python app on GAE: Easy?

[Tom Campbell]

I'm finishing up a web2py app that I will be deploying to paying customers. 
I am not a great sysadmin.

Seems like GAE is a good choice, because while I follow all web2py best 
practices I'm always nervous about forgetting to apply a security patch to 
the OS, or having an apt-get cause unforeseen problems. 

Does GAE free me from these issues? It seems to me I simply need to write 
to Python 2.7 and update web2py when necessary, and GAE takes care of OS 
updates, security, etc.

Is that correct or am I oversimplifying?





-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/e11b8a4f-3962-4dfe-a115-66f9b810b3f3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Getting "Security issue: Signature length not correct: got 256 but was expecting 128" since about one hour ago.

[Tom Edge]

Does this work when using the cloud endpoints @Api annotations to declare 
endpoint api methods? I can't work out how to implement your workaround

On Thursday, June 11, 2015 at 4:29:28 PM UTC+1, Erik Zivkovic wrote:
>
> You can subclass GoogleIdTokenVerifier for now
>
> package com.my.project.package;
>
>
> import java.io.IOException;
> import java.security.GeneralSecurityException;
> import java.security.PublicKey;
>
>
> import com.google.api.client.auth.openidconnect.IdTokenVerifier;
> import com.google.api.client.googleapis.auth.oauth2.GoogleIdToken;
> import com.google.api.client.googleapis.auth.oauth2.GoogleIdTokenVerifier;
> import com.google.api.client.http.HttpTransport;
> import com.google.api.client.json.JsonFactory;
>
>
> @Deprecated
> public class GoogleIdTokenVerifier2 extends GoogleIdTokenVerifier {
>
>
> public GoogleIdTokenVerifier2(HttpTransport transport, JsonFactory 
> jsonFactory) {
> super(transport, jsonFactory);
> }
>
>
> @Override
> public boolean verify(GoogleIdToken googleIdToken) throws 
> GeneralSecurityException, IOException {
> // check the payload
> if (!((IdTokenVerifier)this).verify(googleIdToken)) {
> return false;
> }
> // verify signature
> for (PublicKey publicKey : getPublicKeysManager().getPublicKeys()) 
> {
> try {
> if (googleIdToken.verifySignature(publicKey)) {
> return true;
> }
> } catch (Exception e) {
> System.err.println("Verify Token:" + e);
> }
> }
> return false;
> }
> }
>
>
>
>
-- 
*Latest from the Rugged Blog: 2014, Our Year Without Paper 
*


    
 

o. 01202 665885
a. Communications House. 16 Didcot Road, Nuffield Trading Estate, Poole, 
Dorset. BH17 0GD
w. www.nuffieldtechnologies.com 
w.  www.ruggeddata.co.uk

*Please consider the environment before printing electronic copies of 
information.*
 
Company Reg No. 04619513
This e-mail and all attached files are intended for the named addressee 
only. If you are not the addressee legally indicated in the message (or 
responsible for delivery of the message to such person), you may not copy, 
deliver or disclose it to anyone. It may contain legally privileged or 
confidential information, which may be copyrighted. All efforts are made to 
keep our systems free of viruses, however please note it is your 
responsibility to scan this e-mail and attachments to ensure that no virus 
is present.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/1a6fdb6c-3f83-45de-870f-8e7188ed2901%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Getting "Security issue: Signature length not correct: got 256 but was expecting 128" since about one hour ago.

[Tom Edge]

Yes, I've got the same problem. I've asked a question on Stack Overflow 
here 
https://stackoverflow.com/questions/30781799/google-cloud-endpoints-verifytoken-signature-length-not-correct

Have you found any kind of workaround yet?

Tom

On Thursday, June 11, 2015 at 1:34:30 PM UTC+1, Erik Zivkovic wrote:
>
> I am using code from 
> http://android-developers.blogspot.se/2013/01/verifying-back-end-calls-from-android.html
>  
> and have been using it for a long time to authenticate users for my 
> service, but since about one hour ago I am getting "Security issue: 
> Signature length not correct: got 256 but was expecting 128", it is being 
> thrown from GoogleIdTokenVerifier.verify
>
> There seems to be a similar issue for OAuth: 
> http://stackoverflow.com/questions/30780407/google-oauth2-jwt-token-verification-exception
>
> Someone else experiencing the same issue?
>
> BR Erik
>

-- 
*Latest from the Rugged Blog: 2014, Our Year Without Paper 
<http://www.ruggeddata.co.uk/blog/2014-our-year-without-paper>*


<http://www.nuffieldtechnologies.co.uk/>  <http://www.ruggeddata.co.uk/>  
 

o. 01202 665885
a. Communications House. 16 Didcot Road, Nuffield Trading Estate, Poole, 
Dorset. BH17 0GD
w. www.nuffieldtechnologies.com <http://www.nuffieldtechnologies.com>
w.  www.ruggeddata.co.uk

*Please consider the environment before printing electronic copies of 
information.*
 
Company Reg No. 04619513
This e-mail and all attached files are intended for the named addressee 
only. If you are not the addressee legally indicated in the message (or 
responsible for delivery of the message to such person), you may not copy, 
deliver or disclose it to anyone. It may contain legally privileged or 
confidential information, which may be copyrighted. All efforts are made to 
keep our systems free of viruses, however please note it is your 
responsibility to scan this e-mail and attachments to ensure that no virus 
is present.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/1a4ff8dd-778e-4942-87fa-445ececf173a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Getting "Security issue: Signature length not correct: got 256 but was expecting 128" since about one hour ago.

[Tom Edge]

Yes, I've got the same problem. I've asked a question on Stack Overflow 
here 
https://stackoverflow.com/questions/30781799/google-cloud-endpoints-verifytoken-signature-length-not-correct

Have you found any kind of workaround yet?

Tom

On Thursday, June 11, 2015 at 1:34:30 PM UTC+1, Erik Zivkovic wrote:
>
> I am using code from 
> http://android-developers.blogspot.se/2013/01/verifying-back-end-calls-from-android.html
>  
> and have been using it for a long time to authenticate users for my 
> service, but since about one hour ago I am getting "Security issue: 
> Signature length not correct: got 256 but was expecting 128", it is being 
> thrown from GoogleIdTokenVerifier.verify
>
> There seems to be a similar issue for OAuth: 
> http://stackoverflow.com/questions/30780407/google-oauth2-jwt-token-verification-exception
>
> Someone else experiencing the same issue?
>
> BR Erik
>

-- 
*Latest from the Rugged Blog: 2014, Our Year Without Paper 
<http://www.ruggeddata.co.uk/blog/2014-our-year-without-paper>*


<http://www.nuffieldtechnologies.co.uk/>  <http://www.ruggeddata.co.uk/>  
 

o. 01202 665885
a. Communications House. 16 Didcot Road, Nuffield Trading Estate, Poole, 
Dorset. BH17 0GD
w. www.nuffieldtechnologies.com <http://www.nuffieldtechnologies.com>
w.  www.ruggeddata.co.uk

*Please consider the environment before printing electronic copies of 
information.*
 
Company Reg No. 04619513
This e-mail and all attached files are intended for the named addressee 
only. If you are not the addressee legally indicated in the message (or 
responsible for delivery of the message to such person), you may not copy, 
deliver or disclose it to anyone. It may contain legally privileged or 
confidential information, which may be copyrighted. All efforts are made to 
keep our systems free of viruses, however please note it is your 
responsibility to scan this e-mail and attachments to ensure that no virus 
is present.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/ba68659a-3f2f-406c-88f2-41e31baa2ba6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] App Engine Java Map Reduce Completion Callback

['Tom Kaitchuck' via Google App Engine]

You can use pipelines:
https://github.com/GoogleCloudPlatform/appengine-pipelines
to chain things to run after a MR job. One of the included examples does
this:
https://github.com/GoogleCloudPlatform/appengine-mapreduce/blob/master/java/example/src/com/google/appengine/demos/mapreduce/entitycount/ChainedMapReduceJob.java

On Thu, Jan 29, 2015 at 10:21 AM, Jim  wrote:

> Hello everyone,
>
> I've just started working with Java Map Reduce on App Engine, and I want
> to take advantage of the Completion Callback function so I can write a
> servlet that processes the final output of my Map Reduce job.  I'd like to
> drop the output data into a task and then persist it into my datastore.
> Ikai Lan's tutorial has been very helpful to me as I work my way through
> this, however it was written 4 1/2 years ago and the Map Reduce library has
> progressed a lot since then.
>
> In particular, Ikai's page
> http://ikaisays.com/2010/07/09/using-the-java-mapper-framework-for-app-engine/
> shows how to define a URI for a callback after completion of your Map
> Reduce job.  He shows how to define a job in mapreduce.xml which includes a
> setting for mapreduce.appengine.donecallback.url.   However, with the
> version I'm using, configuration of jobs is done in code using the
> MapReduceSpecification builder.  I can't find any reference there or in any
> of the other configuration objects for input, mapper, reducer, output, etc.
> to set a URI callback like Ikai describes when using the old mapreduce.xml
> config file.
>
> I have a feeling I'm overlooking something simple.  Can somebody point me
> in the right direction?
>
> If the callback method is no longer available, I'm thinking a custom
> output method would be the right place to capture my final output and drop
> to the task queue.  Any other ideas?
>
> Thanks for your help,
>
> Jim
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/cf47c5eb-f8ad-4376-90c7-86e44549f3d6%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAN4PiZE3Myjg2VrojYjDGNGZ8LoB3TWyq2R%2BpMKpc8ycptZyAA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Is there a PHP API for writing to the Datastore?

[Tom Walder]

No problem - I hope it works for you.

Feedback appreciated - enhancement requests / comments / whatever!

Feel free to give me some love on twitter (@tom_walder) / github

Enjoy!

Tom




On 14 January 2015 at 08:40, Anthony Shapley  wrote:

> Hi Tom,
>
> This looks like a fantastic contribution for any PHP App Engine
> developers, I for one have been looking for an easier solution to write to
> datastore from PHP and I thank you for creating and sharing this. I will
> most certainly have a play with it.
>
> Thanks
> Ant
>
> Kind Regards,
> Anthony Shapley
>
> Follow me on Twitter <http://www.twitter.com/anthonyshapley>, Facebook
> <http://www.facebook.com/anthony.shapley> or Google+
> <https://plus.google.com/53690494421503825/>. Visit the website
> <http://www.shapleytech.ltd.uk> to see our work.
>
>
> On 13 January 2015 at 13:46, Tom Walder  wrote:
>
>> I've just released a library that sits on top of the Google PHP API
>> client.
>>
>> It makes reading and writing with Datastore much easier (I hope).
>>
>> Check it out - it might help you if you're still looking.
>>
>> https://github.com/tomwalder/php-gds
>>
>>
>>
>> On Wednesday, 25 June 2014 19:22:33 UTC+1, Nooby wrote:
>>>
>>> Hello,
>>>
>>> I was wondering if there is any PHP documentation about writing to the
>>> datastore? I know PHP is experimental at this stage. So is this something
>>> that will be available in the future?
>>>
>>> Having written websites in PHP, it is the language that makes most sense
>>> to me writing an app engine app. I have got as far as letting the user
>>> upload an image to the blobstore. Now I need to store the list of images in
>>> datastorage. (I don't want to use Cloud SQL because firstly it's not
>>> scalable and secondly there is no free quota).
>>>
>>> Thanks.
>>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to google-appengine+unsubscr...@googlegroups.com.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> Visit this group at http://groups.google.com/group/google-appengine.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "Google App Engine" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/google-appengine/gDD4vNKyFHo/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
*Tom Walder*, CTO*Call*: 0845 521 0444 / *Web*: www.docnet.nu
<http://www.docnet.nu/?utm_source=email_signature&utm_medium=email&utm_campaign=email_signature>This
message is private and confidential. If you have received this message in
error, please notify us and remove it from your system. Doctor Net Limited
t/a Docnet is a company registered in England and Wales. Registered number:
3303804. Registered office: Speakers House, 39 Deansgate, Manchester, M3 2BA

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Is there a PHP API for writing to the Datastore?

[Tom Walder]

I've just released a library that sits on top of the Google PHP API client.

It makes reading and writing with Datastore much easier (I hope).

Check it out - it might help you if you're still looking.

https://github.com/tomwalder/php-gds



On Wednesday, 25 June 2014 19:22:33 UTC+1, Nooby wrote:
>
> Hello,
>
> I was wondering if there is any PHP documentation about writing to the 
> datastore? I know PHP is experimental at this stage. So is this something 
> that will be available in the future?
>
> Having written websites in PHP, it is the language that makes most sense 
> to me writing an app engine app. I have got as far as letting the user 
> upload an image to the blobstore. Now I need to store the list of images in 
> datastorage. (I don't want to use Cloud SQL because firstly it's not 
> scalable and secondly there is no free quota).
>
> Thanks.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] MapReduce import woes

['Tom Kaitchuck' via Google App Engine]

Take a look at the build.sh script:
https://github.com/GoogleCloudPlatform/appengine-mapreduce/blob/master/python/build.sh
It compiles (and runs) the demo application using the checked out Mapreduce.
It currently has an issue that makes it kindof annoying:
https://github.com/GoogleCloudPlatform/appengine-mapreduce/issues/17

However if you just want to depend on MapReduce in your application, the
easiest thing is to get it from Pypi.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Will cloud Dataflows replace the MapReduce/Pipeline API?

['Tom Kaitchuck' via Google App Engine]

Yes. Multiple namespaces are not directly supported. Given a finite known
set, the simplest thing to do would be to construct an InMemoryInput and
have the mapper operate on a namespace. You could also, for instance create
a new input class that iterated over namespaces.

Both of the above are predicated on the idea that the mapper can operate on
a namespace as a single item. If this is not practical, there is not much
that can be done. It is not possible to issue / shard a datastore query
that crosses namespaces. So there is no good way to shard the data.


On Tue, Jul 29, 2014 at 4:26 AM, Aswath Satrasala <
aswath.satras...@gmail.com> wrote:

> That is correct.  It support one namespace.  If there are 1000 namespaces,
> then we have to setup 1000 mapreduce jobs, and monitor them.
> It is more work and not convenient, if you are doing data processing more
> often.
> Please star this issue, if you are using namespace and mapreduce
> https://code.google.com/p/appengine-mapreduce/issues/detail?id=108
>
>
>
>
> On Fri, Jul 25, 2014 at 12:13 AM, 'Tom Kaitchuck' via Google App Engine <
> google-appengine@googlegroups.com> wrote:
>
>> Mapreduce does support namespaces:
>>
>> https://code.google.com/p/appengine-mapreduce/source/browse/trunk/java/src/main/java/com/google/appengine/tools/mapreduce/inputs/DatastoreInput.java#28
>>
>>
>> On Thu, Jul 24, 2014 at 12:28 AM, Aswath Satrasala <
>> aswath.satras...@gmail.com> wrote:
>>
>>> Currently, mapreduce does not support namespaces well and takes takes
>>> longer time to setup and code.
>>> Any idea, on the the dataflow when it will be for public?  Will it
>>> support for processing namespaces data in appengine?
>>>
>>> -Aswath
>>>
>>>
>>>
>>> On Fri, Jun 27, 2014 at 12:18 AM, 'Tom Kaitchuck' via Google App Engine
>>>  wrote:
>>>
>>>> No
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Google App Engine" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to google-appengine+unsubscr...@googlegroups.com.
>>>> To post to this group, send email to google-appengine@googlegroups.com.
>>>> Visit this group at http://groups.google.com/group/google-appengine.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>  --
>>> You received this message because you are subscribed to the Google
>>> Groups "Google App Engine" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to google-appengine+unsubscr...@googlegroups.com.
>>> To post to this group, send email to google-appengine@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/google-appengine.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to google-appengine+unsubscr...@googlegroups.com.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> Visit this group at http://groups.google.com/group/google-appengine.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Will cloud Dataflows replace the MapReduce/Pipeline API?

['Tom Kaitchuck' via Google App Engine]

Mapreduce does support namespaces:
https://code.google.com/p/appengine-mapreduce/source/browse/trunk/java/src/main/java/com/google/appengine/tools/mapreduce/inputs/DatastoreInput.java#28


On Thu, Jul 24, 2014 at 12:28 AM, Aswath Satrasala <
aswath.satras...@gmail.com> wrote:

> Currently, mapreduce does not support namespaces well and takes takes
> longer time to setup and code.
> Any idea, on the the dataflow when it will be for public?  Will it support
> for processing namespaces data in appengine?
>
> -Aswath
>
>
>
> On Fri, Jun 27, 2014 at 12:18 AM, 'Tom Kaitchuck' via Google App Engine <
> google-appengine@googlegroups.com> wrote:
>
>> No
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to google-appengine+unsubscr...@googlegroups.com.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> Visit this group at http://groups.google.com/group/google-appengine.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Will cloud Dataflows replace the MapReduce/Pipeline API?

['Tom Kaitchuck' via Google App Engine]

No

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] HTTP 429 Returned for /mapreduce/workerCallback task

['Tom Kaitchuck' via Google App Engine]

The 429 is the mapreduce framework pushing back to the task queue because
there is not enough memory on the instance to handle the number of requests
it was given. This isn't necessarily a problem, and if there are no other
issues, can be safely ignored. However if it is occurring a lot, you could
improve performance by lowering "max-concurrent-requests" in your modules
configuration:
https://developers.google.com/appengine/docs/java/modules/#Java_Configuration

You should be able to see the progress of the shards in the lower right
panel of the UI. Provided all of the shards are making progress, you
probably have nothing to worry about.


On Mon, Jun 9, 2014 at 9:56 AM, Bill Speirs  wrote:

>
> I'm attempting to run a map-reduce job from Java. The job kicked off and
> started without a hitch, but now it's stuck waiting on ~34 tasks to finish.
> The job is stuck in the "ExamineStatusAndReturnResult" phase, after the
> ShardedJob phase. I see ~34 tasks in my queue all of the
> form: POST /mapreduce/workerCallback/map-*hex-numbers* They all say that
> the previous run returned a 429 Too Many Requests (
> http://tools.ietf.org/html/rfc6585#section-4). I'm guessing I've hit some
> kind of limit/quota, but I cannot tell what/where this quota is.
>
> How can I find out what is causing the 429 response code?
>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Cloud Datastore Dashboard not working

[Tom Edge]

I've got the same thing, but I've just noticed that I'm over my datastore 
read operations quota... Is that the same for everyone else?

On Friday, May 23, 2014 1:42:18 PM UTC+1, Andrew Chen wrote:
>
> My God. Pre production panic. Same here. This really turns me off.
>
> On Monday, April 28, 2014 10:30:45 AM UTC-4, husayt wrote:
>>
>> Is there anybody in google team here, I can address this to?
>>
>> Just to recap: I have a major issue with new google develpers console, is 
>> that it doesn't load Cloud datastore Query nor Dashboard views. 
>> see 
>> https://www.dropbox.com/s/5cmgg34z0pw6852/Screenshot%202014-04-28%2015.21.53.png
>>
>>  I will appreciate if that can be resolved soon, as it is really 
>> affecting us.
>>
>>
>> On Monday, 21 April 2014 05:15:14 UTC+1, King Royal wrote:
>>>
>>> I have had problems same to you,, Please help me!!
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Can I run an AppEngine app with mixed Java and Go runtimes?

[Tom]

yes, a project can encompass multiple runtimes, but naturally you can't 
have code from multiple runtimes in the same instance, so you could 
potentially end up with higher costs (it won't be as efficient at scaling) 
and I'm not surprised to hear that there are issues with running multiple 
runtimes simultaneously on the dev server.

On Thursday, 1 May 2014 14:32:59 UTC-4, Ronoaldo Pereira wrote:
>
> I'm porting the front-end handlers of a large Java app to Go. However, I'm 
> not able to run the Java and Go runtimes in paralell in the development 
> server, to share the same datastore backend.
>
> Is there a way to acomplish that? I'm asking here because I noticed some 
> Java code to launch instances from devappserver2 module in the Python SDK. 
> I tought that it may be possible to run both a Java module and a Go module 
> with that service, if I was using the Java Yaml configuration.
>
> Kind regards,
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] My project doesn't have its default auto-created GCS bucket

[Tom]

On Monday, 7 April 2014 22:31:30 UTC-4, Vinny P wrote:
>
> On Mon, Apr 7, 2014 at 9:05 AM, Tom >
>  wrote:
>
>> Yes, that is correct, so I went ahead and clicked the 'create' button now 
>> it says "Project creation is in progress."
>>
>> But doesn't this mean that it will create a new CC project, so I will now 
>> have 2 CC project's for my GAE project?  And my Google Cloud Storage will 
>> be spread across the two CC's projects.
>>
>
>
> Yes, you will now have two projects.
>
> But you won't get the default bucket with the free quota without 
> connecting your App Engine app to a cloud project. The free quota also only 
> applies to that default bucket, so if you want the free space, you'll have 
> to move data into that bucket.
>
> With that said, there's nothing inherently wrong with having assets 
> striped across 2 or even more cloud projects. As long as you configure the 
> authentication to properly work, there's no difference compared to having 
> all assets within a single cloud project.
>  

Ok, thanks.

It appears that I know have access to the default bucket.

Tom
 

>  
>  
> -
> -Vinny P
> Technology & Media Advisor
> Chicago, IL
>
> App Engine Code Samples: http://www.learntogoogleit.com
>  
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] My project doesn't have its default auto-created GCS bucket

[Tom]

Yes, that is correct, so I went ahead and clicked the 'create' button now 
it says "Project creation is in progress."

But doesn't this mean that it will create a new CC project, so I will now 
have 2 CC project's for my GAE project?  And my Google Cloud Storage will 
be spread across the two CC's projects.

Tom

On Monday, 7 April 2014 04:16:12 UTC-4, Vinny P wrote:
>
> On Wed, Apr 2, 2014 at 12:54 PM, Tom <> wrote:
>
>> I have a cloud console project and a GAE/J project.  They were created 
>> independently, so were not connected at birth.  So, if I go to GAE app 
>> settings, the 'cloud integration' option is not enabled.
>>
>
>
> So to clarify, your console looks like this: http://imgur.com/qpEs2jucorrect? 
>
> If so, go ahead and click the Create button, you should get your default 
> bucket soon after that.
>   
>  
> -
> -Vinny P
> Technology & Media Advisor
> Chicago, IL
>
> App Engine Code Samples: http://www.learntogoogleit.com
>  
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] My project doesn't have its default auto-created GCS bucket

[Tom]

I have a cloud console project and a GAE/J project.  They were created 
independently, so were not connected at birth.  So, if I go to GAE app 
settings, the 'cloud integration' option is not enabled.

But I have connected them via the appspot service account, and that has 
worked fine for me - my GAE project has been successfully using the keys 
and API's defined in my CC project.

Below is some recent advice from Amy Unruh about how to make sure your 
projects are connected and you get your default bucket.  My project is 
already connected in the manner described, but the bucket does not appear 
in the cloud console.

Have they finished rolling out the default bucket to all projects?  Or 
should I just wait?

Thanks, Tom.

There is indeed a default auto-created bucket, with bucket name "
> your-app.appspot.com", that is getting rolled out and backfilled to 
> existing apps and their associated projects.  Not all projects may have it 
> yet.  If the 5GB quota is sufficient for your needs, you ought to be able 
> to just start using it as you would any bucket.
>   
> Go to the Application Settings page for the app engine app you want to use 
> that bucket with, and grab the app's service account address.
> Then go to the cloud console (cloud.google.com/console) and make sure 
> that service account is listed in the Permissions page of the project that 
> includes this default bucket, with 'edit' permissions. 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: 1.9.0 Pre-Release SDKs are now available.

[Tom Kaitchuck]

The new docs are here:
https://developers.google.com/appengine/docs/java/dataprocessing/
These are a replacement for the ones on code.google.com
Kirill: Your compatibility question is addressed here:
https://developers.google.com/appengine/docs/java/dataprocessing/mapreduce_update


On Mon, Feb 10, 2014 at 4:50 PM, Kirill Lebedev
wrote:

> Thanks for this information. Actually one of our appengines is already on
> 1.9.0. And I have a compatibility question based on your notes:
>
>  We are actively using Java version of MapReduce library. We are still on
> 0.2 version (Blobstore-based intermediate storage and InMemory Shuffling).
> We tried to update our system to 0.3 and 0.4 (current SVN state)  and it
> was not successful. More over 0.4 SVN version has import
> com.google.appengine.api.labs.modules.ModulesService; and import
> com.google.appengine.api.labs.modules.ModulesServiceFactory; imports in
> MapReduceJob.java. So how this release will affect MapReduce compatibility?
> Will 0.2 version still work on 1.9.0. Do you have any plans to publicly
> release 0.4 version of Java MapReduce that will reflect 1.9.0 changes? It
> is critical cause our code relies on that libraries.
>
> Thanks,
> Kirill Lebedev
>
> вторник, 4 февраля 2014 г., 17:43:41 UTC-8 пользователь Richmond Manzana
> написал:
>
>> We want to inform you that the pre-release SDKs for Python, PHP and Java
>> are now available.
>>
>> As previously 
>> announcedin
>>  a Google code site announcement, new App Engine Binaries are no longer
>> available at:
>> http://code.google.com/p/googleappengine/downloads/list
>>
>> Older binaries will remain available at the code.google.com site.
>>
>> 1.9.0 Pre-release SDKs are now available at these links:
>>
>> App Engine 1.9.0 Java prerelease 
>> SDK
>>
>> App Engine 1.9.0 Python prerelease 
>> SDK
>>
>> App Engine 1.9.0 PHP prerelease 
>> SDK
>>
>> In the future, please look forward to the finding the latest binaries at
>> https://developers.google.com/appengine/downloads
>>
>>
>> Also, please see the pre-release notes below.
>>
>> Cheers,
>>
>> Richmond Manzana
>> Technical Program Manager
>> Google App Engine
>>
>> App Engine SDK - Pre-Release Notes
>>
>> Version 1.9.0
>>
>> Python & PHP
>> ==
>> - New App Engine Application Identifiers must now start with a letter,
>>   in addition to the existing requirements that the identifier be 6-30
>>   characters which are letters, numbers, and hyphens, and not start or
>> end with
>>   a hyphen.
>>
>> Python
>> ==
>> - The size limit on the Search API is now computed and enforced on a
>> per-index
>>   basis, rather than for the app as a whole. The per-index limit is now
>> 10GB.
>>   There is no fixed limit on the number of indexes, or on the total
>> amount of
>>   Search API storage an application may use.
>> - Users now have the ability to embed images in emails via the Content-Id
>>   attachment header.
>> https://code.google.com/p/googleappengine/issues/detail?id=965
>> https://code.google.com/p/googleappengine/issues/detail?id=10503
>> - Fixed an issue with NDB backup/restore corrupting certain compressed
>>   entities.
>> https://code.google.com/p/googleappengine/issues/detail?id=8599
>>
>> PHP
>> ==
>> - The PHP interpreter has been upgraded from PHP 5.4.19 to PHP 5.4.22.
>> - Autoloading is now available in the SDK so developers will no longer
>> need to
>>   explicitly require SDK files.
>> - Expanded php.ini setting google_appengine.allow_include_gs_buckets to
>> allow
>>   a path filter be included for improved security.
>> - A warning message now appears if an application moves a user uploaded
>> file to
>>   a Google Cloud Storage bucket/path. This is due to the fact that code
>> may be
>>   included and lead to a local file inclusion vulnerability.
>> - Added API functions CloudStorageTools::getMetadata() and
>>   CloudStorageTools::getContentType() for retrieving the metadata and
>> content
>>   type of Google Cloud Storage objects.
>> https://code.google.com/p/googleappengine/issues/detail?id=10182
>> - Fixed an issue with GCS folders not displaying correctly in Developers
>>   Console.
>> - Fixed an issue with PHP_SELF and SCRIPT_NAME not being implemented
>> correctly.
>> https://code.google.com/p/googleappengine/issues/detail?id=9989
>> https://code.google.com/p/googleappengine/issues/detail?id=10478
>>
>> Java
>> ==
>> - Java 6 applications cannot be deployed to Google App Engine from any
>

Re: [google-appengine] Re: Best way to update 400,000 entities at once?

[Tom Kaitchuck]

This is exactly the sort of task the MapReduce was meant for. It should be
really a lot easier than managing the partitioning, error recovery, etc
yourself.
Take a look at our new docs:
https://developers.google.com/appengine/docs/java/dataprocessing/mapreduce_library
hopefully they should make it less overwhelming.


On Thu, Feb 27, 2014 at 5:04 AM, de Witte  wrote:

> Use a backend instance and keep it running until done.
>
> Or
>
> Use two tasks. One for retrieving 1000 keys at the time and a second one
> to update the entities in a batch of 1000.
>
> Done it for 300.000 entities in less than a 20 mins. ~300 tasks
>
> Op vrijdag 7 februari 2014 22:43:33 UTC+1 schreef Keith Lea:
>
>> Hi everyone,
>>
>> I'm a long time App Engine user for my app's backend, but I'm really
>> still a novice about the datastore.
>>
>> I'd like to add a new property (and index) for all entities of a certain
>> type. I have about 400,000 of this type of entity in the datastore, and I'd
>> like to load each one, add a property, and save it back to the datastore.
>> 400,000 times.
>>
>> This will obviously take a long time, so I'd really like to split it up
>> into ~100 tasks that each take 1/100th of the entities (~4,000 entities)
>> and perform this operation.
>>
>> But I really don't know how to do this using queries, and the Java
>> MapReduce library is overwhelmingly complicated.
>>
>> So how can I create 100 tasks that each take a unique chunk of the
>> entities to operate on? Is this called "sharding"? Is there a way for a
>> task to say "give me entity #200,000 thru #204,000"? (My entity's keys are
>> strings, which were generated by my application and generally look like
>> "928348-com.example-iOS".)
>>
>> I'm using Java and Objectify btw. Thanks for any help or guidance!!
>>
>> Keith
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: Issue with Cloud Storage GCS API on local Development Server

[Tom Kaitchuck]

Namespaces are a datastore specific concept. It caused trouble in this
instance because the local mock of GCS uses the datastore for its metadata.
Real GCS does not depend on datastore, so this problem will not occur with
deployed code.


On Fri, Dec 13, 2013 at 6:05 AM, Troy High  wrote:

> I was having the exact same problem with namespaces and the local dev
> server not honoring them for some entries. I applied the same workaround as
> Ken and am able to retrieve files programmatically.
>
> So is this a bug in the Dev server implementation or does this mean cloud
> storage does not officially support namespaces?  I haven't pushed this code
> to the google servers yet so I am not sure if I would encounter the same
> issue without the workaround.
>
> Thanks,
> Troy
>
>
> On Friday, November 22, 2013 12:40:05 PM UTC-5, k...@form-runner.comwrote:
>>
>> Nailed the culprit!
>>> We make extensive use of namespaces, and in my original post, there
>>> turns out to be an unfortunate interaction between namespaces and the GCS
>>> client API.
>>>
>>> Down in my original post, where I laid out all the data I could find in
>>> the Development Console, you will see that there are entries in the empty
>>> namespace and a different namespace:
>>>
>>> In __GsFileInfo__ (in the empty Namespace)
>>> ….blah … blah
>>>
>>> In Namespace 5629499534213120 (where everything ran):
>>> In _ah_FakeCloudStorate_formrunnerbucket-r7yh23nb2:
>>> …. blah…. blah
>>>
>>> The code for writing into GCS ran in namespace 5629499534213120, but it
>>> wrote part of it's data into the empty namespace.
>>>
>>> The (successful) experiment was to ensure that all the GCS API code
>>> (read and write) runs in the empty namespace, conceptually thus:
>>>
>>> private byte[] readFromFile(GcsFilename fullGcsFilename)  throws
>>> IOException
>>> {
>>> String in_ns = NamespaceManager.get();
>>> NamespaceManager.set(null);
>>>  int fileSize = (int) gcsService.getMetadata(
>>> fullGcsFilename).getLength();
>>> ByteBuffer result = ByteBuffer.allocate(fileSize);
>>>  GcsInputChannel readChannel = gcsService.openReadChannel(fullGcsFilename,
>>> 0);
>>> try {
>>>   readChannel.read(result);
>>>  } finally {
>>>   readChannel.close();
>>> }
>>> byte[] toreturn=result.array();
>>>  NamespaceManager.set(in_ns);
>>> return toreturn;
>>> }
>>>
>>> Now the FileNotFound does not occur, and I do get (some) bytes back,
>>> about 1/10 of the 1.2MB.  Presumably when I switch to ObjectStreaming, I'll
>>> get all.
>>> Cheers,
>>> --Ken
>>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] SSL warning, gone

[tom]

It used to be that when I deployed, I saw a warning

WARNING appengine_rpc.py:399 ssl module not 
> found. 
> Without the ssl module, the identity of the remote host cannot be 
> verified, and 
> connections may NOT be secure. To fix this, please install the ssl 
> module from 
> http://pypi.python.org/pypi/ssl . 
>

Now I no longer see that. But I cant find an announcement regarding a 
change in the SDK. Looking at the traffic in wireshark, it looks like its 
all TLS and SSL, but I'm no network expert. 

Please can someone confirm that the SDK deploys the code over a secured 
connection, preferably by linking to an announcement from google?

thanks

tom

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Issue with Cloud Storage GCS API on local Development Server

[Tom Kaitchuck]

Correct. It does not have to be directly in the servlet. For example App
Engine MapReduce: https://code.google.com/p/appengine-mapreduce/
uses the GCS client to write out data, but it is many levels removed from a
servlet.


On Thu, Nov 21, 2013 at 4:44 PM, Ken Bowen  wrote:

> Hi Tom,
>
> Re:
> > you don't need to use the LocalServiceTestHelper
>
> Understood.  I ran LocalExample once when I began, just to follow the
> documentation.
>
> The original description I posted is from a running app on the DevServer,
> no TestHelper involved.  In fact, the test jars are not even in the project.
>
> After posting, I was reviewing LocalExample to make comparisons, and the
> phrase
>
> "...run locally as opposed to in a deployed servlet"
>
> caught my eye.  So I wrote a version of what I posted as a servlet (grabs
> a 1.2MB PDF out of a resource, stores it with the GCS library, retrieves
> it, and writes out the number of bytes it got).  That works fine.
>
> If you say this doesn't /have/ to be directly in a servlet, I'll dig in
> further to my original.
>
> Cheers,
> --Ken
>
> On Nov 21, 2013, at 5:07 PM, Tom Kaitchuck wrote:
>
> > I'm a bit confused by your statement.
> > If you want to run in the DevAppserver or in AppEngine then you don't
> need to use the LocalServiceTestHelper at all. The LocalExample only does
> that as a demo of how to run it as a local executable or if you want to use
> it within Junit.
> > If you don't include such a line, like in GcsExampleServlet then it
> should work as part of your deployed application.
> >
> >
> > On Thu, Nov 21, 2013 at 3:41 PM, Ken Bowen  wrote:
> > Thanks Tom.
> > I went through pretty much the same servlet exercise a couple of hours
> before seeing your post this afternoon.
> > The problem was that I missed the point the the docs that the code
> /must/ run in a servlet.
> > (I'm not clear where it's stated, but I guessed from the comment about
> the test harness in LocalExample.)
> > I assumed that it would be ok to run in a deployed app on AppEngine.
> > Thanks again,
> > --Ken
> >
> > On Nov 21, 2013, at 2:51 PM, Tom Kaitchuck wrote:
> >
> > > Using the 1.8.8 version of the SDK and depending on
> appengine-gcs-client 0.3.3, the following servlet based on your above
> example works:
> > >
> > > public class GcsTest extends HttpServlet {
> > >   private void writeToFile(GcsService gcsService, GcsFilename
> fullGcsFilename, byte[] content)
> > >   throws IOException {
> > > System.out.println("writeToFile:full=" +
> fullGcsFilename.toString());
> > > GcsOutputChannel outputChannel =
> > > gcsService.createOrReplace(fullGcsFilename,
> GcsFileOptions.getDefaultInstance());
> > > outputChannel.write(ByteBuffer.wrap(content));
> > > outputChannel.close();
> > >   }
> > >
> > >
> > >   private byte[] readFromFile(GcsService gcsService, GcsFilename
> fullGcsFilename)
> > >   throws IOException {
> > > System.out.println("readFromFile:full=" +
> fullGcsFilename.toString());
> > > int fileSize = (int)
> gcsService.getMetadata(fullGcsFilename).getLength();
> > > ByteBuffer result = ByteBuffer.allocate(fileSize);
> > > GcsInputChannel readChannel =
> gcsService.openReadChannel(fullGcsFilename, 0);
> > > try {
> > >   readChannel.read(result);
> > > } finally {
> > >   readChannel.close();
> > > }
> > > return result.array();
> > >   }
> > >
> > >   @Override
> > >   public void doGet(HttpServletRequest req, HttpServletResponse resp)
> throws IOException {
> > > GcsService gcsService = GcsServiceFactory.createGcsService();
> > > GcsFilename fullGcsFilename = new GcsFilename("Foo", "Bar");
> > > byte[] content = new byte[] {1, 2, 3, 4, 5, 6, 7, 8, 9, 10};
> > > writeToFile(gcsService, fullGcsFilename, content);
> > > byte[] result = readFromFile(gcsService, fullGcsFilename);
> > > PrintWriter writer = resp.getWriter();
> > > writer.append(Arrays.toString(content));
> > > writer.append(Arrays.toString(result));
> > >   }
> > > }
> > >
> > > I'm not really sure what could be different about your setup to cause
> that. Try creating a new project with the minimal possible dependencies and
> run the servlet above and see what happens.
> > >
> > >
> > &g

Re: [google-appengine] Issue with Cloud Storage GCS API on local Development Server

[Tom Kaitchuck]

I'm a bit confused by your statement.
If you want to run in the DevAppserver or in AppEngine then you don't need
to use the LocalServiceTestHelper at all. The LocalExample only does that
as a demo of how to run it as a local executable or if you want to use it
within Junit.
If you don't include such a line, like in GcsExampleServlet then it should
work as part of your deployed application.


On Thu, Nov 21, 2013 at 3:41 PM, Ken Bowen  wrote:

> Thanks Tom.
> I went through pretty much the same servlet exercise a couple of hours
> before seeing your post this afternoon.
> The problem was that I missed the point the the docs that the code /must/
> run in a servlet.
> (I'm not clear where it's stated, but I guessed from the comment about the
> test harness in LocalExample.)
> I assumed that it would be ok to run in a deployed app on AppEngine.
> Thanks again,
> --Ken
>
> On Nov 21, 2013, at 2:51 PM, Tom Kaitchuck wrote:
>
> > Using the 1.8.8 version of the SDK and depending on appengine-gcs-client
> 0.3.3, the following servlet based on your above example works:
> >
> > public class GcsTest extends HttpServlet {
> >   private void writeToFile(GcsService gcsService, GcsFilename
> fullGcsFilename, byte[] content)
> >   throws IOException {
> > System.out.println("writeToFile:full=" + fullGcsFilename.toString());
> > GcsOutputChannel outputChannel =
> > gcsService.createOrReplace(fullGcsFilename,
> GcsFileOptions.getDefaultInstance());
> > outputChannel.write(ByteBuffer.wrap(content));
> > outputChannel.close();
> >   }
> >
> >
> >   private byte[] readFromFile(GcsService gcsService, GcsFilename
> fullGcsFilename)
> >   throws IOException {
> > System.out.println("readFromFile:full=" +
> fullGcsFilename.toString());
> > int fileSize = (int)
> gcsService.getMetadata(fullGcsFilename).getLength();
> > ByteBuffer result = ByteBuffer.allocate(fileSize);
> > GcsInputChannel readChannel =
> gcsService.openReadChannel(fullGcsFilename, 0);
> > try {
> >   readChannel.read(result);
> > } finally {
> >   readChannel.close();
> > }
> > return result.array();
> >   }
> >
> >   @Override
> >   public void doGet(HttpServletRequest req, HttpServletResponse resp)
> throws IOException {
> > GcsService gcsService = GcsServiceFactory.createGcsService();
> > GcsFilename fullGcsFilename = new GcsFilename("Foo", "Bar");
> > byte[] content = new byte[] {1, 2, 3, 4, 5, 6, 7, 8, 9, 10};
> > writeToFile(gcsService, fullGcsFilename, content);
> > byte[] result = readFromFile(gcsService, fullGcsFilename);
> > PrintWriter writer = resp.getWriter();
> > writer.append(Arrays.toString(content));
> > writer.append(Arrays.toString(result));
> >   }
> > }
> >
> > I'm not really sure what could be different about your setup to cause
> that. Try creating a new project with the minimal possible dependencies and
> run the servlet above and see what happens.
> >
> >
> >
> > On Tue, Nov 19, 2013 at 12:57 PM,  wrote:
> > I've encountered a problem using the Google Cloud Storage GCS Client
> API, running on the local development server.  I'm trying to write the
> bytes from a PDF file, and then read them back.
> >
> > The code appears to write the (local fake)GCS file ok: (1) There appears
> to be an appropriate entry in the Development Console, and (2) there's a
> physical file in ~war/WEB-APP/appengine-generated (details below).
>  However, when I attempt to read the bytes from the GCS file, it throws a
> FileNotFound exception when it attempts to get the metadata (filesize).
> >
> > First, here's the core code, versions of GCS client read/write, with my
> debugging stmts left in for reference:
> >
> > private void writeToFile(GcsFilename fullGcsFilename, byte[] content)
> >   throws IOException
> > {
> > System.out.println("writeToFile:full="+fullGcsFilename.toString());
> > GcsOutputChannel outputChannel =
> >   gcsService.createOrReplace(fullGcsFilename,
> GcsFileOptions.getDefaultInstance());
> > outputChannel.write(ByteBuffer.wrap(content));
> > outputChannel.close();
> > }
> >
> > private byte[] readFromFile(GcsFilename fullGcsFilename)
> >   throws IOException
> > {
> > System.out.println("readFromFile:full="+fullGcsFilename.toString());
> > int fileSize = (int)
> gcsService.getMetadata(fullGcsFilename).getLength();   [*][

Re: [google-appengine] Issue with Cloud Storage GCS API on local Development Server

[Tom Kaitchuck]

Using the 1.8.8 version of the SDK and depending on appengine-gcs-client
0.3.3, the following servlet based on your above example works:

public class GcsTest extends HttpServlet {
  private void writeToFile(GcsService gcsService, GcsFilename
fullGcsFilename, byte[] content)
  throws IOException {
System.out.println("writeToFile:full=" + fullGcsFilename.toString());
GcsOutputChannel outputChannel =
gcsService.createOrReplace(fullGcsFilename,
GcsFileOptions.getDefaultInstance());
outputChannel.write(ByteBuffer.wrap(content));
outputChannel.close();
  }


  private byte[] readFromFile(GcsService gcsService, GcsFilename
fullGcsFilename)
  throws IOException {
System.out.println("readFromFile:full=" + fullGcsFilename.toString());
int fileSize = (int)
gcsService.getMetadata(fullGcsFilename).getLength();
ByteBuffer result = ByteBuffer.allocate(fileSize);
GcsInputChannel readChannel =
gcsService.openReadChannel(fullGcsFilename, 0);
try {
  readChannel.read(result);
} finally {
  readChannel.close();
}
return result.array();
  }

  @Override
  public void doGet(HttpServletRequest req, HttpServletResponse resp)
throws IOException {
GcsService gcsService = GcsServiceFactory.createGcsService();
GcsFilename fullGcsFilename = new GcsFilename("Foo", "Bar");
byte[] content = new byte[] {1, 2, 3, 4, 5, 6, 7, 8, 9, 10};
writeToFile(gcsService, fullGcsFilename, content);
byte[] result = readFromFile(gcsService, fullGcsFilename);
PrintWriter writer = resp.getWriter();
writer.append(Arrays.toString(content));
writer.append(Arrays.toString(result));
  }
}

I'm not really sure what could be different about your setup to cause that.
Try creating a new project with the minimal possible dependencies and run
the servlet above and see what happens.



On Tue, Nov 19, 2013 at 12:57 PM,  wrote:

> I've encountered a problem using the Google Cloud Storage GCS Client API,
> running on the local development server.  I'm trying to write the bytes
> from a PDF file, and then read them back.
>
> The code appears to write the (local fake)GCS file ok: (1) There appears
> to be an appropriate entry in the Development Console, and (2) there's a
> physical file in ~war/WEB-APP/appengine-generated (details below).
>  However, when I attempt to read the bytes from the GCS file, it throws a
> FileNotFound exception when it attempts to get the metadata (filesize).
>
> First, here's the core code, versions of GCS client read/write, with my
> debugging stmts left in for reference:
>
> private void writeToFile(GcsFilename fullGcsFilename, byte[] content)
>  throws IOException
> {
> System.out.println("writeToFile:full="+fullGcsFilename.toString());
> GcsOutputChannel outputChannel =
>   gcsService.createOrReplace(fullGcsFilename,
> GcsFileOptions.getDefaultInstance());
> outputChannel.write(ByteBuffer.wrap(content));
> outputChannel.close();
> }
>
> private byte[] readFromFile(GcsFilename fullGcsFilename)
> throws IOException
> {
> System.out.println("readFromFile:full="+fullGcsFilename.toString());
> int fileSize = (int)
> gcsService.getMetadata(fullGcsFilename).getLength();   [*][Exception thrown
> here]
> ByteBuffer result = ByteBuffer.allocate(fileSize);
> GcsInputChannel readChannel =
> gcsService.openReadChannel(fullGcsFilename, 0);
> try {
>   readChannel.read(result);
> } finally {
>   readChannel.close();
> }
> return result.array();
> }
>
> Here's the debugging output (in/out filenames appear to be the same):
> 
> writeToFile:full=GcsFilename(formrunnerbucket-r7yh23nb2,
> FA/MasterFormStore-6649846324789248)
>
> 
> readFromFile:full=GcsFilename(formrunnerbucket-r7yh23nb2,
> FA/MasterFormStore-6649846324789248)
>
> 
> Here's the observed results:
>
> IN ~war/WEB-APP/appengine-generated:
>
> -rw-r--r--  1 ken  staff  1679407 Nov 19 09:19
> encoded_gs_key:L2dzL2Zvcm1ydW5uZXJidWNrZXQtcjd5aDIzbmIyL0ZBL01hc3RlckZvcm1TdG9yZS02NjQ5ODQ2MzI0Nzg5MjQ4
> This is the expected PDF file, which can be opened (with Preview on a Mac).
>
> 
> In the Development Console (http://localhost:/_ah/admin/datastore):
> In __GsFileInfo__ (in the empty Namespace)
>
> Key:
> ag5mb3JtcnVubmVyLWhyZHJ7CxIOX19Hc0ZpbGVJbmZvX18iZ2VuY29kZWRfZ3Nfa2V5OkwyZHpMMlp2Y20xeWRXNXVaWEppZFdOclpYUXRjamQ1YURJemJtSXlMMFpCTDAxaGMzUmxja1p2Y20xVGRHOXlaUzAyTmpRNU9EUTJNekkwTnpnNU1qUTQM
>
> ID/Name:
> encoded_gs_key:L2dzL2Zvcm1ydW5uZXJidWNrZXQtcjd5aDIzbmIyL0ZBL01hc3RlckZvcm1TdG9yZS02NjQ5ODQ2MzI0Nzg5MjQ4
>
> content_type: application/octet-stream
>
> filename:
>  /gs/formrunnerbucket-r7yh23nb2/FA/MasterFormStore-6649846324789248
>
> size: 1679407
>
> [[The ID/Name appears to be identical to the filename appearing in
> ~war/WEB-APP/appengine-generated]]
>
> ---
> In the Development Console (http://localhost:/_ah/admin/datastore),
> in Namespace 5629499534213120 (where everything ran):
> In _ah_FakeCloudStorate_formru

Re: [google-appengine] Have a front end for Cloud Storage.

[Tom Kaitchuck]

The appengine gcs client: https://code.google.com/p/appengine-gcs-client/
has no size restriction. If you want to upload to/from appengine it is the
prefered solution.
If you want users to upload and download externally the prefered solution
is to use createUploadUrl:
https://developers.google.com/appengine/docs/java/javadoc/com/google/appengine/api/blobstore/BlobstoreService#createUploadUrl(java.lang.String,
com.google.appengine.api.blobstore.UploadOptions)
for uploading data and serve:
https://developers.google.com/appengine/docs/java/javadoc/com/google/appengine/api/blobstore/BlobstoreService#serve(com.google.appengine.api.blobstore.BlobKey,
HttpServletResponse)
for download.

The advantage of createUploadUrl and serve over having the user post and
get from the bucket directly (Which of course works) is that it allows your
application to have more control as you can run code in response to each
request which lets you do things like custom permissions, logging, rate
limiting etc.



On Wed, Nov 6, 2013 at 5:59 AM, Vinny P  wrote:

> On Tue, Nov 5, 2013 at 3:56 PM, Pushpinder Jaswal <
> pushpinder.jas...@gmail.com> wrote:
>
>> I would like to know the best API and practices for this purpose, also
>> how should I proceed in this scenario for example have a client to add
>> objects through App Engine to cloud storage or directly add objects to
>> cloud storage. (I would like to mention that the objects that I am talking
>> about would be more than 32 Mb in size.)
>>
>>
>
> Both options are fine, but the better option would probably be to add
> objects through the App Engine service. You can use the Blobstore upload
> handler to write to GCS. Also, this way there is one central point to
> manage permissioning, bucket management, etc.
>
> Larger than 32MB files are fine, just note that you'll have to use
> Blobstore upload and serving options.
>
>
> On Tue, Nov 5, 2013 at 3:56 PM, Pushpinder Jaswal <
> pushpinder.jas...@gmail.com> wrote:
>
>> I would like have options to maintain the permissions and deleting
>> buckets as well.
>>
>
>
> That can be easily done through the XML/JSON Cloud Storage API, or by
> using the Java client library.
>
>
> -
> -Vinny P
> Technology & Media Advisor
> Chicago, IL
>
> App Engine Code Samples: http://www.learntogoogleit.com
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: handle https request on local machine using google appengine sdk

[tom]

We have the following setup:

 - A digicert signed SSL cert for https://www.foo.com, which is our 
production 'version'
 - A self-signed wildcard cert (free) for https://*.www.foo.com, which we 
use for all our remote testing (e.g. * = dev, staging, beta, etc)
 - No cert for local testing, all local requests are to http://

To enable this mix we do two things:

1. Where 'absolute' urls are necessary, we make them protocol relative - 
i.e. "//www.foo.com/bar"
2. We have some application logic that ensures the right protocol is used 
where necessary (there isnt much of this, and it's factored out)

Between all those things, we dont find it necessary to test SSL/https in a 
local environment.

tom



On Tuesday, October 8, 2013 1:17:08 PM UTC-7, Bryce Cutt wrote:
>
> If you really must do SSL with the SDK you can nginx as a reverse proxy in 
> front of the GAE SDK and have nginx handle HTTPS and serving the 
> certificate.
>
>
> On Tuesday, October 8, 2013 8:10:22 AM UTC-7, Chetan Dhembre wrote:
>>
>> Hi,
>>I have created one appplication using google app engine sdk in java. 
>> And it is handling http request properly. but i unable to handle https 
>> request. I tried to install self signed ssl certificate on my local java 
>> home directory but it does not help me. Any on faced this problem ? how to 
>> tackle this problem ? please help
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Updates for GPE, GAE SDK, and ADT today.

[Tom]

Eclipse notified me of new versions of GPE and ADT today (and a new GAE 
1.8.5 too, but maybe that was already there?).

The problem I have with these updates is that I'm rarely able to find 
anything that tells me what the update is, i.e. release notes.  Today I 
looked at the new g+ cloud platform community as well as the usual places, 
but nothing there either.

These updates can be great (bug fixes) or not so great (e.g. breaking cloud 
endpoints) so it really would be helpful if we could have some release 
notes, even if they just served to tell us the scale and scope of the 
update.  Such a notice would also serve as a sign-post around which users 
could comment and share any issues they are having.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: GAE 1.8.4 release is available on Eclipse update, but not announced yet?

[Tom]

Exactly.  Yes, I am using Endpoints and that is why I'm scared to install. 
 Doesn't appear that they test endpoints before rolling out new SDK's.  Tom

On Saturday, September 7, 2013 5:15:33 AM UTC-4, Johan Compagner wrote:
>
> i would not install it if you are using the cloud enpoint generation
> those are broken (again) it seems that this happens with every new 
> release.. :(
>
>
> On Thursday, September 5, 2013 6:27:07 PM UTC+2, Tom wrote:
>>
>> Did I miss something?  Yesterday I checked Eclipse update and GAE/J 1.8.4 
>> is available.  (I know the pre-release came out a couple weeks ago, but 
>> this is the full release now.)
>>
>> I'm hesitant to install it because I haven't seen any announcement about 
>> it, so I'm not sure if this might be a mistake.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Howto associate my existing API project with existing GAE project

[Tom]

Oh, so the new "cloud integration" projects are actually different then the 
'old' API projects - I hadn't realize that.

I guess then, if I want the integration, I have no choice but to do as you 
suggest: recreate my API project as one of the new projects.

Thanks for the info,
Tom


On Thursday, September 5, 2013 2:13:48 PM UTC-4, Jason Collins wrote:
>
> I don't think you can do what you're looking to do.
>
> Do you want them associated to get some of the nice, automatic service 
> account authorization bits? You can do same (I believe) by adding the GAE 
> service account to the Team tab of your existing API project.
>
> That said, it feels like more and more feature sets will be built around 
> the tight integration of an API project (really a "cloud" project) and your 
> GAE application, so it might be worth your while to migrate to the new 
> "cloud integration" projects. We did this, though it was somewhat painful. 
> Hopefully, I've interpreted the vision correctly!
>
> j
>
> On Thursday, 5 September 2013 11:03:30 UTC-6, Tom wrote:
>>
>> I decided to try out the new Cloud Console and associate my API project 
>> with my GAE/J project - both already existed.
>>
>> After fooling around a bit but not succeeding, I click on the 'cloud 
>> integration' option in my GAE application settings - not surprisingly this 
>> associated my GAE project with a brand new API project.  I can delete that 
>> newly created project, but it just creates another and now things are 
>> starting to get weird so I better stop.
>>
>> Any suggestions?  Thanks.
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Howto associate my existing API project with existing GAE project

[Tom]

I decided to try out the new Cloud Console and associate my API project 
with my GAE/J project - both already existed.

After fooling around a bit but not succeeding, I click on the 'cloud 
integration' option in my GAE application settings - not surprisingly this 
associated my GAE project with a brand new API project.  I can delete that 
newly created project, but it just creates another and now things are 
starting to get weird so I better stop.

Any suggestions?  Thanks.


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] GAE 1.8.4 release is available on Eclipse update, but not announced yet?

[Tom]

Did I miss something?  Yesterday I checked Eclipse update and GAE/J 1.8.4 
is available.  (I know the pre-release came out a couple weeks ago, but 
this is the full release now.)

I'm hesitant to install it because I haven't seen any announcement about 
it, so I'm not sure if this might be a mistake.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: How to encode videos?

[tom]

We use zencoder for transcoding videos. Our work flow is:

1. webpages uploads file direct from browser to AWS S3
2. webpage POST to our AE to notify that upload is complete
3. our AE makes request to zencoder to start transcode
4. zencoder completes job, then POSTs transcoded video to AWS S3
5. zencoder POSTs to our AE to notify us transcode is complete (or 
errored), we update our assets status in DB

Of course we also handle the error cases (#3 and #5 are where zencoder 
notify us if there is a problem). The admins get an email if there is an 
error (rarely, but not never)

There's lots of other ways to skin the cat, of course. Ours works for us in 
production. If you'd like more details then I'm very happy to share - just 
ask. 

BTW, for #1 we use https://github.com/TTLabs/EvaporateJS, which we wrote, 
and is handling 22GB uploads!

tom

On Sunday, August 18, 2013 11:05:26 AM UTC-7, Kaan Soral wrote:
>
> This issue has been on my mind, let's say a user uploads an 2gb .mov file, 
> and let's say one decides to use a service like zencoder
> The upload to external service and the download from external service both 
> sound challenging, if the RAM of an instance is 512mb's for example, and 
> the runtime of a task is 10 minutes, would it be possible to achieve 
> upload/download?
>
> I would really appreciate video encoding advice / file upload/download 
> advice related to videos.
>
> Generally it's easy to upload a file to AppEngine, and download a file 
> from AppEngine, thanks to Blobstore upload/download handlers, however as 
> far as I know, there aren't any helpers that will do the reverse, upload 
> file from AppEngine, and download file to AppEngine from an URL
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Moving a WordPress site to the new GAE/PHP runtime...

[Tom]

Thanks Vinny and alon for the suggestions.  Tom

On Sunday, July 14, 2013 4:55:42 AM UTC-4, alon wrote:
>
> Consider using http://wordpress.org/plugins/google-app-engine/ for a much 
> easier transfer.
>
> -
> Cheers,
>
> def AlonCarmel(request)
>  import simplejson as json
>  contact = {}
>  contant['email'] = 'a...@aloncarmel.me '
>  contact['twitter'] = '@aloncarmel'
>  contact['web'] = 'http://aloncarmel.me'
>  contact['phone'] = '+972-54-4860380'
>  return HttpResponse(json.dumps(contact))
>
> * If you received an unsolicited email from by mistake that wasn't of your 
> matter please delete immediately. All E-mail sent from Alon Carmel is 
> copyrighted to Alon Carmel 2008. Any details revealed in e-mails sent by 
> Alon Carmel are owned by the Author only. Any attempt to duplicate or 
> imitate any of the Content is prohibited under copyright law 2011.
>
> word.
>
>
> On Sun, Jul 14, 2013 at 9:40 AM, Vinny P  >wrote:
>
>> Hello Tom,
>>  
>> On Fri, Jul 12, 2013 at 2:30 PM, Tom <> wrote:
>>
>>> Is Wordpress on GAE/PHP ready enough for me to consider migrating his 
>>> site?  His site is a fairly straight-forward wordpress site
>>>
>>  The App Engine staff run their own blog on GAE/PHP at 
>> http://gaeforphp-blog.appspot.com/, so it certainly seems ready enough.
>>   
>> I have a PHP application on GAE (not a Wordpress instance, but a custom 
>> application) and I find the PHP runtime to be quite stable. As an anecdotal 
>> experience, the PHP runtime is faster with instance startup than the Java 
>> runtime.
>>   
>>
>> On Fri, Jul 12, 2013 at 2:30 PM, Tom <> wrote:
>>
>>> Will the cost of running this low volume site be roughly similar to 
>>> running it in a WPS at godaddy. 
>>
>>  
>>  
>> It's difficult to estimate cost comparisons without knowing what Godaddy 
>> package + any addons you're using now, and without your definition of 
>> "low-volume" (ballpark numbers would be very helpful here). Broadly 
>> speaking, I would bet that App Engine would be reasonably competitive in 
>> regards to price.
>>   
>> I briefly skimmed the site you linked to, and it doesn't look too 
>> complicated in regards to server-side processing. If you installed a 
>> caching plugin such as W3 Super Cache and configured the free version of 
>> Cloudflare CDN to sit in front of the site, you could probably serve a 
>> majority of the site hits entirely from cache/static pages.
>>   
>>  
>> On Fri, Jul 12, 2013 at 2:30 PM, Tom <> wrote:
>>
>>> Is there a forum frequented by GAE/PHP devs where I might find someone 
>>> who I could pay do do this?  I think most activity on this topic is at 
>>> stackoverflow and that is not a suitable place for such a query.
>>
>>   
>>  
>> You said earlier in your email that you know Java/Python, but not PHP. 
>> Still, consider trying this project yourself. Uploading a Wordpress install 
>> to GAE doesn't require too much PHP knowledge, and once Wordpress is 
>> installed, it largely configures itself/offers a GUI to configure most 
>> other things. There's a guide to uploading Wordpress here: 
>> https://developers.google.com/appengine/articles/wordpress . Try it out.
>>  
>>  
>>  
>> -
>> -Vinny P
>> Technology & Media Advisor
>> Chicago, IL
>>
>> App Engine Code Samples: http://www.learntogoogleit.com
>>   
>>  
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to google-appengi...@googlegroups.com .
>> To post to this group, send email to 
>> google-a...@googlegroups.com
>> .
>> Visit this group at http://groups.google.com/group/google-appengine.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>  
>>  
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Moving a WordPress site to the new GAE/PHP runtime...

[Tom]

A friend has come to me because his wordpress on godaddy site got hacked (I 
use GAE/J and GAE/P and am not famililar with PHP or WP).  

Is Wordpress on GAE/PHP ready enough for me to consider migrating his site? 
 His site is a fairly straight-forward wordpress site: joblaw.ca.

Will the cost of running this low volume site be roughly similar to running 
it in a WPS at godaddy.

Is there a forum frequented by GAE/PHP devs where I might find someone who 
I could pay do do this?  I think most activity on this topic is at 
stackoverflow and that is not a suitable place for such a query.

Thanks,
Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] How to upload files to blobstore programmatically , without form.

[Tom Kaitchuck]

If you want to write to blobstore programmatically you can
call createUploadUrl as many times as you like and pass those to whatever
is going to do the upload. This is supported and not going away.
If you want to upload to GCS you can use the Manager as Vinny mentioned, or
you can use gsutil from a command line or script:
https://developers.google.com/storage/docs/gsutil
or you can use the GCS client from within your App Engine application.


On Mon, Jul 1, 2013 at 11:40 AM, Vinny P  wrote:

> On Mon, Jul 1, 2013 at 6:40 AM,  wrote:
>
>> Hi ! i want to upload many files to GAE blob store but i want to do that
>> programmatically , instead of using the form and browsing the file . for
>> example uploading all files in a particular folder to the GAE blobstore.
>> Is there any method to do so ?
>>
>
>
> Writing programmatically to the blobstore is deprecated. What you can do
> is create a Google Cloud Storage bucket, and use the GCS Manager (
> https://developers.google.com/storage/docs/gsmanager ) to upload files.
> GCS Manager supports drag and drop, so you can simply drag files from your
> computer to your storage bucket.
>
> -
> -Vinny P
> Technology & Media Advisor
> Chicago, IL
>
> App Engine Code Samples: http://www.learntogoogleit.com
>
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Will 1.8.2 Datastore Admin Scheduled Backup tool support the new Cloud Storage Client Library?

[Tom Kaitchuck]

You don't have to use the datastore backup format to get data into
bigQuery. (You could if you wanted, the code is open source) But BigQuery
supports Json and CSV directly, which is often easier.
We actually have a guide on extracting data from Datastore and using a
MapReduce to load it into BigQuery here:
https://developers.google.com/bigquery/articles/datastoretobigquery
This guide was specifically made because it is quite common to want to
either import only a subset of the data into BigQuery or to first run some
transform on it.

Following the guide it shouldn't take much effort to get things up and
working. In the SVN there is already an output writer (still being tested)
included in the mapreduce library to write using the GCS client library,
which should allow you to run without using the FilesAPI at all. (You can
use it now, it's name starts with an "_" as we are still testing it, once
that work is completed it will be renamed to remove the "_")

We are actively working on baking all of this in, to make things much
easier, but I can't commit to a specific date. If your only concern is this
going to happen in a timely way, then you should wait. That being said,
running your own MR a la the example above is a good option anyway as it
gives you more control over exactly what your are putting in to BigQuery
and the format it is in.



On Tue, Jul 9, 2013 at 6:30 AM, Jason Collins wrote:

> Thanks Tom.
>
> The next part of my story is that we use the backup files with a BigQuery
> ingestion job - that is, the BigQuery ingestion job uses the native output
> from the Datastore Admin Scheduled Backups stored on Cloud Storage.
> ('sourceFormat': 'DATASTORE_BACKUP')
>
> Presumably, I'd also have to replicate the format if I were to roll my own
> GCS/MapReduce hybrid and continue to use the same BigQuery ingestion
> approach.
>
> Any suggestions on that front? Or maybe just an approximate ETA and save
> me a bunch of work? ;)
>
> j
>
>
> On Monday, 8 July 2013 18:23:38 UTC-6, Tom Kaitchuck wrote:
>
>> This is something we are working hard on. We're updating many code paths
>> to fix a lot of issues and migrate over to the GCS client. Changes won't
>> roll out in one big release, rather updates will be released as they are
>> completed. If you don't want to wait, it is absolutely supported to use the
>> GCS client to write out data from within your own MapReduce.
>>
>>
>> On Thu, Jul 4, 2013 at 9:01 AM, Jason Collins wrote:
>>
>>> Our backups are really flakey and take a long time, apparently because
>>> of the sketchy Files API link.
>>>
>>> Will the GAE 1.8.2 release of Datastore Admin Scheduled Backup tool
>>> support the new Cloud Storage Client Library?
>>>
>>> This is all wrapped up in MapReduce/Pipelines, so there are a lot of
>>> moving parts. My goal is to simply use stock, Google-supplied tools and
>>> have this stuff be resilient and predictable.
>>>
>>> Any info is appreciated. If something in this area is not imminent, we
>>> will have to start looking for alternate ways to achieve reliable backups.
>>>
>>> Thanks,
>>> j
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Google App Engine" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to google-appengi...@**googlegroups.com.
>>> To post to this group, send email to google-a...@googlegroups.**com.
>>>
>>> Visit this group at 
>>> http://groups.google.com/**group/google-appengine<http://groups.google.com/group/google-appengine>
>>> .
>>> For more options, visit 
>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
>>> .
>>>
>>>
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Will 1.8.2 Datastore Admin Scheduled Backup tool support the new Cloud Storage Client Library?

[Tom Kaitchuck]

This is something we are working hard on. We're updating many code paths to
fix a lot of issues and migrate over to the GCS client. Changes won't roll
out in one big release, rather updates will be released as they are
completed. If you don't want to wait, it is absolutely supported to use the
GCS client to write out data from within your own MapReduce.


On Thu, Jul 4, 2013 at 9:01 AM, Jason Collins wrote:

> Our backups are really flakey and take a long time, apparently because of
> the sketchy Files API link.
>
> Will the GAE 1.8.2 release of Datastore Admin Scheduled Backup tool
> support the new Cloud Storage Client Library?
>
> This is all wrapped up in MapReduce/Pipelines, so there are a lot of
> moving parts. My goal is to simply use stock, Google-supplied tools and
> have this stuff be resilient and predictable.
>
> Any info is appreciated. If something in this area is not imminent, we
> will have to start looking for alternate ways to achieve reliable backups.
>
> Thanks,
> j
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: CSS being ignored on local development server

[tom]

Yes, we have the 'static_dir' in our app.yaml. 
This is only a problem on the local development server. The issue is a 
confirmed defect, we just need more people to vote for it :)

tom



On Tuesday, July 2, 2013 9:36:09 AM UTC-7, wiki...@gmail.com wrote:
>
> did you tried adding, static_dir in app.yaml
>
> go to app.yaml file
>
> add under "handlers:"
> """- url: url_of_css(for eg /static) 
>   static_dir: static""
>
> Thank you
> Harsh Bhikadia
> (http://www.wikicious.appspot.com/)
>
> On Friday, June 28, 2013 12:16:18 AM UTC+5:30, tom wrote:
>>
>> We'd been seeing 'ignored' CSS for a while when loading pages from the 
>> local development server. After many months of hard reloading the page, we 
>> finally figured out the issue, which I filed here:
>>
>> https://code.google.com/p/googleappengine/issues/detail?id=9411
>>
>> I'm posting it here in case you're also experience 'ignored' CSS on your 
>> local development server, and want to vote for it :)
>>
>> tom
>>  
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] CSS being ignored on local development server

[tom]

We'd been seeing 'ignored' CSS for a while when loading pages from the 
local development server. After many months of hard reloading the page, we 
finally figured out the issue, which I filed here:

https://code.google.com/p/googleappengine/issues/detail?id=9411

I'm posting it here in case you're also experience 'ignored' CSS on your 
local development server, and want to vote for it :)

tom
 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] appengine-gcs-client library and acl

[Tom Kaitchuck]

It is not currently possible to do that with that library. However you can
set and edit permissions in much more complex ways using this library:
https://developers.google.com/storage/docs/json_api/v1/api-lib/java

There is an ongoing discussion as to how these two libraries should
interrelate. If you have a thoughts on that, please email me off list.


On Thu, Jun 13, 2013 at 2:08 PM, dragan  wrote:

> I analyzed the code a bit. It seems to me that the acl method only sets
> the "x-goog-acl" header which is only good for "applying *predefined* ACLs
> to buckets and objects". I need something like 
> this<https://developers.google.com/storage/docs/accesscontrol#aclquery>.
> Is it possible to do this with the current library?
>
>
> On Wednesday, June 12, 2013 11:57:30 PM UTC+2, Tom Kaitchuck wrote:
>
>> Whatever you specify as part of the GcsFileOptions will be passed
>> directly to GCS. So you can refer to the GCS access control documentation
>> for how to set things and how varrious ACLs will be interpreted:
>> https://developers.google.com/**storage/docs/accesscontrol#**
>> About-Access-Control-Lists<https://developers.google.com/storage/docs/accesscontrol#About-Access-Control-Lists>
>> If you go to the cloud console, you should be able to see the email
>> addresses associated with your account including the one that is used for
>> the App Engine Application's identity.
>>
>>
>> On Wed, Jun 12, 2013 at 4:17 AM, dragan  wrote:
>>
>>> I'm trying to convert my current code from the File API to the Cloud
>>> Storage API (using the 
>>> appengine-gcs-client<https://code.google.com/p/appengine-gcs-client/>library).
>>>  I want to ensure that only the GAE application has access to the
>>> newly created files on the Cloud Storage. How can I do that? Te examples i
>>> have seen all use the following code or something similar:
>>>
>>> GcsFileOptions options = new GcsFileOptions.Builder().mimeT**ype(
>>> "text/xml").acl("public-**read").build();
>>>
>>>
>>> This makes the file accessible to everybody for reading. What should I
>>> put in the acl call to make the file accessible only to the GAE
>>> application. I figured that I can add the application to the team as the
>>> editor but I can not figure out the next step. A code snippet would be nice.
>>> Thanks in advance.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Google App Engine" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to google-appengi...@**googlegroups.com.
>>> To post to this group, send email to google-a...@googlegroups.**com.
>>>
>>> Visit this group at http://groups.google.com/**
>>> group/google-appengine?hl=en<http://groups.google.com/group/google-appengine?hl=en>
>>> .
>>> For more options, visit 
>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
>>> .
>>>
>>>
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
>
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Puzzled by promotion of Go for high performance web serving

[Tom]

I saw the new blog post here promoting Go for "high performance" apps on 
App Engine
http://blog.golang.org/go-and-google-cloud-platform

I remember reading about Go with considerable interest when it was 
announced.  The controversial and particularly unique elements of Go's 
design were justified with the explanation that Go was designed for 
building massive systems.  In particular a number of design concessions 
were made such that massive code base could be built quickly.  

I don't mean to put Go down - I wasn't building massive systems so I didn't 
pursue the language and know nothing of it beyond its high level design 
justification.  For all I know it is a great language.

My point in making this post is that I hope Google will, with time, also 
support C-based languages for web serving on GAE, rather than requiring the 
vast number of us who already know them so well to learn a new language 
that was designed for such a different purpose.

I will add that I'm currently using GAE/J to back an Android app, and the 
ability to share code between client and server is terrific, so I think 
there is a lot of value in adding new GAE runtimes that correspond to 
client technologies.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] appengine-gcs-client library and acl

[Tom Kaitchuck]

Whatever you specify as part of the GcsFileOptions will be passed directly
to GCS. So you can refer to the GCS access control documentation for how to
set things and how varrious ACLs will be interpreted:
https://developers.google.com/storage/docs/accesscontrol#About-Access-Control-Lists
If you go to the cloud console, you should be able to see the email
addresses associated with your account including the one that is used for
the App Engine Application's identity.


On Wed, Jun 12, 2013 at 4:17 AM, dragan  wrote:

> I'm trying to convert my current code from the File API to the Cloud
> Storage API (using the 
> appengine-gcs-clientlibrary).
>  I want to ensure that only the GAE application has access to the
> newly created files on the Cloud Storage. How can I do that? Te examples i
> have seen all use the following code or something similar:
>
> GcsFileOptions options = new GcsFileOptions.Builder().mimeType("text/xml"
> ).acl("public-read").build();
>
>
> This makes the file accessible to everybody for reading. What should I put
> in the acl call to make the file accessible only to the GAE application. I
> figured that I can add the application to the team as the editor but I can
> not figure out the next step. A code snippet would be nice.
> Thanks in advance.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: 1.8.1 Pre-release SDKs Available.

[Tom Kaitchuck]

Jon the three criteria you specify are all available today:
You can use the Images API with files in GCS by getting a blobKey for them
which can be done by calling getBlobKey.
File upload is supported.
Headers can be specified at upload time.

On Sat, Jun 8, 2013 at 5:26 PM, jon  wrote:

> I will migrate to GCS if:
> * All conditions stated by Chris Ramsdale are met
> * It is fully compatible with Blobstore's dynamic resize feature (eg the
> =s parameter still works)
> * It allows for a file upload (especially from mobile apps) to be
> completed in one HTTP request
> * It sets far future expiry header
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: 1.8.1 Pre-release SDKs Available.

[Tom Kaitchuck]

Jeff: Replies inline


On Sat, Jun 8, 2013 at 9:49 AM, Jeff Schnitzer  wrote:
>
>
> Some questions:
>
>  * Will the current upload mechanism be preserved? Looking through the
> docs it appears the answer is that you create a signed url directly into
> GCS and have the client POST/PUT it, which seems like it should be
> compatible with the existing BlobstoreService.getUploadUrl() approach. But
> how do we get notification when the upload is complete? Right now the
> blobstore upload mechanism gives us a callback, and I do important things
> on this callback.
>

This will continue to function as it does now. This api is not affected.


>  * Will this work with the image service the way the blobstore does now? I
> transform, resize, and crop images on the fly - this rarely-lauded feature
> is actually one of my favorite parts of GAE.
>

Yes. In fact you can already use the image service with files in GCS and
blobstore using the same API.


>  * Will existing blobstore-based image urls be preserved? I have a lot of
> these in my datastore.
>

Today's announcement will have no effect on files store in blobstore.


>  * What does the GAE dev environment do with the GCS apis? What about the
> Local Unit Testing framework?
>

It will work and use local disk as the backing store.


> As long as there are sane answers to these questions, I have no objection
> to GCS... although it will require that I rewrite and some code:
>
>  * I read PDF data out of the blobstore using the files api, send it off
> to a service for transformation into an image, then write the image back to
> the blobstore. This sounds pretty straightforward with GCS.
>
>  * I de-dup all uploaded images using the hash, and track image
> references. This means I have a lot of data referencing BlobKeys in the
> datastore. This brings up the question, if data is migrated from Blobstore
> to GCS, what are the new keys? Will it be clear how to migrate this data?
>

You can generate a BlobKey from an Item in GCS so this code would not need
to be changed much.
Data migration is not being done, nor is it necessary for you to plan for
at this time.


> I don't object to rewriting code as long as the migration path is clear. I
> can appreciate consolidating development effort around a single
> blobstore-ish offering.
>
> Thanks,
> Jeff
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: 1.8.1 Pre-release SDKs Available.

[Tom Kaitchuck]

Vinny:
The intended replacement for the FilesAPI is not the library you link to,
but the one here: https://code.google.com/p/appengine-gcs-client/
It is very deliberately modeled after the FilesAPI.


On Sat, Jun 8, 2013 at 8:33 AM, Vinny P  wrote:

> Hi Chris, thanks for stopping by.
>
> *My need: Better libraries.*
>
> What I liked about the Files API (and particularly in regards to the
> Blobstore) is that it made writing files so unbelievably easy. For example,
> in Java all I needed to do was get an instance of FileService, then I could
> write and read using openWriteChannel/openReadChannel. The Files API
> handled the dirty part of configuring access to the datastore, managing the
> write, etc. Frankly, I think the Files API is one of the best engineered
> parts of GAE* (give whoever wrote that API a raise and a promotion
> please!).*
>
> But you look at the javadoc for the Java Cloud Storage library, and it's
> an utter mess. See for yourself:
> https://developers.google.com/resources/api-libraries/documentation/storage/v1beta2/java/latest/
>  .
> For one, there's not enough examples. Two, I have to mess around with
> BucketAccessControls and Builders and a whole mess of things. Chris, I just
> want to write some files to persistent storage, I don't want to have to
> micromanage everything else and deal with extra fluff. I'll micromanage if
> I have to, but the Blobstore took care of that for me.
>
> Get the guy who wrote the Files API and put him to work on writing the GCS
> library.
>
> -
> -Vinny P
> Technology & Media Advisor
> Chicago, IL
>
> My Go side project: http://invalidmail.com/
>
>
> On Saturday, June 8, 2013 1:04:23 AM UTC-5, Chris Ramsdale wrote:
>
>> a bunch of great feedback that we'll continue to address.  in regards to
>> timeline, we have no plans of decommissioning this API before end of year.
>> that said, assuming the following:
>>
>>- App Engine <=> Google Cloud Storage performance is equivalent to
>>(if not better than) App Engine <=> Blobstore
>>- all blobs were auto-migrated over to Cloud Storage (free of charge)
>>- all existing URLs just worked
>>
>> what would keep you from migrating over to a Cloud Storage-based solution?
>>
>> -- Chris
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: 1.8.1 Pre-release SDKs Available.

[Tom Kaitchuck]

Jon: The example you are pointing to is using two APIs. In java things
under the package: com.google.appengine.api.blobstore are _not_ affected by
this announcement.
Things under the java package: com.google.appengine.api.files are affected by
this announcement.
We will be providing a migration guide for how to change code to use the Google
Cloud Storage Client Library.
This library is Preview as of 1.8.1 and as such is guaranteed to move to
GA.

Just to emphasize, this is a deprecation announcement, not a decommission.
So your application will _not_ break with 1.8.1, but you should begin
looking at how to port it to the new library.
If you have questions about how to do this, feel free to create a thread on
this group.



On Wed, Jun 5, 2013 at 3:28 AM, jon  wrote:

> Chris just to clarify, this is *not* being deprecated is it?
> https://developers.google.com/appengine/docs/java/blobstore/overview#Writing_Files_to_the_Blobstore
>
> I sure hope not because I'm using it heavily in my apps.
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: SDK 1.7.7 still has sqlite3 import error

[Tom Piercy]

Yup. I'm having this issue too. Does anyone know of a way to fix it or a 
good path to start down to solving this?

On Wednesday, April 10, 2013 8:37:16 AM UTC+1, Mahron wrote:
>
> Stops when threadsafe is set to "no". Must have something to do with 
> threading I suppose. 
>
>
>
> File "C:\Program Files (x86)\Google\google_appengine\google\appengine 
> \ext\db\__init__.py", line 2326, in next 
> return self.__model_class.from_entity(self.__iterator.next()) 
>   File "C:\Program Files (x86)\Google\google_appengine\google\appengine 
> \datastore\datastore_query.py", line 2892, in next 
> next_batch = self.__batcher.next() 
>   File "C:\Program Files (x86)\Google\google_appengine\google\appengine 
> \datastore\datastore_query.py", line 2754, in next 
> return self.next_batch(self.AT_LEAST_ONE) 
>   File "C:\Program Files (x86)\Google\google_appengine\google\appengine 
> \datastore\datastore_query.py", line 2791, in next_batch 
> batch = self.__next_batch.get_result() 
>   File "C:\Program Files (x86)\Google\google_appengine\google\appengine 
> \api\apiproxy_stub_map.py", line 604, in get_result 
> return self.__get_result_hook(self) 
>   File "C:\Program Files (x86)\Google\google_appengine\google\appengine 
> \datastore\datastore_query.py", line 2528, in __query_result_hook 
> self._batch_shared.conn.check_rpc_success(rpc) 
>   File "C:\Program Files (x86)\Google\google_appengine\google\appengine 
> \datastore\datastore_rpc.py", line 1222, in check_rpc_success 
> rpc.check_success() 
>   File "C:\Program Files (x86)\Google\google_appengine\google\appengine 
> \api\apiproxy_stub_map.py", line 570, in check_success 
> self.__rpc.CheckSuccess() 
>   File "C:\Program Files (x86)\Google\google_appengine\google\appengine 
> \api\apiproxy_rpc.py", line 156, in _WaitImpl 
> self.request, self.response) 
>   File "C:\Program Files (x86)\Google\google_appengine\google\appengine 
> \ext\remote_api\remote_api_stub.py", line 200, in MakeSyncCall 
> self._MakeRealSyncCall(service, call, request, response) 
>   File "C:\Program Files (x86)\Google\google_appengine\google\appengine 
> \ext\remote_api\remote_api_stub.py", line 234, in _MakeRealSyncCall 
> raise pickle.loads(response_pb.exception()) 
>   File "C:\Python27\lib\pickle.py", line 1382, in loads 
> return Unpickler(file).load() 
>   File "C:\Python27\lib\pickle.py", line 858, in load 
> dispatch[key](self) 
>   File "C:\Python27\lib\pickle.py", line 1090, in load_global 
> klass = self.find_class(module, name) 
>   File "C:\Python27\lib\pickle.py", line 1124, in find_class 
> __import__(module) 
>   File "C:\Python27\lib\sqlite3\__init__.py", line 24, in  
> from dbapi2 import * 
>   File "C:\Python27\lib\sqlite3\dbapi2.py", line 27, in  
> from _sqlite3 import * 
>   File "C:\Program Files (x86)\Google\google_appengine\google\appengine 
> \tools\devappserver2\python\sandbox.py", line 856, in load_module 
> raise ImportError('No module named %s' % fullname) 
> ImportError: No module named _sqlite3 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] MapReduce Failures

[Tom Kaitchuck]

A "RetrySliceError" will result in a retry. If it is not, it could be that
you have the max reattempts on your task queue set too low. (Because Map
Reduce manages retries based on it's configuration, it is safe to set this
to unlimited.) Also you may want to take a look at shard retry:
https://code.google.com/p/appengine-mapreduce/wiki/PythonShardRetry which
is a new feature designed to make python Map Reduce more relyable.


On Fri, May 24, 2013 at 8:20 AM, Ranjit Chacko  wrote:

> I'm seeing shards abruptly fail in my MR jobs for no apparent reason and
> without retrying:
>
> task_name=appengine-mrshard-1581047187783C3601732-14-2-retry-0
> app_engine_release=1.8.0 instance=00c61b117c53a40e120ac864168a3fe51c2ce
>
> Shard 1581047187783C3601732-14 failed permanently.
>
> Is there some adjustment I can make to my queue parameters to avoid or
> reduce these issues?
>
> Recently I had been having problems with MR jobs throwing "UnknownErrors"
> and "ApplicationError followed by "RetrySliceErrors", and setting the
> min_backoff_seconds to 1 seemed to help with reducing the retry errors.
>
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Attention Java MapReduce users

[Tom Kaitchuck]

You should be able to sync to the SVN and get this update (r466)


On Wed, May 29, 2013 at 3:52 PM, Tom Kaitchuck wrote:

> My fault. I forgot to update the revision of the GCS client the ant
> build.xml loads. If you want the fix right away just change the build.xml
> to have the line where it defines the property "gcsversion" to be
> "r54".(You will also to need to delete the Jar if it has been previously
> downloaded into the same workspace.)
>
> I'll push out an update later today.
>
>
> On Wed, May 29, 2013 at 1:32 PM, Carter Maslan  wrote:
>
>> We get compile errors when we ant the latest version of the java
>> mapreduce library with java SDK 1.8.0. Error is "Builder() has private
>> access".
>> Is there an existing fix for that?
>>
>> [javac]
>> /mr2/appengine-mapreduce-read-only/java/src/com/google/appengine/tools/mapreduce/outputs/GoogleCloudStorageFileOutputWriter.java:39:
>> Builder() has private access in
>> com.google.appengine.tools.cloudstorage.GcsFileOptions.Builder
>> [javac] GCS_SERVICE.createOrReplace(file, new
>> GcsFileOptions.Builder().mimeType(mimeType).build());
>> [javac]   ^
>> [javac]
>> /mr2/appengine-mapreduce-read-only/java/src/com/google/appengine/tools/mapreduce/outputs/GoogleCloudStorageFileOutputWriter.java:39:
>> cannot find symbol
>> [javac] symbol  : method mimeType(java.lang.String)
>> [javac] location: class
>> com.google.appengine.tools.cloudstorage.GcsFileOptions.Builder
>> [javac] GCS_SERVICE.createOrReplace(file, new
>> GcsFileOptions.Builder().mimeType(mimeType).build());
>> [javac]
>> ^
>>
>>
>>
>>
>>
>> On Wed, May 22, 2013 at 4:48 PM, Tim Jones  wrote:
>>
>>> Awesome, I downloaded a few minutes ago and have been running against it
>>> with no problems.  Thanks!
>>>
>>>
>>> On Wed, May 22, 2013 at 4:46 PM, Tom Kaitchuck wrote:
>>>
>>>> The issue mentioned above (The NPE when the last item was already
>>>> written) has been fixed in the version 464 in the public svn.
>>>> https://code.google.com/p/appengine-mapreduce/source/detail?r=464
>>>>
>>>>
>>>> On Mon, May 20, 2013 at 6:32 PM, Tom Kaitchuck 
>>>> wrote:
>>>>
>>>>> This is a bug I am working on. It occurs when the last record to be
>>>>> written encounters a keyOrderingException (Meaning it was already written
>>>>> but an ACK was not received so it was retried.) So it should be rare and
>>>>> the retry of the shuffle that was added should cause it to be harmless.
>>>>>
>>>>> If you are seeing any broader problems send me a message off-list with
>>>>> your appId.
>>>>>
>>>>> I'll post an update here when a patch has been pushed out.
>>>>>
>>>>>
>>>>>
>>>>> On Mon, May 20, 2013 at 4:28 PM, Tim Jones  wrote:
>>>>>
>>>>>> After upgrading, I'm getting the following NullPointerException in
>>>>>> the InMemoryShuffler.  Is this a known issue?
>>>>>>
>>>>>> Caused by: java.lang.NullPointerException
>>>>>> at
>>>>>> com.google.appengine.tools.mapreduce.impl.InMemoryShuffleJob$1.run(InMemoryShuffleJob.java:234)
>>>>>>  at
>>>>>> com.google.appengine.tools.mapreduce.impl.InMemoryShuffleJob$1.run(InMemoryShuffleJob.java:231)
>>>>>> at
>>>>>> com.google.appengine.tools.mapreduce.impl.util.RetryHelper.doRetry(RetryHelper.java:62)
>>>>>>  at
>>>>>> com.google.appengine.tools.mapreduce.impl.util.RetryHelper.runWithRetries(RetryHelper.java:101)
>>>>>> at
>>>>>> com.google.appengine.tools.mapreduce.impl.InMemoryShuffleJob.closeFinally(InMemoryShuffleJob.java:231)
>>>>>>  at
>>>>>> com.google.appengine.tools.mapreduce.impl.InMemoryShuffleJob.writeOutput(InMemoryShuffleJob.java:227)
>>>>>> at
>>>>>> com.google.appengine.tools.mapreduce.impl.InMemoryShuffleJob.writeOutputs(InMemoryShuffleJob.java:243)
>>>>>>  at
>>>>>> com.google.appengine.tools.mapreduce.impl.InMemoryShuffleJob.run(InMemoryShuffleJob.java:253)
>>>>>> at
>>>>>> com.google.appengine.tools.mapreduce.impl.InMemoryShuffleJob.run(InMemoryShuffleJob.java:42)
>>>>>>

[google-appengine] Re: Attention Java MapReduce users

[Tom Kaitchuck]

My fault. I forgot to update the revision of the GCS client the ant
build.xml loads. If you want the fix right away just change the build.xml
to have the line where it defines the property "gcsversion" to be
"r54".(You will also to need to delete the Jar if it has been previously
downloaded into the same workspace.)

I'll push out an update later today.


On Wed, May 29, 2013 at 1:32 PM, Carter Maslan  wrote:

> We get compile errors when we ant the latest version of the java mapreduce
> library with java SDK 1.8.0. Error is "Builder() has private access".
> Is there an existing fix for that?
>
> [javac]
> /mr2/appengine-mapreduce-read-only/java/src/com/google/appengine/tools/mapreduce/outputs/GoogleCloudStorageFileOutputWriter.java:39:
> Builder() has private access in
> com.google.appengine.tools.cloudstorage.GcsFileOptions.Builder
> [javac] GCS_SERVICE.createOrReplace(file, new
> GcsFileOptions.Builder().mimeType(mimeType).build());
> [javac]   ^
> [javac]
> /mr2/appengine-mapreduce-read-only/java/src/com/google/appengine/tools/mapreduce/outputs/GoogleCloudStorageFileOutputWriter.java:39:
> cannot find symbol
> [javac] symbol  : method mimeType(java.lang.String)
> [javac] location: class
> com.google.appengine.tools.cloudstorage.GcsFileOptions.Builder
> [javac] GCS_SERVICE.createOrReplace(file, new
> GcsFileOptions.Builder().mimeType(mimeType).build());
> [javac]
> ^
>
>
>
>
>
> On Wed, May 22, 2013 at 4:48 PM, Tim Jones  wrote:
>
>> Awesome, I downloaded a few minutes ago and have been running against it
>> with no problems.  Thanks!
>>
>>
>> On Wed, May 22, 2013 at 4:46 PM, Tom Kaitchuck wrote:
>>
>>> The issue mentioned above (The NPE when the last item was already
>>> written) has been fixed in the version 464 in the public svn.
>>> https://code.google.com/p/appengine-mapreduce/source/detail?r=464
>>>
>>>
>>> On Mon, May 20, 2013 at 6:32 PM, Tom Kaitchuck wrote:
>>>
>>>> This is a bug I am working on. It occurs when the last record to be
>>>> written encounters a keyOrderingException (Meaning it was already written
>>>> but an ACK was not received so it was retried.) So it should be rare and
>>>> the retry of the shuffle that was added should cause it to be harmless.
>>>>
>>>> If you are seeing any broader problems send me a message off-list with
>>>> your appId.
>>>>
>>>> I'll post an update here when a patch has been pushed out.
>>>>
>>>>
>>>>
>>>> On Mon, May 20, 2013 at 4:28 PM, Tim Jones  wrote:
>>>>
>>>>> After upgrading, I'm getting the following NullPointerException in the
>>>>> InMemoryShuffler.  Is this a known issue?
>>>>>
>>>>> Caused by: java.lang.NullPointerException
>>>>> at
>>>>> com.google.appengine.tools.mapreduce.impl.InMemoryShuffleJob$1.run(InMemoryShuffleJob.java:234)
>>>>>  at
>>>>> com.google.appengine.tools.mapreduce.impl.InMemoryShuffleJob$1.run(InMemoryShuffleJob.java:231)
>>>>> at
>>>>> com.google.appengine.tools.mapreduce.impl.util.RetryHelper.doRetry(RetryHelper.java:62)
>>>>>  at
>>>>> com.google.appengine.tools.mapreduce.impl.util.RetryHelper.runWithRetries(RetryHelper.java:101)
>>>>> at
>>>>> com.google.appengine.tools.mapreduce.impl.InMemoryShuffleJob.closeFinally(InMemoryShuffleJob.java:231)
>>>>>  at
>>>>> com.google.appengine.tools.mapreduce.impl.InMemoryShuffleJob.writeOutput(InMemoryShuffleJob.java:227)
>>>>> at
>>>>> com.google.appengine.tools.mapreduce.impl.InMemoryShuffleJob.writeOutputs(InMemoryShuffleJob.java:243)
>>>>>  at
>>>>> com.google.appengine.tools.mapreduce.impl.InMemoryShuffleJob.run(InMemoryShuffleJob.java:253)
>>>>> at
>>>>> com.google.appengine.tools.mapreduce.impl.InMemoryShuffleJob.run(InMemoryShuffleJob.java:42)
>>>>>  ... 47 more
>>>>>
>>>>> On Wednesday, May 1, 2013 1:32:51 PM UTC-7, Tom Kaitchuck wrote:
>>>>>>
>>>>>> If you are using the experimental Java MapReduce library for App
>>>>>> Engine, you are strongly encouraged to update to the latest version of 
>>>>>> the
>>>>>> library in the public svn: https://code.google.com/p/**
>>>>>> appengine-mapreduce/source/**checkout<h

Re: [google-appengine] Re: Bad news for GAE/Java from Google I/O

[Tom]

I'm not that experienced with GAE, but I'm wondering whether it really is 
that simple.  If an instance of my app takes 10s to start up, and my app 
can, at peak, receive X requests in a 10s period, then wouldn't I require 
close to X idle instances running all the time to satisfy your criteria. 
 And wouldn't that be prohibitively expensive for most apps?

If my understanding is correct, then I think a simple on/off option (which 
has been suggested) would not be a good solution for most developers.  I 
certainly want them to figure out how to reduce response latencies and 
instance startup times, but I would need to be in control of the balance 
between response time and cost.

As to your point about Heroku/Azure, do you know how they resolved the 
problem?  Did they manage to significantly reduce startup times or have 
they come up with some other scheme?

One final point - I'm baffled at Google's decision to support Go on GAE. 
 When I looked at Go and wondered about design decisions that seemed odd, 
the answer I always got was that it was designed to reduce build times for 
enormous systems.  How does that primary design criteria make it suitable 
for GAE?  If they had instead added a runtime for something like LLVM then 
people like myself (and the vast number of other devs who know C/C++/OC) 
could take advantage of the near-instantaneous startup times that you are 
talking about, with our existing skills and libraries.


Tom

On Thursday, May 23, 2013 2:07:18 PM UTC-4, Ryan Chazen wrote:
>
> And it's possible to not use app engine at all, and then not have to worry 
> about how fast your app starts. As a bonus, it's far cheaper as well! 
> "Possible" is never a useful word to throw around here - it's possible to 
> rewrite your java webapp in Go, it's possible to convince people they don't 
> even need a webapp and just use pen and paper.
>
> The point is, GAE/J pointing user requests to servers that have not 
> finished loading is broken, and Google needs to fix it if they want to 
> compete with Heroku/Azure/etc who all seem to have this issue sorted out. 
> This isn't some kind of technical impossibility here - every other hosting 
> provider of this type manages to solve this issue. And GAE is the most 
> expensive of the lot in terms of compute resources. GAE should be offering 
> the best solution, not a "jump through hoops and pray it works until the 
> next GAE release where everything crawls at half speed for a day" solution.
>
>
>
> On Wed, May 22, 2013 at 11:53 AM, Bediako George <
> bediako...@lucidtechnics.com > wrote:
>
>> So our Airlift application framework starts up pretty much immediately on 
>> App Engine for Java.
>>
>> It runs on the Java runtime, but developers write request handlers in 
>> JavaScript via Rhino.  Even though it is production ready (we do have 
>> several customers using it already) it is not ready for public consumption 
>> (we are looking to release it at the end the summer).
>>
>> I am not mentioning this as an alternative to what you are doing. 
>>  Instead, I want to make the point that it is possible to create a 
>> framework in Java that plays well with App Engine.
>>
>> https://github.com/LucidTechnics/Airlift
>>
>> Bediako
>>
>> On Friday, May 17, 2013 2:20:53 AM UTC-4, Marcel Overdijk wrote:
>>>
>>> This is an interesting question indeed.
>>>
>>> I don't believe startup times for Java will become better on GAE; it's 
>>> also very typical for in Java land that startup times take > 30s for medium 
>>> to large apps or depending on the frameworks chosen.
>>> This is no problem when you up front spin up the number of required 
>>> instances like on a VPS or CloudFoundry PaaS.
>>> One of the great things about GAE is the programming API and the 
>>> autoscaling part, but this autoscaling (and thus spinning up and killing 
>>> instances) bites back unfortunately.
>>>
>>> Couldn't Google prohibit cold requests being served to users? In many 
>>> cases it would better to have latency when a requested is routed to a warm 
>>> instance.
>>> I think this would solve many problems. What do you think?
>>>
>>> To bad Google is not participating in these discussions.
>>>
>>> But with great technologies and frameworks being delivered by Google 
>>> (like Angular JS) Google cannot be taken seriously when they say just go 
>>> back 10 years and use servlets, static factories, hard coded configs etc. 
>>> This is a real development nightmare.
>>>
>>> ^M
>>>
>>>