[appengine-java] Re: When is the ConcurrentModificationException thrown in GAE?
Man, i've been there. Check out sharding - you can only update a
single entity a couple of times a second or else you'll get a
ConcurrentModificationException - you have to shard your entities.
Ben
nimbits.com
On Jan 3, 11:20 pm, Harshad harshad...@gmail.com wrote:
Hi,
I am reading the official GAE documentation on transactions and I
can't understand when a ConcurrentModificationException is thrown.
Look at one of the examples which I am copy-pasting here:
int retries = 3;
while (true) {
Transaction txn = datastore.beginTransaction();
try {
Key boardKey = KeyFactory.createKey(MessageBoard,
boardName);
Entity messageBoard = datastore.get(boardKey);
long count = (Long) messageBoard.getProperty(count);
++count;
messageBoard.setProperty(count, count);
datastore.put(messageBoard);
txn.commit();
break;
} catch (ConcurrentModificationException e) {
if (retries == 0) {
throw e;
}
// Allow retry to occur
--retries;
} finally {
if (txn.isActive()) {
txn.rollback();
}
}
}
Now, all the writes to the datastore (in this example) are wrapped
under a transaction. So why would a ConcurrentModificationException be
thrown?
Does it happen when some other code which is not wrapped in a
transaction updates the same entity that is being modified by the
above code? If I ensure that all code that updates an Entity is always
wrapped in a transaction, is it guaranteed that I won't get a
ConcurrentModificationException?
thanks,
Harshad
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Set password for appcfg.sh update
Hi Ian, Yeah, --email only lets you define your username. --passin is used to make sure that it should _not_ store the password and ask you for the password every time. I'd need something like --password. ;-) Thanks, Ben On Mar 23, 3:24 am, Ian Marshall ianmarshall...@gmail.com wrote: Ben, Have you tried the option(s) (--email and) --passin? Ian On Mar 22, 7:08 pm, Benjamin Muschko benjamin.musc...@gmail.com wrote: Hi, Is there way to use a pre-defined password (e.g. set as a parameter or from some file) for the update task (http://code.google.com/appengine/ docs/java/tools/uploadinganapp.html)? Unfortunately, I couldn't find an appropriate parameter. I'd like to upload my app to App Engine as part of an automated Continuous Integration process without having to enter the password on the command-line. Thanks, Ben -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Set password for appcfg.sh update
Hi, Is there way to use a pre-defined password (e.g. set as a parameter or from some file) for the update task (http://code.google.com/appengine/ docs/java/tools/uploadinganapp.html)? Unfortunately, I couldn't find an appropriate parameter. I'd like to upload my app to App Engine as part of an automated Continuous Integration process without having to enter the password on the command-line. Thanks, Ben -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Local development server classpath - com.google.appengine.tools.KickStart
Didier, Thanks for your reply. I am not using Eclipse. Instead I was trying to run KickStart.main() within an existing Java process for a build plugin. The Ant page you mentioned simply copies the JARs into WEB-INF/ lib. In the meantime I found a workaround for what I was trying to do. Still...I'd still be interested to know if you can add additional JARs when calling KickStart.main(). Thanks, Ben On Mar 19, 1:21 am, Didier Durand durand.did...@gmail.com wrote: Hi, You will be ok if you add not only 1 single gae jar (appengine-tools- api.jar) but all those (about 2) appearing in the GAE sdk library on Eclipse when you create a new project + those needed to run locally. The simplest way for you is to follow he hints given inhttp://code.google.com/appengine/docs/java/tools/ant.html regards didier On Mar 18, 8:55 pm, Benjamin Muschko benjamin.musc...@gmail.com wrote: Hi, I have a question about the runtime classpath that is being used when starting up a local development server. It is required to set appengine-tools-api.jar in the classpath a parameter. My application directory web_app_dir does not include some of the libraries that are required at runtime (see below someother.jar); they sit in a different directory. I tried to add them using the -cp parameter but they don't seem to get evaluated. I get a java.lang.NoClassDefFoundError. This is the call I make: java -cp /home/ben/dev/tools/appengine-java-sdk-1.4.2/lib/appengine- tools-api.jar:/home/ben/dev/someother.jar com.google.appengine.tools.KickStart com.google.appengine.tools.development.DevAppServerMain web_app_dir Can somebody please shed some light on this? Are only the libraries used that sit in web_app_dir/WEB-INF/lib? Is there any way I can add additional libraries using an parameter? Thanks, Ben -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Local development server classpath - com.google.appengine.tools.KickStart
Hi, I have a question about the runtime classpath that is being used when starting up a local development server. It is required to set appengine-tools-api.jar in the classpath a parameter. My application directory web_app_dir does not include some of the libraries that are required at runtime (see below someother.jar); they sit in a different directory. I tried to add them using the -cp parameter but they don't seem to get evaluated. I get a java.lang.NoClassDefFoundError. This is the call I make: java -cp /home/ben/dev/tools/appengine-java-sdk-1.4.2/lib/appengine- tools-api.jar:/home/ben/dev/someother.jar com.google.appengine.tools.KickStart com.google.appengine.tools.development.DevAppServerMain web_app_dir Can somebody please shed some light on this? Are only the libraries used that sit in web_app_dir/WEB-INF/lib? Is there any way I can add additional libraries using an parameter? Thanks, Ben -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: java.lang.OutOfMemoryError: Java heap space
Thanks Peter - i think i'm forced to use a query like the one above
because the only criteria i have for deleting these objects is a
timestamp and the foreign key. So I would have to make a seperate
query to get the object's keys. Please let me know if i'm missing some
trick here or if it makes sense to do this another way
It looks like the only solution that isn't causing errors is to delete
1000 at a time and then restart the delete task until the result is
zero. This is all very expensive though, it looks like it costs about
$2.00 to delete a couple hundred thousand records this way.
The big question is, is it the deletePersistentAll that's costing the
cpu time? Should i get the query and iterate through the result and
use the low level api to delete each key?
Here is what i currently have that is costing a lot of $ to purge a
couple 100K of records from the database
Query q = pm.newQuery(RecordedValue.class,pointFK== k
timestamp
d);
q.declareImports(import java.util.Date);
q.setRange(0,1000);
q.setOrdering(timestamp ascending);
MapString, Object args = new HashMapString,
Object();
args.put(k,pointId);
args.put(d, d.getTime());
ListRecordedValue v = (ListRecordedValue)
q.executeWithMap(args);
long count = v.size();
if (count 0)
{
pm.deletePersistentAll(v);
DataServiceImpl.startDeleteDataTask(pointId,true,expDays);
}
On Feb 12, 1:57 pm, Peter Liu tinyee...@gmail.com wrote:
Try using the low level API and do keys only query with limit of 1000
(and delete them) repeatedly instead of retrieving whole objects.
I am guessing the out of memory is due to large amount of objects
returned by the query. Keys only query also use much less api cpu.
On Feb 12, 9:22 pm, Benjamin bsaut...@gmail.com wrote:
I'm getting errors when a task kicks off to delete a lot of data based
on a timestamp. I enabled billing and already chewed through $0.50 in
CPU time, but i'm still getting the error message. Is there anything
else I should do? I was trying to avoid splitting the task up with a
result limit or something, i really just need to blow away persisted
objects that have a timestamp older than a specified date - this
snippet of code causes the error:
PersistenceManager pm = PMF.get().getPersistenceManager();
Calendar d = Calendar.getInstance();
long retVal = 0;
if (expDays 0)
{
d.add(Calendar.DATE,(expDays * -1));
Query q =
pm.newQuery(RecordedValue.class,pointFK== k timestamp
d);
q.declareImports(import java.util.Date);
MapString, Object args = new HashMapString,
Object();
args.put(k,pointId);
args.put(d, d.getTime());
retVal = q.deletePersistentAll(args);
}
pm.close();
return retVal;
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: java.lang.OutOfMemoryError: Java heap space
update - the whole time i was researching this I kept thinking a key
only query was when you used the low level api to query for an object
using the key. Confusion on my part that what everyone meant was to
write a query that only requested the id of the objects, and that was
much faster/cheaper. Here is the final version of my delete code that
does a key only query and cycles through 1000 results at a time
Thanks!
d.add(Calendar.DATE,(expDays * -1));
Query q = pm.newQuery(select id from +
RecordedValue.class.getName());
q.setFilter(pointFK== k timestamp d);
q.declareImports(import java.util.Date);
q.setRange(0,1000);
q.setOrdering(timestamp ascending);
MapString, Object args = new HashMapString,
Object();
args.put(k,pointId);
args.put(d, d.getTime());
ListRecordedValue v = (ListRecordedValue)
q.executeWithMap(args);
long count = v.size();
if (count 0)
{
pm.deletePersistentAll(v);
DataServiceImpl.startDeleteDataTask(pointId,true,expDays);
}
On Feb 13, 5:23 am, Benjamin bsaut...@gmail.com wrote:
Thanks Peter - i think i'm forced to use a query like the one above
because the only criteria i have for deleting these objects is a
timestamp and the foreign key. So I would have to make a seperate
query to get the object's keys. Please let me know if i'm missing some
trick here or if it makes sense to do this another way
It looks like the only solution that isn't causing errors is to delete
1000 at a time and then restart the delete task until the result is
zero. This is all very expensive though, it looks like it costs about
$2.00 to delete a couple hundred thousand records this way.
The big question is, is it the deletePersistentAll that's costing the
cpu time? Should i get the query and iterate through the result and
use the low level api to delete each key?
Here is what i currently have that is costing a lot of $ to purge a
couple 100K of records from the database
Query q = pm.newQuery(RecordedValue.class,pointFK==
k timestamp
d);
q.declareImports(import java.util.Date);
q.setRange(0,1000);
q.setOrdering(timestamp ascending);
MapString, Object args = new HashMapString,
Object();
args.put(k,pointId);
args.put(d, d.getTime());
ListRecordedValue v = (ListRecordedValue)
q.executeWithMap(args);
long count = v.size();
if (count 0)
{
pm.deletePersistentAll(v);
DataServiceImpl.startDeleteDataTask(pointId,true,expDays);
}
On Feb 12, 1:57 pm, Peter Liu tinyee...@gmail.com wrote:
Try using the low level API and do keys only query with limit of 1000
(and delete them) repeatedly instead of retrieving whole objects.
I am guessing the out of memory is due to large amount of objects
returned by the query. Keys only query also use much less api cpu.
On Feb 12, 9:22 pm, Benjamin bsaut...@gmail.com wrote:
I'm getting errors when a task kicks off to delete a lot of data based
on a timestamp. I enabled billing and already chewed through $0.50 in
CPU time, but i'm still getting the error message. Is there anything
else I should do? I was trying to avoid splitting the task up with a
result limit or something, i really just need to blow away persisted
objects that have a timestamp older than a specified date - this
snippet of code causes the error:
PersistenceManager pm = PMF.get().getPersistenceManager();
Calendar d = Calendar.getInstance();
long retVal = 0;
if (expDays 0)
{
d.add(Calendar.DATE,(expDays * -1));
Query q =
pm.newQuery(RecordedValue.class,pointFK== k timestamp
d);
q.declareImports(import java.util.Date);
MapString, Object args = new HashMapString,
Object();
args.put(k,pointId);
args.put(d, d.getTime());
retVal = q.deletePersistentAll(args);
}
pm.close();
return retVal;
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe
[appengine-java] java.lang.OutOfMemoryError: Java heap space
I'm getting errors when a task kicks off to delete a lot of data based
on a timestamp. I enabled billing and already chewed through $0.50 in
CPU time, but i'm still getting the error message. Is there anything
else I should do? I was trying to avoid splitting the task up with a
result limit or something, i really just need to blow away persisted
objects that have a timestamp older than a specified date - this
snippet of code causes the error:
PersistenceManager pm = PMF.get().getPersistenceManager();
Calendar d = Calendar.getInstance();
long retVal = 0;
if (expDays 0)
{
d.add(Calendar.DATE,(expDays * -1));
Query q = pm.newQuery(RecordedValue.class,pointFK== k
timestamp
d);
q.declareImports(import java.util.Date);
MapString, Object args = new HashMapString,
Object();
args.put(k,pointId);
args.put(d, d.getTime());
retVal = q.deletePersistentAll(args);
}
pm.close();
return retVal;
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Struggling w/datastore relationship
I just came across this document: http://www.scribd.com/doc/24330945/No-Relation-The-Mixed-Blessings-of-Non-Relational-Databases It helps, but doesn't completely answer my question. It would appear that what I'm wanting to do is an EAV (entities/attributes/values) type relationship, however, I'm still not clear on how to do so in GAE/J and still be able to query how I need to. As for adding a list to each of User and Tag, it doesn't keep the relationship between the User, Tag AND URL... just two of the three... that's what's causing me heartburn here... :) Or, perhaps it does, and I'm too relationally-minded to see it? Thanks! -Ben -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Struggling w/datastore relationship
Thanks much for the link! It doesn't directly answer my issue, but it did give me a few more ideas about how to go about this... I've watched it three times (so far)! :) Despite seeing the merge-join in action, I don't think it quite matches my model, so I'm trying to visualize how to make my data match one of these models... and that's not working so far either. Anyone else have ideas? The issue that isn't addressed is the third relationship... everything demonstrated and talked about in these slides only discuss two relationships, and it doesn't appear that a third would work (or at least not very well!). Also, the limit _is_ over 2000, but sounds like it's 5000 per index. The extended index class is pretty interesting, but I'm not completely understanding that either... does that mean I create a new instance of the index class each time I reach 5000 entries? Or does it mean I create a new index class itself (i.e. three instances of userIndexClass or one instance of each of userIndexClassOne, userIndexClassTwo and userIndexClassThree)? Thanks again! -Ben -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: multiple list properties
Bump, I would be interested in learning more about this as well. Thanks! -Ben -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: UserService functionality when run via cron
The bug reference would be nice. If only to vote it up. On Wed, Dec 15, 2010 at 2:10 AM, andrew aute...@gmail.com wrote: I submitted a bug on the subject, and it was accepted. If you want the bug reference I can find it. Andrew -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Datastore metadata property query (SDK 1.4)
Never mind. Got it resolved. Must have been an issue with my environment. Ben -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: How to set Google App Engine cron job using different interval in different period of time?
i'd just stick a check on the current time in the cron's code, so it always runs every 5 minutes but only actually does the thing it's supposed to do when it's the right time of day. On Dec 13, 5:31 am, EtuO nbaer...@gmail.com wrote: How to config a cron job to run every 5 minutes between 9:00am~20:00pm。 -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Any idea on channel api without javascript?
Hi there, I am currently using GAE as our game's server, we are not using javascript as the frontend, but channel api is quite a useful stuff for us. Is there any way we can use the channel api without javascript? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Running Google App Engine from within a HSM.
Hi Ikai, Thank you for your fast response. I'm not sure I understand the question. Are you looking to run the GAE SDK inside your own hardware? Or are you looking to run something like AppScale (http://code.google.com/p/appscale/) with the security module? I am looking to run something like AppScale (http://code.google.com/p/appscale/ ) with the security module? Specifically, I was wondering if it was possible to run AppServer (part of AppScale) inside a hardware security module (HSM), where that HSM is capable of running Java Standard Edition software. So in a hybrid cloud deployment some of my business logic processing particularly sensitive information (social security numbers, credit cards, and so on) is running entirely within the protection of a hardware security module (running AppServer) managed at my site, and the remaining parts of my business logic responsible for processing less sensitive data is running on the public cloud provided by Google. Thanks! Benjamin Gittins On 19/11/2010, at 7:58 PM, Ikai Lan (Google) wrote: I'm not sure I understand the question. Are you looking to run the GAE SDK inside your own hardware? Or are you looking to run something like AppScale (http://code.google.com/p/appscale/) with the security module? If you're talking about deployment in production, you don't have any choice over what it's run on. -- Ikai Lan Developer Programs Engineer, Google App Engine Blogger: http://googleappengine.blogspot.com Reddit: http://www.reddit.com/r/appengine Twitter: http://twitter.com/app_engine On Thu, Nov 18, 2010 at 7:08 AM, Benjamin Gittins gittin...@gmail.com wrote: I am relatively new to Google App Engine. I'd like to know if it is possible to run Google App Engine inside a Java Standard Edition compliant network attached hardware security module, such as SafeNet LUNA-SP? ( www.safenet-inc.com/products/data-protection/hardware-security-modules/luna-sp/ ) Thanks, Benjamin Gittins -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com . To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en . -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com . To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en . -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Running Google App Engine from within a HSM.
I am relatively new to Google App Engine. I'd like to know if it is possible to run Google App Engine inside a Java Standard Edition compliant network attached hardware security module, such as SafeNet LUNA-SP? ( www.safenet-inc.com/products/data-protection/hardware-security-modules/luna-sp/ ) Thanks, Benjamin Gittins -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: mapreduce cron
Hi Jacek, there is an example in mapper project source code: http://code.google.com/p/appengine-mapreduce/source/browse/trunk/java/example/com/google/appengine/demos/mapreduce/TestServlet.java Hope it helps. On Oct 23, 12:03 am, jacek.ambroziak jacek.ambroz...@gmail.com wrote: How to start a MapReduce job as a cron scheduled task? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Newly deployed app doesn’t se em always reflected on the server.
are you changing the version number? You'd need to go to the console and change the version that's being served. Maybe you're caching behind a proxy or viewing a cached version on your end? try loading the url in a browser and hit F5 On Oct 15, 4:21 pm, nnhobbit nnclou...@gmail.com wrote: I deployed an app, in which an URL was originally serving simply a “hello”. Later I changed the content served from that URL to something else and deployed it again. However the new content did not show up and the page still showed the original “hello”. Another (possibly related) issue I'm experiencing is that an URL that was removed from my code remained accessible for about a day or so after the new code was deployed. (The issue could be either from the delayed deployment or some sort of cache?) p.s. everything is working as expected on devel environment. p.s. admin console shows successful deployment. Anyone encountered similar issues? Any suggestions? Thanks -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: App Engine and IP Addresses
Guys - thank you. It's working great. I'm actually impressed at how I can us the IP i get from google.com or appspot.com and as long as I provide the host my POST get's directed to my app engine app. I will post more about this in my blog: http://nimbits.blogspot.com/ I now have an Arduino Microcontroller board with an Ethernet shield that can post it's data directly to my Nimbits Data Logging Service on App Engine : http://www.nimbits.com The Arduino C code is simply a basic web client (using ethernet.h) that does: client.println(GET /service/currentvalue? point=testemail=bsaut...@gmail.comformat=json HTTP/1.1); client.println(Host:nimbits1.appspot.com); client.println(Accept-Language:en-us,en;q=0.5); client.println(Accept-Encoding:gzip,deflate); client.println(Connection:close); client.println(Cache-Control:max-age=0); client.println(); Worth mentioning that I had trouble with http keep alives - i had to add a connection close header. Not sure why. On Oct 10, 5:36 pm, Maxim Veksler ma...@vekslers.org wrote: Exactly. Usehttps://addons.mozilla.org/en-US/firefox/addon/6647/to see what headers your browser send as part of the HTTP GET request and emulate them in C code. Should work :). On Sat, Oct 9, 2010 at 11:45 PM, Peter Ondruska peter.ondru...@gmail.comwrote: When connecting to IP address you need to use HTTP host header so that GAE knows which application/virtual server you want. On Oct 9, 6:26 pm, Benjamin bsaut...@gmail.com wrote: I've been working on a challenge over the past couple of days and I could really use a knowledge transfer on App Engine, Domains and IP addresses. I seem to be missing something. I'm trying to write a library for Arduino Micro-controllers to do HTTP Posts to a servlet hosted on appengine. For example The URL of the servlet is http://nimbits1.appspot.com/service/currentvalue?point=testformat=json Do to limitations on the Arduino device, i need to get an IP Address that will resolve to nimbits1.appspot.com first, before doing my post to /service/currentvalue?point=testformat=json I have the C code to request an IP from DNS of a domain which works without a problem. So far so good. My problem is my requests seem to hit a brick wall when I try to use the IP instead of the Domain in my requests. Let's say I ping nimbits1.appspot.com - I get 74.125.113.121 or 72.14.204.141 back from the DNS Server. This takes me to Google servers, but not my app. I'm guessing that the server want the subdomain in the request but i'm not provided one. I registered a new domain: nimbits.org on godaddy and followed Nick Johnson's fine tutorial on mapping naked domains to have nimbits.org redirect to nimbits1.appspot.com (As a permanent redirect without masking) http://blog.notdot.net/2009/12/Naked-domains-on-App-Engine if i navigate tohttp://nimbits.orgI redirect ok tohttp:// nimbits1.appspot.com Further, if i do a wget in a linux terminal I can see the IP's i'm resolving to: benja...@ben-ubws01:~$ wget nimbits.org --2010-10-09 12:20:43-- http://nimbits.org/ Resolving nimbits.org... 64.202.189.170 Connecting to nimbits.org|64.202.189.170|:80... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location:http://nimbits1.appspot.com[following] --2010-10-09 12:20:44-- http://nimbits1.appspot.com/ Resolving nimbits1.appspot.com... 64.233.169.141 Connecting to nimbits1.appspot.com|64.233.169.141|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] Saving to: `index.html.10' [ = ] 3,376 --.-K/s in 0.003s 2010-10-09 12:20:44 (1.18 MB/s) - `index.html.10' saved [3376] If i try and navigate to any of the above IP Addresses i.ehttp:// 64.233.169.141 I endup on google or godaddy, but not my app. Any help would be greatly appriciated. I may have to resort to having users point their arduino to an internal web server that can forward the request, but having arduino devices post directly to app engine would be very cool. -Ben -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2B unsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http
[appengine-java] App Engine and IP Addresses
I've been working on a challenge over the past couple of days and I could really use a knowledge transfer on App Engine, Domains and IP addresses. I seem to be missing something. I'm trying to write a library for Arduino Micro-controllers to do HTTP Posts to a servlet hosted on appengine. For example The URL of the servlet is http://nimbits1.appspot.com/service/currentvalue?point=testformat=json Do to limitations on the Arduino device, i need to get an IP Address that will resolve to nimbits1.appspot.com first, before doing my post to /service/currentvalue?point=testformat=json I have the C code to request an IP from DNS of a domain which works without a problem. So far so good. My problem is my requests seem to hit a brick wall when I try to use the IP instead of the Domain in my requests. Let's say I ping nimbits1.appspot.com - I get 74.125.113.121 or 72.14.204.141 back from the DNS Server. This takes me to Google servers, but not my app. I'm guessing that the server want the subdomain in the request but i'm not provided one. I registered a new domain: nimbits.org on godaddy and followed Nick Johnson's fine tutorial on mapping naked domains to have nimbits.org redirect to nimbits1.appspot.com (As a permanent redirect without masking) http://blog.notdot.net/2009/12/Naked-domains-on-App-Engine if i navigate to http://nimbits.org I redirect ok to http://nimbits1.appspot.com Further, if i do a wget in a linux terminal I can see the IP's i'm resolving to: benja...@ben-ubws01:~$ wget nimbits.org --2010-10-09 12:20:43-- http://nimbits.org/ Resolving nimbits.org... 64.202.189.170 Connecting to nimbits.org|64.202.189.170|:80... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: http://nimbits1.appspot.com [following] --2010-10-09 12:20:44-- http://nimbits1.appspot.com/ Resolving nimbits1.appspot.com... 64.233.169.141 Connecting to nimbits1.appspot.com|64.233.169.141|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] Saving to: `index.html.10' [ = ] 3,376 --.-K/s in 0.003s 2010-10-09 12:20:44 (1.18 MB/s) - `index.html.10' saved [3376] If i try and navigate to any of the above IP Addresses i.e http://64.233.169.141 I endup on google or godaddy, but not my app. Any help would be greatly appriciated. I may have to resort to having users point their arduino to an internal web server that can forward the request, but having arduino devices post directly to app engine would be very cool. -Ben -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: App Engine and IP Addresses
I did just get an arduino to post to app engine by using a reverse proxy on an apache server on my LAN - but i'd rather hit app engine directly... On Oct 9, 12:26 pm, Benjamin bsaut...@gmail.com wrote: I've been working on a challenge over the past couple of days and I could really use a knowledge transfer on App Engine, Domains and IP addresses. I seem to be missing something. I'm trying to write a library for Arduino Micro-controllers to do HTTP Posts to a servlet hosted on appengine. For example The URL of the servlet is http://nimbits1.appspot.com/service/currentvalue?point=testformat=json Do to limitations on the Arduino device, i need to get an IP Address that will resolve to nimbits1.appspot.com first, before doing my post to /service/currentvalue?point=testformat=json I have the C code to request an IP from DNS of a domain which works without a problem. So far so good. My problem is my requests seem to hit a brick wall when I try to use the IP instead of the Domain in my requests. Let's say I ping nimbits1.appspot.com - I get 74.125.113.121 or 72.14.204.141 back from the DNS Server. This takes me to Google servers, but not my app. I'm guessing that the server want the subdomain in the request but i'm not provided one. I registered a new domain: nimbits.org on godaddy and followed Nick Johnson's fine tutorial on mapping naked domains to have nimbits.org redirect to nimbits1.appspot.com (As a permanent redirect without masking) http://blog.notdot.net/2009/12/Naked-domains-on-App-Engine if i navigate tohttp://nimbits.org I redirect ok tohttp://nimbits1.appspot.com Further, if i do a wget in a linux terminal I can see the IP's i'm resolving to: benja...@ben-ubws01:~$ wget nimbits.org --2010-10-09 12:20:43-- http://nimbits.org/ Resolving nimbits.org... 64.202.189.170 Connecting to nimbits.org|64.202.189.170|:80... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location:http://nimbits1.appspot.com[following] --2010-10-09 12:20:44-- http://nimbits1.appspot.com/ Resolving nimbits1.appspot.com... 64.233.169.141 Connecting to nimbits1.appspot.com|64.233.169.141|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] Saving to: `index.html.10' [ = ] 3,376 --.-K/s in 0.003s 2010-10-09 12:20:44 (1.18 MB/s) - `index.html.10' saved [3376] If i try and navigate to any of the above IP Addresses i.ehttp://64.233.169.141 I endup on google or godaddy, but not my app. Any help would be greatly appriciated. I may have to resort to having users point their arduino to an internal web server that can forward the request, but having arduino devices post directly to app engine would be very cool. -Ben -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] query for null missing properties
Brain freeze guys - I added a property to my object and persisted in the datastore. The older object have now have a column that when viewed in the control panel have a value of missing. I need to query those objects so i can update them - propname == null , propname == - I can't seem to find the right syntax for this. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: query for null missing properties
OK thanks Ikai, I was trying to avoid that but see no problem with it. Ben On Sep 13, 5:15 pm, Ikai Lan (Google) ikai.l+gro...@google.com wrote: You can't. Empty properties are the equivalent of unindexed properties. You'll need to iterate over all the Entities in your datastore and update them. The Mapper API is a very good tool for this: http://code.google.com/p/appengine-mapreduce/ On Mon, Sep 13, 2010 at 12:20 PM, Benjamin bsaut...@gmail.com wrote: Brain freeze guys - I added a property to my object and persisted in the datastore. The older object have now have a column that when viewed in the control panel have a value of missing. I need to query those objects so i can update them - propname == null , propname == - I can't seem to find the right syntax for this. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2B unsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Google Update 1.3.6
I had this problem and found it was from installing the eclipse plug in using the install new software item on the help menu instead of using the install from the new eclipse marketplace item under help or the other way around, i don't remember - but i was installing 64 bit / 32 bit and all sorts of combinations and that turned out to be the fix for me. The symptom was that after i installed everything I didn't have the web application option when creating new projects. Ben http://javagwt.blogspot.com/ On Aug 29, 9:21 am, EarlyDoors jon.bangbang.greenw...@googlemail.com wrote: Hi All I'm using Eclipse 3.5 and have just updated to GAEJ SDK 1.3.6 and GWT SDK 2.0.4 and no longer have the ability to create a Web Application Project. I seem to remember having this problem updating to 1.3.5. Can someone please remind me of what else I need to reconfigure in Eclipse? Thanks in advance -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] facebook graph solution
I spent some time this morning and consolidated how I write app engine apps in java, do oauth with facebook a host my apps in a facebook iframe - interacting with the users profile, wall, newsfeed etc with the facebook graph. Hope this helps - enjoy! http://javagwt.blogspot.com/2010/08/facebook-apps-on-app-engine-without-any.html -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Facebook API Integration
I integrate my app engine apps with facebook without any third part api - just http, oauth and the facebook graph - let me know if you'd like to know more. http://developers.facebook.com/docs/api On Aug 18, 3:40 pm, Peter Simun si...@seges.sk wrote: This can helps youhttp://java.dzone.com/articles/efective-json-google-web Peer On 18. Aug, 13:05 h., Ahmed Shoeib ahmedelsayed.sho...@gmail.com wrote: hi all , i want a simple application explain how to integrate facebook api with my app engine application best regards, ahmed shoeib java developer -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Please help, I'm not able to obtain the oAuth redirect URL from twitter
Sorry to hear of your trouble Nischal I'm hitting twitter without any issue from my prod app engine - if it helps, i've had a lot of success using twitter4j libraries - maybe you can check them out. The only strange thing about getting an auth token from twitter oauth from app engine is how you have to direct the user to the authorization url and hold the token in your session on the server side while they authorize the app - then continue with the process of getting an oauth token on the server. On Jul 22, 10:33 pm, nischalshetty nischalshett...@gmail.com wrote: Was searching twitter, others seem to be facing the same problem. Please treat this on priority, my entire sitehttp://www.justunfollow.com is unusable right now due to the error. A lot of users are stuck due to the error. -Nischal On Jul 23, 7:24 am, nischalshetty nischalshett...@gmail.com wrote: To obtain the redirect URL it's an https call to twitter. I'm just not able to obtain it in production. However, it works in my development environment on my PC. Please help ASAP, my site gets a lot of users and this is causing problems for all of them :( Here's the stack trace : twitter4j.TwitterException: Could not fetch URL:https://twitter.com/oauth/request_token at twitter4j.internal.http.HttpClient.request(HttpClient.java:340) at twitter4j.internal.http.HttpClientWrapper.request(HttpClientWrapper.java: 68) at twitter4j.internal.http.HttpClientWrapper.post(HttpClientWrapper.java: 99) at twitter4j.http.OAuthAuthorization.getOAuthRequestToken(OAuthAuthorization.j ava: 121) at twitter4j.Twitter.getOAuthRequestToken(Twitter.java:1406) at justunfollow.manager.impl.JustUnfollowManagerImpl.redirectURL(JustUnfollowM anagerImpl.java: 60) at justunfollow.action.Login.authenticate(Login.java:56) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at com.google.apphosting.runtime.security.shared.intercept.java.lang.reflect.M ethod_ $1.run(Method_.java:165) at java.security.AccessController.doPrivileged(Native Method) at com.google.apphosting.runtime.security.shared.intercept.java.lang.reflect.M ethod_.privilegedInvoke(Method_.java: 163) at com.google.apphosting.runtime.security.shared.intercept.java.lang.reflect.M ethod_.invoke_(Method_.java: 124) at com.google.apphosting.runtime.security.shared.intercept.java.lang.reflect.M ethod_.invoke(Method_.java: 43) at com.opensymphony.xwork2.DefaultActionInvocation.invokeAction(DefaultActionI nvocation.java: 404) at com.opensymphony.xwork2.DefaultActionInvocation.invokeActionOnly(DefaultAct ionInvocation.java: 267) at justunfollow.interceptor.ExceptionInterceptor.intercept(ExceptionIntercepto r.java: 23) at com.opensymphony.xwork2.DefaultActionInvocation $2.doProfiling(DefaultActionInvocation.java:224) at com.opensymphony.xwork2.DefaultActionInvocation $2.doProfiling(DefaultActionInvocation.java:223) at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStac k.java: 455) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocat ion.java: 221) at com.opensymphony.xwork2.interceptor.DefaultWorkflowInterceptor.doIntercept( DefaultWorkflowInterceptor.java: 221) at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(Metho dFilterInterceptor.java: 86) at com.opensymphony.xwork2.DefaultActionInvocation $2.doProfiling(DefaultActionInvocation.java:224) at com.opensymphony.xwork2.DefaultActionInvocation $2.doProfiling(DefaultActionInvocation.java:223) at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStac k.java: 455) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocat ion.java: 221) at com.opensymphony.xwork2.validator.ValidationInterceptor.doIntercept(Validat ionInterceptor.java: 150) at org.apache.struts2.interceptor.validation.AnnotationValidationInterceptor.d oIntercept(AnnotationValidationInterceptor.java: 48) at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(Metho dFilterInterceptor.java: 86) at com.opensymphony.xwork2.DefaultActionInvocation $2.doProfiling(DefaultActionInvocation.java:224) at com.opensymphony.xwork2.DefaultActionInvocation $2.doProfiling(DefaultActionInvocation.java:223) at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStac k.java:
[appengine-java] Re: Google Plugin for Eclipse 3.6 is now available
Just to Add to Arian's post. I has all sorts of trouble installing the new plugin for eclipse Helios. I did it through the update site and the marketplace says it's installed, but i don't have any of the google options i'm used to. If this helps - i tried this with Eclipse Helios EE x64 and x86, and Eclipse Helios for Java x64 and x86. And tried on two different workstations. Both we're running windows 7 64 bit. Anyone else? On Jun 25, 11:21 am, Arian aerobo...@gmail.com wrote: On Jun 23, 11:14 pm, Jason Parekh jasonpar...@gmail.com wrote: Hey folks, Google Plugin for Eclipse 1.3.3 is out with support for Eclipse 3.6. Install it with Eclipse 3.6's new Eclipse Marketplace feature by going to Help Eclipse Marketplace, and search for Google Plugin for Eclipse. Alternatively, here are the update sites: - Eclipse Helios (3.6):http://dl.google.com/eclipse/plugin/3.6 - Eclipse Galileo (3.5):http://dl.google.com/eclipse/plugin/3.5 - Eclipse Ganymede (3.4):http://dl.google.com/eclipse/plugin/3.4 - Eclipse Europa (3.3):http://dl.google.com/eclipse/plugin/3.3 Need detailed instructions? Check out the quick start guide:http://code.google.com/eclipse/docs/getting_started.html Enjoy! I installed the plugin on Helios, but I don't see the project templates, nor the deploy button, nor any button or function related to Google web toolkit and appengine.. it seems like it fails to load the plugin or something :( -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Limiting the number of requests per IP each minute
It's funny, i can't think of a better way to stop the abuse but i can think of a 100 ways to abuse it. I hate getting stuck in one of those spy vs spy - measure vs counter measure situations so i'm sympathetic to the challenge. Maybe you could randomly redirect a user to one of those prove your human by entering these garbled looking words that only a human can read tests. Like, one in every 100 clicks or so a user gets a random test they have to prove they are human before continuing on. Anyway, that's my 12:30 am $0.02. good luck. - Ben On Jun 28, 7:17 am, mscwd01 mscw...@gmail.com wrote: I assume my idea was the best anyone can come up with? On Jun 27, 10:27 am, mscwd01 mscw...@gmail.com wrote: Hey, I am developing an app which awards users for visiting a specific link. However, I want to ensure this is not abused by people writing scripts to visit the link rather than manually viewing it. Is there a way to check how many times a specific IP address has accessed your application within the last minute and deny access if the visit count is higher than, say, 20. I guess I could use the memcache to insert the users IP (if they have not visited before), then on each subsequent visit check if the IP exists and if so increment the visit count. However, will this work if you have thousands of concurrent users? Is there a better way? Many thanks -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
