hey guys,
We are not experiencing this problem and we execute like 15k+ task per
minute probably.
One tip tho, might worth to mention:
The ones staying in the queue have 0 retry attempts? Retries uses a push
back mechanism so if they fail they are kept on the queue if your
configuration is not s
tastore but kept my Master/Slave test
>>> running. Past couple of days Master/Slave has exhausted Task Queue of
>>> 100,000 but HR is working fine. Typical Task Queue would probably be 5000
>>> for the day. Already Master/Slave has exhausted quota. For those having tas
you can shard by the biggest geocell and cache based on the smallest
geocell.
at then end GeoModel creates a hash based on geoboxes that you can easily
play with
On 14 May 2011 18:10, Devraj Mukherjee wrote:
> Hi all,
>
> Our project http://safesharps.org.au uses GeoModel
> (http://code.google.
i'm also using it right now for some moderation flows i need to finish so i
will test it too
On 9 June 2011 10:14, Jason Collins wrote:
> @Mike_W, I've updated the docs to describe this new feature.
>
> http://code.google.com/p/fantasm/wiki/AdvancedConcepts#Fan-In
>
> j
>
> On Jun 8, 9:44 pm, R
BTW - any intentions on plugging an observer pattern/plugin around it :) ?
It would be awesome to have a list of observers watching for change on
states (subject)
On 9 June 2011 12:22, David Mora wrote:
> i'm also using it right now for some moderation flows i need to finish so i
> w
Depending on your entity group/transaction strategy here you might want to
consider Referencing Users to a Dislike/Like Kind:
http://code.google.com/appengine/docs/python/datastore/datamodeling.html#References
What happens is (and premature i can say so) your like, dislikes properties
are going t
Hi King,
I'm learning this thing here too so don't worry :)
On 9 June 2011 19:54, King wrote:
> Hi David,
> I'm having some trouble understanding your reply
>
> class Like(db.Model):
>> topic = db.StringProperty() #enumeration
>> user = db.ReferenceProperty(User)
>>
>
> By enumeration,
;
> and in fsm.yaml
>
> context_types:
> observers: module.name.of.Observer
>
> This would only act at the FSMAction level, not the state transition
> level. Subclasses of FSMContext/FSMState which override .dispatch()
> can also be very powerful - but these required bigger chan
So zend + right scale are playing with a PaaS already, and there's another
that i forgot the name (or at least trying to).
I worked 5 years on massive PHP systems (+10 mill users, with oracle behind)
as SA and from that i gather many things:
- Hardware consumed tend to be 1.5 times more than o
og: http://googleappengine.blogspot.com
> Twitter: http://twitter.com/app_engine
> Reddit: http://www.reddit.com/r/appengine
>
>
>
> On Sat, Jun 18, 2011 at 4:03 AM, David Mora wrote:
>
>> So zend + right scale are playing with a PaaS already, and there's another
&g
Resource-wise is really expensive to do single-threaded solutions, thats the
major problem.
It is always possible, i don't see any reasons why you would not be able to
run a full PHP environment on GAE since is not that far from what python
requires - at the end you pick GAE as your platform for w
meanwhile and in urge to deploy new versions, the -no_precompilation
is the way to get around this annoying error
On 8 December 2010 16:50, Jason Spitkoski wrote:
> Seems hit and miss. But when it fails, its always during precompilation...
>
> On Tue, Dec 7, 2010 at 4:28 PM, homunq wrote:
>>
>>
I think the problem is at the underlaying appengine.db implementation
(which acts as an ORM)
that property belongs to an entity, you will have to have away to know
when that model is not fully loaded.
Common ORM/Data Gateways problems.
(PS: but yeah, performance wise it would be great)
On 17 De
result as a dict.
>
>
>
>
>
>
>
>
> On Fri, Dec 17, 2010 at 22:41, David Mora wrote:
>> I think the problem is at the underlaying appengine.db implementation
>> (which acts as an ORM)
>>
>> that property belongs to an entity, you will have to have aw
Quoting Ikai
"Just be aware of monotonically increasing indexes like timestamps: if
you have an application with a high write rate that has a timestamp,
this will cause the persistence unit storing the indexes to be unable
to be autosplit easily across multiple hardware instances and you will
take
a
> UUID) or based on something that isn't monotonic, such as email address.
> What level of writes are you experiencing the issue at, and how did you
> determine that this was the cause of your issue?
> -Nick Johnson
>
> On Wed, Jan 19, 2011 at 5:30 AM, David Mora wrote:
>
Hi,
We are in the process or upgrading to SDK 1.4.1, one of the features
we were looking for was the django initialization from the
app_config.py, per SDK log:
"Allow Django initialization to be moved to appengine_config.py to
avoid Django version conflicts when mixing webapp.template with pure
D
Seems like there is a service disruption:
http://api-status.com/6404/117406/Google-App-Engine-API
(ours as well, is timing out even doing code deploy)
On 24 January 2011 16:49, Kangesh Gunaseelan wrote:
> It is just not the dashboard...I am seeing sporadic data store timeouts..
> On Mon, Jan 24
Hi,
So i'm playing around with the map reduce and just faced this problem (more
of a unknown actually)
The one i ran takes about 40GB worth of entities and deletes them (i'm not
using the datastore admin since i actually need to speed the process by
sharding more and increasing the process rat
exciting !!!
thanks !!
(that was a good reminder to look into the matcher API, hah)
On 3 February 2011 13:31, Ikai Lan (Google) wrote:
> Hey everyone, I just wanted to post that the prerelease SDKs for 1.4.2 are
> now available for early testing:
> http://code.google.com/p/googleappengine/downl
have you ran a profiler directly in your GAE instance?
There are many factors that alter your result from a local server
On 4 February 2011 12:59, dudu wrote:
> Note that just 1174 URL Fetch calls uses 8% of my CPU.
>
> --
> You received this message because you are subscribed to the Google Grou
Hi,
So since today there's a small time window for Appengine maintenance, i was
wondering if for those that have background processing happening at that
time (to be specific, mapreduce operations) should abort those processes?
For example, right now we have one running that was triggered from t
same here
(already reported thru appengine billing tho)
It seemed to happend just after the maintenance
On 8 February 2011 10:30, David wrote:
> Somehow this morning my app has reached 100% quota on Task Queue
> Stored Task Bytes and I have no tasks queues. I has never even went
> above a smal
seems to be behaving correctly now
The mapreduce that filled up all our quotas yesterday is currently
running (hour and a half) and is using the right amount of "stored
task bytes"
thanks btw !
On 8 February 2011 14:03, Wesley C (Google) wrote:
> it was something behind the scenes, and you have
i do notice a bit more of latency, which translates to about +1% of CPU.
I guess that in order to accurately report this you should have
metrics for before and after the maintenance.
But looking quickly thru the API status:
http://api-status.com/6404/117406/Google-App-Engine-API
it seems like th
There seems to be a problem (in python) when using the bulk loader:
UnacceptableVersionError: django 1.1 was requested, but 1.2.3.final.0
is already in use
[INFO] An error occurred. Shutting down...
We load django 1.1 directly in app_config.py so it is used across all
mapreduces/pipelines
Wo
seems like there's something going on with the datastore:
http://code.google.com/status/appengine/detail/datastore/2011/02/13#ae-trust-detail-datastore-delete-latency
http://code.google.com/status/appengine/detail/datastore/2011/02/13#ae-trust-detail-datastore-update-latency
http://code.google.com
Now there seems to be a problem with the serving clusters:
"Request was aborted after waiting too long to attempt to service your
request. This may happen sporadically when the App Engine serving
cluster is under unexpectedly high or uneven load. If you see this
message frequently, please contact
Taking a decision of leaving is actually a hard call. There are many
things involved ( + Business decisions)
I guess everything falls at what you expect from your service to end
users: 99.99% uptime? That's _expensive_ everywhere (or eats everyones
life at the sysadmin level resetting load balance
well, i also agree to that.
I mean, my position is really clear. Im with you about the performance
just after the maintenance period (like right now im starting to reply
to all the complains for the last 4 hours today) - has been the worst
in the last 4 months.
was just being clear with a wider o
just out of curiosity:
is it me or i see some implementations of GSA on the Matcher Service? :)
On 12 February 2011 11:16, Bjorn wrote:
> I have the same problem with spreadsheets, though not with feeds such
> as "docs.google.com/feeds/default/private/full" it seems. Only
> occuring when I tried
is this happening also for HR users?
On 21 February 2011 21:51, Dennis wrote:
> I'm still experiencing timeout issues and have been for over 12 hours.
> Simple pages in my application work, but the pages that do 30-40 gae read
> operations are consistently timing out (they were never a problem be
Yeah, we are also looking forward for HR but we need a major redesign of
some components to lower the CPU usage (aim to pay less actually)
I have one HR instance already with a small server that does random inserts
- think im going to leave jmeter running for 2 days and compare the graphs
On 21 F
is a mapreduce (just like the delete)
you can access it thru: http://{version}{host}/_ah/mapreduce
i think it has a callback that cleans up the job after it is done/aborted tho.
On 21 February 2011 23:08, Tom Wu wrote:
> +1
>
> 2011/2/11 Eric Ka Ka Ng
>>
>> The new function to copy data from a
should no longer exist and it's just some task
> entries being left in the datastore. know any ways i could clean up those
> entries too?
>
> - eric
>
> On 22 February 2011 13:14, David Mora wrote:
>>
>> is a mapreduce (just like the delete)
>>
>> you
Just checking
have you checked if the task workers are not returning errors.
It also seems like you are delaying the feeding a lot (100/m means
that 100 items are going to be distributed to 50 buckets every minute
- maybe thats your bottleneck?)
here is a graph that ikai did that helped me a lot
So i left 2 monitors running doing random inserts and 5 others doing location
queries (geobox). After 6 hours has been an steady line with only 1 error.
Im going to keep it running 2 more days and post the results ( im also running
out of cpu since is a free acount)
Sent from my iPhone
On Feb
Hey,
just out of curiosity ... anyone know what could be a good comparison of
virtual instances agains entry level servers in a colo? Comparison as in
hardware. e.g: 20 virtual instances = 1 (8 nucleus, 16 GB mem server)
Thanks !
--
You received this message because you are subscribed to the
My HR instances are fine, but those in M/R are completely down
On 25 February 2011 10:57, Robert Kluin wrote:
> I totally agree with you. It is very difficult to design around the
> large variations in performance; more consistency would make me happy
> as well.
>
> I try to leave lot of room fo
why would it be cheaper if at the end, the datastore admin creates a map
reduce that iterates thru the model via splitting the index and loading each
entity per key name ? Each map is a task queue so it is exactly the same :)
On 9 March 2011 18:22, Bemmu wrote:
> Crossposting my reply from stack
hmmm, well Wesley's option B was merely because the batch operations i
think. One nice feature about the map reduce is the mutation pool which
handles the logic of batching an operation while you yield thru iterations.
I guess in a big dataset like yours make sense (the model retrieval vs key
only)
There are a couple of things you can still do:
- Try always to send high expensive operations to the background (task
queues). And also an expensive operation does not mean a lot of processing
time in your side. If you notice your API CPU usage is high, this could be a
delete/put opera
all those sites/apps looks great! quite amazing.
I see GAE as an emerging platform, the fact google is exposing via API or
modified solutions of their own infrastructure have a lot to think about. The
occasional hiccups are because of the same emerging platform that has to
deal with all our/busine
I tend to agree with the previous comment by Stephen.
The fact that custom clients (read non standard browsers apps built on:
iphone, androids, etc) are forced to close sockets to app engine servers
makes it more obvious to apply a strategy pattern over the interfaces
considering mostly REST. Unle
44 matches
Mail list logo
