Yes, in some cases I might fetch about three entities by their keys
from the datastore. But I doubt this is the problem because even for a
small entity like Label in my model I see the same high CPU on
inserts. And, all get requests in my app are fast and have no
problems.
Here is the definition of this simpler entity:
@PersistenceCapable(identityType = IdentityType.APPLICATION)
public class Label {
@PrimaryKey
@Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
private Key key;
@Persistent
private String label;
@Persistent
private Account account;
@Persistent
@Extension(vendorName="datanucleus", key="gae.parent-pk",
value="true")
private Key accountKey;
@Persistent
private int contactsCounter;
}
And it has one custom index:
label ▲
Includes ancestors
I know this custom index it's on one property and the query should
have used the build in index but, the index definition was auto
generated by app engine, maybe it's because it includes an ancestor.
And still, even for this Label entity, if I insert a Label entity in a
large group I see the same 40 second CPU usage...
On Jan 20, 12:20 am, Robert Kluin <[email protected]> wrote:
> Lucian,
> The only other thing I am wondering about is how much time it might
> be taking to fetch and set those reference (key) properties. Are you
> fetching several entities to populate those fields? I know that
> should not be related to the size of your entity groups, but it might
> be something else to look into.
>
> Robert
>
>
>
> On Tue, Jan 19, 2010 at 4:57 PM, Lucian Baciu <[email protected]> wrote:
> > First of all, thank you both for helping me.
>
> > I don't think I have an exploding index problem. Account entity has no
> > custom indexes defined, and as far as I understand from the docs an
> > exploding index means two or more list properties in the same custom
> > index. No other entity in my model has two or more list properties in
> > a custom index. If a custom index is not defined for those list
> > properties in the same entity, the build in indexes do not have a
> > exploding index problem either.
>
> > My put operations look like, for example:
> > Task dataTask = new Task(....);
> > dataTask = pm.makePersistent(dataTask);
>
> > The keys for all my entities are generated by the App Engine.
> > Properties like attachedContact or responsibleContact are set with
> > existing key values on the new Task entity being created.
>
> > On Jan 19, 11:21 pm, Eli Jones <[email protected]> wrote:
> >> Also, as Robert mentioned.. if you have any indexes defined on those List
> >> properties for the Account Model, that could lead to an exploding index..
>
> >> I'd say you have an exploding index somewhere.. I'm just not clear on where
> >> since I don't know your program setup... and Java on App Engine isn't my
> >> main squeeze.
>
> >> On Tue, Jan 19, 2010 at 3:18 PM, Lucian Baciu <[email protected]>
> >> wrote:
> >> > I have a root entity Account:
>
> >> > @PersistenceCapable(identityType = IdentityType.APPLICATION)
> >> > public class Account implements Serializable{
> >> > �...@primarykey
> >> > �...@persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
> >> > private Long id;
>
> >> > �...@persistent
> >> > private String title;
>
> >> > �...@persistent
> >> > private Date createdDate;
>
> >> > �...@persistent(mappedBy = "account")
> >> > �...@order(extensions = @Extension(vendorName="datanucleus",
> >> > key="list-
> >> > ordering", value="key asc"))
> >> > private List<Contact> contacts;
>
> >> > �...@persistent(mappedBy = "account")
> >> > �...@order(extensions = @Extension(vendorName="datanucleus",
> >> > key="list-
> >> > ordering", value="key asc"))
> >> > private List<Label> labels;
>
> >> > �...@persistent(mappedBy = "account")
> >> > �...@order(extensions = @Extension(vendorName="datanucleus",
> >> > key="list-
> >> > ordering", value="key asc"))
> >> > private List<Task> tasks;
>
> >> > �...@persistent(mappedBy = "account")
> >> > �...@order(extensions = @Extension(vendorName="datanucleus",
> >> > key="list-
> >> > ordering", value="key asc"))
> >> > private List<Note> notes;
>
> >> > //other owned child collections
> >> > }
>
> >> > Task entity:
>
> >> > @PersistenceCapable(identityType = IdentityType.APPLICATION)
> >> > public class Task {
> >> > �...@primarykey
> >> > �...@persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
> >> > private Key key;
>
> >> > �...@persistent
> >> > private Text task;
>
> >> > �...@persistent
> >> > private Date dueDate;
>
> >> > �...@persistent
> >> > private Key createdByContact;
>
> >> > �...@persistent
> >> > private Key responsibleContact;
>
> >> > �...@persistent
> >> > private Key attachedContact;
>
> >> > �...@persistent
> >> > private Key attachedProject;
>
> >> > �...@persistent
> >> > private boolean completed;
>
> >> > �...@persistent
> >> > private Key completedByContact;
>
> >> > �...@persistent
> >> > private Date completedDate;
>
> >> > �...@persistent
> >> > private Key category;
>
> >> > �...@persistent
> >> > private Date createdDate;
>
> >> > �...@persistent
> >> > private Account account;
>
> >> > �...@persistent
> >> > �...@extension(vendorName="datanucleus", key="gae.parent-pk",
> >> > value="true")
> >> > private Key accountKey;
> >> > }
>
> >> > Task entity indexes (this entity has the most indexes):
>
> >> > attachedContact ▲ , completed ▲ , completedDate ▼
> >> > Includes ancestors
>
> >> > attachedContact ▲ , completed ▲ , dueDate ▲
> >> > Includes ancestors
>
> >> > attachedProject ▲ , completed ▲ , completedDate ▼
> >> > Includes ancestors
>
> >> > attachedProject ▲ , completed ▲ , dueDate ▲
> >> > Includes ancestors
>
> >> > completed ▲ , completedDate ▼
> >> > Includes ancestors
>
> >> > completed ▲ , dueDate ▲
> >> > Includes ancestors
>
> >> > completed ▲ , responsibleContact ▲ , completedDate ▼
> >> > Includes ancestors
>
> >> > completed ▲ , responsibleContact ▲ , dueDate ▲
> >> > Includes ancestors
>
> >> > On Jan 19, 10:06 pm, Eli Jones <[email protected]> wrote:
> >> > > What do you mean by "one custom index on the string field"?
>
> >> > > If you have a custom index.. it must be on more than one "property" of
> >> > the
> >> > > model.
>
> >> > > How are the parent and child models defined and what are the
> >> > > definitions
> >> > of
> >> > > the relevant custom indexes? That should help a lot with figuring this
> >> > out.
>
> >> > > On Tue, Jan 19, 2010 at 3:01 PM, Lucian Baciu <[email protected]>
> >> > wrote:
> >> > > > There is no batch, I'm only inserting one entity.
>
> >> > > > For a small group (10 entities) this is the log for an insert of one
> >> > > > entity: 883ms 1290cpu_ms 960api_cpu_ms
> >> > > > For a large group (4000 entities) this is the log for an insert of
> >> > > > one
> >> > > > entity: 8724ms 44123cpu_ms 38854api_cpu_ms
>
> >> > > > And it does not seem to matter which kind of child entity I'm
> >> > > > inserting, the logs are consistent. Even if I'm inserting a small
> >> > > > entity with one string and one integer field, and one custom index on
> >> > > > the string field I'm still getting the same results.
>
> >> > > > Thanks,
> >> > > > Lucian
>
> >> > > > On Jan 19, 9:52 pm, Robert Kluin <[email protected]> wrote:
> >> > > > > Lucian,
>
> >> > > > > How many entities are you batch putting at once? For those puts
> >> > > > > taking 40 _seconds_, are you putting quite a few entities at a
> >> > > > > time?
>
> >> > > > > How long does it take to put one entity not in a large group?
> >> > > > > How
> >> > > > > long does it take to put one entity to a large group?
>
> >> > > > > Robert
>
> >> > > > > On Tue, Jan 19, 2010 at 1:24 PM, Lucian Baciu
> >> > > > > <[email protected]
>
> >> > > > wrote:
> >> > > > > > I have a root entity and several child entities mapped using
> >> > > > > > owned
> >> > one
> >> > > > > > to many relationships. When a new entity group is created put
> >> > > > > > operations in that entity group use reasonable amounts of CPU (1
> >> > > > > > second), but as the number of entities in that entity group
> >> > increase
> >> > > > > > so does the CPU used for put operations. Now I'm seeing over 40
> >> > > > > > seconds CPU used for put operations on entity groups with over
> >> > > > > > one
> >> > > > > > thousand entities. And it continues to increase.
>
> >> > > > > > Child entities have few custom indexes (about 5), but this
> >> > > > > > doesn't
> >> > > > > > seem to cause the problem, because it takes about the same amount
> >> > of
> >> > > > > > CPU (over 40 seconds) for a put operation on any entity kind,
> >> > > > > > even
> >> > for
> >> > > > > > a first put for a specific entity kind.
>
> >> > > > > > I've read all App Engine articles and watched all videos but I
> >> > still
> >> > > > > > can't figure it out. I would really appreciate some help.
>
> >> > > > > > Thanks,
> >> > > > > > Lucian
>
> >> > > > > > --
> >> > > > > > You received this message because you are subscribed to the
> >> > > > > > Google
> >> > > > Groups "Google App Engine" group.
> >> > > > > > To post to this group, send email to
> >> > [email protected]
> >> > > > .
> >> > > > > > To unsubscribe from this group, send email to
> >> > > > [email protected]<google-appengine%2Bunsubscrib
> >> > > > [email protected]><google-appengine%2Bunsubscrib
> >> > [email protected]>
> >> > > > .
> >> > > > > > For more options, visit this group athttp://
> >> > > > groups.google.com/group/google-appengine?hl=en.
>
> >> > > > --
> >> > > > You received this message because you are subscribed to the Google
> >> > Groups
> >> > > > "Google App Engine" group.
> >> > > > To post to this group, send email to
> >> > > > [email protected]
> >> > .
> >> > > > To unsubscribe from this group, send email to
> >> > > > [email protected]<google-appengine%2Bunsubscrib
> >> > > > [email protected]><google-appengine%2Bunsubscrib
> >> > [email protected]>
> >> > > > .
> >> > > > For more options, visit this group at
> >> > > >http://groups.google.com/group/google-appengine?hl=en.
>
> >> > --
> >> > You received this message because you are subscribed to the Google Groups
> >> > "Google App Engine" group.
> >> > To post to this group, send email to [email protected].
> >> > To unsubscribe from this group, send email to
> >> > [email protected]<google-appengine%2Bunsubscrib
> >> > [email protected]>
> >> > .
> >> > For more options, visit this group at
> >> >http://groups.google.com/group/google-appengine?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to [email protected].
> > To unsubscribe from this group, send email to
> > [email protected].
> > For more options, visit this group
> > athttp://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
