Hi Claudio,
You have 12 total fields all of which you have indexed which is the default.
You have 1 key field and 11 non-key fields. This will result in 23 index
entries being written for each entity. You get 1 ascending index on the key
field and by default you get both an ascending and descending index on the
other 11 non-key fields. This will take A LOT of CPU for each entity. Do you
need them all indexed? If not, I would set them to unindexed. That should
help a lot.
Stephen
2011/3/7 Cláudio Coelho <ereb...@gmail.com>
> Since the bottleneck occurs when making objects of Location
> persistent, I assume that is the only relevant class. If anything else
> is relevant, tell me and I'll disclose.
>
> @PersistenceCapable(identityType =
> IdentityType.APPLICATION,detachable="true")
> public class Location{
> @PrimaryKey
> @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
> Long locationId;
> @Persistent
> long countryId;
> @Persistent
> String name;
> @Persistent
> String country=null;
> @Persistent
> double latitude,longitude;
> @Persistent
> int population;
> @Persistent
> long geoBoxL1,geoBoxL2,geoBoxL3,geoBoxL4,geoBoxL5;
> public Location()
> {
>
> }
> public Location(String name, long countryId,String country, double
> latitude, double longitude)
> {
> this.name=name;
> this.countryId=countryId;
> this.latitude=latitude;
> this.longitude=longitude;
> this.country=country;
> }
> public Location(LocationDTO dto) {
> this();
> updateFromDTO(dto);
> }
>
> public void updateFromDTO(LocationDTO dto) {
> setLocationId(dto.getLocationId());
> setCountryId(dto.getCountryId());
> setCountry(dto.getCountry());
> setName(dto.getName());
> setPopulation(dto.getPopulation());
> setLatitude(dto.getLatitude());
> setLongitude(dto.getLongitude());
> setGeoBoxL1(dto.getGeoBoxL1());
> setGeoBoxL2(dto.getGeoBoxL2());
> setGeoBoxL3(dto.getGeoBoxL3());
> setGeoBoxL4(dto.getGeoBoxL4());
> setGeoBoxL5(dto.getGeoBoxL5());
>
>
> }
> public LocationDTO getDTO()
> {
> LocationDTO dto = new LocationDTO();
> dto.setLocationId(getLocationId());
> dto.setCountryId(getCountryId());
> dto.setCountry(getCountry());
> dto.setName(getName());
> dto.setPopulation(getPopulation());
> dto.setLatitude(getLatitude());
> dto.setLongitude(getLongitude());
> dto.setGeoBoxL1(getGeoBoxL1());
> dto.setGeoBoxL2(getGeoBoxL2());
> dto.setGeoBoxL3(getGeoBoxL3());
> dto.setGeoBoxL4(getGeoBoxL4());
> dto.setGeoBoxL5(getGeoBoxL5());
>
> return dto;
> }
>
> public long getLocationId() {
> return locationId;
> }
> public void setLocationId(long locationId) {
> this.locationId = locationId;
> }
> public String getCountry() {
> return country;
> }
>
> public int getPopulation() {
> return population;
> }
> public void setPopulation(int population) {
> this.population = population;
> }
> public void setCountry(String country) {
> this.country = country;
> }
>
> public long getCountryId() {
> return countryId;
> }
> public void setCountryId(long countryId) {
> this.countryId = countryId;
> }
> public String getName() {
> return name;
> }
> public void setName(String name) {
> this.name = name;
> }
> public double getLatitude() {
> return latitude;
> }
> public void setLatitude(double latitude) {
> this.latitude = latitude;
> }
> public double getLongitude() {
> return longitude;
> }
> public void setLongitude(double longitude) {
> this.longitude = longitude;
> }
>
> public long getGeoBoxL1() {
> return geoBoxL1;
> }
> public void setGeoBoxL1(long geoBoxL1) {
> this.geoBoxL1 = geoBoxL1;
> }
> public long getGeoBoxL2() {
> return geoBoxL2;
> }
> public void setGeoBoxL2(long geoBoxL2) {
> this.geoBoxL2 = geoBoxL2;
> }
> public long getGeoBoxL3() {
> return geoBoxL3;
> }
> public void setGeoBoxL3(long geoBoxL3) {
> this.geoBoxL3 = geoBoxL3;
> }
> public long getGeoBoxL4() {
> return geoBoxL4;
> }
> public void setGeoBoxL4(long geoBoxL4) {
> this.geoBoxL4 = geoBoxL4;
> }
> public long getGeoBoxL5() {
> return geoBoxL5;
> }
> public void setGeoBoxL5(long geoBoxL5) {
> this.geoBoxL5 = geoBoxL5;
> }
> }
>
> On Mar 7, 10:23 am, Didier Durand <durand.did...@gmail.com> wrote:
> > Hi,
> >
> > Could you please disclose the source code of your objects stored in
> > the ds ?
> >
> > You maybe right or wrong depending on how they were defined.
> >
> > regards
> >
> > didier
> >
> > On Mar 7, 9:41 am, Cláudio Coelho <ereb...@gmail.com> wrote:
> >
> >
> >
> >
> >
> >
> >
> > > Hi
> > > I'm quite new at GAE, so I'm probably saying something silly, but for
> what I
> > > read on GAE Java and GWT Application Development, I was under the
> impression
> > > that GAE would generate indexes automatically except for some variable
> types
> > > (blobs, text, etc) and for when we explicitly asked it not to.
> > > However, when I look at the datastore-indexes-auto.xml file, it is
> indeed
> > > empty. Through debugging I have been able to isolate that it's the
> > > bottleneck is pm.makeConsistent of each entity to insert. Wouldn't
> having
> > > indexes make the inserting even slower (since these indexes have to be
> > > updated/rebuilt)?
> >
> > > thanks
> >
> > > C.
> >
> > > On Mon, Mar 7, 2011 at 5:44 AM, Didier Durand <durand.did...@gmail.com
> >wrote:
> >
> > > > Hi,
> >
> > > > The first issue that comes to mind in missing indexes: so huge scans
> > > > of all existing data when you upload an additional line.
> >
> > > > What are your indexes ?
> >
> > > > regards
> >
> > > > didier
> >
> > > > On Mar 6, 12:43 pm, Cláudio Coelho <ereb...@gmail.com> wrote:
> > > > > Hi,
> > > > > My application needs a db of the world cities. I have a file that
> has
> > > > about
> > > > > 100k lines and I want to use it to populate the datastore.
> > > > > My first attempt failed because I didn't use tasks, so I was only
> able to
> > > > > upload a limited set of cities. It spent 2% of CPU total time on
> the app
> > > > > engine.
> > > > > I then changed the approach to use tasks.
> > > > > I basically read the file once to determine how many lines does it
> have
> > > > and
> > > > > then invoke tasks to read batches of 15k lines (supplying a start
> and end
> > > > > indexes).
> > > > > These invoked tasks (createCities(int start, int end)) read the
> file
> > > > again,
> > > > > get a list of 15k lines and then process it.
> > > > > The whole process takes about 15 seconds on my machine, however,
> when I
> > > > do
> > > > > this in the app engine, it takes 15 minutes (or more) and maxes out
> the
> > > > 6.5
> > > > > hours of CPU time! I know there are plenty of optimizations I
> should do
> > > > to
> > > > > this process, but that doesn't seem to justify the 6.5 hours, so I
> must
> > > > be
> > > > > doing something seriously wrong.
> >
> > > > > Does anybody have any clue of what I'm doing wrong? (Code attached)
> >
> > > > > Thanks!
> >
> > > > > private void createCities() {
> > > > > try
> > > > > {
> > > > > int count = countCitiesToLoad();
> > > > > final int batchSize = 15000;
> > > > > for(int start=0;start<count;start+=batchSize)
> > > > > {
> > > > > int end=start+batchSize;
> > > > > Queue queue = QueueFactory.getQueue("dbQueue");
> > > > > TaskOptions topt = TaskOptions.Builder.withUrl("/dbservlet");
> > > > > topt.param("command", "createCities");
> > > > > topt.param("start", ""+start);
> > > > > topt.param("end", ""+end);
> > > > > queue.add(topt);
> > > > > logInfo("Dispatched order to create cities "+start+" to "+end);
> > > > > Thread.sleep(500);}
> > > > > }catch(Exception e)
> >
> > > > > {
> > > > > e.printStackTrace();
> > > > > logError(e.getLocalizedMessage());
> >
> > > > > }
> > > > > }
> >
> > > > > private void createCities(int start, int end)
> > > > > {
> > > > > try
> > > > > {
> > > > > logInfo("Reading cities "+start+" to "+end);
> > > > > BufferedReader br= new BufferedReader(new
> FileReader("data/cities.csv"));
> > > > > String line=br.readLine();
> > > > > int counter=0;
> > > > > PersistenceManager pm = PMF.get().getPersistenceManager();
> > > > > ArrayList<String> lines = new ArrayList<String>();
> > > > > while(line!=null || counter<end)
> > > > > {
> > > > > if(counter>=start && counter <end)
> > > > > lines.add(line);
> > > > > counter++;
> > > > > line=br.readLine();}
> >
> > > > > br.close();
> > > > > logInfo("Adding cities "+start+" to "+end);
> > > > > createCities(lines);
> > > > > logInfo("Done cities "+start+" to "+end);}
> >
> > > > > catch(Exception e)
> > > > > {
> > > > > e.printStackTrace();
> > > > > logError(e.getLocalizedMessage());
> >
> > > > > }
> > > > > }
> >
> > > > > private void createCities(ArrayList<String> lines)
> > > > > {
> > > > > if(lines==null)
> > > > > return;
> > > > > PersistenceManager pm = PMF.get().getPersistenceManager();
> > > > > HashMap<String, Country> countryMap = loadCountryMap();
> > > > > try{
> > > > > for(String line : lines)
> > > > > if(line!=null)
> > > > > {
> > > > > String fields[]=line.split(",");
> > > > > if(fields.length<7)
> > > > > logError("length error in line:"+line);
> > > > > else
> > > > > {
> > > > > Location loc = new Location();
> > > > > loc.setName(fields[2]);
> > > > > loc.setLatitude(Double.parseDouble(fields[3]));
> > > > > loc.setLongitude(Double.parseDouble(fields[4]));
> > > > > loc.setPopulation(Integer.parseInt(fields[6]));
> > > > > {
> > > > > Country c = countryMap.get(fields[5]);
> > > > > if(c==null)
> > > > > logError("Failed to get country for:"+line);
> > > > > else
> > > > > {
> > > > > loc.setCountryId(c.getCountryId());
> > > > > loc.setCountry(c.getName());
> > > > > pm.makePersistent(loc);}
> > > > > }
> > > > > }
> > > > > }
> > > > > } catch (Exception e) {
> >
> > > > > e.printStackTrace();
> >
> > > > > } finally {
> > > > > pm.close();
> > > > > }
> > > > > }
> >
> > > > --
> > > > You received this message because you are subscribed to the Google
> Groups
> > > > "Google App Engine for Java" group.
> > > > To post to this group, send email to
> > > > google-appengine-java@googlegroups.com.
> > > > To unsubscribe from this group, send email to
> > > > google-appengine-java+unsubscr...@googlegroups.com.
> > > > For more options, visit this group at
> > > >http://groups.google.com/group/google-appengine-java?hl=en.
> >
> > > --
> > > Cláudio Coelho
> >
> > > "Great spirits have often encountered violent opposition from weak
> minds."
> > > A. Einstein
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine for Java" group.
> To post to this group, send email to
> google-appengine-java@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine-java+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine-java?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
