Hi Claudio Try makePersistentAll()
It will certainly help Cheers CB On Mar 7, 2:57 am, Cláudio Coelho <ereb...@gmail.com> wrote: > Since the bottleneck occurs when making objects of Location > persistent, I assume that is the only relevant class. If anything else > is relevant, tell me and I'll disclose. > > @PersistenceCapable(identityType = > IdentityType.APPLICATION,detachable="true") > public class Location{ > @PrimaryKey > @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY) > Long locationId; > @Persistent > long countryId; > @Persistent > String name; > @Persistent > String country=null; > @Persistent > double latitude,longitude; > @Persistent > int population; > @Persistent > long geoBoxL1,geoBoxL2,geoBoxL3,geoBoxL4,geoBoxL5; > public Location() > { > > } > public Location(String name, long countryId,String country, double > latitude, double longitude) > { > this.name=name; > this.countryId=countryId; > this.latitude=latitude; > this.longitude=longitude; > this.country=country; > } > public Location(LocationDTO dto) { > this(); > updateFromDTO(dto); > } > > public void updateFromDTO(LocationDTO dto) { > setLocationId(dto.getLocationId()); > setCountryId(dto.getCountryId()); > setCountry(dto.getCountry()); > setName(dto.getName()); > setPopulation(dto.getPopulation()); > setLatitude(dto.getLatitude()); > setLongitude(dto.getLongitude()); > setGeoBoxL1(dto.getGeoBoxL1()); > setGeoBoxL2(dto.getGeoBoxL2()); > setGeoBoxL3(dto.getGeoBoxL3()); > setGeoBoxL4(dto.getGeoBoxL4()); > setGeoBoxL5(dto.getGeoBoxL5()); > > } > public LocationDTO getDTO() > { > LocationDTO dto = new LocationDTO(); > dto.setLocationId(getLocationId()); > dto.setCountryId(getCountryId()); > dto.setCountry(getCountry()); > dto.setName(getName()); > dto.setPopulation(getPopulation()); > dto.setLatitude(getLatitude()); > dto.setLongitude(getLongitude()); > dto.setGeoBoxL1(getGeoBoxL1()); > dto.setGeoBoxL2(getGeoBoxL2()); > dto.setGeoBoxL3(getGeoBoxL3()); > dto.setGeoBoxL4(getGeoBoxL4()); > dto.setGeoBoxL5(getGeoBoxL5()); > > return dto; > } > > public long getLocationId() { > return locationId; > } > public void setLocationId(long locationId) { > this.locationId = locationId; > } > public String getCountry() { > return country; > } > > public int getPopulation() { > return population; > } > public void setPopulation(int population) { > this.population = population; > } > public void setCountry(String country) { > this.country = country; > } > > public long getCountryId() { > return countryId; > } > public void setCountryId(long countryId) { > this.countryId = countryId; > } > public String getName() { > return name; > } > public void setName(String name) { > this.name = name; > } > public double getLatitude() { > return latitude; > } > public void setLatitude(double latitude) { > this.latitude = latitude; > } > public double getLongitude() { > return longitude; > } > public void setLongitude(double longitude) { > this.longitude = longitude; > } > > public long getGeoBoxL1() { > return geoBoxL1; > } > public void setGeoBoxL1(long geoBoxL1) { > this.geoBoxL1 = geoBoxL1; > } > public long getGeoBoxL2() { > return geoBoxL2; > } > public void setGeoBoxL2(long geoBoxL2) { > this.geoBoxL2 = geoBoxL2; > } > public long getGeoBoxL3() { > return geoBoxL3; > } > public void setGeoBoxL3(long geoBoxL3) { > this.geoBoxL3 = geoBoxL3; > } > public long getGeoBoxL4() { > return geoBoxL4; > } > public void setGeoBoxL4(long geoBoxL4) { > this.geoBoxL4 = geoBoxL4; > } > public long getGeoBoxL5() { > return geoBoxL5; > } > public void setGeoBoxL5(long geoBoxL5) { > this.geoBoxL5 = geoBoxL5; > } > > } > > On Mar 7, 10:23 am, Didier Durand <durand.did...@gmail.com> wrote:> Hi, > > > Could you please disclose the source code of your objects stored in > > the ds ? > > > You maybe right or wrong depending on how they were defined. > > > regards > > > didier > > > On Mar 7, 9:41 am, Cláudio Coelho <ereb...@gmail.com> wrote: > > > > Hi > > > I'm quite new at GAE, so I'm probably saying something silly, but for > > > what I > > > read on GAE Java and GWT Application Development, I was under the > > > impression > > > that GAE would generate indexes automatically except for some variable > > > types > > > (blobs, text, etc) and for when we explicitly asked it not to. > > > However, when I look at the datastore-indexes-auto.xml file, it is indeed > > > empty. Through debugging I have been able to isolate that it's the > > > bottleneck is pm.makeConsistent of each entity to insert. Wouldn't having > > > indexes make the inserting even slower (since these indexes have to be > > > updated/rebuilt)? > > > > thanks > > > > C. > > > > On Mon, Mar 7, 2011 at 5:44 AM, Didier Durand > > > <durand.did...@gmail.com>wrote: > > > > > Hi, > > > > > The first issue that comes to mind in missing indexes: so huge scans > > > > of all existing data when you upload an additional line. > > > > > What are your indexes ? > > > > > regards > > > > > didier > > > > > On Mar 6, 12:43 pm, Cláudio Coelho <ereb...@gmail.com> wrote: > > > > > Hi, > > > > > My application needs a db of the world cities. I have a file that has > > > > about > > > > > 100k lines and I want to use it to populate the datastore. > > > > > My first attempt failed because I didn't use tasks, so I was only > > > > > able to > > > > > upload a limited set of cities. It spent 2% of CPU total time on the > > > > > app > > > > > engine. > > > > > I then changed the approach to use tasks. > > > > > I basically read the file once to determine how many lines does it > > > > > have > > > > and > > > > > then invoke tasks to read batches of 15k lines (supplying a start and > > > > > end > > > > > indexes). > > > > > These invoked tasks (createCities(int start, int end)) read the file > > > > again, > > > > > get a list of 15k lines and then process it. > > > > > The whole process takes about 15 seconds on my machine, however, when > > > > > I > > > > do > > > > > this in the app engine, it takes 15 minutes (or more) and maxes out > > > > > the > > > > 6.5 > > > > > hours of CPU time! I know there are plenty of optimizations I should > > > > > do > > > > to > > > > > this process, but that doesn't seem to justify the 6.5 hours, so I > > > > > must > > > > be > > > > > doing something seriously wrong. > > > > > > Does anybody have any clue of what I'm doing wrong? (Code attached) > > > > > > Thanks! > > > > > > private void createCities() { > > > > > try > > > > > { > > > > > int count = countCitiesToLoad(); > > > > > final int batchSize = 15000; > > > > > for(int start=0;start<count;start+=batchSize) > > > > > { > > > > > int end=start+batchSize; > > > > > Queue queue = QueueFactory.getQueue("dbQueue"); > > > > > TaskOptions topt = TaskOptions.Builder.withUrl("/dbservlet"); > > > > > topt.param("command", "createCities"); > > > > > topt.param("start", ""+start); > > > > > topt.param("end", ""+end); > > > > > queue.add(topt); > > > > > logInfo("Dispatched order to create cities "+start+" to "+end); > > > > > Thread.sleep(500);} > > > > > }catch(Exception e) > > > > > > { > > > > > e.printStackTrace(); > > > > > logError(e.getLocalizedMessage()); > > > > > > } > > > > > } > > > > > > private void createCities(int start, int end) > > > > > { > > > > > try > > > > > { > > > > > logInfo("Reading cities "+start+" to "+end); > > > > > BufferedReader br= new BufferedReader(new > > > > > FileReader("data/cities.csv")); > > > > > String line=br.readLine(); > > > > > int counter=0; > > > > > PersistenceManager pm = PMF.get().getPersistenceManager(); > > > > > ArrayList<String> lines = new ArrayList<String>(); > > > > > while(line!=null || counter<end) > > > > > { > > > > > if(counter>=start && counter <end) > > > > > lines.add(line); > > > > > counter++; > > > > > line=br.readLine();} > > > > > > br.close(); > > > > > logInfo("Adding cities "+start+" to "+end); > > > > > createCities(lines); > > > > > logInfo("Done cities "+start+" to "+end);} > > > > > > catch(Exception e) > > > > > { > > > > > e.printStackTrace(); > > > > > logError(e.getLocalizedMessage()); > > > > > > } > > > > > } > > > > > > private void createCities(ArrayList<String> lines) > > > > > { > > > > > if(lines==null) > > > > > return; > > > > > PersistenceManager pm = PMF.get().getPersistenceManager(); > > > > > HashMap<String, Country> countryMap = loadCountryMap(); > > > > > try{ > > > > > for(String line : lines) > > > > > if(line!=null) > > > > > { > > > > > String fields[]=line.split(","); > > > > > if(fields.length<7) > > > > > logError("length error in line:"+line); > > > > > else > > > > > { > > > > > Location loc = new Location(); > > > > > loc.setName(fields[2]); > > > > > loc.setLatitude(Double.parseDouble(fields[3])); > > > > > loc.setLongitude(Double.parseDouble(fields[4])); > > > > > loc.setPopulation(Integer.parseInt(fields[6])); > > > > > { > > > > > Country c = countryMap.get(fields[5]); > > > > > if(c==null) > > > > > logError("Failed to get country for:"+line); > > > > > else > > > > > { > > > > > loc.setCountryId(c.getCountryId()); > > > > > loc.setCountry(c.getName()); > > > > > pm.makePersistent(loc);} > > > > > } > > > > > } > > > > > } > > > > > } catch (Exception e) { > > > > > > e.printStackTrace(); > > > > > > } finally { > > > > > pm.close(); > > > > > } > > > > > } > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups > > > > "Google App Engine for Java" group. > > > > To post to this group, send email to > > > > google-appengine-java@googlegroups.com. > > > > To unsubscribe from this group, send email to > > > > google-appengine-java+unsubscr...@googlegroups.com. > > > > For more options, visit this group at > > > >http://groups.google.com/group/google-appengine-java?hl=en. > > > > -- > > > Cláudio Coelho > > > > "Great spirits have often encountered violent opposition from weak minds." > > > A. Einstein -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.