Hello I'm observing huge difference in CPU usage as reported in logs (and on dashboard) and what I see from QuotaService based CPU usage profiling in code. Here're the details:
"investment-advisor.appspot.com" ms=9272 cpu_ms=95562 api_cpu_ms=90569 cpm_usd=2.662842 loading_request=1 instance=00c61b117cca086cac52d0dadb5205551cafea7d As per log (above), CPU_MS=95562 However, as reported by QuotaService: CPU_MS < 3000. This code basically URLFetch some data, and writes to datastore. It writes about 2262 entries into datastore. Is such a high CPU usage expected in writing ~2k entries? Here're the reported cpu_ms by QuotaService for each fragment of code (below): 2011-08-18 20:40:28.517 com.myco.finance.financialanalyst.server.admin.worker.FinanceDataDownloader downloadDayEndData: CPU usage before fetching data: 0.86 I 2011-08-18 20:40:32.689 com.myco.finance.financialanalyst.server.admin.worker.FinanceDataDownloader downloadDayEndData: CPU usage in fetching historical stock data: ^NSEI 1.2858333333333334 I 2011-08-18 20:40:32.768 com.myco.finance.financialanalyst.server.admin.worker.FinanceDataDownloader downloadDayEndData: Historical data present and hence not downloaded for GOLDBEES.NS I 2011-08-18 20:40:32.769 com.myco.finance.financialanalyst.server.admin.worker.FinanceDataDownloader downloadDayEndData: Historical data present and hence not downloaded for LT.NS I 2011-08-18 20:40:32.769 com.myco.finance.financialanalyst.server.admin.worker.FinanceDataDownloader downloadDayEndData: Historical data present and hence not downloaded for SBIN.NS I 2011-08-18 20:40:32.904 com.myco.finance.financialanalyst.server.admin.worker.FinanceDataDownloader downloadDayEndData: CPU usage in fetching EOD stock data: 0.005833333333333334 I 2011-08-18 20:40:32.964 com.myco.finance.financialanalyst.server.admin.worker.FinanceDataDownloader downloadDayEndData: CPU usage in writing EOD stock data: 0.0033333333333333335 I 2011-08-18 20:40:32.965 com.myco.finance.financialanalyst.server.admin.cron.CronJobHttpServlet cronDownloadDailyData: cronDownloadDailyData: Successfully downloaded EOD financial data: 2262 entries here's the code - // Profiling CPU usage QuotaService qs = QuotaServiceFactory.getQuotaService(); // Profiling CPU usage long start = qs.getCpuTimeInMegaCycles(); // Search the list of securities in datastore currently FinancialSecurityDAO financialSecurityDAO = new FinancialSecurityDAO(); QueryResultIterator<FinancialSecurity> securityKeyIterator = financialSecurityDAO.ofy().query(FinancialSecurity.class).fetch().iterator(); // Create a single list, keep adding List<FinancialSecurity> financialSecurityList = new ArrayList<FinancialSecurity>(); while (securityKeyIterator.hasNext()) { FinancialSecurity security = securityKeyIterator.next(); financialSecurityList.add(security); } // If there're no elements in list, generate a new one if (financialSecurityList.size() == 0) { financialSecurityList = initialiseDefaultSecurityList(financialSecurityDAO); } // Obtain data from Data provider StockDataProvider stockDataProvider = new YahooStockDataProvider(); CandleStickDAO candleStickDAO = new CandleStickDAO(); // Profiling CPU usage long end = qs.getCpuTimeInMegaCycles(); log.info("CPU usage before fetching data: " + Double.toString(qs.convertMegacyclesToCpuSeconds(end - start))); List<CandleStick> candleStickList = new ArrayList<CandleStick>(); List<FinancialSecurity> stockFinancialSecurityList = new ArrayList<FinancialSecurity>(); Date incidesDate = getYesterdayDate(); int addedCount = 0; for (FinancialSecurity security : financialSecurityList) { // Download if historical data not downloaded already, or if INDEX if (security.getFinancialSecurityType() == FinancialSecurityType.INDEX || !security.hasHistoricalDataDownloadedAlready()) { // Add immediately to avoid timeout, since historical data can be large // Profiling CPU usage start = qs.getCpuTimeInMegaCycles(); int count = candleStickDAO.putAll( stockDataProvider.getAllDailyCandleStickData(security)).size(); // Profiling CPU usage end = qs.getCpuTimeInMegaCycles(); log.info("CPU usage in fetching historical stock data: " + security.getTickerSymbol() + " " + Double.toString(qs.convertMegacyclesToCpuSeconds(end - start))); // Update, if NOT index if (count > 0) { security.historicalDataDownloaded(); financialSecurityDAO.put(security); } addedCount += count; } else { // Add to EOD list log.info("Historical data present and hence not downloaded for " + security.getTickerSymbol()); stockFinancialSecurityList.add(security); } /* // For indices, since only historical index data available if () { // This may fail, so add check, and we're ok with the failure try { // Profiling CPU usage start = qs.getCpuTimeInMegaCycles(); int count = candleStickDAO.putAll( stockDataProvider.getAllDailyCandleStickData(security)).size(); // Profiling CPU usage end = qs.getCpuTimeInMegaCycles(); log.info("CPU usage in fetching index data for previous day: " + security.getTickerSymbol() + " " + Double.toString(qs.convertMegacyclesToCpuSeconds(end - start))); } catch (DataProviderException e) { // Check for FileNotFoundException, otherwise throw again, need to do this due to wrapping if (e.getCause() instanceof FileNotFoundException) { // Log exception log.warning(e.getMessage()); } else { // Throw any other exception throw e; } } if (security.getFinancialSecurityType() != FinancialSecurityType.INDEX) { stockFinancialSecurityList.add(security); } */ } // Profiling CPU usage start = qs.getCpuTimeInMegaCycles(); // Now for stocks if (stockFinancialSecurityList.size() > 0) { candleStickList.addAll(stockDataProvider.getDayEndCandleStickData(stockFinancialSecurityList)); } // Profiling CPU usage end = qs.getCpuTimeInMegaCycles(); log.info("CPU usage in fetching EOD stock data: " + Double.toString(qs.convertMegacyclesToCpuSeconds(end - start))); // Profiling CPU usage start = qs.getCpuTimeInMegaCycles(); // Write to datastore if (candleStickList.size() > 0) { addedCount += candleStickDAO.putAll(candleStickList).size(); } // Profiling CPU usage end = qs.getCpuTimeInMegaCycles(); log.info("CPU usage in writing EOD stock data: " + Double.toString(qs.convertMegacyclesToCpuSeconds(end - start))); return addedCount; -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.