Hello
I'm observing huge difference in CPU usage as reported in logs (and on
dashboard) and what I see from QuotaService based CPU usage profiling
in code. Here're the details:
"investment-advisor.appspot.com" ms=9272 cpu_ms=95562 api_cpu_ms=90569
cpm_usd=2.662842 loading_request=1
instance=00c61b117cca086cac52d0dadb5205551cafea7d
As per log (above), CPU_MS=95562
However, as reported by QuotaService: CPU_MS < 3000.
This code basically URLFetch some data, and writes to datastore. It
writes about 2262 entries into datastore. Is such a high CPU usage
expected in writing ~2k entries? Here're the reported cpu_ms by
QuotaService for each fragment of code (below):
2011-08-18 20:40:28.517
com.myco.finance.financialanalyst.server.admin.worker.FinanceDataDownloader
downloadDayEndData: CPU usage before fetching data: 0.86
I 2011-08-18 20:40:32.689
com.myco.finance.financialanalyst.server.admin.worker.FinanceDataDownloader
downloadDayEndData: CPU usage in fetching historical stock data: ^NSEI
1.2858333333333334
I 2011-08-18 20:40:32.768
com.myco.finance.financialanalyst.server.admin.worker.FinanceDataDownloader
downloadDayEndData: Historical data present and hence not downloaded
for GOLDBEES.NS
I 2011-08-18 20:40:32.769
com.myco.finance.financialanalyst.server.admin.worker.FinanceDataDownloader
downloadDayEndData: Historical data present and hence not downloaded
for LT.NS
I 2011-08-18 20:40:32.769
com.myco.finance.financialanalyst.server.admin.worker.FinanceDataDownloader
downloadDayEndData: Historical data present and hence not downloaded
for SBIN.NS
I 2011-08-18 20:40:32.904
com.myco.finance.financialanalyst.server.admin.worker.FinanceDataDownloader
downloadDayEndData: CPU usage in fetching EOD stock data:
0.005833333333333334
I 2011-08-18 20:40:32.964
com.myco.finance.financialanalyst.server.admin.worker.FinanceDataDownloader
downloadDayEndData: CPU usage in writing EOD stock data:
0.0033333333333333335
I 2011-08-18 20:40:32.965
com.myco.finance.financialanalyst.server.admin.cron.CronJobHttpServlet
cronDownloadDailyData: cronDownloadDailyData: Successfully downloaded
EOD financial data: 2262 entries
here's the code -
// Profiling CPU usage
QuotaService qs = QuotaServiceFactory.getQuotaService();
// Profiling CPU usage
long start = qs.getCpuTimeInMegaCycles();
// Search the list of securities in datastore currently
FinancialSecurityDAO financialSecurityDAO = new
FinancialSecurityDAO();
QueryResultIterator<FinancialSecurity> securityKeyIterator =
financialSecurityDAO.ofy().query(FinancialSecurity.class).fetch().iterator();
// Create a single list, keep adding
List<FinancialSecurity> financialSecurityList = new
ArrayList<FinancialSecurity>();
while (securityKeyIterator.hasNext()) {
FinancialSecurity security = securityKeyIterator.next();
financialSecurityList.add(security);
}
// If there're no elements in list, generate a new one
if (financialSecurityList.size() == 0) {
financialSecurityList =
initialiseDefaultSecurityList(financialSecurityDAO);
}
// Obtain data from Data provider
StockDataProvider stockDataProvider = new
YahooStockDataProvider();
CandleStickDAO candleStickDAO = new CandleStickDAO();
// Profiling CPU usage
long end = qs.getCpuTimeInMegaCycles();
log.info("CPU usage before fetching data: " +
Double.toString(qs.convertMegacyclesToCpuSeconds(end -
start)));
List<CandleStick> candleStickList = new ArrayList<CandleStick>();
List<FinancialSecurity> stockFinancialSecurityList = new
ArrayList<FinancialSecurity>();
Date incidesDate = getYesterdayDate();
int addedCount = 0;
for (FinancialSecurity security : financialSecurityList) {
// Download if historical data not downloaded already, or if
INDEX
if (security.getFinancialSecurityType() ==
FinancialSecurityType.INDEX ||
!security.hasHistoricalDataDownloadedAlready()) {
// Add immediately to avoid timeout, since historical data can
be large
// Profiling CPU usage
start = qs.getCpuTimeInMegaCycles();
int count = candleStickDAO.putAll(
stockDataProvider.getAllDailyCandleStickData(security)).size();
// Profiling CPU usage
end = qs.getCpuTimeInMegaCycles();
log.info("CPU usage in fetching historical stock data: " +
security.getTickerSymbol() + " " +
Double.toString(qs.convertMegacyclesToCpuSeconds(end -
start)));
// Update, if NOT index
if (count > 0) {
security.historicalDataDownloaded();
financialSecurityDAO.put(security);
}
addedCount += count;
} else {
// Add to EOD list
log.info("Historical data present and hence not downloaded for
" + security.getTickerSymbol());
stockFinancialSecurityList.add(security);
}
/*
// For indices, since only historical index data available
if () {
// This may fail, so add check, and we're ok with the failure
try {
// Profiling CPU usage
start = qs.getCpuTimeInMegaCycles();
int count = candleStickDAO.putAll(
stockDataProvider.getAllDailyCandleStickData(security)).size();
// Profiling CPU usage
end = qs.getCpuTimeInMegaCycles();
log.info("CPU usage in fetching index data for previous day:
" + security.getTickerSymbol() + " " +
Double.toString(qs.convertMegacyclesToCpuSeconds(end -
start)));
} catch (DataProviderException e) {
// Check for FileNotFoundException, otherwise throw again,
need to do this due to wrapping
if (e.getCause() instanceof FileNotFoundException) {
// Log exception
log.warning(e.getMessage());
} else {
// Throw any other exception
throw e;
}
}
if (security.getFinancialSecurityType() !=
FinancialSecurityType.INDEX) {
stockFinancialSecurityList.add(security);
}
*/
}
// Profiling CPU usage
start = qs.getCpuTimeInMegaCycles();
// Now for stocks
if (stockFinancialSecurityList.size() > 0) {
candleStickList.addAll(stockDataProvider.getDayEndCandleStickData(stockFinancialSecurityList));
}
// Profiling CPU usage
end = qs.getCpuTimeInMegaCycles();
log.info("CPU usage in fetching EOD stock data: " +
Double.toString(qs.convertMegacyclesToCpuSeconds(end -
start)));
// Profiling CPU usage
start = qs.getCpuTimeInMegaCycles();
// Write to datastore
if (candleStickList.size() > 0) {
addedCount += candleStickDAO.putAll(candleStickList).size();
}
// Profiling CPU usage
end = qs.getCpuTimeInMegaCycles();
log.info("CPU usage in writing EOD stock data: " +
Double.toString(qs.convertMegacyclesToCpuSeconds(end -
start)));
return addedCount;
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
