Hi,
One large difference is that the quota service does not include API
CPU. Perhaps the speed multipier has changed. If you don't convert
megacycles to cpu ms, is it proportional to the cpu time (adjusting
for API time)?
Robert
On Thu, Aug 18, 2011 at 23:58, Sunny Gupta <sunnymax2...@gmail.com> wrote:
> Hello
>
> I'm observing huge difference in CPU usage as reported in logs (and on
> dashboard) and what I see from QuotaService based CPU usage profiling
> in code. Here're the details:
>
> "investment-advisor.appspot.com" ms=9272 cpu_ms=95562 api_cpu_ms=90569
> cpm_usd=2.662842 loading_request=1
> instance=00c61b117cca086cac52d0dadb5205551cafea7d
>
> As per log (above), CPU_MS=95562
>
> However, as reported by QuotaService: CPU_MS < 3000.
>
> This code basically URLFetch some data, and writes to datastore. It
> writes about 2262 entries into datastore. Is such a high CPU usage
> expected in writing ~2k entries? Here're the reported cpu_ms by
> QuotaService for each fragment of code (below):
>
> 2011-08-18 20:40:28.517
> com.myco.finance.financialanalyst.server.admin.worker.FinanceDataDownloader
> downloadDayEndData: CPU usage before fetching data: 0.86
> I 2011-08-18 20:40:32.689
> com.myco.finance.financialanalyst.server.admin.worker.FinanceDataDownloader
> downloadDayEndData: CPU usage in fetching historical stock data: ^NSEI
> 1.2858333333333334
> I 2011-08-18 20:40:32.768
> com.myco.finance.financialanalyst.server.admin.worker.FinanceDataDownloader
> downloadDayEndData: Historical data present and hence not downloaded
> for GOLDBEES.NS
> I 2011-08-18 20:40:32.769
> com.myco.finance.financialanalyst.server.admin.worker.FinanceDataDownloader
> downloadDayEndData: Historical data present and hence not downloaded
> for LT.NS
> I 2011-08-18 20:40:32.769
> com.myco.finance.financialanalyst.server.admin.worker.FinanceDataDownloader
> downloadDayEndData: Historical data present and hence not downloaded
> for SBIN.NS
> I 2011-08-18 20:40:32.904
> com.myco.finance.financialanalyst.server.admin.worker.FinanceDataDownloader
> downloadDayEndData: CPU usage in fetching EOD stock data:
> 0.005833333333333334
> I 2011-08-18 20:40:32.964
> com.myco.finance.financialanalyst.server.admin.worker.FinanceDataDownloader
> downloadDayEndData: CPU usage in writing EOD stock data:
> 0.0033333333333333335
> I 2011-08-18 20:40:32.965
> com.myco.finance.financialanalyst.server.admin.cron.CronJobHttpServlet
> cronDownloadDailyData: cronDownloadDailyData: Successfully downloaded
> EOD financial data: 2262 entries
>
> here's the code -
>
> // Profiling CPU usage
> QuotaService qs = QuotaServiceFactory.getQuotaService();
>
> // Profiling CPU usage
> long start = qs.getCpuTimeInMegaCycles();
>
> // Search the list of securities in datastore currently
> FinancialSecurityDAO financialSecurityDAO = new
> FinancialSecurityDAO();
> QueryResultIterator<FinancialSecurity> securityKeyIterator =
>
> financialSecurityDAO.ofy().query(FinancialSecurity.class).fetch().iterator();
>
> // Create a single list, keep adding
> List<FinancialSecurity> financialSecurityList = new
> ArrayList<FinancialSecurity>();
> while (securityKeyIterator.hasNext()) {
> FinancialSecurity security = securityKeyIterator.next();
> financialSecurityList.add(security);
> }
>
> // If there're no elements in list, generate a new one
> if (financialSecurityList.size() == 0) {
> financialSecurityList =
> initialiseDefaultSecurityList(financialSecurityDAO);
> }
>
> // Obtain data from Data provider
> StockDataProvider stockDataProvider = new
> YahooStockDataProvider();
> CandleStickDAO candleStickDAO = new CandleStickDAO();
>
> // Profiling CPU usage
> long end = qs.getCpuTimeInMegaCycles();
> log.info("CPU usage before fetching data: " +
> Double.toString(qs.convertMegacyclesToCpuSeconds(end -
> start)));
>
> List<CandleStick> candleStickList = new ArrayList<CandleStick>();
> List<FinancialSecurity> stockFinancialSecurityList = new
> ArrayList<FinancialSecurity>();
> Date incidesDate = getYesterdayDate();
> int addedCount = 0;
> for (FinancialSecurity security : financialSecurityList) {
> // Download if historical data not downloaded already, or if
> INDEX
> if (security.getFinancialSecurityType() ==
> FinancialSecurityType.INDEX ||
> !security.hasHistoricalDataDownloadedAlready()) {
> // Add immediately to avoid timeout, since historical data can
> be large
>
> // Profiling CPU usage
> start = qs.getCpuTimeInMegaCycles();
>
> int count = candleStickDAO.putAll(
>
> stockDataProvider.getAllDailyCandleStickData(security)).size();
>
> // Profiling CPU usage
> end = qs.getCpuTimeInMegaCycles();
> log.info("CPU usage in fetching historical stock data: " +
> security.getTickerSymbol() + " " +
> Double.toString(qs.convertMegacyclesToCpuSeconds(end -
> start)));
>
> // Update, if NOT index
> if (count > 0) {
> security.historicalDataDownloaded();
> financialSecurityDAO.put(security);
> }
>
> addedCount += count;
> } else {
> // Add to EOD list
> log.info("Historical data present and hence not downloaded for
> " + security.getTickerSymbol());
> stockFinancialSecurityList.add(security);
> }
>
> /*
> // For indices, since only historical index data available
> if () {
> // This may fail, so add check, and we're ok with the failure
> try {
>
> // Profiling CPU usage
> start = qs.getCpuTimeInMegaCycles();
>
> int count = candleStickDAO.putAll(
>
> stockDataProvider.getAllDailyCandleStickData(security)).size();
>
>
> // Profiling CPU usage
> end = qs.getCpuTimeInMegaCycles();
> log.info("CPU usage in fetching index data for previous day:
> " + security.getTickerSymbol() + " " +
> Double.toString(qs.convertMegacyclesToCpuSeconds(end -
> start)));
>
> } catch (DataProviderException e) {
> // Check for FileNotFoundException, otherwise throw again,
> need to do this due to wrapping
> if (e.getCause() instanceof FileNotFoundException) {
> // Log exception
> log.warning(e.getMessage());
> } else {
> // Throw any other exception
> throw e;
> }
> }
>
> if (security.getFinancialSecurityType() !=
> FinancialSecurityType.INDEX) {
> stockFinancialSecurityList.add(security);
> }
> */
> }
>
> // Profiling CPU usage
> start = qs.getCpuTimeInMegaCycles();
>
> // Now for stocks
> if (stockFinancialSecurityList.size() > 0) {
>
> candleStickList.addAll(stockDataProvider.getDayEndCandleStickData(stockFinancialSecurityList));
> }
>
> // Profiling CPU usage
> end = qs.getCpuTimeInMegaCycles();
> log.info("CPU usage in fetching EOD stock data: " +
> Double.toString(qs.convertMegacyclesToCpuSeconds(end -
> start)));
>
> // Profiling CPU usage
> start = qs.getCpuTimeInMegaCycles();
>
> // Write to datastore
> if (candleStickList.size() > 0) {
> addedCount += candleStickDAO.putAll(candleStickList).size();
> }
>
> // Profiling CPU usage
> end = qs.getCpuTimeInMegaCycles();
> log.info("CPU usage in writing EOD stock data: " +
> Double.toString(qs.convertMegacyclesToCpuSeconds(end -
> start)));
>
> return addedCount;
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
