Kalpesh, Bruce posted at https://github.com/finance-quote/finance-quote/issues/335#issuecomment-1676443182: "While the fix did not make it into v1.58, you can download and install BPSCHUCK/Finance-Quote-1.58_01.tar.gz from CPAN for testing/verification. It also has what I hope is a fix for TSP. It was not returning the hash if the GET failed or if the body of the response produced a CSV file."
That's the BSEIndia issue; https://github.com/finance-quote/finance-quote/issues/338 is the TSP one. If you find more problems with TSP (or anything else in F::Q) and want to help debug it the best way to communicate with Bruce and stay up to date would be to open an issue at https://github.com/finance-quote/finance-quote/issues/. You can post here too if you want, but a Github issue is a better place to put debugging results. Regards, John Ralls > On Aug 14, 2023, at 1:54 PM, Kalpesh Patel <kalpesh.pa...@usa.net> wrote: > > Some good find in debugging ... Strawberry Perl cannot resolve the FQDN name > one out of three times for www.tsp.gov on my system (this might be > obfuscating the client behavior in response to real backend issue). It might > be that I am reaching servers that are in error state (rc = 500 which is > Internal Server Error; my observation on F::Q could be cause of concern on > amount of data being returned). Below is the output from the failed run: > > C:\Users\kalpesh\OneDrive\QuickenStuff\HELPERS>set DEBUG=1 > > C:\Users\kalpesh\OneDrive\QuickenStuff\HELPERS>test2.pl > > ### [Mon Aug 14 16:26:56 2023] YIND_URL_HEAD : > 'https://query1.finance.yahoo.com/v8/finance/chart/' > ### [Mon Aug 14 16:26:56 2023] YIND_URL_TAIL : > '?interval=1d&period1=1691440016&period2=1692044816' > > ### AlphaVantage->new args : {} > > ### COUNT_URL: 'http://www.panix.com/~hd-fxsts/finance-quote.html?tsp' > ### Code: '200' > > ### [Mon Aug 14 16:26:56 2023] url : > 'https://www.tsp.gov/data/fund-price-history.csv?startdate=2023-08-07&enddate=2023-08-14&Lfunds=1&InvFunds=1&download=1' > ### [Mon Aug 14 16:26:56 2023] reply: bless( { > ### _content => 'Can\'t connect to > www.tsp.gov:443 (nodename nor servname provided, or not known) > > nodename nor servname provided, or not known at > C:/Strawberry/perl/site/lib/LWP/Protocol/http.pm line 50. > ', > ### _headers => bless( { > ### > '::std_case' => { > ### > 'client-date' => 'Client-Date', > ### > 'client-warning' => 'Client-Warning' > ### > }, > ### > 'client-date' => 'Mon, 14 Aug 2023 20:26:56 GMT', > ### > 'client-warning' => 'Internal response', > ### > 'content-type' => 'text/plain' > ### }, > 'HTTP::Headers' ), > ### _msg => 'Can\'t connect to > www.tsp.gov:443 (nodename nor servname provided, or not known)', > ### _rc => 500, > ### _request => bless( { > ### _content > => '', > ### _headers > => bless( { > ### > 'user-agent' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36' > ### > }, 'HTTP::Headers' ), > ### _method > => 'GET', > ### _uri => > bless( do{\(my $o = > 'https://www.tsp.gov/data/fund-price-history.csv?startdate=2023-08-07&enddate=2023-08-14&Lfunds=1&InvFunds=1&download=1')}, > 'URI::https' ) > ### }, > 'HTTP::Request' ) > ### }, 'HTTP::Response' ) > > C:\Users\kalpesh\OneDrive\QuickenStuff\HELPERS> > > My test file (test2.pl) is as follows: > > #test2.pl - start > > use Finance::Quote; > use Data::Dumper; > > $q = Finance::Quote->new; > > %info = $q->fetch('tsp', "L2030"); > > print Dumper(%info); > > # test2.pl - end > > > Interesting side notes for GNC (I am on Windows 11 with GNC v4.14 and now > F::Q v1.58 so might have some subtle from it): > > I was able to put the module in debug mode (set DEBUG=1 in cmd.exe; of course > install the Smart::Comment Perl module first) and have them write out the > smart comment messages to cmd console as you can see from above when I run a > small test file that I've created. However, when I lunch gnc in normal > fashion from the same console where debugging is enabled, the screen does not > display anything, not even the smart comment; it is almost like it is > ignoring all env settings or not being passed to sub-shell. Windows is > notorious for this so there may not be a "clean" way of doing so. There is a > new window that pops up and then disappears at the start up (this process > seems to be part normal start up - check if F::Q is there or not) and does > not display anything also. I attempted to start gnc with debug (with > --log=gnc.scm=debug option) as well but that also does not emit anything > either. On the note of specifying a file name for logging (--logto= > parameter), it does not create nor write any file anywhere on the file system when only a fixed file name is given. I took away that option and it was happy to write a file to %LOCALAPPDATA% path with gnucash.trace.<random_six_alphanumeric_characters>.log. > > > Interesting side notes for F::Q TSP module: > > When it does work properly, the return payload from HTTP seem to be hundreds > of lines if not thousands ... Here is the snippet of successful run: > > C:\Users\kalpesh\OneDrive\QuickenStuff\HELPERS>test2.pl > > ### [Mon Aug 14 15:42:00 2023] YIND_URL_HEAD : > 'https://query1.finance.yahoo.com/v8/finance/chart/' > ### [Mon Aug 14 15:42:00 2023] YIND_URL_TAIL : > '?interval=1d&period1=1691437320&period2=1692042120' > > ### AlphaVantage->new args : {} > > ### COUNT_URL: 'http://www.panix.com/~hd-fxsts/finance-quote.html?tsp' > ### Code: '200' > > ### [Mon Aug 14 15:42:01 2023] url : > 'https://www.tsp.gov/data/fund-price-history.csv?startdate=2023-08-07&enddate=2023-08-14&Lfunds=1&InvFunds=1&download=1' > ### [Mon Aug 14 15:42:01 2023] reply: bless( { > ### _content => 'Date,L Income,L > 2025,L 2030,L 2035,L 2040,L 2045,L 2050,L 2055,L 2060,L 2065,G Fund,F Fund,C > Fund,S Fund,I Fund > 2023-08-11,24.0849,12.2327,42.9435,12.8255,48.3009,13.1625,28.7245,14.0971,14.0954,14.0934,17.6469,18.3534,69.1565,70.2391,37.9419 > 2023-08-10,24.1082,12.2494,43.0380,12.8567,48.4288,13.1999,28.8108,14.1442,14.1425,14.1406,17.6449,18.4205,69.2207,70.3124,38.2391 > 2023-08-09,24.1016,12.2437,42.9976,12.8439,48.3758,13.1845,28.7742,14.1155,14.1139,14.1120,17.6429,18.5294,69.1951,70.4343,38.0083 > 2023-08-08,24.1272,12.2634,43.1181,12.8835,48.5391,13.2323,28.8859,14.1834,14.1818,14.1799,17.6409,18.5100,69.6795,71.0740,38.0127 > 2023-08-07,24.1541,12.2847,43.2515,12.9273,48.7198,13.2853,29.0100,14.2616,14.2599,14.2580,17.6389,18.4477,69.9736,71.7269,38.2463 > 2023-08-04,24.1053,12.2504,43.0581,12.8645,48.4638,13.2110,28.8379,14.1583,14.1566,14.1547,17.6328,18.4709,69.3457,71.4617,38.0032 > 2023-08-03,24.1034,12.2517,43.0826,12.8723,48.4974,13.2209,28.8626,14.1833,14.1815,14.1796,17.6308,18.3188,69.7102,71.6307,37.8756 > 2023-08-02,24.1277,12.2685,43.1728,12.9023,48.6200,13.2568,28.9449,14.2243,14.2226,14.2207,17.6289,18.4426,69.8850,71.8330,38.0070 > 2023-08-01,24.2277,12.3420,43.6036,13.0440,49.2020,13.4269,29.3407,14.4570,14.4553,14.4534,17.6269,18.4987,70.8648,73.1095,38.7558 > 2023-07-31,24.2660,12.3694,43.7577,13.0949,49.4108,13.4880,29.4819,14.5344,14.5328,14.5310,17.6249,18.6043,71.0523,73.4061,39.1413 > 2023-07-28,24.2467,12.3571,43.6964,13.0751,49.3307,13.4647,29.4283,14.5045,14.5029,14.5010,17.6191,18.5820,70.9459,72.6930,39.1484 > 2023-07-27,24.1833,12.3118,43.4387,12.9905,48.9842,13.3635,29.1936,14.3691,14.3675,14.3656,17.6172,18.5164,70.2457,71.6543,38.8820 > 2023-07-26,24.2258,12.3417,43.6042,13.0453,49.2086,13.4293,29.3454,14.4499,14.4484,14.4465,17.6152,18.6653,70.6948,72.5871,38.9458 > 2023-07-25,24.2163,12.3359,43.5775,13.0365,49.1733,13.4189,29.3220,14.4395,14.4379,14.4361,17.6133,18.6144,70.7047,72.2457,38.9313 > 2023-07-24,24.1974,12.3223,43.4989,13.0110,49.0687,13.3885,29.2512,14.3963,14.3947,14.3929,17.6114,18.6334,70.5044,72.1292,38.7806 > 2023-07-21,24.1841,12.3139,43.4569,12.9977,49.0157,13.3735,29.2167,14.3747,14.3732,14.3714,17.6055,18.6571,70.2201,72.1244,38.8425 > 2023-07-20,24.1808,12.3123,43.4515,12.9961,49.0096,13.3719,29.2133,14.3737,14.3722,14.3704,17.6036,18.6455,70.1974,72.2722,38.8218 > 2023-07-19,24.2266,12.3452,43.6382,13.0578,49.2628,13.4460,29.3849,14.4692,14.4677,14.4659,17.6017,18.7521,70.6685,73.1168,39.0026 > 2023-07-18,24.2119,12.3358,43.5911,13.0423,49.2001,13.4277,29.3433,14.4485,14.4469,14.4451,17.5998,18.6938,70.5015,72.8049,39.0442 > 2023-07-17,24.1639,12.3012,43.3926,12.9773,48.9337,13.3500,29.1627,14.3422,14.3407,14.3389,17.5978,18.6769,70.0036,72.0267,38.7854 > 2023-07-14,24.1430,12.2877,43.3244,12.9552,48.8445,13.3242,29.1032,14.3088,14.3072,14.3055,17.5920,18.6543,69.7345,71.3406,38.8917 > 2023-07-13,24.1672,12.3051,43.4224,12.9878,48.9782,13.3634,29.1941,14.3581,14.3566,14.3548,17.5901,18.7318,69.8035,72.0356,39.0693 > 2023-07-12,24.0841,12.2462,43.0912,12.8792,48.5343,13.2341,28.8948,14.1877,14.1862,14.1844,17.5881,18.6216,69.2107,71.2102,38.4014 > 2023-07-11,24.0007,12.1877,42.7678,12.7731,48.1017,13.1082,28.6041,14.0255,14.0239,14.0220,17.5862,18.4682,68.7006,70.6988,37.6648 > 2023-07-10,23.9468,12.1490,42.5478,12.7011,47.8070,13.0223,28.4048,13.9094,13.9078,13.9059,17.5843,18.4385,68.2399,69.6977,37.3488 > 2023-07-07,23.9111,12.1253,42.4239,12.6607,47.6429,12.9745,28.2947,13.8488,13.8471,13.8453,17.5785,18.3742,68.0764,68.6900,37.2301 > 2023-07-06,23.8994,12.1170,42.3776,12.6455,47.5804,12.9561,28.2516,13.8230,13.8214,13.8195,17.5765,18.3788,68.2614,68.0180,37.0282 > [snip] > > > Three observation on F::Q: > > 1) This request downloads CSVs that goes back to 2003-05-31 (20+ years!) even > though request is only for 7 days in the URL. It is like enddate= parameter > is ignored or incorrect parameter passed. Possibly a bug in TSP.pm module? > 2) Is the anonymous data collection of modules utilized still being > collected? Code execution is not harming anything if not being collected but > this is more curiosity on my part for v1.58 which was released few days back. > 3) Looks like Smart::Comment when activated emits messages for all modules > rather than the single one that is being instantiated for the source and > called to action. > > On to debugging the Perl LWP module... > > > > > -----Original Message----- > From: Bruce Schuck <bsch...@asgard-systems.com> > Sent: Friday, August 11, 2023 9:40 PM > To: gnucash-user@gnucash.org > Cc: Kalpesh Patel <kalpesh.pa...@usa.net>; john <jra...@ceridwen.us> > Subject: Re: [GNC] Strange prices update issue (possibly with TSP) > > On Aug 10, 2023, at 12:13, Kalpesh Patel <kalpesh.pa...@usa.net > <mailto:kalpesh.pa...@usa.net> > wrote: > >> The difficult part in debugging this is that when I do query three >> times from the command line, it succeeds without any issue for me as >> well, but when I update prices from GNC, it fails two times and then >> succeeds third time consistently. So this is not an issue with >> throttling from TSP source which you validated as well. Is there more >> in terms of debugging this by any chance? Can I use STDERR and STDOUT >> from Perl to emanate messages to console but not interfere with what >> JSON comes back or butcher the response back to gnc-fq-helper so it >> doesn't bomb to run a full cycles from GNC to test? Any help in debugging is >> appreciated. > > Kalpesh, > > Many, but not all, of the F::Q modules make use of the Smart::Comments > module. TSP.pm is one of them. The environment variable DEBUG will need to be > set, and with no changes to the Perl code you should see helpful information > printed to STDERR. In the code, these are lines that begin with "###" (note, > just 3 pound signs) in those modules using Smart::Comments. > > A quick look at the TSP module and I see two where the "tsp" method will > return *without* setting "success" and "errormsg" for any of the symbols > passed to it. This will happen if the http get call fails, or whatever is > downloaded is not 2 or more lines. > > my $reply = $ua->get($url, @HEADERS); > ### [<now>] url : $url > ### [<now>] reply: $reply > > return unless ($reply->is_success); > > my @line = split(/\n/, $reply->content); > > return unless (@line > 1); > > In the code snippet, there are two examples Smart::Comments use which could > be useful in your case. With DEBUG set the code would print the url to > STDERR, and entire $reply object (status, headers, cookies, body). > > Whether or not this helps you, the omission of not returning the proper > failure data is an issue. > > Hope this helps. > > Bruce S > _______________________________________________ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user ----- Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
