Re: [GNC-dev] Alphavantage strategies
On 22/01/2019 15:16, John Ralls wrote: Wm, F::Q has had a commit to add 6 seconds between requests in its git repo for almost a year, but the maintainer hasn’t been able to make time to do a new release. This has been discussed several times on the user list. Naughty F::Q ... don't be silly, the notion exists, it got used but not implemented. I don't think we (gnc and other people that use F::Q) should presume they should protect us from being bad citizens. And something else, the 6 seconds is growing, 16 seconds, maybe 61 seconds soon, I don't know what the back off should be but I know for sure gnc is not suitable for trading (by that I mean prices within a day or maybe within a few days). There’s also already a price-priority built into GnuCash. > >There are several levels, but the significant ones are transaction-creation, I certainly know about that one and expect most people experience it ordinarily in buying food, etc finance-quote, I think you have the order wrong price-editor. I think an actual price overrides. Sources to the right can replace sources to their left my reading of what you are saying is that the price-editor should override the actual, but I'm not convinced that is really what you mean. and themselves but not sources to the right, so e.g. once you’ve created a price in the price editor >that’s the one for the day. Hmmmn, I'm going to pause and think about that. I think that is a wrong-thing: I think the actual human level price is the price for the day for a person or small or modest sized organisation rather than the price editor price, if you toss in the ordinary stuff like some people and organisations can't get prices at the moment, what do you think is the right price for the day? Is it the market price (big wide world) or the price they could achieve at the time on the day they attempted to get a price? I am genuinely unsure about this, JohnR [99] , I think the price I have for something must be the price for the day *because* the gnc model of last price falls apart if it is supplanted. [99] and other sensible commentators, remote as they may be === I live in London, I don't vote for Trump, if it matters I think Brexit is dumb, I have access to market prices. I'm generally ok with a price a person creates being prime (if I buy an option on DogFood and Wall at USD200 per kg of federal_employee that is my price for supporting Trump, right?) [1] I don’t want GnuCash to make assumptions about what F::Q does internally, My break: I agree I also think it will be very expensive for gnc and similar applications that utilize F::Q to start from scratch. so I think the algorithm you’re proposing would look like: 1 Make list of commodities to retrieve from Alphavantage doesn't gnc do that anyway? if that wasn't true we wouldn't be fucking about choosing which source we wanted a quote from, FFS 2 Check for prices for today, splitting the above list into have-price and don’t-have-price I think this is a good approach but it breaks if many people ask the same question at or about the same time or repeatedly. I don't know the answer to gnc and F::Q *not* being asked === alphabet share price === I don't use gnc or F::Q for that but I expect many people do. 3 Request the don’t-have-price list. If it partly fails, wait 60 seconds and return to 2. Almost, I think the person should get a list, similar to the one presented at presented that says, "we couldn't get the price for FuckwitAndCompany MayAndIdiots CorbynAndWeird" but instead of asking the dumb question about storing the prices it may have obtained think about this at the moment the gnc model gets good prices and throws them away! How fucking idiotic and Trump like is that? You have information? What is the best thing to do? I know! Discard it! Duh Sensible people store information, prices, etc why is gnc *discarding* what it does get? I simply don't understand this. Maybe someone else is whacking F::Q for a perfect complete set of all their data and someone else is just trying to work out if their small business is likely to get fucked by Brexit. Why does gnc not store the prices it did get and then ask "do you want me to try again, the sources may be fucked, I may not be able to get a price again for a day or so because the fuckwit Trump is holding things up" or whatever excuse you have but here is the question WHY IS GNC *NOT* STORING PRICES IT DOES GET? WHY IS GNC *DISCARDING* PRICES WHEN WE KNOW THEY ARE BECOMING MORE EXPENSIVE? 4 Wait 60 seconds and request the have-price list. if someone needs minute by minute prices I think they should pay for that, I'm ok with once a day at best. The code would go into libgnucash/scm/price-quotes.scm. at which point it would be obscure to me. I'd know if it worked or not afterwards so who are you trying to impress? === [1] Trump has moved beyond stupid to
Re: [GNC-dev] Alphavantage strategies
> On Jan 22, 2019, at 1:58 AM, Wm via gnucash-devel > wrote: > > On 21/01/2019 20:10, Derek Atkins wrote: >> Wm via gnucash-devel writes: >> [snip] > Wm's suggestion > === > only collect if the most recent price is more than 1 day (or 12 hours, > etc) old [2]. this sort of fits in with the existing post collection >> [snip] >>> Do you really miss the point on purposes? If gnc collects the last >>> price it doesn't have that is a good thing. >> All GnuCash knows is the time of the last price it received/stored. > > Yup, it also can't know the significance of one price vs another for the > user. If you're suggesting a price priority option I'm not against that but > I don't think it solves the problem you don't perceive. It would be a lot of > work UI wise and I'm pretty sure as soon as it appeared someone would > strongly object. > >> It >> has to make the request to A-V to know if there is a new price. > > There is always a new price in modern markets. That doesn't mean gnc should > be requesting a price every second or millisecond. The db design simply > doesn't support that sort of trading. There is other software for that. > >> So that >> means, by definition, GnuCash will automatically request any prices > >> old, whether or not there is a "new price", because GnuCash >> can't know if there is a new price until it makes the request. > > gnc can be a good citizen and not ask for a price it already has. I am > suggesting it asks for prices it doesn't have first. Why is this suggestion > so troubling to you? > >> Sure, this is a "Good Thing", but it's definitely not going to limit the >> number of price requests GnuCash makes Unless, of course, you're >> assuming people are requesting prices every few minutes? > > this has been done repeatedly in the user list, people *are* trying to use > gnc for intra day trading and generally find it doesn't work. > > I may be misreading you, Derek > It is possible you are playing devil's advocate > It appears to me you are against gnc (or possibly F::Q) backing off and being > a good citizen or doing a good thing. Wm, F::Q has had a commit to add 6 seconds between requests in its git repo for almost a year, but the maintainer hasn’t been able to make time to do a new release. This has been discussed several times on the user list. There’s also already a price-priority built into GnuCash. There are several levels, but the significant ones are transaction-creation, finance-quote, price-editor. Sources to the right can replace sources to their left and themselves but not sources to the right, so e.g. once you’ve created a price in the price editor that’s the one for the day. I don’t want GnuCash to make assumptions about what F::Q does internally, so I think the algorithm you’re proposing would look like: 1 Make list of commodities to retrieve from Alphavantage 2 Check for prices for today, splitting the above list into have-price and don’t-have-price 3 Request the don’t-have-price list. If it partly fails, wait 60 seconds and return to 2. 4 Wait 60 seconds and request the have-price list. The code would go into libgnucash/scm/price-quotes.scm. Regards, John Ralls ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: [GNC-dev] Alphavantage strategies
On 21/01/2019 20:10, Derek Atkins wrote: Wm via gnucash-devel writes: [snip] Wm's suggestion === only collect if the most recent price is more than 1 day (or 12 hours, etc) old [2]. this sort of fits in with the existing post collection [snip] Do you really miss the point on purposes? If gnc collects the last price it doesn't have that is a good thing. All GnuCash knows is the time of the last price it received/stored. Yup, it also can't know the significance of one price vs another for the user. If you're suggesting a price priority option I'm not against that but I don't think it solves the problem you don't perceive. It would be a lot of work UI wise and I'm pretty sure as soon as it appeared someone would strongly object. It has to make the request to A-V to know if there is a new price. There is always a new price in modern markets. That doesn't mean gnc should be requesting a price every second or millisecond. The db design simply doesn't support that sort of trading. There is other software for that. So that means, by definition, GnuCash will automatically request any prices > old, whether or not there is a "new price", because GnuCash can't know if there is a new price until it makes the request. gnc can be a good citizen and not ask for a price it already has. I am suggesting it asks for prices it doesn't have first. Why is this suggestion so troubling to you? Sure, this is a "Good Thing", but it's definitely not going to limit the number of price requests GnuCash makes Unless, of course, you're assuming people are requesting prices every few minutes? this has been done repeatedly in the user list, people *are* trying to use gnc for intra day trading and generally find it doesn't work. I may be misreading you, Derek It is possible you are playing devil's advocate It appears to me you are against gnc (or possibly F::Q) backing off and being a good citizen or doing a good thing. -- Wm ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: [GNC-dev] Alphavantage strategies
Wm via gnucash-devel writes: [snip] >>> Wm's suggestion >>> === >>> only collect if the most recent price is more than 1 day (or 12 hours, >>> etc) old [2]. this sort of fits in with the existing post collection [snip] > Do you really miss the point on purposes? If gnc collects the last > price it doesn't have that is a good thing. All GnuCash knows is the time of the last price it received/stored. It has to make the request to A-V to know if there is a new price. So that means, by definition, GnuCash will automatically request any prices > old, whether or not there is a "new price", because GnuCash can't know if there is a new price until it makes the request. Sure, this is a "Good Thing", but it's definitely not going to limit the number of price requests GnuCash makes Unless, of course, you're assuming people are requesting prices every few minutes? -derek -- Derek Atkins 617-623-3745 de...@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: [GNC-dev] Alphavantage strategies
On 16/01/2019 14:12, Derek Atkins wrote: Hi, Wm via gnucash-devel writes: [snip] Let's consider some simple backoff strategies that gnc could implement, feel free to add your own. Wm's suggestion === only collect if the most recent price is more than 1 day (or 12 hours, etc) old [2]. this sort of fits in with the existing post collection query, i.e. always store the good prices that have been obtained, instead ask the user, "do you want to try and fetch the other prices / rates again". this provides a natural pause to Alphavantage as gnc won't be hitting it for prices it just got a second time and would only be requesting the prices it didn't get last time. [2] I use this in my own collections (external to gnc) run by cron, some more obscure currency pairs are only available a few hours a day, I don't even try if the most recent is less than a day old. Unless it's changed, the only real "backoff" strategy required is to sleep for 16 seconds between requests. Basically, assuming this hasn't changed recently, they allow 5 requests per minute. So by sleeping 16 seconds you ensure that 5th request occurs after 60 seconds is up. It should be pretty easy to implement this delay and still allow full pulls. The downside, of course, is that if you have lots of prices to pull it can take N/4 minutes to get them all, so sure, only refreshing prices > 12 hours old is probably fine, but I suspect this will still be an all-or-nothing exercise in general. Enjoy! Do you really miss the point on purposes? If gnc collects the last price it doesn't have that is a good thing. -- Wm Good evening Mr Bogle. Sleep well. ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: [GNC-dev] Alphavantage strategies
Hi, Wm via gnucash-devel writes: [snip] > Let's consider some simple backoff strategies that gnc could > implement, feel free to add your own. > > Wm's suggestion > === > only collect if the most recent price is more than 1 day (or 12 hours, > etc) old [2]. this sort of fits in with the existing post collection > query, i.e. always store the good prices that have been obtained, > instead ask the user, "do you want to try and fetch the other prices / > rates again". this provides a natural pause to Alphavantage as gnc > won't be hitting it for prices it just got a second time and would > only be requesting the prices it didn't get last time. > > [2] I use this in my own collections (external to gnc) run by cron, > some more obscure currency pairs are only available a few hours a day, > I don't even try if the most recent is less than a day old. Unless it's changed, the only real "backoff" strategy required is to sleep for 16 seconds between requests. Basically, assuming this hasn't changed recently, they allow 5 requests per minute. So by sleeping 16 seconds you ensure that 5th request occurs after 60 seconds is up. It should be pretty easy to implement this delay and still allow full pulls. The downside, of course, is that if you have lots of prices to pull it can take N/4 minutes to get them all, so sure, only refreshing prices > 12 hours old is probably fine, but I suspect this will still be an all-or-nothing exercise in general. Enjoy! -derek -- Derek Atkins 617-623-3745 de...@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
[GNC-dev] Alphavantage strategies
I was unaware [1] until recently that Alphavantage wanted individuals / non-subscribers / poor people / whoever doesn't pay / to back off a bit. I don't have a problem with this, it is a good free service and it has been obvious they're struggling. [1] but not surprised to discover There have been postings in the .user list (one of them being mine) that involve editing files in the F:Q install to mitigate this. Not a good idea as the next update overwrites any changes, users shouldn't have to get involved, may not be reading the list, sleep(15) is arbitrary, etc. Let's consider some simple backoff strategies that gnc could implement, feel free to add your own. Wm's suggestion === only collect if the most recent price is more than 1 day (or 12 hours, etc) old [2]. this sort of fits in with the existing post collection query, i.e. always store the good prices that have been obtained, instead ask the user, "do you want to try and fetch the other prices / rates again". this provides a natural pause to Alphavantage as gnc won't be hitting it for prices it just got a second time and would only be requesting the prices it didn't get last time. [2] I use this in my own collections (external to gnc) run by cron, some more obscure currency pairs are only available a few hours a day, I don't even try if the most recent is less than a day old. -- Wm ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
