Re: handling fingerprints
Hi Jason! > On Nov 13, 2021, at 2:52 PM, JB2Cool wrote: > > I think i understand what i'm meant to be testing but i can't seem to get any > functional difference between using 5.0.4 and 5.0.4-27-gb87cbed53cd6. Well, for the thing that I want you to test, there should be a "day and night" style difference. 5.0.4 certainly doesn't do it (store fingerprints in the cloud). And 5.0.4.27 /should/ be doing it... > Can you spell it out for dummies how to test this for you (I know you've > already written it out but i'm talking farm animal stupid instructions). I > think what's confusing me is your mention of deleting a dive from the > logbook, do you really want the dive computer to have more current info than > the saved dive log? I'm doing this as an entirely separate computer user > using a different cloud storage account so there is no fear/issue with > messing up anything in my everyday log, I'm using Windows desktop application > and Linux AppImage and an OSTC Plus if it makes any difference. So testing this with a separate account is of course "safest". But even without it you should be able to test it. Here's the general idea. Subsurface now saves the dive computer fingerprint of the last download with the dive log (btw, it does that both for XML and cloud storage, but it's of course easier to test the cloud storage part). So first thing we need to do is to create a successful download. That's why I'm suggesting to delete the last dive that you have - that way you will actually download something from the dive computer. After that, when you save the dive data to the cloud, it should show up in the git repo. For those who know how to peek into your local repo, it will be in 00-Subsurface as a line like this fingerprint model=ac48a543 serial=042c1ef9 deviceid= diveid=d85e8de4 data="84782d49" If you don't get a line like this, I want to know about that :) Now sync that to the cloud server and go to a different computer or device. Sync there as well. When you now download from that same dive computer, it should almost immediately tell you that there are no new dives - because it will receive that fingerprint and find it and not try to download things from the dive computer. This is most visible with a G2 (which without fingerprint downloads EVERYTHING -- and with BLE that easily takes 15+ minutes... vs. responding in a few seconds), but I have noticed a significant difference with several other dive computers that I tested with (but haven't tried my OSTC... reading through the sources there the difference may be almost invisible as the fingerprint isn't used until after the data was downloaded - so the fingerprint test is only marginally faster than the second test we do (which is the manual check if we already have the dive once it was parsed). > Sorry to be a pest You are, as always, extremely helpful and I am very grateful that you are trying to test this. Because there are so many different scenarios here, and because there are so many different dive computers who all handle fingerprints somewhat differently, it's really hard to write good instructions how to test this. The whole point is that this is supposed to be invisible to the user - so if you can download and nothing breaks, I guess that's already good news :) /D ___ subsurface mailing list subsurface@subsurface-divelog.org http://lists.subsurface-divelog.org/cgi-bin/mailman/listinfo/subsurface
Re: handling fingerprints
I think i understand what i'm meant to be testing but i can't seem to get any functional difference between using 5.0.4 and 5.0.4-27-gb87cbed53cd6. Can you spell it out for dummies how to test this for you (I know you've already written it out but i'm talking farm animal stupid instructions). I think what's confusing me is your mention of deleting a dive from the logbook, do you really want the dive computer to have more current info than the saved dive log? I'm doing this as an entirely separate computer user using a different cloud storage account so there is no fear/issue with messing up anything in my everyday log, I'm using Windows desktop application and Linux AppImage and an OSTC Plus if it makes any difference. Sorry to be a pest Jason On Sat, 13 Nov 2021 at 00:19, Dirk Hohndel via subsurface < subsurface@subsurface-divelog.org> wrote: > To avoid confusion: > > you'll need Subsurface-mobile 3.4.1(or newer) and Subsurface 5.0.4.25 (or > newer) to test this... > > /D > > > On Nov 12, 2021, at 3:46 PM, Dirk Hohndel via subsurface < > subsurface@subsurface-divelog.org> wrote: > > > > My code to share fingerprints via the cloud has been merged into master. > > Android test binaries with that new code are pending review to be pushed > into the beta channel. > > I'll work on iOS test binaries in a moment. > > > > It would be extremely helpful if this could get some testing. > > This new feature is extremely useful for Scubapro G2 users (download via > cable on your PC and when you next attempt to download via BLE on your > phone the right thing happens). > > The G2 is certainly the most egregious case - but I'm sure this will > help with some others that I'm not aware of where having the correct > fingerprint also makes a difference in download duration on mobile. > > > > In order to test this you need a dive computer that you can download > from on two devices - doesn't matter if its mobile or desktop. > > Test scenario: force a download on one computer/device (either delete > the last dive or force download all dives). Sync with cloud, sync with > cloud on the other computer/device. > > a fresh download should successfully use the fingerprint and immediately > tell you that there are no new dives. > > > > /D > > > >> On Oct 23, 2021, at 5:12 PM, Dirk Hohndel wrote: > >> > >> > >> Hey Linus, > >> > >> The more I think about our current fingerprint handling (I think that's > our third attempt), the more I think we are still not doing this right. > >> > >> Right now we consider fingerprint files specific to a host (so the > specific system - PC, laptop, phone, tablet) that Subsurface runs on. > >> So let's say I have a PC and a phone. I use the cloud to store my dive > data. And I also want to download my dives for a research project into an > XML file. > >> > >> (1) on my PC I download from my dive computer and store the dives in > the cloud > >> -> there is a fingerprint file in the PC file system and the dives are > in the cloud > >> (2) after a couple more dives, I download again > >> -> Subsurface finds the fingerprint file and only downloads the new > dives; all good, fingerprint file on the PC and cloud dives get updated > >> (3) close the cloud storage, open a local .xml file; download again > >> -> fingerprint file is found, but the matching dive is not in the dive > list, so fingerprint gets ignored (but why are we even looking at it???) > >> -> dives are downloaded, fingerprint is written again > >> (4) more diving, this time I only brought the phone > >> -> no fingerprint file on the phone, all dives are downloaded > >> -> new dives go in the cloud, fingerprint file on the phone > >> (5) no additional diving, back on the PC > >> -> those last dives would attempt to download again (the fingerprint > is outdated), but after the first one downloads we'd recognize that we > already have it and give up -- without updating the fingerprint file on the > PC > >> (6) I buy a new phone, connect to the cloud, then try downloading from > the dive computer > >> -> even though all the dives are in the cloud, because there is no > fingerprint file, we download from the dive computer > >> -> because no new dives were downloaded, no fingerprint file is > written - so we do this every time until the next time we actually have a > new dive that's downloaded on that phone > >> > >> (3), (4), (5), and (6) all indicate that what we are doing isn't really > right > >> The fingerprint information really is tightly related to the dive log > file. That's why we have this check in (3) to prevent us from ignoring the > dives on the dive computer if the fingerprinted on isn't in the dive list. > >> I can actually turn (4) and especially (6) into a really annoying > situation with a dive computers like the G2 where we keep downloading all > dive data if we don't have a fingerprint (there is no "give me the newest > dive" function - if you don't have a fingerprint you download everything) > -- and of course
Re: handling fingerprints
To avoid confusion: you'll need Subsurface-mobile 3.4.1(or newer) and Subsurface 5.0.4.25 (or newer) to test this... /D > On Nov 12, 2021, at 3:46 PM, Dirk Hohndel via subsurface > wrote: > > My code to share fingerprints via the cloud has been merged into master. > Android test binaries with that new code are pending review to be pushed into > the beta channel. > I'll work on iOS test binaries in a moment. > > It would be extremely helpful if this could get some testing. > This new feature is extremely useful for Scubapro G2 users (download via > cable on your PC and when you next attempt to download via BLE on your phone > the right thing happens). > The G2 is certainly the most egregious case - but I'm sure this will help > with some others that I'm not aware of where having the correct fingerprint > also makes a difference in download duration on mobile. > > In order to test this you need a dive computer that you can download from on > two devices - doesn't matter if its mobile or desktop. > Test scenario: force a download on one computer/device (either delete the > last dive or force download all dives). Sync with cloud, sync with cloud on > the other computer/device. > a fresh download should successfully use the fingerprint and immediately tell > you that there are no new dives. > > /D > >> On Oct 23, 2021, at 5:12 PM, Dirk Hohndel wrote: >> >> >> Hey Linus, >> >> The more I think about our current fingerprint handling (I think that's our >> third attempt), the more I think we are still not doing this right. >> >> Right now we consider fingerprint files specific to a host (so the specific >> system - PC, laptop, phone, tablet) that Subsurface runs on. >> So let's say I have a PC and a phone. I use the cloud to store my dive data. >> And I also want to download my dives for a research project into an XML file. >> >> (1) on my PC I download from my dive computer and store the dives in the >> cloud >> -> there is a fingerprint file in the PC file system and the dives are in >> the cloud >> (2) after a couple more dives, I download again >> -> Subsurface finds the fingerprint file and only downloads the new dives; >> all good, fingerprint file on the PC and cloud dives get updated >> (3) close the cloud storage, open a local .xml file; download again >> -> fingerprint file is found, but the matching dive is not in the dive >> list, so fingerprint gets ignored (but why are we even looking at it???) >> -> dives are downloaded, fingerprint is written again >> (4) more diving, this time I only brought the phone >> -> no fingerprint file on the phone, all dives are downloaded >> -> new dives go in the cloud, fingerprint file on the phone >> (5) no additional diving, back on the PC >> -> those last dives would attempt to download again (the fingerprint is >> outdated), but after the first one downloads we'd recognize that we already >> have it and give up -- without updating the fingerprint file on the PC >> (6) I buy a new phone, connect to the cloud, then try downloading from the >> dive computer >> -> even though all the dives are in the cloud, because there is no >> fingerprint file, we download from the dive computer >> -> because no new dives were downloaded, no fingerprint file is written - >> so we do this every time until the next time we actually have a new dive >> that's downloaded on that phone >> >> (3), (4), (5), and (6) all indicate that what we are doing isn't really right >> The fingerprint information really is tightly related to the dive log file. >> That's why we have this check in (3) to prevent us from ignoring the dives >> on the dive computer if the fingerprinted on isn't in the dive list. >> I can actually turn (4) and especially (6) into a really annoying situation >> with a dive computers like the G2 where we keep downloading all dive data if >> we don't have a fingerprint (there is no "give me the newest dive" function >> - if you don't have a fingerprint you download everything) -- and of course >> all this is especially frustrating when this is a BLE download that takes >> for ever and is super flaky. >> >> I am back to thinking that we should consider the fingerprint a property of >> a specific dive log. So both an XML file and the git storage should remember >> the last fingerprint for each dive computer from which we did a direct >> download. And we should index that exactly like we name these files in the >> filesystem right now - model.devinfo-serial - and then have the fingerprint >> data be a hex encoded byte stream of fingerprint-deviceid-diveid. >> >> Sadly, this still causes pointless re-download orgies when a firmware change >> or some other change messes with the devinfo-serial number which we have to >> use as lookup parameter - but it seems like it would get more cases right >> than what we have today. >> >> >> Of course, this all is triggered by the plight of specifically G2 users. And >> the
Re: handling fingerprints
My code to share fingerprints via the cloud has been merged into master. Android test binaries with that new code are pending review to be pushed into the beta channel. I'll work on iOS test binaries in a moment. It would be extremely helpful if this could get some testing. This new feature is extremely useful for Scubapro G2 users (download via cable on your PC and when you next attempt to download via BLE on your phone the right thing happens). The G2 is certainly the most egregious case - but I'm sure this will help with some others that I'm not aware of where having the correct fingerprint also makes a difference in download duration on mobile. In order to test this you need a dive computer that you can download from on two devices - doesn't matter if its mobile or desktop. Test scenario: force a download on one computer/device (either delete the last dive or force download all dives). Sync with cloud, sync with cloud on the other computer/device. a fresh download should successfully use the fingerprint and immediately tell you that there are no new dives. /D > On Oct 23, 2021, at 5:12 PM, Dirk Hohndel wrote: > > > Hey Linus, > > The more I think about our current fingerprint handling (I think that's our > third attempt), the more I think we are still not doing this right. > > Right now we consider fingerprint files specific to a host (so the specific > system - PC, laptop, phone, tablet) that Subsurface runs on. > So let's say I have a PC and a phone. I use the cloud to store my dive data. > And I also want to download my dives for a research project into an XML file. > > (1) on my PC I download from my dive computer and store the dives in the cloud > -> there is a fingerprint file in the PC file system and the dives are in > the cloud > (2) after a couple more dives, I download again > -> Subsurface finds the fingerprint file and only downloads the new dives; > all good, fingerprint file on the PC and cloud dives get updated > (3) close the cloud storage, open a local .xml file; download again > -> fingerprint file is found, but the matching dive is not in the dive > list, so fingerprint gets ignored (but why are we even looking at it???) > -> dives are downloaded, fingerprint is written again > (4) more diving, this time I only brought the phone > -> no fingerprint file on the phone, all dives are downloaded > -> new dives go in the cloud, fingerprint file on the phone > (5) no additional diving, back on the PC > -> those last dives would attempt to download again (the fingerprint is > outdated), but after the first one downloads we'd recognize that we already > have it and give up -- without updating the fingerprint file on the PC > (6) I buy a new phone, connect to the cloud, then try downloading from the > dive computer > -> even though all the dives are in the cloud, because there is no > fingerprint file, we download from the dive computer > -> because no new dives were downloaded, no fingerprint file is written - > so we do this every time until the next time we actually have a new dive > that's downloaded on that phone > > (3), (4), (5), and (6) all indicate that what we are doing isn't really right > The fingerprint information really is tightly related to the dive log file. > That's why we have this check in (3) to prevent us from ignoring the dives on > the dive computer if the fingerprinted on isn't in the dive list. > I can actually turn (4) and especially (6) into a really annoying situation > with a dive computers like the G2 where we keep downloading all dive data if > we don't have a fingerprint (there is no "give me the newest dive" function - > if you don't have a fingerprint you download everything) -- and of course all > this is especially frustrating when this is a BLE download that takes for > ever and is super flaky. > > I am back to thinking that we should consider the fingerprint a property of a > specific dive log. So both an XML file and the git storage should remember > the last fingerprint for each dive computer from which we did a direct > download. And we should index that exactly like we name these files in the > filesystem right now - model.devinfo-serial - and then have the fingerprint > data be a hex encoded byte stream of fingerprint-deviceid-diveid. > > Sadly, this still causes pointless re-download orgies when a firmware change > or some other change messes with the devinfo-serial number which we have to > use as lookup parameter - but it seems like it would get more cases right > than what we have today. > > > Of course, this all is triggered by the plight of specifically G2 users. And > the annoying fact that even if they do the full download on a PC via cable, > they still have to get a full download to work on their phone via BLE... > > I'd much rather come up with a way to not rely on the devinfo-serial number, > but at least on the G2 that one looks like it should be stable (there's
Re: handling fingerprints
On 25/10/2021 19:25, Linus Torvalds wrote: On Mon, Oct 25, 2021 at 10:13 AM Jef Driesen wrote: What's the reason for ignoring the fingerprint if you don't have the corresponding dive anymore? Simple: you successfully downloaded all the dives, but had some problems, and never saved them. Maybe just a "oops, screwed up editing" and exit without saving. Or maybe a crash, whatever. So then you restart subsurface, and need to re-download. If you don't actually _have_ the dive the fingerprint points to, then you should re-download it, and not ignore it "becasue I already downloaded it". That makes sense, but it's also a case that would disappear if the fingerprint is saved together with the dives. Jef ___ subsurface mailing list subsurface@subsurface-divelog.org http://lists.subsurface-divelog.org/cgi-bin/mailman/listinfo/subsurface
Re: handling fingerprints
On Mon, Oct 25, 2021 at 10:13 AM Jef Driesen wrote: > > What's the reason for ignoring the fingerprint if you don't have the > corresponding dive anymore? Simple: you successfully downloaded all the dives, but had some problems, and never saved them. Maybe just a "oops, screwed up editing" and exit without saving. Or maybe a crash, whatever. So then you restart subsurface, and need to re-download. If you don't actually _have_ the dive the fingerprint points to, then you should re-download it, and not ignore it "becasue I already downloaded it". Linus ___ subsurface mailing list subsurface@subsurface-divelog.org http://lists.subsurface-divelog.org/cgi-bin/mailman/listinfo/subsurface
Re: handling fingerprints
On 24/10/2021 02:12, Dirk Hohndel via subsurface wrote: (3) close the cloud storage, open a local .xml file; download again -> fingerprint file is found, but the matching dive is not in the dive list, so fingerprint gets ignored (but why are we even looking at it???) -> dives are downloaded, fingerprint is written again What's the reason for ignoring the fingerprint if you don't have the corresponding dive anymore? Assume you download your dives, and for some reason you don't want to keep the most recent dive in your logbook. Now, according to the above logic, the next time you download, you ignore the fingerprint because the dive isn't in the logbook and thus you'll re-download those old dives again, including the unwanted dive. For the dives that were already in your logbook, that's just inefficient, but they won't get added twice (I believe subsurface has some logic to prevent that). But that unwanted dive will re-appear, and you'll have to manually remove it again. As a user, I would expect that the download resumes at the point where it stopped last time, regardless of whether I kept the previously downloaded dives or not. If I really want to re-download previously download dives, I can use the "Force download all dives" option, which disables the fingerprint logic. Some example scenarios where you don't want to keep every dive recorded by your dive computer: If you have a watch style dive computer that you also uses as a watch, the watch may record activity in the pool as dives. Or someone doing both scuba and freedives may want to keep two separate logs. I am back to thinking that we should consider the fingerprint a property of a specific dive log. So both an XML file and the git storage should remember the last fingerprint for each dive computer from which we did a direct download. And we should index that exactly like we name these files in the filesystem right now - model.devinfo-serial - and then have the fingerprint data be a hex encoded byte stream of fingerprint-deviceid-diveid. Sadly, this still causes pointless re-download orgies when a firmware change or some other change messes with the devinfo-serial number which we have to use as lookup parameter - but it seems like it would get more cases right than what we have today. Of course, this all is triggered by the plight of specifically G2 users. And the annoying fact that even if they do the full download on a PC via cable, they still have to get a full download to work on their phone via BLE... I'd much rather come up with a way to not rely on the devinfo-serial number, but at least on the G2 that one looks like it should be stable (there's no way to get the firmware version on the G2, it seems). Why would a change in the firmware version cause a re-download? The serial number in the devinfo event is completely independent from the firmware version. Unless the device suddenly starts to report a different number, the serial number will not change after a firmware update. So it's as stable as it can be. The devinfo serial number may indeed not match exactly with the serial number as printed on the device (due to some additional formatting which can't be represented by a simple 32bit number). But that's another story. The primary purpose of the serial number in the devinfo event (and the devinfo event itself) is to support the fingerprint feature: provide a unique ID to identify an individual device. For that purpose, it doesn't matter it doesn't exactly match the human readable serial number. It that sense it's like more an opaque ID, just like the fingerprint itself. Of course, where possible we try to decode the serial number correctly. The reason why it's a 32bit number is historic. At the time this feature was added (more than 10 years ago), representing serial numbers as a 32bit integer was an obvious choice, because that's how dive computers stored it. Since then, there are many new dive computers that do things a bit different... I wonder if I am missing something in my model here... is there a scenario where storing the fingerprint with the dive log file creates incorrect behavior? At least for the important case (dive computer that supports both wired and BLE transfer - do the fast wired transfer on the PC and have the correct fingerprint for the next BLE transfer both locally AND and on the phone) this seems like it would be an improvement... One scenario where you have to be careful is when the download failed. Because dives are downloaded in reverse order, you get the fingerprint of the most recent dives immediately with the first dive. But if you do update the fingerprint when some of the older dives failed to download, then you won't be able to retry and download those older dives without disabling the fingerprint again. So you should only update the fingerprint after a successful download, and deal with the duplicates after retrying. Jef
