[fpc-pascal] Documentation patch for fcl/netdb
I've added a documentation patch to bug 0037906 in the bug tracker. This covers the new code that I added to fcl/netdb for DNS resolution as well most of the other functions and procedures in the unit. The patch also adds examples that showcase the new functions. I've tested these on Fedora 32 only. https://bugs.freepascal.org/view.php?id=37906#c130418 ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Trouble integrating examples into fpcdoc
On 5/04/21 18:53, Michael Van Canneyt via fpc-pascal wrote: On Mon, 5 Apr 2021, Noel Duffy via fpc-pascal wrote: In the fpcdocs subversion project, there are a number of example directories each containing one or more small programs and a Makefile. My question is, is that Makefile automatically generated, or is it hand-crafted? It is hand-crafted. Great, thanks. That makes the way ahead quite straightforward. I'm writing documentation for the netdb module and want to add some example programs. I've created the first program and used an example tag to reference it within netdb.xml, and this works to show the example code in the generated html. But the example program isn't compiled. None of the examples are compiled when you make the documentation. Yes, I realised after I'd posted it that I should have said when running "make examples" or "make linuxexamples". Anyway, I have enough information now to start adding example programs. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
[fpc-pascal] Trouble integrating examples into fpcdoc
In the fpcdocs subversion project, there are a number of example directories each containing one or more small programs and a Makefile. My question is, is that Makefile automatically generated, or is it hand-crafted? I'm writing documentation for the netdb module and want to add some example programs. I've created the first program and used an example tag to reference it within netdb.xml, and this works to show the example code in the generated html. But the example program isn't compiled. Taking the fpcdocs/linuxex example directory, the Makefile references the single defined example, ex64.pp, at line 54: OBJECTS=ex64 This line occurs after this warning: ### # No need to edit after this line. ### Which suggests to me that the Makefiles are generated. But if so, how? I can't find anything in the Makefile.fpc at the fpcdocs root to reference them. On the other hand, the Makefiles are committed to the repository, which does somewhat suggest that they're not generated. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Patches for Issue #37906 - DNS over TCP
On 30/01/21 11:43 pm, Michael Van Canneyt via fpc-pascal wrote: On Sat, 30 Jan 2021, Noel Duffy via fpc-pascal wrote: Lastly, a minor point: in the source for netdb.pp there's a comment warning of stringfromlabel's lack of checks. Since it now has a good few checks, I think this warning is superfluous. But I didn't want to remove it without checking, in case there's more to it than meets the eye. I also noticed this, but decided to leave it for the moment. if you feel it can be removed, I'll trust your judgment on that. My vote is to remove the comment. While it may have been justified in the past, the current version of stringfromlabel is reasonably robust and resilient to the most obvious kinds of attack, in my opinion. Now, I'm not a security researcher and haven't done any serious fuzzing attacks against it, so all I can say is that stringfromlabel is about as safe as the rest of the code in netdb.pp. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Patches for Issue #37906 - DNS over TCP
On 30/01/21 9:45 pm, Michael Van Canneyt via fpc-pascal wrote: On Sat, 30 Jan 2021, Noel Duffy via fpc-pascal wrote: I've added patches to resolve issue #37906, DNS over TCP, to the bug tracker: https://bugs.freepascal.org/view.php?id=37906 I've broken the changes into three patches: 1. netdb.patch, which updates packages/fcl-net/src/netdb.pp, adding support for DNS over TCP 2. netdb-example.patch, which adds an example program to packages/fcl-net/examples and adds it to packages/fcl-net/examples/Makefile.fpc 3. netdb-tests.patch, which adds unit tests to packages/fcl-net/tests and updates tests/Makefile.fpc to add the new tests to the build system I've created separate patches to simplify applying them. If there's a problem with one, the others should still be OK. In addition to new code for DNS over TCP, this latest set of patches adds checks for writes beyond buffer boundaries and other potential security issues. Since the last update I've also added support for SRV records, as defined in RFC 2782. Very nice indeed !! I checked your patches, they're all applied. Great stuff! Thanks Michael. As always, I wonder if you ever sleep! :-D Thank you very much for this contribution ! Happy to do so. With regard to documentation, I've not written anything yet, and it seems I'm as well placed as anyone to do it. That said, I'm completely unfamiliar with generating documentation from source in FPC. But there's just one way to learn, I suppose! Lastly, a minor point: in the source for netdb.pp there's a comment warning of stringfromlabel's lack of checks. Since it now has a good few checks, I think this warning is superfluous. But I didn't want to remove it without checking, in case there's more to it than meets the eye. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
[fpc-pascal] Patches for Issue #37906 - DNS over TCP
I've added patches to resolve issue #37906, DNS over TCP, to the bug tracker: https://bugs.freepascal.org/view.php?id=37906 I've broken the changes into three patches: 1. netdb.patch, which updates packages/fcl-net/src/netdb.pp, adding support for DNS over TCP 2. netdb-example.patch, which adds an example program to packages/fcl-net/examples and adds it to packages/fcl-net/examples/Makefile.fpc 3. netdb-tests.patch, which adds unit tests to packages/fcl-net/tests and updates tests/Makefile.fpc to add the new tests to the build system I've created separate patches to simplify applying them. If there's a problem with one, the others should still be OK. In addition to new code for DNS over TCP, this latest set of patches adds checks for writes beyond buffer boundaries and other potential security issues. Since the last update I've also added support for SRV records, as defined in RFC 2782. Cheers, Noel. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Unit testing for RTL packages
On 24/01/21 11:53 pm, Sven Barth via fpc-pascal wrote: Am 24.01.2021 um 08:37 schrieb Noel Duffy via fpc-pascal: How should one go about adding unit tests to a package in the RTL? In many packages, for instance fcl-db, there is a tests folder with fpcunit units and programs in them, and they're referenced in fpmake.pp as example programs, but these do not appear to be built or run during a standard build of the RTL. Separately, there are tests for some packages in the top-level tests/test folder, which do appear to be run with make full TEST_FPC=path_to_your_compiler None of these latter tests use fpcunit. Do any of the fpcunit tests get run as part of a test cycle? Historically the tests for packages reside in tests/test/packages/. If you need to add a new package you need to add it to the TESTPACKAGESDIRS variable in tests/Makefile.fpc and regenerate the Makefile.(*) Two, three years ago I also added the ability to have the tests directly in packages//tests (it started with the Rtti unit). If you add a new package this way you need to add it to the TESTPACKAGESDIRECTDIRS variable in tests/Makefile.fpc and regenerate the Makefile.(*) In *any* case you have to keep in mind the following: - for a test to be picked up by the testsuite they need to start with a "t" and end in ".pp" (this means that you should prefix units that you don't want the testsuite to compile independantly with a "u" instead of a "t" or suffix them with ".pas" instead of ".pp") - if it's a program it should return exit code 0 on success and 1 on error (libraries and units are not executed, they are only checkey for correct compilation) - all rtl/packages units are available (which means that you can use fcl-fpcunit for example, though you should use the "consoletestrunner" (see also packages/rtl-objpas/tests/testrunner.rtlobjpas.pp)) (*) invoke trunks' fpcmake utility (located in utils/fpcm/bin/- after a complete build) with the parameter "-Tall" inside the tests directory Thanks for the detailed reply. I will try this out later today. I'm not actually adding a new package, just making changes to fcl-net, and my tests already use consoletestrunner, so it should be straightforward. (I know, famous last words!) ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
[fpc-pascal] Unit testing for RTL packages
How should one go about adding unit tests to a package in the RTL? In many packages, for instance fcl-db, there is a tests folder with fpcunit units and programs in them, and they're referenced in fpmake.pp as example programs, but these do not appear to be built or run during a standard build of the RTL. Separately, there are tests for some packages in the top-level tests/test folder, which do appear to be run with make full TEST_FPC=path_to_your_compiler None of these latter tests use fpcunit. Do any of the fpcunit tests get run as part of a test cycle? I've written a bunch of tests, and now want to know where to put them in order to generate a patch. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] FPDoc now with Markdown support
On 3/01/21 3:31 am, Michael Van Canneyt via fpc-pascal wrote: Hello ! I didn't make it quite in time for the new year, but still: The fpdoc engine (what is used to document the FPC & Lazarus units) is now capable of outputting the documentation in markdown. This can be used as input for mkdocs or another engine such as sphinx. As a first engine, I tackled mkdocs. Great work! Thought I recognised some of these themes, then saw readthedocs in the list of themes. Now we can start a long bitter bikeshedding thread about which theme to use for the official docs ;-) [snipped] I've been looking at allowing markdown for the description files (they would be less verbose then), but there seems to be no decent markdown parser available for pascal. If somone cares to contribute one... I remember needing something like Markdown about a year or two ago, but gave up after reading the spec and seeing how vague and hand-wavey it was. I ended up going with a custom format instead. Not much help to you, I'm afraid. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] netdb, TCP, and DNS queries
On 7/12/20 11:29 am, Michael Van Canneyt via fpc-pascal wrote:
On Mon, 7 Dec 2020, Noel Duffy via fpc-pascal wrote:
All of these are in the new version as well. Perhaps I need to redo the changes
against trunk to avoid confusion.
Yes, please.
Line 55 contains:
{$if defined(android)}
{$define FPC_USE_LIBC}
{$endif}
OK, I've added a patch to the bug tracker.
https://bugs.freepascal.org/view.php?id=37906#c127584
This is against trunk.
I'm still working on tests.
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] netdb, TCP, and DNS queries
On 7/12/20 12:41 am, Michael Van Canneyt via fpc-pascal wrote:
On Sun, 6 Dec 2020, Noel Duffy via fpc-pascal wrote:
Hi all,
Following up on the work I've being doing to add support for DNS queries over
TCP to netdb and to add support for querying all of the standard resource types
such as MX, SOA, TXT, NS, etc, I've added the proposed changes to a note in the
Bugtracker:
https://bugs.freepascal.org/view.php?id=37906#c127368
This isn't yet in patch form. I've just created a copy of netdb.pp and renamed
it to newnetdb.pp. I'm posting about it now to get some feedback on it.
In general it looks good. 2 remarks:
I did see you removed the comments about byte order, I don't think that's a
good idea, these comments have their use :-)
Which comments are you referring to? See my comments below about using 3.2.0.
The second point is that you removed the USE_LIBC define.
The netdb unit works only with USE_LIBC on Android.
I've gone back and done a diff with the original netdb.pp and I'm not seeing
any mention of libc. I based my version on the one that shipped with 3.2.0, not
trunk. 3.2.0 has these defines:
{$DEFINE ETC_BY_ENV}
{$DEFINE SFN_VERSION}
{$DEFINE ETC_BY_ENV}
{$DEFINE SFN_VERSION}
{$DEFINE ETC_BY_ENV}
{$DEFINE SFN_VERSION}
{$DEFINE UNIX_ETC}
All of these are in the new version as well. Perhaps I need to redo the changes
against trunk to avoid confusion.
So it is imperative that this keeps working too. Maybe with a reduced API,
but it should work nonetheless.
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
[fpc-pascal] netdb, TCP, and DNS queries
Hi all, Following up on the work I've being doing to add support for DNS queries over TCP to netdb and to add support for querying all of the standard resource types such as MX, SOA, TXT, NS, etc, I've added the proposed changes to a note in the Bugtracker: https://bugs.freepascal.org/view.php?id=37906#c127368 This isn't yet in patch form. I've just created a copy of netdb.pp and renamed it to newnetdb.pp. I'm posting about it now to get some feedback on it. The second file attached is a standalone Pascal project named dnsq.lpr that exercises the dns functions. It works just like the Unix program "dig". You give it a DNS resource type, one of A, , CNAME, PTR, NS, SOA, MX, or TXT, and a domain name. Some example usage: ./dnsq txt freepascal.org - freepascal.org (TXT) Answers: 1 NS: 0 Additional: 0 Length: 87 AnsStart: 20 -- Answers freepascal.org 43200 TXT v=spf1 mx a:lists.freepascal.org a:mail.freepascal.org $ ./dnsq ns freepascal.org - freepascal.org (NS) Answers: 3 NS: 0 Additional: 5 Length: 213 AnsStart: 20 -- Answers freepascal.org 170198 NS dns1.easydns.com freepascal.org 170198 NS dns3.easydns.ca freepascal.org 170198 NS dns2.easydns.net -- Additional Section dns2.easydns.net 502 A 198.41.222.254 dns2.easydns.net 468 2400:CB00:2049:0001::C629:DEFE dns1.easydns.com 174 A 64.68.192.10 dns1.easydns.com 83 2400:CB00:2049:0001::A29F:1835 dns3.easydns.ca 6 A 64.68.196.10 $ ./dnsq soa freepascal.org - freepascal.org (SOA) Answers: 1 NS: 0 Additional: 0 Length: 77 AnsStart: 20 -- Answers freepascal.org 167820 SOA dns1.easydns.com zone.easydns.com, serial:1598197381, refresh:3600, retry:600, min:300, expire:604800 The code for dnsq is quite short and straightforward, imho. My hope is that, with a bit of polish, it may be suitable for addition to the set of sample projects included with fpc. In addition to showing queries for different resource types, it shows how to detect truncation and fall back to TCP. I'm still working on tests for the code. Testing network code isn't easy to do, so I'm having to figure it out as I go. As a general rule I've tried to minimize changes to existing code and if changes are necessary, to add new types rather than change old ones. Where I've added record types, I've tried to compose from existing record types so things are still defined just once. For instance, where I needed a TQueryData equivalent with a length field, I did it like this: TQueryDataLength = packed record length: Word; hpl: TQueryData; end; I also created a new RR record type. I needed a resource record that preserves the RR name, because many queries return A and records in the additional section and it's impossible to use these without the name. For instance, see the NS query example I showed above, where five A and records get returned in the additional section. Using the API is straightforward enough. You call DnsLookup with a domain name and a resource type. If that succeeds, you call GetRRRecords to get a dynamic array of resource records. To get the actual RRData, there are functions for each resource type. E.g, DNSRRGetText, DNSRRGetMX, and so on. OK, that's probably enough for now. I'll be happy to go into more details or answer any questions about this code. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] netdb, DNS and TCP
On 11/10/20 9:42 pm, Michael Van Canneyt via fpc-pascal wrote: On Sun, 11 Oct 2020, Noel Duffy via fpc-pascal wrote: To avoid that duplication, the code would need to use a dynamic buffer and fill it just before socket writing. The TCP variant would include its length field in the buffer. Additionally, after reading the response from the socket, the buffer contents would have to be copied into a record data structure. It's a tradeoff between execution efficiency and simplicity. It always is. But I am inclined to think that DNS queries benefit more from execution efficiency, hence the preference for avoiding the copying of buffers as much as possible. One could of course argue that with TCP, the speed suffers in each case. Therein lies the rub. Once your code reads or writes to the network, all bets are off with regard to speed. If you have suggestions for a better implementation, I am all ears... Well, better depends on what you're optimizing for, of course. In many places the DRY principle is considered a Holy Commandment graven on stone tablets and breaking it tantamount to heresy. If the community thinks breaking DRY is acceptable to preserve speed in this case then I'm willing to accept that and run with it. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] netdb, DNS and TCP
On 11/10/20 9:42 pm, Michael Van Canneyt via fpc-pascal wrote: On Sun, 11 Oct 2020, Noel Duffy via fpc-pascal wrote: To avoid that duplication, the code would need to use a dynamic buffer and fill it just before socket writing. The TCP variant would include its length field in the buffer. Additionally, after reading the response from the socket, the buffer contents would have to be copied into a record data structure. It's a tradeoff between execution efficiency and simplicity. It always is. But I am inclined to think that DNS queries benefit more from execution efficiency, hence the preference for avoiding the copying of buffers as much as possible. One could of course argue that with TCP, the speed suffers in each case. Therein lies the rub. Once your code reads or writes to the network, all bets are off with regard to speed. If you have suggestions for a better implementation, I am all ears... Well, better depends on what you're optimizing for, of course. In many places the DRY principle is considered a Holy Commandment graven on stone tablets and breaking it tantamount to heresy. If the community thinks breaking DRY is acceptable to preserve speed in this case then I'm willing to accept that and run with it. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
[fpc-pascal] netdb, DNS and TCP
Hi all, To keep track of the work I've been doing on DNS in netdb, I created bug #37906 in Mantis. https://bugs.freepascal.org/view.php?id=37906 I've attached to a note on that bug report a proof-of-concept version of netdb.pp and a small program that uses it to make text queries over TCP. This version of netdb has different record types for TCP and UDP (technically, it has three, one for UDP queries and replies, one for TCP queries, one for TCP replies). The wire protocol for DNS over TCP has a mandatory two-octet length field at the start for both queries and replies, and I didn't want to have to use a full-length 64k buffer for both when a smaller 512 byte buffer suffices. Overall I'm not really enthused with this implementation. Having separate record types for the different payload types means having separate versions of a number of helper functions as well. Functions like BuildPayload, NextRR, and SkipAnsQueries. To avoid that duplication, the code would need to use a dynamic buffer and fill it just before socket writing. The TCP variant would include its length field in the buffer. Additionally, after reading the response from the socket, the buffer contents would have to be copied into a record data structure. It's a tradeoff between execution efficiency and simplicity. On a related point, does anyone have any tips for testing low-level network functions? I'm not sure if there's an easy way, perhaps with add-on test libraries, to simulate things like network errors, packet loss, and the like. I'm currently thinking the only way to do it would be to create mocked versions of the fpsend and fprecv calls. I know about tc netem on linux, but something that can be used inside unit tests would be ideal. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Buffer size for TCP DNS queries in netdb
On 26/09/20 9:07 pm, Michael Van Canneyt via fpc-pascal wrote: On Sat, 26 Sep 2020, Noel Duffy via fpc-pascal wrote: To restate the question, is 64k too much overhead for DNS queries? That overhead would be present on all queries, regardless of whether UDP or TCP was used. I would then prefer to allocate a dynamic buffer. A simple "Array of Byte" can easily be managed with SetLength() and it is automatically disposed of when not needed. An unintended consequence of attempting to replace the TPayLoad static array with a dynamic one is that the TQueryData record can no longer be passed to fpSendTo as a stream of octets by passing a pointer to it. I.e, Function Query(Resolver : Integer; Var Qry,Ans : TQueryData; QryLen : Integer; Var AnsLen : Integer) : Boolean; begin [...] fpsendto(sock,@qry,qrylen+12,0,@SA,SizeOf(SA)); When TPayLoad is made dynamic, the final member of TQueryData changes from an array of octets to a pointer. Now passing a pointer to Qry results in the pointer to the dynamic array being serialized instead of the query payload. Naturally the DNS server will turn up its nose at that and return a sternly-worded error code in protest. The payload data will have to be manually serialized to octets, I think. At least I don't know of any way to serialize a record that contains pointers using any feature of the language or api in the RTL. Doing manual serialization will also require more memory, of course. The quantities aren't huge, though, so I don't think it matters hugely. But more experienced FPC developers may be able to recommend a better way, of course! ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Buffer size for TCP DNS queries in netdb
On 26/09/20 9:07 pm, Michael Van Canneyt via fpc-pascal wrote: On Sat, 26 Sep 2020, Noel Duffy via fpc-pascal wrote: To restate the question, is 64k too much overhead for DNS queries? That overhead would be present on all queries, regardless of whether UDP or TCP was used. I would then prefer to allocate a dynamic buffer. A simple "Array of Byte" can easily be managed with SetLength() and it is automatically disposed of when not needed. Fair enough. Dynamic it is. Thanks Michael. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
[fpc-pascal] Buffer size for TCP DNS queries in netdb
Hi all, I've spent some time this past couple of weeks reading through the DNS resolver code in netdb and poring over RFCs that specify the protocol with a view to adding support for additional resource record queries. Things like TXT, SOA, and so forth. I'm using netdb from fpc 3.2.0 as the starting point. That code only supports querying DNS over UDP. Because TXT records can be up to 64k in size, they frequently get too big to fit inside a UDP packet. The RFCs mandate that resolvers must fall back to TCP in that case. That's the part I'm currently looking at. The maximum size for a DNS response over TCP is 65,535 octets. The netdb code defines a type TPayload that's a 512 octet buffer for receiving DNS responses. That's the largest response possible over UDP. The simplest path forward for me is to extend that buffer from 512 octets to 65,535. While that increase in size would hardly register on most machines, I wonder whether it might be too much on some of the more memory-restricted platforms that FPC supports. DOS, for instance. I also see IFDEFs for Android in this code. The alternative is to use different types for TCP queries, or to have the buffer allocated dynamically. To restate the question, is 64k too much overhead for DNS queries? That overhead would be present on all queries, regardless of whether UDP or TCP was used. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] netdb and DNS queries for MX, TXT, and other resource types
On 16/09/20 8:43 pm, Michael Van Canneyt via fpc-pascal wrote: On Wed, 16 Sep 2020, Noel Duffy via fpc-pascal wrote: I'm in need of a library to perform DNS lookups for a range of different resource types such as MX, TXT, PTR and so on. At first I thought that netdb would do the job, but it turns out that it only supports direct A and AAA lookups. However, the code actually has the ability to query all resource types, but the module's API doesn't expose this functionality. The functions Query and BuildQuery, and the types TQueryData and TRRData are not made available to callers. cnetdb is similar in that it exposes just a subset of what libresolv provides. There's no interface to things like res_nquery, for instance. While it's theoretically possible for me to interface directly to libresolv, in reality the API is a horrible hairball of C macros and pointers and this is not a path I want to go down. Is it intended that netdb will be capable of querying all resource types? It appears that most of the work is already done, with the only major pieces missing the parsing of the additional resource records and the exposure of additional functions for the public API. It was intended, but in practice I just added the records I needed. If you feel like adding new records, I'll gladly accept patches. I can certainly do that. The first question, though, is what should the API look like? From my initial reading of the code the simplest path might be to expose Query and BuildQuery publicly, so that users can compose queries for arbitrary resource types similarly to libresolv, but there will still need to be new methods to parse the RR data. Such an API is low-level but offers the most flexibility. That's what a number of other languages do. I'll need to do a close read of RFC 1035 to get a clearer idea of the RR data parsing requirements before I can suggest anything useful there. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
[fpc-pascal] netdb and DNS queries for MX, TXT, and other resource types
I'm in need of a library to perform DNS lookups for a range of different resource types such as MX, TXT, PTR and so on. At first I thought that netdb would do the job, but it turns out that it only supports direct A and AAA lookups. However, the code actually has the ability to query all resource types, but the module's API doesn't expose this functionality. The functions Query and BuildQuery, and the types TQueryData and TRRData are not made available to callers. cnetdb is similar in that it exposes just a subset of what libresolv provides. There's no interface to things like res_nquery, for instance. While it's theoretically possible for me to interface directly to libresolv, in reality the API is a horrible hairball of C macros and pointers and this is not a path I want to go down. Is it intended that netdb will be capable of querying all resource types? It appears that most of the work is already done, with the only major pieces missing the parsing of the additional resource records and the exposure of additional functions for the public API. What other libraries are there for DNS? I'm aware of Indy and Synapse, but have used neither. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Bug 37080 -- StrToHostAddr accepts all Pascal number notations
On 17/05/20 3:00 am, Michael Van Canneyt wrote: On Sat, 16 May 2020, Michael Van Canneyt wrote: On Sat, 16 May 2020, Jonas Maebe wrote: On 15/05/2020 12:39, Noel Duffy via fpc-pascal wrote: While doing some work on bug 37060, the refactoring of StrToHostAddr and StrToHostAddr6 in the sockets unit,(https://bugs.freepascal.org/view.php?id=37060), I found that StrToHostAddr is doing no validation at all on input address characters before calling the function Val, so any Pascal notation that Val accepts, such as 0x and $ for hexadecimal, % for binary, & for octal, and mathematical signs are all accepted in ipv4 octets. I added a note to https://bugs.freepascal.org/view.php?id=37013 about the fact that this test program fails if range checking is enabled (I don't know if the range error is in StrToHostAddr6 or in the test program itself). I will check. I never use range checking, so that could have easily escaped me. Fixed 2 occurrences of a range check. Looking at the code in SVN, I spotted a couple of still extant references to s6_addr16 instead of u6_addr16 in StrToHostAddr6. It's possible to trigger range check errors for them. All s6_addr16s should be u6_addr16s. If you try parsing an address like '::' it will hit this part of the code. Sorry for the screw-up. Normally I religiously use range checks in my own code, but it completely went out of my head when working on this. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Bug 37080 -- StrToHostAddr accepts all Pascal number notations
On 17/05/20 1:21 am, Jonas Maebe wrote: On 15/05/2020 12:39, Noel Duffy via fpc-pascal wrote: While doing some work on bug 37060, the refactoring of StrToHostAddr and StrToHostAddr6 in the sockets unit,(https://bugs.freepascal.org/view.php?id=37060), I found that StrToHostAddr is doing no validation at all on input address characters before calling the function Val, so any Pascal notation that Val accepts, such as 0x and $ for hexadecimal, % for binary, & for octal, and mathematical signs are all accepted in ipv4 octets. I added a note to https://bugs.freepascal.org/view.php?id=37013 about the fact that this test program fails if range checking is enabled (I don't know if the range error is in StrToHostAddr6 or in the test program itself). D'oh! Normally I'm scrupulous about turning on range checks in my own programs, but I completely forgot to do that here. But I see Michael has already fixed it. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Return value of StrToHostAddr and StrToHostAddr6 in sockets unit
Per discussions, I've posted a proposed patch on bug 37060 https://bugs.freepascal.org/view.php?id=37060 I've also attached a test program that exercises the new functions. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Bug 37080 -- StrToHostAddr accepts all Pascal number notations
On 15/05/20 11:25 pm, Michael Van Canneyt wrote: On Fri, 15 May 2020, Noel Duffy via fpc-pascal wrote: That must be a new record in bug fix speed. You fixed that within ten minutes of my message to the list! I forgot to mention in my message, but did mention in the bug report, that leading zeroes are also allowed, so you can parse: 01.1.1.1 Personally I don't think this is a problem (mathematically, you can add as many zeroes as you like), but I can imagine some do, so I fixed it. Well, the same argument could be made for allowing hexadecimal or other notations. But ip address representation formats aren't just numbers, they're textual representations meant for easy transmission between programs and across networks, and the standards documents are quite clear about what's allowed in them and what's not. If StrToHostAddr breaks with those standards and accepts addresses that other implementations won't, or rejects addresses others accept, that makes it unusable for programs that must operate in mixed language environments. In my own case, I'm writing a program to parse and validate SPF records, as defined in RFC7208. SPF records are read and processed by SMTP servers as they receive mail, so any tools that attempt to validate SPF records must not accept as valid any ipv4 or ipv6 address that the SMTP server won't also accept. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Bug 37080 -- StrToHostAddr accepts all Pascal number notations
On 15/05/20 10:48 pm, Michael Van Canneyt wrote: On Fri, 15 May 2020, Noel Duffy via fpc-pascal wrote: While doing some work on bug 37060, the refactoring of StrToHostAddr and StrToHostAddr6 in the sockets unit,(https://bugs.freepascal.org/view.php?id=37060), I found that StrToHostAddr is doing no validation at all on input address characters before calling the function Val, so any Pascal notation that Val accepts, such as 0x and $ for hexadecimal, % for binary, & for octal, and mathematical signs are all accepted in ipv4 octets. https://bugs.freepascal.org/view.php?id=37080 As a consequence, StrToHostAddr will happily parse addresses like these: 0xa.3.4.5 9.$c.4.5 %.%11001110.30.4 &7.&5.30.4 12.+4.1.1 Any number in any notation will be accepted as long as byte overflow isn't triggered. Thus, + in octets is accepted, but - is not, because the function detects that the octet is negative which then triggers overflow. Fixing this is outside the scope of the patch I'm preparing for 37060, but if no-one else feels inclined to look into this by the time I'm finished with that patch then I'll submit a new patch for it. Fixed. That must be a new record in bug fix speed. You fixed that within ten minutes of my message to the list! I forgot to mention in my message, but did mention in the bug report, that leading zeroes are also allowed, so you can parse: 01.1.1.1 ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
[fpc-pascal] Bug 37080 -- StrToHostAddr accepts all Pascal number notations
While doing some work on bug 37060, the refactoring of StrToHostAddr and StrToHostAddr6 in the sockets unit,(https://bugs.freepascal.org/view.php?id=37060), I found that StrToHostAddr is doing no validation at all on input address characters before calling the function Val, so any Pascal notation that Val accepts, such as 0x and $ for hexadecimal, % for binary, & for octal, and mathematical signs are all accepted in ipv4 octets. https://bugs.freepascal.org/view.php?id=37080 As a consequence, StrToHostAddr will happily parse addresses like these: 0xa.3.4.5 9.$c.4.5 %.%11001110.30.4 &7.&5.30.4 12.+4.1.1 Any number in any notation will be accepted as long as byte overflow isn't triggered. Thus, + in octets is accepted, but - is not, because the function detects that the octet is negative which then triggers overflow. Fixing this is outside the scope of the patch I'm preparing for 37060, but if no-one else feels inclined to look into this by the time I'm finished with that patch then I'll submit a new patch for it. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Return value of StrToHostAddr and StrToHostAddr6 in sockets unit
On 14/05/20 12:36 am, Michael Van Canneyt wrote:
On Wed, 13 May 2020, Noel Duffy via fpc-pascal wrote:
On 13/05/20 9:24 pm, Bart via fpc-pascal wrote:
On Wed, May 13, 2020 at 10:40 AM Noel Duffy via fpc-pascal
As I've already learned the hard way, the sockets unit is compiled in
"fpc" mode, so there's no "out" parameters. Otherwise I would
definitely
have used one.
{$modeswicth out} perhaps?
I didn't know this existed. I'm not sure of the implications of its
use, though, so I would like to get the opinion of more experienced
FPC developers before I use it.
It's OK to use this. It just enables the use of the out keyword for the
current unit, nothing more.
It's safest to add it before the 'interface' keyword.
OK, I've made a note on the bug report to use {$modeswitch out} and to
use out parameters for the function definitions.
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Return value of StrToHostAddr and StrToHostAddr6 in sockets unit
On 13/05/20 9:24 pm, Bart via fpc-pascal wrote:
On Wed, May 13, 2020 at 10:40 AM Noel Duffy via fpc-pascal
As I've already learned the hard way, the sockets unit is compiled in
"fpc" mode, so there's no "out" parameters. Otherwise I would definitely
have used one.
{$modeswicth out} perhaps?
I didn't know this existed. I'm not sure of the implications of its use,
though, so I would like to get the opinion of more experienced FPC
developers before I use it.
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Return value of StrToHostAddr and StrToHostAddr6 in sockets unit
On 13/05/20 8:36 pm, Michael Van Canneyt wrote: On Wed, 13 May 2020, Noel Duffy via fpc-pascal wrote: On 13/05/20 7:26 pm, Michael Van Canneyt wrote: BTW, I didn't see an obvious way within Mantis to link one bug to another, or to indicate any kind of relationship. Is this possible? I've just mentioned the bug number in the bug text of the new one so that anyone reading it can see the link. You can after you saved it, see the 'Relationships' box below the 'Issue Details' box. I added the relation. I see the box now. But it appears I can't add anything to it. Maybe my user account lacks sufficient privileges. But since you've added the relation, I won't worry about it. Thanks. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Return value of StrToHostAddr and StrToHostAddr6 in sockets unit
On 13/05/20 6:41 pm, Marc Weustink via fpc-pascal wrote: On 12-5-2020 14:56, Marco van de Voort wrote: Op 2020-05-12 om 12:32 schreef Michael Van Canneyt: I'm all for it, but please use the following names: function TryStrToHostAddr(IP: String; var ip4: in_addr): Boolean; function TryStrToHostAddr6(IP: String; var ip6: in6_addr): Boolean; Rationale for this request: All conversion calls in sysutils and dateutils use this naming scheme, it's good for consistency. (and I hate underscores, but that's beside the point ;)) Yes to the principle and yes to the TryStrTo naming. But I would make the IP param CONST ;-) And maybe change the var into an out As I've already learned the hard way, the sockets unit is compiled in "fpc" mode, so there's no "out" parameters. Otherwise I would definitely have used one. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Return value of StrToHostAddr and StrToHostAddr6 in sockets unit
On 13/05/20 7:26 pm, Michael Van Canneyt wrote: Always a new issue for new changes. Makes it easier to track where something went wrong if something is wrong. Done. https://bugs.freepascal.org/view.php?id=37060 I will try to get a patch ready for this in the next couple of days. BTW, I didn't see an obvious way within Mantis to link one bug to another, or to indicate any kind of relationship. Is this possible? I've just mentioned the bug number in the bug text of the new one so that anyone reading it can see the link. I have applied and committed your patch and resolved the current bugreport. Your test program went into the testsuite. Many thanks for your efforts ! Excellent. Thanks for that. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Return value of StrToHostAddr and StrToHostAddr6 in sockets unit
On 12/05/20 10:32 pm, Michael Van Canneyt wrote:> On Tue, 12 May 2020, Noel Duffy via fpc-pascal wrote:>>>> A simple solution is to add functions to the sockets unit which accep a var record into which the output will be written and which return a boolean to indicate success or failure. E.g, function inet_pton4(IP: String; var ip4: in_addr): Boolean; function inet_pton6(IP: String; var ip6: in6_addr): Boolean; I'm all for it, but please use the following names: function TryStrToHostAddr(IP: String; var ip4: in_addr): Boolean; function TryStrToHostAddr6(IP: String; var ip6: in6_addr): Boolean; Rationale for this request: All conversion calls in sysutils and dateutils use this naming scheme, it's good for consistency. Sure, I can do that. I will look at creating a patch for this. For your purposes, do you prefer to have a new bug in the bug tracker opened for tracking this change? (and I hate underscores, but that's beside the point ;)) Uh-oh! (looking at sea of underscores in my Pascal code!) ;) ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Return value of StrToHostAddr and StrToHostAddr6 in sockets unit
On 13/05/20 12:56 am, Marco van de Voort wrote:
Op 2020-05-12 om 12:32 schreef Michael Van Canneyt:
The names I use here are the libc names, which many other languages
also use.
Now, StrToHostAddr and StrToHostAddr6 can call the appropriate
function, and programs that use those functions won't notice any
change. But new code will be able to use the new functions and have
an unambiguous answer as to whether the address is valid or not.
The change is quite straightforward to make, and it won't break
existing code, so I think it's worthwhile to do.
Any thoughts or opinions on this?
I'm all for it, but please use the following names:
function TryStrToHostAddr(IP: String; var ip4: in_addr): Boolean;
function TryStrToHostAddr6(IP: String; var ip6: in6_addr): Boolean;
Rationale for this request: All conversion calls in sysutils and
dateutils use this naming scheme, it's good for consistency.
(and I hate underscores, but that's beside the point ;))
Yes to the principle and yes to the TryStrTo naming. But I would make
the IP param CONST ;-)
Good idea, and normally I would, but the current implementation of
StrToHostAddr (see below) does Delete on IP as it processes it. As it
currently stands this function must accept a non-const.
function StrToHostAddr(IP : AnsiString) : in_addr ;
Var
Dummy : AnsiString;
I,j,k : Longint;
Temp : in_addr;
begin
strtohostaddr.s_addr:=0; //:=NoAddress;
For I:=1 to 4 do
begin
If I<4 Then
begin
J:=Pos('.',IP);
If J=0 then
exit;
Dummy:=Copy(IP,1,J-1);
Delete (IP,1,J);
end
else
Dummy:=IP;
Val (Dummy,k,J);
array4int(temp.s_addr)[i]:=k;
If J<>0 then Exit;
end;
strtohostaddr.s_addr:=ntohl(Temp.s_addr);
end;
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
[fpc-pascal] Return value of StrToHostAddr and StrToHostAddr6 in sockets unit
While working with StrToHostAddr and StrToHostAddr6 over the past couple of weeks I've run into issues caused by the functions returning all zero addresses to indicate errors. All-zero addresses are technically valid according to the RFCs, so an all-zero address shouldn't be used as an error sentinel. E,g: 0.0.0.0 :: ::0.0.0.0 Are considered valid if non-routable addresses. libc's inet_pton will successfully parse these. A simple solution is to add functions to the sockets unit which accept a var record into which the output will be written and which return a boolean to indicate success or failure. E.g, function inet_pton4(IP: String; var ip4: in_addr): Boolean; function inet_pton6(IP: String; var ip6: in6_addr): Boolean; The names I use here are the libc names, which many other languages also use. Now, StrToHostAddr and StrToHostAddr6 can call the appropriate function, and programs that use those functions won't notice any change. But new code will be able to use the new functions and have an unambiguous answer as to whether the address is valid or not. The change is quite straightforward to make, and it won't break existing code, so I think it's worthwhile to do. Any thoughts or opinions on this? ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Bugs in StrToHostAddr6 in sockets unit
I've submitted a test program and proposed patch to fix the IPv6 parsing issues with StrToHostAddr6. https://bugs.freepascal.org/view.php?id=37013 Because I only checked out the packages and rtl directories from Subversion, my patch's root directory is packages. I hope this isn't a problem, but if it is I'll re-do it. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Compiler mode for code in rtl and rtl-extra
On 11/05/20 7:29 pm, Michael Van Canneyt wrote: On Mon, 11 May 2020, Noel Duffy via fpc-pascal wrote: Example error messages: sockets.inc(497,5) Error: Identifier not found "Result" sockets.inc(510,53) Fatal: Syntax error, ":" expected but "identifier RES" found for this definition: function convert_hextet(const s: ShortString; out res: Word): Boolean; You are correct. The unit is compiled in fpc mode, which means no "result" variable, no out param (replace with var). Great. If var is available then the job is easy. We can envision changing this mode, but then the whole unit needs to be verified and checked for correct functioning, since it will redefine basic types as integer from 16 to 32 bit, change the meaning of string. I don't think you will want to undertake that effort :) I don't think I want to do it either! This certainly falls under the "If it ain't broke" maxim. Thanks for the fast response. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
[fpc-pascal] Compiler mode for code in rtl and rtl-extra
I've been beavering away at a fix for https://bugs.freepascal.org/view.php?id=37013 and I've run into something unexpected. When I try to build rtl-extra with my proposed patch applied, I get compiler errors for things like the use of result variables and var/out variables. I'm so used to these being available that I quite forgot there are modes where they're not. My question is, first, what mode is the compiler using, and second, is this required for portability to other platforms? Or am I completely misunderstanding this? Example error messages: sockets.inc(497,5) Error: Identifier not found "Result" sockets.inc(510,53) Fatal: Syntax error, ":" expected but "identifier RES" found for this definition: function convert_hextet(const s: ShortString; out res: Word): Boolean; It's not a big deal to rewrite around these, but first I want to make sure I am correctly diagnosing the issue. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Bugs in StrToHostAddr6 in sockets unit
On 7/05/20 9:43 pm, Michael Van Canneyt wrote: On Thu, 7 May 2020, Noel Duffy via fpc-pascal wrote: Well, considering the amount of RFCs I implemented for FPC, I think I'll join you. Let me know what golf course you picked. If your email address is any indication, new zealand is a bit far off, but has always been on my bucket list ;-) Yes, I'm in New Zealand. Right now there's a two-week quarantine for all new arrivals though, and the golf courses are of course shut! Still, plenty of time with nothing to do but write Pascal code and obsess over the minutiae of IETF standards! ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Bugs in StrToHostAddr6 in sockets unit
On 7/05/20 9:17 pm, Michael Van Canneyt wrote: On Thu, 7 May 2020, Noel Duffy via fpc-pascal wrote: Great. My worry was that the sockets unit might fall into the same category as the libc one, available but deprecated and not recommended for use. Plus, when rewriting something that's been unchanged for years, there's a risk of breaking programs that depend on the bug. I think the processing of wrong IP addresses and missing IPv4-ending addresses support is not really something one depends on :) So fixing this is quite OK. Heh! Guess I really am the only one who implements RFCs in Object Pascal in his spare time. Maybe I should have learned to play golf :D ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Bugs in StrToHostAddr6 in sockets unit
On 6/05/20 11:40 pm, Karoly Balogh (Charlie/SGR) wrote: As much as we use the compiler and associated libraries ourselves, we can't cover every use case, or find every bug. A lot of packages are there for completeness, or because someone submitted it, or for compatibility with Delphi, not because we necessarily use it ourselves. So if you find weak spots like this where you feel you can do better, do it and submit it! Sure, I do understand this, but it can be difficult sometimes to tell whether a unit is actively used or whether it's a relic of some long abandoned idea, like the libc unit for instance. Added to that, I am aware that rewriting functions in old units risks breaking things, even if the fix is correct, and so I feel it's prudent (and, hopefully, polite) to ask before writing the code. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Bugs in StrToHostAddr6 in sockets unit
On 6/05/20 11:21 pm, Christo Crause via fpc-pascal wrote: > On Wed, May 6, 2020 at 12:19 PM Noel Duffy via fpc-pascal > <mailto:fpc-pascal@lists.freepascal.org>> > wrote: > > So I guess the question is, is it worth the effort to make > StrToHostAddr6 RFC4291 compliant? Is that something the FPC team would > want, or do they just not use the sockets unit? > > > There have been 2 changes applied to sockets.inc in the last two > months so it is not neglected. I am convinced a patch to ensure > compliance will be considered for inclusion, especially if you > provide test cases demonstrating shortcomings in the current > implementation. Great. My worry was that the sockets unit might fall into the same category as the libc one, available but deprecated and not recommended for use. Plus, when rewriting something that's been unchanged for years, there's a risk of breaking programs that depend on the bug. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Bugs in StrToHostAddr6 in sockets unit
On 3/05/20 10:28 pm, Michael Van Canneyt wrote: On Sun, 3 May 2020, Noel Duffy via fpc-pascal wrote: On Sun, 3 May 2020 09:57:46 +0200 (CEST) Michael Van Canneyt wrote: Yes, please open a bug report. If you attach a small console test program that demonstrates the bug (and subsequently the fix) then I will make sure it ends up in the correct place. If you make sure it exits with exit code 0 if all is well, and a nonzero exit code if there is an error, that will save me some work. If you post the URL for the bug here, I will look at it at once. To follow-up on this issue I raised in bug #37013 (https://bugs.freepascal.org/view.php?id=37013), I've been digging into RFC4291 to see what address formats for IPv6 exist, and I see that the scope of this problem is bigger than I envisioned. To summarize, the current function, StrToHostAddr6, from the sockets unit, will successfully parse many address formats expressly forbidden by RFC4291, and it fails to parse others that the RFC expressly requires. An example of the latter is this ugly form: fe80:caf1:906e:9d2f:a520:4172:172.16.31.14 This apparently exists to make IPv4 to IPv6 transitions easier. Anyway, back to the bug report. I can't make StrToHostAddr6 RFC4291 compliant without rewriting it, and I don't want to undertake that unless I know it's something the FPC community actually wants. I am happy to do the work, don't get me wrong, but I realise that a wholesale rewrite of the function is much wider in scope and has more potential for drawbacks. Testing would also have to be fairly rigorous to make sure all the different formats are handled correctly. I've already done some work to make it easy to compare StrToHostAddr6's output to that of inet_pton in the C library, so this would at least provide a good method to quickly spot problems. So I guess the question is, is it worth the effort to make StrToHostAddr6 RFC4291 compliant? Is that something the FPC team would want, or do they just not use the sockets unit? ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Bugs in StrToHostAddr6 in sockets unit
On 3/05/20 10:28 pm, Michael Van Canneyt wrote: Yes, please open a bug report. If you attach a small console test program that demonstrates the bug (and subsequently the fix) then I will make sure it ends up in the correct place. If you make sure it exits with exit code 0 if all is well, and a nonzero exit code if there is an error, that will save me some work. If you post the URL for the bug here, I will look at it at once. Done. Bug #37013. https://bugs.freepascal.org/view.php?id=37013 ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Bugs in StrToHostAddr6 in sockets unit
On Sun, 3 May 2020 09:57:46 +0200 (CEST) Michael Van Canneyt wrote: On Sun, 3 May 2020, Noel Duffy via fpc-pascal wrote: The problem is that the StrToHostAddr6 function doesn't set its return value until the end of the function. If a parse error occurs mid-function, it zeroes the record in which it's writing the result but exits without setting the function return value. What gets returned depends on what's on the stack. I could have sworn that fpc would detect a function exiting without setting a return value, but clearly 3.0.4 doesn't. It checks if the result is assigned. This is done; But it does not check this for every exit, as far as I know it never has. That seems like it would be a valuable feature to have. I am not certain what's contained in an uninitialized variable that gets returned from a function but I'd guess there's a risk of information leakage. The way return values from functions get set is one part of the Pascal language I dislike. 2. The second problem is that a colon-separated string containing hexadecimal values of any length will be parsed and treated as valid by StrToHostAddr6. E.g, it will parse a string like "fe80ca2f1::906e:9d2f:a520:4172". The sample program produces this output for this string: "fe80ca2f1::906e:9d2f:a520:4172" -> A2-F1-00-00-00-00-00-00-90-6E-9D-2F-A5-20-41-72- For the part before the first colon, it has discarded all but the last two bytes, A2 and F1. But it should not have accepted this string at all. There can be only four characters between the colons. If you send a patch for this, I'll apply it. Sure I can take a stab at this. Do you normally get people to open a bug against which to post the patch? I'm not at all familiar with contributing to fpc, I'm afraid, so please bear with me! Also, if there's information on unit testing for library functions and procedures, that would be very helpful. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
[fpc-pascal] Bugs in StrToHostAddr6 in sockets unit
Using fpc 3.0.4 on Fedora 30.
I've just started using the StrToHostAddr6 function in the sockets unit
to parse IPv6 addresses. I've found a couple of issues with it.
1. Even if address parsing fails, StrToHostAddr6 doesn't return an
all-zero result in the in6_addr return value. The documentation states
that it does, but my tests show it doesn't.
This can be trivially demonstrated with a program like so:
program ip6bug;
uses
SysUtils, sockets;
procedure parse_print_ip6(addr: String);
var
ip6: in6_addr;
idx: Cardinal;
begin
ip6 := StrToHostAddr6(addr);
write('"' + addr+'" ' + '-> ');
for idx := 0 to 15 do
write(IntToHex(byte(ip6.s6_addr8[idx]),2)+'-');
writeln();
end;
const
addr = 'fxg%::906e:9d2f:a520:4172';
begin
parse_print_ip6(addr);
end.
On my machine this produces:
"fxg%::906e:9d2f:a520:4172" ->
00-00-00-60-68-74-E0-00-00-00-00-61-4F-2B-60-00-
The problem is that the StrToHostAddr6 function doesn't set its return
value until the end of the function. If a parse error occurs
mid-function, it zeroes the record in which it's writing the result but
exits without setting the function return value. What gets returned
depends on what's on the stack.
I could have sworn that fpc would detect a function exiting without
setting a return value, but clearly 3.0.4 doesn't.
2. The second problem is that a colon-separated string containing
hexadecimal values of any length will be parsed and treated as valid by
StrToHostAddr6. E.g, it will parse a string like
"fe80ca2f1::906e:9d2f:a520:4172". The sample program produces this
output for this string:
"fe80ca2f1::906e:9d2f:a520:4172" ->
A2-F1-00-00-00-00-00-00-90-6E-9D-2F-A5-20-41-72-
For the part before the first colon, it has discarded all but the last
two bytes, A2 and F1. But it should not have accepted this string at
all. There can be only four characters between the colons.
Neither of these are difficult to fix. But I am curious to know why the
compiler lets a function return without setting a return value and
doesn't at least issue a warning. I know it does in some cases, because
I've often seen the message "Function result doesn't appear to be set".
But maybe that's only if you never set the result, as opposed to setting
it only for some paths through the code.
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] StrToHostAddr in sockets unit accepts negative octets
On 20/04/20 7:41 pm, Karoly Balogh (Charlie/SGR) wrote: Hi, On Mon, 20 Apr 2020, Noel Duffy via fpc-pascal wrote: I have already fixed it, with a simplistic fix, I think. Committed as SVN r44845. Great, good to know it's fixed. Not having the SVN repo to hand, I can't tell how recent that revision is. Has this change made it to any released version of the compiler? No. I just fixed it yesterday. :) So the fix is new. I'll probably get it merged to the soon-to-be released 3.2, if no one objects. Oh I see. I thought you meant it had already been fixed before I raised it. If it hasn't then I will have to either roll a custom function or look to falling back on the system native libc's inet_pton function. You can compile yourself a custom version of the sockets unit from current trunk too in the mean time. Or you can even patch 3.0.4, if you want. Here's what's changed: https://svn.freepascal.org/cgi-bin/viewvc.cgi/trunk/packages/rtl-extra/src/inc/sockets.inc?r1=44845&r2=44844&pathrev=44845 That's simple enough. I will probably just make a custom copy of that function for now and leave the sockets unit alone so that future compiler upgrades don't cause issues. Falling back to inet_pton() is also an option indeed, if your program already links against libc (for threading or other reasons), that might be the easiest option. It doesn't yet, and I generally try to avoid libc if I can, but the next part of my program has to parse IPv6 addresses and do CIDR calculations against them, and so I half suspect that I'm going to end up using libc anyway. But that's a bridge to cross when I come to it. Thank you again for the blazingly fast bug fix. Cheers, Noel. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] StrToHostAddr in sockets unit accepts negative octets
On 19/04/20 11:48 pm, Karoly Balogh (Charlie/SGR) wrote: Hi, On Sun, 19 Apr 2020, Michael Van Canneyt wrote: The StrToHostAddr function in the sockets unit accepts negative octets in the source ip address. Consider this program: This should indeed be ckecked. Please report this in the bugtracker, so we don't forget to look at it! I have already fixed it, with a simplistic fix, I think. Committed as SVN r44845. Basically it was wrapping everything in the longint number range silently to a byte. So for example a string saying '172.300.32.14' was accepted as IP 172.44.32.14. My patch fixes this too, not just the negatives, but please feel free to verify, or improve on it if you feel like. This was an ancient bug, the code in question was unchanged since SVN rev 1, so 2005 at least, but probably longer. Great, good to know it's fixed. Not having the SVN repo to hand, I can't tell how recent that revision is. Has this change made it to any released version of the compiler? If it hasn't then I will have to either roll a custom function or look to falling back on the system native libc's inet_pton function. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
[fpc-pascal] StrToHostAddr in sockets unit accepts negative octets
Running fpc 3.0.4 on Fedora 30.
The StrToHostAddr function in the sockets unit accepts negative octets
in the source ip address. Consider this program:
program ip4_neg;
{$mode objfpc}{$H+}
uses
sockets;
var
ip4_addr: in_addr;
begin
ip4_addr := StrToHostAddr('172.-16.32.14');
if ip4_addr.s_addr = 0 then
writeln('invalid ip.')
else
writeln('ip is valid: ' + HostAddrToStr(ip4_addr));
end.
This program parses an invalid ip address. But when run:
$ ./ip4_neg
ip is valid: 172.240.32.14
This seems to me to be a bug. I tested this in Python using its
socket.inet_aton (a wrapper around the C library function) and also
directly in C and both returned an error for this address.
E.g,
$ python
Python 2.7.17 (default, Oct 21 2019, 17:20:57)
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.inet_aton('172.-16.32.14')
Traceback (most recent call last):
File "", line 1, in
socket.error: illegal IP address string passed to inet_aton
>>>
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Ncurses unit and newer releases of ncurses
On 30/03/20 8:17 pm, Michael Van Canneyt wrote: On Sun, 29 Mar 2020, Noel Duffy via fpc-pascal wrote: Running fpc 3.0.4 on Fedora 30. The ncurses unit is based on version 5.6 of ncurses, according to constants defined in the source. The upstream ncurses is now at 6.2, and I wondered if there are plans afoot to upgrade the ncurses unit to support a more recent ncurses version. Newer releases support 256 colours and additional attributes like italic on supported terminals. These plans exist, I even did some work on it already. Great! It's not critical but it certainly would be nice to have. I don't know if some part of the ncurses unit automatically generated from from the C sources, so I don't know how difficult this job actually is. Well, it's a manual job. The C sources proved too messy for h2pas... I was afraid that that might be the case. My C has long since rusted away but if I can help to get this done let me know. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
[fpc-pascal] Ncurses unit and newer releases of ncurses
Running fpc 3.0.4 on Fedora 30. The ncurses unit is based on version 5.6 of ncurses, according to constants defined in the source. The upstream ncurses is now at 6.2, and I wondered if there are plans afoot to upgrade the ncurses unit to support a more recent ncurses version. Newer releases support 256 colours and additional attributes like italic on supported terminals. I don't know if some part of the ncurses unit automatically generated from from the C sources, so I don't know how difficult this job actually is. Cheers, Noel ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
