Re: [twsocket] OK to Receiving >2GB ?
About 1500 bytes give or take a few over ethernet -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Kei Sent: Saturday, October 29, 2005 3:26 PM To: ICS support mailing Subject: Re: [twsocket] OK to Receiving >2GB ? Hi! I think I'm going to accept the reality that... TCP packets are splitted into arbitrary sizes.. but!!! How do I know the maximum size possible? What is the maximum size that a packet can be? I certainly don't want to malloc 100KB for a TCP packet... Thanks! David Wilfried Mestdagh wrote: >Hello David, > > > >>Hi! I'm new to ICS! >> >> > >Welcome to the group :) > > > >>A->>B: msg hello >>B->>A: msg yo! how's it going? >> >> > >Yes that's the way to go. Design a user made proto for what you intend >to do. > > > >>If A is larger than the default buffer size (256 chars) then the A >>(sender) will warn B in advance >> >> > >You can do that, but is not nececary. Also you will maybe have a very >mutch allocation / deallocation of memory and you can eventually end up >with fragmented memory where you have not a nice large block in it at >the moment you need it. But it can work, just think over carefully. A >better idea is often to make a receive buffer that grows automatically >if (and only if) needed, and then just reuse that buffer over and over >again. Then you have some (re)allocation in begin but then stable. > > > >>TWSocket will automatically split it into packets, >> >> > >Winsock will split in packets as large as the MTU (around 1500 bytes). >Eventually data can (and will if high speed) arrive as 1 large packet >but not necacarely. > > > >>I really want the sender side to send the 1MB file all at once, since >>I do the FileWrite() right after Receive() >> >> > >No you cannot. Winsock does not respect packet boundaries, but (see >prior paragraph) there are no megabytes TCP packets. > >You have to receive all data chuncks into a buffer, and when you >received them all then you save to file (or save every packet direct to >disk). There is no problem to know the moment of close the file because >you know the length of the data from your protocol. > >--- >Rgds, Wilfried [TeamICS] >http://www.overbyte.be/eng/overbyte/teamics.html >http://www.mestdagh.biz > >Saturday, October 29, 2005, 11:01, Kei wrote: > > > >>Hi! I'm new to ICS! >> >> > > > >>I am designing a simple protocol that will be mainly used locally (as >>a database server backend)..so I'm guessing I could send up to 2GB of >>stuff without hassle (BLOBs, for example). Right now I'm just >>experimenting with the facility for two parties to effectively "talk" >>to each other, even with long long messages, and with binary data transfer. >>For example, for sending a message, the command is "msg [text]" >> >> > >A->>B: msg hello >B->>A: msg yo! how's it going? > > > >>If A is larger than the default buffer size (256 chars) then the A >>(sender) will warn B in advance, like this: "hey! I'm gonna send you >>1 bytes of text" >> >> > >A->>B: longmsg 1 >B->>A: ready msg >A->>B: msg blahblahblah...blah! > > > >>In this case, B will be notified of the text size, then when >>OnClientDataAvailable() event comes, it will malloc a bigger buffer, >>then Receive(CustomSized_Buffer, SizeHeToldMe). Similarly Im >>considering the same mechanism to send binary data. But if the file is >>slightly >> >> >larger (>>10KB) then TWSocket will automatically split it into packets, > > >>which I don't want it to do: >> >> > >A->>B: upload 1048576 picture.jpg >B->>A: ready upload >A->>B: 01001010101010.. (10720 bytes) >A->>B: 1010101012.. (10720 bytes) > > >>: >>: >> >> >A->>B: 01001010101010.. (4023 bytes) > > > >>I really want the sender side to send the 1MB file all at once, since >>I do the FileWrite() right after Receive() >> >> > > > >>Could anybody please help me on this issue? >> >> > > > >>Thanks! >> >> > > > >>David >> >> > > > > > > > > > > > > > > > > -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] OK to Receiving >2GB ?
http://www.faqs.org/rfcs/rfc879.html -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Kei Sent: Saturday, October 29, 2005 2:26 PM To: ICS support mailing Subject: Re: [twsocket] OK to Receiving >2GB ? Hi! I think I'm going to accept the reality that... TCP packets are splitted into arbitrary sizes.. but!!! How do I know the maximum size possible? What is the maximum size that a packet can be? I certainly don't want to malloc 100KB for a TCP packet... Thanks! David Wilfried Mestdagh wrote: >Hello David, > > > >>Hi! I'm new to ICS! >> >> > >Welcome to the group :) > > > >>A->>B: msg hello >>B->>A: msg yo! how's it going? >> >> > >Yes that's the way to go. Design a user made proto for what you intend >to do. > > > >>If A is larger than the default buffer size (256 chars) then the A >>(sender) will warn B in advance >> >> > >You can do that, but is not nececary. Also you will maybe have a very >mutch allocation / deallocation of memory and you can eventually end up >with fragmented memory where you have not a nice large block in it at >the moment you need it. But it can work, just think over carefully. A >better idea is often to make a receive buffer that grows automatically >if (and only if) needed, and then just reuse that buffer over and over >again. Then you have some (re)allocation in begin but then stable. > > > >>TWSocket will automatically split it into packets, >> >> > >Winsock will split in packets as large as the MTU (around 1500 bytes). >Eventually data can (and will if high speed) arrive as 1 large packet >but not necacarely. > > > >>I really want the sender side to send the 1MB file all at once, since >>I do the FileWrite() right after Receive() >> >> > >No you cannot. Winsock does not respect packet boundaries, but (see >prior paragraph) there are no megabytes TCP packets. > >You have to receive all data chuncks into a buffer, and when you >received them all then you save to file (or save every packet direct to >disk). There is no problem to know the moment of close the file because >you know the length of the data from your protocol. > >--- >Rgds, Wilfried [TeamICS] >http://www.overbyte.be/eng/overbyte/teamics.html >http://www.mestdagh.biz > >Saturday, October 29, 2005, 11:01, Kei wrote: > > > >>Hi! I'm new to ICS! >> >> > > > >>I am designing a simple protocol that will be mainly used locally (as >>a database server backend)..so I'm guessing I could send up to 2GB of >>stuff without hassle (BLOBs, for example). Right now I'm just >>experimenting with the facility for two parties to effectively "talk" >>to each other, even with long long messages, and with binary data transfer. >>For example, for sending a message, the command is "msg [text]" >> >> > >A->>B: msg hello >B->>A: msg yo! how's it going? > > > >>If A is larger than the default buffer size (256 chars) then the A >>(sender) will warn B in advance, like this: "hey! I'm gonna send you >>1 bytes of text" >> >> > >A->>B: longmsg 1 >B->>A: ready msg >A->>B: msg blahblahblah...blah! > > > >>In this case, B will be notified of the text size, then when >>OnClientDataAvailable() event comes, it will malloc a bigger buffer, >>then Receive(CustomSized_Buffer, SizeHeToldMe). Similarly Im >>considering the same mechanism to send binary data. But if the file is >>slightly >> >> >larger (>>10KB) then TWSocket will automatically split it into packets, > > >>which I don't want it to do: >> >> > >A->>B: upload 1048576 picture.jpg >B->>A: ready upload >A->>B: 01001010101010.. (10720 bytes) >A->>B: 1010101012.. (10720 bytes) > > >>: >>: >> >> >A->>B: 01001010101010.. (4023 bytes) > > > >>I really want the sender side to send the 1MB file all at once, since >>I do the FileWrite() right after Receive() >> >> > > > >>Could anybody please help me on this issue? >> >> > > > >>Thanks! >> >> > > > >>David >> >> > > > > > > > > > > > > > > > > -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] OK to Receiving >2GB ?
Hi! I think I'm going to accept the reality that... TCP packets are splitted into arbitrary sizes.. but!!! How do I know the maximum size possible? What is the maximum size that a packet can be? I certainly don't want to malloc 100KB for a TCP packet... Thanks! David Wilfried Mestdagh wrote: >Hello David, > > > >>Hi! I'm new to ICS! >> >> > >Welcome to the group :) > > > >>A->>B: msg hello >>B->>A: msg yo! how's it going? >> >> > >Yes that's the way to go. Design a user made proto for what you intend >to do. > > > >>If A is larger than the default buffer size (256 chars) then the A >>(sender) will warn B in advance >> >> > >You can do that, but is not nececary. Also you will maybe have a very >mutch allocation / deallocation of memory and you can eventually end up >with fragmented memory where you have not a nice large block in it at >the moment you need it. But it can work, just think over carefully. A >better idea is often to make a receive buffer that grows automatically >if (and only if) needed, and then just reuse that buffer over and over >again. Then you have some (re)allocation in begin but then stable. > > > >>TWSocket will automatically split it into packets, >> >> > >Winsock will split in packets as large as the MTU (around 1500 bytes). >Eventually data can (and will if high speed) arrive as 1 large packet >but not necacarely. > > > >>I really want the sender side to send the 1MB file all at once, since I >>do the FileWrite() right after Receive() >> >> > >No you cannot. Winsock does not respect packet boundaries, but (see >prior paragraph) there are no megabytes TCP packets. > >You have to receive all data chuncks into a buffer, and when you >received them all then you save to file (or save every packet direct to >disk). There is no problem to know the moment of close the file because >you know the length of the data from your protocol. > >--- >Rgds, Wilfried [TeamICS] >http://www.overbyte.be/eng/overbyte/teamics.html >http://www.mestdagh.biz > >Saturday, October 29, 2005, 11:01, Kei wrote: > > > >>Hi! I'm new to ICS! >> >> > > > >>I am designing a simple protocol that will be mainly used locally (as a >>database server backend)..so I'm guessing I could send up to 2GB of >>stuff without hassle (BLOBs, for example). Right now I'm just >>experimenting with the facility for two parties to effectively "talk" to >>each other, even with long long messages, and with binary data transfer. >>For example, for sending a message, the command is "msg [text]" >> >> > >A->>B: msg hello >B->>A: msg yo! how's it going? > > > >>If A is larger than the default buffer size (256 chars) then the A >>(sender) will warn B in advance, like this: "hey! I'm gonna send you >>1 bytes of text" >> >> > >A->>B: longmsg 1 >B->>A: ready msg >A->>B: msg blahblahblah...blah! > > > >>In this case, B will be notified of the text size, then when >>OnClientDataAvailable() event comes, it will malloc a bigger buffer, >>then Receive(CustomSized_Buffer, SizeHeToldMe). Similarly Im considering >>the same mechanism to send binary data. But if the file is slightly >> >> >larger (>>10KB) then TWSocket will automatically split it into packets, > > >>which I don't want it to do: >> >> > >A->>B: upload 1048576 picture.jpg >B->>A: ready upload >A->>B: 01001010101010.. (10720 bytes) >A->>B: 1010101012.. (10720 bytes) > > >>: >>: >> >> >A->>B: 01001010101010.. (4023 bytes) > > > >>I really want the sender side to send the 1MB file all at once, since I >>do the FileWrite() right after Receive() >> >> > > > >>Could anybody please help me on this issue? >> >> > > > >>Thanks! >> >> > > > >>David >> >> > > > > > > > > > > > > > > > > -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Downloading a webpage, including itspictures.How??
True. Would this be called a web-spider? >It actually takes more than one THttpCli.Get(). >My understanding is that once you get the initial html, you have >to parse it, get all the links, and request them, until you exhaust >the links. > >Jack -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Downloading a webpage, including its pictures. How??
It actually takes more than one THttpCli.Get(). My understanding is that once you get the initial html, you have to parse it, get all the links, and request them, until you exhaust the links. Jack > Do you have any examples of how to use THttpCli for tasks like this? > /Perry > Arno Garrels wrote: >>Public wrote: >> >> >>>Hello, >>> >>>I would like to download a webpage, including all its pictures, to my >>>local harddrive. >>>I will later use the local file and display this in a Twebbrowser. >>> >>>How can I do this? >>>Which components do you recommend me to use? >>> >>> >> >>THttpCli is your friend. The rest is elaboration. >> >>--- >>Arno Garrels [ICSTeam] >> >> >> >> >> >> -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Downloading a webpage, including its pictures. How??
Do you have any examples of how to use THttpCli for tasks like this? /Perry Arno Garrels wrote: >Public wrote: > > >>Hello, >> >>I would like to download a webpage, including all its pictures, to my >>local harddrive. >>I will later use the local file and display this in a Twebbrowser. >> >>How can I do this? >>Which components do you recommend me to use? >> >> > >THttpCli is your friend. The rest is elaboration. > >--- >Arno Garrels [ICSTeam] > > > > > > -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] OK to Receiving >2GB ?
Hello David, > Hi! I'm new to ICS! Welcome to the group :) > A->>B: msg hello > B->>A: msg yo! how's it going? Yes that's the way to go. Design a user made proto for what you intend to do. > If A is larger than the default buffer size (256 chars) then the A > (sender) will warn B in advance You can do that, but is not nececary. Also you will maybe have a very mutch allocation / deallocation of memory and you can eventually end up with fragmented memory where you have not a nice large block in it at the moment you need it. But it can work, just think over carefully. A better idea is often to make a receive buffer that grows automatically if (and only if) needed, and then just reuse that buffer over and over again. Then you have some (re)allocation in begin but then stable. > TWSocket will automatically split it into packets, Winsock will split in packets as large as the MTU (around 1500 bytes). Eventually data can (and will if high speed) arrive as 1 large packet but not necacarely. > I really want the sender side to send the 1MB file all at once, since I > do the FileWrite() right after Receive() No you cannot. Winsock does not respect packet boundaries, but (see prior paragraph) there are no megabytes TCP packets. You have to receive all data chuncks into a buffer, and when you received them all then you save to file (or save every packet direct to disk). There is no problem to know the moment of close the file because you know the length of the data from your protocol. --- Rgds, Wilfried [TeamICS] http://www.overbyte.be/eng/overbyte/teamics.html http://www.mestdagh.biz Saturday, October 29, 2005, 11:01, Kei wrote: > Hi! I'm new to ICS! > I am designing a simple protocol that will be mainly used locally (as a > database server backend)..so I'm guessing I could send up to 2GB of > stuff without hassle (BLOBs, for example). Right now I'm just > experimenting with the facility for two parties to effectively "talk" to > each other, even with long long messages, and with binary data transfer. > For example, for sending a message, the command is "msg [text]" A->>B: msg hello B->>A: msg yo! how's it going? > If A is larger than the default buffer size (256 chars) then the A > (sender) will warn B in advance, like this: "hey! I'm gonna send you > 1 bytes of text" A->>B: longmsg 1 B->>A: ready msg A->>B: msg blahblahblah...blah! > In this case, B will be notified of the text size, then when > OnClientDataAvailable() event comes, it will malloc a bigger buffer, > then Receive(CustomSized_Buffer, SizeHeToldMe). Similarly Im considering > the same mechanism to send binary data. But if the file is slightly larger (>>10KB) then TWSocket will automatically split it into packets, > which I don't want it to do: A->>B: upload 1048576 picture.jpg B->>A: ready upload A->>B: 01001010101010.. (10720 bytes) A->>B: 1010101012.. (10720 bytes) > : > : A->>B: 01001010101010.. (4023 bytes) > I really want the sender side to send the 1MB file all at once, since I > do the FileWrite() right after Receive() > Could anybody please help me on this issue? > Thanks! > David -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
[twsocket] OK to Receiving >2GB ?
Hi! I'm new to ICS! I am designing a simple protocol that will be mainly used locally (as a database server backend)..so I'm guessing I could send up to 2GB of stuff without hassle (BLOBs, for example). Right now I'm just experimenting with the facility for two parties to effectively "talk" to each other, even with long long messages, and with binary data transfer. For example, for sending a message, the command is "msg [text]" A->B: msg hello B->A: msg yo! how's it going? If A is larger than the default buffer size (256 chars) then the A (sender) will warn B in advance, like this: "hey! I'm gonna send you 1 bytes of text" A->B: longmsg 1 B->A: ready msg A->B: msg blahblahblah...blah! In this case, B will be notified of the text size, then when OnClientDataAvailable() event comes, it will malloc a bigger buffer, then Receive(CustomSized_Buffer, SizeHeToldMe). Similarly Im considering the same mechanism to send binary data. But if the file is slightly larger (>10KB) then TWSocket will automatically split it into packets, which I don't want it to do: A->B: upload 1048576 picture.jpg B->A: ready upload A->B: 01001010101010.. (10720 bytes) A->B: 1010101012.. (10720 bytes) : : A->B: 01001010101010.. (4023 bytes) I really want the sender side to send the 1MB file all at once, since I do the FileWrite() right after Receive() Could anybody please help me on this issue? Thanks! David -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
