Re: `Socket.receive` providing arbitrary packet sizes and hanging without sending EOF
On 12/16/2017 05:21 PM, Unazed Spectaculum wrote: > 1) Starting program > unazed@unazed /home/d/storage-server dmd -debug -run app.d Although I don't normally use the -run switch, as expected, it works with -run as well. (More below.) > Listening: 0.0.0.0:6969 > > 2) telnet to the server > unazed@unazed ~ telnet 0.0.0.0 6969 > Trying 0.0.0.0... > Connected to 0.0.0.0. > Escape character is '^]'. Note that line! ;) > > 3) paste '{"filename":"/tmp/foo","method":"bar"}' Please also press Enter after that. I don't know what code is at fault but the json parser was not happy without that newline at the end. > unazed@unazed /home/d/storage-server dmd -debug -run app.d > Listening: 0.0.0.0:6969 > Client: 127.0.0.1:55014 > Chunk 0 > [123, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 34, 47, 116, > 109, 112, 47, 102, 111, 111, 34, 44, 34, 109, 101, 116, 104, 111, 100, > 34, 58, 34, 98, 97, 114, 34, 125, 13, 10] > Chunk 1 > [pause] Same here... > 4-5) press ESC + CTRL+D (I'm on Linux) Aha! That's the problem! You should enter the "Escape character", which is ^] as telnet indicates. You should press Ctrl-] (stay away from the ESC key). Otherwise, you're injecting an ESC character to the communication stream. (The 27 below.) By the way, I'm on Linux as well! :) (Mint, Ubuntu based.) And finally, press Ctrl-D to end the stream. It really works: [...] Chunk 1 Exiting "receiveAll" {"filename":"/tmp/foo","method":"bar"} [...] Ali
Re: `Socket.receive` providing arbitrary packet sizes and hanging without sending EOF
On Thursday, 14 December 2017 at 20:27:36 UTC, Ali Çehreli wrote: On 12/14/2017 11:55 AM, Unazed Spectaculum wrote: This is the only function which has a call to `receiveAll` I marked my changes with [Ali]: import std.stdio; import std.socket; import std.conv; import std.json; ubyte[] receiveBytes(T)(T socket, size_t receiveCount) { ubyte[] buffer = new ubyte[receiveCount]; size_t count = socket.receive(buffer); // [Ali] Return null when there is no data if (count == Socket.ERROR || count == 0) { return null; } return buffer[0 .. count]; } string receiveAll(T)(T socket, size_t segmentSize = 1024) { ubyte[][] data; size_t count = 0; do { debug(1) writefln("Chunk %s", count); auto part = receiveBytes(socket, segmentSize); // [Ali] Done when there is no data if (!part) { break; } data ~= part; writeln(data[count]); if (!data) break; } while(data[count++]); char[] stringData; foreach (elem; data) stringData ~= elem; debug(1) writeln(`Exiting "receiveAll"`); return to!string(stringData); } void main() { auto socket = new TcpSocket(); // [Ali]setupSocket(socket, "0.0.0.0", 6969); // The following had worked in a test program: socket.setOption(SocketOptionLevel.SOCKET, SocketOption.REUSEADDR, true); socket.bind(new InternetAddress(6969)); socket.listen(1); writefln("Listening: %s", socket.localAddress); while(true) { Socket client = socket.accept(); debug(1) writefln("Client: %s", client.remoteAddress); auto data = receiveAll(client); writeln(data); JSONValue json; try { json = parseJSON(data); } catch (JSONException e) { debug(1) writefln("Failed parsing data as JSON, aborting.\n|| %s", e); client.close(); continue; } catch (Exception e) { debug(1) writefln("Client caused exception:\n||%s", e); client.close(); continue; } /+ [Ali] Not important for this test if (!verifyValues(json)) { debug(1) writefln("Client missed out important key fields: %s", client.remoteAddress); client.close(); continue; } +/ debug(1) writeln("Client transacted successful JSON packet."); writefln("%s:\n\tFilename: %s\n\tMethod: %s\n\tData length: %d", client.remoteAddress, json["filename"], json["method"], data.length ); // [Ali] Not important for this test /+ if (json["method"].str == "store") storeData(json["filename"].str, json["data"].str); else if (json["method"].str == "retrieve") retrieveData(client, json["filename"].str); +/ client.close(); } } 1) Start the program 2) $ telnet 0.0.0.0 6969 Trying 0.0.0.0... Connected to 0.0.0.0. Escape character is '^]'. 3) Paste expected json to terminal: {"filename":"/tmp/foo","method":"bar"} 4) Press the escape character ^] 5) Press Ctrl-D (Ctrl-Z on Windows) to terminate the connection telnet> Connection closed. Here is the output from the program: Listening: 0.0.0.0:6969 [123, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 34, 47, 116, 109, 112, 47, 102, 111, 111, 34, 44, 34, 109, 101, 116, 104, 111, 100, 34, 58, 34, 98, 97, 114, 34, 125, 13, 10] {"filename":"/tmp/foo","method":"bar"} 127.0.0.1:47704: Filename: "\/tmp\/foo" Method: "bar" Data length: 40 I think it works! :) Ali I've tried to integrate your code with mine; to no avail, I've directly tested your code; and yet again to no avail, so this should conclude the thread as it pinpoints an error with my actual OS. Just in case, as a last resort in case I'm actually that dumb, here's me reproducing your steps: 1) Starting program unazed@unazed /home/d/storage-server dmd -debug -run app.d Listening: 0.0.0.0:6969 2) telnet to the server unazed@unazed ~ telnet 0.0.0.0 6969 Trying 0.0.0.0... Connected to 0.0.0.0. Escape character is '^]'. 3) paste '{"filename":"/tmp/foo","method":"bar"}' unazed@unazed /home/d/storage-server dmd -debug -run app.d Listening: 0.0.0.0:6969 Client: 127.0.0.1:55014 Chunk 0 [123, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 34, 47, 116, 109, 112, 47, 102, 111, 111, 34, 44, 34, 109, 101, 116, 104, 111, 100, 34, 58, 34, 98, 97, 114, 34, 125, 13, 10] Chunk 1 [pause] 4-5) press ESC + CTRL+D (I'm on Linux) Chunk 1 [27] Chunk 2 [pause] So, big thanks , and sorry for taking longer to respond; I just get anxious about solving things sometimes and now it's kind of annoying I have to do some more research to fix this issue.
Re: `Socket.receive` providing arbitrary packet sizes and hanging without sending EOF
On 12/14/2017 11:55 AM, Unazed Spectaculum wrote: This is the only function which has a call to `receiveAll` I marked my changes with [Ali]: import std.stdio; import std.socket; import std.conv; import std.json; ubyte[] receiveBytes(T)(T socket, size_t receiveCount) { ubyte[] buffer = new ubyte[receiveCount]; size_t count = socket.receive(buffer); // [Ali] Return null when there is no data if (count == Socket.ERROR || count == 0) { return null; } return buffer[0 .. count]; } string receiveAll(T)(T socket, size_t segmentSize = 1024) { ubyte[][] data; size_t count = 0; do { debug(1) writefln("Chunk %s", count); auto part = receiveBytes(socket, segmentSize); // [Ali] Done when there is no data if (!part) { break; } data ~= part; writeln(data[count]); if (!data) break; } while(data[count++]); char[] stringData; foreach (elem; data) stringData ~= elem; debug(1) writeln(`Exiting "receiveAll"`); return to!string(stringData); } void main() { auto socket = new TcpSocket(); // [Ali]setupSocket(socket, "0.0.0.0", 6969); // The following had worked in a test program: socket.setOption(SocketOptionLevel.SOCKET, SocketOption.REUSEADDR, true); socket.bind(new InternetAddress(6969)); socket.listen(1); writefln("Listening: %s", socket.localAddress); while(true) { Socket client = socket.accept(); debug(1) writefln("Client: %s", client.remoteAddress); auto data = receiveAll(client); writeln(data); JSONValue json; try { json = parseJSON(data); } catch (JSONException e) { debug(1) writefln("Failed parsing data as JSON, aborting.\n|| %s", e); client.close(); continue; } catch (Exception e) { debug(1) writefln("Client caused exception:\n||%s", e); client.close(); continue; } /+ [Ali] Not important for this test if (!verifyValues(json)) { debug(1) writefln("Client missed out important key fields: %s", client.remoteAddress); client.close(); continue; } +/ debug(1) writeln("Client transacted successful JSON packet."); writefln("%s:\n\tFilename: %s\n\tMethod: %s\n\tData length: %d", client.remoteAddress, json["filename"], json["method"], data.length ); // [Ali] Not important for this test /+ if (json["method"].str == "store") storeData(json["filename"].str, json["data"].str); else if (json["method"].str == "retrieve") retrieveData(client, json["filename"].str); +/ client.close(); } } 1) Start the program 2) $ telnet 0.0.0.0 6969 Trying 0.0.0.0... Connected to 0.0.0.0. Escape character is '^]'. 3) Paste expected json to terminal: {"filename":"/tmp/foo","method":"bar"} 4) Press the escape character ^] 5) Press Ctrl-D (Ctrl-Z on Windows) to terminate the connection telnet> Connection closed. Here is the output from the program: Listening: 0.0.0.0:6969 [123, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 34, 47, 116, 109, 112, 47, 102, 111, 111, 34, 44, 34, 109, 101, 116, 104, 111, 100, 34, 58, 34, 98, 97, 114, 34, 125, 13, 10] {"filename":"/tmp/foo","method":"bar"} 127.0.0.1:47704: Filename: "\/tmp\/foo" Method: "bar" Data length: 40 I think it works! :) Ali
Re: `Socket.receive` providing arbitrary packet sizes and hanging without sending EOF
On Thursday, 14 December 2017 at 00:09:39 UTC, Ali Çehreli wrote: On 12/13/2017 11:39 AM, Unazed Spectaculum wrote: > ubyte[] receiveBytes(T)(T socket, size_t receiveCount) > { > ubyte[] buffer = new ubyte[receiveCount]; > size_t count = socket.receive(buffer); Don't trust code you find on newsgroups. :o) You have to check the returned value first. According to documentation, it can return Socket.ERROR: https://dlang.org/phobos/std_socket.html#.Socket.receive > there is always a superfluous chunk > which is awaiting data. Can you show with complete code? Perhaps the stream is in blocking mode? > No matter what way I try; my code doesn't seem to know when to quit > regardless of the check. Also for the arbitrary packet sizes, I would've > expected that if I received N bytes X times, the first X-1 times would > be perfectly N not some unusual integer. > Simply put, say I'm receiving 1024 bytes 5 times. The length of each > item on the stack looks like: > > [720, > 490, > 1024, > 103 > ] Posix read(2) man page says "It is not an error if this number is smaller than the number of bytes requested; this may happen for example because fewer bytes are actually available right now (maybe because we were close to end-of-file, or because we are reading from a pipe, or from a terminal), or because read() was interrupted by a signal." Ali void main() { auto socket = new TcpSocket(); setupSocket(socket, "0.0.0.0", 6969); writefln("Listening: %s", socket.localAddress); while(true) { Socket client = socket.accept(); debug(1) writefln("Client: %s", client.remoteAddress); auto data = receiveAll(client); writeln(data); JSONValue json; try { json = parseJSON(data); } catch (JSONException e) { debug(1) writefln("Failed parsing data as JSON, aborting.\n|| %s", e); client.close(); continue; } catch (Exception e) { debug(1) writefln("Client caused exception:\n||%s", e); client.close(); continue; } if (!verifyValues(json)) { debug(1) writefln("Client missed out important key fields: %s", client.remoteAddress); client.close(); continue; } debug(1) writeln("Client transacted successful JSON packet."); writefln("%s:\n\tFilename: %s\n\tMethod: %s\n\tData length: %d", client.remoteAddress, json["filename"], json["method"], data.length ); if (json["method"].str == "store") storeData(json["filename"].str, json["data"].str); else if (json["method"].str == "retrieve") retrieveData(client, json["filename"].str); client.close(); } } This is the only function which has a call to `receiveAll`, also yeah I don't typically check return codes for error values, I just assume it'll all work and if it doesn't a fresh restart will fix it; but I'll include it in my code just in case.
Re: `Socket.receive` providing arbitrary packet sizes and hanging without sending EOF
On 12/13/2017 11:39 AM, Unazed Spectaculum wrote: > ubyte[] receiveBytes(T)(T socket, size_t receiveCount) > { > ubyte[] buffer = new ubyte[receiveCount]; > size_t count = socket.receive(buffer); Don't trust code you find on newsgroups. :o) You have to check the returned value first. According to documentation, it can return Socket.ERROR: https://dlang.org/phobos/std_socket.html#.Socket.receive > there is always a superfluous chunk > which is awaiting data. Can you show with complete code? Perhaps the stream is in blocking mode? > No matter what way I try; my code doesn't seem to know when to quit > regardless of the check. Also for the arbitrary packet sizes, I would've > expected that if I received N bytes X times, the first X-1 times would > be perfectly N not some unusual integer. > Simply put, say I'm receiving 1024 bytes 5 times. The length of each > item on the stack looks like: > > [720, > 490, > 1024, > 103 > ] Posix read(2) man page says "It is not an error if this number is smaller than the number of bytes requested; this may happen for example because fewer bytes are actually available right now (maybe because we were close to end-of-file, or because we are reading from a pipe, or from a terminal), or because read() was interrupted by a signal." Ali