Re: `Socket.receive` providing arbitrary packet sizes and hanging without sending EOF
On 12/16/2017 05:21 PM, Unazed Spectaculum wrote:
> 1) Starting program
> unazed@unazed /home/d/storage-server dmd -debug -run app.d
Although I don't normally use the -run switch, as expected, it works
with -run as well. (More below.)
> Listening: 0.0.0.0:6969
>
> 2) telnet to the server
> unazed@unazed ~ telnet 0.0.0.0 6969
> Trying 0.0.0.0...
> Connected to 0.0.0.0.
> Escape character is '^]'.
Note that line! ;)
>
> 3) paste '{"filename":"/tmp/foo","method":"bar"}'
Please also press Enter after that. I don't know what code is at fault
but the json parser was not happy without that newline at the end.
> unazed@unazed /home/d/storage-server dmd -debug -run app.d
> Listening: 0.0.0.0:6969
> Client: 127.0.0.1:55014
> Chunk 0
> [123, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 34, 47, 116,
> 109, 112, 47, 102, 111, 111, 34, 44, 34, 109, 101, 116, 104, 111, 100,
> 34, 58, 34, 98, 97, 114, 34, 125, 13, 10]
> Chunk 1
> [pause]
Same here...
> 4-5) press ESC + CTRL+D (I'm on Linux)
Aha! That's the problem! You should enter the "Escape character", which
is ^] as telnet indicates. You should press Ctrl-] (stay away from the
ESC key). Otherwise, you're injecting an ESC character to the
communication stream. (The 27 below.)
By the way, I'm on Linux as well! :) (Mint, Ubuntu based.)
And finally, press Ctrl-D to end the stream. It really works:
[...]
Chunk 1
Exiting "receiveAll"
{"filename":"/tmp/foo","method":"bar"}
[...]
Ali
Re: `Socket.receive` providing arbitrary packet sizes and hanging without sending EOF
On Thursday, 14 December 2017 at 20:27:36 UTC, Ali Çehreli wrote:
On 12/14/2017 11:55 AM, Unazed Spectaculum wrote:
This is the only function which has a call to `receiveAll`
I marked my changes with [Ali]:
import std.stdio;
import std.socket;
import std.conv;
import std.json;
ubyte[] receiveBytes(T)(T socket, size_t receiveCount)
{
ubyte[] buffer = new ubyte[receiveCount];
size_t count = socket.receive(buffer);
// [Ali] Return null when there is no data
if (count == Socket.ERROR || count == 0) {
return null;
}
return buffer[0 .. count];
}
string receiveAll(T)(T socket, size_t segmentSize = 1024)
{
ubyte[][] data;
size_t count = 0;
do
{
debug(1) writefln("Chunk %s", count);
auto part = receiveBytes(socket, segmentSize);
// [Ali] Done when there is no data
if (!part) {
break;
}
data ~= part;
writeln(data[count]);
if (!data)
break;
} while(data[count++]);
char[] stringData;
foreach (elem; data)
stringData ~= elem;
debug(1) writeln(`Exiting "receiveAll"`);
return to!string(stringData);
}
void main()
{
auto socket = new TcpSocket();
// [Ali]setupSocket(socket, "0.0.0.0", 6969);
// The following had worked in a test program:
socket.setOption(SocketOptionLevel.SOCKET,
SocketOption.REUSEADDR, true);
socket.bind(new InternetAddress(6969));
socket.listen(1);
writefln("Listening: %s", socket.localAddress);
while(true)
{
Socket client = socket.accept();
debug(1) writefln("Client: %s", client.remoteAddress);
auto data = receiveAll(client);
writeln(data);
JSONValue json;
try {
json = parseJSON(data);
} catch (JSONException e) {
debug(1) writefln("Failed parsing data as JSON,
aborting.\n|| %s", e);
client.close();
continue;
} catch (Exception e) {
debug(1) writefln("Client caused exception:\n||%s",
e);
client.close();
continue;
}
/+ [Ali] Not important for this test
if (!verifyValues(json))
{
debug(1) writefln("Client missed out important key
fields: %s", client.remoteAddress);
client.close();
continue;
}
+/
debug(1) writeln("Client transacted successful JSON
packet.");
writefln("%s:\n\tFilename: %s\n\tMethod: %s\n\tData
length: %d",
client.remoteAddress,
json["filename"],
json["method"],
data.length
);
// [Ali] Not important for this test
/+
if (json["method"].str == "store")
storeData(json["filename"].str, json["data"].str);
else if (json["method"].str == "retrieve")
retrieveData(client, json["filename"].str);
+/
client.close();
}
}
1) Start the program
2) $ telnet 0.0.0.0 6969
Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.
3) Paste expected json to terminal:
{"filename":"/tmp/foo","method":"bar"}
4) Press the escape character
^]
5) Press Ctrl-D (Ctrl-Z on Windows) to terminate the connection
telnet> Connection closed.
Here is the output from the program:
Listening: 0.0.0.0:6969
[123, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 34,
47, 116, 109, 112, 47, 102, 111, 111, 34, 44, 34, 109, 101,
116, 104, 111, 100, 34, 58, 34, 98, 97, 114, 34, 125, 13, 10]
{"filename":"/tmp/foo","method":"bar"}
127.0.0.1:47704:
Filename: "\/tmp\/foo"
Method: "bar"
Data length: 40
I think it works! :)
Ali
I've tried to integrate your code with mine; to no avail, I've
directly tested your code; and yet again to no avail, so this
should conclude the thread as it pinpoints an error with my
actual OS.
Just in case, as a last resort in case I'm actually that dumb,
here's me reproducing your steps:
1) Starting program
unazed@unazed /home/d/storage-server dmd -debug -run app.d
Listening: 0.0.0.0:6969
2) telnet to the server
unazed@unazed ~ telnet 0.0.0.0 6969
Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.
3) paste '{"filename":"/tmp/foo","method":"bar"}'
unazed@unazed /home/d/storage-server dmd -debug -run app.d
Listening: 0.0.0.0:6969
Client: 127.0.0.1:55014
Chunk 0
[123, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 34, 47,
116, 109, 112, 47, 102, 111, 111, 34, 44, 34, 109, 101, 116, 104,
111, 100, 34, 58, 34, 98, 97, 114, 34, 125, 13, 10]
Chunk 1
[pause]
4-5) press ESC + CTRL+D (I'm on Linux)
Chunk 1
[27]
Chunk 2
[pause]
So, big thanks , and sorry for taking longer to respond; I just
get anxious about solving things sometimes and now it's kind of
annoying I have to do some more research to fix this issue.
Re: `Socket.receive` providing arbitrary packet sizes and hanging without sending EOF
On 12/14/2017 11:55 AM, Unazed Spectaculum wrote:
This is the only function which has a call to `receiveAll`
I marked my changes with [Ali]:
import std.stdio;
import std.socket;
import std.conv;
import std.json;
ubyte[] receiveBytes(T)(T socket, size_t receiveCount)
{
ubyte[] buffer = new ubyte[receiveCount];
size_t count = socket.receive(buffer);
// [Ali] Return null when there is no data
if (count == Socket.ERROR || count == 0) {
return null;
}
return buffer[0 .. count];
}
string receiveAll(T)(T socket, size_t segmentSize = 1024)
{
ubyte[][] data;
size_t count = 0;
do
{
debug(1) writefln("Chunk %s", count);
auto part = receiveBytes(socket, segmentSize);
// [Ali] Done when there is no data
if (!part) {
break;
}
data ~= part;
writeln(data[count]);
if (!data)
break;
} while(data[count++]);
char[] stringData;
foreach (elem; data)
stringData ~= elem;
debug(1) writeln(`Exiting "receiveAll"`);
return to!string(stringData);
}
void main()
{
auto socket = new TcpSocket();
// [Ali]setupSocket(socket, "0.0.0.0", 6969);
// The following had worked in a test program:
socket.setOption(SocketOptionLevel.SOCKET, SocketOption.REUSEADDR,
true);
socket.bind(new InternetAddress(6969));
socket.listen(1);
writefln("Listening: %s", socket.localAddress);
while(true)
{
Socket client = socket.accept();
debug(1) writefln("Client: %s", client.remoteAddress);
auto data = receiveAll(client);
writeln(data);
JSONValue json;
try {
json = parseJSON(data);
} catch (JSONException e) {
debug(1) writefln("Failed parsing data as JSON,
aborting.\n|| %s", e);
client.close();
continue;
} catch (Exception e) {
debug(1) writefln("Client caused exception:\n||%s", e);
client.close();
continue;
}
/+ [Ali] Not important for this test
if (!verifyValues(json))
{
debug(1) writefln("Client missed out important key fields:
%s", client.remoteAddress);
client.close();
continue;
}
+/
debug(1) writeln("Client transacted successful JSON packet.");
writefln("%s:\n\tFilename: %s\n\tMethod: %s\n\tData length: %d",
client.remoteAddress,
json["filename"],
json["method"],
data.length
);
// [Ali] Not important for this test
/+
if (json["method"].str == "store")
storeData(json["filename"].str, json["data"].str);
else if (json["method"].str == "retrieve")
retrieveData(client, json["filename"].str);
+/
client.close();
}
}
1) Start the program
2) $ telnet 0.0.0.0 6969
Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.
3) Paste expected json to terminal:
{"filename":"/tmp/foo","method":"bar"}
4) Press the escape character
^]
5) Press Ctrl-D (Ctrl-Z on Windows) to terminate the connection
telnet> Connection closed.
Here is the output from the program:
Listening: 0.0.0.0:6969
[123, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 34, 47, 116,
109, 112, 47, 102, 111, 111, 34, 44, 34, 109, 101, 116, 104, 111, 100,
34, 58, 34, 98, 97, 114, 34, 125, 13, 10]
{"filename":"/tmp/foo","method":"bar"}
127.0.0.1:47704:
Filename: "\/tmp\/foo"
Method: "bar"
Data length: 40
I think it works! :)
Ali
Re: `Socket.receive` providing arbitrary packet sizes and hanging without sending EOF
On Thursday, 14 December 2017 at 00:09:39 UTC, Ali Çehreli wrote:
On 12/13/2017 11:39 AM, Unazed Spectaculum wrote:
> ubyte[] receiveBytes(T)(T socket, size_t receiveCount)
> {
> ubyte[] buffer = new ubyte[receiveCount];
> size_t count = socket.receive(buffer);
Don't trust code you find on newsgroups. :o) You have to check
the returned value first. According to documentation, it can
return Socket.ERROR:
https://dlang.org/phobos/std_socket.html#.Socket.receive
> there is always a superfluous chunk
> which is awaiting data.
Can you show with complete code? Perhaps the stream is in
blocking mode?
> No matter what way I try; my code doesn't seem to know when
to quit
> regardless of the check. Also for the arbitrary packet sizes,
I would've
> expected that if I received N bytes X times, the first X-1
times would
> be perfectly N not some unusual integer.
> Simply put, say I'm receiving 1024 bytes 5 times. The length
of each
> item on the stack looks like:
>
> [720,
> 490,
> 1024,
> 103
> ]
Posix read(2) man page says
"It is not an error if this number is smaller than the number
of bytes
requested; this may happen for example because fewer bytes are
actually
available right now (maybe because we were close to
end-of-file, or because
we are reading from a pipe, or from a terminal), or because
read() was
interrupted by a signal."
Ali
void main()
{
auto socket = new TcpSocket();
setupSocket(socket, "0.0.0.0", 6969);
writefln("Listening: %s", socket.localAddress);
while(true)
{
Socket client = socket.accept();
debug(1) writefln("Client: %s", client.remoteAddress);
auto data = receiveAll(client);
writeln(data);
JSONValue json;
try {
json = parseJSON(data);
} catch (JSONException e) {
debug(1) writefln("Failed parsing data as JSON, aborting.\n||
%s", e);
client.close();
continue;
} catch (Exception e) {
debug(1) writefln("Client caused exception:\n||%s", e);
client.close();
continue;
}
if (!verifyValues(json))
{
debug(1) writefln("Client missed out important key fields:
%s", client.remoteAddress);
client.close();
continue;
}
debug(1) writeln("Client transacted successful JSON packet.");
writefln("%s:\n\tFilename: %s\n\tMethod: %s\n\tData length: %d",
client.remoteAddress,
json["filename"],
json["method"],
data.length
);
if (json["method"].str == "store")
storeData(json["filename"].str, json["data"].str);
else if (json["method"].str == "retrieve")
retrieveData(client, json["filename"].str);
client.close();
}
}
This is the only function which has a call to `receiveAll`, also
yeah I don't typically check return codes for error values, I
just assume it'll all work and if it doesn't a fresh restart will
fix it; but I'll include it in my code just in case.
Re: `Socket.receive` providing arbitrary packet sizes and hanging without sending EOF
On 12/13/2017 11:39 AM, Unazed Spectaculum wrote:
> ubyte[] receiveBytes(T)(T socket, size_t receiveCount)
> {
> ubyte[] buffer = new ubyte[receiveCount];
> size_t count = socket.receive(buffer);
Don't trust code you find on newsgroups. :o) You have to check the
returned value first. According to documentation, it can return
Socket.ERROR:
https://dlang.org/phobos/std_socket.html#.Socket.receive
> there is always a superfluous chunk
> which is awaiting data.
Can you show with complete code? Perhaps the stream is in blocking mode?
> No matter what way I try; my code doesn't seem to know when to quit
> regardless of the check. Also for the arbitrary packet sizes, I would've
> expected that if I received N bytes X times, the first X-1 times would
> be perfectly N not some unusual integer.
> Simply put, say I'm receiving 1024 bytes 5 times. The length of each
> item on the stack looks like:
>
> [720,
> 490,
> 1024,
> 103
> ]
Posix read(2) man page says
"It is not an error if this number is smaller than the number of bytes
requested; this may happen for example because fewer bytes are actually
available right now (maybe because we were close to end-of-file, or because
we are reading from a pipe, or from a terminal), or because read() was
interrupted by a signal."
Ali
`Socket.receive` providing arbitrary packet sizes and hanging without sending EOF
ubyte[] receiveBytes(T)(T socket, size_t receiveCount)
{
ubyte[] buffer = new ubyte[receiveCount];
size_t count = socket.receive(buffer);
return buffer[0 .. count];
}
string receiveAll(T)(T socket, size_t segmentSize = 1024)
{
ubyte[][] data;
size_t count = 0;
do
{
debug(1) writefln("Chunk %s", count);
data ~= receiveBytes(socket, segmentSize);
writeln(data[count]);
if (!data)
break;
} while(data[count++]);
char[] stringData;
foreach (elem; data)
stringData ~= elem;
debug(1) writeln(`Exiting "receiveAll"`);
return to!string(stringData);
}
I've tried many variations of the above code; both with the
retrieve integrated into the do-while loop in receiveAll; however
I was just seeing whether abstracting the code would let me spot
my issues a bit faster because it won't require me to make two
variables for the buffer and amount of bytes received, more like
the typical interface you get.
Issue is; when receiving any size buffer from the end-point
(tested with >1024 byte and <1024 byte buffers); there is always
a superfluous chunk which is awaiting data.
Listening: 0.0.0.0:6969
Client: somebody:58769
Chunk 0
[123, 34, 109, 101, 116, 104, 111, 100, 34, 58, 32, 34, 114, 101,
116, 114, 105, 101, 118, 101, 34, 44, 32, 34, 102, 105, 108, 101,
110, 97, 109, 101, 34, 58, 32, 34, 51, 48, 50, 54, 57, 50, 48,
49, 55, 50, 51, 54, 56, 54, 57, 49, 50, 49, 95, 97, 97, 97, 97,
97, 34, 44, 32, 34, 100, 97, 116, 97, 34, 58, 32, 34, 34, 125]
Chunk 1
[PAUSE]
Listening: 0.0.0.0:6969
Client: somebody:58767
Chunk 0
[123, 34, 109, 101, 116, 104, 111, 100, 34, 58, 32, 34, 114, 101,
116, 114, 105, 101, 118, 101, 34, 44, 32, 34, 102, 105, 108, 101,
110, 97, 109, 101, 34, 58, 32, 34, ... 97]
Chunk 1
[97, ... 125]
Chunk 2
[PAUSE]
No matter what way I try; my code doesn't seem to know when to
quit regardless of the check. Also for the arbitrary packet
sizes, I would've expected that if I received N bytes X times,
the first X-1 times would be perfectly N not some unusual integer.
Simply put, say I'm receiving 1024 bytes 5 times. The length of
each item on the stack looks like:
[720,
490,
1024,
103
]
Although I'd assume it'd be more like:
[1024,
1024,
289
]
What's up with this? It makes working with sockets so damn
tedious since there's no room for assumptions; can anybody
suggest an abstracted library for sockets in D or help with my
former issue?
