Re: `Socket.receive` providing arbitrary packet sizes and hanging without sending EOF
On Thursday, 14 December 2017 at 20:27:36 UTC, Ali Çehreli wrote:
On 12/14/2017 11:55 AM, Unazed Spectaculum wrote:
This is the only function which has a call to `receiveAll`
I marked my changes with [Ali]:
import std.stdio;
import std.socket;
import std.conv;
import std.json;
ubyte[] receiveBytes(T)(T socket, size_t receiveCount)
{
ubyte[] buffer = new ubyte[receiveCount];
size_t count = socket.receive(buffer);
// [Ali] Return null when there is no data
if (count == Socket.ERROR || count == 0) {
return null;
}
return buffer[0 .. count];
}
string receiveAll(T)(T socket, size_t segmentSize = 1024)
{
ubyte[][] data;
size_t count = 0;
do
{
debug(1) writefln("Chunk %s", count);
auto part = receiveBytes(socket, segmentSize);
// [Ali] Done when there is no data
if (!part) {
break;
}
data ~= part;
writeln(data[count]);
if (!data)
break;
} while(data[count++]);
char[] stringData;
foreach (elem; data)
stringData ~= elem;
debug(1) writeln(`Exiting "receiveAll"`);
return to!string(stringData);
}
void main()
{
auto socket = new TcpSocket();
// [Ali]setupSocket(socket, "0.0.0.0", 6969);
// The following had worked in a test program:
socket.setOption(SocketOptionLevel.SOCKET,
SocketOption.REUSEADDR, true);
socket.bind(new InternetAddress(6969));
socket.listen(1);
writefln("Listening: %s", socket.localAddress);
while(true)
{
Socket client = socket.accept();
debug(1) writefln("Client: %s", client.remoteAddress);
auto data = receiveAll(client);
writeln(data);
JSONValue json;
try {
json = parseJSON(data);
} catch (JSONException e) {
debug(1) writefln("Failed parsing data as JSON,
aborting.\n|| %s", e);
client.close();
continue;
} catch (Exception e) {
debug(1) writefln("Client caused exception:\n||%s",
e);
client.close();
continue;
}
/+ [Ali] Not important for this test
if (!verifyValues(json))
{
debug(1) writefln("Client missed out important key
fields: %s", client.remoteAddress);
client.close();
continue;
}
+/
debug(1) writeln("Client transacted successful JSON
packet.");
writefln("%s:\n\tFilename: %s\n\tMethod: %s\n\tData
length: %d",
client.remoteAddress,
json["filename"],
json["method"],
data.length
);
// [Ali] Not important for this test
/+
if (json["method"].str == "store")
storeData(json["filename"].str, json["data"].str);
else if (json["method"].str == "retrieve")
retrieveData(client, json["filename"].str);
+/
client.close();
}
}
1) Start the program
2) $ telnet 0.0.0.0 6969
Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.
3) Paste expected json to terminal:
{"filename":"/tmp/foo","method":"bar"}
4) Press the escape character
^]
5) Press Ctrl-D (Ctrl-Z on Windows) to terminate the connection
telnet> Connection closed.
Here is the output from the program:
Listening: 0.0.0.0:6969
[123, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 34,
47, 116, 109, 112, 47, 102, 111, 111, 34, 44, 34, 109, 101,
116, 104, 111, 100, 34, 58, 34, 98, 97, 114, 34, 125, 13, 10]
{"filename":"/tmp/foo","method":"bar"}
127.0.0.1:47704:
Filename: "\/tmp\/foo"
Method: "bar"
Data length: 40
I think it works! :)
Ali
I've tried to integrate your code with mine; to no avail, I've
directly tested your code; and yet again to no avail, so this
should conclude the thread as it pinpoints an error with my
actual OS.
Just in case, as a last resort in case I'm actually that dumb,
here's me reproducing your steps:
1) Starting program
unazed@unazed /home/d/storage-server dmd -debug -run app.d
Listening: 0.0.0.0:6969
2) telnet to the server
unazed@unazed ~ telnet 0.0.0.0 6969
Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.
3) paste '{"filename":"/tmp/foo","method":"bar"}'
unazed@unazed /home/d/storage-server dmd -debug -run app.d
Listening: 0.0.0.0:6969
Client: 127.0.0.1:55014
Chunk 0
[123, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 34, 47,
116, 109, 112, 47, 102, 111, 111, 34, 44, 34, 109, 101, 116, 104,
111, 100, 34, 58, 34, 98, 97, 114, 34, 125, 13, 10]
Chunk 1
[pause]
4-5) press ESC + CTRL+D (I'm on Linux)
Chunk 1
[27]
Chunk 2
[pause]
So, big thanks , and sorry for taking longer to respond; I just
get anxious about solving things sometimes and now it's kind of
annoying I have to do some more research to fix this issue.
Re: `Socket.receive` providing arbitrary packet sizes and hanging without sending EOF
On Thursday, 14 December 2017 at 00:09:39 UTC, Ali Çehreli wrote:
On 12/13/2017 11:39 AM, Unazed Spectaculum wrote:
> ubyte[] receiveBytes(T)(T socket, size_t receiveCount)
> {
> ubyte[] buffer = new ubyte[receiveCount];
> size_t count = socket.receive(buffer);
Don't trust code you find on newsgroups. :o) You have to check
the returned value first. According to documentation, it can
return Socket.ERROR:
https://dlang.org/phobos/std_socket.html#.Socket.receive
> there is always a superfluous chunk
> which is awaiting data.
Can you show with complete code? Perhaps the stream is in
blocking mode?
> No matter what way I try; my code doesn't seem to know when
to quit
> regardless of the check. Also for the arbitrary packet sizes,
I would've
> expected that if I received N bytes X times, the first X-1
times would
> be perfectly N not some unusual integer.
> Simply put, say I'm receiving 1024 bytes 5 times. The length
of each
> item on the stack looks like:
>
> [720,
> 490,
> 1024,
> 103
> ]
Posix read(2) man page says
"It is not an error if this number is smaller than the number
of bytes
requested; this may happen for example because fewer bytes are
actually
available right now (maybe because we were close to
end-of-file, or because
we are reading from a pipe, or from a terminal), or because
read() was
interrupted by a signal."
Ali
void main()
{
auto socket = new TcpSocket();
setupSocket(socket, "0.0.0.0", 6969);
writefln("Listening: %s", socket.localAddress);
while(true)
{
Socket client = socket.accept();
debug(1) writefln("Client: %s", client.remoteAddress);
auto data = receiveAll(client);
writeln(data);
JSONValue json;
try {
json = parseJSON(data);
} catch (JSONException e) {
debug(1) writefln("Failed parsing data as JSON, aborting.\n||
%s", e);
client.close();
continue;
} catch (Exception e) {
debug(1) writefln("Client caused exception:\n||%s", e);
client.close();
continue;
}
if (!verifyValues(json))
{
debug(1) writefln("Client missed out important key fields:
%s", client.remoteAddress);
client.close();
continue;
}
debug(1) writeln("Client transacted successful JSON packet.");
writefln("%s:\n\tFilename: %s\n\tMethod: %s\n\tData length: %d",
client.remoteAddress,
json["filename"],
json["method"],
data.length
);
if (json["method"].str == "store")
storeData(json["filename"].str, json["data"].str);
else if (json["method"].str == "retrieve")
retrieveData(client, json["filename"].str);
client.close();
}
}
This is the only function which has a call to `receiveAll`, also
yeah I don't typically check return codes for error values, I
just assume it'll all work and if it doesn't a fresh restart will
fix it; but I'll include it in my code just in case.
`Socket.receive` providing arbitrary packet sizes and hanging without sending EOF
ubyte[] receiveBytes(T)(T socket, size_t receiveCount)
{
ubyte[] buffer = new ubyte[receiveCount];
size_t count = socket.receive(buffer);
return buffer[0 .. count];
}
string receiveAll(T)(T socket, size_t segmentSize = 1024)
{
ubyte[][] data;
size_t count = 0;
do
{
debug(1) writefln("Chunk %s", count);
data ~= receiveBytes(socket, segmentSize);
writeln(data[count]);
if (!data)
break;
} while(data[count++]);
char[] stringData;
foreach (elem; data)
stringData ~= elem;
debug(1) writeln(`Exiting "receiveAll"`);
return to!string(stringData);
}
I've tried many variations of the above code; both with the
retrieve integrated into the do-while loop in receiveAll; however
I was just seeing whether abstracting the code would let me spot
my issues a bit faster because it won't require me to make two
variables for the buffer and amount of bytes received, more like
the typical interface you get.
Issue is; when receiving any size buffer from the end-point
(tested with >1024 byte and <1024 byte buffers); there is always
a superfluous chunk which is awaiting data.
Listening: 0.0.0.0:6969
Client: somebody:58769
Chunk 0
[123, 34, 109, 101, 116, 104, 111, 100, 34, 58, 32, 34, 114, 101,
116, 114, 105, 101, 118, 101, 34, 44, 32, 34, 102, 105, 108, 101,
110, 97, 109, 101, 34, 58, 32, 34, 51, 48, 50, 54, 57, 50, 48,
49, 55, 50, 51, 54, 56, 54, 57, 49, 50, 49, 95, 97, 97, 97, 97,
97, 34, 44, 32, 34, 100, 97, 116, 97, 34, 58, 32, 34, 34, 125]
Chunk 1
[PAUSE]
Listening: 0.0.0.0:6969
Client: somebody:58767
Chunk 0
[123, 34, 109, 101, 116, 104, 111, 100, 34, 58, 32, 34, 114, 101,
116, 114, 105, 101, 118, 101, 34, 44, 32, 34, 102, 105, 108, 101,
110, 97, 109, 101, 34, 58, 32, 34, ... 97]
Chunk 1
[97, ... 125]
Chunk 2
[PAUSE]
No matter what way I try; my code doesn't seem to know when to
quit regardless of the check. Also for the arbitrary packet
sizes, I would've expected that if I received N bytes X times,
the first X-1 times would be perfectly N not some unusual integer.
Simply put, say I'm receiving 1024 bytes 5 times. The length of
each item on the stack looks like:
[720,
490,
1024,
103
]
Although I'd assume it'd be more like:
[1024,
1024,
289
]
What's up with this? It makes working with sockets so damn
tedious since there's no room for assumptions; can anybody
suggest an abstracted library for sockets in D or help with my
former issue?
Re: how would I go about creating a Socket receiveAll method?
On Tuesday, 12 December 2017 at 21:14:30 UTC, Adam D. Ruppe wrote:
On Tuesday, 12 December 2017 at 21:03:54 UTC, Unazed
Spectaculum wrote:
I've decided to go for the run-time approach to this, it works
fine with all of my tests so you have my greatest gratitude.
the way I'd do it btw is simply:
ubyte[] result;
ubyte[1024] buffer;
auto got = socket.receive(buffer[]);
while(got > 0) {
result ~= buffer[0 .. got];
}
if(got < 0)
throw new Exception(lastSocketError());
return result;
so it uses the one static buffer to receive the stuff one block
at a time but just copies it over to the dynamic array with the
~= operator
Oh, thanks; it's definitely shorter however I do enjoy the
ability to generalize/ambiguate functions by providing optional
parameters, however thanks for showing another way; I enjoy
knowing multiple ways of performing one task so anything helps :)
Re: how would I go about creating a Socket receiveAll method?
On Tuesday, 12 December 2017 at 20:27:04 UTC, Ali Çehreli wrote:
On 12/12/2017 12:10 PM, Unazed Spectaculum wrote:
> string receiveAll(T)(T socket, int segment_size = 1024)
> {
> char[segment_size][] data;
Unrelated, you most likely want to use ubyte. (char is for
UTF-8.)
The problem is, char[segment_size] is a static array, where the
length must be known at compile time because length is a part
of its type.
So, depending on what you need you have two options:
a) Use dynamic array if the length is known at run time
b) Although (a) will work just fine, use template parameter for
length if the length is known at compile time and you want to
avoid dynamic allocation.
However, too large arrays won't fit on the stack. (Further
however, your 'data' is a slice anyway, just the elements are
static.)
The following program shows the two options with
one-dimensional arrays:
// Size is known at run time
void foo(T)(T t, size_t size = 1024) {
auto data = new ubyte[](size);
}
// Size is known at compile time
void bar(size_t size = 1024, T)(T t) {
ubyte[size] data;
}
void main() {
int i;
foo(i, 10);
bar!20(i);
}
Here is one with two-dimensional arrays:
import std.stdio;
size_t counter = 0;
bool done() {
return (++counter % 4) == 0;
}
// Size is known at run time
void foo(T)(T t, size_t size = 1024) {
ubyte[][] data;
while (!done) {
data ~= new ubyte[size];
// Use data[$-1]
writeln("a) Will read here: ", data[$-1]);
}
}
// Size is known at compile time
void bar(size_t size = 1024, T)(T t) {
ubyte[size][] data;
while (!done) {
++data.length;
writeln("b) Will read here: ", data[$-1]);
}
}
void main() {
int i;
foo(i, 10);
bar!20(i);
}
Ali
string receiveAll(T)(T socket, size_t segment_size = 1024)
{
ubyte[][] data;
size_t count = 0;
while (true)
{
data ~= new ubyte[segment_size];
auto received = socket.receive(data[count]);
data[count] = data[count][0 .. received];
if (!received)
break;
else if (received < segment_size)
break; /* early exit */
++count;
}
char[] stringData;
foreach (elem; data)
stringData ~= elem;
return to!string(stringData);
}
I've decided to go for the run-time approach to this, it works
fine with all of my tests so you have my greatest gratitude.
I might have created some weird inefficiencies but don't worry
take time telling me about them unless they're going to blow up
my program since I think you've explained enough already :D.
Since I'm only a few days into D I wouldn't expect much of my
code, I'm moreover from the generic Python and thereabouts C-ish
background.
again, thanks.
how would I go about creating a Socket receiveAll method?
string receiveAll(T)(T socket, int segment_size = 1024)
{
char[segment_size][] data;
int cnt = 0;
while(true)
{
auto received = socket.receive(data[cnt]);
if (received < segment_size)
break; /* early exit */
else if (!received)
break;
++cnt;
}
return data;
}
This is my theoretical function, it errors at
`char[segment_size][] data;` with the painful `app.d(20): Error:
variable segment_size cannot be read at compile time` and I
recall having an issue similar to this earlier (yesterday) but I
don't think any of my solutions seemed valid for this situation.
I understand it's to do with CTFE or some form of compile-time
checking but that's really the only thing that annoys me about D,
perhaps somebody could link to some resource that explains (not
shortly) how to make the D compiler evaluate some things at
run-time opposed to compile time.
Perhaps somebody can link some resources e.g. socket servers in D
so I can learn how it's implemented by somebody with a bit more
experience, or some resources on how to use sockets properly
¯\_(ツ)_/¯
Re: `string[] args` can't be read at compile time?
On Sunday, 10 December 2017 at 15:46:57 UTC, Adam D. Ruppe wrote: On Sunday, 10 December 2017 at 15:28:19 UTC, Unazed Spectaculum wrote: if (!exists!(args[1])) That should be `!exists(args[1])`. You had an extra ! in there by the (. Generally speaking, when there's a "cannot be read at compile time", it is because something is initialized in a static context and/or there's an extra ! in the arg list to get rid of. Thanks! That fixed it. I'll take it into account whenever I see any other future errors like it.
`string[] args` can't be read at compile time?
import std.stdio;
import std.file;
void main(string[] args)
{
if (args.length != 2)
{
writeln("usage: ./app ");
return;
}
if (!exists!(args[1]))
{
writefln("fatal error: %s doesn't exist", args[1]);
}
}
Line 13 (if (!exists!(args[1]))) produces error:
'app.d(13): Error: variable args cannot be read at compile time'
With 'dmd -run app.d' compilation command. I've seen another
Stackoverflow post with a similar issue
(https://stackoverflow.com/questions/39920780/variable-i-cannot-be-read-at-compile-time) but I don't believe it to be related.
Any help?
