Re: Am I doing this right? (File byChunk)
I'll be sure to check it out once I get my phone line back (forgot to pay the bills..). I'm currently on expensive wireless (Why is wireless so expensive anyway? What's so special about radiowaves that were invented half a century ago?..) Thanks for the heads up! On 9/14/10, Jonathan M Davis jmdavisp...@gmx.com wrote: gmx.com seems to deal with mailing list e-mails correctly (by actually putting the ones you sent to the list in your inbox when they come from the list), and it has free imap and lots of disk space just like gmail (not to mention that it uses proper folders instead of labels), so I've switched over to it. It's much more pleasant to deal with (at least when using it from an e-mail client; whether the user interface for the site itself is better is more debatable). And switching e-mail addresses gives me a chance to reduce the spam that I get. :) - Jonathan M Davis
Re: Am I doing this right? (File byChunk)
gmx.com seems to deal with mailing list e-mails correctly (by actually putting the ones you sent to the list in your inbox when they come from the list), and it has free imap and lots of disk space just like gmail (not to mention that it uses proper folders instead of labels), so I've switched over to it. It's much more pleasant to deal with (at least when using it from an e-mail client; whether the user interface for the site itself is better is more debatable). And switching e-mail addresses gives me a chance to reduce the spam that I get. :) - Jonathan M Davis
Re: Am I doing this right? (File byChunk)
Andrej Mitrovic:
foreach (ubyte[] buf; file.byChunk(4096))
{
sendEditor(SCI_ADDTEXT, buf.length, cast(LPARAM)buf.ptr);
}
...
SCI_ADDTEXT(int length, const char *s)
Keep in mind that the length of a D array is a size_t, this means a 32 or 64
bit long unsigned word.
The ADDTEXT function creates a copy of the pointed-to contents to it's
internal scintilla buffers, so I don't have to keep any pointers
around after the call.
In byChunk() the content of buffer is reused across calls, so you are not
wasting allocations. I don't know if it's possible to use a fixed-size
char[4096] array to remove the first memory allocation too. I think byChunk()
needs a second optional argument, to give it a preallocated buffer (like a
slice of a fixed-size array).
Bye,
bearophile
Re: Am I doing this right? (File byChunk)
Yes, because LPARAM is defined in the DFL library as a long. Actually,
it's hardcoded, there's no static if or versioning. I'll keep an eye
on that for when DMD is able to build 64bit binaries.
On Mon, Sep 13, 2010 at 1:09 PM, bearophile bearophileh...@lycos.com wrote:
Andrej Mitrovic:
foreach (ubyte[] buf; file.byChunk(4096))
{
sendEditor(SCI_ADDTEXT, buf.length, cast(LPARAM)buf.ptr);
}
...
SCI_ADDTEXT(int length, const char *s)
Keep in mind that the length of a D array is a size_t, this means a 32 or 64
bit long unsigned word.
Re: Am I doing this right? (File byChunk)
That could be a good idea. On Mon, Sep 13, 2010 at 1:09 PM, bearophile bearophileh...@lycos.com wrote: In byChunk() the content of buffer is reused across calls, so you are not wasting allocations. I don't know if it's possible to use a fixed-size char[4096] array to remove the first memory allocation too. I think byChunk() needs a second optional argument, to give it a preallocated buffer (like a slice of a fixed-size array). Bye, bearophile
Am I doing this right? (File byChunk)
Here's a little snippet of code that interfaces with Scintilla (it works btw.):
File file = File(test.txt, r);
foreach (ubyte[] buf; file.byChunk(4096))
{
sendEditor(SCI_ADDTEXT, buf.length, (cast(char[])buf).idup);
}
The cast looks ugly, but I *have* to send a copy. Am I doing it right, or can I
make this a bit simpler? Otherwise I can use file.byLine, but I think this
would be much slower (unless D reads the entire contents at once into memory,
but I think it does not).
Actually, I shouldn't even complain. I can't believe how easy it is to open a
file, use buffers and send it all to a C interface. This is a 4-liner, the C
equivalent spans 20+ lines. Ha!
Re: Am I doing this right? (File byChunk)
Hello Andrej,
Here's a little snippet of code that interfaces with Scintilla (it
works btw.):
File file = File(test.txt, r);
foreach (ubyte[] buf; file.byChunk(4096))
{
sendEditor(SCI_ADDTEXT, buf.length, (cast(char[])buf).idup);
}
The cast looks ugly, but I *have* to send a copy.
What do those have to do with each other?
Am I doing it right,
It doesn't seem right to me either, because you are allocating memory in
chunks and passing it to C without keeping pointers around. Also what does
that function do with them? If it keeps them around, you might be faster
to load the whole file in one go and make a single call (fewer allocations,
fewer calls, no-extra memory usage, etc.)
or can I make this a bit simpler? Otherwise I can use file.byLine, but
I think this would be much slower (unless D reads the entire contents
at once into memory, but I think it does not).
Actually, I shouldn't even complain. I can't believe how easy it is to
open a file, use buffers and send it all to a C interface. This is a
4-liner, the C equivalent spans 20+ lines. Ha!
--
... IXOYE
Re: Am I doing this right? (File byChunk)
On Sunday 12 September 2010 18:04:14 Andrej Mitrovic wrote:
Here's a little snippet of code that interfaces with Scintilla (it works
btw.):
File file = File(test.txt, r);
foreach (ubyte[] buf; file.byChunk(4096))
{
sendEditor(SCI_ADDTEXT, buf.length, (cast(char[])buf).idup);
}
The cast looks ugly, but I *have* to send a copy. Am I doing it right, or
can I make this a bit simpler? Otherwise I can use file.byLine, but I
think this would be much slower (unless D reads the entire contents at
once into memory, but I think it does not).
I'd have to check, but I believe that byLine() reads the whole line into memory
but not the whole file.
- Jonathan M Davis
Re: Am I doing this right? (File byChunk)
Ah, silly me, disregard that code. I was working around the issue but
I didn't have to, here's the correct code:
foreach (ubyte[] buf; file.byChunk(4096))
{
sendEditor(SCI_ADDTEXT, buf.length, cast(LPARAM)buf.ptr);
}
Now, the explanation. The sendEditor function takes an opcode (uint),
and 2 arguments. It uses message passing to communicate with
scintilla, the two arguments are aliased to uint (which is usually a
length argument but is sometimes ignored) and LPARAM (a pointer's
address as ULONG).
Scintilla converts the address back to a pointer, and passes it (and
the first argument) to the relevant function based on the opcode.
Here's the header of that function:
SCI_ADDTEXT(int length, const char *s)
The reason my first code was working is because the wrapper has two
overloaded sendEditor functions. With one you can pass an opcode, a
length and pointer address, with the other you can pass an opcode, a
length and a string, and the function makes a copy of the string, adds
a null terminator and uses its address and the other arguments to send
a message.
The ADDTEXT function creates a copy of the pointed-to contents to it's
internal scintilla buffers, so I don't have to keep any pointers
around after the call.
I hope I've explained this right (and didn't mess something up). :p
On Mon, Sep 13, 2010 at 3:47 AM, BCS n...@anon.com wrote:
Hello Andrej,
Here's a little snippet of code that interfaces with Scintilla (it
works btw.):
File file = File(test.txt, r);
foreach (ubyte[] buf; file.byChunk(4096))
{
sendEditor(SCI_ADDTEXT, buf.length, (cast(char[])buf).idup);
}
The cast looks ugly, but I *have* to send a copy.
What do those have to do with each other?
Am I doing it right,
It doesn't seem right to me either, because you are allocating memory in
chunks and passing it to C without keeping pointers around. Also what does
that function do with them? If it keeps them around, you might be faster to
load the whole file in one go and make a single call (fewer allocations,
fewer calls, no-extra memory usage, etc.)
or can I make this a bit simpler? Otherwise I can use file.byLine, but
I think this would be much slower (unless D reads the entire contents
at once into memory, but I think it does not).
Actually, I shouldn't even complain. I can't believe how easy it is to
open a file, use buffers and send it all to a C interface. This is a
4-liner, the C equivalent spans 20+ lines. Ha!
--
... IXOYE
Re: Am I doing this right? (File byChunk)
I was refering to the open method, or when calling the constructor of
Text. But it doesn't actually *read* the contents until I ask it to
(which is a good thing). In other news it's late and I'm talking crazy
tonight, sorry. :)
On Mon, Sep 13, 2010 at 4:01 AM, Jonathan M Davis jmdavisp...@gmail.com wrote:
On Sunday 12 September 2010 18:04:14 Andrej Mitrovic wrote:
Here's a little snippet of code that interfaces with Scintilla (it works
btw.):
File file = File(test.txt, r);
foreach (ubyte[] buf; file.byChunk(4096))
{
sendEditor(SCI_ADDTEXT, buf.length, (cast(char[])buf).idup);
}
The cast looks ugly, but I *have* to send a copy. Am I doing it right, or
can I make this a bit simpler? Otherwise I can use file.byLine, but I
think this would be much slower (unless D reads the entire contents at
once into memory, but I think it does not).
I'd have to check, but I believe that byLine() reads the whole line into
memory
but not the whole file.
- Jonathan M Davis
Re: Am I doing this right? (File byChunk)
On Sunday 12 September 2010 19:31:23 Andrej Mitrovic wrote: I was refering to the open method, or when calling the constructor of Text. But it doesn't actually *read* the contents until I ask it to (which is a good thing). In other news it's late and I'm talking crazy tonight, sorry. :) Well, you do appear to be 9 hours ahead of me, time zone-wise, and you do appear to be up rather late. Entertainingly, the time difference makes for interesting timestamps in the messages as our respective clients put in the local time that a message is sent: On Mon, Sep 13, 2010 at 4:01 AM, Jonathan M Davis jmdavisp...@gmail.com wrote: On Sunday 12 September 2010 18:04:14 Andrej Mitrovic wrote: Mine also appears to be in 24 time, while yours is in 12 hour time, and mine seems to want to put second-precision, while yours only cares about minute- precision. The Date format is a fair bit different as well. Not that it really matters, but it's interesting nonetheless. - Jonathan M Davis
Re: Am I doing this right? (File byChunk)
I prefer 24-hour timestamps, but I've no idea where to set this in Gmail. And I'm not using a special newsgroup reader because they're all clumsy in their own little ways. On Mon, Sep 13, 2010 at 4:46 AM, Jonathan M Davis jmdavisp...@gmail.com wrote: On Sunday 12 September 2010 19:31:23 Andrej Mitrovic wrote: I was refering to the open method, or when calling the constructor of Text. But it doesn't actually *read* the contents until I ask it to (which is a good thing). In other news it's late and I'm talking crazy tonight, sorry. :) Well, you do appear to be 9 hours ahead of me, time zone-wise, and you do appear to be up rather late. Entertainingly, the time difference makes for interesting timestamps in the messages as our respective clients put in the local time that a message is sent: On Mon, Sep 13, 2010 at 4:01 AM, Jonathan M Davis jmdavisp...@gmail.com wrote: On Sunday 12 September 2010 18:04:14 Andrej Mitrovic wrote: Mine also appears to be in 24 time, while yours is in 12 hour time, and mine seems to want to put second-precision, while yours only cares about minute- precision. The Date format is a fair bit different as well. Not that it really matters, but it's interesting nonetheless. - Jonathan M Davis
Re: Am I doing this right? (File byChunk)
On Sunday 12 September 2010 19:56:28 Andrej Mitrovic wrote: I prefer 24-hour timestamps, but I've no idea where to set this in Gmail. And I'm not using a special newsgroup reader because they're all clumsy in their own little ways. I'm using a gmail account, but I'm using kmail to manage my mail. So, the timestamps and whatnot are going to depend on my system settings. I like knode (KDE's newsgroup reader) overall, but I stopped using it because I had no way of syncing between machines without copying all of knode's files over every time, which is not pleasant. Using gmail/kmail with IMAP allows for keeping things in sync. Unfortunately, since gmail never sends you messages from the list that you sent to it, it doesn't really work properly to send messages which reply to your own messages, and kmail threads replies to my messages poorly (usually associating them with the message that I responded to). So, I'd prefer to use knode, but the lack of syncing was too painful. I suppose that it would work better if I used a non-gmail account though. I wouldn't know what to use though, since I'd want free IMAP support. And another provider would use proper folders rather than the annoying labels which mail programs like kmail don't deal with quite properly. So maybe I should think of finding a better e-mail provider than gmail... - Jonathan M Davis
