Re: malloc error when trying to assign the returned pointer to a struct field
On Friday, 8 September 2023 at 19:14:47 UTC, H. S. Teoh wrote: My guess is that you have a double-free somewhere, or there's a buffer overrun. Or maybe some bad interaction with the GC, e.g. if you tried to free a pointer from the GC heap. (Note that this may not immediately show up; free() could've assumed that everything was OK when it has in fact messed up its internal data structures; the problem would only show up later on in code that's actually unrelated to the real problem.) Wait! I did searched on the web! I cannot find anything about a "buffer overrun"! I only find about "buffer overflow".
Re: malloc error when trying to assign the returned pointer to a struct field
On Friday, 8 September 2023 at 19:14:47 UTC, H. S. Teoh wrote: The error message looks to me like a corruption of the malloc heap. These kinds of bugs are very hard to trace, because they may go undetected and only show up in specific circumstances, so small perturbations of completely unrelated code may make the bug appear or disappear -- just because the bug doesn't show up when you disable some code does not prove that that's where the problem is; it could be that corruption is still happening, it just so happens that it goes unnoticed when the behaviour of the code changes slightly. Yep! That's what I guess as well! Tbh, I knew that playing with memory and trying to write a system library will be a touch job and I'm up for it! But these bugs really burn me out because like you said, nobody can truly help because they are so weird... My guess is that you have a double-free somewhere, or there's a buffer overrun. Or maybe some bad interaction with the GC, e.g. if you tried to free a pointer from the GC heap. (Note that this may not immediately show up; free() could've assumed that everything was OK when it has in fact messed up its internal data structures; the problem would only show up later on in code that's actually unrelated to the real problem.) I have commented out every `free` at this point just to be sure and still the problem remains. I don't know what a "buffer overrun" is, I will make my research and I will reply you when I try. The GC does not exist as I'm `betterC`. If I were in your shoes I'd use Valgrind / Memcheck to try to find the real cause of the problem. Chances are, it may have nothing to do with the bit of code you quoted at all. You could try to insert extra malloc/free's in various places around the code (in places along the code path, but unrelated to the problematic code) to see if that changes the behaviour of the bug. If it does, your corruption is likely somewhere other than the _ptr code you showed. T Thanks for the advice! I already used Valgrind before I bother you guys but because at this point of development, I didn't cared about freeing the memory, "valgrind" points so many errors that it isn't useful to help me identify what's wrong.
Re: malloc error when trying to assign the returned pointer to a struct field
On Friday, 8 September 2023 at 19:48:33 UTC, Basile B. wrote: My idea was that if you dont have defined a copy constructor and if an instance is assigned to another, then that other instance share the same pointer, which can cause memory errors. To eliminate that risk and to detect where default post-blitting may happen you can add ```d @disable this(this); ``` to the struct. Thank you! I did and I got a bunch of errors! Seeing where they are located, it doesn't seem to be in the file and instances that give me the error. Something really weird happens. I'll probably return to my own memory allocated and fix the bugs I had there, it will probably take me less time...
Re: malloc error when trying to assign the returned pointer to a struct field
On 9/8/2023 12:59 AM, rempas via Digitalmars-d-learn wrote: u64 _cap = 0; // Total amount of elements (not bytes) we can this._ptr = cast(T*)malloc(size); I'm pretty sure this is your problem. You're allocating size bytes which is only going to work where sizeof(T) == 1. Changing to malloc(size * sizeof(T)) is likely going to work better.
Re: malloc error when trying to assign the returned pointer to a struct field
On Friday, 8 September 2023 at 07:59:37 UTC, rempas wrote:
I do have the following struct:
...
That's some minimal code that I do have just to showcase it.
This is not ideal. Why? Because 99% of the time, a poster has
come here with a problem they don't know how to solve, and have
focused in on where they *think* the problem is. However, the
problem isn't there. But us reading the description can only see
what the poster sees, and either don't see a problem ("I'm just
as confused as you are!") or know there is more to the story.
Not only that, but frequently not-complete code is... not
complete. And people tend to focus on problems they can see (e.g.
where is that `_len` defined?), frustrating the poster with
"trivial" problems that are solved "in the real code".
Inevitably, there is a subsequent post with the real code, and
that contains the problem.
The best thing to post is a minimally reproducing example. The
next best thing is a link to a complex reproducing example. Which
you have done later (I will take a look). I just wanted to point
this out because it's a frequent problem on these forums.
So, some times, this work will works, some others, it will give
me the following error:
`Fatal glibc error: malloc.c:2594 (sysmalloc): assertion
failed: (old_top == initial_top (av) && old_size == 0) ||
((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top)
&& ((unsigned long) old_end & (pagesize - 1)) == 0)`
This is an internal message from glibc. It seems the malloc
structure is corrupted.
Is there any possible that there is a compiler bug? I do use
ldc2 and `betterC`!
There is always a chance...
Now, critiquing your original code, I see red flags here:
```d
u64 _cap = 0; // Total amount of elements (not bytes) we
can store
```
and then later:
```d
this(i64 size) {
this._len = 0;
this._cap = size;
```
ok, so `size` must mean the number of elements, not the number of
bytes.
```d
static if (is(T == char)) { size += 1; } // Additional
space for the null terminator
this._ptr = cast(T*)malloc(size);
```
Here, you have allocated `size` bytes for the array. Is this what
is intended? Your comments suggest otherwise! If `T.sizeof` is 2
or more, then you still only allocate e.g. 20 bytes for a `_cap`
of 20. but an array of 20 T's would require 40 bytes.
-Steve
Re: malloc error when trying to assign the returned pointer to a struct field
On Saturday, 9 September 2023 at 08:54:14 UTC, Brad Roberts wrote: I'm pretty sure this is your problem. You're allocating size bytes which is only going to work where sizeof(T) == 1. Changing to malloc(size * sizeof(T)) is likely going to work better. Oh man That was it! I had forget about that! Funny enough, the reallocation tests I do letter when expanding the vector do include that but I had forgot to place it in the new (because I had the an old one and it included this) constructor I had made that only allocates memory! Now, if only one could expect how and why "libc" knows that and doesn't just care to give me the memory I asked it for? Or it could be than D does something additional without telling us? Which can explain when this memory is only present when I assign the value to the "this._ptr` field! Hope I don't have other memory related errors until I finish the compiler... How can I thank you
Re: malloc error when trying to assign the returned pointer to a struct field
On Saturday, 9 September 2023 at 09:04:18 UTC, Steven
Schveighoffer wrote:
This is not ideal. Why? Because 99% of the time, a poster has
come here with a problem they don't know how to solve, and have
focused in on where they *think* the problem is. However, the
problem isn't there. But us reading the description can only
see what the poster sees, and either don't see a problem ("I'm
just as confused as you are!") or know there is more to the
story.
Not only that, but frequently not-complete code is... not
complete. And people tend to focus on problems they can see
(e.g. where is that `_len` defined?), frustrating the poster
with "trivial" problems that are solved "in the real code".
Inevitably, there is a subsequent post with the real code, and
that contains the problem.
The best thing to post is a minimally reproducing example. The
next best thing is a link to a complex reproducing example.
Which you have done later (I will take a look). I just wanted
to point this out because it's a frequent problem on these
forums.
Yeah... I got my lesson! Tbh, I wanted to avoid posting my source
as much as possible because I do have this "perfectionist"
mindset (which I am aware is bad and trying to get rid of, and
have already done progress) and I want the source to sound as
"professional" as possible as I take serious my project and have
hopes for it! But yeah, NEVER again!
Here, you have allocated `size` bytes for the array. Is this
what is intended? Your comments suggest otherwise! If
`T.sizeof` is 2 or more, then you still only allocate e.g. 20
bytes for a `_cap` of 20. but an array of 20 T's would require
40 bytes.
-Steve
Bingo! You and Brad found out! And like I said to him, it's funny
that I had a previous function that just allocates memory and I
removed it and then created that one and in that one, I forgot
about that. It's super funny cause I even wrote the comment that
explains it and my reallocation function (that gets triggered
when someone wants to expand the collection) has that
multiplication one it.
Thank you and everyone else for the help! Hope I can at least
finish and share something that people will enjoy using!
Re: malloc error when trying to assign the returned pointer to a struct field
On Saturday, 9 September 2023 at 09:21:32 UTC, rempas wrote: Now, if only one could expect how and why "libc" knows that and doesn't just care to give me the memory I asked it for? Or it could be than D does something additional without telling us? Which can explain when this memory is only present when I assign the value to the "this._ptr` field! You are focusing on the wrong problem. You asked for size bytes, and malloc gave you size bytes. It doesn't "know" anything special. Then you proceeded at some point to write *past* the size bytes. What did you overwrite? Probably some internal malloc implementation structure. Then it later noticed "hey, this structure doesn't make sense, I'm going to report it to the user!" That's why you see the message. Memory problems are very difficult to find, and typically an error is triggered far away from the source, in seemingly unrelated code. This is why whenever I see an error that smells like memory corruption, I stop all other work and find it. Memory errors can come and go based on random chance or how the compiler lays out functions. So having it "just go away" isn't enough. Very very infrequently, this happens because of a codegen issue, but most of the time it's pilot error. -Steve
Re: malloc error when trying to assign the returned pointer to a struct field
On Sat, Sep 09, 2023 at 09:21:32AM +, rempas via Digitalmars-d-learn wrote: > On Saturday, 9 September 2023 at 08:54:14 UTC, Brad Roberts wrote: > > I'm pretty sure this is your problem. You're allocating size bytes > > which is only going to work where sizeof(T) == 1. Changing to > > malloc(size * sizeof(T)) is likely going to work better. > > Oh man That was it! I had forget about that! Funny enough, the > reallocation tests I do letter when expanding the vector do include > that but I had forgot to place it in the new (because I had the an old > one and it included this) constructor I had made that only allocates > memory! > > Now, if only one could expect how and why "libc" knows that and > doesn't just care to give me the memory I asked it for? Or it could be > than D does something additional without telling us? Which can explain > when this memory is only present when I assign the value to the > "this._ptr` field! libc doesn't know what you intended. All it knows is that you asked it for 20 bytes (even though you actually needed 40), then later on its internal structures are corrupted (because you thought you got 40 bytes; storing data past the 20 bytes overwrote some of malloc's internal data -- this is the buffer overrun / buffer overflow I referred to). So it aborts the program instead of continuing to run in a compromised state. T -- There are four kinds of lies: lies, damn lies, and statistics.
Re: malloc error when trying to assign the returned pointer to a struct field
On Saturday, 9 September 2023 at 09:47:14 UTC, Steven Schveighoffer wrote: You are focusing on the wrong problem. You asked for size bytes, and malloc gave you size bytes. It doesn't "know" anything special. Then you proceeded at some point to write *past* the size bytes. What did you overwrite? Probably some internal malloc implementation structure. Then it later noticed "hey, this structure doesn't make sense, I'm going to report it to the user!" That's why you see the message. Memory problems are very difficult to find, and typically an error is triggered far away from the source, in seemingly unrelated code. This is why whenever I see an error that smells like memory corruption, I stop all other work and find it. Memory errors can come and go based on random chance or how the compiler lays out functions. So having it "just go away" isn't enough. Very very infrequently, this happens because of a codegen issue, but most of the time it's pilot error. -Steve I understand! Thank you for the valuable information. I do have lots of things to learn it seems. But that means that I also won't get bored anytime soon ;)
Re: malloc error when trying to assign the returned pointer to a struct field
On Saturday, 9 September 2023 at 09:56:59 UTC, H. S. Teoh wrote: libc doesn't know what you intended. All it knows is that you asked it for 20 bytes (even though you actually needed 40), then later on its internal structures are corrupted (because you thought you got 40 bytes; storing data past the 20 bytes overwrote some of malloc's internal data -- this is the buffer overrun / buffer overflow I referred to). So it aborts the program instead of continuing to run in a compromised state. T Thank you! I fully realize now what's the problem! And that was indeed a very sneaky problem. The good news is that I'm mostly done with these memory structures and functions so I will probably take a while since I find something similar. I'm lucky there are people smarter, more experience that are willing to help. Bless you all and have a great day!
Re: malloc error when trying to assign the returned pointer to a struct field
On Saturday, 9 September 2023 at 09:30:10 UTC, rempas wrote: Bingo! You and Brad found out! Hate to be that guy, but I posted a link to a stackoverflow question with the exact error message you were getting, and the solution. And I told you I had experienced the same error and that question fixed it.
Re: malloc error when trying to assign the returned pointer to a struct field
On Saturday, 9 September 2023 at 10:54:34 UTC, bachmeier wrote: Hate to be that guy, but I posted a link to a stackoverflow question with the exact error message you were getting, and the solution. And I told you I had experienced the same error and that question fixed it. No reason to not says something, always speak you mind my friend! I did read it and all the replies but the problem is that, I was so focused on thinking that something is going wrong with the compiler or with libc or something unexpected. For that reason, I made the mistake to not see other things that might had gotten wrong even tho you literally gave me the answer! That's another problem of mine that I am aware and trying to fix (not only when it comes to coding but in general) and I do apologize for wasting both mine and everyone else's time! Now that people explained to me, I fully understood what's going on and how I made the mistake. Have a great day my friend and I hope I can repay one day! ;)
Re: Hipreme's Tip of of the day #09 - Using D shared libraries with dub
On Monday, 4 September 2023 at 23:57:03 UTC, Hipreme wrote: Hello again! -- As of some requests in DConf, I'll post here some things related (or not) to dub recipes. Since there is so many ways to build D and dub is quite the main way, I'll try to show other uncommon ways to use it, this is more recommended to bigger projects since the standard one is enough for most. [...] Thanks for these Hipreme. I always enjoy reading them.
pipeProcess output to hash string
Hi All,
Request your help on how to convert the output of
std.process.pipeProcess to hash string
```
auto test(in Redirect redirect=Redirect.stdout | Redirect.stderr)
{
import std.process;
import std.digest.crc;
import std.stdio: writeln;
result = std.process.pipeProcess("whoami" ~ "/?", redirect);
auto content = result.stdout.readln;
auto hash = crc32Of(content);
return hash;
}
void main() {
writeln(test);
}
```
Output: All writes the below.
```
[0, 0, 0, 0]
```
Required: Read the completed output and convert it to a single
hash string.
From,
Vino.B
Re: pipeProcess output to hash string
On Saturday, 9 September 2023 at 15:44:44 UTC, Vino wrote:
Hi All,
Request your help on how to convert the output of
std.process.pipeProcess to hash string
```
auto test(in Redirect redirect=Redirect.stdout |
Redirect.stderr) {
import std.process;
import std.digest.crc;
import std.stdio: writeln;
result = std.process.pipeProcess("whoami" ~ "/?",
redirect);
auto content = result.stdout.readln;
auto hash = crc32Of(content);
return hash;
}
void main() {
writeln(test);
}
```
Output: All writes the below.
```
[0, 0, 0, 0]
```
Required: Read the completed output and convert it to a single
hash string.
From,
Vino.B
With a slightly modified command line that works on linux here :
```d
import std.process, std.stdio;
auto test(in Redirect redirect=Redirect.stdout | Redirect.stderr)
{
import std.digest.crc;
import std.stdio: writeln;
auto result = std.process.pipeProcess(["whoami"], redirect);
auto content = result.stdout.readln;
auto hash = crc32Of(content);
return hash;
}
void main() {
writeln(test);
}
```
not sure why you append "/?" to the program name.
Re: pipeProcess output to hash string
On Saturday, 9 September 2023 at 16:49:30 UTC, user1234 wrote: not sure why you append "/?" to the program name. Windows maybe? Try this. auto result = std.process.pipeProcess(["whoami", "/?"], redirect);
