Re: Getting a safe path for a temporary file
On Friday, October 27, 2017 21:38:55 Jonathan M Davis via Digitalmars-d- learn wrote: > Also, toStringz specifically returns an immutable(char)* - though looking > it over right now, I'd say that that's a bug for the overload that takes > const(char)[] instead of string. It really should return const(char)* in > that case. Actually, looking it over again, the return type is fine. That's the overload that always allocates a new string (since it assumes that the array could not have been from a string literal, since that would be a string and would go to the other overload), so even though the argument could have been const or mutable, what's returned is a pointer to a string, so immutable(char)* is correct. But https://issues.dlang.org/show_bug.cgi?id=15136 is still a problem for the other overload. - Jonathan M Davis
Re: Getting a safe path for a temporary file
On Saturday, October 28, 2017 02:46:00 Shriramana Sharma via Digitalmars-d-
learn wrote:
> On Wednesday, 25 October 2017 at 00:35:29 UTC, Ali Çehreli wrote:
> > > char[] name = "/tmp/XX".dup;
> >
> > remain valid. The actual issue is the missing '\0'. So,
> >
> > consider toStringz in this case:
> > https://dlang.org/library/std/string/to_stringz.html
>
> Thanks for your reply, but can you clarify exactly I should use
> this?
>
> char[] name = "/tmp/XX".toStringz;
>
> gives
>
> (13): Error: cannot implicitly convert expression
> `toStringz("/tmp/XX")` of type `immutable(char)*` to `char[]`
>
> So I tried:
>
> char[] name = "/tmp/XX".toStringz.dup;
>
> which gives
>
> (13): Error: template object.dup cannot deduce function from
> argument types !()(immutable(char)*), candidates are:
> /usr/include/dmd/druntime/import/object.d(1943):
> object.dup(T : V[K], K, V)(T aa)
> /usr/include/dmd/druntime/import/object.d(1979):
> object.dup(T : V[K], K, V)(T* aa)
> /usr/include/dmd/druntime/import/object.d(3764):
> object.dup(T)(T[] a) if (!is(const(T) : T))
> /usr/include/dmd/druntime/import/object.d(3780):
> object.dup(T)(const(T)[] a) if (is(const(T) : T))
>
> And:
>
> char[] name = "/tmp/XX".dup.toStringz;
>
> gives the error (13): Error: cannot implicitly convert
> expression `toStringz(dup("/tmp/XX"))` of type
> `immutable(char)*` to `char[]`
>
> So I'm not sure what to do?!
Well, for starters, toStringz returns a pointer, not a dynamic array. If you
want a dynamic array that's null-terminated, then you'll need to explicitly
put a null character on the end unless you're just going to use a string
literal (which would mean a string, not a char[], so that won't work here).
Also, toStringz specifically returns an immutable(char)* - though looking it
over right now, I'd say that that's a bug for the overload that takes
const(char)[] instead of string. It really should return const(char)* in
that case.
But regardless, that means that even changing char[] to char* wouldn't cut
it if you're using toStringz, since you're starting with a string. If you
want a specific constness, then you can use the more general function,
std.utf.toUTFz, which works with multiple character types and differing
constness rather than being designed specifically for string like toStringz
is.
However, something to take into account is that toStringz and toUTFz don't
always return the same string (and in fact, they probably never should,
because the trick they use to check for the null character one past the end
of the string doesn't always work correctly). So, if you're passing a string
to a C function that's then going to mutate it, you don't want toStringz or
toUTFZ. In that case, it's better to just manually put the null terminator
at the end of the string. So, you probably would end up with something like
char[] name = "/tmp/XX\0".dup;
auto fd = mkstemp(name.ptr);
Then when you return the name, you do something like
name[0 .. $ - 1].idup;
Alternatively, you could use a static array, but either way, you'd want to
put the null terminator in there manually. And fromStringz really isn't
necessary, since mkstemp is just going to fill in the array that you gave
it, and you know exactly where the null terminator is going to be, since
it's just filling in the X's rather than doing something that could end up
putting a null terminator anywhere in the array. You can just slice the
array to chop off the null terminator, and then dup it or idup if it's a
static array (so that you don't return a slice of a local variable), or if
it's a dynamic array, then either return it as-is or idup it if you want a
string; you could even use std.exception.assumeUnique to just cast it to
string if you know that no other references to that data exists (which they
wouldn't if you allocated the string inside of the function).
Also, you _really_ wouldn't want to use fromStringz if you used a static
array, since fromStringz always returns a slice of the original input:
inout(char)[] fromStringz(inout(char)* cString) @nogc @system pure nothrow {
import core.stdc.string : strlen;
return cString ? cString[0 .. strlen(cString)] : null;
}
But if I were you, I'd just create a char[] with an explicit null
terminator, and then afterwards, slice off the null character, and pass it
to assumeUnique to get a string, since then you allocate only once and don't
have to copy the contents of the array.
- Jonathan M Davis
Re: Getting a safe path for a temporary file
On Wednesday, 25 October 2017 at 00:35:29 UTC, Ali Çehreli wrote:
> char[] name = "/tmp/XX".dup;
remain valid. The actual issue is the missing '\0'. So,
consider toStringz in this case:
https://dlang.org/library/std/string/to_stringz.html
Thanks for your reply, but can you clarify exactly I should use
this?
char[] name = "/tmp/XX".toStringz;
gives
(13): Error: cannot implicitly convert expression
`toStringz("/tmp/XX")` of type `immutable(char)*` to `char[]`
So I tried:
char[] name = "/tmp/XX".toStringz.dup;
which gives
(13): Error: template object.dup cannot deduce function from
argument types !()(immutable(char)*), candidates are:
/usr/include/dmd/druntime/import/object.d(1943):
object.dup(T : V[K], K, V)(T aa)
/usr/include/dmd/druntime/import/object.d(1979):
object.dup(T : V[K], K, V)(T* aa)
/usr/include/dmd/druntime/import/object.d(3764):
object.dup(T)(T[] a) if (!is(const(T) : T))
/usr/include/dmd/druntime/import/object.d(3780):
object.dup(T)(const(T)[] a) if (is(const(T) : T))
And:
char[] name = "/tmp/XX".dup.toStringz;
gives the error (13): Error: cannot implicitly convert
expression `toStringz(dup("/tmp/XX"))` of type
`immutable(char)*` to `char[]`
So I'm not sure what to do?!
Re: Getting a safe path for a temporary file
On Sunday, 18 January 2015 at 00:51:37 UTC, Laeeth Isharc wrote: On Saturday, 17 January 2015 at 16:55:42 UTC, Marc Schütz wrote: On Saturday, 17 January 2015 at 14:37:00 UTC, Laeeth Isharc wrote: On Saturday, 17 January 2015 at 13:47:39 UTC, Marc Schütz wrote: [...] I agree that it would be useful. This is what I used, although there may be a better option: http://dlang.org/phobos/std_uuid.html Nice idea, but it still allows for intentional collision attacks :-( The only really safe solution is one that generates (probably) unique names, then opens the file with O_EXCL|O_CREAT (or whatever other means the OS provides), and if it fails, retries with a different name. `std.stdio.tmpfile()` already does that (it uses `tmpfile(3)` under the hood), but doesn't allow access to the name. I don't follow why a collision attack is applicable in this case. Your stage 1 of generating unique names: how is this different from using a random uuid? UUIDs are defined to be unique, not unpredictable. UUID that use random number generation (UUID4) should use a cryptographically secure random number generator but are not required to. Therefore it shouldn't be blindly trusted against someone actively trying to get a collision.
Re: Getting a safe path for a temporary file
On Sunday, 18 January 2015 at 16:00:32 UTC, Kagamin wrote: On Sunday, 18 January 2015 at 11:21:52 UTC, Marc Schütz wrote: It's not different, and if you're still doing the O_EXCL open afterwards, it's safe. I just assumed you were going to use the generated filename without a further check. This is then unsafe, no matter how the UUID is generated, and depending on the RNG that's been used, they can be quite predictable. Granted, the risk is low, but still... tmpfile is more predictable: it generates sequential file names. Being predictable is only an issue if the file is wrongly used (ie: no check that it might already exist, or be a symlink or check at the wrong time leaving an exploitable time frame etc). Sequential file names are a good way to provide uniqueness over a single system after all.
Re: Getting a safe path for a temporary file
On 10/22/2017 06:41 PM, Shriramana Sharma wrote: > On Sunday, 22 October 2017 at 15:21:37 UTC, Shriramana Sharma wrote: >> For my program right now I'm using a souped-up version using a static >> array: >> >> char[20] name = "/tmp/XX"; Literal strings have a '\0' attached, which does not become a part of the slice. > Hmm I was wondering if I needed it to be static, and verily, substituting: > > char[] name = "/tmp/XX".dup; However, .dup is not be aware of the special literal treatment so it does not copy the '\0' character. > Is it because the D slice is subject to relocation and C is occasionally > not able to access the proper pointer? As long as you have a reference on the D side, the slice will remain valid. The actual issue is the missing '\0'. So, consider toStringz in this case: https://dlang.org/library/std/string/to_stringz.html Ali
Re: Getting a safe path for a temporary file
On Sunday, 22 October 2017 at 15:21:37 UTC, Shriramana Sharma wrote: For my program right now I'm using a souped-up version using a static array: char[20] name = "/tmp/XX"; Hmm I was wondering if I needed it to be static, and verily, substituting: char[] name = "/tmp/XX".dup; instead gives a proper output *some* of the time but mostly gives the error: std.exception.ErrnoException@std/stdio.d(630): (Bad file descriptor) ??:? @safe shared(core.stdc.stdio._IO_FILE)* std.exception.errnoEnforce!(shared(core.stdc.stdio._IO_FILE)*, "std/stdio.d", 630uL).errnoEnforce(shared(core.stdc.stdio._IO_FILE)*, lazy immutable(char)[]) [0x44bd31] ??:? @trusted void std.stdio.File.fdopen(int, const(char[]), immutable(char)[]) [0x44294a] ??:? @safe void std.stdio.File.fdopen(int, const(char[])) [0x4428b1] ??:? .TempFile .tempFileOpen() [0x43d4fd] ??:? _Dmain [0x43d5ce] Is it because the D slice is subject to relocation and C is occasionally not able to access the proper pointer?
Re: Getting a safe path for a temporary file
On Sunday, October 22, 2017 15:21:37 Shriramana Sharma via Digitalmars-d-
learn wrote:
> On Saturday, 17 January 2015 at 17:16:41 UTC, Tobias Pankrath
>
> wrote:
> > You're looking for core.sys.posix.stdlib : mkstemp.
> >
> > I think that should be used by std.stdio.File as well, care to
> > create an enhancement request in bugzilla?
>
> Though this thread is old, I ran into the issue when wanting to
> create a temporary file in my D program and so filed this:
> https://issues.dlang.org/show_bug.cgi?id=17926
>
> For my program right now I'm using a souped-up version using a
> static array:
>
> import std.stdio;
>
> struct TempFile
> {
> string name;
> private File handle;
> alias handle this;
> }
>
> TempFile tempFileOpen()
> {
> char[20] name = "/tmp/XX";
> File handle;
>
> import core.sys.posix.stdlib: mkstemp;
> handle.fdopen(mkstemp(name.ptr), "w");
>
> import std.string: fromStringz;
> return TempFile(fromStringz(name.ptr).idup, handle);
> }
>
> void main()
> {
> TempFile tfile;
> tfile = tempFileOpen();
> writeln(tfile.name);
> tfile = tempFileOpen();
> writeln(tfile.name);
> }
We did have a function for creating a temporary file and returning it as a
File, but it quickly got canned, because adding that functionality to
std.stdio resulted in "hello world" increasing in size, which apparently was
considered unacceptable:
https://issues.dlang.org/show_bug.cgi?id=14599
I did recently create this enhancement request:
https://issues.dlang.org/show_bug.cgi?id=17912
and this PR:
https://github.com/dlang/phobos/pull/5788
It creates a function in std.file that acts similar to std.file.write but
which writes to a randomly generated file name instead of the one you tell
it. So, you don't get an open file out of the deal, but you can write data
to it when it's first opened, and you can reopen it afterwards (which is
usually what I've seen the need for anyway - e.g. a test where you need to
pass a filename to a function that you're testing so that it can open it and
do whatever it does normally).
- Jonathan M Davis
Re: Getting a safe path for a temporary file
On Saturday, 17 January 2015 at 17:16:41 UTC, Tobias Pankrath
wrote:
You're looking for core.sys.posix.stdlib : mkstemp.
I think that should be used by std.stdio.File as well, care to
create an enhancement request in bugzilla?
Though this thread is old, I ran into the issue when wanting to
create a temporary file in my D program and so filed this:
https://issues.dlang.org/show_bug.cgi?id=17926
For my program right now I'm using a souped-up version using a
static array:
import std.stdio;
struct TempFile
{
string name;
private File handle;
alias handle this;
}
TempFile tempFileOpen()
{
char[20] name = "/tmp/XX";
File handle;
import core.sys.posix.stdlib: mkstemp;
handle.fdopen(mkstemp(name.ptr), "w");
import std.string: fromStringz;
return TempFile(fromStringz(name.ptr).idup, handle);
}
void main()
{
TempFile tfile;
tfile = tempFileOpen();
writeln(tfile.name);
tfile = tempFileOpen();
writeln(tfile.name);
}
Re: Getting a safe path for a temporary file
On Sunday, 18 January 2015 at 11:21:52 UTC, Marc Schütz wrote: It's not different, and if you're still doing the O_EXCL open afterwards, it's safe. I just assumed you were going to use the generated filename without a further check. This is then unsafe, no matter how the UUID is generated, and depending on the RNG that's been used, they can be quite predictable. Granted, the risk is low, but still... tmpfile is more predictable: it generates sequential file names.
Re: Getting a safe path for a temporary file
On Saturday, 17 January 2015 at 21:32:18 UTC, Jonathan M Davis via Digitalmars-d-learn wrote: But I think that what we need is a function in std.stdio (e.g tempFile insteaf of tmpfile) which returns an open File with a randomly generated name and gives you access to its name rather than using C's tmpfile, which does not give you access to the name and deletes the file on you when it's closed. Right - I overlooked this fact. The bad thing is that you might even be forced to close the file before another program can open it, if either of the programs wants to open it exclusively (probably most likely to happen on Windows).
Re: Getting a safe path for a temporary file
On Sunday, 18 January 2015 at 00:51:37 UTC, Laeeth Isharc wrote: I don't follow why a collision attack is applicable in this case. Your stage 1 of generating unique names: how is this different from using a random uuid? It's not different, and if you're still doing the O_EXCL open afterwards, it's safe. I just assumed you were going to use the generated filename without a further check. This is then unsafe, no matter how the UUID is generated, and depending on the RNG that's been used, they can be quite predictable. Granted, the risk is low, but still...
Re: Getting a safe path for a temporary file
On Saturday, 17 January 2015 at 17:16:41 UTC, Tobias Pankrath wrote: On Saturday, 17 January 2015 at 16:55:42 UTC, Marc Schütz wrote: On Saturday, 17 January 2015 at 14:37:00 UTC, Laeeth Isharc wrote: On Saturday, 17 January 2015 at 13:47:39 UTC, Marc Schütz wrote: Is it currently possible to get the path to a safe temporary file, i.e. one that is guaranteed to be freshly created and will not override another existing file? There's `std.file.tempDir`, which doesn't create a unique file. Then there's `std.stdio.tmpfile()`, which does, but it returns a `File` object, and its `name` property is `null`. Did I miss something? IMO this is very import functionality. One use case is passing these names as command line arguments to an external program that doesn't support stdin/stdout. I agree that it would be useful. This is what I used, although there may be a better option: http://dlang.org/phobos/std_uuid.html Nice idea, but it still allows for intentional collision attacks :-( The only really safe solution is one that generates (probably) unique names, then opens the file with O_EXCL|O_CREAT (or whatever other means the OS provides), and if it fails, retries with a different name. `std.stdio.tmpfile()` already does that (it uses `tmpfile(3)` under the hood), but doesn't allow access to the name. You're looking for core.sys.posix.stdlib : mkstemp. I think that should be used by std.stdio.File as well, care to create an enhancement request in bugzilla? But it's POSIX only :-(
Re: Getting a safe path for a temporary file
On Saturday, 17 January 2015 at 16:55:42 UTC, Marc Schütz wrote: On Saturday, 17 January 2015 at 14:37:00 UTC, Laeeth Isharc wrote: On Saturday, 17 January 2015 at 13:47:39 UTC, Marc Schütz wrote: Is it currently possible to get the path to a safe temporary file, i.e. one that is guaranteed to be freshly created and will not override another existing file? There's `std.file.tempDir`, which doesn't create a unique file. Then there's `std.stdio.tmpfile()`, which does, but it returns a `File` object, and its `name` property is `null`. Did I miss something? IMO this is very import functionality. One use case is passing these names as command line arguments to an external program that doesn't support stdin/stdout. I agree that it would be useful. This is what I used, although there may be a better option: http://dlang.org/phobos/std_uuid.html Nice idea, but it still allows for intentional collision attacks :-( The only really safe solution is one that generates (probably) unique names, then opens the file with O_EXCL|O_CREAT (or whatever other means the OS provides), and if it fails, retries with a different name. `std.stdio.tmpfile()` already does that (it uses `tmpfile(3)` under the hood), but doesn't allow access to the name. I don't follow why a collision attack is applicable in this case. Your stage 1 of generating unique names: how is this different from using a random uuid?
Re: Getting a safe path for a temporary file
On Saturday, January 17, 2015 13:47:37 via Digitalmars-d-learn wrote: > Is it currently possible to get the path to a safe temporary > file, i.e. one that is guaranteed to be freshly created and will > not override another existing file? > > There's `std.file.tempDir`, which doesn't create a unique file. > Then there's `std.stdio.tmpfile()`, which does, but it returns a > `File` object, and its `name` property is `null`. > > Did I miss something? IMO this is very import functionality. One > use case is passing these names as command line arguments to an > external program that doesn't support stdin/stdout. The _only_ way to this write is to randomly generate a file name and then open the file with O_CREAT | O_EXCL (or more likely, O_RDWR | O_CREAT | O_EXCL), and then retry with a new name if the creation fails (good enough random name generation would likely require only two attempts at most). Simply randomly generating a file name is not enough, because it's still technically possible for the file to already exist (even if it's unlikely), and even checking for the file's existence prior to opening it isn't enough, because technically, the file could be created by another program in the small amount of time between when you checked for the file's existence and tried to create it. POSIX actually has mkstemp for doing this for you, but on some operating systems, it restricts the number of random files that it can generate to as little as 26 (at least, that's what I recall the number being). I don't think that any of the POSIX systems that we currently support have an implementation of mkstemp that's quite that bad, but all in all, I don't think that using mkstemp is a good idea. The problem is solved simply enough by randomly generating a file name (e.g. with rndGen()) and then using the correct flags with open. And I actually have code that does this that I was working on getting into Phobos, but the problem was getting access to the correct function on Windows (_wsopen_s, I believe). It wasn't available in druntime, and I didn't get around to fixing that (IIRC, because I started looking into the whole problem of how to deal with windows bindings in druntime in general and going down a rat hole that I didn't have time for). So, I never finished that pull request, and I really should get back to it. But I think that what we need is a function in std.stdio (e.g tempFile insteaf of tmpfile) which returns an open File with a randomly generated name and gives you access to its name rather than using C's tmpfile, which does not give you access to the name and deletes the file on you when it's closed. IMHO, tmpfile is pretty useless - especially when it comes to unit tests. - Jonathan M Davis
Re: Getting a safe path for a temporary file
On Saturday, 17 January 2015 at 16:55:42 UTC, Marc Schütz wrote: On Saturday, 17 January 2015 at 14:37:00 UTC, Laeeth Isharc wrote: On Saturday, 17 January 2015 at 13:47:39 UTC, Marc Schütz wrote: Is it currently possible to get the path to a safe temporary file, i.e. one that is guaranteed to be freshly created and will not override another existing file? There's `std.file.tempDir`, which doesn't create a unique file. Then there's `std.stdio.tmpfile()`, which does, but it returns a `File` object, and its `name` property is `null`. Did I miss something? IMO this is very import functionality. One use case is passing these names as command line arguments to an external program that doesn't support stdin/stdout. I agree that it would be useful. This is what I used, although there may be a better option: http://dlang.org/phobos/std_uuid.html Nice idea, but it still allows for intentional collision attacks :-( The only really safe solution is one that generates (probably) unique names, then opens the file with O_EXCL|O_CREAT (or whatever other means the OS provides), and if it fails, retries with a different name. `std.stdio.tmpfile()` already does that (it uses `tmpfile(3)` under the hood), but doesn't allow access to the name. You're looking for core.sys.posix.stdlib : mkstemp. I think that should be used by std.stdio.File as well, care to create an enhancement request in bugzilla?
Re: Getting a safe path for a temporary file
On Saturday, 17 January 2015 at 14:37:00 UTC, Laeeth Isharc wrote: On Saturday, 17 January 2015 at 13:47:39 UTC, Marc Schütz wrote: Is it currently possible to get the path to a safe temporary file, i.e. one that is guaranteed to be freshly created and will not override another existing file? There's `std.file.tempDir`, which doesn't create a unique file. Then there's `std.stdio.tmpfile()`, which does, but it returns a `File` object, and its `name` property is `null`. Did I miss something? IMO this is very import functionality. One use case is passing these names as command line arguments to an external program that doesn't support stdin/stdout. I agree that it would be useful. This is what I used, although there may be a better option: http://dlang.org/phobos/std_uuid.html Nice idea, but it still allows for intentional collision attacks :-( The only really safe solution is one that generates (probably) unique names, then opens the file with O_EXCL|O_CREAT (or whatever other means the OS provides), and if it fails, retries with a different name. `std.stdio.tmpfile()` already does that (it uses `tmpfile(3)` under the hood), but doesn't allow access to the name.
Re: Getting a safe path for a temporary file
On Saturday, 17 January 2015 at 13:47:39 UTC, Marc Schütz wrote: Is it currently possible to get the path to a safe temporary file, i.e. one that is guaranteed to be freshly created and will not override another existing file? There's `std.file.tempDir`, which doesn't create a unique file. Then there's `std.stdio.tmpfile()`, which does, but it returns a `File` object, and its `name` property is `null`. Did I miss something? IMO this is very import functionality. One use case is passing these names as command line arguments to an external program that doesn't support stdin/stdout. I agree that it would be useful. This is what I used, although there may be a better option: http://dlang.org/phobos/std_uuid.html
Getting a safe path for a temporary file
Is it currently possible to get the path to a safe temporary file, i.e. one that is guaranteed to be freshly created and will not override another existing file? There's `std.file.tempDir`, which doesn't create a unique file. Then there's `std.stdio.tmpfile()`, which does, but it returns a `File` object, and its `name` property is `null`. Did I miss something? IMO this is very import functionality. One use case is passing these names as command line arguments to an external program that doesn't support stdin/stdout.
