Nothrow std.conv.to with explicit default value

[Per Nordlöw via Digitalmars-d-learn]

I have a nothrow variant of std.conv.to defined as follows:
T toDefaulted(T, S, U)(S value, /*lazy*/ U defaultValue)
if (is(typeof(() { T r = defaultValue; }))) // TODO use 
std.traits.isAssignable!(T, U) ?
{
    try
    {
        import std.conv : to;
        return value.to!T;
    }
    catch (Exception e) // assume `ConvException`. TODO can we 
capture `ConvException` instead make it inferred `nothrow`
    {
        return defaultValue;
    }
}

The problem with this code is that throwing exceptions for the 
default case is costly at least with dmd. Is there another way to 
do this?