{"id":283,"date":"2016-09-28T13:59:14","date_gmt":"2016-09-28T13:59:14","guid":{"rendered":"http:\/\/dlang.org\/blog\/?p=283"},"modified":"2021-10-08T11:09:59","modified_gmt":"2021-10-08T11:09:59","slug":"how-to-write-trusted-code-in-d","status":"publish","type":"post","link":"https:\/\/dlang.org\/blog\/2016\/09\/28\/how-to-write-trusted-code-in-d\/","title":{"rendered":"How to Write @trusted Code in D"},"content":{"rendered":"

Steven Schveighoffer is the creator and maintainer of the dcollections<\/a> and iopipe<\/a> libraries. He was the primary instigator of D’s inout<\/a> feature and the architect of a major rewrite of the language’s built-in arrays. He also authored the oft-recommended introductory article<\/a> on the latter.
\n<\/em><\/p>\n

\n

\"d6\"In computer programming, there is a concept of memory-safe<\/strong> code<\/a>, which is guaranteed at some level not to cause memory corruption issues. The ultimate holy grail of memory safety is to be able to mechanically verify you will not corrupt memory no matter what. This would provide immunity from attacks via buffer overflows and the like. The D language provides a definition of memory safety that allows quite a bit of useful code, but conservatively forbids things that are sketchy. In practice, the compiler is not omnipotent, and it lacks the context that we humans are so good at seeing (most of the time), so there is often the need to allow otherwise risky behavior. Because the compiler is very rigid on memory safety, we need the equivalent of a cast to say “yes, I know this is normally forbidden, but I’m guaranteeing that it is fine”. That tool is called @trusted<\/code>.<\/p>\n

Because it’s very difficult to explain why @trusted<\/code> code might be incorrect without first discussing memory safety and D’s @safe<\/code> mechanism, I’ll go over that first.<\/p>\n

What is Memory Safe Code?<\/h3>\n

The easiest way to explain what is safe, is to examine what results in unsafe code. There are generally 3 main ways to create a safety violation in a statically-typed language:<\/p>\n

    \n
  1. Write or read from a buffer outside the valid segment of memory that you have access to.<\/li>\n
  2. Cast some value to a type that allows you to treat a piece of memory that is not a pointer as a pointer.<\/li>\n
  3. Use a pointer that is dangling, or no longer valid.<\/li>\n<\/ol>\n

    The first item is quite simple to achieve in D:<\/p>\n

    auto buf = new int[1]; \r\nbuf[2] = 1;<\/pre>\n
    With default bounds checks on, this results in an exception at runtime, even in code that is not checked for safety. But D allows circumventing this by accessing the pointer of the array:<\/p>\n
    buf.ptr[2] = 1;<\/pre>\n
    For an example of the second, all that is needed is a cast:<\/p>\n
    *cast(int*)(0xdeadbeef) = 5;<\/pre>\n
    And the third is relatively simple as well:<\/p>\n
    auto buf = new int[1];\r\nauto buf2 = buf;\r\ndelete buf;  \/\/ sets buf to null\r\nbuf2[0] = 5; \/\/ but not buf2.<\/pre>\n
    Dangling pointers also frequently manifest by pointing at stack data that is no longer in use (or is being used for a different reason). It’s very simple to achieve:<\/p>\n
    int[] foo()\r\n{\r\n    int[4] buf;\r\n    int[] result = buf[];\r\n    return result;\r\n}<\/pre>\n
    So simply put, safe code avoids doing things that could potentially result in memory corruption. To that end, we must follow some rules that prohibit such behavior.<\/p>\n
    Note: dereferencing a null<\/code> pointer in user-space is not<\/em> considered a memory safety issue in D. Why not? Because this triggers a hardware exception, and generally does not leave the program in an undefined state with corrupted memory. It simply aborts the program. This may seem undesirable to the user or the programmer, but it’s perfectly fine in terms of preventing exploits. There are potential memory issues possible with null<\/code> pointers, if one has a null<\/code> pointer to a very large memory space. But for safe D, this requires an unusually large struct to even begin to worry about it. In the eyes of the D language, instrumenting all pointer dereferences to check for null<\/code> is not worth the performance degradation for these rare situations.<\/p>\n
    D’s @safe rules<\/h3>\n
    D provides the @safe<\/code><\/strong> attribute<\/a> that tags a function to be mechanically checked by the compiler to follow rules that should prevent all possible memory safety problems. Of course, there are cases where developers need to make exceptions in order to get some meaningful work done.<\/p>\n
    The following rules are geared to prevent issues like the ones discussed above (listed in the spec here<\/a>).<\/p>\n
      \n
    1. Changing a raw pointer value is not allowed. If @safe<\/code> D code has a pointer, it has access only<\/em> to the value pointed at, no others. This includes indexing a pointer.<\/li>\n
    2. Casting pointers to any type other than void*<\/code> is not allowed. Casting from any non-pointer type to a pointer type is not allowed. All other casts are OK (e.g. casting from float<\/code> to int<\/code>) as long as they are valid. Casting a dynamic array to a void[]<\/code> is also allowed.<\/li>\n
    3. Unions that have pointer types that overlap other types cannot be accessed. This is similar to rules 1 and 2 above.<\/li>\n
    4. Accessing an element in or taking a slice from a dynamic array must<\/em> be either proven safe by the compiler, or incur a bounds check during runtime. This even happens in release mode, when bounds checks are normally omitted (note: dmd’s option -boundscheck=off<\/strong> will override this, so use with extreme caution).<\/li>\n
    5. In normal D, you can create a dynamic array from a pointer by slicing<\/a> the pointer. In @safe<\/code> D, this is not allowed, since the compiler has no idea how much space you actually have available via that pointer.<\/li>\n
    6. Taking a pointer to a local variable or function parameter (variables that are stored on the stack) or taking a pointer to a reference parameter are forbidden. An exception is slicing a local static array, including the function foo<\/code> above. This is a known issue<\/a>.<\/li>\n
    7. Explicit casting between immutable and mutable types that are or contain references is not allowed. Casting value-types between immutable and mutable can be done implicitly and is perfectly fine.<\/li>\n
    8. Explicit casting between thread-local and shared types that are or contain references is not allowed. Again, casting value-types is fine (and can be done implicitly).<\/li>\n
    9. The inline assembler<\/a> feature of D is not allowed in @safe<\/code> code.<\/li>\n
    10. Catching thrown objects that are not derived from class Exception<\/code> is not allowed.<\/li>\n
    11. In D, all variables are default initialized. However, this can be changed to uninitialized by using a void initializer<\/a>:\n
      int *s = void;<\/pre>\n
      Such usage is not allowed in @safe<\/code> D. The above pointer would point to random memory and create an obvious dangling pointer.<\/li>\n
    12. __gshared<\/code> variables are static variables that are not properly typed as shared<\/code>, but are still in global space. Often these are used when interfacing with C code. Accessing such variables is not allowed in @safe<\/code> D.<\/li>\n
    13. Using the ptr<\/code> property of a dynamic array is forbidden (a new rule that will be released in version 2.072 of the compiler).<\/li>\n
    14. Writing to void[]<\/code> data by means of slice-assigning from another void[]<\/code> is not allowed (this rule is also new, and will be released in 2.072).<\/li>\n
    15. Only @safe<\/code> functions or those inferred to be @safe<\/code> can be called.<\/li>\n<\/ol>\n
      The need for @trusted<\/h3>\n
      The above rules work well to prevent memory corruption, but they prevent a lot of valid, and actually safe, code. For example, consider a function that wants to use the system call read<\/code><\/a>, which is prototyped like this:<\/p>\n
      ssize_t read(int fd, void* ptr, size_t nBytes);<\/pre>\n
      For those unfamiliar with this function, it reads data from the given file descriptor, and puts it into the buffer pointed at by ptr<\/code> and expected to be nBytes<\/code> bytes long. It returns the number of bytes actually read, or a negative value if an error occurs.<\/p>\n
      Using this function to read data into a stack-allocated buffer might look like this:<\/p>\n
      ubyte[128] buf;\r\nauto nread = read(fd, buf.ptr, buf.length);<\/pre>\n
      How is this done inside a @safe<\/code> function? The main issue with using read<\/code> in @safe<\/code> code is that pointers can only pass a single value, in this case a single ubyte<\/code>. read<\/code> expects to store more bytes of the buffer. In D, we would normally pass the data to be read as a dynamic array. However, read<\/code> is not D code, and uses a common C idiom of passing the buffer and length separately, so it cannot be marked @safe<\/code>. Consider the following call from @safe<\/code> code:<\/p>\n
      auto nread = read(fd, buf.ptr, 10_000);<\/pre>\n
      This call is definitely not<\/em> safe. What is safe in the above read<\/code> example is only the one call, where the understanding of the read<\/code> function and calling context assures memory outside the buffer will not be written.<\/p>\n
      To solve this situation, D provides the\u00a0@trusted<\/code>\u00a0 attribute<\/a>, which tells the compiler that the code inside the function is assumed to be @safe<\/code>, but will not be mechanically checked. It’s on you, the developer, to make sure the code is actually @safe<\/code>.<\/p>\n
      A function that solves the problem might look like this in D:<\/p>\n
      auto safeRead(int fd, ubyte[] buf) @trusted\r\n{\r\n    return read(fd, buf.ptr, buf.length);\r\n}<\/pre>\n
      Whenever marking an entire function @trusted<\/code>, consider if code could call this function from any context<\/em> that would compromise memory safety. If so, this function should not be marked @trusted<\/code> under any circumstances<\/strong>. Even if the intention is to only call it in safe ways, the compiler will not prevent unsafe usage by others. safeRead<\/code> should be fine to call from any @safe<\/code> context, so it’s a great candidate to mark @trusted<\/code>.<\/p>\n
      A more liberal API for the safeRead<\/code> function might take a void[]<\/code> array as the buffer. However, recall that in @safe<\/code> code, one can cast any dynamic array to a void[]<\/code> array — including an array of pointers. Reading file data into an array of pointers could result in an array of dangling pointers. This is why ubyte[]<\/code> is used instead.<\/p>\n
      @trusted escapes<\/h3>\n
      A @trusted<\/code> escape is a single expression that allows @system<\/code> (the unsafe default in D) calls such as read<\/code> without exposing the potentially unsafe call to any other part of the program. Instead of writing the safeRead<\/code> function, the same feat can be accomplished inline within a @safe<\/code> function:<\/p>\n
      auto nread = ( () @trusted => read(fd, buf.ptr, buf.length) )();<\/pre>\n
      Let’s take a closer look at this escape to see what is actually happening. D allows declaring a lambda function<\/a> that evaluates and returns a single expression, with the () => expr<\/code> syntax<\/a>. In order to call the lambda function, parentheses are appended to the lambda. However, operator precedence will apply those parentheses to the expression and not the lambda, so the entire lambda must be wrapped in parentheses to clarify the call. And finally, the lambda can be tagged @trusted<\/code> as shown, so the call is now usable from the @safe<\/code> context that contains it.<\/p>\n
      In addition to simple lambdas, whole nested functions<\/a> or multi-statement lambdas can be used. However, remember that adding a trusted nested function or saving a lambda to a variable exposes the rest of the function to potential safety concerns! Take care not to expose the escape too much because this risks having to manually verify code that should just be mechanically checked.<\/p>\n
      Rules of Thumb for @trusted<\/h3>\n
      The previous examples show that tagging something as @trusted<\/code> has huge implications. If you are disabling memory safety checks, but allowing any @safe<\/code> code to call it, then you must be sure that it cannot result in memory corruption. These rules should give guidance on where to put @trusted<\/code> marks and avoid getting into trouble:<\/p>\n
      Keep @trusted code small<\/h4>\n
      @trusted<\/code> code is never mechanically checked for safety, so every line must be reviewed for correctness. For this reason, it’s always advisable to keep the code that is @trusted<\/code> as small as possible.<\/p>\n
      Apply @trusted to entire functions when the unsafe calls are leaky<\/h4>\n
      Code that modifies or uses data that @safe<\/code> code also uses creates the potential for unsafe calls to leak into the mechanically checked portion of a @safe<\/code> function. This means that portion of the code must be manually reviewed for safety issues. It’s better to mark the whole thing @trusted<\/code>, as that’s more in line with the truth. This is not a hard and fast rule; for example, the read<\/code> call from the earlier example is perfectly safe, even though it will affect data that is used later by the function in @safe<\/code> mode.<\/p>\n
      A pointer allocated with C’s malloc<\/code> in the beginning of the function, and free<\/code>‘d later, could have been copied somewhere in between. In this case, the dangling pointer may violate @safe<\/code>, even in the mechanically checked part. Instead, try wrapping the entire portion that uses the pointer as @trusted<\/code>, or even the entire function. Alternatively, use scope guards<\/a> to guarantee the lifetime of the data until the end of the function.<\/p>\n
      Never use @trusted on template functions that accept arbitrary types<\/h4>\n
      D is smart enough to infer<\/a> @safe<\/code> for template functions that follow the rules. This includes member functions of templated types. Just let the compiler do its job here. To ensure the function is actually @safe<\/code> in the right contexts, create an @safe unittest<\/code>\u00a0 to call it. Marking the function @trusted<\/code> allows any operator overloads or members that might violate memory safety to be ignored by the safety checker! Some tricky ones to remember are postblit<\/a> and opCast<\/a>.<\/p>\n
      It’s still OK to use @trusted<\/code> escapes here, but be very careful. Consider especially possible types that contain pointers when thinking about how such a function could be abused. A common mistake is to mark a range function or range usage @trusted<\/code>. Remember that most ranges are templates, and can be easily inferred as @system<\/code> when the type being iterated has a @system<\/code> postblit or constructor\/destructor, or is generated from a user-provided lambda.<\/p>\n
      Use @safe to find the parts you need to mark as @trusted<\/h4>\n
      Sometimes, a template intended to be @safe<\/code> may not be inferred @safe<\/code>, and it’s not clear why. In this case, try temporarily marking the template function @safe<\/code> to see where the compiler complains. That’s where @trusted<\/code> escapes should be inserted if appropriate.<\/p>\n
      In some cases, a template is used pervasively, and tagging it as @safe<\/code> may make too many parts break. Make a copy of the template under a different name that you mark @safe<\/code>, and change the calls that are to be checked so that they call the alternative template instead.<\/p>\n
      Consider how the function may be edited in the future<\/h4>\n
      When writing a trusted function, always think about how it could be called with the given API, and ensure that it should be @safe<\/code>. A good example from above is making sure safeRead<\/code> cannot accept an array of pointers. However, another possibility for unsafe code to creep in is when someone edits a part of the function later, invalidating the previous verification, and the whole function needs to be rechecked. Insert comments to explain the danger of changing something that would then violate safety. Remember, pull request diffs don’t always show the entire context, including that a long function being edited is @trusted<\/code>!<\/p>\n
      Use types to encapsulate @trusted operations with defined lifetimes<\/h4>\n
      Sometimes, a resource is only dangerous to create and\/or destroy, but not to use during its lifetime. The dangerous operations can be encapsulated into a type’s constructor and destructor, marked @trusted<\/code>, which allows @safe<\/code> code to use the resource in between those calls. This takes a lot of planning and care. At no time can you allow @safe<\/code> code to ferret out the actual resource so that it can keep a copy past the lifetime of the managing struct! It is essential to make sure the resource is alive as long as @safe<\/code> code has a reference to it.<\/p>\n
      For example, a reference-counted type can be perfectly safe, as long as a raw pointer to the payload data is never available. D’s\u00a0std.typecons.RefCounted<\/code><\/a> cannot be marked @safe<\/code>, since it uses\u00a0alias this<\/code><\/a> to devolve to the protected allocated struct in order to function, and any calls into this struct are unaware of the reference counting. One copy of that payload pointer, and then when the struct is free<\/code>‘d, a dangling pointer is present.<\/p>\n
      This can’t be @safe!<\/h3>\n
      Sometimes, the compiler allows a function to be @safe<\/code>, or is inferred @safe<\/code>, and it’s obvious that shouldn’t be allowed. This is caused by one of two things: either a function that is called by the @safe<\/code> function (or some deeper function) is marked @trusted<\/code> but allows unsafe calls, or there is a bug or hole in the @safe<\/code> system. Most of the time, it is the former. @trusted<\/code> is a very tricky attribute to get correct, as is shown by most of this post. Frequently, developers will mark a function @trusted<\/code> only thinking of some uses of their function, not realizing the dangers it allows. Even core D developers make this mistake! There can be template functions that are inferred safe because of this, and sometimes it’s difficult to even find the culprit. Even after the root cause is discovered, it’s often difficult to remove the @trusted<\/code> tag as it will break many users of the function. However, it’s better to break code that is expecting a promise of memory safety than subject it to possible memory exploits. The sooner you can deprecate and remove the tag, the better. Then insert trusted escapes for cases that can be proven safe.<\/p>\n
      If it does happen to be a hole in the system, please report the issue<\/a>, or ask questions on the D forums<\/a>. The D community is generally happy to help, and memory safety is a particular focus for Walter Bright, the creator of the language.<\/p>\n","protected":false},"excerpt":{"rendered":"
      Steven Schveighoffer is the creator and maintainer of the dcollections and iopipe libraries. He was the primary instigator of D’s inout feature and the architect of a major rewrite of the language’s built-in arrays. He also authored the oft-recommended introductory article on the latter. In computer programming, there is a concept of memory-safe code, which […]<\/p>\n","protected":false},"author":9,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[9,20],"tags":[],"_links":{"self":[{"href":"https:\/\/dlang.org\/blog\/wp-json\/wp\/v2\/posts\/283"}],"collection":[{"href":"https:\/\/dlang.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dlang.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dlang.org\/blog\/wp-json\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"https:\/\/dlang.org\/blog\/wp-json\/wp\/v2\/comments?post=283"}],"version-history":[{"count":19,"href":"https:\/\/dlang.org\/blog\/wp-json\/wp\/v2\/posts\/283\/revisions"}],"predecessor-version":[{"id":302,"href":"https:\/\/dlang.org\/blog\/wp-json\/wp\/v2\/posts\/283\/revisions\/302"}],"wp:attachment":[{"href":"https:\/\/dlang.org\/blog\/wp-json\/wp\/v2\/media?parent=283"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dlang.org\/blog\/wp-json\/wp\/v2\/categories?post=283"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dlang.org\/blog\/wp-json\/wp\/v2\/tags?post=283"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}