![\"Digital](\"https:\/\/dlang.org\/blog\/wp-content\/uploads\/2016\/08\/d6.png\")
<\/p>\n<\/p>\n
This post is part of an ongoing series<\/a> on working with both D and C in the same project. The previous two posts looked into interfacing D and C arrays. Here, we focus on a special kind of array: strings. Readers are advised to read Arrays Part One<\/a> and Arrays Part Two<\/a> before continuing with this one.<\/p>\n D strings and C strings are both implemented as arrays of character types, but they have nothing more in common. Even that one similarity is only superficial. We’ve seen in previous blog posts that D arrays and C arrays are different under the hood: a C array is effectively a pointer to the first element of the array (or, in C parlance, C arrays decay to pointers<\/a>, except when they don’t<\/a>); a D dynamic array is a fat pointer<\/em>, i.e., a length and pointer pair. A D array does not decay to a pointer, i.e., it cannot be implicitly assigned to a pointer or bound to a pointer parameter in an argument list. Example:<\/p>\n Beyond that, we’ve got further incompatibilities:<\/p>\n It may appear at first blush as if passing D and C strings back and forth can be a major headache. In practice, that isn’t the case at all. In this and subsequent posts, we’ll see how easy it can be. In this post, we start by looking at how we can deal with NUL termination and wrap up by digging deeper into the related topic of how string literals are stored in memory.<\/p>\n Let’s get this out of the way first: when passing a D string to C, the programmer must ensure it is terminated with Forgetting to use However, because strings in D are immutable, None of this applies when we’re dealing directly with string literals, as they get a bit of special treatment from the compiler that makes D programmers very often find themselves passing string literals to C functions. Walter Bright recognized early on how common this would be and decided that it needed to be just as seamless in D as it is in C. So he implemented string literals in a way that mitigates the two major incompatibilities that arise from NUL terminators and differences in array internals:<\/p>\n These two features may seem minor, but they are quite major in terms of convenience. That’s why I didn’t pass a literal to No need for I want to emphasize that this only applies to string literals<\/em> (of type But there is a little more to this story.<\/p>\n Normal array literals will usually trigger a GC allocation (unless the compiler can elide the allocation, such as when assigning the literal to a static array). Let’s do a bit of digging to see what happens with a D string literal:<\/p>\n To make use of a command-line tool particularly convenient for this example, I compiled the above on 64-bit Linux with all three major compilers using the following command lines:<\/p>\n If we were compiling C or C++, we could expect to find string literals in the read-only data segment, In all three cases, the string is right there in the read-only data segment. The D spec explicitly avoids specifying where a string literal will be stored, but in practice, we can bank on the following: it might be in the binary’s read-only segment, or it might be in the normal data segment, but it won’t trigger a GC allocation, and it won’t be allocated on the stack.<\/p>\n Wherever it is, there’s a positive consequence that we can sometimes take advantage of. Notice in the If you’ve read the GC series on this blog<\/a>, you are aware that the GC can only have a chance to run a collection if an attempt is made to allocate from the GC heap. More allocations mean a higher chance to trigger a collection and more memory that needs to be scanned when a collection runs. Many applications may never notice, but it’s a good policy to avoid GC allocations when it’s easy to do so. The above is a good example of just that: To be very clear: this is only true for string variables that have been directly initialized with a string literal or assigned one. If the value of the variable was the result of any other operation, then it cannot be considered NUL-terminated. Examples:<\/p>\n None of these strings can be considered NUL-terminated. Each case will trigger a GC allocation. The runtime pays no mind to the NUL terminator of any of the literals during the append operations or in the Given that you’re reading a D blog, you’re probably adventurous or like experimenting. That may lead you to discover another case that looks reliable:<\/p>\n The above very much looks like appending multiple string literals in an initialization or assignment is just as reliable as using a single string literal. We can strengthen that assumption with the following:<\/p>\n Compiling the above, again on 64-bit Linux:<\/p>\n Now let’s execute them all:<\/p>\n We see that Well, not so fast.<\/p>\n What’s happening is that all three compilers are performing an optimization known as constant folding<\/em><\/a>. In short, they can recognize when all operands of an append expression are compile-time constants, so they can perform the append at compile-time to produce a single string literal. In this case, the net effect is the same as To be clear: this only works because all of the operands are string literals. No matter how many string literals are involved in an operation, if only one operand is a variable, then the operation triggers a GC allocation.<\/p>\n Although we see that the result of an append operation involving string literals can be passed directly to C just fine, and we’ve proven that it’s stored in read-only memory alongside its NUL terminator, this is not something we should consider reliable<\/em>. It’s an optimization that no compiler is required to perform. Though it’s unlikely that any of the three major D compilers will suddenly stop constant folding string literals, a future D compiler could possibly be released without this particular optimization and instead trigger a GC allocation.<\/p>\n In short: rely on this at your own risk.<\/p>\n Addendum<\/strong>: Compile The same but different<\/h3>\n
extern(C) void metamorphose(int* a, size_t len);\n\nvoid main() {\n int[] a = [8, 4, 30];\n metamorphose(a, a.length); \/\/ Error - a is not int*\n metamorphose(a.ptr, a.length); \/\/ Okay\n}<\/pre>\n\n
string<\/code>, wstring<\/code>, and dstring<\/code>, are encoded as Unicode: UTF-8, UTF-16, and UTF-32 respectively. The C char*<\/code> can<\/em> be encoded as UTF-8, but it isn’t required to be. Then there’s the C wchar_t*<\/code>, which differs in bit size between implementations, never mind encoding.<\/li>\nstring<\/code> is an alias to immutable(char)[]<\/code>. C strings are mutable by default.<\/li>\n\\0<\/code>, which is encoded as 0<\/code> in most character sets); D strings are not required to be NUL-terminated.<\/li>\n<\/ul>\nNUL termination<\/h3>\n
\\0<\/code>. std.string.toStringz<\/code><\/a>, a simple utility function in the D standard library (Phobos)<\/a>, can be employed for this:<\/p>\nimport core.stdc.stdio : puts;\nimport std.string : toStringz;\n\nvoid main() {\n string s0 = "Hello C ";\n string s1 = s0 ~ "from D!";\n puts(s1.toStringz());\n}<\/pre>\ntoStringz<\/code> takes a single argument of type const(char)[]<\/code> and returns immutable(char)*<\/code> (there’s more about const<\/code> vs. immutable<\/code> in Part Two). The form s1.toStringz<\/code>, known as UFCS (Uniform Function Call Syntax)<\/a>, is lowered by the compiler into toStringz(s1)<\/code>.<\/p>\ntoStringz<\/code> is the idiomatic approach, but it’s also possible to append "\\0"<\/code> manually. In that case, puts<\/code> can be supplied with the string’s pointer directly:<\/p>\nimport core.stdc.stdio : puts;\n\nvoid main() {\n string s0 = "Hello C ";\n string s1 = s0 ~ "from D!" ~ "\\0";\n puts(s1.ptr);\n}<\/pre>\n.ptr<\/code> will result in a compilation error, but forget to append the "\\0"<\/code> and who knows when someone will catch it (possibly after a crash in production and one of those marathon debugging sessions which can make some programmers wish they had never heard of programming). So prefer toStringz<\/code> to avoid such headaches.<\/p>\ntoStringz<\/code> does allocate memory from the GC heap. The same is true when manually appending \"\\0\"<\/code> with the append operator. If there’s a requirement to avoid garbage collection at the point where the C function is called, e.g., in a @nogc<\/code> function or when -betterC<\/code> is enabled, it will have to be done in the same manner as in C, e.g., by allocating\/reallocating space with malloc\/realloc<\/code> (or some other allocator) and copying the NUL terminator. (Also note that, in some situations, passing pointers to GC-managed memory from D to C can result in unintended consequences. We’ll dig into what that means, and how to avoid it, in Part Two.)<\/p>\nputs("Hello D from C!".toStringz)<\/code> redundant. Let’s see why.<\/p>\nString literals in D are special<\/h4>\n
\n
const(char)*<\/code>.<\/li>\n<\/ol>\nputs<\/code> in the toStringz<\/code> example. With a literal, it would look like this:<\/p>\nimport core.stdc.stdio : puts;\n\nvoid main() {\n puts("Hello C from D!");\n}<\/pre>\ntoStringz<\/code>. No need for manual NUL termination or .ptr<\/code>. It just works.<\/p>\nstring<\/code>, wstring<\/code>, and dstring<\/code>) and not to string variables; once a string literal is included in an expression, the NUL-termination guarantee goes out the window. Also, no other array literal type is implicitly convertible to a pointer, so the .ptr<\/code> property must be used to bind them to a pointer function parameter, e.g., `giveMeIntPointer([1, 2, 3].ptr)<\/code>.<\/p>\nString literals in memory<\/h4>\n
import std.stdio;\n\nvoid main() {\n writeln("Where am I?");\n}<\/pre>\ndmd -ofdmd-memloc memloc.d\ngdc -o gdc-memloc memloc.d\nldc2 -ofldc-memloc memloc.d<\/pre>\n
.rodata<\/code>, of the binary. So let’s look there via the readelf<\/code> command, which allows us to extract specific data from binaries in the elf object file format, to see if the same thing happens with D. The following is abbreviated output for each binary:<\/p>\nreadelf -x .rodata .\/dmd-memloc | less\nHex dump of section '.rodata':\n 0x0008e000 01000200 00000000 00000000 00000000 ................\n 0x0008e010 04100000 00000000 6d656d6c 6f630000 ........memloc..\n 0x0008e020 57686572 6520616d 20493f00 2f757372 Where am I?.\/usr\n 0x0008e030 2f696e63 6c756465 2f646d64 2f70686f \/include\/dmd\/pho\n...\n\nreadelf -x .rodata .\/gdc-memloc | less\nHex dump of section '.rodata':\n 0x00003000 01000200 00000000 57686572 6520616d ........Where am\n 0x00003010 20493f00 00000000 2f757372 2f6c6962 I?.....\/usr\/lib\n...\n\nreadelf -x .rodata .\/ldc-memloc | less\nHex dump of section '.rodata':\n 0x00001e40 57686572 6520616d 20493f00 00000000 Where am I?.....\n 0x00001e50 2f757372 2f6c6962 2f6c6463 2f783836 \/usr\/lib\/ldc\/x86<\/pre>\n
readelf<\/code> output that there is a dot (.<\/code>) immediately following the question mark at the end of each string. That represents the NUL terminator. It is not counted in the string’s .length<\/code> (so "Where am I?".length<\/code> is 11 and not 12), but it’s still there. So when we initialize a string variable with a string literal or assign a string literal to a variable, the lack of an allocation also means there’s no copying, which in turn means the variable is pointing to the literal’s location in memory. And that means we can safely do this:<\/p>\nimport core.stdc.stdio: puts;\n\nvoid main() {\n string s = "I'm NUL-terminated.";\n puts(s.ptr);\n s = "And so am I.";\n puts(s.ptr);\n}<\/pre>\ntoStringz<\/code> allocates, we don’t need it in either call to puts<\/code> because we can trust that s<\/code> is NUL-terminated, so we don’t use it.<\/p>\nstring s1 = s ~ "...I'm Unreliable!!";\nstring s2 = s ~ s1;\nstring s3 = format("I'm %s!!", "Unreliable");<\/pre>\nformat<\/code> function, so the programmer can’t trust it will be part of the result. Pass any one of these strings to C without first terminating it and trouble will eventually come knocking.<\/p>\nBut hold on…<\/h4>\n
import core.stdc.stdio: puts;\n\nvoid main() {\n string s = "Am I " ~ "reliable?";\n puts(s.ptr);\n}<\/pre>\nimport std.stdio : writeln;\n\nvoid main() {\n writeln("Am I reliable?".ptr);\n\n string s = "Am I " ~ "reliable?";\n writeln(s.ptr);\n}<\/pre>\nwriteln<\/code> is a templated function that recognizes when it’s being given a pointer; rather than treating it as a string and printing what it points to, it prints the pointer’s value. So we can print memory addresses in D without a format string.<\/p>\ndmd -ofdmd-rely rely.d\ngdc -o gdc-rely rely.d\nldc2 -ofldc-rely rely.d<\/pre>\n
.\/dmd-rely\n562363F63010\n562363F63030\n\n.\/gdc-rely\n5566145E0008\n5566145E0008\n\n.\/ldc-rely\n55C63CFB461C\n55C63CFB461C<\/pre>\n
dmd-rely<\/code> prints two different addresses, but they’re very close together. Both gdc-rely<\/code> and ldc-rely<\/code> print a single address in both cases. And if we make use of readelf<\/code> as we did with the memloc<\/code> example above, we’ll find that, in every case, the literals are in the read-only data segment. Case closed!<\/p>\ns = "Am I reliable?"<\/code>. LDC and GDC go further and recognize that the resulting literal is identical to the one used earlier, so they reuse the existing literal’s address (a.k.a. string interning<\/a>). (Note that DMD also performs string interning, but currently it only kicks in when a string literal appears more than twice.)<\/p>\nrely.d<\/code> on Windows with dmd and the binary will yield some very different output:<\/p>\ndmd -m64 -ofwin-rely.exe rely.d\n.\/win-rely\n7FF76701D440\n7FF76702BB30<\/pre>\n