{"id":2029,"date":"2019-04-08T10:16:24","date_gmt":"2019-04-08T10:16:24","guid":{"rendered":"http:\/\/dlang.org\/blog\/?p=2029"},"modified":"2021-09-30T13:39:46","modified_gmt":"2021-09-30T13:39:46","slug":"project-highlight-dpp","status":"publish","type":"post","link":"https:\/\/dlang.org\/blog\/2019\/04\/08\/project-highlight-dpp\/","title":{"rendered":"Project Highlight: DPP"},"content":{"rendered":"

\"\"D was designed from the beginning to be ABI compatible with C. Translate the declarations from a C header file into a D module and you can link directly with the corresponding C library or object files. The same is true in the other direction as long as the functions in the D code are annotated with the appropriate linkage attribute<\/a>. These days, it\u2019s possible to bind with C++ and even Objective-C.<\/p>\n

Binding with C<\/a> is easy, but can sometimes be a bit tedious, particularly when done by hand. I can speak to this personally as I originally implemented the Derelict collection of bindings<\/a> by hand and, though I slapped together some automation when I ported it all over to its successor project, BindBC<\/a>, everything there is maintained by hand. Tools like dstep exist<\/a> and can work well enough, though they come with limitations which require careful attention to and massaging of the output.<\/p>\n

Tediousness is an enemy of productivity. That\u2019s why several pages of discussion were generated from \u00c1tila Neves\u2019s casual announcement<\/a> a few weeks before DConf 2018 that it was now possible to #include<\/code> C headers in D code.<\/p>\n

dpp is a compiler wrapper<\/a> that will parse a D source file with the .dpp<\/code> extension and expand in place any #include<\/code> directives it encounters, translating all of the C or C++ symbols to D, and then pass the result to a D compiler (DMD by default). Says \u00c1tila:<\/p>\n

What motivated the project was a day at Cisco when I wanted to use D but ended up choosing C++ for the task at hand. Why? Because with C++ I could include the relevant header and be on my way, whereas with D (or any other language really) I\u2019d have to somehow translate the header and all its transitive dependencies somehow. I tried dstep and it failed miserably. Then there\u2019s the fact that the preprocessor is nearly always needed to properly use a C API. I wanted to remove one advantage C++ has over D, so I wrote dpp.<\/p><\/blockquote>\n

Here\u2019s the example he presented in the blog post<\/a> accompanying the initial announcement:<\/p>\n

\/\/ stdlib.dpp\r\n#include <stdio.h>\r\n#include <stdlib.h>\r\n\r\nvoid main() {\r\n    printf(\"Hello world\\n\".ptr);\r\n\r\n    enum numInts = 4;\r\n    auto ints = cast(int*) malloc(int.sizeof * numInts);\r\n    scope(exit) free(ints);\r\n\r\n    foreach(int i; 0 .. numInts) {\r\n        ints[i] = i;\r\n        printf(\"ints[%d]: %d \".ptr, i, ints[i]);\r\n    }\r\n\r\n    printf(\"\\n\".ptr);\r\n}<\/pre>\n
Three months later, dpp was successfully compiling the julia.h<\/code> header<\/a> allowing the Julia language to be embedded in a D program. The following month, it was enabled by default on run.dlang.io<\/a>.<\/p>\n
C support is fairly solid, though not perfect.<\/p>\n
Although preprocessor macro support is one of dpp\u2019s key features, some macros just can\u2019t be translated because they expand to C\/C++ code fragments. I can\u2019t parse them because they\u2019re not actual code yet (they only make sense in context), and I can\u2019t guess what the macro parameters are. Strings? Ints? Quoted strings? How does one programmatically determine that #define FOO(S) (S)<\/code> is meant to be a C cast? Did you know that in C macros can have the same name as functions and it all works? Me neither until I got a bug report!<\/p><\/blockquote>\n
Push the stdlib.dpp<\/code> code block from above through run.dlang.io<\/a> and read the output to see an example of translation difficulties.<\/p>\n
The C++ story is more challenging.  \u00c1tila recently wrote about<\/a> one of the problems he faced. That one he managed to solve, but others remain.<\/p>\n
dpp can\u2019t translate C++ template specialisations on reference types because reference types don\u2019t exist in D. I don\u2019t know how to translate anything that depends on SFINAE because it also doesn\u2019t exist in D.<\/p><\/blockquote>\n
For those not in the know, classes in D are reference types in the same way that Java classes are reference types, and function parameters annotated with ref<\/code> accept arguments by reference, but when it comes to variable declarations, D has no equivalent for the C++ lvalue reference declarator, e.g. int& someRef = i;<\/code>.<\/p>\n
Despite the difficulties, \u00c1tila persists.<\/p>\n
The holy grail is to be able to #include a C++ standard library header, but that\u2019s so difficult that I\u2019m currently concentrating on a much easier problem first: being able to successfully translate C++ headers from a much simpler library that happens to use standard library types (std::string<\/code>, std::vector<\/code>, std::map<\/code>, the usual suspects). The idea there is to treat all types that dpp can\u2019t currently handle as opaque binary blobs, focusing instead on the production library types and member functions. This sounds<\/em> simple, but in practice I\u2019ve run into issues with LLVM IR with ldc, ABI issues with dmd, mangling issues, and the most fun of all: how to create instances of C++ stdlib types in D to pass back into C++? If a function takes a reference to std::string<\/code>, how do I give it one? I did find a hacky way to pass D slices to C++ functions though, so that was cool!<\/p><\/blockquote>\n
On the plus side, he\u2019s found some of D\u2019s features particularly helpful in implementing dpp, though he did say that \u201cthis is harder for me to recall since at this point I mostly take D\u2019s advantages for granted.\u201d The first thing that came to mind was a combination of built-in unit tests and token strings:<\/p>\n
unittest {\r\n    shouldCompile(\r\n        C(q{ struct Foo { int i; } }),\r\n        D(q{ static assert(is(typeof(Foo.i) == int)); })\r\n    );\r\n}<\/pre>\n
It\u2019s almost self-explanatory: the first parameter to shouldCompile<\/code> is C code (a header), and the second D code to be compiled after translating the C header. D\u2019s token strings allow the editor to highlight the code inside, and the fact that C syntax is so similar to D lets me use them on C code as well!<\/p><\/blockquote>\n
He also found help from D\u2019s contracts and the garbage collector.<\/p>\n
libclang is a C library and as such has hardly any abstractions or invariant enforcement. All nodes in the AST are represented by a libclang \u201ccursor\u201d, which can have several \u201ckinds\u201d. D\u2019s contracts allowed me to document and enforce at runtime which kind(s) of cursors a function expects, preventing bugs. Also, libclang in certain places requires the client code to manually manage memory. D\u2019s GC makes for a wrapper API in which that is never a concern.<\/p><\/blockquote>\n
During development, he exposed some bugs in the DMD frontend.<\/p>\n
I tried using sumtype<\/code> in a separate branch of dpp to first convert libclang AST entities into types that are actually enforced at compile time instead of run time. Unfortunately that caused me to have to switch to compiling all code at once since sumtype<\/code> behaves differently in separate compilation, triggering previously unseen frontend bugs.<\/p><\/blockquote>\n
For unit testing, he uses unit-threaded, a library he created<\/a> to augment D\u2019s built-in unit testing feature with advanced functionality. To achieve this, the library makes use of D\u2019s compile-time reflection features. But dpp has a lot<\/em> of unit tests.<\/p>\n
Given the number of tests I wrote for dpp, compiling takes a very long time. This is exacerbated by -unittest<\/code>, which is a known issue. Not using unit-threaded\u2019s runner would speed up compilation, but then I\u2019d lose all the features. It\u2019d be better if the compile-time reflection required were made faster.<\/p><\/blockquote>\n
Perhaps he\u2019ll see some joy there when Stefan Koch\u2019s ongoing work on NewCTFE<\/a> is completed.<\/p>\n
\u00c1tila will be speaking about dpp on May 9 as part of his presentation at DConf 2019 in London<\/a>. The conference runs from May 8\u201311, so as I write there\u2019s still plenty of time to register<\/a>. For those who can\u2019t make it, you can watch the livestream (a link to which will be provided in the D forums each day of the conference) or see the videos of all the talks on the D Language Foundation\u2019s YouTube channel<\/a> after DConf is complete.<\/p>\n","protected":false},"excerpt":{"rendered":"
D was designed from the beginning to be ABI compatible with C. Translate the declarations from a C header file into a D module and you can link directly with the corresponding C library or object files. The same is true in the other direction as long as the functions in the D code are […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[26,12,13],"tags":[],"_links":{"self":[{"href":"https:\/\/dlang.org\/blog\/wp-json\/wp\/v2\/posts\/2029"}],"collection":[{"href":"https:\/\/dlang.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dlang.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dlang.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dlang.org\/blog\/wp-json\/wp\/v2\/comments?post=2029"}],"version-history":[{"count":2,"href":"https:\/\/dlang.org\/blog\/wp-json\/wp\/v2\/posts\/2029\/revisions"}],"predecessor-version":[{"id":2031,"href":"https:\/\/dlang.org\/blog\/wp-json\/wp\/v2\/posts\/2029\/revisions\/2031"}],"wp:attachment":[{"href":"https:\/\/dlang.org\/blog\/wp-json\/wp\/v2\/media?parent=2029"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dlang.org\/blog\/wp-json\/wp\/v2\/categories?post=2029"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dlang.org\/blog\/wp-json\/wp\/v2\/tags?post=2029"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}