Ownership and Borrowing in D

Digital Mars logo Nearly all non-trivial programs allocate and manage memory. Getting it right is becoming increasingly important, as programs get ever more complex and mistakes get ever more costly. The usual problems are:

memory leaks (failure to free memory when no longer in use)
double frees (freeing memory more than once)
use-after-free (continuing to refer to memory already freed)

The challenge is in keeping track of which pointers are responsible for freeing the memory (i.e. owning the memory), which pointers are merely referring to the memory, where they are, and which are active (in scope).

The common solutions are:

Garbage Collection – The GC owns the memory and periodically scans memory looking for any pointers to that memory. If none are found, the memory is released. This scheme is reliable and in common use in languages like Go and Java. It tends to use much more memory than strictly necessary, have pauses, and slow down code because of inserted write gates.
Reference Counting – The RC object owns the memory and keeps a count of how many pointers point to it. When that count goes to zero, the memory is released. This is also reliable and is commonly used in languages like C++ and ObjectiveC. RC is memory efficient, needing only a slot for the count. The downside of RC is the expense of maintaining the count, building an exception handler to ensure the decrement is done, and the locking for all this needed for objects shared between threads. To regain efficiency, sometimes the programmer will cheat and temporarily refer to the RC object without dealing with the count, engendering a risk that this is not done correctly.
Manual – Manual memory management is exemplified by C’s malloc and free. It is fast and memory efficient, but there’s no language help at all in using them correctly. It’s entirely up to the programmer’s skill and diligence in using it. I’ve been using malloc and free for 35 years, and through bitter and endless experience rarely make a mistake with them anymore. But that’s not the sort of thing a programming shop can rely on, and note I said “rarely” and not “never”.

Solutions 2 and 3 more or less rely on faith in the programmer to do it right. Faith-based systems do not scale well, and memory management issues have proven to be very difficult to audit (so difficult that some coding standards prohibit use of memory allocation).

But there is a fourth way – Ownership and Borrowing. It’s memory efficient, as performant as manual management, and mechanically auditable. It has been recently popularized by the Rust programming language. It has its downsides, too, in the form of a reputation for having to rethink how one composes algorithms and data structures.

The downsides are manageable, and the rest of this article is an outline of how the ownership/borrowing (OB) system works, and how we propose to fit it into D. I had originally thought this would be impossible, but after spending a lot of time thinking about it I’ve found a way to fit it in, much like we’ve fit functional programming into D (with transitive immutability and function purity).

Ownership

The solution to who owns the memory object is ridiculously simple—there is only one pointer to it, so that pointer must be the owner. It is responsible for releasing the memory, after which it will cease to be valid. It follows that any pointers in the memory object are the owners of what they point to, there are no other pointers into the data structure, and the data structure therefore forms a tree.

It also follows that pointers are not copied, they are moved:

T* f();
void g(T*);
T* p = f();
T* q = p; // value of p is moved to q, not copied
g(p);     // error, p has invalid value

Moving a pointer out of a data structure is not allowed:

struct S { T* p; }
S* f();
S* s = f();
T* q = s.p; // error, can't have two pointers to s.p

Why not just mark s.p as being invalid? The trouble there is one would need to do so with a runtime mark, and this is supposed to be a compile-time solution, so attempting it is simply flagged as an error.

Having an owning pointer fall out of scope is also an error:

void h() {
  T* p = f();
} // error, forgot to release p?

It’s necessary to move the pointer somewhere else:

void g(T*);
void h() {
  T* p = f();
  g(p);  // move to g(), it's now g()'s problem
}

This neatly solves memory leaks and use-after-free problems. (Hint: to make it clearer, replace f() with malloc(), and g() with free().)

This can all be tracked at compile time through a function by using Data Flow Analysis (DFA) techniques, like those used to compute Common Subexpressions. DFA can unravel whatever rat’s nest of gotos happen to be there.

Borrowing

The ownership system described above is sound, but it is a little too restrictive. Consider:

struct S { void car(); void bar(); }
struct S* f();
S* s = f();
s.car();  // s is moved to car()
s.bar();  // error, s is now invalid

To make it work, s.car() would have to have some way of moving the pointer value back into s when s.car() returns.

In a way, this is how borrowing works. s.car() borrows a copy of s for the duration of the execution of s.car(). s is invalid during that execution and becomes valid again when s.car() returns.

In D, struct member functions take the this by reference, so we can accommodate borrowing through an enhancement: taking an argument by ref borrows it.

D also supports scope pointers, which are also a natural fit for borrowing:

void g(scope T*);
T* f();
T* p = f();
g(p);      // g() borrows p
g(p);      // we can use p again after g() returns

(When functions take arguments by ref, or pointers by scope, they are not allowed to escape the ref or the pointer. This fits right in with borrow semantics.)

Borrowing in this way fulfills the promise that only one pointer to the memory object exists at any one time, so it works.

Borrowing can be enhanced further with a little insight that the ownership system is still safe if there are multiple const pointers to it, as long as there are no mutable pointers. (Const pointers can neither release their memory nor mutate it.) That means multiple const pointers can be borrowed from the owning mutable pointer, as long as the owning mutable pointer cannot be used while the const pointers are active.

For example:

T* f();
void g(T*);
T* p = f();  // p becomes owner
{
  scope const T* q = p; // borrow const pointer
  scope const T* r = p; // borrow another one
  g(p); // error, p is invalid while q and r are in scope
}
g(p); // ok

Principles

The above can be distilled into the notion that a memory object behaves as if it is in one of two states:

there exists exactly one mutable pointer to it
there exist one or more const pointers to it

The careful reader will notice something peculiar in what I wrote: “as if”. What do I mean by that weasel wording? Is there some skullduggery going on? Why yes, there is. Computer languages are full of “as if” dirty deeds under the hood, like the money you deposit in your bank account isn’t actually there (I apologize if this is a rude shock to anyone), and this isn’t any different. Read on!

But first, a bit more necessary exposition.

Folding Ownership/Borrowing into D

Isn’t this scheme incompatible with the way people normally write D code, and won’t it break pretty much every D program in existence? And not break them in an easily fixed way, but break them so badly they’ll have to redesign their algorithms from the ground up?

Yup, it sure is. Except that D has a (not so) secret weapon: function attributes. It turns out that the semantics for the Ownership/Borrowing (aka OB) system can be run on a per-function basis after the usual semantic pass has been run. The careful reader may have noticed that no new syntax is added, just restrictions on existing code. D has a history of using function attributes to alter the semantics of a function—for example, adding the pure attribute causes a function to behave as if it were pure. To enable OB semantics for a function, an attribute @live is added.

This means that OB can be added to D code incrementally, as needed, and as time and resources permit. It becomes possible to add OB while, and this is critical, keeping your project in a fully functioning, tested, and releasable state. It’s mechanically auditable how much of the project is memory safe in this manner. It adds to the list of D’s many other memory-safe guarantees (such as no pointers to the stack escaping).

As If

Some necessary things cannot be done with strict OB, such as reference counted memory objects. After all, the whole point of an RC object is to have multiple pointers to it. Since RC objects are memory safe (if built correctly), they can work with OB without negatively impinging on memory safety. They just cannot be built with OB. The solution is that D has other attributes for functions, like @system. @system is where much of the safety checking is turned off. Of course, OB will also be turned off in @system code. It’s there that the RC object’s implementation hides from the OB checker.

But in OB code, the RC object looks to the OB checker like it is obeying the rules, so no problemo!

A number of such library types will be needed to successfully use OB.

Conclusion

This article is a basic overview of OB. I am working on a much more comprehensive specification. It’s always possible I’ve missed something and that there’s a hole below the waterline, but so far it’s looking good. It’s a very exciting development for D and I’m looking forward to getting it implemented.

For further discussion and comments from Walter, see the discussion threads on the /r/programming subreddit and at Hacker News.

DMD 2.087.0 Released

Digital Mars logo The latest release of the Digital Mars D compiler (DMD) is now available. Version 2.087.0 marks 44 closed Bugzilla issues and 22 major changes courtesy of 63 contributors. See the changelog for the details and related links. Visit the Digital Mars Downloads page to get the release package for your platform(s).

One of the changes in this release is the end of a transitional period regarding imports, another involves a certain compiler switch and the compilation of Phobos. There’s also something developers on Windows will find useful, and more options for documenting code with Ddoc.

Endings and beginnings

Once upon a time, two related compiler issues were reported in the D bug tracker, where they remained for years beyond measure (it was actually just shy of a decade). These bugs allowed symbols to sometimes be accessed inside a scope in which they weren’t supposed to be visible. Eventually, once the bugs were fixed, two switches were introduced to help users maintain their existing code: -transtion=import caused the code to compile under the old, incorrect behavior, and -transition=checkimport would report on all occurences of the erroneous behavior in a code base. Steven Schveighoffer did a write up about it all on his blog at the time, which is a good read for anyone interested in the details.

In DMD 2.087.0, the transitional period is over. The -transition=import and -transition=checkimports switches no longer have any effect. Henceforth, if you have any existing code you’ve been compiling with -transition=import, your code will break with the new release if you are still relying on the old buggy behavior.

As one period ends, another begins. A new deprecation will give you warnings if you are initializing immutable global data via a static constructor like so:

immutable int bar;
static this()
{
    bar = 42;
}

This behavior is deprecated. Static constructors and destructors are called once per thread. Given that immutable global data is implicitly shared across threads rather than being thread-local like normal D variables, data like bar would be overwritten every time a new thread is spawned. The fix is to ensure that the static constructor is also shared across threads:

immutable int bar;
shared static this()
{
    bar = 42;
}

Static constructors and destructors marked shared are invoked once per process rather than once per thread.

DIP 1000 and Phobos

DIP 1000 was the first D Improvement Proposal submitted after the DIP process was transformed from an informal, wiki-based approach, to a formal, managed approach with structured review periods. It proposed a feature called “Scoped Pointers” intended to “provide a mechanism to guarantee that a reference cannot escape lexical scope”. Unfortunately, the document itself remained in a sort of limbo as the proposed feature was implemented and evolved. Eventually, the implementation diverged from the proposal to such a degree that the DIP was marked as Superseded and retired. But not the feature!

The -preview=dip1000 flag has been available for some time now, but it has been a bit tricky to use given that the standard library could not be compiled with it. With DMD 2.087.0, that is true no more. Phobos now compiles with -preview=dip1000 and D programmers can now more easily make use of the feature.

Given that this is still in preview mode and hasn’t yet seen enough wide-spread use, please don’t be surprised if you uncover any bugs. Please do report any bugs you find to the issue tracker so that they can be squashed as soon as possible.

Explicitly choose the LLD linker on Windows

From the beginning, DMD shipped with self-contained support for 32-bit development on Windows. This was possible because Walter Bright made use of the existing platform libraries and linker (OPTLINK) that he was already shipping with his Digital Mars C and C++ compiler. Unfortunately, OPTLINK only supports the OMF object format, so the output of DMD on Windows was incompatible with the greater Windows ecosystem which is primarily built around the PE COFF output (see this PDF of the specification) of the Microsoft build tools. PE COFF, known as MSCOFF in the D universe, is a Microsoft-specific version of the COFF format.

Eventually, Walter added MSCOFF support to DMD for 64-bit (and later, 32-bit) development, but that required that developers have the Microsoft linker and platform libraries installed. In the past, that meant installing either Visual Studio or the Microsoft Build Tools package along with the Windows SDK. In recent years, installing Visual Studio Community edition would provide everything necessary. When compiling with -m64 or -m32mscoff, OPTLINK would be ignored in favor of the Microsoft linker.

With the release of DMD 2.079.0, the compiler began shipping with the LLVM linker (LLD), a set of platform libraries derived from those that ship with the MinGW compiler, and a wrapper library for the Visual C++ 2010 runtime. From that point forward, when given -m64 or -m32mscoff on Windows, the compiler would search for the Microsoft installation and, if not found, fallback on the bundled linker and libraries if they were installed. For the first time, DMD had self-contained 64-bit output on Windows. (Interestingly, it’s never been self-contained on other platforms, where DMD relies on the system linker and libraries, the presence of which is a given and not a source of complaint as the dependence on the MS linker has been.)

Now that the new set up has been put through its paces for a while and some of the kinks have been ironed out, DMD 2.087.0 makes it possible to explicitly select the bundled MSCOFF import libraries and LLD linker via the command line switch -mscrtlib=msvcrt100.

Markdown support in Ddoc

Ddoc was originally designed as a macro-based system for documenting source code, but its use has expanded beyond that scenario in the years since. Most sections of the dlang.org website are created with Ddoc, as is the dconf.org site. Ali Çehreli used it to write his book, Programming in D.

Support for Markdown-like syntax has been requested now and again in the forums. Now that’s available in DMD 2.087. It’s currently in preview mode, so it requires the -preview=markdown flag. There are some differences from the other flavors of Markdown you may be familiar with, so be sure to read the list of supported features before putting it to use.

Supporting the development of D

Much of the blood, sweat, and tears that went into this and every release of the compiler was provided by volunteers. They do the work they can within the constraints of their knowledge, skills, experience and, the mother of all limiters, time. There are some tasks that have yet to fit within the bounds of any volunteer’s constraints. These will require dedicated effort to strike off the list.

To that end, I’d like to remind everyone that we are raising money for a Human Resource Fund that we will use to bring in some folks specifically to tackle these difficult tasks. WekaIO seeded the fund with a generous contribution and a few of our community members have thrown some of their own resources into the pot with the gratitude of the D Language Foundation.

But we need more! We still have DConf 2019 t-shirts for anyone who wants to throw $60 at the HR fund, as well as DMan shirts and DConf 2020 registration discounts for those able and willing to donate more. See my recent blog post on the topic for details to make sure you donate to the correct campaign in order to get the shirt you want.

Another reminder: if you want to become a Gold Donor or Personal Sponsor on our Open Collective page (which is separate from the HR Fund – again, read my recent blog post to avoid confusion), please either ensure your email address is included in your profile or contact me directly to let me know who you are. Otherwise, I can’t send you a DMan shirt!

Get Ready for Symmetry Autumn of Code 2019

Last year, Symmetry Investments sponsored three programmers to work on projects in the D community. Two completed successfully. Francesco Gallà worked on adding support for HTTP/2 to vibe.d and got the grand prize of a bonus payment and a free trip to DConf 2019, where he gave a talk about the experience and his project. His compatriot, Franceso Mecca, worked on porting an old D1 forking GC to D2 for integration into DRuntime. He has a blog post drafted on that already that I will publish here as soon as he finishes the next round of work on the project and updates the draft with some new data.

Now, on behalf of the D Language Foundation and Symmetry Investments, I’m happy to announce that we are go for Symmetry Autumn of Code 2019! You’ll find the details on the SAoC page here at the blog.

This year it’s the same but different! It’s open once again to any programmer, with a preference for students. Participants will receive $1000 USD for each of three successful milestones for a total of $3000. One participant will be selected for a final $1000 payment and a free trip to DConf 2020.

Thanks to feedback from last year’s participants, we’ve made some adjustments this time around. The most visible is that we’re allowing more time to ramp up the projects. Officially, the project planning phase begins September 1, but realistically it begins a week earlier when the applicants are notified of their application status. The official launch of SAoC 2019 is on September 15, two weeks later than last year. We’ll also require participants to submit weekly status updates to the forums this year to encourage more interaction with the community.

Behind the scenes, we’ll be a little more hands on this year in the areas where it matters. The participants last year were happy that we weren’t looking over their shoulders, but there were areas where they would have found our intrusion be a boon. So this year, we’ll be taking a more active role in helping participants find mentors and ramp up their projects, and we’ll communicate more frequently with each student to determine any other specific support we can provide. We’ll stay out of their code and their workflow–that’s what mentors are for–but we’ll do what we can to help ensure a successful project for each participant.

We’re also doing something different for the mentors this year. The SAoC 2018 mentors received the gratitude of the participants and the organizers, and nothing else. Mentors in SAoC 2019 will get all that gratitude, but they’ll also each receive a one-time payment of $500 at the end of the event for a successful project. If, for whatever reason, a participant is unable to complete the event, the mentor may still be eligible for the payment, but it will have to be evaluated on a case-by-case basis.

Following in the tradition of the Google Summer of Code events we’ve participated in over the years, we used a Wiki page for SAoC 2018 project ideas. This year, we’re using the new dlang/projects repository. Anyone with an idea for a SAoC project is welcome to submit an issue there. We’ll tag suitable projects with a SAOC tag so that SAoC 2019 applicants can easily search for projects that interest them. Please read the SAoC page for some guidelines on the sort of information a good project idea should contain.

We’re looking forward to seeing the applications this year. We’d like to expand the number of participants from last year’s three, but that depends on how many applications we get.

If you are interested in participating, please start thinking about your project ideas! The more detailed your application, the better the odds of being selected. We’ll get the tags going on dlang/projects soon to help get the gears cranking.

For the rest of you, please submit project ideas or consider being a mentor. Potential mentors should either contact me directly or announce their interest in the forums. Applicants in search of a mentor can do the same. The details are on the SAoC 2019 page.

Good luck!

Fuzzing Your D Application with LDC and AFL

Fuzzing, or fuzz testing, is a powerful method to find hidden bugs in your application. The basic idea is to present random input to your application and monitor how it behaves. If it crashes or shows some other unusual behavior then you have found a bug.

The use of true random input is not very effective, as most applications reject such input. Therefore many fuzz testing tools mutate valid input, e.g. flipping one or two bits, and present this mutated input to the application. This approach is easy to automate. A fuzz test can run for hours or days until an input is found which crashes your application.

Fuzz testing is very popular. A lot of security bugs have been found with this method. So it’s better to fuzz test your application by yourself instead of waiting for your users to report serious bugs!

Johan Engelen showed at DConf 2018 and in more detail in a blog post how you can use LLVM libFuzzer to fuzz test your application. For libFuzzer, you need to write a test driver. This is powerful because you can make decisions about the function to test. The downside is that you have to code the test driver.

AFL (short for American Fuzzy Lop, a rabbit breed) is another tool to fuzz test an application. AFL has a different approach than libFuzzer and does not require coding. The application under test has to read its data from stdin or from a file. The binary must be instrumented, which requires a recompile of the application. In case you have no source code for the application you can use AFL together with QUEMU. No instrumentation is required but the tests run much slower.

Because random input is not a good choice, you give AFL one or more valid input files, preferably of a small size. AFL mutates this input file, e.g. by flipping a single bit. This new input file is presented to your application and the reaction on it is observed. With the instrumentation in place, AFL discovers the path the data takes through your application. The relationship between bit flips and different code paths that run because of the bit flips is recorded and used to discover new paths and to trigger unexpected behavior. Input which causes crashes is saved in a directory. The main UI gives a lot of information, including how many unique crashes occured in the test session.

AFL works best if the input is a small binary, e.g. a PNG or a ZIP file. If your application has a more verbose and structured input (e.g. a programming language) then you can provide a dictionary which helps AFL with the basic syntax.

The latest release of AFL has an interesting feature. For instrumenting code compiled with clang, a small LLVM plugin is used. This plugin can also be used with LDC, making it possible to fuzz test your D application!

I used AFL to fuzz test LLtool, my recursive-descent parser generator presented at DConf 2019. LLtool expects a grammar description as a file or on stdin. If no error is found, then a D fragment of a recursive-descent parser is produced. Here, I show my approach.

First of all, you need to install AFL. It is included in most Linux distributions, e.g. Ubuntu. A FreeBSD port is also available. One caveat here: please make sure that the AFL plugin is compiled with the same LLVM version as LDC. Otherwise you will see an error message like

ld-elf.so.1: /usr/local/lib/afl/afl-llvm-pass.so: Undefined symbol "...."

during compilation. In this case, download AFL from the link above and compile it yourself.

Different distributions install AFL in different locations. You need to find out the path. E.g. Ubuntu uses /usr/lib/afl, FreeBSD uses /usr/local/lib/afl. I use an environment variable to record this value for later use (bash syntax):

export AFL_PATH=`/ust/lib/afl`

To instrument your code you have to specify the AFL plugin on the LDC command line:

ldc2 -plugin=$AFL_PATH/afl-llvm-pass.so *.d

You will see a short statistic emitted by the new pass:

afl-llvm-pass 2.52b by <lszekeres@google.com>
[+] Instrumented 16118 locations (non-hardened mode, ratio 100%).

For LLVM instrumentation, AFL requires a small runtime library. You need to link the object file $AFL_PATH/afl-llvm-rt.o into your application.

In my dub.sdl file I created a special build type for AFL. This puts all the steps above into a single place. Plus, you can copy and paste this build type directly to your own dub.sdl file because the only dependencies are AFL and LDC!

buildType "afl" {
    toolchainRequirements dmd="no" gdc="no" ldc=">=1.0.0"
    dflags "-plugin=$AFL_PATH/afl-llvm-pass.so"
    sourceFiles "$AFL_PATH/afl-llvm-rt.o"
    versions "AFL"
    buildOptions "debugMode" "debugInfo" "unittests"
}

Now you can type dub build -b=afl on the command line to instrument your application for use with afl. Do not forget to set the AFL_PATH environment variable, otherwise dub will complain.

Now create two new directories called testcases and findings. Put a small, valid input file into the testcases directory. For example save this

%token number
%%
expr: term "+" term;
term: factor "*" factor;
factor: number;

as file t1.g in the testcases folder. Inputs which crash the application will be saved in the findings directory.

To call AFL, you type on the command line:

afl-fuzz -i testcases -o findings ./LLtool --DRT-trapExceptions=0 @@

Two parts of the command line require further explanation. If the application requires a file for input, you specify the file path as @@. Otherwise AFL assumes that the application reads the input from stdin.

If the application crashes, then AFL saves the input causing the crash in the findings/crashes directory. But the D runtime is very friendly. Exceptions uncaught by the application are caught by the D runtime, a stack trace is printed, and the application terminates. This does not count as a crash for AFL. To produce a crash you have to specify the D runtime option --DRT-trapExceptions=0. For more information, read the relevant edition of This week in D.

It is worth reading the AFL documentation because there it provides a lot of tips and background information. Enjoy watching AFL crashing your application and producing test cases for you!

A long-time contributor to the D community, Kai Nacke is the author of ‘D Web Development‘ and a maintainer of LDC, the LLVM D Compiler.

DConf T-Shirts and the Human Resource Fund

I announced the Human Resource Fund back in April as part of an initiative we’re working on with companies using D in production. It was seeded by WekaIO. Now we need to see it grow.

We always have an excess of t-shirts after DConf. Often, they are given away. This year, we intentionally ordered many more than necessary. Some were given away to “Friends of D” (such as the venue staff), but the rest were intended to help us raise money.

Some of the shirts remained in London. The remainder were given to several volunteers to carry with them back home, the goal being to position them around the world so that we can minimize shipping costs. Now, we’re ready to start sending the shirts through the mail.

Help us by donating $60 to the Human Resource Fund and one of our volunteers will send you a DConf 2019 t-shirt. To take advantage of this, you have to donate through the following campaign:

The D Language Foundation will reimburse each volunteer for the shipping costs. In order to help us minimize those costs and maximize the amount we get to keep, we’ve set up a menu organized by shirt size and region, as shown in the following screenshot.

Screenshot of donation form. This campaign will help us minimize shipping costs and keep track in real time of the number of shirts remaining. Once the shirts are gone, the campaign is closed. So please, when you make your donation, help us out by selecting the region in which you live if there are still shirts available.

Those of you who have seen our Open Collective page may be aware that Gold Donors, those who make a one-time donation of $100 or more, receive a DMan t-shirt. Personal Sponsors, those who commit to an annually recurring donation of $200 or more, receive a DMan t-shirt and a guaranteed Early-Bird discount to the next DConf. We’re extending both of those packages to the Human Resource Fund campaign.

To take advantage of this, please donate through the HR Fund campaign page:

After I receive notification from Flipcause of your donation, I’ll contact you to verify your shipping address and determine the size and color you’d prefer for your new DMan t-shirt.

Dman t-shirts.

Please help us grow this fund! The General Fund is also important and we still welcome donations there, but the Human Resource Fund will be our primary focus for the next few months. The sooner we reach the levels we need, the sooner we can hire people to work on important and difficult tasks in the D ecosystem.

DConf 2019 Recap

In late November of last year, Laeeth Isharc of Symmetry Investments expressed interest in hosting DConf 2019 in London. On a personal note, I had been looking for an excuse to get back to London since my brief visit at the end of the first Berlin DConf in 2016, so as my inbox filled with emails discussing possible venues, my excitement started to build. At one point, the Royal Institution was among the list of candidates.

I couldn’t make an announcement yet as nothing was certain, but I did start teasing it on Twitter and here on the blog. At our first D Language Foundation quarterly meeting on December 1, there was unanimous agreement that London was the place to be. As the days passed and it seemed to be a near certainty, I was eager to make the announcement, but near certainty is not certainty. I had to wait until Symmetry had selected a venue. That news reached my inbox on December 21. I announced it on the blog the next day.

After that, it was time to get into the details.

Planning a DConf

Past editions of DConf were organized either by hosts with employees who regularly organize conferences as part of their job descriptions, or, in the case of DConf 2018, an event planner hired by the host. This year there were no event planners and no dedicated conference organizers. It was a very different experience compared to my first peek behind the DConf curtain last year. Most of the details were hashed out in numerous emails and phone calls with Belinda Liao. Though we can thank Laeeth and Symmetry for making DConf 2019 happen, we owe a big thanks to Belinda for making it work.

I first became acquainted with Belinda–who Laeeth introduced as his “chief of staff in London”, who Symmetry’s tech team affectionately refer to as “the official nag”, and whose official title is Business Manager for Technology at Symmetry–during last year’s Symmetry Autumn of Code. Throughout the planning for DConf, she was the one doing all of the legwork. She also made sure we covered all of the bases, querying me for our requirements, pointing out anything I overlooked, and bringing new ideas.

The venue told us they would handle the live stream, but we also wanted a separate solution for recording and producing the individual talk videos. Belinda hired Stage Engage, who sent a single technician, Rowan While, to get the job done. He set up multiple cameras and sat at his primary camera in a back corner for the entirety of the three days of talks. He and his colleagues did an excellent job and three weeks after DConf the link to download the videos was sitting in my inbox. They’re all available on our YouTube channel and are accessible, along with the slides for each talk, through the DConf 2019 schedule.

During the talk submission period, Ethan Watson reached out to tell us he could submit a talk, help us in reviewing drafts of the speakers’ slides, or volunteer to be the emcee. Andrei suggested he do all three. So he did!

This year was the first time we asked the speakers to submit drafts of their slides. Last year, Andrei participated in a conference where the speakers were required to present their talks via Skype for review prior to the conference. He suggested to me at the end of DConf 2018 that we might want to consider that this year. When Ethan came onboard, he suggested instead simply reviewing drafts of the slides, which is standard procedure at the Game Developer’s Conference where he had previously presented. So we set up a deadline for the speakers to send us their drafts. Ethan reviewed them and provided feedback.

Planning the peripherals

When I first heard we might be going to London, I wanted to find some places to see other than the well-known tourist spots. On my first visit, I’d only had a day to be a tourist. This time, my wife was coming along and we were making a two-week trip out of it. So I hit YouTube to search for some video guides. That’s where I found Joolz Guides.

Julian McDonnell uses the channel to post London history walks and travel films. A filmmaker and actor, he also makes himself available for private guided walks through his web site. The idea came to me later than it should have, but in late February I contacted Julian to see if he would be available for a couple of pre-DConf walks. He doesn’t generally work with large groups, though he told me he had scheduled a pub crawl with 18 employees of a company. So we set 18 as the maximum size of a group, worked out a payment structure based on the total number of people, and I got Andrei’s approval for the foundation to cover the cost.

Around the same time, I got in touch with a pub near the venue. Finding a nightly gathering spot was a concern from the beginning. At past conferences, it was either the “official” hotel or, last year, the venue itself. There were several hotel options around the venue, many of them rather pricey. The budget hotels didn’t strike me as places where we could be holding our nightly “BeerConf”. I wanted to avoid the situation that happened in 2017 in Berlin when the hotel staff kicked us out of the lobby and relegated us to a back room. Belinda offered suggestions and I also sought advice from Russel Winder. Ultimately, I stumbled upon the Prince Arthur Pub while poking around Google Maps one night.

The pub has a second-floor space available for private hire. Getting it booked for three nights was a simple process. It also presented an opportunity for sponsorship. Ali Çehreli had been hoping to get his employer, Mercedes Benz Research and Development North America, to sponsor us in one form or another. We had been looking at potential swag, but now that we had the pub, he got approval for the company to cover the booking fee and a couple of rounds of drinks for each person who joined us each night.

Later on, Symmetry rented a different space nearby for the third night, including beer and food. So we had a cozy neighborhood pub for the first two nights of BeerConf and a more upscale bar for the third. Two different atmospheres that both allowed us all to have a good time.

The tours

The first of the two tours we booked with Julian took place two days before the conference, starting outside the Ritz at Green Park station (which, as it turns out, is right up the street from Symmetry) and ended up at the Strand near Charring Cross. The next day we met at Temple station and went through Temple (the legal district named for Temple Church, which was built by the Knights Templar) and on a winding route through the City of London.

Julian pointed out a number of sites we most likely would not have picked out on our own, giving us little nuggets of history for each. For me, some of the highlights were the building where the Beatles did their famous rooftop concert, the tailor shop that served as a front in the movie Kingsman: The Secret Service, the story of Temple Church (which we decided at the time not to go inside, but which I will visit on my next London trip to see the tomb of William Marshal), St. Etheldreda’s Church in Ely Place (dating back to the reign of Edward I), and the Charterhouse. Oh, and an interesting bit of trivia about the origin of the Japanese word for suit.

Julian is an entertaining guide and I believe everyone enjoyed the tours. If you’re ever in London with time to spare, I recommend you contact him about a private walk or a historical pub crawl.

99 City Road

The conference took place on the second floor of Inmarsat’s Old Street office building. Some of us arrived before 8:00 am on the first day and were directed by the security staff to a cozy little waiting area on the first floor. When the time came, we were guided to a side entrance and issued ID cards that would allow us access through the main entrance through Saturday.

As far as I’m aware, the conference went smoothly for just about everyone on site. There were a few hiccups along the way, most of which were noticed by few, if any, of the attendees. For example, those of us early arrivals on the first day found that the power outlets were located under trapdoors scattered throughout the room, but they were out of range of many of the seats. Before it became an issue, Belinda appeared with two of the venue staff, all bearing power strips. Belinda put out most fires before anyone smelled smoke.

Speaking of fires, we started the third day with a test of the building’s fire alarm system. It wasn’t a drill, just a test, so we didn’t have to go anywhere. All was well. Until Steven Schveighoffer got about 75% through his talk. The Stage Engage team edited it out of their recording, but in the live stream, you can see the point where Steve was interrupted by the fire alarm. This time, it was no test. We had to evacuate the building. Several folks got outside and were told to move down the street before abruptly being called back. Five minutes later, everyone filed back into the conference hall and Steve was able to finish his talk.

The venue staff encountered their own minor issue on the first day. At every DConf, we have mid-morning and mid-afternoon snacks, but the coffee is generally available all day. At 99 City Road, they’re used to events with “coffee breaks” in the mid-morning and mid-afternoon, where the coffee is set up and taken down along with the snacks. They soon learned that many DConf attendees are powered by caffeine, so they adapted and left the coffee out all day for the duration of the conference.

The food provided for our lunch each day was fantastic. Speaking for myself, it’s the best food I’ve ever had at a DConf (I did not attend the 2014 and 2015 editions, but I doubt that they compare). It was so good that I went back for seconds each day, which was possible because we had an abundance of food. If we find ourselves at this venue again, the quality of lunch is something we know we don’t have to worry about.

We did encounter one major issue this year, though it did not affect the conference attendees. A number of remote viewers on the first day encountered issues with the live stream, with some unable to see it and others having audio trouble. The venue was using Webex to handle the live stream. Sinisa Poznanovic, the venue’s A/V tech, attempted to switch to YouTube during the lunch break, but the video was oddly being flipped horizontally. He was unable to resolve the issue before the afternoon session, but he promised to stay after we left in the evening until he got it working. When we came in the next morning, the YouTube live stream was set up and working properly. I have to say it was a pleasure working with Sinisa and he has our gratitude for the great work he did throughout the conference.

The Webex issue is something none of us foresaw, but it’s possible we could have. Belinda had sent me a PDF with the links and login information several days before the conference. At the time, I was in Canterbury hanging out with a couple of old friends. I had never heard of Webex, but a cursory search on my phone showed that it’s owned by Cisco. That and the fact that it’s what the venue crew always use were enough to satisfy me, so I searched no further and went back to my vacation. In hindsight, had I dug more deeply into the search results (which I have since done), I would have learned that 64-bit Linux is not officially supported. Had I posted the login details to the forums as soon as I got them from Belinda, those with negative Webex experiences could have spoken up prior to the conference. Such red flags might have motivated us to insist on using YouTube instead.

In the future, we’ll require YouTube for all of our live streaming and, if we encounter anything new, I’ll enlist some help to do more thorough vetting in an effort to uncover potential problems.

The AGM

One of the proposals that came in during the talk submission period was from Nicholas Wilson outlining an Annual General Meeting. When the selection committee met to select this year’s speakers, we decided it would not be feasible to have an AGM as part of the regular schedule. We agreed instead to hold it before the Hackathon.

Just as the DConf Hackathon isn’t the sort of event most people think of when they hear the term, we didn’t envision the AGM as the sort of meeting corporate shareholders would be familiar with. We wanted to limit it to two hours so that we would have time for people to discuss their Hackathon plans before lunch. Nicholas had the idea and put together the agenda, so he would be the moderator. Ethan and I would roam the room with mics so attendees could ask questions. Initially, we had no plans to live stream the event, but in the end, we decided to do it anyway.

The meeting began with an announcement from Andrei. For those who haven’t heard, he is stepping away from his leadership role in the D Language Foundation. He’s still involved in the D community and still manages the foundation’s finances, but for personal reasons, he can no longer devote the time and attention a leadership role requires. Átila Neves was invited to join the team and take over that role. To what I’m certain will be the benefit of the D community, he accepted. It was made possible because Laeeth, his employer, agreed to allow him to do foundation work on Symmetry time.

One of the benefits of DConf is face-to-face communication. Some of the conversations that take place lead to new ideas, collaborations, and projects, but the majority of them are lost to time and memory. In our first AGM, we have not only the benefit of face-to-face communication but also a video record. We covered a lot of ground in the meeting: DUB, DIP 1000, the PR queue, shared, @property, std.experimental, DMD as a library, the DIP process, and more. The ideas put forward are there on video so they won’t be lost. And, thanks to the note-taking skills of Johannes Loher, we have a nice list of action items to work with.

One direct result of the AGM is that I recently revised the DIP process to address some concerns that have been raised in recent times. More items will be ticked off the action list over time.

The quarterly D Language Foundation meeting

The first two quarterly meetings took place over Skype. This time, we were able to hold it face-to-face. Andrei, Walter, Ali, and I were joined on site by representatives from a handful of D shops. We had scheduled the meeting during one of the talks so that we could ensure we’d have a quiet spot for those who were participating remotely. Unfortunately, we had issues with Google Talk for one remote participant while others who were to participate via Skype were too busy to attend.

Skyping these meetings is better than nothing, but meeting face-to-face was a tremendously more efficient and enjoyable experience. We had productive discussions on several topics that garnered more participation than the Skype meetings do, where the discussions tend to be less animated. The company reps aired their issues, we talked about some future plans, and all went well. Most of the items discussed will benefit the community at large when they are finally realized, e.g. Bugzilla issues and new tooling.

I expect our quarterly meetings will become a regular sideline event at future DConfs.

Until next year…

I thoroughly enjoyed myself at DConf this year. Last year, I was too busy emceeing to have much fun. As Ethan discovered this year, there’s more to the emcee job than one might expect (and I would say he’s much more suited to it than I am). When I did it, I was worried about drinking in the evening and wanted to get in bed early each night, so I only fully participated in BeerConf the final night. This year, I had no such concerns, though I did leave early the second night to surprise my wife for dinner.

I hope that everyone at DConf 2019 enjoyed it as much as I did. I also hope that those who were unable to attend this year, especially those who have never attended a DConf, can make the trip next year no matter where in the world we end up. Just think, it’s only been a few weeks since the conference, but we should be talking about DConf 2020 in just five more months.

The countdown is on!

Revisions to the DIP Process

At the AGM that was held prior to the Hackathon at DConf 2019 in London, I announced that I would be making revisions to the DIP progress aimed at shortening the length of time required to go from the Community Review to a final verdict. I also, in response to Joseph Rushton Wakeling’s feedback about guidance for reviewers, agreed to enhance the existing documentation to clarify what is expected of reviewers during the Community and Final Review stages and to provide guidance on how to provide a good review.

The new documentation is now live in four documents under the new docs subdirectory in the DIP repository (all of which are linked in the README). The PROCEDURE and GUIDELINES documents are still there so that existing links remain valid, but all of their content has been replaced with links to the new documentation. Please consider the new documentation to be in draft form. I have not yet subjected them to intense editing, so any corrections are welcome, as are suggestions on how to enhance them.

Anyone intending to participate in a DIP review by leaving feedback in a review thread should familiarize themselves with the DIP Reviewer Guidelines. I can’t force anyone to do so, but I do consider this mandatory. In my role as DIP manager, trying to summarize long threads that have gone off on in-depth discussions of one thing or another, reading through post after post for any sign of actual feedback, is a time-consuming (and highly annoying!) process. Henceforth, I will be deleting any posts in Community and Final review discussion threads that do not adhere to the guidelines laid out in the above document. As I declare in the document, such posts will be copied and pasted in a separate thread where off-topic discussions of the DIP may continue. It’s not my intention to stifle debate or censor anyone or in any other ray restrict discussion of the DIP–I just want to make my job, and the DIP author’s, easier. So please, for my sake and yours, read and understand the reviewer guidelines.

The content of the GUIDELINES document, which was targeted at DIP authors, has been moved (and slightly modified) into the DIP Author Guidelines. The portion of the PROCEDURE document that was aimed at DIP authors is now in The DIP Authoring Process document. All potential DIP authors should read and understand both documents before submitting a DIP. Failure to do so may result in surprises. For example, I’m going to be more proactive in closing pull requests that are submitted while a DIP is still in development and not yet in a first draft state. So please read!

Finally, the portion of the PROCEDURE document that described the different review stages is now found in the document titled The DIP Review Process. Everyone should read this. The primary difference from the previous document is that I’ve explicitly declared that Community Reviews will always begin in the first seven days of a month and Final Reviews will always begin in the third week of a month. Additionally, where I would formerly allow multiple Community Reviews to take place simultaneously, I now restrict it to one at any given time. The goal is to streamline the process and minimize the time it takes to go from Community Review to a final verdict.

As the new document outlines, the best-case scenario, in which only one round of Community Review is required and no DIPs are in active consideration of the language maintainers when another DIP finishes the Final Review, should look like this:

the DIP enters Community Review in the first week of Month A.
after Community Review, the DIP author will have four weeks to complete any required revisions.
in the third week of Month B, the Final Review begins.
after the Final Review, no revisions are required and no other DIP is under active consideration, so the DIP may immediately move into Formal Assessment.
the language maintainers have enough information to render a verdict on the DIP within 30 days.

So it should take between two and three months for the review process to complete. Again, this is the best-case scenario. I expect it more likely that it will typically take between four and five months, given that some DIPs will need multiple Community Review rounds and some will require revision during Formal Assessment.

As I said at the AGM, I’m always open to improving the process to the extent I can within the boundaries of the current framework. Any fundamental structural changes will need approval by Walter and Átila. If you have any suggestions to strengthen the documentation, please let me know.

Flexible Default Function Parameters via structs with Nullable Fields

The problem

Sometimes we need to combine an aggregate of a set of values with an aggregate of the corresponding set of default values to create a combined result. The result for each member is either the explicitly specified value or, where no value is specified, the default value. This is similar to default function arguments in D. However, D forces one to always specify the first N values in function parameters, but I want to be able to specify an arbitrary subset of the values. Example:

Explicit values: a: 1, b: 2

Default values: a: 3, b: 4, c: 5

Combined result: a: 1, b: 2, c: 5

Possible solutions

The first idea is to use associative arrays. This approach is inefficient, however, because it combines values with string (or enum at best) names at runtime with associative array lookups and stores. It could be used like this (untested) example:

combine(["a": 1, "b": 2], ["a": 3, "b": 4, "c": 5])

I was advised to instead use structs with nullable values (see below) to pass multiple values. This is nearly as efficient as possible because the members of the structs are enumerated at compile time (in fact, I use static foreach in my implementation). So I implemented this solution. The source code is quite useful and released under the Apache 2.0 license.

To represent the (explicit) values of type T, we use a struct member of type Nullable!T. If it is null, this means that the explicit value is missing and the default value is used instead; otherwise the specified value is used.

Example of definition and combination

First, install the struct-params package with DUB (see the DUB documentation; I strongly recommend using DUB to build D projects) or clone my GitHub repository.

Then add the following import to your source:

import struct_params;

Example code:

mixin StructParams!("S", int, "x", float, "y");
immutable S.WithDefaults combinedMain = { x: 12 }; // note y is default initialized to null
immutable S.Regular combinedDefault = { x: 11, y: 3.0 };
immutable combined = combine(combinedMain, combinedDefault);
assert(combined.x == 12 && combined.y == 3.0);

StructParams is a string mixin, a D construct which generates D code at compile time and mixes it in at the point of declaration.

mixin StructParams!("S", int, "x", float, "y"); effectively defines the following struct:

struct S {
  struct Regular {
    int x;
    float y;
  }
  struct WithDefaults {
    Nullable!int x;
    Nullable!float y;
  }
}

WithDefaults is the struct to pass, for example, explicit values (which can be present (non-null) or missing (null) to be replaced with default values). For this, the D template type Nullable is used to represent either a value of a type or a null denoting a missing value.

Regular is just a standard struct with fields. Regular is a type which can be used to pass default values to the function combine.

This function combine combines explicit values with default values (as described above).

Then we assert that the result is correct.

Calling functions

Finally, we will do the main thing for which all the above was intended and call a function with combined values:

float f(int a, float b) {
    return a + b;
}
assert(callFunctionWithParamsStruct!f(combined) == combined.x + combined.y);

The structure combined is “split” into members and the members are passed as parameters to f in the order the fields of the struct are defined, that is in the order their names are specified as arguments to the StructParams string mixin. For those interested in the details: I use the built-in D struct and class property .tupleof to split the structure into a tuple of members (e.g. explicitly calling f with an instance reg of Regular would look like this: f(reg.tupleof);).

We can also call a member function of a struct or class instance (t in the example below):

struct Test {
    float f(int a, float b) {
        return a + b;
    }
}
Test t;
assert(callMemberFunctionWithParamsStruct!(t, "f")(combined) == combined.x + combined.y);

It is very unnatural to call the member f using a string of its name, but I have not found a better solution.

Another variant would be to use callFunctionWithParamsStruct!((int a, float b) => t.f(a, b))(combined), but this way is inconvenient as it requires specifying arguments explicitly.

Final considerations

Note that we cannot currently use struct initializers with named arguments, like S.Regular(x: 11, y: 3.0), as the current version of D does not have this feature. There is a draft D Improvement Proposal (DIP) to introduce the feature, but I hear its author is going to replace it with a more general-case proposal after DConf 2019 in London.

It would be beneficial to implement such structs with default values, but this seems impossible, but representing every possible D value as a literal is apparently impossible (for example, a structure with circular references to other structures seems not to be representable as a literal). We can attempt to implement it for a subset of types of values, or even for some values of types and not for others, but this would require further consideration.

Victor Porton is an open source developer, a math researcher, and a Christian writer. He earns his living as a programmer.

DStep 1.0.0

DStep is a tool for automatically generating D bindings for C and Objective-C libraries. This is implemented by processing C or Objective-C header files and outputting D modules. DStep uses the Clang compiler as a library (libclang) to process the header files.

Background

The first version of DStep was released on the 7th of July, 2012. There have been four subsequent releases, the last of which was on the 16th of January, 2016. Quite a lot has happened in the D world and with DStep since then.

After the release of DStep 0.2.1 in January 2016, there wasn’t much progress on DStep. I had a limited amount of time and chose to spend it on other projects. Fortunately, in 2016, DStep got picked as one of four D-related projects for Google Summer of Code (GSoC). The student who chose to work on DStep was Wojciech Szęszoł. He did a tremendous amount of work and pushed DStep forward by years compared to the time it would have taken me. In fact, I was often a blocker because I couldn’t keep up with reviewing all the changes he made.

New Release

The latest release of DStep contains a huge number of new features and bug fixes. A lot of the new features add support for translating C preprocessor macros in various forms. DStep also gained support for one more platform: Windows. Here follow some of the new features available in DStep 1.0.0:

Support for Simple Defines

This feature adds support for translating a simple form of #define to a manifest constant in D. Example:

#define FOO 1

The above C code is translated to the following D code:

enum FOO = 1;

DStep will try to translate the C code so that the D code looks as much as possible like the original C code. If a #define contains an expression instead of a single literal, DStep will try to preserve the original expression:

#define FOO 1 + 3

Instead of translating this to a manifest constant with the value of 4 (which would be semantically correct), DStep will preserve the original expression and translate it to:

enum FOO = 1 + 3;

This also goes for other types of literals, like hexadecimal literals:

#define FOO 0x1

Here DStep will preserve the hexadecimal literal and translate it to:

enum FOO = 0x1;

Function-Like Macros

DStep is now able to translate function-like macros. This is a pretty advanced feature that requires a small parser for the macros. DStep uses libclang to tokenize the macros and the parses them to be able to do the proper translations. The most basic example looks like:

#define FOO() 0 + 1

The above macro will translate to the following D code:

extern (D) int FOO()
{
    return 0 + 1;
}

Although not shown here (to minimize the examples), DStep will output extern (C): at the top of each file. Therefore, for macros translated to functions, DStep will add extern (D) to give the functions D linkage and mangling.

Here’s an example of a C macro containing parameters:

#define FOO(a, b) a + b

Unfortunately, in C, a and b can be basically anything. D doesn’t have an exact corresponding feature. DStep will translate this as accurately as
possible by outputting a templated function:

extern (D) auto FOO(T0, T1)(auto ref T0 a, auto ref T1 b)
{
    return a + b;
}

The assumption in this translation is that a and b will be a value of some kind of type. They can either be of the same type or of different types. To
avoid copying any of the values, ref parameters are used. Since an rvalue cannot be passed to a ref parameter, auto ref is used instead to properly handle both rvalues and lvalues.

More advanced expressions are supported as well:

#define BAR 4
#define FOO(a, b) a + 3 + (b + BAR) - sizeof(b)

In the above example there’s a combination of parameters, literals, parenthesized expression, usages of other macros, and built-in operators. DStep handles all those and translates it to:

enum BAR = 4;

extern (D) auto FOO(T0, T1)(auto ref T0 a, auto ref T1 b)
{
    return a + 3 + (b + BAR) - b.sizeof;
}

Again, the expression is preserved as closely as possible to the original source code. The parentheses, the reference to the BAR macro, all are preserved.

Token Concatenation

This feature adds support for translating the token concatenation, or token pasting, operator to a D string concatenation:

#define CONCAT(prefix, name) prefix ## name

The above function-like macro concatenates the two given tokens. DStep translates that to a function that converts the arguments to strings and concatenates the two resulting strings. This can then be used together with the string mixin statement to give the same behavior as in C.

extern (D) string CONCAT(T0, T1)(auto ref T0 prefix, auto ref T1 name)
{
    import std.conv : to;

    return to!string(prefix) ~ to!string(name);
}

Another example is parameters combined with tokens:

#define CONCAT(prefix) prefix ## name

This translates similarly to the previous example, but since name is not a parameter this will be translated to a string literal:

extern (D) string CONCAT(T)(auto ref T prefix)
{
    import std.conv : to;

    return to!string(prefix) ~ "name";
}

Preprocessor Constants in Array Sizes

DStep will now preserve preprocessor constants for the size of arrays:

#define Foo 3
int a[Foo];

In previous versions of DStep the translation would just output the size of the array a as 3. This would be semantically accurate but the generated source
code would look less like the original C source code. Now DStep is able to translate preprocessor constants and can, therefore, use the preprocessor constant as the size of the array:

enum Foo = 3;
extern __gshared int[Foo] a;

In the above example, the manifest constant Foo is used as the size of a instead of a plain 3. This more closely matches the original C source code.

Preserving Comments

In previous versions of DStep comments were completely stripped out. With this release DStep is able to preserve comments in the D code from the original C code:

// This comment describes this whole file

// Documentation for the symbol `foo`
void foo();

/* Loose comment */ /* Loose comment */

/*
    Multi-line loose comment.
    Multi-line loose comment.
    Multi-line loose comment.
    Multi-line loose comment.
    Multi-line loose comment.
*/ /* Loose comment */

int a; // this is `a`

In the above example there are three types of comments:

A header comment for the whole file
A preceding comment for the symbol foo
Loose comments not belonging to any symbol
A trailing comment for the symbol a

All of these comments are now properly preserved:

// This comment describes this whole file

extern (C):

// Documentation for the symbol `foo`
void foo ();

/* Loose comment */ /* Loose comment */

/*
    Multi-line loose comment.
    Multi-line loose comment.
    Multi-line loose comment.
    Multi-line loose comment.
    Multi-line loose comment.
*/ /* Loose comment */

extern __gshared int a; // this is `a`

In the above example, notice how the header comment is placed above the extern (C): line. If a module declaration is output in the D file (when the --package flag is used), the header comment will be placed above that as well:

// This comment describes this whole file

module bar.foo;

extern (C):

// Documentation for the symbol `foo`
void foo ();

/* Loose comment */ /* Loose comment */

/*
    Multi-line loose comment.
    Multi-line loose comment.
    Multi-line loose comment.
    Multi-line loose comment.
    Multi-line loose comment.
*/ /* Loose comment */

extern __gshared int a; // this is `a`

It’s also possible to disable the preservation of comments using the --comments=false flag.

Package Prefix

To better help organize bindings, DStep supports the
--package <name.of.package> flag. When this flag is enabled DStep will put all the translated modules in the <name> package. Note, this will only add the package prefix to the module declaration of the translated file. It will not place the output file in a directory corresponding to the package.

Removing Excessive Newlines

DStep will now remove excessive newlines but still preserve spacing for the original C code. This is best illustrated with an example:

int a;


int b;

In the above example there are two newlines between the declarations of a and b. DStep will remove the excessive newline and only output one to still preserve the spacing:

extern __gshared int a;

extern __gshared int b;

But if there is no spacing between the declarations, that is respected as well:

int a;
int b;

In the above example there is no newline between the declarations and DStep will preserve that:

extern __gshared int a;
extern __gshared int b;

Preserving Order of Declarations

In previous versions of DStep the declarations in the translated D code would follow a certain order defined by DStep, aliases first, then constants, then types and last functions. With this release, DStep will now preserve the order of the declarations of the original C code:

void bar();

struct Foo
{
    int a;
};

Previous versions would translate the above to:

struct Foo
{
    int a;
}

void bar ();

with the struct first and then the function declaration. With this release, the order is preserved:

void bar ();

struct Foo
{
    int a;
}

Multiple Input Files

Previous versions of DStep only allowed a single header file as input. With this release, multiple files can be passed to DStep at once. Each input file will produce one D source file as input. To pass multiple input files to DStep, just pass the filenames when invoking DStep.

$ dstep foo.h bar.h

Running the above command will produce two D source files: foo.d and bar.d.

If multiple input files and the -o flag are given, the -o flag specifies the output directory where the D source files will be placed. When multiple input
files are given it’s not possible to specify the names of the D source files.

$ dstep foo.h bar.h -o foobar
$ ls foobar
bar.d foo.d

The above command will place the two D source files in the directory foobar.

`--reduce-aliases` Flag

Normally when DStep translates a header file to a D module it will reduce aliases if possible. DStep contains a set of common typedefs that can be reduced to native D types. That means that code like this:

#include ;
int32_t a = 3;

Will be translated to the following D code:

extern __gshared int a;

In this release of DStep, there’s a new flag, --reduce-aliases. This flag allows the reduce aliases feature to be enabled or disabled. By default it’s enabled, but can be disabled by invoking DStep with the following command: dstep --reduce-aliases=false. When this feature is disabled, it will translate the above example to the following D code:

import core.stdc.stdint;
extern __gshared int32_t a;

It will keep the name int32_t as the type of the variable declaration and add the import for the module that contains the declaration of int32_t.

`--alias-enum-members` Flag

In C, enum members are accessible directly in the global scope. Example:

enum Foo
{
    foo,
    bar
};

enum Foo a = foo;

In D however, enum members need to be qualified with the enum name. The correct translation of the above would be:

enum Foo
{
    foo = 0,
    bar = 1
}


Foo a = Foo.foo;

In this release of DStep a new flag has been added, --alias-enum-members, that enables the generation of aliases to enum members at module scope. This will allow keeping the translation more closely to the original C code:

enum Foo
{
    foo = 0,
    bar = 1
}

alias foo = Foo.foo;
alias bar = Foo.bar;

By default this feature is not enabled for the translated D code to more closely follow the D conventions.

`--translate-macros` Flag

In this release, DStep can now translate several kinds of C macros to their equivalent in D. This might not always be desirable because the translations are not bulletproof. Therefore, there’s a new flag, --translate-macros, which will enable or disable the translation of macros. By default translation of macros is enabled.

libclang Bindings

DStep uses Clang as a library to process the C code. There two main ways of using Clang as a library. One is to use the C++ APIs directly. This will give full access to what Clang can do. The problem with this API is that is not a stable API. It’s also a C++ API and when DStep was first implemented, the C++ integration in D was quite lacking. One of my requirements when implementing DStep was to implement it in D. Therefore, the natural choice was to use the C API, called “libclang”, which is also provided. This library is the main interface intended to be used by editors and IDEs that want to leverage Clang as a library. It’s also recommended to use libclang when accessing Clang from a language other than C++—D in my case.

Since libclang is a C library only exposing C header files, I needed bindings to be able to use from it within D. Up until now these bindings were hand made. Fortunately, these headers are the most forgiving I have ever seen when it comes to translating into D. Only a single header file was needed containing hardly any macros at all. It was quite a quick job of translating these headers by using search-and-replace.

With this release of DStep, since DStep has improved so much, the bindings are now self-hosted. That is, DStep has been used to generate the bindings. In addition to that, generation of the bindings has now been added to the test suite to make sure it doesn’t break.

Support for Windows

DStep was originally developed on macOS since that is my main development platform. Thanks to the Posix standard it was easy to port to Linux as well. The first release of DStep was available for both macOS and Linux. Back in 2011 when the development of DStep first started, Clang did not support Windows. There seems to have been some support for MinGW, but that was not a target supported by DMD. The LLVM and Clang team has made huge progress since then and in 2016 when DStep got picked as a GSoC project, support for Windows was available, targeting compatibility with the Visual Studio compiler.

In this release, thanks to Wojciech Szęszoł, DStep is now available on Windows. Due to Clang being compatible with Visual Studio, DStep needs to be built to use the same object format. When compiling with DMD, that means compiling for 64-bit (via the -m64 flag) or using the -m32mscoff flag when compiling for 32 bit. The Dub package automatically takes care of this.

Continuous Integration/Deployment

In the area of CI/CD quite a few things have happened. Originally the test suite of DStep was implemented using Cucumber and Ruby. These tests were a form of end-to-end test and failed to take advantage of the reasons to use Cucumber in the first place. These tests have now been replaced with a combination of unit tests and end-to-end tests, all implemented in D.

LDC

LDC has been added as a supported compiler. That means that DStep is compiled with LDC in addition to DMD as part of the CI pipelines. Every commit and every pull request is now tested with LDC as well.

Upgrade of Compilers

Both DMD and LDC have been upgraded to their latest versions. In addition, beta and nightly releases are being tested in the CI pipelines. A scheduled job has been added to the CI pipelines as well, which will run once every day to make sure new releases of the compilers won’t break DStep even if no changes have been made to DStep. This also means that only the latest versions of LDC and DMD are supported for building DStep.

Testing Windows using AppVeyor

Since DStep now supports Windows as an additional platform a new CI pipeline has been added in the form of AppVeyor. This is a CI service that provides Windows as a platform to run builds on. This build run compiles DStep both using DMD and LDC and it also builds both 32-bit and 64-bit versions.

Complex Floating-Point Types

Another feature that is new in this version of DStep is that complex floating-point types are now supported. There are three complex types that are supported:
float _Complex, double _Complex and long double _Complex.

float _Complex a;
double _Complex b;
long double _Complex c;

The above code snippet in C is translated to the following D code:

extern __gshared cfloat a;
extern __gshared cdouble b;
extern __gshared creal c;

New Alias Syntax

Typedefs in C header files are translated to alias declarations in the D code. Up until this release they used the old alias syntax: alias oldName newName. Since the previous release of DStep the D language has improved and gained new features. One of them is a new (now considered the standard) alias syntax: alias newName = oldName. It’s easier to follow which name is the alias and which is the original when it’s using a more familiar syntax similar to variable declarations. Here’s an example of how the C code is translated into D code with the new alias syntax:

typedef int foo;

The above C code is translated to the following D code:

alias foo = int;

Custom Global Attributes

By default DStep doesn’t add any attributes like @nogc or nothrow to the translated code. In this release of DStep, support for attributes has been added. Custom global attributes can be enabled with the --global-attribute flag. For a C header file with the following content:

int a;

And invoking DStep with the following command:

$ dstep foo.h --global-attribute @nogc --global-attribute nothrow

Will output the following D code:

@nogc:
nothrow:

extern __gshared int a;

Rename Enums

Unlike in D, enums in C don’t create a new scope for their members, even if a name is given to the enum. Example:

enum
{
    a,
    b
};

enum Foo
{
    c,
    d
};

int e = a;
int f = c;

In the above example it’s possible to access the enum members from both of the enums without qualifying the type. In D, this is not the case for named enums. They require the qualifying the enum member with the type name:

enum
{
    a,
    b
}

enum Foo
{
    c,
    d
}

int e = a; // ok since the first enum is anonymous
int f = Foo.c; // need to qualifying the enum member with the type name

To reduce the risk of symbol conflict it’s quite common for C libraries to prefix enum members with the name of the type:

enum Foo
{
    FooC,
    FooD
};

While the following is perfectly fine in D as well, it gets a bit redundant and verbose to have to specify Foo twice:

enum Foo
{
    FooC,
    FooD
}

int c = Foo.FooC;
int d = Foo.FooD;

For this reason DStep now supports a new flag, --rename-enum-members, which when enabled will try to remove any prefix of the enum member names. Given the
following C header file:

enum Foo
{
    FooA,
    FooB
};

And running DStep as follows:

$ dstep foo.h --rename-enum-members

It will produce the following D code:

enum Foo
{
    a = 0,
    b = 1
}

DStep identified the Foo prefix and removed it from the enum member names. It also converted the names to lowercase to better match the standard D naming
conventions.

By default this feature is not enabled to more closely match the original C code.

Normalize Modules

When the --package flag is specified DStep will add a module declaration to all D modules. By default, it will use the name of the input file as the name of the module. In the C world there’s no direct file-naming convention. Some libraries will use all lowercase letters, some will use snake case, some will use camel case, some will use Pascal case, and so on.

The standard D naming convention for modules (and therefore files) is to use only lowercase letters and underscores, i.e snake case. To help with following this convention DStep now supports the new flag --normalize-modules. When this flag is enabled (and the --package flag is used) DStep will try to convert the name of the input file to a name matching the D conventions.

Given a C header file named Foo.h and only invoking DStep with the --package flag:

$ dstep Foo.h --package bar

DStep will produce the following D code:

module bar.Foo;

When the --normalize-modules flag is used as well:

$ dstep Foo.h --package bar --normalize-modules

DStep will output the following D code:

module bar.foo;

Note that Foo has been converted to foo.

Another example with a file using the Pascal naming convention:

$ dstep NSString.h --package bar --normalize-modules

And the result:

module bar.ns_string;

By default this feature is not enabled to more closely match the original C code.

Bit Fields

Another new feature that has been added in this release of DStep is support for bit fields. The bit field is a built-in language construct in C, but there’s no language support for it in D. Fortunately, with the help of D’s metaprogramming capabilities, the bit field has been implemented as a library construct and is available in the standard library [3]. The library construct will generate getters and setters that perform the same bit manipulation that
the C compiler would have generated.

The following snippet in C:

struct Foo
{
    unsigned int a : 1;
    unsigned int b : 2;
    unsigned int c : 5;
};

Is translated to D:

struct Foo
{
    import std.bitmanip : bitfields;

    mixin(bitfields!(
        uint, "a", 1,
        uint, "b", 2,
        uint, "c", 5));
}

The D translation makes use of the bitfields template from the standard library. It’s automatically imported, directly inside the struct, to minimize the scope of where the symbol is available.

In his day job, Jacob Carlborg is a DevOps engineer for Derivco Sweden, but he’s been using D on his own time since 2006. He is the maintainer of numerous open source projects, including DStep, a utility that generates D bindings from C and Objective-C headers, DWT, a port of the Java GUI library SWT, and DVM, the topic of another post on this blog. He implemented native Thread Local Storage support for DMD on OS X and contributed, along with Michel Fortin, to the integration of Objective-C in D.

Human Resources in the D Ecosystem (or Resources, Resources, Resources)

In November of last year, I announced here that we were launching a Pull Request Manager campaign. We wanted to raise $3000 in three months as compensation for Nicholas Wilson to make a dent in the pull request queues of the core D projects. The community answered the call and Nicholas got to work. It was a successful run, so we decided to go another round.

About a month remains in the second round campaign, but donations have come in at a slower pace. Still, we’re confident the community will once again help us reach the finish line so we can compensate Nicholas for his time. We intend to launch a third round at some point in the not-too-distant future, but before we do that we’ve got some different fish to fry.

The Human Resource Share

In December 2018, the first Quarterly D Language Foundation meeting was held online. It came together at the prodding of the aforementioned Nicholas Wilson. Some well-known D shops—Dunhumby (Sociomantic), Funkwerk, WekaIO, and Symmetry Investments—were invited to send representatives to join Walter, Andrei, Ali Çehreli, Nicholas, and myself in a Skype call. Given that it was the inaugural meeting, the agenda was light. We primarily wanted to hear what the companies’ biggest D issues were at the time so that we could prioritize them in the work pipeline. However, we did raise one idea that we wanted the companies to consider for our benefit.

A persistent problem in the D ecosystem is a lack of available human resources to work on the issues that don’t fall into the realm of personal or corporate interests. By that I mean that volunteers tend to contribute where they have a personal interest and contributions from the companies tend to be aimed almost exclusively at areas that have a direct benefit to their projects. The result is that a large number of issues that do not fit into either category fall by the wayside. This is only to be expected and we aren’t complaining. What we are doing is trying to determine how to direct energy toward those neglected issues without the need for raising money.

So we asked the companies to consider a form of “human resource sharing”, the idea being that each company would periodically designate one employee to spend a day on the company dime working on D ecosystem tasks that don’t necessarily have a direct impact on the company’s interests. The representatives promised to take it back to their bosses and give us an answer at the next meeting.

In the interim, I wrote up a more concrete proposal that outlined two options for approaching it: a monthly rotation where each company takes turns doing the work, and a quarterly system where each company commits to completing at least one item on the task list per quarter. I asked the companies to provide us with their preference at our second quarterly meeting in March.

At the March meeting, we invited a few more companies to join us. Given that these are either in the startup phase or aren’t using D exclusively, I’ll play it safe and keep quiet about who they are for now. Overall, the response to the Human Resource Share was positive. Unfortunately, most of the companies are already short on human resources as it is and cannot commit to our quarterly scheme. However, one company did commit to starting immediately and another committed to providing work as they can. All of them committed to helping in other ways, which includes the provision of funds (see below), and hope to have the human resources to spare in the future.

To that end, we’ve set up the ecotasks repository to house our Ecosystem Task List. The list was initially envisioned as a collection of specific tasks, e.g. specific Bugzilla issues, but that makes it more difficult for each tasked worker to decide what to do. Instead, we’ve cobbled together a set of task groups. For now, that consists primarily of links to the GitHub Issues page for different projects and a request to “close as many issues as possible”. The idea is that workers can go to an issues page and work on solving those they can squeeze into their allotted time.

We’ve put this on GitHub not just as a means of transparency, but also because we would like to invite the entire community to participate. The list is loosely sorted by priority in that items higher on the list are considered higher priority than those lower on the list, but there’s no relative priority between specific tasks. The ordering is sure to change over time.

We ask that anyone working through the list to, at the start of the work session, open an issue and leave a comment indicating which item is being worked on. For example:

I’m working on the dub registry issues right now. Specifically, I plan to tackle issues #I, #J, and #K.

Then, at the end of the work session, close the issue with a note indicating what was accomplished.

If you have a little time to spare one weekend, please consider visiting the ecotasks repository and taking on one or two issues. Even better, challenge yourself to go through it once a month and see what you can accomplish. D depends on volunteer effort to thrive. We have a lot of it already, but we always need more. This is one of many ways to make an impact even if it isn’t an enjoyable or very visible one.

The Human Resource Fund

There are some tasks in the D ecosystem that no amount of cajoling and begging will get done because they’re too complex, too time consuming, require a specific skill set to properly complete, or all of the above. When the companies offered to throw money at us in place of human resources, that led us to a new idea.

We are now running a permanent fundraising campaign specifically aimed at solving the bigger issues. The Human Resource Fund for D Ecosystem Tasks is intended to grow and grow and grow. We’re currently in talks with some of the companies about how often and how much they can contribute toward it and in what amounts. We also invite the community at large to donate to it now and again.

Recently, Andrei mentioned in the forums that we need to put together a qualified team to complete the spec and implementation of shared. That’s an example of the sort of big issue we want to use this fund to solve. Donations small and large are equally welcome. The sooner we can get a nice pile built up, the sooner we can start prioritizing issues and finding the people to solve them.

For now, we want to focus on beefing this fund up a bit so we’re going to hold off temporarily on the next Pull Request Manager campaign, but we’ll definitely come back to that again before too long. Anyway, the current campaign is still in need of your attention!

Effecting Change

Every D user has different priorities and goals, different needs and desires. A full-featured IDE is important to one person but not even worth mentioning for another. One programmer expects to see a native D GUI, another is happy with bindings to an existing C or C++ library, and yet another has no need of a GUI at all. One person contributes to a certain D project, but never to any others, while another person has a different set they contribute to, or starts their own. Somewhere in the middle are the issues that are too boring or too complex, the issues that never rise to anyone’s attention or are considered undoable for whatever reason.

In the time I’ve been following and involved with D, I’ve seen the leadership try a number of different ways to drive energy toward some of these unsolved issues. What every approach they’ve tried has had in common is that they depend on the community. When you don’t have the human resources to do the job, you need the money to hire the human resources. When you don’t have the money, you need to ask for and rely upon the charity and goodwill of others. And if there’s no one with the bandwidth to continuously push the issue, fewer people step up. When the community doesn’t step up, then either someone on the core team has to (at the expense of time taken from their normal workload) or the issue languishes.

The two initiatives I’ve described above, the Human Resource Share and the Human Resource Fund, are the latest attempts to make things happen. Again, all contributions are welcome and appreciated! I don’t want this to come off as a complaint, because that’s not what is intended at all.

This is a call to arms! We’re asking members of the community to roll up their sleeves and do the dirty work they normally wouldn’t think to do, or even would prefer not to do. We’re asking for crowdsourced effort in solving problems that will make the D ecosystem better for all of us. A few dozen people spending an hour here or a weekend there will mean more issues closed and more members of the core team can stay focused more often on the work in their purview, which is another big win.

If you can’t help us out with your time, help us out with your money! The Human Resource fund will always need boosting. So, too, the General Fund, the current PR Manager campaign, and any other campaigns we launch in the future. You can also support us when you shop at Amazon by doing so via smile.amazon.com and selecting “D Language Foundation” as your supported charity. When you buy products marked “Eligible for AmazonSmile donation” through smile.amazon.com, the foundation will receive 0.5% of the purchase price. Finally, you’ll soon be able to support D through the DLang Swag Emporium, where you’ll be able to purchase D-themed t-shirts, coffee mugs, and more.

We’re always open to ideas on how to get more things done. If you have anything you’d like to suggest, bring it to the forums for community discussion or email me directly at aldacron@gmail.com.

Now, let’s make things happen!

Ownership

Borrowing

Principles

Folding Ownership/Borrowing into D

As If

Conclusion

Endings and beginnings

DIP 1000 and Phobos

Explicitly choose the LLD linker on Windows

Markdown support in Ddoc

Supporting the development of D

Planning a DConf

Planning the peripherals

The tours

99 City Road

The AGM

The quarterly D Language Foundation meeting

Until next year…

Possible solutions

Example of definition and combination

Calling functions

Final considerations

Background

New Release

Support for Simple Defines

Function-Like Macros

Token Concatenation

Preprocessor Constants in Array Sizes

Preserving Comments

Package Prefix

Removing Excessive Newlines

Preserving Order of Declarations

Multiple Input Files

--reduce-aliases Flag

--alias-enum-members Flag

--translate-macros Flag

libclang Bindings

Support for Windows

Continuous Integration/Deployment

LDC

Upgrade of Compilers

Testing Windows using AppVeyor

Complex Floating-Point Types

New Alias Syntax

Custom Global Attributes

Rename Enums

Normalize Modules

Bit Fields

The Human Resource Share

The Human Resource Fund

Effecting Change

`--reduce-aliases` Flag

`--alias-enum-members` Flag

`--translate-macros` Flag