Author Archives: Joakim

Liran Zvibel of WekaIO on using D to Create the World’s Fastest File System

Matrix, the world’s fastest file system, was written in D and recently posted impressive numbers in the IO-500 Node Challenge. It was created by WekaIO, a San Jose, CA, based startup with engineering in Tel Aviv, Israel. Liran Zvibel, the co-founder and CEO of WekaIO, has been a regular speaker at DConf, talking about their use of D at DConf 2015, 2016, and 2018.

WekaIO is an expression of a design goal of D, that you can write your prototype quickly and easily in D, then continue working on the same codebase until it reaches production quality, as opposed to prototyping in a different high-level programming language. Liran took some time out of his busy schedule to answer some questions about WekaIO and their use of D.

Liran Zvibel at DConf 2018

Joakim: Tell us about the enterprise storage market that WekaIO is in. How do you use D in your product?

Liran: Any compute environment has a mix of CPU power, networking, and storage: this is uniform across many organization types and sizes, and whether the infrastructure is in-house (“on premises”) or in the public cloud (“The cloud is just somebody else’s computer,” as they say).

The storage component provides the current state and also the history needed by compute. While compute is stateless (how many times have you rebooted your computer to fix a problem?) and the network is ephemeral, storage must be able to keep its state consistently and coherently while providing enough performance for the compute to do its job (otherwise the running jobs are IO-bound, and nobody likes that).

There are three main types of centralized storage systems:

  • Block storage systems, that provide the abstraction of a local drive that sits remotely. Several systems may get access to their own “volume” on the centralized system. The AWS equivalent for such a system is Elastic Block Store. These volumes are usually not shared between more than one server, and the reason to use them is failure resiliency, reliability, and performance (also some advanced features such as taking a point in time, backup, integration into a VM environment, etc.).
  • File systems, centralized storage that is also shared, and allows several
    servers on the network to access the same data. This is the kind of system
    that WekaIO provides. Traditionally, people have turned to block-based solutions if they needed high performance, then created a local file system based on that shared volume, but with WekaIO we show that we enable a shareable file system that is even faster than a local file system over block storage.
  • Object storage solutions, these enable storing objects with reduced semantics (no ability to modify, data stored is only eventually consistent, etc) to enable cost savings, and generally don’t care about performance.

When I review storage systems here, I talk mainly about block-based and file system solutions as object storage is much simpler and is implemented
using different methods.

Requirements for storage systems are:

  • Reliability
  • Performance (low-latency IOPS and throughput)
  • Features

Traditionally, these systems have a “data path” that cares about reliability and performance, and then a “control path” or “management path” that takes care of higher-level features and making reliability work at a higher level.

For many storage systems, the data path is implemented in some system programming language, such as C or C++, and the management path is implemented in a higher-level language such as Python, Java, Go, etc. Our previous company, XIV (that IBM acquired in 2007) used a private version of C that had polymorphism, generic programming, and integrated RPC, for code that was a mixture of XML, C, and weird header files.

At WekaIO, we use the D programming Language to implement both the data path and the control path, as we can use a single language to get both machine-level access with high performance when needed, and a higher-level language for the features.

Joakim: How did you end up choosing D? You mentioned at DConf that you
initially tried a combination of Python and C++.

Liran: The first part of the work we had done on the C++/Python combo was to work on an efficient in-process RPC mechanism between Python and C++, that would
allow us to refer to the same object from C++ and Python in an efficient way. That way the same object could have run in the C++ context when needing performance, and in Python when needing brevity. The idea was to start implementing the system in Python then convert pieces to C++ as needed, where inter-process communication (inter or intra servers) would be done in C++ only. At that time, we had a prototype of the system implemented in Python and we started working on the C++ part, and the RPC definition first.

When we discovered D, we realized we may be able to get a single language to handle both the high-level and performance requirements, and we started by running some pet projects to verify that this was indeed a working language. The next phase was implementing our RPC over D (which was much more elegant than the C++ version), and a tracing system that would allow us to debug (the tracing system was reviewed in my DConf 2015 talk).

The biggest limitation we had initially was the availability of an optimizing compiler, as the reference compiler, DMD, does not provide assembly that is equivalent to LLVM or even GCC when running on modern x86 processors. After DConf 2015, and with the help of David Nadlinger, we were able to get LDC (the D compiler with an LLVM backend) to compile our code and generate results that met our needs.

Joakim: Andrei Alexandrescu, one of the co-architects of the D language, blogged about visiting you in Tel Aviv: how your code is very compact and that you added new features without growing the codebase much. What key features of D do you use to accomplish this and the speed and other benefits of your storage system?

Liran: The strongest part of D is its generic programming. We use generic programming in two ways, that usually are contradictory but with D they actually work well. One way we use generic programming is to achieve higher performance, some examples:

  • Using static polymorphism, object types get fully assembled by the compiler and runtime runs the correct code.
  • In many cases, passing compile-time arguments saves expensive memory loads and branches leading to much faster execution.
  • Compile-time introspection allows placing objects in memory differently, and also makes code run faster based on static decisions.

Then we use static if, generics, foreach and introspection to allow us to write higher-level code much easier, and also write code once that applies for many cases (watch Andrei’s 2017 DConf keynote – Design by Introspection).

We probably use most of the language features: including User-Defined Attributes, to mark code that our introspection chooses to handle differently later or just for debugging purposes; the built-in unit tests;
obviously ranges (these are clever!); and even some contract programming.

Joakim: D is not that widely known yet, how do you bring developers on board? Any common hangups?

Liran: Our internal environment has a very proprietary form to it, with dependencies and a build environment that is unique to us, but the biggest grief we get from users trying to start using D (even WekaIO employees that try to run some pet projects) is that the first few minutes and out-of-the-box experience with DUB as a package manager and build environment are not streamlined enough. We often get comparisons to other upcoming languages where the on-boarding process is easier and more inviting to new users to write the second project after they compiled the obligatory “hello, world!” program.

Newcomers are usually unaware of D features, but as our code is so full of them, they adapt to it extremely quickly during their welcome project, so it’s not a problem at all.

Joakim: You mention contracting some LDC devs to do work for you. What kinds of fixes have they had to put in and could you talk more generally about the flaws in D
that you’ve had to solve or work around?

Liran: The biggest problem we had initially (over 3 years ago) was that LDC did not even compile our code. The compiler choked on it and failed. The first few iterations were just David getting the compiler to compile. Then after we got a running binary, we had a long series of changes to make sure that the code generated by LDC worked the same way as the DMD-generated code, then we got into adding performance improvements to our code and to LDC itself. The WekaIO codebase is larger than the standard project, and the current LDC that we use (which is available on GitHub as a fork) contains some template instantiation changes that are not part of the standard frontend.

Johan Engelen runs our current LDC efforts and has spent a lot of time on link-time optimization and performance-guided optimizations to get LDC and our code to run faster. Now, Johan works on running our code and tests before advancing our LDC to the next frontend releases, to make sure things still run well. We care deeply about performance, and also about the binary representation of our data structures, and some of the work Johan does each release is to verify that they don’t change.

The biggest issue with D to work around is reliance on the Garbage Collector (GC) throughout the runtime and standard library, Phobos. We have had to work on making sure we can leverage exceptions with no GC, and had to create our own non-GC standard library, as we cannot rely on Phobos for it.

We were able to get our LDC and libraries to a state that we have a very efficient runtime environment that produces extremely fast code (with no GC jitter). Our only grief is around compilation time for our large project.

Joakim: You’ve talked about sponsoring D meetups in Tel Aviv publicizing the D language. What kind of reception have you had? More generally, Tel Aviv is famous for being a startup hotbed: how does WekaIO find being situated there?

Liran: Tel Aviv and Israel are fertile ground for startups. We have run some D meetups and one of them even had Andrei Alexandrescu giving a talk at the Google Campus (our offices were not big enough to fit all the people coming). The reception of the concepts is very good, but unfortunately, we were not able to get other startups to actually start using the D Language. Having a startup in Tel Aviv is both a blessing and a curse, as there is a lot of very strong talent with a lot of experience, especially around infrastructure, but also there is a lot of demand. Alongside the startups, there are many large corporations (Google, Apple, Microsoft, Amazon, and many more) with engineering offices offering very lucrative positions, so even though there is a lot of good “supply,” the “demand” is even stronger and filling positions is not easy.

Joakim is the resident interviewer for the D Blog. He has also interviewed members of the D community for This Week in D and is responsible for the Android port of LDC.

On Tilix and D: An Interview with Gerald Nunn

Joakim is the resident interviewer for the D Blog. He has also interviewed members of the D community for This Week in D and is responsible for the Android port of LDC.

Gerald Nunn is the developer of Tilix (formerly called Terminix), an advanced, open-source, Gtk3-based tiling terminal emulator, which is the most-starred D project on github, recently surpassing even the D reference compiler, DMD. Earlier this year at DConf in Berlin, he talked about how he chose D. His slides and a recorded video are available. In his day job, which has nothing to do with desktop GUI applications, he is a Senior Middleware Solutions Architect at Red Hat.

Joakim: What is a tiling terminal emulator?

Gerald: A tiling terminal emulator allows you to split the terminal into multiple tiles and rearrange them in any layout that makes the most sense for the particular task you are working on. People that work in multiple terminals simultaneously typically find them the most useful, particularly with ever-increasing monitor sizes and resolutions.

While the tiling is cool, the primary reason I created Tilix was that I wanted a terminal emulator that followed the Gnome Human Interface Guidelines (HIG) and used Client-Side Decorations (CSD). Tilix follows the Gnome HIG published here, which means adhering to spacing, layout and other recommendations. Following the HIG is important in order for your application to be consistent with the desktop experience as a whole.

CSD refers to the titlebar of the window, where the client assumes responsibility for it, rather than the display manager, and can populate the titlebar with buttons and other controls. This is part of the Gnome HIG and most default Gnome applications (gedit, files, videos, etc.) use this. The one exception is gnome-terminal which does not use the CSD at all.

Joakim: Can you give some examples of how you conform better to the Gnome HIG?

Gerald: The Gnome HIG specifies a specific design language with regards to how applications running in Gnome should look and feel. Some examples of Tilix following the HIG include the use of CSDs and application menus, following various guidelines such as spacing and layouts, etc. Additionally, the Gnome designers have put together a variety of mockups of how they think various applications should look. Tilix uses the mockups developed by Gnome designers for the terminal where feasible. For example, in Tilix the preferences and profiles dialog used to be separate, however one of the Gnome designers proposed this mockup for gnome-terminal. I went ahead and implemented it in Tilix, for a much better experience than what I had previously.

As a result of using CSDs and following the Gnome HIG, hopefully the experience of using Tilix in Gnome feels more organic to its users.

The interesting thing is the tension between people who use Tilix on Gnome and those who use it on other distributions. While I make no bones about the fact that Gnome is my primary target, I do try to make the experience better in other desktop environments by allowing the user to disable the CSD in favor of a normal titlebar if they so wish.

Joakim: You come to D from primarily a Java background. Do you still write Java-style code in D? Was that easy, i.e. how much did you have to change to write D instead?

Gerald: Yes, my background is in Java. I found it quite interesting at DConf when I asked how many people came from a non C/C++ background that only one other fellow raised his hand.

If you look at my code in Tilix, it does look a lot like Java code. Some of that is due to my background and some is due to GtkD being a class-based wrapper. I find switching between D and Java to be pretty seamless for the most part; there is much less cognitive friction between the two than say, switching between Java and Python.

The biggest D idiom I had to learn was ranges, since it is a pretty foundational feature of D. However that wasn’t overly complicated. Compile-Time Function Evaluation (CTFE) is still something that doesn’t come naturally. I have to look it up again every time I need to use it and my current attempts at CTFE from a code perspective are pretty ugh. I’d like to leverage CTFE more in Tilix as I continue to get more experience with D.

Finally, my lack of C experience means that the other area I struggle with a bit is interfacing with C code. While for the most part it’s pretty straightforward, when I have to start deciphering something complex, it gets a bit hoary. Support for flatpack is currently being held up, as I haven’t summoned up the mental energy to work through some of the issues I am having interfacing with C code.

Having said that, I do have some experience with native code development, as I spent a fair amount of time writing code in Delphi and Object Pascal many, many years ago. This is where most of my GUI experience comes from as well.

Joakim: From your DConf talk, you’re obviously not worried much about the Garbage Collector (GC). Have you had to think about it at all when you’re coding Tilix? Any issues with the GC causing stuttering in the GUI?

Gerald: Coming from Java, the GC is pretty natural for me and I definitely do not consider it a negative for D. I think the GC in D gets a lot of bad press based on Java experiences, but it’s important to remember that the GC in D is quite different than the one in Java. The biggest difference to me is that you have more options to control the GC, since it’s well understood when it can kick off a GC cycle. I’ve been very happy to see more people pushing back on the various reddit threads and forum posts complaining about the use of GC.

I haven’t had any issues with the GC in terms of pauses, and no Tilix users have opened cases about it. I have had a few GC-related issues, primarily around leaking memory due to holding references, but these have all been programmer error rather than an issue with D’s implementation of GC. I have another GTK D application, Visual Grep, where I ran into abysmal performance when loading a large amount of matches in a tight loop. However, simply disabling the GC for that section of the code sped things up immensely.

Joakim: The github repository for Tilix is remarkably clean, no open Pull Requests (PRs) and a low percentage of issues still open. How much time do you spend weekly on Tilix? Is Tilix strictly a hobby or has it become something more?

Gerald: Tilix is strictly a hobby. I probably spend 5 to 10 hours a week on it. At this point it’s a mature application, hence the relatively low number of issues. I also prioritize fixing bugs over adding new features, which helps keep the list manageable.

With regards to pull requests, I’m a firm believer in being responsive, so I will typically respond to a PR within a day or two. As a contributor myself, I know nothing kills interest like seeing your PR languish for weeks, months, or even years. If you want people to contribute, which I definitely do, then I feel you owe contributors the courtesy of responding in a timely fashion.

Now, Tilix is a relatively small project so it’s easy for me to adopt that approach. I can understand why larger projects may have more difficulty in this area.

Joakim: D has a lot of features, how well do you know it? You mentioned that you want to use some of the compile-time capabilities more: which D features do you think Tilix would benefit from in the future and how?

Gerald: I don’t think I know it that well to be honest, beyond the core set of features I use in Tilix. I’m always amazed at the guys on the forum that can argue the merits/drawbacks of the low-level details of the language. That’s definitely not me. I’d like to get better at it, but the reality is that I only use D as a hobby so I can’t invest the same amount of time I do with Java. In addition, my job at Red Hat has more of an infrastructure component than my previous jobs, so a lot of my learning time is spent ramping up on that side of the house.

In terms of features that Tilix would benefit from, I think using CTFE and ranges would be very beneficial for exposing some of the capabilities in GtkD in a more idiomatic way. I have a fair amount of code where it could be a lot more concise with appropriate usage of CTFE. As a simple example, using ranges to support iterating over various artifacts using foreach rather than a classical for loop. However, I think some of the more complex use cases, like supporting D-Bus and GObject, would be very useful.

For those not familiar with GObject, it’s the base level object in GTK and is reference-counted. Being able to create GObjects easily in D, like you can in Python, would make it much easier to interface with some of the APIs; right now doing so is the equivalent of writing it in raw C and it’s somewhat laborious. Mike Wey, the GtkD maintainer, has started doing some work on this.

Joakim: You mentioned at DConf that D has a quick edit-compile cycle: how do you enable that, i.e. what IDE, compilers, toolchain do you use, both for development and then for release?

Gerald: I use MS Visual Studio Code on Linux with the excellent code-d plugin, written by Jan “WebFreak” Jurzitza. It gives me all of the features I need (code completion, tooltip hints, etc.). The only thing I find missing from Java IDEs is a refactoring capability. For development, I use DMD as it has the fastest compile time. Release builds are done using LDC, the D compiler with an LLVM backend, since it generates a smaller and faster binary. I rarely need to fire up a debugger, but when I do I just use GDB from the command-line.

Joakim: What problems have you had with D? What features do you dislike?

Gerald: No major problems from my perspective. I’m generally very happy with the language and find it strikes the right balance between ease of use and capabilities. Most of my dislikes would be centered on the standard library, Phobos, rather than the language itself, and those dislikes directly correlate to lack of manpower.

No major issues with Phobos, but rather a bunch of irritants. For example, you cannot easily use immutable with send in std.concurrency, std.experimental.logger is still experimental, the json parser has issues if localization is set to use a comma instead of a period for decimals, etc., etc. None of them in and of themselves are showstoppers, and most of these are really about having the manpower to polish things up. I’m actually somewhat reluctant to complain about them because I could probably fix some of them myself and submit PRs.

I tend to get more annoyed about the negativity in the forums with regards to GC. I do feel that sometimes people get so wrapped up in what D needs for it to be a perfect systems language (i.e. no GC, memory safety, etc.), it gets overlooked that it is a very good language for building native applications as it is now. While D is often compared to Rust, in some ways the comparison to Go is more interesting to me. Both are GC-based languages and both started as systems languages, however Go pivoted and doubled down on the GC and has seen success. One of the Red Hat products I support, OpenShift, leverages Kubernetes (a Google project) for container orchestration and it’s written in Go.

I think D as a language is far superior to Go, and I wish we would toot our horn a little more in this regard instead of the constant negative discussion around systems programming. Now in fairness, Go has a large corporate sponsor whereas D does not, however the contrast in positioning is still interesting to me.

Joakim: What are your future plans for Tilix?

Gerald: The two biggest features I’d like to add is support for tmux control mode and adding the ability for the popout sidebar to be permanently displayed.

For those not familiar with tmux, it is a terminal multiplexer; it essentially does terminal tiling, but within the terminal itself. It also supports a number of other features, but the most interesting is keeping terminal sessions alive outside of the terminal. Since it does tiling within the terminal, there is a bit of a performance hit, plus from a GUI perspective it can’t leverage native widgets like scrollbars. To mitigate this, it supports something called control mode that enables it to integrate with a tiling terminal emulator, so that spawning of new terminals, i.e. tiling, is managed outside of tmux. This greatly improves its performance, while allowing users to leverage other features that tmux supports. At the moment, only iterm2 on OSX supports this AFAIK.

The sidebar in Tilix is one of the more controversial UI elements. I opted not to implement a tabbed interface, because I found them useless in terms of figuring out which tab I wanted; there simply isn’t enough room on the tab to distinguish them and manually renaming them is a pain. The sidebar is my attempt at an alternative, it renders a thumbnail of each session (aka tab) in a sidebar that can be popped out as needed to switch between sessions.

The Tilix sidebar in action.

Some people have a strong preference for something that is permanently available, unfortunately making the sidebar always visible isn’t easy, as generating the thumbnails takes a significant amount of time due to the way GTK is structured. There are potentially ways to make it work, but there is a time investment required to try different options to see what is feasible and then what is the most effective.

In terms of a dream feature, I’d love to switch to using a terminal emulator that is written natively in D, rather than the GTK VTE (Virtual Terminal Emulator) that is written in C that I’m using now. For those not familiar with it, the VTE is the terminal emulation widget used by Gnome Terminal and is available as a reusable widget. Many terminal emulators in Linux use this widget (gnome-terminal, guake, terminator, tilix, etc.), as it provides a production-ready emulator that has been through a huge amount of testing.

The downside with using it is that any custom features you want to implement that involve the actual terminal emulation layer require modifying the VTE and getting those modifications in upstream. I have a few patches that Tilix supports (for triggers and badges), but frankly I’ve done a poor job of getting them into upstream. Part of the reason for this is VTE is written in C and getting up to speed enough with C to make a quality patch is time-consuming.

So having the terminal emulation written in D would make this much easier, however it’s a huge time investment as people underestimate the amount of work involved. There’s a lot of edge cases in terminal emulation, plus adding all of the stuff to make it user-friendly (search, clicking links, etc.), it’s a lot to take on. I simply don’t have the time to make this a reality unless I win the lottery. If someone wanted to take that part on and create a GTK terminal emulation widget that has all of the needed features and stick with it for the long haul, I’d be happy to work with them to get it integrated with Tilix. Adam Ruppe has already created one that works quite well from my testing of it; if someone wanted to work on converting it to a GTK widget, add the necessary improvements and agree to maintain it, feel free to ping me. 🙂

Joakim: Please take us from your experience first discovering and using D to writing Tilix.

Gerald: Throughout my working career, I’ve always had hobby projects going on. Some things I worked on included a popular add-in for Delphi called Gexperts, a Windows file explorer replacement, a Java IDE called Gel and a popular Android app called OnTrack Diabetes that I sold a few years ago. A couple of years ago, I was looking for something new to work on as my hobby program and settled on the idea of building a desktop application for Linux. I knew it needed to use the GTK toolkit, since Gnome is my preferred desktop environment, and particularly with my past Delphi experience in GUIs which I could leverage. I also knew I wasn’t interested in coding in C or C++, so I had a look at what the alternatives were.

I started with Python, as it has excellent support for GTK and I had done a bit of Jython programming in WebLogic, as the Weblogic Scripting Tool (WLST) used it. However, most of my previous work had been small scripts and I quickly realized that at larger scales Python wasn’t for me. I really prefer statically-typed languages in general and Python’s dynamic typing drove me batty, particularly on a hobby program where I constantly needed to rebuild my mental stack since I worked on it infrequently.

I also looked at Rust and Go, but at the time neither of them had feature-complete GTK bindings. Also, while Rust had a lot of positive press, it had a formidable learning curve and I was less than convinced that its focus on memory safety via the borrow checker was a better approach than GC.

I was aware of D, as I had looked at D previously many years ago and liked the language, but the ecosystem was so weak it just wasn’t that useful for practical work. I gave it a second look and found that it had greatly improved and amazingly, full GtkD bindings were available. It also helped that D and Java are similar enough that picking up D was incredibly easy. Once I learned how ranges worked, it was easy to start cranking out code.

I first created a small application called Visual Grep that wraps grep into a GUI. When I was consulting, I often had to grep large code bases looking for specific patterns and a GUI that made browsing the matches was an absolute necessity. This was a great first application, as I learned quite a few things about D. The application performance was initially shit with large result sets, because the GC was constantly kicking in when loading results due to the constant allocation. Disabling the GC during that tight loop improved the performance immeasurably. I also learned about integrating GTK with D’s multi-threading capabilities.

I was inspired by the UI from Gnome Builder, an IDE, and thought that it would work quite well for a terminal emulator. Thus Tilix was born. Well, actually at first it was called Terminix, but once it started getting popular I received a polite cease-and-desist order from Terminix, the American pest control company. Being Canadian, I wasn’t overly aware of them, hence why I didn’t think too much about the name. Lesson learned: spend time choosing a good name in case your app does become more popular than you expect.

I’ve enjoyed my time with D and it’s been a great language for creating desktop applications in GTK. In many ways, I feel like D is a natural successor to Vala, which was a language created specifically for building GTK applications but has been slowly dying, largely due to its single focus. If you are not building GTK apps, you aren’t using Vala, which means the pool of people using it and working on it is by definition very small.

I also feel that D is a natural successor to Delphi, at least with GtkD, as a potent tool for creating desktop applications. With its fast compile time and as an easy-to-learn language, it brings many of Delphi’s best attributes into the modern age. Plus I can type { and } instead of Begin and End and increase my efficiency by 75% or so. 🙂

Ruminations on D: An Interview with Walter Bright

Joakim is the resident interviewer for the D Blog. He has also interviewed members of the D community for This Week in D and is responsible for the Android port of LDC.

d6Walter Bright is the creator and first implementer of the D programming language. He was an early developer of C++ compilers starting from the mid-’80s, including the first C++ compiler to translate source code directly to object code without using C as an intermediate, and has written compilers for ABEL, C, Java, and Javascript. He believes he is the only person to have written a full C++98 compiler by himself. Empire, one of the first computer strategy games, was written by Walter at Caltech. Before getting into computer programming full-time, Walter worked for Boeing from 1979-1982 on the 757’s flight controls, particularly the stabilizer trim gearbox and system and the elevators.

Joakim: D appears to be picking up speed. The fourth straight DConf took place earlier this year, Wired wrote a nice article a couple years ago, Gartner ranked it in the top 20 languages soon afterwards, and downloads of the reference compiler, DMD, average 1200 per day in recent years. Talk about the current popularity and status of the language and what you’re doing to take it to the next level.

Walter: I don’t worry too much about that. I spend my efforts making D the best language possible, and let the metrics take care of themselves. It’s like being a CEO; he shouldn’t be sweating the stock price, he should be working on making money for the company, then the stock price will take care of itself.

There’s always a stack of things to do, more or less with the most important one on the top, and I pop the top off and work on it, mostly the one with the maximum benefit/cost ratio. The ordering in the stack changes all the time. If an item has others strongly interested in working on it, I defer to them.

I get the jobs that nobody else wants to do. 🙂 Regression fixes for complex problems is a big one. It’s much more fun designing new stuff than maintenance on the existing stuff, dealing with technical debt, etc.

Joakim: In the last year, @nogc and interfacing with C++ have been at the top of your stack. Why? You always used to say that interfacing more with C++ would be too much work.

Walter: It became clear that the garbage collector wasn’t needed to be embedded in most things, that memory allocation could be decided separately from the algorithm. Letting the user decide seemed like a great way forward.

As for C++, I figured out a way to support it that avoided the problems I thought were impractical to deal with. I did a talk about it– PDF slides here– and a Rust user wrote up a partial transcript.

Since that talk, I’ve come up with a solution for exceptions. C++ can throw/catch exceptions of any type. But few people do anything other than throw/catch a reference to a class, like std::exception. All D needed to do was allow throw/catch of a class marked with the extern (C++) attribute. Throw/catch of other types remains unsupported, and most likely will remain unsupported.

In order to make that work, the custom exception handling code in the code generation and the library had to change to work with the DWARF exception handling mechanism. That turned out to be a fair amount of work, as it is rather under-documented. But it’s pretty simple for the user.

Joakim: What do you plan on working on in D for the rest of the year? Work on @nogc and C++ support seems to be ongoing and you’ve tried to increase the use of ranges in the standard library for some time now. Anything else? What is the status of those efforts?

Walter: Those are still high-priority and ongoing efforts. But also we’re flexible; if a major opportunity comes up that needs us to push in a different direction, we’ll adapt. The pervasive use of ranges is advancing rapidly, I’ve been very pleased with the results.

The current focus is on improving memory safety. It’s become more and more important to have verifiable memory safety in a programming language, as the expenses involved when unsafe code is exploited in order to install malware become greater and greater. D has always supported memory safety, but recently we’ve embarked on a much more comprehensive review of memory safety in the core language and are making changes to close the gaps.

Joakim: How much time do you spend on D and what is your daily routine?

Walter: I work full time on D. I probably spend half my time working on the language, and the other half helping others with it, discussing things, doing interviews (!), writing articles, doing conferences, etc.

Joakim: You were 42 when you started working on D and I guess it is the first language you designed? Talk about why you started working on it so late–you were probably older then than most of the D contributors now–and what insights your experience gave you that your younger self or other young contributors may not have.

Walter: ABEL is the first language I designed, and was a solid success for Data I/O. It’s obsolete now, because the electronic devices it was aimed at are now obsolete, but it had a great run for 10 years or so.

Having been writing compilers my whole career, doing tech support for them, and following the various changes in the languages inevitably gave me much insight on what worked and what didn’t. Probably the biggest thing is that simpler is better. But making something simple is actually quite difficult. Anybody can (and does) design a complex solution, but few can see through the complexity to find the underlying simplicity.

Many successful languages were designed by older engineers.

Joakim: Please give some examples of such simplicity and how you were able to find it.

Walter: We nailed it with arrays (Jan Knepper’s idea), the basic template design, compile-time function execution (CTFE), and static if. I have no idea what the thought process is in any repeatable manner. If anything, it’s simply a dogged sense that there’s got to be a better way. It took me years to suddenly realize that a template function is nothing more than a function with two sets of parameters –compile time and run time–and then everything just fell into place.

I was more or less blinded by the complexity of templates such that I had simply missed what they fundamentally were. There was also a bit of the “gee, templates are hard” that predisposes one to believe they actually are hard, and then confirmation bias sets in.

I once attended a Scott Meyers presentation on type lists. He took an hour to explain it, with slide after slide of complexity. I had the thought that if it was an array of ints, his presentation would be 2 minutes long. I realized that an array of types should be equally straightforward.

With CTFE, we just went straight in the front door from asking the question “why can’t we execute this function at compile time, just like constant folding?” I did the initial one just by extending the constant folding logic. Don Clugston took it much further, but at its heart it’s still a modified dwarf. Stefan Koch is currently working on making a real interpreter out of it.

Joakim: Why do you think there hasn’t been a killer app for D yet? For example, Ruby was kind of an obscure language for a dozen years till Rails propelled it into the spotlight.

Walter: I suspect the age of the killer app is behind us. There is so much software existing and being written, for every imaginable purpose, that it’s hard to believe there will even be another killer app of any sort. Of course, predictions are notoriously unreliable.

Joakim: D has a unique approach in the compiled languages market, being mostly community-developed without a major corporate sponsor. C++ had Bell Labs, Rust has Mozilla, Go has Google; all have full-time paid devs on the language, even if they also take open-source contributions. Do you think this is a problem and is D being left behind?

Walter: We recently started a D foundation which will make it a lot easier for corporations to sponsor D.

Joakim: Do you make money off D? I know you’ve contracted with Facebook to write a C++ linter, Flint, and preprocessor, Warp, and that you work with Sociomantic and other companies using D.

Walter: I do some paid consulting work for D, but am careful not to take on so much that it interferes with working on D itself.

Joakim: Do you write much code in D outside of the standard library? If so, talk about recent stuff you’ve written and how the experience has been, plus a bit about using some of the new features.

Walter: D consumes so much of my efforts, there’s not much time to write other D apps other than smallish utilities. Currently, of course, the D compiler front end itself is now in D. But it’s been translated from C++, so isn’t idiomatic D.

Joakim: OK, so I guess Warp is the largest program you’ve written in D lately, about 10 klocs. Can you talk about the experience of actually coding that in D, as opposed to C/C++? What stood out for you?

Walter: What stood out is the speed with which it went together, and the remarkably small number of bugs that surfaced in it after it was released. I credit the extensive use of unit tests for the latter.

Having written another preprocessor before certainly helped, and it would be hard to tease that out as a separate effect. But I still believe that unit testing made the difference, because the way the preprocessor worked was very different from the one I’d done before.

What stood out with D was the ease of changing the data structures to try different ways, compared with doing this in other languages.

Joakim: When you think back to your vision for D as you were starting out in 1999, does it at all resemble that today? Anything big missing that you wanted back then?

Walter: D is far more advanced than what I thought of 15 years ago. Programming language ideas have certainly advanced since then, and D along with it.

D originally wasn’t going to have templates at all, based on my earlier experience with them. But finding a simple way to do them changed everything – even to the point where well over half of a modern D program is templates!

The idea of ranges slowly evolved over time, we’re still learning how to do it right.

bit as a basic type was unworkable. complex as a basic type turned out to be pointless (it works better as a library type). Auto-decoding of UTF-8 to code points turned out not nearly as useful as expected.

Transitive const was a leap of faith, and is consistently overlooked by other languages looking to adopt D features. I have a lot of faith in transitive const, and it is already paying off in making it possible to have pure functions, a key feature for modern programming.