D in Games: Ethan Watson of Remedy Games Goes to GDC Europe

Posted on

At DConf 2013 at the Facebook HQ, Manu Evans, then of Remedy Games, gave a talk titled, “Using D Alongside a Game Engine” (follow the link for the slides and video). He talked about Remedy’s experience getting D to work with their existing C++ game engine. The D landscape was a bit different when they got underway than it is today. For starters, DMD did not support 64-bit targets on Windows and the language had not yet gained support for directly binding to C++. Manu’s talk discusses the steps they took to work around such issues and achieve a working system.

Fast forward to 2016. Remedy releases Quantum Break, the first commercial AAA game to ship with bits implemented in D. Ethan Watson comes to DConf 2016 with a talk titled, “Quantum Break: AAA Gaming with Some D Code” (follow the link for the slides, go to YouTube for the video). In it, he expanded on what Manu had talked about before. Now, he’s getting ready to take that message to Game Developers Conference Europe.

On the 16th of August this year in Cologne, Germany, I will be presenting a talk titled, “D: Using an Emerging Language in Quantum Break“. For those who have tuned in to DConf previously, it will ostensibly be a combination of both my talk from 2016 and Manu Evans’s talk from 2013, but necessarily cut down due to the fact that it will need to fit two hours of content into one.

He has more in mind than just discussing the technical aspects of Remedy’s solution. During informal discussions at DConf, Ethan spoke of how he would like to bring D to the attention of his colleagues at other studios in the game industry. His upcoming talk presents just such an opportunity.

The approach I’ll be taking is something of a sales pitch to other game developers. My talk at DConf this year was a bit apologetic on my part as I felt we did not use D anywhere near as much as we should, and that we still had unsolved problems at the end of it. GDC Europe will be a platform for me to talk about the journey toward shipping a game with an emerging language in use; a way to show what benefits the language has over the industry stalwart C++ and other modern languages like C# and Rust; and as a bit of a way to drum up support from other developers and get them to at least look at the language and evaluate it for their own usage.

Additionally, a future event Ethan hinted at in his DConf talk will become reality in time for GDC Europe.

We will also be open sourcing our binding solution for GDC. I’ve been cleaning it up over the last couple of weeks, to the point where it’s effectively a complete reimplementation designed for ease of use, ease of reading, and to support some of the additional ideas I’ve had for the system since I took over from Manu. It’ll basically be our way of saying, “D is great! Here’s some code we prepared earlier to help you get up to speed.”

That’s awesome news for the D community as a whole. D has had built-in support for binding to C++ for a little while and it has improved over several releases, though it’s not as fully functional as the support for binding to C (a far simpler task) and likely never will be. There is also an experimental third-party solution called Calypso, which is a fork of the LDC compiler that ultimately aims to provide full and direct interfacing with C++. The availability of an open-source alternative that has been battle-tested in production code can only be a good thing.

Unfortunately, there is one downside to Ethan’s presentation.

They’ve scheduled my talk at the same time as John Romero’s talk. Never thought I’d be competing with him for attention.

Who would want to compete for an audience with one of the co-founders of id software and co-creator of franchises like Doom and Quake, who is giving a talk about the programming principles the id team developed in their early years? Well, if you ask Ethan, the downside is that he can’t skip out of his own talk for Romero’s. Ah, well. As they say, them’s the breaks.

The Why and Wherefore of the New D Improvement Proposal Process

Posted on

When a programmer sees fellow programmers contributing code to open source projects, it is unlikely that the question of why will ever come to mind in terms of motive. Regarding technical merit or usefulness, sure. But motive? Coders who are inclined to contribute bug fixes, new features, build systems, or other resources to open source projects do so for a variety of reasons, most of which other programmers will intuitively understand. But when a programmer volunteers to take over what is solely an administrative process for an open source project, that’s a different story altogether.

Code contributions are usually one-off affairs. They require a time investment up front, but once the PR or patch is merged, that’s often the end of it. Managing an administrative process, on the other hand, is an ongoing commitment. Time that could be spent coding is instead spent doing the proverbial paperwork. And the benefit to the volunteer may not always be clear. Implementing a ray tracer and going bungee jumping are things you might do for the hell of it. Administrative tasks? Not so much.

The most recent post on this blog highlighted two non-coding initiatives which were kicked off by members of the D community. One of them is a new process for handling D Improvement Proposals, or DIPs. This was the work of Mihails Strasuns. Not only did he put together a new process, he also volunteered to manage it. What on earth was he thinking?

There isn’t anything special or unique about DIPs. Most programming languages that rely heavily on community input get something similar at some point. Once a language matures, changing even small bits of it becomes challenging and needs careful consideration. All of the relevant information has to be published somewhere.

The previous DIP process looked like a collection of articles on a community wiki. It seemed to work pretty well initially, when the DIP count was low enough, but with the number of improvement proposals reaching 100 and most of them still remaining in “draft” status, scalability issues became noticeable.

He highlights three issues specifically. First up is quality control.

There was no quality control for submitted proposals and, as a result, submissions were often lacking detailed information about language impact.

With the DIPs being Wiki-based, anyone with a D Wiki account could add a page for a new DIP, but there was no one designated to ensure that new DIPs met any sort of standard. This resulted in quality ranging from a detailed treatise that covered all the bases, to something that looked like notes on a cocktail napkin. Some remain in draft status, having never been finalized by their authors.

Another issue is the lack of response from the leadership.

It was very rare to get a formal response from Andrei or Walter regarding a DIP, partially because proper reviewing of incomplete proposals was demanding too much time.

Walter and Andrei already have their hands full. The last thing they need is to spend time reviewing a DIP that isn’t ready for review. A DIP should be ready for their consideration before it is brought to their attention and not require them to ask for information that should already be there. Without a designated arbiter to decide when a DIP is complete, how are they to know? The author of a DIP may believe it ready, but all too often will be wrong.

Finally, there is the issue of maintaining the list of DIPs.

Abandoned or rejected proposals often were not marked as such because editing the list was purely a community effort.

With no one person responsible for maintaining the list, it was updated only when someone in the community noticed an update was needed and was willing to make the appropriate edits. The result was that there were only a small number of people who ever touched it.

So the solution Mihails came up with to address all of these problems, which he says is similar to that used with other modern programming languages, is described in the README of the new github project he set up to replace the Wiki. It outlines the procedure, including how to submit a DIP and how to get it approved, gives advice on how to write a great DIP, and even outlines the responsibilities of the new DIP manager. Which, for now, is him.

And that brings us back to the topic of the first couple of paragraphs. We now know the problems he saw with the process and how he decided to improve it, but we still don’t know why. So, why, Mihails?

There is, of course, a personal reason which caused me to step up with this process proposal. In a way, this is an attempt to solve a conflict of interest that arises from being employed by one of the largest D commercial users while being actively engaged in the language on my own. I want for the language to change and improve, but, at the same time, I have to ensure any such changes don’t cause too much disruption to our systems at work.

That commercial user of which he speaks is Sociomantic, hosts of the excellent DConf 2016 in Berlin and maintainers of the ocean and makd projects they opened up for the D community.

Having a formalized process for language changes helps a lot in that regard. It means there is a single information feed to follow if one doesn’t want to miss anything of major impact. It’s much more concise than a newsgroup or mailing list. And volunteering to handle the DIP bureaucracy allows me to make timely suggestions for better ways to handle any changes so that they don’t cause too much trouble to existing code.

The DIP process arose organically to fill a void. Its was, perhaps, inevitable that its ad hoc existence take another step in its evolution. It was just waiting for the right person with the right ideas and the willingness to make it happen. That person turned out to be Mihails Strasuns. The D communnity will be better off thanks to his efforts.

The DLang Vision and Improvement Process

Posted on

The evolution of D has been a process of steady, if not always speedy, improvement. In the early days, Walter was the only person working on the language, the reference compiler, and the standard library, though he did accept contributions. Today, numerous people contribute to or maintain a piece of every aspect of D’s development. Semi-formal processes exist where none did before. Things have come a long way.

As with anything in life, those who come later are unaware of the improvements that came before. If you’ve just stepped off the boat into D Land, you don’t remember the days before core D development moved to github, or the time before the first release of Visual D, the D plugin for Visual Studio developed and maintained by Rainer Schuetze. All you’re going to see is that your pet Pull Request hasn’t yet been merged, or that Visual D doesn’t allow you to do all of the things that the core of Visual Studio allows. You’ll rightly head over to the forums to complain about the PR merge process, or request a new feature for Visual D (or gripe about its absence).

And so the story goes, ad infinitum. Along the way, the community loses some members who once found promise in the language, but become frustrated because their pet peeves have not been addressed. And herein lies the point of this post.

The D Programming Language is a community driven language. As such, there is no formal process for creating opportunities to fix and improve the areas that need fixing and improving. Walter has his head down in compiler code and often barely has time to come up for air. Andrei is stretched thin in the additions he has planned for Phobos, the work he does on behalf of the D Foundation, and his numerous other commitments. It’s up to the community to get things moving on other fronts.

Most often, improvements come at the instigation of one highly motivated individual, who either gets busy with the rolling up of sleeves or keeps pushing until someone better positioned takes up the challenge. Still others want to contribute, but aren’t sure where to begin. One place is the documentation, as described the last time this blog touched on this theme. For other areas, the place to look is the D Vision Document.

Andrei started the Vision Document as a means to let the community know the direction in which he and Walter expect the language to go, with plans to release an updated document biannually. Each document lists progress that has been made in the preceding six months and the goals for the coming six. The first document focused on goals the D leadership will personally enable or make happenThe latest document, which was put up for feedback recently in the forums, includes additional goals the leadership strongly believe are important for the success of the D language. This new addition is the result of an initiative by community member Robert Schadek.

Robert took the simple step of creating a Wiki page to collate the wishlist items that Walter and Andrei mention in the forum now and again. These are things they would like to see completed, but have no time to make happen themselves. Andrei merged these items into the Vision Document so that there is only one document to maintain and one source for the information. This is a perfect example of a low-barrier community contribution that has led to an overall improvement (though Robert is no stranger to higher-barrier contributions) and shows how there are ways other than submitting Pull Requests to improve the D Programming Language experience.

A higher-level means of helping to improve the language is through D Improvement Proposals. A DIP is a formal proposal to make changes or additions to the language. The author of the DIP need not be the implementer. For some time, these have primarily been loosely maintained in an informal process at the D Wiki. Recently, Mihails Strasuns decided the process needed better management, so he created a github repository for DIPs as part of a new process to address some shortcomings of the old system. Then he made a forum announcement about it, including the news that he is, for now, volunteering his services as the new DIP manager.

So, to recap, if you are itching to contribute to D in some way that you know will have an impact, the latest Vision Document is the place to go for inspiration. As I write, that’s the 2016H2 document. If you have an idea for a new feature or improvement to the language, creating a DIP is the way to go (a future post here will say more about the DIP process). There are numerous other areas where contributions are welcome, like fixing or reporting bugs, reviewing PRs, and taking the initiative to contribute to the D process like Robert and Mihails have done.

If you don’t have the time, motivation, or inclination to contribute, that’s fine, too. Just keep in mind when you come to the forums to vent about one frustration or another (and please, feel free to do so) that much of what exists in the D ecosystem today, along with nearly every improvement that will come for the foreseeable future, is the result of someone in the community expending their own time and effort to make it so, usually with no financial incentive (the exception being bug bounties). Please let that knowledge color your perspective and your expectations. If you can’t, or won’t, work to resolve shortcomings in the language, the processes, or the ecosystem yourself, your goal in bringing them to the community should be to motivate someone else to do so. Not only will that increase the odds of fixing your issues, it will help all D users inch closer to the ultimate D Nirvana.

Project Highlight: Auburn Sounds

Posted on

One of the questions some people ask when evaluating a language for the first time is, who’s making money with it? That’s no exception with D. While there are a number of companies using D in production, there are also people making money with the language out of their homes. The last Project Highlight looked at a freely available open source project. This one is about a proprietary set of audio plugins developed and sold by one programmer.

Guillaume Piolat has been using D since 2007. In that time, he has been an active member of the community, maintaining and contributing to several open source projects including Derelict and GFM. Now, he has begun to build an audio plugin business that he calls Auburn Sounds.

I have a long love/hate relationship with D and honestly I prefer the programs I’ve made with it over other languages. It’s the language I feel the most free with and it has always served me well. I’m just not comfortable with the low productivity level of C++, so I was willing to take a long term bet.

That bet is now paying off for him, but it hasn’t been completely free of challenges.

I did not know how hard wrapping plugin formats would be, especially on OS X. It turned out you could “derelictify” Carbon and Cocoa and call the Obj-C runtime without much difficulty, just with some work.

By derelictify, he means creating dynamic bindings in D to the C APIs of shared libraries that can then be loaded at run time via system APIs like dlopen. The DerelictUtil library can be used to create loaders for such bindings, hiding the platform-specific details behind a simple API, and the Derelict project provides a number of bindings that do just that. Hence the term derelictify. It’s worth noting here that D has some limited support for interfacing directly with Objective-C.

Other issues included bugs in the tool chain and less-than-ideal code generation.

I stumbled upon some DMD backend bugs and some bugs in DUB, but all of them were fixed in the end. Additionally, DMD codegen wasn’t competitive. Fortunately, LDC has made some incredible progress, bringing top codegen to both Windows and OS X . It’s something that I was expecting, but not so soon! This particular bet really paid off.

It’s commonly recommended in the D community to use DMD for its blazing fast compile times during development and, for the projects that really need it, LDC or GDC for production to take advantage of the fact that they typically produce binaries with better performance.

DUB is a build tool and package manager for D projects. A number of libraries have been registered in the DUB Registry, all of which are available to use as dependencies in any DUB-managed project. The tool will soon be shipping with DMD in an upcoming release of the compiler.

Now that he has several completed plugins available, how does Guillaume feel about having chosen D?

Auburn Sounds has existed for fifteen months, and in the past nine I’ve not thought of going back to C++ a single time. I don’t use D-specific features aggressively. At first, I was thinking that I would need D’s meta-programming support, like Design by Introspection, to create an efficient audio library, but it turned out having almost no abstraction worked well enough. The thing that matters most for this project is codegen quality, speed of development, platform support and low mental overhead.

Other benefits include the fluidity of DUB, the availability of VisualD, and the quality of some third-party libraries like imageformats, DerelictUtil, and especially ae.utils.graphics. Speed of compilation and development count a lot, but they aren’t something you really notice once you’ve grown accustomed to them.

Guillaume intends to continue to use D to develop more audio plugins and improve the ones he has already made available. His latest, Panagement, has both free and paid versions.

Panagement solves two problems in audio mixing: giving stereo content to a track quickly and fixing regular panning, which doesn’t sound that great on headphones. It’s a top dog in a sub-niche that traditionally doesn’t interest people a lot. It’s also the first plugin I’ve released entirely built with LDC.

In addition to Panagement, you can also currently find a voice octaver for sale and three other plugins freely available in their full versions: a binaural panner, a physical synthesizer, and a distortion plugin.

The D community undoubtedly collectively wishes Guillaume, long one of their own, the best of luck. If you are a one man shop or a small team using D to produce commercial software, let us know in the D Forums!

Programming in D: A Happy Accident

Posted on

This is a guest post from Ali Çehreli, who not only uses D as a hobby, but also gets to use it as an employee of Weka.io. He is the author of Programming in D and is frequently found in the D Learn forum with ready answers to questions on using the language. He also is an officer of the D Foundation.


progindI consider the book Programming in D a happy accident, because initially it was not intended to be a book.

I used to be a frequent contributor to Turkish C and C++ forums. Although there were many smart and motivated members on those forums, most of them were not well-versed enough to follow programming resources in English. If they were patient enough to wait about ten years and if a publisher decided to have them translated, then they might get their hands on Turkish versions of their favorite books.

In 2009, around the time when my interest in C++ had started to diminish, I read with great excitement Andrei Alexandrescu’s The Case for D article in ACCU’s C Vu magazine (also available at Dr. Dobb’s). To a person coming from a C++ background, D was a fresh breath of air, removing some of C++’s warts and bringing many new features, some unique, some borrowed from other languages.

I was instantly hooked. I immediately created the Turkish D site ddili.org, translated Andrei’s article to Turkish, and published it there. One of the reasons for my excitement was the potential that D could be one software technology that Turkish programmers would not be left in the dark about. Since D was still being designed and implemented, there was time to write fresh Turkish documentation for it. I translated other D articles and started writing an HTML tutorial that would later become the book.

I knew very well that attempting to teach a topic is one of the best ways of learning that topic. I knew that I would be learning D myself. Little did I know then that this project would make me a better software engineer in general as well.

Teaching programming is a notoriously difficult task. According to some academic papers I found when I started the tutorial, one of the difficulties comes from the fact that different people model new concepts in their minds in different ways, rendering particular teaching methods inefficient at least for some students. Encouraged by the lack of one correct way of teaching programming, I picked one that was the easiest for me: introducing concepts in linear fashion with as few forward references as possible, starting with the most basic concepts like the = character confusingly meaning something different than is equal to.

Starting from the basics made it necessary for me to introduce lower-level concepts before higher-level concepts. For example, although the foreach statement is much more commonly used in practice, while, for, and foreach statements are introduced in the book in that order. I think that choice created a better foundation for the reader.

It took me two years to finish writing a flow of chapters from the assignment operator all the way to the garbage collector. It was very challenging and very rewarding to find a natural flow of presentation not only throughout the book but also within each chapter. The method I used for each chapter was to think about the presentation of the topic along with non-trivial examples beforehand, without touching the computer. I then wrote the chapter fairly quickly without much attention to detail, put it aside for a couple of days, then came back to review it from the point of view of a reader. I repeated that process perhaps five to ten times for each chapter until I thought it was fairly acceptable. Likely as a result of that process, a common feedback I receive is about how to-the-point my writing style has been.

Based on feedback from the Turkish community and encouragement from Andrei Alexandrescu, I started translating the book to English in early 2011. The translation continued along with new chapter additions, many corrections, and some chapter rewrites.

I made a PDF version available in January 2012 and the translation was finally completed in July 2014. Not only had I achieved my initial goal of providing fresh Turkish documentation for D, this book might have been the first software resource that was translated in the other direction.

I readily agreed with the suggestion that the book should be available in paper form as well. That decision brought many different challenges related to self-publishing like layout, cover design, pricing, the printing company, etc. The first print publication was in August 2015. Surprisingly, producing an ebook version turned out to be even more challenging. In addition to different kinds of layout issues, all ebook formats require special attention.

I am awestruck that my humble idea of a humble tutorial turned into a well known resource in the D ecosystem. It makes me very happy that people actually find the book useful. I am also happy that, periodically, people express interest in translating it to other languages. As of this writing, in addition to the completed Turkish and English versions, there are ongoing translations by volunteers to French and Chinese (German, Korean, Portuguese, and Russian translations were started but not continued).

As for future directions, I would like to add more chapters; definitely one on allocators once they’re added to the standard library (they currently live in the std.experimental.allocator package).

One thing that bothers me about the book is that most code samples don’t take full advantage of D’s universal function call syntax (UFCS), mainly because that feature was added to the language only after most of the book was already written. I would like to move the UFCS chapter to an earlier point in the book so that more code samples can be in the idiomatic D style.

The book will always be freely available online, allowing me to make frequent updates and corrections. Fortunately, my Inglish leaves a lot to improve on, so there will always be grammar and typo corrections as well.