If you spend any time over at r/VoxelGameDev, you may have seen posts about Voxelman, the plugin-driven game engine MrSmith33 is developing with D. His real name is Andrey Penechko, and he started work on Voxelman after he was inspired by Minecraft to think about all the cool things he could do with a voxel engine, particularly the low-level optimization tricks he could use in implementing one. Then he jumped in and started figuring things out.

I started the project somewhere in 2011 or 2012. It began with creating an SDL window and getting some triangles on the screen. Then I did cubes, then a single chunk. It was a simple, single-threaded thing. I did it all with a fixed camera and only had rudimentary camera controls.

For that initial version of the project, he was using C++, but he found himself stuck from a lack of knowledge about the language. So he started searching to see what else was out there. That led him to D.

I don’t really remember how I found D. I was in need of some statically typed compiled language other than C++. I was frustrated about all the source file organisation, the need of forward declarations, header separation and the include system. In D, it was as simple as writing code. I bought a cheap 10 inch tablet just to read Andrei’s book, because my 3.2″ PPC was too small to read the whole thing. I enjoyed reading every single bit of it.

His ultimate goal with the project is to provide a platform for which people can create and share plugins and game worlds.

Ideally a complete project build should have the engine source and tools (launcher, source editor, compiler). Players should be able to initiate a connection to any server in the server list, then the launcher will download any missing plugins, compile a new executable and start the engine with the list of plugins. Currently, a build of Voxelman is less than 3MB in size. I think that this is a good property to have.

The major sticking point he sees with this approach is the dependency DMD has on the Microsoft tools for 64-bit (and 32-bit COFF) support on Windows (specifically the Windows SDK and the Microsoft linker). Even though the MS linker is considered the system linker, it’s not uncommon to see Cygwin and or one of the various distributions of MinGW installed instead of the MS tools. In a perfect world, he could tell people to download the D compiler and they would have everything they need. But it’s not a deal-breaker, so he’s not letting it stop him.

Voxelman uses a client-server architecture, where the server can be launched in a dedicated process or as part of the client’s. This is managed by a launcher which, in addition to launching the game, can be used to compile projects, manage the world, and find servers to connect with.

World and mesh generation is multi-threaded and, as in most such engines, the model is chunk-based. The chunk management implementation is informed by the concept of entity component systems, with a chunk’s world position serving as its entity ID and layers functioning as components.

Each dimension is broken into chunks. A chunk is a 32³ array of blocks. Each chunk can have a set of data layers (currently blocks and block entities). Each layer is essentially an immutable snapshot. It can be of different storage types (uniform, where all blocks are the same,  or a compressed or full array, where the layer stores an array of data). Those layers then can be freely transmitted between threads, with reference counting done in the main thread. When a layer is no longer needed it’s deleted.

Immutable chunk data makes for fast auto saves of chunk snapshots in a separate IO thread.

When a chunk is received on the client side, it can be sent to a worker thread and the geometry will be generated. Snapshots are sent to the IO thread when save points occur, and they can still be used in the main thread, sent to the client, or processed by other worker threads. One can easily use an old snapshot while several new ones are in use. Whenever a layer is being modified, data is copied into a write buffer, changes are made, and at a commit point at the end of the frame, all write buffers are committed to chunk storage.

Andrey calls his plugin system “semi-hackish”.

All plugins inherit from an IPlugin interface. Then, each plugin registers itself in a global table of plugins from a shared static constructor. The global table has lists for server and client plugins. The engine adds those plugins to the plugin manager based on a provided plugin pack. The plugin manager implements the initialization sequence. When starting initialization, you have lots of dependencies, so you need to run things in a specific order.

He has found a lot of things to like about D. As major pros, he cites the module system (“no forward declarations”), foreach loops (“99% of loops in my code are these guys”), associative arrays, delegates, and templates (“They’re beautiful; you simply add another set of parentheses and you’re done”). He also loves D’s dynamic arrays (slices).

They are a perfect design, with the pointer and the length bundled together. You can append to them, concatenate them, and change their length.

As minor pros, he lists D’s Compile-Time Function Execution and its code generation and compile-time introspection features. Unlike some D users, he also counts the garbage collector in that group. He has implemented a mix of GC-ed and non-GCed memory in Voxelman.

High-level stuff is fully in GC memory. I call something high-level if it has only one instance, so I use interfaces/classes for the high-level parts. Low-level things are mostly stack allocated, using structs (which are POD in D), and the most performance sensitive and memory consuming parts use manual memory management (via Mallocator). This includes chunk storage and chunk meshes.

He also has a list of rough corners. He doesn’t like that support for DLLs is not yet fully functional and reliable. He has found problems when trying to use shared (for example, the Mutex class cannot be used with it). He also finds all the use cases of the is expression confusing, saying the syntax “feels like regular expressions for templates; very powerful and concise, but hard to understand.”

His difficulties with shared actually took him down an interesting path that ultimately had a positive impact on performance.

I started my multi-threading by using the send and receive functions from std.concurrency. I found that I needed to send messages of variable length. For example, when loading or saving chunks, you need to send all the layers to another thread. This involved allocating arrays for all the layers and also required the use of shared.

This situation led me to the implementation of a lock-free message queue, where each message is just a stream of bytes. You write variables on one end and read them from the other. This is obviously a single producer, single consumer queue.

A disadvantage was the use of a fixed-size circular array. You need to make sure that the queue doesn’t fill up. This was a point where I found a good book that explains how atomics work: C++ Concurency in Action: Practical Multithreading. This is one of the places in D’s documentation where you feel a lack of pointers on where to find relevant information on a specific topic.

So the new solution doesn’t require any allocations and is actually faster than the built-in one. Later I added a notification system via Semaphore, so that worker threads wait when out of work.

If you’re looking for an open source D game to contribute to, Voxelman is waiting for you. You can read more about some of its internals on reddit, check out some images on imgur, and watch some videos on YouTube. I’ll leave you with this example of it in action:

To begin with, let’s be clear that Timur Gafarov is a person and not a project. The impetus for this post was an open source first-person shooter, called Atrium, that he develops and maintains. In the course of making the game, he has created a few other D projects, each of which could be the focus of its own post. So this time around, we’re going to do a plural Project Highlight and introduce you to the GitHub repository of Timur Gafarov.

When Timur first came to D six years ago, you might say it was love at first sight.

As an indie game developer with a strong bias toward graphics engines and rendering tech, I always try to keep track of modern compiled languages effective enough for writing real-time stuff. The most obvious choice in this field is C++, and I actually used it for several years until I found D in 2010. I immediately fell in love with the language’s clean, beautiful syntax, its powerful template system, the numerous built-in features absent in C++, and the rich and easy to use standard library.

Interestingly, he was actually attracted by one of the things often cited as a turn-off about D back then: the lack of libraries. It was a situation in which he saw opportunities to create things from scratch, without worrying about reinventing the wheel. At the time, DUB was not yet a thing, so the first task he set himself was coding up a build system called Cook, which he still uses sometimes instead of DUB.

After that, he was ready to start making a game. He wanted to use OpenGL, and found an existing binding in Derelict (a collection of dynamic bindings maintained by this post’s author — there also exists a collection of static bindings called Deimos) that allowed him to do so in D. With that off his plate, he next sat down to write a game engine. The first few steps in that direction resulted in a set of utility packages that coalesced into dlib.

At first, there were no clear plans or goals. After a previous period of using third-party engines, I had some experience with low-level graphics coding in C++ and just wanted to port my stuff to D for a start. I began with vector/matrix algebra and image I/O. These efforts resulted in dlib, a general purpose library.

He next turned his attention to 3D physics. Even though there were existing libraries with bindable C interfaces, like ODE (with a dynamic binding that existed then in the form of DerelictODE and a static binding in Deimos) and Newton (for which Deimos-like and Derelict-like bindings have since been created by third parties), he just couldn’t help himself. Enter dmech.

Atrium was born as part of my experiment with writing a 3D physics engine, called dmech. OK, this wasn’t strictly necessary, but the chance of being the first one to write this kind of thing in D was so attractive that I couldn’t stand it 🙂 Of course, I didn’t dare to compete with such industry standards as Bullet, but nevertheless it was an amazing experience. I’ve learned a lot.

A screenshot from Atrium.

A physics engine and utility library weren’t all he needed. That’s where DGL comes in.

The next milestone was writing a graphics engine that I call DGL. I can’t consider this step fully completed, because I’m never happy with my abstractions and design solutions. Finally, I ended up with some kind of simplified Physically-Based Rendering pipeline, Percentage Closer Filtering shadows and multipass rendering, with which I’m sort of satisfied for now.

There was a period when I had to use outdated hardware, so DGL uses OpenGL 1.x, relying on extensions to utilize modern GPU technologies. Yeah, in 2016 this sounds funny 🙂 But recently I started experimenting with OpenGL 4, so the engine is likely to be rewritten. Again!

But for Timur, the graphics system isn’t the most important part of Atrium.

It’s the collision detection and character kinematics. I think that believable character behavior and interactions with the virtual world are crucial for any realistic game. In Atrium, these are fully physics-based, with as few hacks and workarounds as possible. Rigid body dynamics natively ‘talk’ with player-controllable kinematics. That means, for example, that a character can be pushed with moving objects and can push other objects himself. A stack of dynamic boxes can be transported via a kinematic moving platform. A gravity gun can be used to move things. And so on. I’m deeply inspired by Valve’s masterpieces, Half-Life 2 and Portal, so I want to make my own ‘physical’ first person puzzle.

Working on this project has certainly been a labor of love.

It has already taken me about six years of spare-time work and it still exists only in the form of an early gameplay demo. Of course, if I’d used an existing mature engine, like Unity or UE, things would be much simpler. Constraining myself to use D for such a complex task may look strange, but it’s fun. And that’s that.

Through those years, he has learned a good deal about the ups and downs of game development with D. Particularly that ever popular bugbear known as the GC.

Modern D is a very attractive choice as a language for game development. Even the garbage collector is not a problem, because you can use object pools, custom allocators, or simply malloc and free. The key point is to know when the GC is invoked and try to avoid those cases in performance critical code. Personally, I prefer using malloc so that I can free the memory when I want, since delete has been deprecated and destroy  just releases all the references to an object instance without actually deleting it. Using manual memory management imposes some restrictions on the code–for example, you can’t use closures or D’s built-in containers–but that, again, is not a big problem. A large effort is currently underway to lessen GC usage in dlib, so that you can use it to write fully unmanaged applications with ease. It has GC-free containers, file I/O streams, image decoders, and so on.

If you are interested in game development with D, Timur’s GitHub repository should be an early point of call. Even if you aren’t making a game or a game engine, you may well find something useful in dlib. With its growing list of contributors, it’s getting a good deal of care and attention.

Thanks to Timur for taking the time to contribute to this post. We wish him luck with all of his projects!