Project Highlight: DlangUI

Posted on

Vadim Lopatin is an active D user who, like many in the D community, comes from a Java and C++ background.

My current job is writing a Java backend for a virtual call center . I’ve also worked as a C++ developer on IP PBX devices. Programming is my hobby as well. My biggest hobby project, which I’ve rewritten from scratch twice in the last 15 years, is CoolReader, a cross platform e-book reader written in C++.

He kept hearing news about D and, over time, became more interested in its “cool features”, like CTFE and code generation. So, three years ago, he decided to initiate a couple of projects to learn its features.

DDBC is a database connector similar to Java’s JDBC, with an API close to the original. HibernateD is an ORM library, similar to the Java-based Hibernate. Unlike Java, D allows the use of compile-time code introspection and code generation. It was interesting work, and I was impressed by the power of D.

Both projects proved to be no more than learning exercises, however, as he never used either himself and neither became popular in the community. Now they are largely abandoned, but he has since found another area where he could apply his talents and, as it turns out, where community interest has been much higher.

The new idea came about as he surveyed the state of available GUI libraries in D. While there were several options to choose from, he wasn’t satisfied by the fact that they were all either non-native wrappers or not cross-platform. He had already written a cross-platform GUI in C++ for CoolReader GL, a version of his ebook reader that uses the same GUI on all supported platforms. Why not implement another one in D?

He has a long list of items he thinks are important for a GUI library to check off. A few of them are:

  • Cross-platform — the same code should work on all platforms with simple recompilation.
  • Internationalization — it should be easy to write multilingual apps. Unicode everywhere. Strings externalized to resources.
  • Hardware acceleration — take advantage of DirectX or OpenGL where available, but it should be possible to use software rendering where they aren’t.
  • Resolution independence — flexible layouts must be used instead of fixed pixel-by-pixel positioning of controls.

A markup language for describing layouts, touch screen support, 3D rendering, customizable look-and-feel, easy event handling, and several other items complete the list. A big set of requirements for one person to work on alone, but he already had a good deal of experience with the GUI he wrote for CoolReader. So when he got going with his DlangUI project, his previous work is where he started.

Part of DlangUI is a direct port of the CoolReader GL GUI. It was easy to reuse big parts of C++ code thanks to the similarity of D and C++ syntax.

So he set about checking items off of his list. Such as support for hardware acceleration via an interface that easily supports different rendering backends, one of which is implemented using OpenGL. But as things got under way, he discovered that there is one particular issue with porting C++ to D that arises in the parts that can’t be directly reused.

The D GC does not bring any help for resource management, since object destructors may be called in any thread, in any order, or never at all. If an object owns some resources, it ought to be destroyed in a predictable way. Therefore, widgets and other objects holding resources must be destroyed manually by their owners

DlangUI uses reference counting for easy freeing of owned objects. Widgets remove their children on destroy. Windows remove their widgets when closing. I had to add debug mode instance counts for various objects, and corresponding messages in the log, to make sure all resources are freed gracefully.

Some resources (e.g. images) are cached. Their references may be taken from the cache, used, and then released often. To allow cleanup of caches, all such resources have usage flags. The cache provides a checkpoint method which removes the usage flag from all items, and a cleanup method which frees all cache items which have not been used since the last checkpoint.

He has worked on a number of items from his list, such as theme customization.

DlangUI themes are inspired by the Android API. It borrows Android’s state drawables (they may be even used as is), nine-patch PNGs, and resource versions for different screen sizes or resolutions. Usually widgets don’t use a hardcoded look and feel or layout properties. Instead, they use a style ID referencing to currently selected theme. If the theme is changed in runtime, all widgets receive a corresponding notification so that they can reload any cached values from the new theme. Simply providing a new theme changes the  look and feel significally.

Currently, two standard themes are provided in DlangUI: default (light) and dark. Applications may specify a standard theme as a parent, and override only the styles it needs. Standard theme resources are usually embedded into the application executable using the cool D feature import("filename"). Applications may embed their own resources as well. This allows creating a single file app withoug any additional resource files needing to be shipped with the executable.

Another check mark can  be place next to layouts. Here, he again looked to Android.

To support multiple screen resolutions and sizes, widgets must be placed and resized using layouts instead of direct pixel-based positioning. DlangUI uses Android API-like layouts for grouping, placing and resizing widgets, based on a two-phase measure/layout scheme.

And, while a GUI can be assembled entirely in code, he took inspiration from elsewhere for ideas to knock the markup item off his list.

Manually writing code to create a widget hierarchy and setting their properties is a bit boring. DlangUI offers the possibility to create widgets using a JSON-based description similar to Qt QML. I call it DML. Currently, only the creation of widgets and the setting of their properties are supported. In future, I hope to add the ability to describe signal handlers in DML, and automatically assign signals to handlers, and widget instances to variables. There is a GUI app, dlangui:dmledit, which helps to write DML. It combines a text editor for DML and a preview window to see the results.

When it comes to being cross-platform, a lot has been done so far, thanks to different backend implementations: Win32, SDL2, DSFML, X11, and Android. Not long ago, Vadim even announced a text-based interface which works in the Linux terminal or Windows console.

It was a real surprise for me how few changes were required to implement text-mode support. Besides the backend code and the text-mode drawing buffer implementation, most of the changes came in the form of a Console theme. Only a few fixes were required in the widgets, removing several hardcoded margins and sizes. Even DlangIDE, a DlangUI-based IDE for the D programming language, is now usable in terminals.

Here’s what DlangUI’s components normally look like on Windows.
screenshot-example1-windows

And this is what DlangIDE looks like running in the Windows console.

dlangide

When compared to screenshots of programs running with different DlangUI backends, seeing it in a terminal like that is pretty darn cool.

DlangUI manages event handling via signals and has built-in support for 3D graphics, including a 3D scene package. And work still continues on making Vadim’s list smaller, as well as addressing the problems with the library.

The most mentioned issue the non-native look and feel of widgets. Although it’s possible to make a theme looking exactly like native one, it would not track system theme changes anyway. There’s no system menu support on OS X and in Gnome (where a common menu is used for all apps). The documentation is poor. There is some DDOX-generated documentation, but it’s not detailed enough and I seldom update. I need more tutorials and examples. And some advanced controls are missing, e.g. an HTML view.

He also says that there are too few developers working on the project. While some users have submitted PRs, the majority of the work has been done by Vadim alone. Given what he has produced so far, that’s a pretty impressive achievement. But, in addition to solving the problems above, he’s got a lot more he wants to implement, such as:

  • An XML+CSS rendering widget to show/edit HTML or rich text
  • Refactoring DlangUI to extract window creation, OpenGL context creation, drawing, font support, input events code from widget set – for cases when no widgets are needed
  •  Mobile platforms support improvements – add iOS backend, improve android support, improve touch mode support
  •  Native system menu support on OS X and Gnome
  •  Support for fallback fonts in font engines, from which to get missing symbols
  •  A native OSX backend based on Cocoa instead of libSDL2
  • Improvements in Scene3D to make it suitable for writing 3D games

If you need a GUI for your D app, DlangUI is a viable option today. More importantly, if you’re able and willing to help out here and there, Vadim sure could use a few more hands on a few more keyboards!

 

 

 

Project Highlight: Timur Gafarov

Posted on

dlib-logoTo begin with, let’s be clear that Timur Gafarov is a person and not a project. The impetus for this post was an open source first-person shooter, called Atrium, that he develops and maintains. In the course of making the game, he has created a few other D projects, each of which could be the focus of its own post. So this time around, we’re going to do a plural Project Highlight and introduce you to the GitHub repository of Timur Gafarov.

When Timur first came to D six years ago, you might say it was love at first sight.

As an indie game developer with a strong bias toward graphics engines and rendering tech, I always try to keep track of modern compiled languages effective enough for writing real-time stuff. The most obvious choice in this field is C++, and I actually used it for several years until I found D in 2010. I immediately fell in love with the language’s clean, beautiful syntax, its powerful template system, the numerous built-in features absent in C++, and the rich and easy to use standard library.

Interestingly, he was actually attracted by one of the things often cited as a turn-off about D back then: the lack of libraries. It was a situation in which he saw opportunities to create things from scratch, without worrying about reinventing the wheel. At the time, DUB was not yet a thing, so the first task he set himself was coding up a build system called Cook, which he still uses sometimes instead of DUB.

After that, he was ready to start making a game. He wanted to use OpenGL, and found an existing binding in Derelict (a collection of dynamic bindings maintained by this post’s author — there also exists a collection of static bindings called Deimos) that allowed him to do so in D. With that off his plate, he next sat down to write a game engine. The first few steps in that direction resulted in a set of utility packages that coalesced into dlib.

At first, there were no clear plans or goals. After a previous period of using third-party engines, I had some experience with low-level graphics coding in C++ and just wanted to port my stuff to D for a start. I began with vector/matrix algebra and image I/O. These efforts resulted in dlib, a general purpose library.

He next turned his attention to 3D physics. Even though there were existing libraries with bindable C interfaces, like ODE (with a dynamic binding that existed then in the form of DerelictODE and a static binding in Deimos) and Newton (for which Deimos-like and Derelict-like bindings have since been created by third parties), he just couldn’t help himself. Enter dmech.

Atrium was born as part of my experiment with writing a 3D physics engine, called dmech. OK, this wasn’t strictly necessary, but the chance of being the first one to write this kind of thing in D was so attractive that I couldn’t stand it 🙂 Of course, I didn’t dare to compete with such industry standards as Bullet, but nevertheless it was an amazing experience. I’ve learned a lot.

A screenshot from Atrium.
A screenshot from Atrium.

A physics engine and utility library weren’t all he needed. That’s where DGL comes in.

The next milestone was writing a graphics engine that I call DGL. I can’t consider this step fully completed, because I’m never happy with my abstractions and design solutions. Finally, I ended up with some kind of simplified Physically-Based Rendering pipeline, Percentage Closer Filtering shadows and multipass rendering, with which I’m sort of satisfied for now.

There was a period when I had to use outdated hardware, so DGL uses OpenGL 1.x, relying on extensions to utilize modern GPU technologies. Yeah, in 2016 this sounds funny 🙂 But recently I started experimenting with OpenGL 4, so the engine is likely to be rewritten. Again!

But for Timur, the graphics system isn’t the most important part of Atrium.

It’s the collision detection and character kinematics. I think that believable character behavior and interactions with the virtual world are crucial for any realistic game. In Atrium, these are fully physics-based, with as few hacks and workarounds as possible. Rigid body dynamics natively ‘talk’ with player-controllable kinematics. That means, for example, that a character can be pushed with moving objects and can push other objects himself. A stack of dynamic boxes can be transported via a kinematic moving platform. A gravity gun can be used to move things. And so on. I’m deeply inspired by Valve’s masterpieces, Half-Life 2 and Portal, so I want to make my own ‘physical’ first person puzzle.

Working on this project has certainly been a labor of love.

It has already taken me about six years of spare-time work and it still exists only in the form of an early gameplay demo. Of course, if I’d used an existing mature engine, like Unity or UE, things would be much simpler. Constraining myself to use D for such a complex task may look strange, but it’s fun. And that’s that.

Through those years, he has learned a good deal about the ups and downs of game development with D. Particularly that ever popular bugbear known as the GC.

Modern D is a very attractive choice as a language for game development. Even the garbage collector is not a problem, because you can use object pools, custom allocators, or simply malloc and free. The key point is to know when the GC is invoked and try to avoid those cases in performance critical code. Personally, I prefer using malloc so that I can free the memory when I want, since delete has been deprecated and destroy  just releases all the references to an object instance without actually deleting it. Using manual memory management imposes some restrictions on the code–for example, you can’t use closures or D’s built-in containers–but that, again, is not a big problem. A large effort is currently underway to lessen GC usage in dlib, so that you can use it to write fully unmanaged applications with ease. It has GC-free containers, file I/O streams, image decoders, and so on.

If you are interested in game development with D, Timur’s GitHub repository should be an early point of call. Even if you aren’t making a game or a game engine, you may well find something useful in dlib. With its growing list of contributors, it’s getting a good deal of care and attention.

Thanks to Timur for taking the time to contribute to this post. We wish him luck with all of his projects!

Project Highlight: Visual D

Posted on

In the world of modern software development, a language that is not supported in any of the major Integrated Development Environments is not going to gain very much traction. For better or worse, the IDE has become a widespread and permanent fixture. Programmers are free to use any editor or environment they want in their own time, but those who write software for a living and are not self-employed are going to have to work in the environment(s) allowed by their employer. In Windows shops, that often means Visual Studio.

For the first several years of D’s existence, not only was there no support for it in Visual Studio, the toolchain on Windows produced binaries only in the OMF format, making them incompatible with the Microsoft tools out of the box. Today, that has changed, and both DMD and LDC can output COFF files that are compatible with the MS linker. But even before DMD got support for COFF, one community member initiated Visual D, a project to bring D to the Visual Studio environment.

Rainer Schuetze first discovered D shortly after work on D2 began.

It must have been an article in the German c’t magazine (maybe this one) that brought my attention to the D language. The meta programming features of D2 caught my eye, namely templates and compile-time function evaluation. It was refreshing to see these elements being neatly integrated into the language, not bolted on top of it with a different language as in C++.

He decided to give D a try. The road that led him to Visual D originated not from a longing for an IDE, but from his attempts at debugging DMD’s output.

The options D had to offer on Windows were rather disappointing: the version of Microsoft’s Windbg distributed with DMD was barely usable as it was already 12 years old back then in 2008. Newer versions didn’t work at all with the executables generated by DMD. The debugger being one of the highlights of Visual Studio, I tried it, too, but anything newer than VS.NET 2003 failed miserably, with the latter being almost acceptable. It didn’t stop execution on breakpoints. So it seemed that there might be just a small tweak to make debugging work considerably better with Visual Studio.

It was this desire to debug D programs that led Rainer to start his first open source project, cv2pdb. Rainer’s tool can convert CodeView format output by DMD to the PDB format used by the Visual Studio debugger. Once that was working, it was no great leap to want better integration with the Visual Studio environment, such as build support and code completion.

Rainer found one abandoned project that had been started toward that end already, but he couln’t get it functional. He also experimented with modifying an existing language service written in C#, but had trouble getting it to work with a parser written in D. In the end, he decided that spending his time with D “should not mean writing code in other languages.” So, in 2009, Visual D was born. He ran into some difficulties almost immediately, as anyone using D in those days was bound to.

At the time, you would hit a compiler bug every few hundred lines of code, but these were not the biggest obstacles on the way to getting this extension to work. Visual Studio is a Win32 application and loads its extensions dynamically as DLLs. D very much relies on Thread Local Storage (TLS) as it is the default for global variables. Under Windows this uses “implicit TLS” built into the binary. Unfortunately, Windows Vista was the first version to support this for dynamically loaded DLLs, but Windows XP was still the most widely used version. It only supports TLS for the DLLs that are loaded implicitly with the application.

Eventually, after a lot of debugging, he managed to work around his Windows XP problems by tricking the system into believing a manually loaded DLL had been implicitly loaded with the application. The result of these efforts can be seen in the DRuntime modules core.sys.windows.dll and core.sys.windows.threadaux. This implementation comes with the drawback that DLLs using it cannot be unloaded. An improved version by Denis Shelomovskii works around this. Given the decline of XP usage, the need for this will eventually fade away.

TLS wasn’t his only problem.

Another issue turned up with the interfaces that create the API between Visual Studio and its extensions: these are all COM interfaces. Fortunately, D supports COM out of the box, virtual function calls work nicely. Unfortunately, the implementation in the library is currently unusable. The allocation of a COM object (every class derived from IUnknown) is done using C’s malloc, assuming the usual approach to free this memory in the Release() function when the reference count goes down to zero. Unfortunately, this is not possible from within a member function of a D class as the invariant is still called before the function returns. That’s why the runtime implementation just leaks the memory.

As a solution, he initially implemented his own ComObject class and overloaded new. Later, when the overloading of new was deprecated in the language, he switched to using a factory function for COM objects. He wasn’t finished yet.

The Visual Studio SDK provides COM interface definitions for C++ that I started to convert manually on a per case basis whenever I needed a declaration in D. This proved rather tedious as dependencies grew, so I considered converting the interface definitions automatically. These are given by Interface Definition Language (IDL) files or C++ header files.

So he created a conversion tool.

It uses a tokenizer followed by some custom conversion functions (mostly dealing with C pre-processor code) and a long list of replacement rules using the function now accessible within Visual D as Search/Replace Tokens. Each token comes with the comment preceding it, so the D files very much resemble the original files and still contain documentation, which is very helpful for the VS SDK. The conversion now works for all files of the VS SDK versions 2008 to 2015 and a selection of required files from the Windows SDKs 6.0 to 10 (about 90 header files converted, some more stubbed to be mostly empty).

Visual D has come a long way since Rainer first started working on it, but the journey is by no means complete. The next release will integrate the expression evaluator of the Mago debugger with the Visual Studio debug engine, thanks to Microsoft’s publication of information about the debug engine’s extensibility. Additionally, Visual D users will gain the ability to integrate with C and C++ projects.

Just drop your D files into a C/C++ project and they compile with the rest of your application without further ado. Enjoy settings integration instead of the ancient looking project dialogs of visualdproj files. More importantly, all the additional tools like the Manifest Tool, MIDL Compiler and even Build Extensions (e.g. Assembler) are easily available to D projects, too.

These new features are available in a preview version right now. Rainer has been short on time, so the features haven’t improved much since the preview release, but once he finds the time you can be sure to see these new features in the next release.

Project Highlight: Auburn Sounds

Posted on

One of the questions some people ask when evaluating a language for the first time is, who’s making money with it? That’s no exception with D. While there are a number of companies using D in production, there are also people making money with the language out of their homes. The last Project Highlight looked at a freely available open source project. This one is about a proprietary set of audio plugins developed and sold by one programmer.

Guillaume Piolat has been using D since 2007. In that time, he has been an active member of the community, maintaining and contributing to several open source projects including Derelict and GFM. Now, he has begun to build an audio plugin business that he calls Auburn Sounds.

I have a long love/hate relationship with D and honestly I prefer the programs I’ve made with it over other languages. It’s the language I feel the most free with and it has always served me well. I’m just not comfortable with the low productivity level of C++, so I was willing to take a long term bet.

That bet is now paying off for him, but it hasn’t been completely free of challenges.

I did not know how hard wrapping plugin formats would be, especially on OS X. It turned out you could “derelictify” Carbon and Cocoa and call the Obj-C runtime without much difficulty, just with some work.

By derelictify, he means creating dynamic bindings in D to the C APIs of shared libraries that can then be loaded at run time via system APIs like dlopen. The DerelictUtil library can be used to create loaders for such bindings, hiding the platform-specific details behind a simple API, and the Derelict project provides a number of bindings that do just that. Hence the term derelictify. It’s worth noting here that D has some limited support for interfacing directly with Objective-C.

Other issues included bugs in the tool chain and less-than-ideal code generation.

I stumbled upon some DMD backend bugs and some bugs in DUB, but all of them were fixed in the end. Additionally, DMD codegen wasn’t competitive. Fortunately, LDC has made some incredible progress, bringing top codegen to both Windows and OS X . It’s something that I was expecting, but not so soon! This particular bet really paid off.

It’s commonly recommended in the D community to use DMD for its blazing fast compile times during development and, for the projects that really need it, LDC or GDC for production to take advantage of the fact that they typically produce binaries with better performance.

DUB is a build tool and package manager for D projects. A number of libraries have been registered in the DUB Registry, all of which are available to use as dependencies in any DUB-managed project. The tool will soon be shipping with DMD in an upcoming release of the compiler.

Now that he has several completed plugins available, how does Guillaume feel about having chosen D?

Auburn Sounds has existed for fifteen months, and in the past nine I’ve not thought of going back to C++ a single time. I don’t use D-specific features aggressively. At first, I was thinking that I would need D’s meta-programming support, like Design by Introspection, to create an efficient audio library, but it turned out having almost no abstraction worked well enough. The thing that matters most for this project is codegen quality, speed of development, platform support and low mental overhead.

Other benefits include the fluidity of DUB, the availability of VisualD, and the quality of some third-party libraries like imageformats, DerelictUtil, and especially ae.utils.graphics. Speed of compilation and development count a lot, but they aren’t something you really notice once you’ve grown accustomed to them.

Guillaume intends to continue to use D to develop more audio plugins and improve the ones he has already made available. His latest, Panagement, has both free and paid versions.

Panagement solves two problems in audio mixing: giving stereo content to a track quickly and fixing regular panning, which doesn’t sound that great on headphones. It’s a top dog in a sub-niche that traditionally doesn’t interest people a lot. It’s also the first plugin I’ve released entirely built with LDC.

In addition to Panagement, you can also currently find a voice octaver for sale and three other plugins freely available in their full versions: a binaural panner, a physical synthesizer, and a distortion plugin.

The D community undoubtedly collectively wishes Guillaume, long one of their own, the best of luck. If you are a one man shop or a small team using D to produce commercial software, let us know in the D Forums!

Project Highlight: The PowerNex Kernel

Posted on

Hang around the D community long enough and you’ll discover that people are using the language in a variety of fields for a variety of projects, both professionally and personally. Computer games, scientific software, web apps, economic modelling, even scripted utilities. There have even been a few open source kernel projects using D, the most recent of which is PowerNex by Dan Printzell.

As hobby projects go, an OS kernel is one of the more complex projects a programmer could tackle. It requires a certain level of motivation and dedication that isn’t needed for other types of projects. Dan insists that it’s “fun, rewarding and hard, like big projects should be.”

I have always been interested in OS development and have been trying to write my own OS for years now. Each attempt was written in C, but none of them worked well because I mostly just copy-pasted code with no real knowledge of how it worked. Back in November 2015, I decided to start writing yet another kernel, but this time in D. I also challenged myself to make it 64-bit. The reason I chose D is simple. I love the language. I love that you can write nice looking code with the help of string mixins and templates and that the code can interface easily with C.

The D programming language ships with a runtime, appropriately named DRuntime, which manages the garbage collector, ensures static constructors and destructors are called, and more. Some language features depend on the runtime being present. When developing an OS kernel, making use of the full runtime is not an option. Dan took the minimal D runtime for bare metal D development that Adam Ruppe described in his book, the D Cookbook, and used that as the basis for his kernel.

It is still pretty much the same thing Adam provided, with some patches to fix deprecated stuff and to connect it to the rest of the kernel.

It hasn’t all been a walk through the roses, though. By default, variables in D are in Thread Local Storage (TLS). In order to force variables to become globally shared across threads, they must be marked as either shared or __gshared. The former is intended to tell the compiler to restrict certain operations on the variable (you can read about it in the freely available concurrency chapter from Andrei Alexandrescu’s book, The D Programming Language). The latter essentially causes the compiler to treat it as a global C variable, with no guarantees and no protection. Normally, TLS variables are a good thing for D programs, but not when starting out in the early stages of kernel development.

The biggest problem I’ve encountered is that the compiler expects that TLS is enabled, which I haven’t done yet, so I need to append __gshared to all the global variables. If I don’t write __gshared, the kernel will try and access random memory addresses and do undefined stuff. Sometimes it crashes, sometimes it doesn’t. This is the thing that is most often behind PowerNex bugs.

Did I mention that Dan loves D’s string mixins and templates?

String mixins and templates are the best thing in the language. Without these I would probably write the kernel in C instead. One place where they are used is in the Interrupt Service Routines (ISR) handler. The problem with the ISRs is that they don’t provide their ID to the handler. So I need to make 256 different functions just to know which ISR was triggered. This could be really error prone, but with some help from templates and string mixins, I can generate those and be sure that the content for each function is correct.

To compile PowerNex, Dan uses a cross-compiled GNU Binutils, a patched version of DMD, and his own build system, called Wild.

The GNU Binutils is for compiling the assembly files and for linking the final executable. The patch for DMD that I currently use basically just adds PowerNex as a target and as a predefined version (which is active when compiling). It is really hackily implemented because I’m not too familiar with the DMD source code. I want to implement these better and get it upstream in the future when I will be able to compile userspace programs.

The build system is not that much to look at currently. It is written in D and uses a JSON file as a frontend to define a set of file processors, rules and targets. With the help of these, Wild can compile PowerNex. I’m currently working on conversion from JSON to a custom format to be able to provide the features needed for the compilation of the kernel and all its userspace programs.

He has a few specific goals in mind before he’s ready to brand a PowerNex 1.0 release.

One of my first short term goals is to be able to run a simple ELF executable. Next, I want to port druntime and phobos; once I have that done I will be able to run almost any D program natively. Finally, I will port either DMD or SDC (the Stupid D Compiler), depending on what state SDC is in when I get there.

You can see a couple of screenshots of PowerNex in action via a post from one D community member in Dan’s forum announcement thread. If the idea of kernel development with D gives you goosebumps, go have some fun!