Author Archives: Michael Parker

Project Highlight: Visual D

In the world of modern software development, a language that is not supported in any of the major Integrated Development Environments is not going to gain very much traction. For better or worse, the IDE has become a widespread and permanent fixture. Programmers are free to use any editor or environment they want in their own time, but those who write software for a living and are not self-employed are going to have to work in the environment(s) allowed by their employer. In Windows shops, that often means Visual Studio.

For the first several years of D’s existence, not only was there no support for it in Visual Studio, the toolchain on Windows produced binaries only in the OMF format, making them incompatible with the Microsoft tools out of the box. Today, that has changed, and both DMD and LDC can output COFF files that are compatible with the MS linker. But even before DMD got support for COFF, one community member initiated Visual D, a project to bring D to the Visual Studio environment.

Rainer Schuetze first discovered D shortly after work on D2 began.

It must have been an article in the German c’t magazine (maybe this one) that brought my attention to the D language. The meta programming features of D2 caught my eye, namely templates and compile-time function evaluation. It was refreshing to see these elements being neatly integrated into the language, not bolted on top of it with a different language as in C++.

He decided to give D a try. The road that led him to Visual D originated not from a longing for an IDE, but from his attempts at debugging DMD’s output.

The options D had to offer on Windows were rather disappointing: the version of Microsoft’s Windbg distributed with DMD was barely usable as it was already 12 years old back then in 2008. Newer versions didn’t work at all with the executables generated by DMD. The debugger being one of the highlights of Visual Studio, I tried it, too, but anything newer than VS.NET 2003 failed miserably, with the latter being almost acceptable. It didn’t stop execution on breakpoints. So it seemed that there might be just a small tweak to make debugging work considerably better with Visual Studio.

It was this desire to debug D programs that led Rainer to start his first open source project, cv2pdb. Rainer’s tool can convert CodeView format output by DMD to the PDB format used by the Visual Studio debugger. Once that was working, it was no great leap to want better integration with the Visual Studio environment, such as build support and code completion.

Rainer found one abandoned project that had been started toward that end already, but he couln’t get it functional. He also experimented with modifying an existing language service written in C#, but had trouble getting it to work with a parser written in D. In the end, he decided that spending his time with D “should not mean writing code in other languages.” So, in 2009, Visual D was born. He ran into some difficulties almost immediately, as anyone using D in those days was bound to.

At the time, you would hit a compiler bug every few hundred lines of code, but these were not the biggest obstacles on the way to getting this extension to work. Visual Studio is a Win32 application and loads its extensions dynamically as DLLs. D very much relies on Thread Local Storage (TLS) as it is the default for global variables. Under Windows this uses “implicit TLS” built into the binary. Unfortunately, Windows Vista was the first version to support this for dynamically loaded DLLs, but Windows XP was still the most widely used version. It only supports TLS for the DLLs that are loaded implicitly with the application.

Eventually, after a lot of debugging, he managed to work around his Windows XP problems by tricking the system into believing a manually loaded DLL had been implicitly loaded with the application. The result of these efforts can be seen in the DRuntime modules core.sys.windows.dll and core.sys.windows.threadaux. This implementation comes with the drawback that DLLs using it cannot be unloaded. An improved version by Denis Shelomovskii works around this. Given the decline of XP usage, the need for this will eventually fade away.

TLS wasn’t his only problem.

Another issue turned up with the interfaces that create the API between Visual Studio and its extensions: these are all COM interfaces. Fortunately, D supports COM out of the box, virtual function calls work nicely. Unfortunately, the implementation in the library is currently unusable. The allocation of a COM object (every class derived from IUnknown) is done using C’s malloc, assuming the usual approach to free this memory in the Release() function when the reference count goes down to zero. Unfortunately, this is not possible from within a member function of a D class as the invariant is still called before the function returns. That’s why the runtime implementation just leaks the memory.

As a solution, he initially implemented his own ComObject class and overloaded new. Later, when the overloading of new was deprecated in the language, he switched to using a factory function for COM objects. He wasn’t finished yet.

The Visual Studio SDK provides COM interface definitions for C++ that I started to convert manually on a per case basis whenever I needed a declaration in D. This proved rather tedious as dependencies grew, so I considered converting the interface definitions automatically. These are given by Interface Definition Language (IDL) files or C++ header files.

So he created a conversion tool.

It uses a tokenizer followed by some custom conversion functions (mostly dealing with C pre-processor code) and a long list of replacement rules using the function now accessible within Visual D as Search/Replace Tokens. Each token comes with the comment preceding it, so the D files very much resemble the original files and still contain documentation, which is very helpful for the VS SDK. The conversion now works for all files of the VS SDK versions 2008 to 2015 and a selection of required files from the Windows SDKs 6.0 to 10 (about 90 header files converted, some more stubbed to be mostly empty).

Visual D has come a long way since Rainer first started working on it, but the journey is by no means complete. The next release will integrate the expression evaluator of the Mago debugger with the Visual Studio debug engine, thanks to Microsoft’s publication of information about the debug engine’s extensibility. Additionally, Visual D users will gain the ability to integrate with C and C++ projects.

Just drop your D files into a C/C++ project and they compile with the rest of your application without further ado. Enjoy settings integration instead of the ancient looking project dialogs of visualdproj files. More importantly, all the additional tools like the Manifest Tool, MIDL Compiler and even Build Extensions (e.g. Assembler) are easily available to D projects, too.

These new features are available in a preview version right now. Rainer has been short on time, so the features haven’t improved much since the preview release, but once he finds the time you can be sure to see these new features in the next release.

Core Team Update: Martin Nowak

In the early days of DMD, new releases were put out when Walter decided they were ready. There was no formal process, no commonly accepted set of criteria that defined the point at which a new compiler version was due or the steps involved in building the release packages. In the early days, that was just fine. As time passed, not so much.

According to Martin Nowak:

The old release process was completely opaque, inconsistent, irregular, and also very time-consuming for Walter. So at some point it more or less failed and Walter could no longer manage to build the next release.

Martin was eager to do something about it, but he wasn’t the first person to take action on the issue. He decided to start with the work that had come before.

At this point, I took a fairly complex script from Nick Sabalausky, which tried to emulate Walter’s process, and started to improve upon it. It got trimmed down to a much smaller size over time.

But that was just the beginning.

Next, I prepared OS images for Vagrant for all supported platforms. That alone took more than a week. After that, I wired up the release script with Vagrant. From there on we had kind of reproducible builds.

That was a major step forward and eliminated some of the common mistakes that had crept in now and again with the previous, no-so-reproducible, process. With the pieces in place, Martin got some help from another community member.

Andrew Edwards took over the actual release building, but I was still doing the main work of keeping the scripts running and managing bugfixes. At some point, Andrew no longer had time. As the back and forth between us was almost as much work as the release build process itself, I completely took over. Nowadays, we have the script running fully automated to build nightlies.

Thanks to Martin, the process for building DMD release packages is described step-by-step in the D Wiki so that anyone can do it if necessary. Moreover, he coauthored and implemented a D Improvement Proposal to clearly define a release schedule. This process was adopted beginning with DMD 2.068 and continues today.

With the improved release schedule, DMD users can plan ahead to anticipate new releases, or take advantage of nightly builds and point releases to test out bug fixes or, whenever they come around, new features in the language or the standard library. However, the schedule is not etched in stone, as any particular release may be delayed for one reason or another. For example, the 2.071 release introduced major changes to D’s import and symbol lookup to fix some long-standing annoyances, with the subsequent 2.071.1 fixing some regressions they introduced. The release of 2.072 has been delayed until all of the known issues related to the changes have been fixed.

Those who were around before Martin stepped up to take charge of the release process surely notice the difference. He and the others who have contributed along the way (like Nick and Andrew) have done the D community a major service. As a result, Martin has become an essential member of the core D team.

D in Games: Ethan Watson of Remedy Games Goes to GDC Europe

At DConf 2013 at the Facebook HQ, Manu Evans, then of Remedy Games, gave a talk titled, “Using D Alongside a Game Engine” (follow the link for the slides and video). He talked about Remedy’s experience getting D to work with their existing C++ game engine. The D landscape was a bit different when they got underway than it is today. For starters, DMD did not support 64-bit targets on Windows and the language had not yet gained support for directly binding to C++. Manu’s talk discusses the steps they took to work around such issues and achieve a working system.

Fast forward to 2016. Remedy releases Quantum Break, the first commercial AAA game to ship with bits implemented in D. Ethan Watson comes to DConf 2016 with a talk titled, “Quantum Break: AAA Gaming with Some D Code” (follow the link for the slides, go to YouTube for the video). In it, he expanded on what Manu had talked about before. Now, he’s getting ready to take that message to Game Developers Conference Europe.

On the 16th of August this year in Cologne, Germany, I will be presenting a talk titled, “D: Using an Emerging Language in Quantum Break“. For those who have tuned in to DConf previously, it will ostensibly be a combination of both my talk from 2016 and Manu Evans’s talk from 2013, but necessarily cut down due to the fact that it will need to fit two hours of content into one.

He has more in mind than just discussing the technical aspects of Remedy’s solution. During informal discussions at DConf, Ethan spoke of how he would like to bring D to the attention of his colleagues at other studios in the game industry. His upcoming talk presents just such an opportunity.

The approach I’ll be taking is something of a sales pitch to other game developers. My talk at DConf this year was a bit apologetic on my part as I felt we did not use D anywhere near as much as we should, and that we still had unsolved problems at the end of it. GDC Europe will be a platform for me to talk about the journey toward shipping a game with an emerging language in use; a way to show what benefits the language has over the industry stalwart C++ and other modern languages like C# and Rust; and as a bit of a way to drum up support from other developers and get them to at least look at the language and evaluate it for their own usage.

Additionally, a future event Ethan hinted at in his DConf talk will become reality in time for GDC Europe.

We will also be open sourcing our binding solution for GDC. I’ve been cleaning it up over the last couple of weeks, to the point where it’s effectively a complete reimplementation designed for ease of use, ease of reading, and to support some of the additional ideas I’ve had for the system since I took over from Manu. It’ll basically be our way of saying, “D is great! Here’s some code we prepared earlier to help you get up to speed.”

That’s awesome news for the D community as a whole. D has had built-in support for binding to C++ for a little while and it has improved over several releases, though it’s not as fully functional as the support for binding to C (a far simpler task) and likely never will be. There is also an experimental third-party solution called Calypso, which is a fork of the LDC compiler that ultimately aims to provide full and direct interfacing with C++. The availability of an open-source alternative that has been battle-tested in production code can only be a good thing.

Unfortunately, there is one downside to Ethan’s presentation.

They’ve scheduled my talk at the same time as John Romero’s talk. Never thought I’d be competing with him for attention.

Who would want to compete for an audience with one of the co-founders of id software and co-creator of franchises like Doom and Quake, who is giving a talk about the programming principles the id team developed in their early years? Well, if you ask Ethan, the downside is that he can’t skip out of his own talk for Romero’s. Ah, well. As they say, them’s the breaks.

The Why and Wherefore of the New D Improvement Proposal Process

When a programmer sees fellow programmers contributing code to open source projects, it is unlikely that the question of why will ever come to mind in terms of motive. Regarding technical merit or usefulness, sure. But motive? Coders who are inclined to contribute bug fixes, new features, build systems, or other resources to open source projects do so for a variety of reasons, most of which other programmers will intuitively understand. But when a programmer volunteers to take over what is solely an administrative process for an open source project, that’s a different story altogether.

Code contributions are usually one-off affairs. They require a time investment up front, but once the PR or patch is merged, that’s often the end of it. Managing an administrative process, on the other hand, is an ongoing commitment. Time that could be spent coding is instead spent doing the proverbial paperwork. And the benefit to the volunteer may not always be clear. Implementing a ray tracer and going bungee jumping are things you might do for the hell of it. Administrative tasks? Not so much.

The most recent post on this blog highlighted two non-coding initiatives which were kicked off by members of the D community. One of them is a new process for handling D Improvement Proposals, or DIPs. This was the work of Mihails Strasuns. Not only did he put together a new process, he also volunteered to manage it. What on earth was he thinking?

There isn’t anything special or unique about DIPs. Most programming languages that rely heavily on community input get something similar at some point. Once a language matures, changing even small bits of it becomes challenging and needs careful consideration. All of the relevant information has to be published somewhere.

The previous DIP process looked like a collection of articles on a community wiki. It seemed to work pretty well initially, when the DIP count was low enough, but with the number of improvement proposals reaching 100 and most of them still remaining in “draft” status, scalability issues became noticeable.

He highlights three issues specifically. First up is quality control.

There was no quality control for submitted proposals and, as a result, submissions were often lacking detailed information about language impact.

With the DIPs being Wiki-based, anyone with a D Wiki account could add a page for a new DIP, but there was no one designated to ensure that new DIPs met any sort of standard. This resulted in quality ranging from a detailed treatise that covered all the bases, to something that looked like notes on a cocktail napkin. Some remain in draft status, having never been finalized by their authors.

Another issue is the lack of response from the leadership.

It was very rare to get a formal response from Andrei or Walter regarding a DIP, partially because proper reviewing of incomplete proposals was demanding too much time.

Walter and Andrei already have their hands full. The last thing they need is to spend time reviewing a DIP that isn’t ready for review. A DIP should be ready for their consideration before it is brought to their attention and not require them to ask for information that should already be there. Without a designated arbiter to decide when a DIP is complete, how are they to know? The author of a DIP may believe it ready, but all too often will be wrong.

Finally, there is the issue of maintaining the list of DIPs.

Abandoned or rejected proposals often were not marked as such because editing the list was purely a community effort.

With no one person responsible for maintaining the list, it was updated only when someone in the community noticed an update was needed and was willing to make the appropriate edits. The result was that there were only a small number of people who ever touched it.

So the solution Mihails came up with to address all of these problems, which he says is similar to that used with other modern programming languages, is described in the README of the new github project he set up to replace the Wiki. It outlines the procedure, including how to submit a DIP and how to get it approved, gives advice on how to write a great DIP, and even outlines the responsibilities of the new DIP manager. Which, for now, is him.

And that brings us back to the topic of the first couple of paragraphs. We now know the problems he saw with the process and how he decided to improve it, but we still don’t know why. So, why, Mihails?

There is, of course, a personal reason which caused me to step up with this process proposal. In a way, this is an attempt to solve a conflict of interest that arises from being employed by one of the largest D commercial users while being actively engaged in the language on my own. I want for the language to change and improve, but, at the same time, I have to ensure any such changes don’t cause too much disruption to our systems at work.

That commercial user of which he speaks is Sociomantic, hosts of the excellent DConf 2016 in Berlin and maintainers of the ocean and makd projects they opened up for the D community.

Having a formalized process for language changes helps a lot in that regard. It means there is a single information feed to follow if one doesn’t want to miss anything of major impact. It’s much more concise than a newsgroup or mailing list. And volunteering to handle the DIP bureaucracy allows me to make timely suggestions for better ways to handle any changes so that they don’t cause too much trouble to existing code.

The DIP process arose organically to fill a void. Its was, perhaps, inevitable that its ad hoc existence take another step in its evolution. It was just waiting for the right person with the right ideas and the willingness to make it happen. That person turned out to be Mihails Strasuns. The D communnity will be better off thanks to his efforts.

The DLang Vision and Improvement Process

The evolution of D has been a process of steady, if not always speedy, improvement. In the early days, Walter was the only person working on the language, the reference compiler, and the standard library, though he did accept contributions. Today, numerous people contribute to or maintain a piece of every aspect of D’s development. Semi-formal processes exist where none did before. Things have come a long way.

As with anything in life, those who come later are unaware of the improvements that came before. If you’ve just stepped off the boat into D Land, you don’t remember the days before core D development moved to github, or the time before the first release of Visual D, the D plugin for Visual Studio developed and maintained by Rainer Schuetze. All you’re going to see is that your pet Pull Request hasn’t yet been merged, or that Visual D doesn’t allow you to do all of the things that the core of Visual Studio allows. You’ll rightly head over to the forums to complain about the PR merge process, or request a new feature for Visual D (or gripe about its absence).

And so the story goes, ad infinitum. Along the way, the community loses some members who once found promise in the language, but become frustrated because their pet peeves have not been addressed. And herein lies the point of this post.

The D Programming Language is a community driven language. As such, there is no formal process for creating opportunities to fix and improve the areas that need fixing and improving. Walter has his head down in compiler code and often barely has time to come up for air. Andrei is stretched thin in the additions he has planned for Phobos, the work he does on behalf of the D Foundation, and his numerous other commitments. It’s up to the community to get things moving on other fronts.

Most often, improvements come at the instigation of one highly motivated individual, who either gets busy with the rolling up of sleeves or keeps pushing until someone better positioned takes up the challenge. Still others want to contribute, but aren’t sure where to begin. One place is the documentation, as described the last time this blog touched on this theme. For other areas, the place to look is the D Vision Document.

Andrei started the Vision Document as a means to let the community know the direction in which he and Walter expect the language to go, with plans to release an updated document biannually. Each document lists progress that has been made in the preceding six months and the goals for the coming six. The first document focused on goals the D leadership will personally enable or make happen. The latest document, which was put up for feedback recently in the forums, includes additional goals the leadership strongly believe are important for the success of the D language. This new addition is the result of an initiative by community member Robert Schadek.

Robert took the simple step of creating a Wiki page to collate the wishlist items that Walter and Andrei mention in the forum now and again. These are things they would like to see completed, but have no time to make happen themselves. Andrei merged these items into the Vision Document so that there is only one document to maintain and one source for the information. This is a perfect example of a low-barrier community contribution that has led to an overall improvement (though Robert is no stranger to higher-barrier contributions) and shows how there are ways other than submitting Pull Requests to improve the D Programming Language experience.

A higher-level means of helping to improve the language is through D Improvement Proposals. A DIP is a formal proposal to make changes or additions to the language. The author of the DIP need not be the implementer. For some time, these have primarily been loosely maintained in an informal process at the D Wiki. Recently, Mihails Strasuns decided the process needed better management, so he created a github repository for DIPs as part of a new process to address some shortcomings of the old system. Then he made a forum announcement about it, including the news that he is, for now, volunteering his services as the new DIP manager.

So, to recap, if you are itching to contribute to D in some way that you know will have an impact, the latest Vision Document is the place to go for inspiration. As I write, that’s the 2016H2 document. If you have an idea for a new feature or improvement to the language, creating a DIP is the way to go (a future post here will say more about the DIP process). There are numerous other areas where contributions are welcome, like fixing or reporting bugs, reviewing PRs, and taking the initiative to contribute to the D process like Robert and Mihails have done.

If you don’t have the time, motivation, or inclination to contribute, that’s fine, too. Just keep in mind when you come to the forums to vent about one frustration or another (and please, feel free to do so) that much of what exists in the D ecosystem today, along with nearly every improvement that will come for the foreseeable future, is the result of someone in the community expending their own time and effort to make it so, usually with no financial incentive (the exception being bug bounties). Please let that knowledge color your perspective and your expectations. If you can’t, or won’t, work to resolve shortcomings in the language, the processes, or the ecosystem yourself, your goal in bringing them to the community should be to motivate someone else to do so. Not only will that increase the odds of fixing your issues, it will help all D users inch closer to the ultimate D Nirvana.

Project Highlight: Auburn Sounds

One of the questions some people ask when evaluating a language for the first time is, who’s making money with it? That’s no exception with D. While there are a number of companies using D in production, there are also people making money with the language out of their homes. The last Project Highlight looked at a freely available open source project. This one is about a proprietary set of audio plugins developed and sold by one programmer.

Guillaume Piolat has been using D since 2007. In that time, he has been an active member of the community, maintaining and contributing to several open source projects including Derelict and GFM. Now, he has begun to build an audio plugin business that he calls Auburn Sounds.

I have a long love/hate relationship with D and honestly I prefer the programs I’ve made with it over other languages. It’s the language I feel the most free with and it has always served me well. I’m just not comfortable with the low productivity level of C++, so I was willing to take a long term bet.

That bet is now paying off for him, but it hasn’t been completely free of challenges.

I did not know how hard wrapping plugin formats would be, especially on OS X. It turned out you could “derelictify” Carbon and Cocoa and call the Obj-C runtime without much difficulty, just with some work.

By derelictify, he means creating dynamic bindings in D to the C APIs of shared libraries that can then be loaded at run time via system APIs like dlopen. The DerelictUtil library can be used to create loaders for such bindings, hiding the platform-specific details behind a simple API, and the Derelict project provides a number of bindings that do just that. Hence the term derelictify. It’s worth noting here that D has some limited support for interfacing directly with Objective-C.

Other issues included bugs in the tool chain and less-than-ideal code generation.

I stumbled upon some DMD backend bugs and some bugs in DUB, but all of them were fixed in the end. Additionally, DMD codegen wasn’t competitive. Fortunately, LDC has made some incredible progress, bringing top codegen to both Windows and OS X . It’s something that I was expecting, but not so soon! This particular bet really paid off.

It’s commonly recommended in the D community to use DMD for its blazing fast compile times during development and, for the projects that really need it, LDC or GDC for production to take advantage of the fact that they typically produce binaries with better performance.

DUB is a build tool and package manager for D projects. A number of libraries have been registered in the DUB Registry, all of which are available to use as dependencies in any DUB-managed project. The tool will soon be shipping with DMD in an upcoming release of the compiler.

Now that he has several completed plugins available, how does Guillaume feel about having chosen D?

Auburn Sounds has existed for fifteen months, and in the past nine I’ve not thought of going back to C++ a single time. I don’t use D-specific features aggressively. At first, I was thinking that I would need D’s meta-programming support, like Design by Introspection, to create an efficient audio library, but it turned out having almost no abstraction worked well enough. The thing that matters most for this project is codegen quality, speed of development, platform support and low mental overhead.

Other benefits include the fluidity of DUB, the availability of VisualD, and the quality of some third-party libraries like imageformats, DerelictUtil, and especially ae.utils.graphics. Speed of compilation and development count a lot, but they aren’t something you really notice once you’ve grown accustomed to them.

Guillaume intends to continue to use D to develop more audio plugins and improve the ones he has already made available. His latest, Panagement, has both free and paid versions.

Panagement solves two problems in audio mixing: giving stereo content to a track quickly and fixing regular panning, which doesn’t sound that great on headphones. It’s a top dog in a sub-niche that traditionally doesn’t interest people a lot. It’s also the first plugin I’ve released entirely built with LDC.

In addition to Panagement, you can also currently find a voice octaver for sale and three other plugins freely available in their full versions: a binaural panner, a physical synthesizer, and a distortion plugin.

The D community undoubtedly collectively wishes Guillaume, long one of their own, the best of luck. If you are a one man shop or a small team using D to produce commercial software, let us know in the D Forums!

Project Highlight: The PowerNex Kernel

Hang around the D community long enough and you’ll discover that people are using the language in a variety of fields for a variety of projects, both professionally and personally. Computer games, scientific software, web apps, economic modelling, even scripted utilities. There have even been a few open source kernel projects using D, the most recent of which is PowerNex by Dan Printzell.

As hobby projects go, an OS kernel is one of the more complex projects a programmer could tackle. It requires a certain level of motivation and dedication that isn’t needed for other types of projects. Dan insists that it’s “fun, rewarding and hard, like big projects should be.”

I have always been interested in OS development and have been trying to write my own OS for years now. Each attempt was written in C, but none of them worked well because I mostly just copy-pasted code with no real knowledge of how it worked. Back in November 2015, I decided to start writing yet another kernel, but this time in D. I also challenged myself to make it 64-bit. The reason I chose D is simple. I love the language. I love that you can write nice looking code with the help of string mixins and templates and that the code can interface easily with C.

The D programming language ships with a runtime, appropriately named DRuntime, which manages the garbage collector, ensures static constructors and destructors are called, and more. Some language features depend on the runtime being present. When developing an OS kernel, making use of the full runtime is not an option. Dan took the minimal D runtime for bare metal D development that Adam Ruppe described in his book, the D Cookbook, and used that as the basis for his kernel.

It is still pretty much the same thing Adam provided, with some patches to fix deprecated stuff and to connect it to the rest of the kernel.

It hasn’t all been a walk through the roses, though. By default, variables in D are in Thread Local Storage (TLS). In order to force variables to become globally shared across threads, they must be marked as either shared or __gshared. The former is intended to tell the compiler to restrict certain operations on the variable (you can read about it in the freely available concurrency chapter from Andrei Alexandrescu’s book, The D Programming Language). The latter essentially causes the compiler to treat it as a global C variable, with no guarantees and no protection. Normally, TLS variables are a good thing for D programs, but not when starting out in the early stages of kernel development.

The biggest problem I’ve encountered is that the compiler expects that TLS is enabled, which I haven’t done yet, so I need to append __gshared to all the global variables. If I don’t write __gshared, the kernel will try and access random memory addresses and do undefined stuff. Sometimes it crashes, sometimes it doesn’t. This is the thing that is most often behind PowerNex bugs.

Did I mention that Dan loves D’s string mixins and templates?

String mixins and templates are the best thing in the language. Without these I would probably write the kernel in C instead. One place where they are used is in the Interrupt Service Routines (ISR) handler. The problem with the ISRs is that they don’t provide their ID to the handler. So I need to make 256 different functions just to know which ISR was triggered. This could be really error prone, but with some help from templates and string mixins, I can generate those and be sure that the content for each function is correct.

To compile PowerNex, Dan uses a cross-compiled GNU Binutils, a patched version of DMD, and his own build system, called Wild.

The GNU Binutils is for compiling the assembly files and for linking the final executable. The patch for DMD that I currently use basically just adds PowerNex as a target and as a predefined version (which is active when compiling). It is really hackily implemented because I’m not too familiar with the DMD source code. I want to implement these better and get it upstream in the future when I will be able to compile userspace programs.

The build system is not that much to look at currently. It is written in D and uses a JSON file as a frontend to define a set of file processors, rules and targets. With the help of these, Wild can compile PowerNex. I’m currently working on conversion from JSON to a custom format to be able to provide the features needed for the compilation of the kernel and all its userspace programs.

He has a few specific goals in mind before he’s ready to brand a PowerNex 1.0 release.

One of my first short term goals is to be able to run a simple ELF executable. Next, I want to port druntime and phobos; once I have that done I will be able to run almost any D program natively. Finally, I will port either DMD or SDC (the Stupid D Compiler), depending on what state SDC is in when I get there.

You can see a couple of screenshots of PowerNex in action via a post from one D community member in Dan’s forum announcement thread. If the idea of kernel development with D gives you goosebumps, go have some fun!

Core Team Update: Vladimir Panteleev

When Walter Bright unleashed the first public release of DMD on the world, the whole release and maintenance process surrounding the language and the compiler was largely a one man show. Today, there are a number of regular contributors at every step of the process, but at the heart, in addition to Walter, are three core team members who keep the engine running. One of them is Vladimir Panteleev.

Vladimir maintains a number of D projects that have proven useful to D users over the years. Today’s entry focuses on one that had as much of an impact as any other D project you could name at the time upon its release, if not more.

If the name DFeed isn’t familiar to you, it’s the software that powers the D Forums. Posts pop up now and again from new users (and even some old ones!) asking for features like post editing or deletion, or other capabilities that are common in forum software found around the web. What they don’t realize is that DFeed is actually not the traditional sort of forum package they may be familiar with.

Go back a few years and the primary vehicle powering D discussions was an NNTP server that most people accessed via newsreaders. Over time, there were additional interfaces added, as Vladimir points out:

D only had an NNTP news server, a mailing list connected to it, and two shoddy PHP web interfaces for the NNTP server, both somehow worse than the other. A lot of people were asking for a regular web forum, such as phpBB. However, Walter would always give the same answer – they were inferior in many aspects to NNTP, thus he wouldn’t use them. What’s the point of an official forum if the community leaders don’t visit them?

Vladimir decided to take the initiative and do something about it. He took a project he had already been working on and enhanced it.

The program was originally simply an IRC bot (and called “DIrcFeed”), which announced newsgroup posts, GitHub activity, Wikipedia edits and such to the #d channel on FreeNode. Adding a local message cache and a web interface thus turned it into a forum.

And so DFeed was born. Walter announced that it had gone live on Valentine’s Day 2012. It even managed to get a positive reddit thread behind it, which was a big deal for D in those days. The NNTP server and the mailing list interface to it are still alive and well, but it’s probably a safe assumption that a number of users eventually ditched their newsreaders for DFeed’s web interface and never looked back. I know I did.

So if you’ve ever wondered why you can’t delete your posts in the D Forums, you now have the reason. DFeed does not have the authoritative database. That belongs to the NNTP server. To illustrate two major problems this brings about, consider the idea of adding Markdown support to DFeed.

First, people using NNTP/email won’t see the rendered versions. Which isn’t a big deal by itself since it’s just text, but does create feature imparity. It *is* possible to write Markdown that looks fine when rendered but is unreadable in source form, especially with some common extensions such as GitHub Flavored Markdown.

Second, unless we’re careful with this, people using NNTP/mailing lists might trigger Markdown formatting that could make their post unreadable. This could be avoided, though, by only rendering messages with Markdown if they originate from the web interface, which allows previewing posts.

Even so, he’s hoping to add support for Markdown at some point in the future. Another enhancement he’s eyeing is optional delayed NNTP/email posting.

A lot of younger people communicate mainly through web forums, and are very used to being able to edit their post after sending it. This is not supported on forum.dlang.org for the simple reason that you can’t unsend an email. One solution to this problem is to save all messages sent from the web interface locally, but delay their propagation to NNTP/email for a few minutes. This creates a window during which the message can still be edited, or even deleted.

Other features coming in a future update are OAuth support, a thread overview widget, and performance improvements. He says DFeed isn’t as fast as it used to be. He wants to change that. Don’t look for the new version of DFeed too soon, though. Right now, Vladimir’s attention is turned in directions that he believes will have a bigger impact on D. As soon as I know what that means, you’ll read about it here.

In the meantime, if you have any ideas on how to improve the forums, keeping in mind that any new features have to play nice with both the NNTP server and the mailing list, don’t hesitate to bring it up.

Thanks to Vladimir for taking the time to provide me with an update and for maintaining DFeed over the years. Here’s to many more!

The D Website and You

For those who have been around the D community for a long time, it’s all too easy to look at the website we have today and think how much better it is than anything we’ve had in the past. It’s miles ahead. Unfortunately, that perspective doesn’t lend itself well to recognizing actual problems that newcomers might face when visiting the site for the first time. Their point of reference is quite often the current website of another language, making their perspective on what does and doesn’t work very different.

It goes without saying that D is primarily a community-driven language. It doesn’t have a large company with a dedicated team of paid workers pushing its development. The website is one of the areas where this has a major impact. Its quality is almost entirely dependent on community contributions. Quite often, it’s recent additions to the community, those who have fresh eyes, who step up and push for improvements to the site. Several have been implemented in the past few weeks, thanks to community members who saw a void and took the initiative to get it filled.

André Stein put together an interactive tour of the language and set it up online. Now, that is part of the official site as tour.dlang.org, available from the menu bar at the top of most dlang.org pages by clicking on Learn. This is a tremendous improvement over what existed before, when users had to browse through different link categories to find the resources they needed, none of which were such a quick introduction to the language.

Sebastian Wilzbach has initiated a number of additions and improvements to the website in the short time he has been active in the community. One such addition is a page listing a number of organizations currently using D. Before, this information was only available on the Wiki. Now, it’s a first-class citizen of dlang.org.

Sebastian also put forth a suggestion for a major change in how the website is deployed. Previously, changes to the site had to be manually deployed, a process which caused a delay between when the changes were made and when they became visible. This led to a situation where news on the front page could become horribly outdated. Sebastian’s suggestion was approved and now updates to the site are deployed automatically when they are merged.

Another user, Ozan Nurettin Süel, was looking for the D Foundation website and coming up empty. A post about it in the forum led to a new page being added to the site. Anyone who wants to learn about the Foundation and, once memberships are open, how to join now has somewhere to go for that information under the dlang.org umbrella.

As Andrei said in his DConf keynote this year, the first five minutes become the next five years. The website plays a major role in that first five minutes. It’s up to us as a community to ensure that it meets the needs of potential new users instead of getting in their way. Small changes can often have a big impact. Whether you are an old timer or a newcomer, if you see something that is missing from the site, or have an idea for how to improve it, please let us know. If you notice broken links, incorrect text, or other such issues, please report them. Only then can dlang.org evolve to fully meet the needs of those coming to it for the first time.

Recent D Foundation Activities

Welcome to the official D Blog!

My name is Michael Parker. With the launching of The D Blog, I’ve been fortunate enough to land the role of Blog Author, which means I’ll be making most of the posts around here. This blog will become your source for DLang news, inside looks at projects in the D community, and updates on happenings behind the scenes of language development and at the D Foundation. Consider it a complement to Adam Ruppe’s excellent weekly summary of D goings on, This Week in D. For this inaugural post, I’ve got three news items to pass along about some of the ways the D Foundation is working to promote D.

Lately, Andrei Alexandrescu has been doing a lot of talking in places other than conference stages. For one, he’s been having discussions with the Tech Lounge Foundation in Romania, an organization which, among other endeavors, works to help CS and engineering students and aspiring professionals move beyond their education. Specifically, the D Foundation is interviewing recent graduates and current graduate-level students to explore ideas for high-impact academic and industrial D projects. Not only could this result in a killer project or two for D, it could bring new faces into the community.

A second line of communication is active between Andrei and the University “Politehnica” of Bucharest, where he has had talks with the university’s Vice President Corneliu Burileanu and Director of the Telecommunications Department Eduard-Cristian Popovici. The focus of these discussions has been on finding opportunities for the university and the D Foundation to collaborate. Potential areas of cooperation in this realm include D-centric courses and joint projects developed with D.

Moving out of Romania and up into Scandanavia, Andrei was scheduled to host a D workshop on June 6 at NDC Oslo. That has now been canceled because of a scheduling conflict. Instead, he will be presenting a day-long consulting session at Cisco Norway. He will also be presenting a talk on D during lunch. All proceeds from the event will go straight to the D Foundation.

If you have any feedback on the blog theme, I would love to see it over in the forums. Be sure to keep an eye on this space, or subscribe to the feed, so you can stay up to date. And if you maintain a D project large or small, be on the lookout for an email from me. I’m getting ready to pick a target for the D Blog’s first project highlight.