Category Archives: User Stories

How I Taught the D Programming Language at a Russian University

This article was originally published in Russian by Grigorii Smorkalov. It was translated to English for the D Blog by Georgy Markov and lightly revised from the original by Michael Parker.

This is the fourth year I’m teaching my D Programming Language course at a very real university in Russia. It’s a full-term course with lectures, practical lessons, and exams, although it’s all remote now. This is the story about how I got there, the challenges I encountered, and how students sometimes surpass their teachers.

What’s in D for university students

The job market for D is very small. As I always say during the first lesson, it’s unlikely that students are going to write D code for a salary, but that doesn’t mean that learning D is useless. Firstly, it’s much easier to learn how to program with D than with C or C++. This is important because many students don’t know how to program even after a full C/C++ course. Secondly, a broader outlook makes for better code. My familiarity with D improved my C++ skills and made it much easier to learn Python, especially its iterators. Most importantly, D is the future—of C++ and beyond. Many C++11/17/20 novelties were first battle-tested in D, and even today D is a much more modern and feature-rich language than C++.

Who am I and what’s in D for me

For simplicity, I would say that I am a C++ programmer. This is my main line of work. Before this course, I’d never been a teacher of any sort. Even on my job, I’m not involved much in mentoring. After earning my master’s degree in Computational Mathematics and Cybernetics at UNN (Lobachevsky State University of Nizhny Novgorod) in 2014, I had no relation to academics at all.

In my first months of university, I entertained the thought of being a school teacher. Now I realize that this was just a call for justice of sorts. There is a stark difference between a high school and a university, and for me, the latter was a much better experience. It felt like in just one month at university I learned more than in one term in high school, and that was so cool. Why couldn’t they explain it this way back in high school, was my common thought. If only educators applied the same educational methods in a regular school, it would make the experience much better. My desire to become a teacher vanished the second I received my first paycheck as a programmer (in Russia, ‘programmer’ is a very well-paid job and ‘school teacher’ is the total opposite), but some memory of that desire remained.

This was also the time when I became interested in D. Compared to C++ it looked like a perfect programming language. You can write code that would be as fast, but without all those C atavisms. I used D for my master’s thesis, and I loved it. My program was twice as small and simple as the older C++ version while performing better. Implementing complex and more efficient algorithms in D was much easier; doing the same in C++ would be too much work, and, like any student, I always struggled with my deadlines.

Since then I’ve followed D and used it for my little pet projects. I’ve always wanted to help the D community in some way, but I couldn’t write any useful library, nor could I find the motivation for contributing to open source.

Beginnings

In 2018, an unusual offer surfaced on the D mailing list: does anyone by any chance want to teach students in Moscow, Russia?

The initiator was Dmitry Olshansky, a well-known member of the D community, the creator of std.regex and more. I contacted him and said I was interested. I didn’t see myself as a full-fledged lecturer and expected to be just an assistant. At the beginning that was the plan, with someone else acting as a lecturer. He was going to give lectures remotely via Skype, and I would assist him on site.

To my surprise, the university in question was RSUH, Russian State University for the Humanities. As it turned out, they do have technical faculty there, and the students do actually code. I checked their program: D was introduced for third-term students, and during the first two terms they learned C, C++, Prolog, and even some Lisp, I think (a bit too much, but why not). Their math course was solid, too (yes, I am among those who think that math is important for programmers).

Preparations

I was introduced to the department staff. They explained everyone’s responsibilities and even offered the opportunity to join in scientific work as a programmer. We started working on the course program, although I barely included myself in that “we”. That was a mistake. With one month left until the classes started, the lecturer was suddenly leaving us. The news took me by surprise, but… there was still plenty of time, right?

This was happening when it was time for me to complete all the formalities and start work. Though I knew they were hiring only me, for some reason I was still under the impression that I wouldn’t be alone. I can’t say why I thought so. Everything was saying that there would be no help and that I had to do the whole course by myself, but my impression was hard to shake. The grave realization only came one week before Day One. Only then did I start to prepare for real.

Bureaucracy

This is supposed to be the part about the trials and tribulations of the endless bureaucracy awaiting a poor programmer’s soul. The amount of paperwork required to sign the contract was indeed an entertaining story to tell to my peers. And everyone was, as they say, “rolling on the floor laughing” when I described applying for a Mir payroll card (Mir is the Russian national payment system mandated in the state-funded sector). But that’s it, actually.

The next bureaucratic task was composing the formal program, an official paper including the course program and the materials. There were indeed a lot of formalities there, but I got some help: they showed me the paper for a very similar course. At the end of the day, it was easier than I expected. For this, I should thank the university staff. It was a one-year contract. Renewing it every year only takes one piece of paper and a couple of pen strokes.

The first class

The schedule was set up such that my first class happened to be a seminar (a.k.a. a practical lesson) instead of a lecture. This is only on paper, though with only one group consisting of just 14 people it didn’t make any difference. The schedule was adjusted later. I got two classes in a row and could decide which was a lecture and which was a seminar. I came up with the following arrangement: in the first class, I would lecture and answer general questions, and in the second, the students would program and ask practical questions.

For the first lesson, I prepared a brief description of the language, a syntax overview, a compiler, and several problems to solve: calculating the area of a triangle, solving a quadratic equation, and similar problems that yield a simple output for simple input. The idea was to immerse students in the language and immediately give them something to do with it. The plan was a success. By the end of the class, most of the students had gotten the hang of using a command-line compiler, written some code in a text editor, and solved some problems.

Notepad and the command line

I bet many people would take issue with the command line and text editor part. Seriously? No IDE? IDEs for D exist, but I left it up to the students if they wanted to use them. The reason was my own experience in learning C++ and programming in general. Knowing how a compiler works and how to link several files into a project is integral to understanding the language as a whole.

This is especially true for C and C++. Things like the difference between declaration and definition lose their meaning without understanding the build process. In D, such nuances are fewer. For example, header files are not required. Still, I meant to teach how to program in D, not how to use an IDE. Things like the principles of import are easier to grok when using a command-line compiler rather than some “intuitive interface”. And you learn the syntax faster if you write the code by hand without autocomplete doing it for you.

Further development

Lifted by the success of the first seminar, I was slammed to the ground by the first lecture. It contained the full theoretical explanation of type systems and their various types, compared D with other languages, brought out the problems of C and C++, and demonstrated what makes D different. It was a total failure.

I expected the lecture to take the whole 80-minute class, including questions, but it finished in less than an hour without a single question during or after the speech. I even asked if the students couldn’t hear or understand me, or if there was a problem with my diction. But the problem was with the lecture itself. First, it was too much for one class. Second, the lecture was based on the talks I did for my job that were intended for seasoned programmers. I realized that everything that I’d prepared for my future lectures must be tossed aside and rewritten from scratch.

The new program

The first candidate for simplification, and by that I mean expunging, was metaprogramming. Getting rid of templates was impossible since even the most basic language features and algorithms are tied to them, but code generation of all sorts was the first to be removed. Following that was anything that required external libraries. What was left were things that D code can’t be written without.

Since I had better success with live communication during actual programming, I decided to focus on that. Rewriting the course on the fly was tricky, but I came up with a solid plan: throughout the semester we would write a complex calculating program, during lectures we would study language features required for a given task, and then during seminars the ideas would be transformed into code.

The semester assignment

If you’re a nerd like me, you’ve probably heard of the 10,958 problem. The gist of it is that you need to put signs and parentheses between numbers composed of the digits 1 to 9 so that the result would be exactly 10,958. The solution exists for any number up to 11,000, except 10,958. There’s no proof that it doesn’t exist either, hence the 10,958 problem.

I gave the students an assignment to write a program that would find the solution, brute-forcing any possible combination of signs and parenthesis. All calculations were done with double instead of some sort of bignum, so it’s not a real solution, but it’s simpler this way.

I had several reasons to think that this was a well-fitting problem. First, I could write a solution I expected to see from the students in five hours or two evenings. Compensating for the students’ level, it looked like a good project for a semester-long assignment. Second, the solution isn’t too straightforward. Simple brute force would take too much time so you need to cut off the equal variants in the beginning. And third, I was fascinated by this problem myself, so I thought the students would feel the same. I couldn’t be more wrong.

Not only were a majority of students not into such mathematics, most of them couldn’t even grasp what the problem was about and what this 10,958 was for. I failed to get them interested.

When I realized the problem, it was too late to change anything. So the first semester wasn’t very engaging. Only a few students could finish the assignment to its fullest. For the rest, it was impossible.

The second semester

Since it was a full-term course, I had time to make up for it. For the second semester, I tried to come up with something practical and interactive. I gave the students an assignment to write a game. They were learning networking, so it was to be a multiplayer game. I recalled playing “tic-tac-toe on an infinite field” with my peers during boring lectures. The smarter name for this game is Gomoku. Two players can play online, and the game logic is simple. I was hoping to spark students’ interest. This assignment turned out much better than the previous one, but I still can’t call it a full success.

I really wanted to show them how coroutines (fibers, as they are called in D) make async programming much more manageable, how nice the vibe.d framework is, and how easy it is to use external libraries with the dub package manager. Lesson One: don’t try to sell coroutines to those who don’t know what callback hell is. Lesson Two: always keep hardware limitations in mind.

Problems out of nowhere

I would never have thought that single-threaded compilation could be thwarted by the memory limit. It happened because the computers at the university were equipped with only 2 GB of memory, and some students’ netbooks worked on a mere 1 GB. Simple programs build just fine even on weak machines—D is much better than C++ in terms of compilation time—but a large framework like vibe.d requires a lot of memory for its compilation. Now imagine, I was just telling them how everything is so easy-peasy and then half the students couldn’t even compile a networked version of Hello World.

In my defense, I checked everything in advance. I set up a virtual machine with 2 GB of memory and made sure that it worked with the special compiler flags. Theoretically, I was prepared. But dealing with a new library and a new build system and new compiler flags all at once was just too much for the students. Their brains were getting DoS’d and shut down. So even though I demonstrated how to compile a program on a low-memory machine, I still had to explain to them individually why the usual method of compilation wasn’t working. For those who had only 1 GB of memory, I didn’t even have a ready solution.

But still, the results of the second semester were much better: absolutely everyone could write the client side of the game. Some had problems with coroutines on the server side, but in general, they got this part just fine. For me, that wasn’t enough. So I gave them a supertask: write a simple AI for the game. This would help them understand the advantages of the client-server architecture. They had to realize that an AI is just another client, so the server code should be left as it is, and on the client side, the only required modification was move polling. This was a good problem on architecture design, suitable for students who already had gotten a grip on programming.

We also had a little contest. I invited the students to play some code golf, solving a simple problem with the smallest program possible. For encouragement, I promised to free the best-achieving students from writing a semester report. And if anyone could beat my solution, they would pass the semester examination automatically (in Russian universities, teachers are allowed to set arbitrary conditions, like participating in side projects, for passing the semester without the usual examination).

As I expected, nobody could beat me, though a couple of students came up with some interesting tricks, like writing 10 instead of "\0" to save two symbols. Some would say this is an abuse of the type system, but clever hacks like this speak to a student’s knowledge.

The results of the first year

I was asking too much for a single semester; only one student could write an AI that kind of worked (she said that the first time it made a sensible move, it made her really happy). Another student who complained the whole year that programming was not her thing and that she couldn’t understand anything managed to write her own client and server. At the end of the day, all was not for nothing.

The second try

Everything described until this point happened during the 2018/2019 academic year. The higher-ups had no issues with the course, and I was able to continue on the following year. New students, new possibilities, a new program. This time I was much better prepared. I had materials for most of the lectures, no more need to redo everything on the fly, and during the summer I had time to fix the problems with the course.

The 10,958 problem was expelled on the charge of being boring. Instead, the Gomoku game was expanded, now lasting almost the whole term. “Almost” because the first assignment was a problem regarding OOP: students had to implement various geometrical shapes as classes and draw them on screen with some customization. I made this the first step after Hello World so everyone could learn how to use the tools, and so I could judge their level.

It’s not surprising that students are very different. Some are already working as programmers and attend the course out of curiosity, while some still struggle with variables and loops. From the beginning, I decided that my course would be not just for everyone, but for those who are interested. However, disregarding the others would be wrong, so we needed assignments for different levels.

To jump-start the network study, I set up a server with the reference implementation, and everyone could connect to it and play. Even telnet would do. Allotting more time for an assignment and providing better explanations served students well. Many could finish the supertask. We even had a little contest between AIs. A human could still beat them with ease, but it was still a big improvement over the previous year.

A pleasant surprise

I had the same code golf contest again. And again, I promised to give a free pass to whoever could beat me. I’m sure you can see where this is going, but first I must explain the problem in detail. The task was to implement a function, challenge, defined as follows:

import std;

/**
* Params:
* s = Multiline string, each line containing positive integers separated by whitespace.
* Returns: An array containing the sums of each line and the grand total as the last element.
*/
uint[] challenge(string s);

unittest {
    assert(challenge("0") == [0, 0]);
    assert(challenge("1\n1") == [1, 1, 2]);
    assert(challenge("2\n2\n3") == [2, 2, 3, 7]);
    assert(challenge("2\n2 0 3\n3 1 1 4") == [2, 5, 9, 16]);
}

Only the symbols inside the function body count. Using any Phobos functionality is allowed, but no renamed imports.

My solution from the previous year was pretty straightforward:

auto r=s.split('\n').map!(a=>a.split.map!(i=>i.to!uint).sum); return r.array~r.sum; // 84 characters

I didn’t show them my solution, I just told them its character count. I was absolutely sure that, just like the previous year, nobody could beat it. Imagine my shock when I was outdone in a mere week:

auto r=split(src,"\n").map!"sum(map!(to!uint)(a.split))".array;return r~[sum(r)]; // 82 characters

Even with the unneeded brackets, this solution was better, thanks to the shortened map syntax: passing a lambda as a string in the first case and passing to directly in the second. The latter was just my oversight, but the former was a typical teacher’s mistake. You see, back when I first wrote my solution, it wasn’t possible to pass a string to map like that. I don’t remember well what the problem was, something about scopes. I missed that it was fixed at some point. That’s how in just one year you become an old geezer teacher who can’t keep up with the times.

I stayed true to my word and granted a free pass to the resourceful student at the end of the semester. I was concerned that securing the pass in the middle of the semester would demotivate him to attend classes, but fortunately, I was wrong.

COVID-19 strikes

Of course, I can’t omit how the pandemic impacted the educational system. In the spring of 2020, all classes were moved online. There are some pros and cons to this.

Giving lectures remotely turned out to be very convenient. You can mutually agree on a suitable time, and you don’t need to find a free lecture room. Another very good thing is that students can either ask questions vocally—as when they raise their hands offline—or write them in the chat so that the teacher can answer them when it’s most appropriate. I really think that this system works very well.

Practical lessons are problematic though, especially with those students who aren’t involved enough. When they sit in a classroom they at least do something. Why show up at all if you’re not going to do anything? When it’s online, they just skip class. The chemistry of a group coding session with a teacher ready to help won’t kick in. I encouraged them to ask questions not only during class but at any time, hoping that this would allow me to assist them whenever they are coding. But this only worked for a couple of people.

Lessons of the second year

The second year was better than the first but still less than ideal. I saw no need to modify the course, but its presentation had to be addressed.

We need to tear down this wall in communication. Mutual trust between the teacher and the student makes for a better educational process. It’s difficult to work with those who are wary of the teacher even when doing alright. I don’t yet know a robust solution; each case is tackled individually.

Even those who are doing well need some control. I used to think that attendance scrutiny is for those students who don’t really want to learn, to intimidate them to show up. When I was a student, the best teachers never bothered about absentees. So I too was liberal with these things, even when half the group was missing. But everything has its limits. Bad students will skip classes anyway, but lazy B-graders could benefit from a little scolding.

Fast forward to the present

I don’t have much to say about the last year-and-a-half. Teaching D is now part of my life. Due to the pandemic, we’ve had to move online completely. I couldn’t even meet my students face-to-face. Aside from what I said earlier about how this affects the educational process, this also disrupted one of the ideas I had.

I need some way to track students’ progress to be sure that they actually follow the course and don’t just show up for classes. The way I handled this during the previous two years worked for some students: if they had a problem, they asked questions during a lecture or a practical lesson. But some students just keep quiet when they have problems, so I need some means to identify them. I couldn’t do written tests for programming; that would be nonsense. So during the summer break, I came up with the idea of doing some small quizzes if COVID restrictions were to be lifted. These quizzes would affect the final grade, but not much, as the intention behind them was to help students who are having problems, not to be the final straw that breaks the camel’s back. Unfortunately, remote lessons made this nearly impossible.

My students keep surprising me, coming up with newer and better solutions to the code golf puzzle. Unfortunately, I don’t have the exact code of the student who won in the third year (it appears that she deleted her Github account), but it was something like:

auto r=s.split('\n').map!(a=>a.split.to!(uint[]).sum); return r.array~r.sum; // 74 characters 

Again, it was a feature I didn’t know about: to converts between arrays, too.

I thought that this problem was done and that there was no room for further improvement. Oh, how wrong I was. This is what they’ve brought me this year:

return(s.split('\n')~s).map!(a=>a.split.to!(uint[]).sum).array; // 63 characters

This completely destroys my solution with all the improvements! Note that the trick is not in the syntax, but in the logic. This works by concatenating the split lines with the original string, making it unnecessary to declare an intermediate array, which allows implementing this function as a single statement. I would never have thought to do that! Algorithms win over micro-optimizations, even in code golf.

The biggest change this year is a new, formalized evaluation system. In the semester, students earn points for doing assignments and writing reports on them. The first and simplest assignment is worth 5 points, and the last and hardest is worth 30. The maximum number of points a student can score without participating in code golf is 100. Code golf is scored by the formula 110 − length. The code golf winner this year got 47 points for his solution which earned him an exemption from writing any reports. We have a table listing every student’s points so everybody knows how many points they need to score. Everything is very transparent, so I don’t need to worry about not being objective when evaluating students.

A Gas Dynamics Toolkit in D

The Eilmer flow simulation code is the main simulation program in our collection of gas dynamics simulation tools. An example of its application is shown here with the simulation of the hypersonic flow over the BoLT-II research vehicle that is to be flown in 2022.

BoLT-II simulation with steady-state variant of the Eilmer code. Flow is from bottom-left to top-right of the picture. Only one quarter of the vehicle surface, coloured grey, is shown. Several slices through the flow domain are coloured with the local Mach number, with blue for low values and red for high values. Several streamlines, drawn in black, start at the blunt leading edge of the vehicle and follow the gas flow along the vehicle surface. Image produced by Kyle Damm.

Some history

This simulation program, originally called cns4u, started as a relatively small C program that ran on the Cray-Y/MP supercomputer at NASA Langley Research Center in 1991. A PDF of an early report with the title ‘Single-Block Navier-Stokes Integrator’ can be found here. It describes the simple finite-volume formulation of the code that allows simulation of a nonreacting gas on a single, structured grid. Thirty years on, many capabilities have been added through the efforts of a number of academic staff and students. These capabilities include high-temperature thermochemical effects with reacting gases and distributed-memory parallel simulations on cluster computers. The language in which the program was written changed from C to C++, with connections to Tcl, Python and Lua.

The motivation for using C++ in combination with the scripting languages was to allow many code variations and user programmability so that we could tackle any number of initially unimagined gas-dynamic processes as new PhD students arrived to do their studies. By 2010, the Eilmer3 code (as it was called by then) was sitting at about 100k lines of code and was growing. We were, and still are, mechanical engineers and students of gas-dynamics first and programmers second. C++ was a lot of trouble for us. Over the next 4 years, C++ became even more trouble for us as the Eilmer3 code grew to about 250k lines of code and many PhD students used it to do all manner of simulations for their thesis studies.

Also in 2010, a couple of us (PAJ and RJG) living in different parts of the world (Queensland, Australia and Virginia, USA) came across the D programming language and took note of Andrei Alexandrescu’s promise of stability into the future. Here was the promise of a C++ replacement that we could use to rebuild our code and remain somewhat sane. We each bought a copy of Andrei’s book and experimented with the D language to see if it really was the C++-done-right that we wished for. One of us still has the copy of the initial printing of Andrei’s book without his name on the front cover.

Rebuilding in D

In 2014 we got serious about using D for the next iteration of Eilmer and started porting the core gas dynamics code from C++ to D. Over the next four years, in between university teaching activities, we reimplemented much of the Eilmer3 C++ code in D and extended it. We think that this was done to good effect. This conference paper, from late 2015, documents our effort at the initial port of the structured grid solver. (A preprint is hosted on our site.) The Eilmer4 program is as fast as the earlier C++ program but is far more versatile while being implemented in fewer lines of code. It now works with unstructured as well as structured grids and has a new flexible boundary condition model, a high-temperature thermochemistry module, and in the past two years we have added the Newton-Krylov-accelerated steady-state solver that was used to do the simulation shown above. And importantly for us, with the code now being in D, we now have have many fewer WTF moments.

If you want more details on our development of the Eilmer4 code in D, we have the slides from a number of presentations given to the Centre for Hypersonics over the past six years.

Features of D that have been of benefit to us include:

  • Template programming that other Mechanical Engineers can understand (thanks Walter!). Many of our numerical routines are defined to work with numbers that we define as an alias to either double or Complex!double values. This has been important to us because we can use the same basic update code and get the sensitivity coefficients via finite differences in the complex direction. We think this saved us a large number of lines of code.

  • String mixins have replaced our use of the M4 preprocessor to generate C++ code in Eilmer3. We still have to do a bit of head-scratching while building the code with mixins, but we have retained most of our hair—something that we did not expect to do if we continued to work with C++.

  • Good error messages from the compiler. We often used to be overwhelmed by the C++ template error messages that could run to hundreds of lines. The D compilers have been much nicer to us and we have found the “did you mean” suggestions to be quite useful.

  • A comprehensive standard library in combination with language features such as delegates and closures that allow us to write less code and instead concentrate on our gas dynamics calculations. I think that having to use C++ Functors was about the tipping point in our 25-year adventure with C++.

  • Ranges and the foreach loops make our D code so much tidier than our equivalent C++ code.

  • Low-barrier shared-memory parallelism. We do many of the flow update calculations in parallel over blocks of cells and we like to take advantage of the many cores that are available on a typical workstation.

  • Simple and direct linkage to C libraries. We make extensive use of Lua for our configuration and do large simulations by using many processors in parallel via the OpenMPI library.

  • The garbage collector is wonderful, even if other people complain about it. It makes life simpler for us. For the input and output, we take the comfortable path of letting the compiler manage the memory and then tell the garbage collector to tidy up after us. Of course, we don’t want to overuse it. @nogc is used in the core of the code to force us not to generate garbage. We allocate much of our data storage at the start of a simulation and then pass references to parts of it into the core functions.

  • Fast compilation and good optimizing compilers. Nearly an hour’s build time was fairly common for our old C++ code, and now we would expect a DMD or LDC debug build in about a quarter of a minute. This builds a basic version of the main simulation code on a Lenovo ThinkPad with Core i7 processor. An optimized build can take a little over a minute but the benefit of the faster simulation is paid back by orders of magnitude when a simulation job is run for several hours over hundreds of processors.

  • version(xxxx) { ... } has been a good way to have variants of the code. Some use complex numbers and others are just double numbers. Also, some variants of the code can have multiple chemical species and/or just work with a single-species nonreacting gas. This reduction in physical modelling allows us to reduce the memory required by the simulation. For big simulations of 3D flows, the required memory can be on the order of hundreds of gigabytes.

  • debug { ... } gets used to hide IO code in @nogc functions. If a simulation fails, our first action is often to run the debug-flavor of the code to get more information and then, if needed, run the debug flavor of the code under the control of gdb to dig into the details.

We have a very specialized application and so don’t make use of much of the software ecosystem that has built up around the D language. For a build tool, we use make and for an IDE, we use emacs. The D major mode is very convenient.

There are lots of other features that just work together to make our programming lives a bit better. We are six years in on our adventure with the D programming language and we are still liking it.

Teaching D from Scratch: Is it a viable first language?

About two-and-a-half years ago, one of my son’s homeschooling friends asked if I would teach a coding class. Both of my kids have shown interest in coding via Scratch, and I also need to pull my weight when it comes to homeschooling them (my wife doing most of the instruction). So I thought, surely, this can’t be too hard. The challenges I would encounter over the course of the last few years surprised me more often than I would have expected!

First, a bit about my background. I’ve been writing software professionally for over 20 years, about half using system languages and half doing web development. I’ve dabbled in a lot of programming projects, including embedded microcontrollers with 256 bytes of RAM, collaborative video systems using high-end Unix workstations, and automating a factory testing rack-mounted appliances. What I haven’t ever done to any official degree is teach programming. For sure, this is a challenge that is novel to me, and also one that I have found very satisfying.

A bit about the kids. I had ten students, aged 9 – 15, all of whom had very little experience writing real software. Since this class was going to be a side project for me on top of my normal job, we went rather slowly, meeting once every two weeks.

One of the goals of my class was to try not to dumb down coding into an abstraction. I wanted the kids to experience real programming without too much of a pre-built hand-holding framework, and I believed they could do it. I’ve seen many (and purchased some) online resources for teaching to code that hide most of the real work and just focus on using custom libraries or simplified languages to learn the “basics”, while not actually teaching to be productive in a standard environment. I also witnessed firsthand my own kids getting bored when the material was presented too slowly or too abstractly.

Note: you can see some of my homework assignments and review pages on a dedicated website that I created for this class

First Try: Javascript and Lua

The first language we worked with, because many of the kids didn’t have actual full-blown computers to work on (this was pre-Covid so it was at my house), was Javascript. Javascript is possible to develop in a browser, using a tablet or a Chromebook, so all the kids I was teaching could participate. My focus in these early days was basic concepts like loops, branching, and basic statements. I think Javascript can be a great first language, though the things you can do with it as a beginner are a bit bland, and to do anything exciting you have to start learning the DOM (including how objects and methods work).

After half a year we moved on to our second language: Lua, specifically in the video game system Roblox. I almost wish we had skipped this one because it was a bit too advanced for these students, and I was not familiar with the language and the system to begin with. The kids did have fun with the model-building UI. However, I can see an experienced teacher finding good ways to instruct using this environment. Roblox, with its pre-built rendering and physics engines, has the nice benefit of allowing one to generate a really cool game with relatively little code—something new coders quite enjoy.

Just use D

And that leads us to the second year, where I decided to just try teaching what I know best: the D programming language. Since all the kids have a keen interest in building games, I searched for a simple library I could use to build games. I have to give a shout-out to the library I discovered, Raylib, and the YouTuber Ki Rill, whose very straightforward and well-made videos gave me the confidence that this also would be something I could teach.

For a quick demo of what Raylib looks like, here is a program I wrote in a few minutes that bounces a Raylib-drawn D-man around the window.

Everyone in the class is using Windows, so the toolchain that I am having them use is the stock DMD compiler and the Visual Studio Code development environment with the code-d extension. Throughout the last year-and-a-half, I have had the kids build such classics as hangman and tic-tac-toe using text-based “graphics”. After that came the introduction to 2D graphics and Raylib, where we built a brick-breaking game similar to Arkanoid and others.

Bricker

Finally, I let the students pick what they wanted for the last project, and their pick was a rock-paper-scissors online tournament game. I’m not sure of the lasting power of that one…

Keep in mind that my goal was to teach them programming, not necessarily D. Working through all these projects allowed me to introduce a lot of the typical programming concepts (branching, looping, functions, data types, arrays, etc.). This year we are focusing on aggregates (specifically arrays and structs), and how they can be useful to encapsulate your coding solution into pieces that are easier to deal with. The game we are writing this year is a snake clone. I’m hoping to wrap things up on that by the end of the year, and then next year add in networking to have the game playable between the students (all the kids these days are into online tournament-style games).

Snake

My Take on Teaching With D

So how is D as a beginner’s programming language? I can say with confidence that the kids understand the flavor of D that I have taught just as well as the other languages. What flavor is that you may ask? Why vanilla of course! As D was the first typed language they were exposed to, I had to first explain types and why they are important. I also had to pick and choose which concepts in D I did not want to confuse them with. To that end, I’ve left out a lot of the advanced features (so far), such as:

  • Templates
  • Type inference
  • Operator overloading
  • Overloading in general!
  • __traits
  • Classes/Interfaces/OOP
  • Pointers/references (mostly)
  • Memory management in general (thank you GC!)

This means that a lot of language features are missing from “Vanilla D”, and a lot of those are pieces I like. I had to force myself to avoid those things while teaching so as not to confuse them. I also try to avoid shortcuts if I can, as I want them to master the basic concepts before learning how to do it quicker or less verbosely. I tried hard to always use curly braces for scope blocks, for instance, instead of just writing single statements without them.

The features that I found the kids handled well were basic types like ints and strings (though what a funny name “string” is, that took a while to sink in), if statements and foreach/while loops, structs, and functions. They had a harder time understanding arrays and associative arrays, which might be due to my lack of experience teaching. I’ll also note that teaching virtually has some significant challenges—there’s just no substitute to being able to walk over to a student’s computer and observe and help their progress or draw out on a whiteboard how data is laid out. One surprising thing (to me anyway) that they handled completely in stride, was nested functions. That’s a nice D feature that is not available in C, barely in C++, and something that I personally took a while to get used to.

What Could Go Wrong?

The challenges were numerous. First I had to try and remember what it’s like to know nothing about programming. I did not take any classes on teaching programming and did not look up the “proper” way to teach it, I just went with what I thought would be the next best thing to do based on their skills and experience. Having class only once every two weeks is a big drawback since the students’ brains tend to garbage-collect some of the stuff we worked on last time. For sure, a daily or maybe twice-weekly class schedule would improve the situation, but that would mean I’d have to generate material to teach that many classes, which I unfortunately don’t have the time for.

Secondly, I previously had no experience with games programming. Raylib made this pretty easy, and as long as you can understand event loops, it’s pretty straightforward (and fun—I wish I had learned it sooner).

Aside from my personal and environmental shortcomings, I have some suggestions for the D language and community that would make D a much better “first language”. In no particular order:

Error Messages

Error messages in D are sometimes esoteric, and I’m not just talking about templates. The most common error the students make is bad punctuation. This leads to some of the worst messages. Most of the time, I need to explain what is happening, but even when explaining it myself, I am sometimes thwarted by the compiler. For instance, if you forget a semicolon:

void main()
{
    import std.stdio;
    int x = 5
    writeln(x); // line 5
}

The resulting error is:

foo.d(5): Error: semicolon expected, not `writeln`

But wait, there is a semicolon on line 5! So why is it having a problem? Technically, putting an extra semicolon just before the writeln call on line 5 would solve the error (after all, D doesn’t have significant whitespace). But really, the compiler shouldn’t be suggesting that.

I’d rather see something like:

foo.d(5): Error: previous statement not terminated, perhaps you need 
a semicolon on line 4?

What if you are missing a brace or have an extra one? That spits out a lot of error vomit that is sometimes really difficult to decipher. I know this isn’t as easy a problem to solve, but since the compiler is not even to the tricky semantic parts, what if it used some other hints of the source to try and tease out a suggested solution? Like maybe taking cues from the indentation.

A great feature of the D compiler is the spelling correction suggestions. But sometimes it’s not easy to see what has changed in the suggested spelling correction (especially when it’s just different capitalization). Why not highlight the changed letters? Since D has gained colorized output for errors, things have become much clearer in my opinion.

I do have to relay a comment from one of my students, who said he appreciated the ability of VS Code to jump to the line that the compiler error is referencing. This helps put into perspective where their mindset is.

Debugger on VSCode for Windows

Another challenge I’ve faced is the lack of a good debugging experience for VS Code on Windows. The C debugger for VS Code displays, for example, a string as a length and a C string (which might have trailing garbage). I’d rather not confuse them with this. I have read that Visual D has a better debugging experience, but I need dub support for these projects, and Visual D focuses on Visual Studio integration, something I don’t necessarily want to deal with in teaching these kids. I’m hoping that a recent focus on debugging (e.g., the LLDB integration project) will help.

I think WebFreak has done a great job on the VS Code plugin, and for the most part, it works amazingly well. Thank you! I know debugging is a piece that is hard to get right.

Linking External Libraries

My next challenge is using dub for linking. Dub is great if you are only doing things with D and OS-supplied libraries. As soon as you need to depend on an external C library that is not part of your OS (such as Raylib), now you are not just adding dependencies but downloading pre-built libraries, or trying to build them yourself. I’m not sure if I have great suggestions on this, but I would like some way for dub to be told where to download these libs or have a central place to put them where the linker can find them outside of specifying the location in your dub.json file. Many libraries are released on GitHub, which makes it a prime candidate for downloading and installing external libraries (maybe even when ImportC becomes viable, it can automatically generate the bindings as well).

I spend an inordinate amount of time helping the kids set up their build environments, and having this built-in to the IDE or dub would be immensely helpful for using D for learning.

Suggestions for the Future

I am confident that D can be a viable first language. I have students that, prior to my class, had never before written real code now creating 2D games using D and Raylib (with a lot of hand-holding, darn it!). What I think D needs for this success to be scalable is a set of resources that assumes no programming experience. Most of D’s learning resources are aimed at teaching experienced programmers how D works in terms of their existing knowledge. They answer the question, “How do I do <insert feature> from <insert language> in D?” What we need is to answer the question, “How do I write code?”, that just happens to be using D.

Editor’s Note:Some readers may be thinking at this point of Ali Çehreli’s Programming in D. Though written for beginner-level programmers, it’s focused more on teaching the language than teaching to program generally.

This shouldn’t be too hard to create, as there are a ton of resources on learning to program in C already. D is so similar you could almost just replace C with D and call it a day. In fact, I’m considering creating a YouTube series on the topic with the experience I’ve gained. One important tool that needs creating is a map of D features and where they fall on the difficulty scale. This allows one to write D tutorials that follow a gradual introduction of features. Broaching the subject of complex features is probably needed early, as they will experience some of them (you are using a template whenever you use writeln, or to), but explanation of the entire feature may be deferred to a later level.

And finally, such materials should be engaging! I can’t stress enough how important it is to be doing something fun when you are learning to program. The kids definitely are more engaged since we are creating games that they might actually want to play vs. simple hello world examples or even Javascript web pages. A nice way to proceed might be to write a game in Vanilla D and then introduce features that allow you to refine and improve the code.

I hope this becomes a reality because I think this area of development has been relegated to either frameworks that attempt to water down programming to get kids engaged or sterile hello world-style programming that is intended for rote instruction or reference. I think we can show the full potential of D and, at the same time, engage eager learners with interesting and fun projects to trick them into learning code with our favorite language!

I Wrote a High-Frequency Trading Platform In D

I’ve used the D programming language to implement a high-frequency trading (HFT) platform. I’ve been quite satisfied with the experience and thought I’d share how I got here. It wasn’t a direct path.

In 2008, I stumbled across a book on Amazon called Learn to Tango with D. That grabbed my curiosity, so I decided to research D further. That led me to Digital Mars and Walter Bright. I had first heard of Walter when I learned about Zortech C++, the first native C++ compiler. His work had been a huge influence on my C++ learning experience. So I was immediately interested in the language just because it was his, and excited to learn that he was working with Andrei Alexandrescu on version 2. Still, I decided to wait until they were further along with the new version before I dove in.

In 2010, I bought Andrei’s The D Programming Language as soon as it was published and started reading. At the time, I was working at BNP Paribas using C++ to optimize their HFT platform, so high performance was prevalent in my thoughts. When I saw that D’s classes were reference types, with functions that are virtual by default, I was disappointed. I didn’t see how this could be useful for low-latency programming. I became too busy with work to explore further at the time, so I put the book and the language aside.

In 2014, I began preparing for a new adventure. As part of that, I started working on a feed handler framework from scratch in C++, using my own long-maintained C++ library of low-level components useful in low-latency, high-performance applications. Andrei’s book came to my attention again, so I decided to give it another look.

This time, I read the book through to the end and learned that my initial impression had been misplaced. I found that I liked D’s metaprogramming features and its support for programming in a functional style. By the end of the book, I was ready to give D a try.

I started by porting my C++ library and feed handler to D. It wasn’t difficult. I use very little inheritance in my C++ code, preferring composition and concrete classes. I found myself quite productive with D’s structs, templates, and mixins. All the while, I kept a close eye on performance benchmarks. When D turned out to give me the same performance as my C++ code, I was sold. I found D to be much more elegant, cleaner, more readable, and easier to maintain. I made the switch and never looked back.

My goal was to develop a complete HFT system using D. The system would consist of different subsystems:

  • Feed-Handler Framework: receives market data from exchanges; builds the books for all securities; publishes the updates to the other subsystems.
  • Strategies Framework: receives market data updates from feed handlers; facilitates communications with the Order Management System; allows for plugging into it strategies that make decisions on stock trades.
  • Order Management System: communicates with the exchange and the strategies framework; maintains a database of orders.
  • Signal Generator: receives market data updates from feed handlers; generates different signals as indicator values, predictions of stock prices, etc.; sends the different signals to strategies.

Ultimately, I found a new data structure and better design for my feed-handler framework. I developed the new version completely in D. This implementation heavily uses templates. I like D’s template syntax and generally find the error messages clearer than the complex error messages I was used to from C++. I needed to drop down to assembly for some specific x86 instructions and it was straightforward to do in D.

Later, I needed to work with configuration files. I prefer to write my config files in Lua, a lightweight scripting language that is easy to integrate into a program as an extension via its C API. For this, I found a D Lua binding called DerelictLua. Using, again, D’s metaprogramming facilities, I developed a very easy and practical way to interface with Lua on top of DerelictLua. Editor’s Note: DerelictLua has since been discontinued; new projects should use its successor, bindbc-lua, instead.

The feed handler on the Bats market comes on 31 simultaneous channels, so it is more efficient to use multithreading. For this, I chose not to use the multithreading facilities provided by Phobos. I felt I needed more control in such a low-latency environment, particularly the ability to map each thread to a specific core. I opted to use the pthreads library and its affinity feature. D’s C ABI compatibility made it a straightforward thing to do.

I’m running on FreeBSD. For my intercommunication needs, I’m using kernel queues and sockets. The same functionality is available on macOS, my preferred development platform. D did not get in my way in using these APIs on either macOS or FreeBSD. It was as seamless as using kernel queues from C.

A few notes about problems and limitations:

  • I encountered one compiler bug. I found a workaround, so it wasn’t a blocker. I was able to reproduce it with a few lines of code and contacted the D community. They solved the problem and had a fix in a later version of the compiler.
  • I did not use D’s garbage collector. This is not a strike against D or its GC, though. In a low-latency system like this, even the use of malloc and free can be costly, so I’m not going to take a chance on a nondeterministic system with unpredictable latency. Instead, I used my library to handle allocation/deallocation via free lists, with memory preallocated upfront. As a consequence, I also decided not to use D’s standard library for anything.
  • I had to work with fixed-size ASCII strings that are not NUL-terminated and are, instead, padded with spaces at the end. Without the standard library, I found it easier to manipulate them C-style via pointers.

I was the sole developer on this project but completed it successfully in a relatively short period. Big credit to D and its productivity, readability, and ease of modifications.

Driving with D

Here is what comes to mind when I think of D: fast, expressive, easy, and… driving? That’s right, I drive with D.

Enter my venerable Holden VZ Ute daily driver. From the factory, it came with a rubbish four-speed automatic gearbox. During 18 months of ownership, I destroyed four gearboxes. I could not afford a new vehicle at the time (I’m a 20-year-old Australian computer science student at Monash University), so I had to get creative. I purchased a rock-solid, bulletproof, six-speed automatic gearbox from another car. But that’s where the solutions ended. To make it work, I had to build my own circuit board, computer system, and firmware to control the solenoids, hydraulics, and clutches inside the gearbox, handle user input, perform shifting decisions, and interface to my car by pretending to be the four-speed automatic.

I’m quite proud of my solution. It can perform a shift in 250 milliseconds, which is great for racing. It has a steep first gear, giving it a swift takeoff. It has given some more powerful cars a run for their money. It’s got flappy paddles, diagnostic data on the screen, and the ability to go ahead and change the way it works whenever I want.

Here’s a very old video of the system working. It’s not representative of the current system—that ghastly blue screen is gone, the speedo works, and shifting has improved.

The computer is split into two parts: the user interface board, which drives an OLED display and uses an STM32F042, and the mainboard, which handles everything else, utilizing an STM32F407. The two cooperate over a CAN bus (Controller Area Network). All the firmware to handle this is written in D.

I picked D (as -betterC) because of its ingenious Uniform Function Call Syntax (UFCS), design by contract, metaprogramming, ease of interfacing with C, unit testing, portability, shared, @safe, and codebase maintenance features. Another bonus is the helpful, welcoming community. It has genuinely been a joy discussing D on the forums, and with the founders and community leaders.

The Advantages of D

Uniform Function Call Syntax (UFCS)
This has made my code significantly clearer. My code can accurately follow the flow of data without polluting my stack with single-use variables, nesting many function calls, or other sorts of clutter.
Here is an example of how you could potentially use it in an ECU (Engine Control Unit):

immutable injectorTime = airStoich(100.kpa, 25.degCelsius)
    .airMass
    .fuelMass((14.7f).afr)
    .fuelMol
    .calculateInjectorWidth;

This is equivalent to:

immutable injectorTime =
    calculateInjectorWidth(
        fuelMol(
            fuelMass(
                airMass(
                    airStoich(kpa(100), degCelsius(25))
                ),
                afr(14.7)
            )
        )
    ); // brackets have been expanded for reading clarity

Please note: the values in this example are hardcoded to simplify the code and demonstrate how UFCS can give a unit of measurement to a value.

Both representations are valid D code; you can use either.

With UFCS, there’s no need to read the code backward or count your brackets, no need to use a gazillion single-use variables. Function calls mirror the flow of data. It’s concise.

Design by Contract
D’s contract programming is quite similar to Ada’s. Functions can be marked with preconditions, designated by in, and postconditions, designated by out. Should a contract fail, an assertion is thrown.

// This demonstrates D’s contract programming for a function.
// It uses the short-hand expression based syntax.
int iHaveAContract(void* ptr)
in(ptr !is null) // this is a precondition, if ptr is null, an assertion is raised
in(ptr !is null, "ptr is null :(") // this is a precondition, if ptr is null, an assertion with the error message "ptr is null :(" is raised
out(result; result > 0) // this is a post condition, it captures the return value as result and tests it
out(result; result > 0, "result was too low") // this is a post condition, it captures the return value as result, tests it, and if it fails, raise an assertion with the message "result was too low"
{
    // normal function code here
}

If you want to do something a bit more complex in your contracts, an alternative syntax is available:

int iHaveAContract(void* ptr)
in {
    assert(ptr !is null); // this is equivalent to in(ptr !is null) from above.
    MyStruct* ms = cast(MyStruct*)ptr; // we can introduce variables local to this contract
    assert(ms.blah == 2, "MyStruct.blah must be 2!"); // test ms.blah, if it fails, raise an assert with an error message
}
out(result) { // captures the return value as result
    int squareIt = result * result;
    assert(squareIt == 4);
}
do { // this designates the function body
    // normal function code here
}

void main(string[] args)
{
    auto i = iHaveAContract(null); // this will violate the contract
}

Structs and classes can include contracts, called invariants, that sanity check the state of an instance for its whole lifetime. Invariants are checked after the constructor is run, before the destructor is run, and before and after public function calls.

struct MyStruct
{
    int blah;

    invariant {
        assert(blah == 2, "blah must always be 2 for some reason!");
    }
    // shorthand:
    invariant(blah == 2, "blah must be 2 for some reason!");

    // The invariant is checked at function entry and exit. If value is anything other than 2, the invariant will fail when the function exits
    void setBlah(int value)
    {
        blah = value;
    }
}

void main(string[] args)
{
    MyStruct s; // invariant is run after construction
    s.setBlah(3); // invariant contract will be violated.
}

Metaprogramming
The Don’t Repeat Yourself (DRY) principle is often touted by programmers. D’s metaprogramming is an incredible tool to achieve that goal. I use it in my CAN bus implementation. For example:

struct CANPacket(ushort ID) {
    enum id = ID;
    ubyte[8] data;
}
alias HeartbeatPacket = CANPacket!10;
alias BeepHornPacket = CANPacket!140;

I’ve got specific aliased types as HeartbeatPacket and BeepHornPacket, but I haven’t needed to repeat any code. They all follow the same underlying structure, so if I modify CANPacket, every alias is also updated.

Maybe you want a more descriptive CAN packet? Mixin templates can help with that!

struct GenericCanPacket
{
    ushort id;
    ubyte[8] data; // 8 bytes to store the CAN packet payload
}

struct HeartbeatPacket
{
    ubyte deviceID; // first byte is the device ID
    ubyte statusID; // second byte is the status
}

To translate a GenericCanPacket to the descriptive HeartbeatPacket, we can use mixin templates.

mixin template CanPacketHelperFunctions(ushort ID)
{
    enum id = ID;

    // typeof(this) means that the return type of readFromGeneric will be the struct this template is instantiated in.
    static typeof(this) readFromGeneric(const ref GenericCanPacket p)
    in(p.id == ID, "Generic packet cannot be converted!")
    {
        // do some cast
    }
}

struct HeartbeatPacket
{
    // the stuff declared in CanPacketHelperFunctions is pretty much copy-pasted (not literally) into here
    mixin CanPacketHelperFunctions!10;

    ubyte deviceID; // first byte is the device ID
    ubyte statusID; // second byte is the status
}

void main()
{
    GenericCanPacket generic;
    generic.id = 10;
    generic.data = [ 2 /* deviceID */, 3 /* statusID */ ];

    HeartbeatPacket heartbeat = HeartbeatPacket.readFromGeneric(generic);
    writeln(HeartbeatPacket.id); // the packet ID is 10
    writeln(heartbeat.deviceID); // 2
    writeln(heratbeat.statusID); // 3
}

The mixin template CanPacketHelperFunctions can be used over and over for all sorts of packet representations, and since it is only declared once, the implementation remains consistent across all types that use it.

Interfacing to C
I frequently must communicate with my microcontroller’s HAL and RTOS; D’s C interface made that a breeze. Just add an extern(C) and it’s good to go.

extern(C) c_setPwm(int solenoid, void* userData); // declaration
c_setPwm(4, null); // usage, pretty easy!

Unit Testing
D’s built-in unit testing has saved me from blowing my foot off a few times. I can run all my unit tests on Windows to guarantee logical correctness, and then build a final target for my microcontroller. Here is an example:

struct MyStruct
{
    int x;
    int squareIt() { return x * x; }
}

unittest
{
    MyStruct s;
    s.x = 9;
    assert(s.squareIt == 9 * 9); // If for some reason the implementation breaks, then the unit test fails
}

Deprecation
Codebases can be ever-changing, and sometimes certain functionality may no longer be considered good practice but must be retained for legacy reasons. D provides a way to explicitly mark this in code. Any use of such deprecated code will trigger a deprecation warning from the compiler. Example:

deprecated struct Example
{
    // ...
}

// This deprecation includes a reason
deprecated("This is the reason for deprecation..") struct ExampleWithMessage
{
    // ...
}

void main()
{
    // This will generate the following compiler warning:
    // "Deprecation: struct `Example` is deprecated"
    Example e;

    // This will generate the following compiler warning:
    // "Deprecation: struct `ExampleWithMessage` is deprecated - This is the reason for deprecation.."
    ExampleWithMessage ewm;
}

Of course, deprecated can be applied to all sorts of things, not just structs.

Portability
Following on from above, D supports a surprisingly large number of targets via GDC and LDC. If it weren’t for D’s portability, I would have had to write my project in C++ (ugh). I use LDC, and cross-compiling can be performed by simply adjusting my command line arguments.

Shared
shared is D’s way of guarding against multi-threaded access of code. It’s not perfect, but I use it as-is, and I think it works well. I do have multiple threads in my codebase, and they need to synchronize data. I mark certain variables as shared, which means I must take special care accessing that data. It works with system locks and mutexes. While locked, I can cast shared away and use it like a normal variable. This is handy with structs and classes.

shared int sensorValue;
sensorValue = 4; // using it like a single-thread variable, error
atomicStore(sensorValue, 4); // works with atomics

SafeD
@safe exists to prohibit sketchy memory activities and enforce best behavior. I haven’t had to fight @safe much yet because I don’t do anything wicked with my memory, but it is comfortable knowing that if I am going to make a mistake, the compiler can assist me in stopping it.

Mental Friction
Adam D. Ruppe puts it succinctly: D has low mental friction. The flexibility and expressiveness of the language make it easy to translate one’s thoughts into written code and maintain productivity. I don’t have to fight D much. This is my personal opinion, but I feel like D is the language in which I’m most productive.

Final Thoughts

D is the perfect fit for this sort of project— I think it’s going to have a bright future ahead in the embedded world. I’m going to continue using D for my projects. I’ve got another D-powered automotive project in the works which I hope to show off in the future. Even if D isn’t yet suitable for your project, keep an eye on it. D has been making enormous strides in the past few years, especially in regards to memory safety.

The examples shown in this article are purely meant to demonstrate how D’s features can be used in the real world. Do not take them as gospel as to how you should program.

D-lighted, I’m Sure

For me, finding D is the most recent step along a road winding back at least a dozen years. I’d been searching for a cross-platform development language/environment (POSIX and Windows, but not so much mobile since my search began before mobile was really a thing) and at this point, D fits better than anything else I’ve tried. I won’t go out on a limb and say it’s the Holy Grail of X-Plat, but at the very least, it’s brought some fun back into coding for me. And whenever I massage a hunk of code until it finally works… well, I’m addicted to those little victories.

The Road to D

The first language I tackled back in the mid-oughts to meet this end was PHP. I’d been a web developer for a few years when I found out that PHP had a standalone desktop version. When I then stumbled across Andrei Zmievski’s PHP-GTK, lightbulbs went on and fireworks went off. For a while. The big drawback I found with PHP-GTK was that no PHP compiler could handle the GTK end. So after a few years of patiently hoping someone would tackle this while I wrote nearly 40 blog posts on its use, I started looking elsewhere.

Back in the 1990’s, I’d been steeped in Javascript and HTML, writing simple online apps for banking clients out of Vancouver, BC and later, Bancroft, Ontario. With such a background, when I ran across Electron a few years ago, it seemed like a good fit. I assumed learning it would be easy-peasy-lemon-squeezy, but a lot about the nature of web development languages had changed. To boot, several more languages, standards, and paradigms had been thrown into the mix, so to say the least, I found it all confusing and more than a little intimidating. What I really wanted was one language, an easy distribution system, and a GUI toolkit that didn’t necessitate balancing style sheets with front- and backend code as well as JSON files. And Electron, unfortunately, needs to drag Chromium along for the ride in a little metaphorical red wagon. It’s a solution, but not the one I was looking for.

Then last year, I stumbled onto D. I’d been hearing about it for years, but I hadn’t read much about it. I didn’t realize it was available across so many POSIX platforms. I also didn’t realize it embraced the OOP paradigm and so hadn’t given it much thought. I liked the subtle humor of D being next in line after B, C and A (which oddly enough came along later than B and C), but after a brief smile, I paid no more attention until last October.

When I finally took a good look, I realized that with its curly braces and OOP propensity, D runs right up my street. But before rolling up my sleeves, I made sure there was a GUI toolkit I could use, something that didn’t necessitate balancing three differing paradigms at once. I found a list of GUI toolkits on the D Wiki and was gratified to see GtkD among them. So for the last two plus months, I’ve been putting most of my effort into learning how D and GtkD work together.

Perspiration and Refreshments

It may or may not be important to know this, but I don’t have a CS degree. I’m completely self-taught, a process that started while stuck in the middle of a frozen nowhere for three weeks more than 30 years ago. But that’s a whole other story. My point is, there are holes in my education. That’s what happens when you follow your nose instead of a syllabus.

Because of that, some of the intricacies of D elude me and may always do so. Although I’ve read Ali Çehreli’s chapter on the subject (from Programming in D) I have no idea what mixins are or why I’d want to use them. And templates seem like a good idea, but I don’t know why. I blame my lack of formal CS education for this, but I’m quite comfortable with classes and objects, so I’m not sweating it.

I was first introduced to OOP and the Gang of Four when I was learning PHP, so D covers familiar territory in that regard. Curly braces are another thing I feel quite at home with, having used them for most of my programming life.

But I’m finding the familiarity of D to be a bit of a stumbling block as well. It’s just different enough from C and PHP to mean I have to work hard at pounding those differences into my brain. I deal with it through rote typing of example code. I figure if I copy out enough D code instead of lazing along with copy-n-paste, eventually I’ll push C and PHP far enough into the background that I can see past them. And I keep Mr. Çehreli’s book handy so I don’t go completely off the rails. It’s been quite helpful. And speaking of helpful…

The Ecology of D

Forums

So far, I’ve signed up for two D-oriented forums, one at dlang.org and the other at gtkd.org. I have yet to find an unfriendly avatar. Everyone I’ve encountered seems willing to jump in and help. To contrast these forums with one I’ve been active on for a few years (not mentioning names, but this other forum is related to art and graphics, not programming) on the D forums I haven’t been insulted, nor have I been questioned for looking at things from a (warning: film term) Dutch angle—which is one of the things I do best. That comes from my art background, I suspect. I was quite the rabble rouser in art college. (Just ask Alan Wood the installation artist about our knock-down-drag-out shouting match in the cafeteria of Emily Carr College if you don’t believe me. But again, that’s a whole other story.)

On dlang.org, I mostly hang in the Learn sub-forum, which I suppose is only natural at this stage. It’s probably the most polite bunch I’ve run across on a forum ever, and I’ve been frequenting forums since the hoary days of the BBS when 1200 BAUD was lauded as the fastest thing since the 427 hemi.

Mike Wey seems quite patient for someone who answers more-or-less the same question over and over again on the GtkD forum. The only negative I found with that forum was technical. I signed up, made some posts, and when I went back to sign in a second time, I had to reregister. But I was still identified as the same Ron Tarrant who signed up the first time (I think) so perhaps that’s how things are supposed to be. It’s unusual, but it works.

I will also mention that the GtkD documentation pages are a mind-bender, but because this is where I’m planning to spend a lot of time over the next while, I’ve decided to pitch in and help make things more accessible.

I’m drawing on experiences writing my PHP-GTK blog back in 2006 and porting a bunch of code examples and tutorials into GtkD. I went so far as to buy a domain name (gtkDcoding.com) in preparation for launching a site and blog covering all this. That’s how I deal with learning curves and have done since I wrote that series of BASIC tutorials for my sister while freezing in a Newfoundland outport back in 1985.

Tools

I got up and running with dmd quite quickly. Installation on Windows 10 and FreeBSD was straightforward. A few quick questions on the D forum, and I had everything I needed to do single- or multi-file projects. A few more questions answered on the GtkD forum and I was comfortable enough to start porting my PHP-GTK code.

But I have to say, DUB eludes me. I don’t know if it’s because Electron left me feeling like JSON files are a disgruntled engineer’s revenge plot or if it’s just the way my brain works. Perhaps if I put my mind to it, I could learn it, but since I’m getting the results I want from plain ole dmd, I’ve done no more than skim DUB’s docs up until now. Just for the record, back in my C days, make files were a caution for me, too. I eventually licked them, and if it ever becomes important enough to me to figure out DUB, I guess I will. But for now, my heart’s not in it.

Finding a text editor that supports D (and especially GtkD) syntax highlighting rather than—as a few people on the forum stated—supporting C and getting ‘good enough’ support for D, led me to abandon the search and roll my own. So far, I’ve done more-or-less complete highlighters for PSPad and CodeBlocks. To be fair, they both support D out of the box, but not GtkD which is important to me, mostly as an aid to remembering module names.

I haven’t even looked at other tools. I’m dimly aware of some other sub-forums for what look like other tools, but to be honest I haven’t read them. As I said, dmd does the job, so I’m satisfied for now.

Conclusion

And that’s a quick summary of my first two months with D. On the one hand, I don’t really understand some of D’s paradigms, but on the other, the ones I do relate to are meeting my expectations. The only two things I haven’t found yet are:

  • information about how I would go about packaging a D app (with GtkD) for distribution, and
  • how to build on one platform for distribution on another (if that’s even possible).

I’m willing to put in some time on this and eventually get gtkDcoding.com off the ground as my way of giving back. It’s been a long time since those frozen weeks with a Coleco Adam when I tried to explain BASIC to my thirteen-year-old sister (who is now a paramedic and doesn’t care any more). And I must say, I’m as excited now about D and GtkD as I was about BASIC and general computing way back then. Being retired, I have the time to pursue it and I’m looking forward to becoming a regular member of the D community.

And in case you don’t know what a Dutch angle is, I urge you to watch some episodes of Batman from the 1960’s. See how the camera tilts when things go wrong for our heroes? That’s a Dutch angle.

Bio

I was inspired to learn programming while vacationing in a frozen Newfoundland outport in April of 1985. It started as a desperate attempt to keep from stripping down to my shorts and disappearing into a blizzard, but became a lifelong passion along with acting, writing, and music. In keeping with this non-typical start, it was in art college where I learned my first serious programming language (6502 assembly) and later it was on a job as a technical writer and artist that I wrote my first serious code for a client, an online mortgage calculator for a credit union in British Columbia. The culmination of my programming career was finishing the PHP-GTK app, Corkboard, a writer’s tool for story planning. (Don’t bother looking. I couldn’t come up with a distribution system, so it languishes here on a back-up drive.)

Since dropping out of high school in 1972, I’ve made a living as a taxi driver, musician, screenwriter, technical writer, artist, sound reinforcement equipment salesman, and biology lab technician among other things. I’ve also made money acting, programming, and promoting concerts. I retired from Statistics Canada in 2010 and have since divided my time between acting, writing lame novels, pursuing the elusive X-Plat beast, and keeping house for my wife of 33 years.

How an Engineering Company Chose to Migrate to D

Bastiaan Veelo is the lead developer of a specialised program for the
computer aided geometric design of ship hulls called Fairway, for the
company SARC in the Netherlands.


Imagine there is this little-known programming language in which you enjoy programming in your free time. You know it is ready for prime time and you dream about using it at work everyday. This is the story about how I made a dream like that come true.

My early acquaintance with D

Back when “google” was not yet a common verb, I was doing a web search for “parsing C++”. The reason was that writing a report for an assignment had derailed into writing a syntax highlighter for noweb using bison and flex, and I found out firsthand that C++ is not easy to parse. That web search brought up this page (present version) with an overview of the D Programming Language, and the following statement has had me hooked ever since:

D’s lexical analyzer and parser are totally independent of each other and of the semantic analyzer. This means it is easy to write simple tools to manipulate D source perfectly without having to build a full compiler. It also means that source code can be transmitted in tokenized form for specialized applications.

“Genius,” I thought, “here we have someone who knows what he’s doing.” This is representative of the pragmatic professionalism that still radiates from the D community, and it combines with an unpretentious flair that makes it pleasant to be around. This funny quote decorated its homepage for many years:

“Great, just what I need.. another D in programming.” – Segfault

Nevertheless, I didn’t have many opportunities to use the language and I largely remained sitting on the fence, observing its development.

Programming professionally

With mostly academic programming experience, I started programming professionally in 2006 for SARC, a Dutch engineering company serving the maritime industry. Since the early ’80s they have been developing software for ship design and onboard loading calculations, which today amounts to roughly half a million lines of code. I think their success can partly be attributed to their choice of programming language: Extended Pascal (the ISO 10206 standard, not one of the many proprietary extensions of Pascal).

Extended Pascal was a great improvement over ISO 7185 Pascal. Its compiler, by Prospero Software from England, was fast and well documented. The language is small enough and its syntax appropriately verbose to make engineering professionals quickly productive in programming. Personally though, I spent most of my time programming in C++, modernizing their system for computer aided design of ship hulls using Qt and Coin3D.

When your company outlives a programming language

Although selecting an ISO standard in favor of a proprietary Pascal dialect seemed wise at the time, it is apparent now that the company has outlived the language. Prospero Development Software Ltd was officially dissolved 15 years ago. Still, its former director, Tony Hetherington, continued giving support many years after, but he’d be close to 86 years old now and can no longer be reached. Its website is gone, last archived in 2013. There’s GNU Pascal, which also supports ISO 10206, but that project has stopped moving and long ago lost synchrony with gcc. Although there is no immediate crisis, it is clear that something needs to happen sometime if the company wants to continue its activities in the coming decades.

Changing the odds

A couple of years ago, I secretly started playing with the fantasy of replacing Extended Pascal with D. Even though D’s syntax is somewhat different from Pascal, it shares at least four important similarities: support for nested functions, boundary checking, modules, and compilation speed. In addition, it has many traits that make the language attractive to engineers: good focus on performance and numerics, garbage collection, dynamic arrays, easy parallelization, understandable templates, contract programming, memory safety, unit tests, and even wysiwyg strings and formatted numerals. D’s language features encourage experimentation, which resonates well with engineers.

So I wondered what I could do to highlight D’s significance to my employer and show it’s an attractive language to switch to. I thought I could make a compelling case if I could write a parser in D that would take Extended Pascal source and transpile it to D source. At least I would have fun trying!

So I went over to code.dlang.org to see if there were any D alternatives to flex and bison. There, I found Pegged, and instantly the fun began. Pegged combines the functionality of flex and bison in one incredibly easy to use package, for which its creator Philippe Sigaud obviously enjoyed writing excellent documentation. Nowadays, Pegged is part of the D language tour and you can try it out on-line without having to install a thing. The beauty is that the grammar from the Extended Pascal language specification maps almost linearly to the PEG from which Pegged generates the parser. For this it makes heavy use of D’s generic programming capabilities and compile-time function evaluation — it can generate a parser at compile time if you want it to!

However, it wasn’t smooth sailing all along. As I was testing D, I suddenly found myself being tested as well. I learned the hard way that there is a phenomenon called left-recursion, from which a PEG parser typically cannot break out of. And the Extended Pascal grammar is left-recursive in several ways. Consequently, I spent many evenings and weekends researching parsing theory, until eventually I managed to extend Pegged with support for all kinds of left-recursion! From one thing came another, and I added longest match alternation, case insensitive literals, the toHTML() method for dynamically browsing the syntax tree, and a tracer for logging the parsing process.

Obviously, I was having fun. But more importantly, I was demonstrating that the D programming language is accessible enough that a naval architect can understand other people’s code and expand it in non-trivial ways. The icing on the cake came when I was asked to present my experiences at DConf 2017 in Berlin, which you can watch here (and here’s the extra bit I presented at lunch time for the livestream audience).

At this time, I was able to automatically translate the following trivial example:

program hello(output);

begin
    writeln('Hello D''s "World"!');
end.

into D:

import std.stdio;

// Program name: hello
void main(string[] args)
{
    writeln("Hello D's \"World\"!");
}

Language competition

Having come this far, the founder of SARC agreed that it was time to investigate the merits of various alternative programming languages. We would do a thorough and objective comparison based on trial translations of a comprehensive set of language features. Due to the amount of manual labor that this requires, we had to drastically prune the space of programming languages in an initial review round. Note that what I am about to present does not declare which programming language is the best in our industry. What we are looking for is a language that allows an efficient transition from Extended Pascal without interrupting our business, and which enables us to take advantage of modern insights and tools.

In the initial review round we looked at general language characteristics. Here I’ll just highlight what fell through the sieve and why.

Performance is important to us, which is why we did not consider interpreted languages. C++ is in use for one component of our software, but that was written from the ground up. We feel that the options for translation are not favorable, that its long compile times are a serious hindrance to productivity, and that there are too many ways in which one can shoot one’s self in the foot. We cannot require our expert naval architects to also become experts in C++.

Nowadays, whenever D is publicly evaluated, the younger languages Go and Rust are often brought up as alternatives. Here, we need not go into an in-depth comparison of these languages because both Rust and Go lack one feature that we rely on heavily: nested functions with access to variables in their enclosing scope. Solutions for eliminating nested functions, like bringing them into global scope and passing extra variables, or breaking files up into smaller modules, we find unattractive because it would complicate automated translation, and we’d like to preserve the structure and style of our current code. GNU C does offer nested functions, but it is a non-standard extension and it has been predicted that many will move away from C due to its unsafe features. After this initial pruning, three languages remained on our shortlist: Free Pascal, Ada and D.

As a basis for our detailed comparison, we wrote fifteen small programs that each used a specific feature of Extended Pascal that is important in our current code base. We then translated those programs into each language on our shortlist. We kept a simple score board on how well these features were represented in each language: +1 if the feature is supported or can be implemented, 0 if the lack of the feature can be worked around, and -1 if it can’t. This is what came out of that evaluation:

Test Free Pascal Ada D
Arrays beginning at arbitrary indices +1 +1 +1
Sets 0 0 +1
Schema types 0 0 +1
Types with custom initial values -1 0 +1
Classes +1 +1 +1
Casts +1 +1 +1
Protection against use of dangling pointers -1 +1 +1
Thread safe memory [de]allocation +1 +1 +1
Calling into Windows API +1 +1 +1
Forwarding Windows callbacks to nested functions +1 +1 +1
Speed of calculations +1 +1 +1
Calling procedures written in assembly +1 0 +1
Calling procedures in a DLL +1 +1 +1
Binary compatibility of strings 0 +1 +1
Binary compatible file i/o -1 0 0
Score 6 10 14

So, Free Pascal is the only candidate with negative scores, Ada positions itself in the middle, and D achieves an almost perfect score. Not effortlessly, though; we’ll talk about some of the technical challenges later. Because Free Pascal is, like D, fully Open Source and written in itself, extending the language and filling in the gaps is theoretically possible. Although some of its deficiencies could certainly be resolved that way, others would be quite complicated and/or unlikely to be accepted upstream.

We also estimated the productivity of the languages. Free Pascal scored high because it is closest to what we are used to. Despite its dissimilar syntax, D scored high because of its expressiveness and flexibility. Ada scored lowest because of its rigidity and because of the extra work the programmer has to put in (most importantly casts and conversions). Ada is more verbose than Pascal which was disliked by some of us because it can somewhat obscure the essence of what a piece of code tries to express, and frequently the code became not only verbose but cryptic, which was unanimously disliked.

Third, we estimated the future prospects and the advantages each language could bring to the table. Although Free Pascal has a more active community than we expected it to have, we do not see great potential for growth. Ada is renowned for its support for writing reliable code (although it has no monopoly in that field) but it does come at a cost and requires real effort. D has a dynamic and open community, supports both script-like productivity and high performance, includes various features for writing reliable software (approaching Ada but at a much lower cost), and offers some unique advanced features with which wonders can be accomplished.

Finally, we estimated the effort of translation. Although Free Pascal is very similar to Extended Pascal, missing features pose a real problem and would require a high degree of manual translation and rewriting. Although p2ada exists, it only works partially in our case and does not fully support Extended Pascal. Because Ada frequently requires additional code (casting to the correct type, pulling in a package, instantiating a generic type, adding a pragma, splitting up Put_Lines etc.), writing or extending a reliable transpiler into Ada would be more difficult than doing the same into D.

Selecting a winner

I gave away the winner in the title, but we landed at that conclusion as follows. Ada was the first language to be dropped. We really felt that the extra work that the programmer has to put in is a brake on productivity and creativity. Although it barely played a role in our evaluation, illustrative is the difference between the Ada and D equivalents to the Expressive C++17 Challenge. The D solution is both concise and expressive, the Ada solution is hardly expressive and consists of more lines than I want to write or read. Also of secondary importance, but difficult to ignore, is the difference between the communities surrounding the languages, which in Ada’s case is AdaCore Support, who has no problems demanding annual five-figure subscription fees.

Although akin to our current language, Free Pascal was mainly dropped due to its porting challenges and our estimation that its potential is lower and its future outlook is less optimistic than that of D. If we were to choose Free Pascal, we would basically invest a lot of effort only to arrive at a technological solution that we felt would be of lower quality than we currently have.

And that’s were I saw a dream come true: A clap on the table by the company founder and it was decided to commit to the effort of bringing twenty-five years worth of Extended Pascal code to D!

What makes a difference

In short, my experience is that if a feature is not present in the language, D is powerful enough that the feature can be implemented in a library. Translating each sample program by hand has really helped to focus on replicating functionality, leaving the translation process for later concern. This has led to writing a compatibility library with types and functions that are vital for the conversion. Now that equivalents are known and the parser is done, I just have to implement code generation.

Below follows another example that currently translates automatically and executes identically. It iterates over a fixed length array running from 2 to 20 inclusive, fills it with values, prints the memory footprint and writes it to binary file:

program arraybase(input,output);

type t = array[2..20] of integer;
var a : t;
    n : integer;
    f : bindable file of t;

begin
  for n := 2 to 20 do
    a[n] := n;
  writeln('Size of t in bytes is ',sizeof(a):1); { 76 }
  if openwrite(f,'array.dat') then
    begin
      write(f,a);
      close(f);
    end;
end.

Transpiled to D (or should I say Dascal?) and post-processed by dfmt to fix up formatting:

import epcompat;
import std.stdio;

// Program name: arraybase
alias t = StaticArray!(int, 2, 20);

t a;
int n;
Bindable!t f;

void main(string[] args)
{
    for (n = 2; n <= 20; n++)
        a[n] = n;
    writeln("Size of t in bytes is ", a.sizeof); // 76
    if (openwrite(f, "array.dat"))
    {
        epcompat.write(f, a);
        close(f);
    }
}

Of course this is by no means idiomatic D, but the fact that it is recognizable and readable is nice, especially for my colleagues who will have to go through an unusual transition. By the way, did you notice that code comments are preserved?

One very-nice-to-have feature is binary file compatibility; In fact it may have been the killer feature, without which D might not have been so victorious. The case is that whenever a persistent data structure is extended in our software, we make sure that we can still read and convert that structure from its prior format. That way, if a client pulls out an old design from its archives and runs it through our current software, it will still work without the user even being aware that conversion occurs, possibly in multiple steps. Not having to give up that ability is very attractive.

But it wasn’t easy to get there. The main difficulty is the difference in how strings are represented in D and the Prospero implementation of Extended Pascal, in memory and on file. This presented the challenge of how to preserve binary compatibility in file I/O with data structures that contain string members.

Strings

In Prospero Extended Pascal, strings are implemented as a schema type, which is a parameterized type that can be used in the following ways:

type string80 = string(80);
var str1 : string80;
    str2 : string(60);
procedure foo(s : string);

This defines string80 to be an alias for a string type discriminated to have a capacity of 80 characters. Discriminated string variables, like str1 and str2, can be passed to functions and procedures that take undiscriminated strings as arguments, like foo, which thereby work on strings of any capacity. In memory, str1 is laid out as a sequence of 80 chars, followed by a ushort that encodes the length of the string. I say encodes because a shorter string is padded with \0s up to the capacity and the ushort actually contains the length of that padding. This way, when a pointer to the string is passed to a C function and the contents of the string occupy its full capacity, the 0 in the padding length doubles as the terminating \0 of the C string.

My first thought was to mimic this data representation with a D template. But that would require procedures like foo to be turned into templates as well, which would escalate horribly into template bloat, a problem with multiple string arguments and argument ordering, and would complicate translation. Besides, schema types can also be discriminated at run time, which does not translate to a template.

Could some sort of inheritance scheme be the solution? Not really, because instances of D classes live on the heap, so a string embedded in a struct would just be a pointer instead of the char array and ushort.

But binary layout is actually only relevant in files, and in a stroke of insight I realized that this must be why user-defined attributes, or UDAs, exist. If I annotate the string with the correct capacity for file I/O, then I can just use native D strings everywhere, which genuinely must be the best possible translation and solves the function argument issue. Annotation can be done with an instance of a struct like

struct EPString
{
    ushort capacity;
}

The above Pascal snippet then translates to D like so:

@EPString(80) struct string80 { string _; alias _ this; }
string80 str1;
@EPString(60) string str2;
void foo(string s);

Notice how the string80 alias is translated into the slightly convoluted struct instead of a normal D alias, which would have looked like

@EPString(80) alias string80 = string;
</code>

Although that compiles, there is no way to retrieve the UDA in that case because plain alias does not introduce a symbol. Then hasUDA!(typeof(str1), EPString) would have been equivalent to hasUDA!(string, EPString) which evaluates to false. By using the struct, string80 is a symbol so typeof(str1) gives string80, and hasUDA!(string80, EPString) evaluates to true in this example.

There is one side effect that we will have to learn to accept, and that is that taking a slice of a string does not produce the same result in D as it does in Extended Pascal. That is because string indices start at 1 in Extended Pascal and at 0 in D. My strategy is to eliminate slices from the source and replace them with a call to the standard substr function, which I can implement with index correction. Finding all string slices can be accomplished with a switch in the transpiler that makes it insert a static if to test if the slice is being taken on a string, and abort compilation if it is. (Arrays are transpiled into a custom array type that handles slices and indices compatibly with Extended Pascal.)

Binary compatible file I/O

Now, to write structs to file and handle any embedded @EPString()-annotated strings specially, we can use compile-time introspection in an overload to toFile that acts on structs as shown below. I have left out handling of aliased strings for clarity, as well as shortstring, which is a legacy string type with yet a different binary format.

void toFile(S)(S s, File f) if (is(S == struct))
{
    import std.traits;
    static if (!hasIndirections!S)
        f.lockingBinaryWriter.put(s);
    else
        // TODO unions
        foreach(field; FieldNameTuple!S)
        {
            // If the member has itself a toFile method, call it.
            static if (hasMember!(typeof(__traits(getMember, s, field)), "toFile") &&
                       __traits(compiles, __traits(getMember, s, field).toFile(f)))
                __traits(getMember, s, field).toFile(f);
            // If the member is a struct, recurse.
            else static if (is(typeof(__traits(getMember, s, field)) == struct))
                toFile(__traits(getMember, s, field), f);
            // Treat strings specially.
            else static if (is(typeof(__traits(getMember, s, field)) == string))
            {
                // Look for a UDA on the member string.
                static if (hasUDA!(__traits(getMember, s, field), EPString))
                {
                    enum capacity = getUDAs!(__traits(getMember, s, field), EPString)[0].capacity;
                    static assert(capacity > 0);
                    writeAsEPString(__traits(getMember, s, field), capacity, f);
                }
                else static assert(false, `Need an @EPString(n) in front of ` ~ fullyQualifiedName!S ~ `.` ~ field );
            }
            // Just write other data members.
            else static if(!isFunction!(__traits(getMember, s, field)))
                f.lockingBinaryWriter.put(__traits(getMember, s, field));
        }
}

At the time of writing, I still have work to do for unions, which are used in the translation of variant records (including considering the use of one of the seven existing library solutions 1, 2, 3, 4, 5, 6, 7).

Currently, detecting unions is a bit involved . Also, there is a complication in the determination of the size of a union when the largest variant contains strings: the D version of that variant may not be the largest because D strings are just slices. I’ll probably work around this by adding a dummy variant that is a fixed size array of bytes to force the size of the union to be compatible with Extended Pascal. This is the reason why D scored a mere 0 in file format compatibility. It is amazing what D allows you to do though, so I may be able to do all of that automatically and award D a perfect score retroactively. On the other hand, it is probably easiest to just add the dummy variant in the Pascal source at the few places where it matters and be done with it.

The way forward

Obviously, this is long term planning. It has taken years to grow into D; it will possibly take a year, and probably longer, to migrate to D. Unless others turn up who are in the same boat as us (please contribute!) it’ll be me who has to row this ship to D-land and I still have my regular duties to attend to. My colleagues will continue to develop in Extended Pascal as usual, and once my transpiler is able to translate all or almost all of it, we will make the switch to D overnight. From then on, we’ll be in it for the long run. We trust to be with D and D to be with us for decades to come!

User Stories: Funkwerk

The deadline for the early-bird registration for DConf 2018 in Munich is coming up on March 17th. The price will go up from $340 to $400. If you’d like to go, hurry and sign up to save yourself $60. And remember, the NH Munich Messe hotel, the conference venue, is offering a special deal on single rooms plus breakfast for attendees.


A few of the DConf attendees are coming from a local company called Funkwerk. They’re a D shop that we’ve highlighted here on this blog in a series of posts about their projects (you’ll see one of their products in action if you take the subway or local train service in Munich).

In this post, we cap off the Funkwerk series with the launch of a new feature we creatively call “User Stories”. Now and again, we’ll publish a post in which D users talk of their experiences with D, not about specific projects, but about the language itself. They’ll tell of things like their favorite features, why they use it, how it has changed the way they write code, or anything they’d like to say that expresses how they feel about programming in D.

For this inaugural post, we’ve got three programmers from Funkwerk. First up, Michael Schnelle talks about the power of ranges. Next, Ronny Spiegel tells why generated code is better code. Finally, Stefan Rohe enlightens us on Funkwerk’s community outreach.

The power of ranges

Michael Schnelle has been working as a software developer for about 5 years. Before starting with D 3 years ago, he worked in (Web)Application Development, mostly with Java, Ruby on Rails, and C++, and did Thread Modeling for Applications. He enjoys coding in D and likes how it helps programmers write clean code.

In my experience, no matter what I am programming, I always end up applying functions to a set of data and filter this set of data. Occasionally I also execute something with side effects in between. Let’s look at a simplified use case: the transformation of a given set of data and filtering for a condition afterwards. I could simply write:

foreach(element; elements) {
  auto transformed = transform(element);
  if (metCondition(transformed) {
     results ~= transformed
  } 
}

Using the power from std.algorithm, I can instead write:

filter!(element => metCondition(element))
       (map!(element => transform(element))(elements));

At this point, we have a mixture of functional and object-oriented code, which is quite nice, but still not quite as readable or easy to understand as it could be. Let’s combine it with UFCS (Uniform Function Call Syntax):

elements.map!(element => element.transform)
        .filter!(element => element.metCondition);

I really like this kind of code, because it is clearly self-explanatory. The foreach loop, on the other hand, only tells me how it is being done. If I look through our code at Funkwerk, it is almost impossible to find traditional loops.

But this only takes you one step further. In many cases, there happen to be side effects which need to be executed during the workflow of the program. For this kind of thing, the library provides functions like std.range.tee. Let’s say I want to execute something external with the transformed value before filtering:

elements
  .map!(element => element.transform)
  .tee!(element => operation(element))
  .filter!(element => element.metCondition)
  .array;

It is crucial that operations with side effects are only executed with higher-order functions that are built for that purpose.

int square(int a) { writefln("square value"); return a*a; }

[4, 5, 8]
  .map!(a => square(a))
  .tee!(a => writeln(a))
  .array;

The code above would print out the square value six times, because tee calls range.front twice. It is possible to avoid this by using functions like std.algorithm.iteration.cache, but in my opinion, the nice way would be to avoid side effects in functions that are not meant for that.

In the end, D gives you the possibility to combine the advantages of object-oriented and functional programming, resulting in more readable and maintainable code.

Generated code is better code

Ronny Spiegel has worked as a professional software developer for almost 20 years. He started out using C and C++, but when he joined Funkwerk he really started to love the D language and the tools it provides to introspect code and to automate things at compile time.

In a previous blog post, I gave a short overview of the evolution of the accessors library. As you might imagine, I really like the idea of using the compiler to generate code; in the end this usually results in less work for me and, as a direct result, causes fewer errors.

The establishment of coding guidelines is crucial for a team in order to create maintainable software, and so we have them here at Funkwerk. There is a rule that every value object (or entity) has to implement the toString method in order to provide diagnostic output. The provided string shall be unambiguous so that it’s more like Python’s __repr__ than __str__.

Example:

StationMessage(GeneralMessage(4711, 2017-12-12T10:00:00Z), station="BAR", …)

The generated string should follow some conventions:

  • provide a way to uniquely reconstruct data from a string
    • start with the class name
    • continue with any potential superclasses
    • list all fields providing their name and value separated by a comma
  • be compact but still human readable (for developers)
    • skip the name where it matches the type (e.g. a field of type SysTime is called time)
    • skip the name if the field is called id (usually there’s an IdType used for type safety)
    • there’s some special output format defined for types like Date and SysTime
    • Nullable!T’s will be skipped if null etc.

To format output in a consistent manner, we implemented a SinkWriter wrapping formattedWrite in a way that follows the listed conventions. If this SinkWriter is used everywhere, this is the first step to fully generate the toString method.

Unfortunately that’s not enough; it’s very common to forget something when adding a new field to a class. Today I stumbled across some code where a field was missing in the diagnostics output and that led to some confusion.

Using (template) mixins together with CTFE (Compile Time Function Execution) and the provided type traits, D provides a powerful toolset which enables us to generate such functions automatically.

We usually implement an alternative toString method which uses a sink delegate as described in https://wiki.dlang.org/Defining_custom_print_format_specifiers. The implementation is a no-brainer and looks like this:

public void toString(scope void delegate(const(char)[]) sink) const
{
    alias MySelf = Unqual!(typeof(this));

    sink(MySelf.stringof);
    sink("(");

    with (SinkWriter(sink))
    {
        write("%s", this.id_);
        write("station=%s", this.station_);
        // ...
    }

    sink(")");
}

This code seems to be so easy that it might be generalized like this:

public void toString(scope void delegate(const(char)[]) sink) const
{
    import std.traits : FieldNameTuple, Unqual;

    alias MySelf = Unqual!(typeof(this));

    sink(MySelf.stringof);
    sink("(");

    with (SinkWriter(sink))
    {
        static foreach (fieldName; FieldNameTuple!MySelf)
        {{
            mixin("const value = this." ~ fieldName ~ ";");
            write!"%s=%s"(fieldName, value);
        }}
    }

    sink(")");
}

The above is just a rough sketch of how such a generic function might look. For a class to use this generation approach, simply call something like

mixin(GenerateToString);

inside the class declaration, and that’s it. Never again will a field be missing in the class’s toString output.

Generating the toString method automatically might also help us to switch from the common toString method to an alternative implementation. If there will be more conventions over time, we will only have to extend the SinkWriter and/or the toString-template, and that’s it.

As a summary: Try to generate code if possible – it is less error prone and D supports you with a great set of tools!

Funkwerk and the D-Community

Stefan Rohe started the D-train at Funkwerk back in 2008. They have loved DLang since then and replaced D1-Tango with D2-Phobos in 2013. They are strong believers in open source and local communities, and are thrilled to see you all in Munich at DConf 2018.

Funkwerk is the largest D shop in south Germany, so we hire D-velopers, mainly just through being known for programming in D. In order to give a little bit back to the D community at large and help the local community grow, Funkwerk hosted the foundational edition of the Munich D Meetup.

The local community is important …

Munich Meetup at Brainlab

The meetup was founded in August 2016, 8 years after the first line of D code at Funkwerk was written. Since then, the Meetup has grown steadily to ~350 members. At that number, it is still not the biggest D Meetup, but it is the most visited and the most active. It provides a chance for locals in Munich to interact with like-minded D-interested people each month. And with an alternating level of detail and a different location each month, it stays interesting and attracts different participants.

… and so is the global community

To engage with the global community, Funkwerk is willing to open source some of its general-purpose D libraries. They can all be found under github.com/funkwerk, and some are registered in the DUB registry.

To mention are:

  • accessors – a library to auto generate getters and setters with UDAs
  • depend – a tool that checks actual import dependencies against a UML model of target dependencies
  • d2uml – reverse engineering of D source code into PlantUML class outlines

Feel free to use these and let us know how you like them.