# Symmetry Investments and the D Language Foundation are Hiring

The D Language Foundation is hiring! Thanks to generous funding from Symmetry Investments, we are looking to fill two (mostly) non-programming positions geared toward improving the D ecosystem. Symmetry is also offering a bounty for a specific improvement to DUB, the D build tool and package manager. And on top of all of that, they are hiring D programmers.

## D Pull Request/Issue Manager

A lot of good work goes into the D Programming Language GitHub repositories. Unfortunately, some of that good work sometimes gets left behind. A similar story can be told for our Bugzilla database, where some issues are fixed almost as soon as they’re reported and others fall victim to a lack of attention. Efforts have been made in the past to tidy things up, but without someone in a position to permanently keep at it, it’s a task that is never complete.

The D Language Foundation is looking for one or two motivated individuals to take on that permanent position, get the work done, and keep things running smoothly. Symmetry Investements is generously funding this role with $50,000 per year for one person, or$25,000 per year for each of two.

The ideal candidate is someone who:

• is familiar with git, GitHub, and Bugzilla;
• is familiar enough with D to be able to review simple pull requests;
• is able to recognize when more specialized reviews are required and
• is able to proofread English text (for reviewing documentation and web site pull requests).

Examples of the role’s responsibilities include:

• ensuring all pull requests follow procedure;
• reviewing simple pull requests;
• finding appropriate reviewers for more complex pull requests;
• ensuring that pull requests are reviewed in a timely manner;
• reviving stale pull requests;
• coordinating between pull request submitters and reviewers to prevent pull requests from going stale;
• closing pull requests that are no longer valid;
• identifying Bugzilla issues that are duplicates or invalid;
• identifying Bugzilla issues that are candidates for bounties;
• publicizing Bugzilla issues in need of a champion and
• other related tasks.

We are hoping to hire from within the D community, though we will accept queries from anyone. If you are interested in taking on the role, please send your resume to social@dlang.org. You should also indicate if you are willing to do the job full time (just you) or part time (share the responsibilities with someone else).

I’ve been working with the D Language Foundation for the past three years. Much of what I do falls loosely in the category of Community Relations. These days, I’m in need of an assistant. Symmetry Investments is providing $600 per month for the role. The job will involve a number of different activities as the need arises, such as: • seeking out guest authors and projects to highlight for the D Blog; • monitoring our social media accounts; • sending out messages from the D Language Foundation (such as thank you notes to new donors); • assisting with maintenance of pages at dlang.org and dconf.org; • assisting with the organization of events like DConf and SAOC and • any odd jobs that pop up now and again. If you have good communication skills, an optimistic disposition, and enthusiasm for the D Programming Language, I’d like to talk to you. I don’t need a resume. Instead, please send an email to social@dlang.org explaining why you’re the right person for the job. ## DUB Bounty DUB has become a critical component in the D ecosystem. A significant number of projects depend on it and we need it to be able to meet a wide range of project needs. To that end, there are certainly improvements to be made. One such is in how DUB determines which of a project’s source files are in need of recompilation. Currently, DUB follows in the tradition of the venerable make and uses timestamp comparisons to make that determination. A new generation of version control and build tools (git, buck, bazel, scons, waf, plz, and more) rely on file checksums to assess the need for action. This is a much more robust approach because it detects actual changes in file content. Timestamps can change in any number of irrelevant ways. Robustness is important if one is to depend on a build working properly even when files are moved, copied, and shared across people, machines, and teams. As hashes are fast to compute on modern hardware, the impact on speed is very low. Symmetry Investments is offering a$2,000 bounty to the programmer who either converts DUB’s use of timestamp-dependent builds to use SHA-1 hashing throughout, or implements it as a global option to preserve the current behavior.

For inspiration, see this clip from Linus Torvald’s Google talk, and the article Build-Systems Should Use Hashes Over Timestamps. Note that shasum $(git ls-files) in Phobos takes 0.05 seconds on a warm SSD drive in a desktop machine. Anyone interested in taking on this bounty should contact social@dlang.org beforehand. Anyone interested in contributing to the bounty amount can do so via the bounty card Support for Hash-Based Recompilation in DUB at our Task Bounties page. # Saving Money by Switching from PHP to D 2night was born in 2000 as an online magazine focused on nightlife and restaurants in Italy. Over the years, we have evolved into a full-blown experiential marketing agency, keeping up our vocation of spreading what’s cool to do when you go out, but specialized in producing brand events and below-the-line unconventional marketing campaigns. We started using D at 2night in 2012 when we developed a webservice used by our Android and iOS apps. It has worked fine since then, but it was just a small experiment. In 2019, after many other experiments, we decided to take the big step: we switched the complete website from PHP to D. The time was right; we had been planning to give our website a new look and we took this opportunity to rewrite the entire infrastructure. ### Development The job turned out to be easier than we had imagined. We implemented a small D backend over our Mongo database in a few hundred lines. We created a Simple Common Gateway Interface (SCGI) library to interface with the NGINX server and another library to work with the DOM. Using the HTML DOM instead of an obscure HTML template language helped us speed up development a lot. In this way, someone who works on HTML or JavaScript is not required to know D or any template language and can deploy plain HTML and CSS files. On the other hand, someone who works on the backend does not care so much about HTML tags since he can simply access elements by ID, class, etc.; if some HTML tags are moved around the page the whole thing still works. HTML+CSS+JavaScript on the frontend and D on the backend are totally independent. Writing code in this way is quite simple. Let’s say we want to build a blog page. We start from a simple HTML file like this: <!DOCTYPE html> <html lang="en"> <head><title>Test page</title></head> <body> <!-- Main post --> <h1>Post title</h1> <h2>The optional subheading</h2> <p> Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin a velit tempus, eleifend ex non, aliquam ipsum. Nullam molestie enim leo, viverra finibus diam faucibus a. Ut dapibus a orci in eleifend. </p> <!-- Two more posts --> <div id="others"> <h3>Other posts</h3> <div> <h4>Post#2</h4> <p> Morbi tempus pretium metus, et aliquet dolor. Duis venenatis convallis nisi, auctor elementum augue rutrum in. Quisque euismod vestibulum velit id pharetra. Morbi hendrerit faucibus sem, ac tristique libero... </p> </div> <div> <h4>Post #3</h4> <p>Sed sit amet vehicula nisl. Nulla in mi est. Vivamus mollis purus eu magna ullamcorper, eget posuere metus sodales. Vestibulum ipsum ligula, vehicula sit amet libero at, elementum vestibulum mi. </p> </div> </div> </body> </html>  This is a valid HTML5 file that can be edited by anyone who knows HTML. Now we have to fill this template with real data from a database, which we can represent as an array in this example for the sake of simplicity: // A blog post struct SimplePost { string heading; string subheading; string text; string uri; } SimplePost[] posts = [ SimplePost("D is awesome!", "This is a real subheading", "Original content was replaced", "http://dlang.org"), SimplePost("Example post #1", "Example subheading #1", "Random text #1"), SimplePost("Example post #2", "Example subheading #2", "Random text #2"), SimplePost("Example post #3", "Example subheading #3", "This will never be shown") ]; First, we must read our HTML template just as it is and parse it using our html5 library:  auto page = readText("html/test.html"); // Parse the source auto dom = parser.parse(page); Then we replace the content of the main article with data from the first element of our array. We use the tag name in order to select the correct HTML element:  // Take the first element from our data source auto mainPost = posts.front; // Update rendered data of main post dom.byTagName("h1").front.innerText = mainPost.heading; dom.byTagName("p").front.innerText = mainPost.text; dom.byTagName("a").front["href"] = mainPost.uri; We want to check if our article has a subtitle. If it doesn’t we’re going to remove the related tag.  // If we have a subtitle we show it. If not, we remove the node from our page if (mainPost.subheading.empty) dom.byTagName("h2").front.detach(); else dom.byTagName("h2").front.innerText = mainPost.subheading; If you wanted to get the same result with a template language, you’d probably need to mess up the HTML with something like this: <!-- We don't like this! --> <? if(!post.subheading.isEmpty) ?> <h2><?= post.subheading ?></h2> <? endif ?>  This mixes logic inside the view and it disrupts the whole HTML file. Anyone who works on the HTML frontend is supposed to know what post is, the logic behind this object, and the template language itself. Last but not least, many HTML editors would probably be driven crazy by any custom syntax. And this is still a simple case! Going back to our example, to fill the last part of our page we must get the container from the DOM. All we need is to perform a search by ID on the DOM: auto container = dom.byId("others").front; Now we use the first element inside the container as a template. So we clone it and we empty the container itself:  // Use the first children as template auto containerItems = container.byCssSelector(div[id="others"] > div); auto otherPostTemplate = containerItems.front.clone(); // Remove all existing children from container containerItems.each!(item => item.detach); Finally we add a new child to the container for each post in our data source:  // Take 2 more posts from list. We drop the first, it's the main one. foreach(post; posts.drop(1).take(2)) { // Clone our html template auto newOtherPost = otherPostTemplate.clone(); // Update it with our data newOtherPost.byTagName("h4").front.innerText = post.heading; newOtherPost.byTagName("p").front.innerText = post.text; // Add it to html container container.appendChild(newOtherPost); } Putting it all together: import std; import arrogant; // Init auto parser = Arrogant(); // A blog post struct SimplePost { string heading; string subheading; string text; string uri; } /* Of course real data should come from a db query. We're using an array for simplicity */ SimplePost[] posts = [ SimplePost("D is awesome!", "This is a real subheading", "Original content was replaced", "http://dlang.org"), SimplePost("Example post #1", "Example subheading #1", "Random text #1"), SimplePost("Example post #2", "Example subheading #2", "Random text #2"), SimplePost("Example post #3", "Example subheading #3", "This will never be shown") ]; void main() { // Our template from disk auto page = readText("html/test.html"); // Parse the source auto dom = parser.parse(page); // Take the first element from our data source auto mainPost = posts.front; // Update rendered data of main post dom.byTagName("h1").front.innerText = mainPost.heading; dom.byTagName("p").front.innerText = mainPost.text; dom.byTagName("a").front["href"] = mainPost.uri; // If we have a subtitle we show it. If not, we remove the node from our page if (mainPost.subheading.empty) dom.byTagName("h2").front.detach(); else dom.byTagName("h2").front.innerText = mainPost.subheading; // ----- // Other articles // ----- // Get the container auto container = dom.byId("others").front; // Use the first children as template auto containerItems = container.byCssSelector(div[id="others"] > div); auto otherPostTemplate = containerItems.front.clone(); containerItems.each!(item => item.detach); // Take 2 more posts from list. We drop the first, it's the main one. foreach(post; posts.drop(1).take(2)) { // Clone our html template auto newOtherPost = otherPostTemplate.clone(); // Update it with our data newOtherPost.byTagName("h4").front.innerText = post.heading; newOtherPost.byTagName("p").front.innerText = post.text; // Add it to html container container.appendChild(newOtherPost); } writeln(dom.document); } This program will output a new valid HTML5 page like this: <!DOCTYPE html> <html lang="en"> <head><title>Test page</title></head> <body> <h1>D is awesome!</h1> <h2>This is a real subheading</h2> <p>Original content was replaced</p> <a href="http://dlang.org">More...</a> <h3>Other posts</h3> <div id="others"> <div> <h4>Example post #1</h4> <p>Random text #1</p> </div> <div> <h4>Example post #2</h4> <p>Random text #2</p> </div> </div> </body> </html>  Of course, the same results could be achieved in many other ways and in other languages too. Our library is just a wrapper over a plain C library named Modest. But what really makes the difference is how easy it is to write and read code thanks to D’s powerful and easy-to-understand syntax. The code shown above can be easily understood by anyone has some programming experience. I’ve received pull requests for our project from colleagues who had never heard of D at all. That’s only one part of the big picture since we’re using many different libraries for different purposes. ### Performance Obviously, performance was a big win. The website felt like it was running on local machines, bringing a dramatic increase to speed and lower latency across the board. After the switch, at first the load on our cloud servers was so low that we thought the website was down! Switching from PHP to D meant we could cut in half the instance size of each Amazon AWS machine in our cloud. And these machines are still underloaded. Our database cloud was highly affected by this too. We now use one quarter of its original computational power. All of this brought an instantaneous and dramatic cost savings, down to more than half of what our costs used to be. ### One more thing… A few days after launch we realized that some of our costs were rising anyway. We were relying on a third-party service to host and cut the pictures we display on the website. This is not a simple task; in order to crop a picture correctly, you need to know where the subjects of the picture are located and you must try to keep them inside the trimmed frame. On the legacy website we mostly used a fixed proportion for images and we used a third-party service for some special cases. The new version of 2night.it has several different possible cuts for each “master” picture, and this raised the costs by 15x! Luckily, we found that a D binding to the OpenCV API is available. We used this to develop a smart algorithm that can cut any photo while preserving the subject of the picture. And again, the performance of our service is so impressive that we do not need a new machine to host it. In a week or so the costs for pictures dropped from some thousands of euros per month to almost 0. # Get Ready for Symmetry Autumn of Code 2019 Last year, Symmetry Investments sponsored three programmers to work on projects in the D community. Two completed successfully. Francesco Gallà worked on adding support for HTTP/2 to vibe.d and got the grand prize of a bonus payment and a free trip to DConf 2019, where he gave a talk about the experience and his project. His compatriot, Franceso Mecca, worked on porting an old D1 forking GC to D2 for integration into DRuntime. He has a blog post drafted on that already that I will publish here as soon as he finishes the next round of work on the project and updates the draft with some new data. Now, on behalf of the D Language Foundation and Symmetry Investments, I’m happy to announce that we are go for Symmetry Autumn of Code 2019! You’ll find the details on the SAoC page here at the blog. This year it’s the same but different! It’s open once again to any programmer, with a preference for students. Participants will receive$1000 USD for each of three successful milestones for a total of $3000. One participant will be selected for a final$1000 payment and a free trip to DConf 2020.

Thanks to feedback from last year’s participants, we’ve made some adjustments this time around. The most visible is that we’re allowing more time to ramp up the projects. Officially, the project planning phase begins September 1, but realistically it begins a week earlier when the applicants are notified of their application status. The official launch of SAoC 2019 is on September 15, two weeks later than last year. We’ll also require participants to submit weekly status updates to the forums this year to encourage more interaction with the community.

Behind the scenes, we’ll be a little more hands on this year in the areas where it matters. The participants last year were happy that we weren’t looking over their shoulders, but there were areas where they would have found our intrusion be a boon. So this year, we’ll be taking a more active role in helping participants find mentors and ramp up their projects, and we’ll communicate more frequently with each student to determine any other specific support we can provide. We’ll stay out of their code and their workflow–that’s what mentors are for–but we’ll do what we can to help ensure a successful project for each participant.

We’re also doing something different for the mentors this year. The SAoC 2018 mentors received the gratitude of the participants and the organizers, and nothing else. Mentors in SAoC 2019 will get all that gratitude, but they’ll also each receive a one-time payment of $500 at the end of the event for a successful project. If, for whatever reason, a participant is unable to complete the event, the mentor may still be eligible for the payment, but it will have to be evaluated on a case-by-case basis. Following in the tradition of the Google Summer of Code events we’ve participated in over the years, we used a Wiki page for SAoC 2018 project ideas. This year, we’re using the new dlang/projects repository. Anyone with an idea for a SAoC project is welcome to submit an issue there. We’ll tag suitable projects with a SAOC tag so that SAoC 2019 applicants can easily search for projects that interest them. Please read the SAoC page for some guidelines on the sort of information a good project idea should contain. We’re looking forward to seeing the applications this year. We’d like to expand the number of participants from last year’s three, but that depends on how many applications we get. If you are interested in participating, please start thinking about your project ideas! The more detailed your application, the better the odds of being selected. We’ll get the tags going on dlang/projects soon to help get the gears cranking. For the rest of you, please submit project ideas or consider being a mentor. Potential mentors should either contact me directly or announce their interest in the forums. Applicants in search of a mentor can do the same. The details are on the SAoC 2019 page. Good luck! # Liran Zvibel of WekaIO on using D to Create the World’s Fastest File System Matrix, the world’s fastest file system, was written in D and recently posted impressive numbers in the IO-500 Node Challenge. It was created by WekaIO, a San Jose, CA, based startup with engineering in Tel Aviv, Israel. Liran Zvibel, the co-founder and CEO of WekaIO, has been a regular speaker at DConf, talking about their use of D at DConf 2015, 2016, and 2018. WekaIO is an expression of a design goal of D, that you can write your prototype quickly and easily in D, then continue working on the same codebase until it reaches production quality, as opposed to prototyping in a different high-level programming language. Liran took some time out of his busy schedule to answer some questions about WekaIO and their use of D. Liran Zvibel at DConf 2018 Joakim: Tell us about the enterprise storage market that WekaIO is in. How do you use D in your product? Liran: Any compute environment has a mix of CPU power, networking, and storage: this is uniform across many organization types and sizes, and whether the infrastructure is in-house (“on premises”) or in the public cloud (“The cloud is just somebody else’s computer,” as they say). The storage component provides the current state and also the history needed by compute. While compute is stateless (how many times have you rebooted your computer to fix a problem?) and the network is ephemeral, storage must be able to keep its state consistently and coherently while providing enough performance for the compute to do its job (otherwise the running jobs are IO-bound, and nobody likes that). There are three main types of centralized storage systems: • Block storage systems, that provide the abstraction of a local drive that sits remotely. Several systems may get access to their own “volume” on the centralized system. The AWS equivalent for such a system is Elastic Block Store. These volumes are usually not shared between more than one server, and the reason to use them is failure resiliency, reliability, and performance (also some advanced features such as taking a point in time, backup, integration into a VM environment, etc.). • File systems, centralized storage that is also shared, and allows several servers on the network to access the same data. This is the kind of system that WekaIO provides. Traditionally, people have turned to block-based solutions if they needed high performance, then created a local file system based on that shared volume, but with WekaIO we show that we enable a shareable file system that is even faster than a local file system over block storage. • Object storage solutions, these enable storing objects with reduced semantics (no ability to modify, data stored is only eventually consistent, etc) to enable cost savings, and generally don’t care about performance. When I review storage systems here, I talk mainly about block-based and file system solutions as object storage is much simpler and is implemented using different methods. Requirements for storage systems are: • Reliability • Performance (low-latency IOPS and throughput) • Features Traditionally, these systems have a “data path” that cares about reliability and performance, and then a “control path” or “management path” that takes care of higher-level features and making reliability work at a higher level. For many storage systems, the data path is implemented in some system programming language, such as C or C++, and the management path is implemented in a higher-level language such as Python, Java, Go, etc. Our previous company, XIV (that IBM acquired in 2007) used a private version of C that had polymorphism, generic programming, and integrated RPC, for code that was a mixture of XML, C, and weird header files. At WekaIO, we use the D programming Language to implement both the data path and the control path, as we can use a single language to get both machine-level access with high performance when needed, and a higher-level language for the features. Joakim: How did you end up choosing D? You mentioned at DConf that you initially tried a combination of Python and C++. Liran: The first part of the work we had done on the C++/Python combo was to work on an efficient in-process RPC mechanism between Python and C++, that would allow us to refer to the same object from C++ and Python in an efficient way. That way the same object could have run in the C++ context when needing performance, and in Python when needing brevity. The idea was to start implementing the system in Python then convert pieces to C++ as needed, where inter-process communication (inter or intra servers) would be done in C++ only. At that time, we had a prototype of the system implemented in Python and we started working on the C++ part, and the RPC definition first. When we discovered D, we realized we may be able to get a single language to handle both the high-level and performance requirements, and we started by running some pet projects to verify that this was indeed a working language. The next phase was implementing our RPC over D (which was much more elegant than the C++ version), and a tracing system that would allow us to debug (the tracing system was reviewed in my DConf 2015 talk). The biggest limitation we had initially was the availability of an optimizing compiler, as the reference compiler, DMD, does not provide assembly that is equivalent to LLVM or even GCC when running on modern x86 processors. After DConf 2015, and with the help of David Nadlinger, we were able to get LDC (the D compiler with an LLVM backend) to compile our code and generate results that met our needs. Joakim: Andrei Alexandrescu, one of the co-architects of the D language, blogged about visiting you in Tel Aviv: how your code is very compact and that you added new features without growing the codebase much. What key features of D do you use to accomplish this and the speed and other benefits of your storage system? Liran: The strongest part of D is its generic programming. We use generic programming in two ways, that usually are contradictory but with D they actually work well. One way we use generic programming is to achieve higher performance, some examples: • Using static polymorphism, object types get fully assembled by the compiler and runtime runs the correct code. • In many cases, passing compile-time arguments saves expensive memory loads and branches leading to much faster execution. • Compile-time introspection allows placing objects in memory differently, and also makes code run faster based on static decisions. Then we use static if, generics, foreach and introspection to allow us to write higher-level code much easier, and also write code once that applies for many cases (watch Andrei’s 2017 DConf keynote – Design by Introspection). We probably use most of the language features: including User-Defined Attributes, to mark code that our introspection chooses to handle differently later or just for debugging purposes; the built-in unit tests; obviously ranges (these are clever!); and even some contract programming. Joakim: D is not that widely known yet, how do you bring developers on board? Any common hangups? Liran: Our internal environment has a very proprietary form to it, with dependencies and a build environment that is unique to us, but the biggest grief we get from users trying to start using D (even WekaIO employees that try to run some pet projects) is that the first few minutes and out-of-the-box experience with DUB as a package manager and build environment are not streamlined enough. We often get comparisons to other upcoming languages where the on-boarding process is easier and more inviting to new users to write the second project after they compiled the obligatory “hello, world!” program. Newcomers are usually unaware of D features, but as our code is so full of them, they adapt to it extremely quickly during their welcome project, so it’s not a problem at all. Joakim: You mention contracting some LDC devs to do work for you. What kinds of fixes have they had to put in and could you talk more generally about the flaws in D that you’ve had to solve or work around? Liran: The biggest problem we had initially (over 3 years ago) was that LDC did not even compile our code. The compiler choked on it and failed. The first few iterations were just David getting the compiler to compile. Then after we got a running binary, we had a long series of changes to make sure that the code generated by LDC worked the same way as the DMD-generated code, then we got into adding performance improvements to our code and to LDC itself. The WekaIO codebase is larger than the standard project, and the current LDC that we use (which is available on GitHub as a fork) contains some template instantiation changes that are not part of the standard frontend. Johan Engelen runs our current LDC efforts and has spent a lot of time on link-time optimization and performance-guided optimizations to get LDC and our code to run faster. Now, Johan works on running our code and tests before advancing our LDC to the next frontend releases, to make sure things still run well. We care deeply about performance, and also about the binary representation of our data structures, and some of the work Johan does each release is to verify that they don’t change. The biggest issue with D to work around is reliance on the Garbage Collector (GC) throughout the runtime and standard library, Phobos. We have had to work on making sure we can leverage exceptions with no GC, and had to create our own non-GC standard library, as we cannot rely on Phobos for it. We were able to get our LDC and libraries to a state that we have a very efficient runtime environment that produces extremely fast code (with no GC jitter). Our only grief is around compilation time for our large project. Joakim: You’ve talked about sponsoring D meetups in Tel Aviv publicizing the D language. What kind of reception have you had? More generally, Tel Aviv is famous for being a startup hotbed: how does WekaIO find being situated there? Liran: Tel Aviv and Israel are fertile ground for startups. We have run some D meetups and one of them even had Andrei Alexandrescu giving a talk at the Google Campus (our offices were not big enough to fit all the people coming). The reception of the concepts is very good, but unfortunately, we were not able to get other startups to actually start using the D Language. Having a startup in Tel Aviv is both a blessing and a curse, as there is a lot of very strong talent with a lot of experience, especially around infrastructure, but also there is a lot of demand. Alongside the startups, there are many large corporations (Google, Apple, Microsoft, Amazon, and many more) with engineering offices offering very lucrative positions, so even though there is a lot of good “supply,” the “demand” is even stronger and filling positions is not easy. Joakim is the resident interviewer for the D Blog. He has also interviewed members of the D community for This Week in D and is responsible for the Android port of LDC. # Symmetry Autumn of Code is Underway Earlier this year, Laeeth Isharc brought an idea to the D Foundation for Symmetry Investments to sponsor a summer of code. He was eager to provide a few motivated individuals the incentive to get some great work done for D and the D community. Sporadic email discussions preceded some chatting at DConf 2018 Munich and the idea subsequently began to pick up steam. By the time the details were sketched out, it had transformed into the Symmetry Autumn of Code. ## The SAoC projects Eight applicants submitted their resumes and project proposals for three slots. Nearly all of the proposals were taken from the SAoC suggestions page at the D Wiki. Given the limited window, the selection process went fairly quick. Of the three selected, two had mentors attached. It took a little while to find a mentor for the third selection, so we extended the milestone deadline for that participant. Now I can happily say that all three are well underway. ### A fork-based concurrent GC for DRuntime Francesco Mecca proposed this project. The goal is to take Leandro Lucarella’s original D1 fork-based GC, port it to D2, adapt it to DRuntime’s GC interface, and culminate with a pull request for DRuntime to present the work and open discussion. Leandro agreed to mentor this project and is working with Francesco to develop a test suite that the port must pass as part of Milestone 2. They have also included documentation in their milestone list, which is good news. ### vibe.d HTTP/2 implementation This one came from Francesco Galla, who is currently pursuing a MSc in Network and Security. The goal here is to enhance the vibe-http library to support HTTP/2. Fittingly, Sönke Ludwig, the maintainer of vibe.d, agreed to mentor the project, but requested someone to share the load due to his schedule. Sebastian Wilzbach stepped up as co-mentor. This project involves rewriting the current HTTP/1 API, ensuring it works as expected, then incrementally adding support for HTTP/2. Portions of the rewrite were already completed before SAoC came along, but had not yet been tested. As such, testing and bug fixing will be a significant portion of the first milestone. ### Porting the Mago debugger to D This project was proposed by László Szerémi. He’s a heavy user of debuggers in D and wants to see the situation improved. He believes that porting the Mago debugger to D is a major step in that direction. His first two milestones are concerned with translating, testing, and bug fixing. To cap off the SAoC event, he intends to get a GUI frontend up and running with some basic features. László had no mentor when he applied, and no one had specifically volunteered to mentor any debugger projects, so we put out a call in the forums and I reached out directly to a few people. In the end, Stefan Koch agreed to take it on. ## SAoC is not the end I know that one of the goals Laeeth had behind his initial suggestion for this event was to enhance the D ecosystem. None of the selected projects are simple, one-shot tasks. These are projects which will all require attention, care, and effort beyond SAoC. The participants are getting them started and we hope they’ll continue to maintain them for some time to come, but in the end, these are projects for the community. When SAoC 2018 is behind us, it will ultimately be up to the community to determine if the projects live long and prosper or die young. I’ll post more about SAoC and the participants as the event goes on. We wish them the best in meeting their milestones! # How an Engineering Company Chose to Migrate to D Bastiaan Veelo is the lead developer of a specialised program for the computer aided geometric design of ship hulls called Fairway, for the company SARC in the Netherlands. Imagine there is this little-known programming language in which you enjoy programming in your free time. You know it is ready for prime time and you dream about using it at work everyday. This is the story about how I made a dream like that come true. ## My early acquaintance with D Back when “google” was not yet a common verb, I was doing a web search for “parsing C++”. The reason was that writing a report for an assignment had derailed into writing a syntax highlighter for noweb using bison and flex, and I found out firsthand that C++ is not easy to parse. That web search brought up this page (present version) with an overview of the D Programming Language, and the following statement has had me hooked ever since: D’s lexical analyzer and parser are totally independent of each other and of the semantic analyzer. This means it is easy to write simple tools to manipulate D source perfectly without having to build a full compiler. It also means that source code can be transmitted in tokenized form for specialized applications. “Genius,” I thought, “here we have someone who knows what he’s doing.” This is representative of the pragmatic professionalism that still radiates from the D community, and it combines with an unpretentious flair that makes it pleasant to be around. This funny quote decorated its homepage for many years: “Great, just what I need.. another D in programming.” – Segfault Nevertheless, I didn’t have many opportunities to use the language and I largely remained sitting on the fence, observing its development. ## Programming professionally With mostly academic programming experience, I started programming professionally in 2006 for SARC, a Dutch engineering company serving the maritime industry. Since the early ’80s they have been developing software for ship design and onboard loading calculations, which today amounts to roughly half a million lines of code. I think their success can partly be attributed to their choice of programming language: Extended Pascal (the ISO 10206 standard, not one of the many proprietary extensions of Pascal). Extended Pascal was a great improvement over ISO 7185 Pascal. Its compiler, by Prospero Software from England, was fast and well documented. The language is small enough and its syntax appropriately verbose to make engineering professionals quickly productive in programming. Personally though, I spent most of my time programming in C++, modernizing their system for computer aided design of ship hulls using Qt and Coin3D. ## When your company outlives a programming language Although selecting an ISO standard in favor of a proprietary Pascal dialect seemed wise at the time, it is apparent now that the company has outlived the language. Prospero Development Software Ltd was officially dissolved 15 years ago. Still, its former director, Tony Hetherington, continued giving support many years after, but he’d be close to 86 years old now and can no longer be reached. Its website is gone, last archived in 2013. There’s GNU Pascal, which also supports ISO 10206, but that project has stopped moving and long ago lost synchrony with gcc. Although there is no immediate crisis, it is clear that something needs to happen sometime if the company wants to continue its activities in the coming decades. ## Changing the odds A couple of years ago, I secretly started playing with the fantasy of replacing Extended Pascal with D. Even though D’s syntax is somewhat different from Pascal, it shares at least four important similarities: support for nested functions, boundary checking, modules, and compilation speed. In addition, it has many traits that make the language attractive to engineers: good focus on performance and numerics, garbage collection, dynamic arrays, easy parallelization, understandable templates, contract programming, memory safety, unit tests, and even wysiwyg strings and formatted numerals. D’s language features encourage experimentation, which resonates well with engineers. So I wondered what I could do to highlight D’s significance to my employer and show it’s an attractive language to switch to. I thought I could make a compelling case if I could write a parser in D that would take Extended Pascal source and transpile it to D source. At least I would have fun trying! So I went over to code.dlang.org to see if there were any D alternatives to flex and bison. There, I found Pegged, and instantly the fun began. Pegged combines the functionality of flex and bison in one incredibly easy to use package, for which its creator Philippe Sigaud obviously enjoyed writing excellent documentation. Nowadays, Pegged is part of the D language tour and you can try it out on-line without having to install a thing. The beauty is that the grammar from the Extended Pascal language specification maps almost linearly to the PEG from which Pegged generates the parser. For this it makes heavy use of D’s generic programming capabilities and compile-time function evaluation — it can generate a parser at compile time if you want it to! However, it wasn’t smooth sailing all along. As I was testing D, I suddenly found myself being tested as well. I learned the hard way that there is a phenomenon called left-recursion, from which a PEG parser typically cannot break out of. And the Extended Pascal grammar is left-recursive in several ways. Consequently, I spent many evenings and weekends researching parsing theory, until eventually I managed to extend Pegged with support for all kinds of left-recursion! From one thing came another, and I added longest match alternation, case insensitive literals, the toHTML() method for dynamically browsing the syntax tree, and a tracer for logging the parsing process. Obviously, I was having fun. But more importantly, I was demonstrating that the D programming language is accessible enough that a naval architect can understand other people’s code and expand it in non-trivial ways. The icing on the cake came when I was asked to present my experiences at DConf 2017 in Berlin, which you can watch here (and here’s the extra bit I presented at lunch time for the livestream audience). At this time, I was able to automatically translate the following trivial example: program hello(output); begin writeln('Hello D''s "World"!'); end. into D: import std.stdio; // Program name: hello void main(string[] args) { writeln("Hello D's \"World\"!"); } # Language competition Having come this far, the founder of SARC agreed that it was time to investigate the merits of various alternative programming languages. We would do a thorough and objective comparison based on trial translations of a comprehensive set of language features. Due to the amount of manual labor that this requires, we had to drastically prune the space of programming languages in an initial review round. Note that what I am about to present does not declare which programming language is the best in our industry. What we are looking for is a language that allows an efficient transition from Extended Pascal without interrupting our business, and which enables us to take advantage of modern insights and tools. In the initial review round we looked at general language characteristics. Here I’ll just highlight what fell through the sieve and why. Performance is important to us, which is why we did not consider interpreted languages. C++ is in use for one component of our software, but that was written from the ground up. We feel that the options for translation are not favorable, that its long compile times are a serious hindrance to productivity, and that there are too many ways in which one can shoot one’s self in the foot. We cannot require our expert naval architects to also become experts in C++. Nowadays, whenever D is publicly evaluated, the younger languages Go and Rust are often brought up as alternatives. Here, we need not go into an in-depth comparison of these languages because both Rust and Go lack one feature that we rely on heavily: nested functions with access to variables in their enclosing scope. Solutions for eliminating nested functions, like bringing them into global scope and passing extra variables, or breaking files up into smaller modules, we find unattractive because it would complicate automated translation, and we’d like to preserve the structure and style of our current code. GNU C does offer nested functions, but it is a non-standard extension and it has been predicted that many will move away from C due to its unsafe features. After this initial pruning, three languages remained on our shortlist: Free Pascal, Ada and D. As a basis for our detailed comparison, we wrote fifteen small programs that each used a specific feature of Extended Pascal that is important in our current code base. We then translated those programs into each language on our shortlist. We kept a simple score board on how well these features were represented in each language: +1 if the feature is supported or can be implemented, 0 if the lack of the feature can be worked around, and -1 if it can’t. This is what came out of that evaluation: Test Free Pascal Ada D Arrays beginning at arbitrary indices +1 +1 +1 Sets 0 0 +1 Schema types 0 0 +1 Types with custom initial values -1 0 +1 Classes +1 +1 +1 Casts +1 +1 +1 Protection against use of dangling pointers -1 +1 +1 Thread safe memory [de]allocation +1 +1 +1 Calling into Windows API +1 +1 +1 Forwarding Windows callbacks to nested functions +1 +1 +1 Speed of calculations +1 +1 +1 Calling procedures written in assembly +1 0 +1 Calling procedures in a DLL +1 +1 +1 Binary compatibility of strings 0 +1 +1 Binary compatible file i/o -1 0 0 Score 6 10 14 So, Free Pascal is the only candidate with negative scores, Ada positions itself in the middle, and D achieves an almost perfect score. Not effortlessly, though; we’ll talk about some of the technical challenges later. Because Free Pascal is, like D, fully Open Source and written in itself, extending the language and filling in the gaps is theoretically possible. Although some of its deficiencies could certainly be resolved that way, others would be quite complicated and/or unlikely to be accepted upstream. We also estimated the productivity of the languages. Free Pascal scored high because it is closest to what we are used to. Despite its dissimilar syntax, D scored high because of its expressiveness and flexibility. Ada scored lowest because of its rigidity and because of the extra work the programmer has to put in (most importantly casts and conversions). Ada is more verbose than Pascal which was disliked by some of us because it can somewhat obscure the essence of what a piece of code tries to express, and frequently the code became not only verbose but cryptic, which was unanimously disliked. Third, we estimated the future prospects and the advantages each language could bring to the table. Although Free Pascal has a more active community than we expected it to have, we do not see great potential for growth. Ada is renowned for its support for writing reliable code (although it has no monopoly in that field) but it does come at a cost and requires real effort. D has a dynamic and open community, supports both script-like productivity and high performance, includes various features for writing reliable software (approaching Ada but at a much lower cost), and offers some unique advanced features with which wonders can be accomplished. Finally, we estimated the effort of translation. Although Free Pascal is very similar to Extended Pascal, missing features pose a real problem and would require a high degree of manual translation and rewriting. Although p2ada exists, it only works partially in our case and does not fully support Extended Pascal. Because Ada frequently requires additional code (casting to the correct type, pulling in a package, instantiating a generic type, adding a pragma, splitting up Put_Lines etc.), writing or extending a reliable transpiler into Ada would be more difficult than doing the same into D. # Selecting a winner I gave away the winner in the title, but we landed at that conclusion as follows. Ada was the first language to be dropped. We really felt that the extra work that the programmer has to put in is a brake on productivity and creativity. Although it barely played a role in our evaluation, illustrative is the difference between the Ada and D equivalents to the Expressive C++17 Challenge. The D solution is both concise and expressive, the Ada solution is hardly expressive and consists of more lines than I want to write or read. Also of secondary importance, but difficult to ignore, is the difference between the communities surrounding the languages, which in Ada’s case is AdaCore Support, who has no problems demanding annual five-figure subscription fees. Although akin to our current language, Free Pascal was mainly dropped due to its porting challenges and our estimation that its potential is lower and its future outlook is less optimistic than that of D. If we were to choose Free Pascal, we would basically invest a lot of effort only to arrive at a technological solution that we felt would be of lower quality than we currently have. And that’s were I saw a dream come true: A clap on the table by the company founder and it was decided to commit to the effort of bringing twenty-five years worth of Extended Pascal code to D! # What makes a difference In short, my experience is that if a feature is not present in the language, D is powerful enough that the feature can be implemented in a library. Translating each sample program by hand has really helped to focus on replicating functionality, leaving the translation process for later concern. This has led to writing a compatibility library with types and functions that are vital for the conversion. Now that equivalents are known and the parser is done, I just have to implement code generation. Below follows another example that currently translates automatically and executes identically. It iterates over a fixed length array running from 2 to 20 inclusive, fills it with values, prints the memory footprint and writes it to binary file: program arraybase(input,output); type t = array[2..20] of integer; var a : t; n : integer; f : bindable file of t; begin for n := 2 to 20 do a[n] := n; writeln('Size of t in bytes is ',sizeof(a):1); { 76 } if openwrite(f,'array.dat') then begin write(f,a); close(f); end; end. Transpiled to D (or should I say Dascal?) and post-processed by dfmt to fix up formatting: import epcompat; import std.stdio; // Program name: arraybase alias t = StaticArray!(int, 2, 20); t a; int n; Bindable!t f; void main(string[] args) { for (n = 2; n <= 20; n++) a[n] = n; writeln("Size of t in bytes is ", a.sizeof); // 76 if (openwrite(f, "array.dat")) { epcompat.write(f, a); close(f); } } Of course this is by no means idiomatic D, but the fact that it is recognizable and readable is nice, especially for my colleagues who will have to go through an unusual transition. By the way, did you notice that code comments are preserved? One very-nice-to-have feature is binary file compatibility; In fact it may have been the killer feature, without which D might not have been so victorious. The case is that whenever a persistent data structure is extended in our software, we make sure that we can still read and convert that structure from its prior format. That way, if a client pulls out an old design from its archives and runs it through our current software, it will still work without the user even being aware that conversion occurs, possibly in multiple steps. Not having to give up that ability is very attractive. But it wasn’t easy to get there. The main difficulty is the difference in how strings are represented in D and the Prospero implementation of Extended Pascal, in memory and on file. This presented the challenge of how to preserve binary compatibility in file I/O with data structures that contain string members. ## Strings In Prospero Extended Pascal, strings are implemented as a schema type, which is a parameterized type that can be used in the following ways: type string80 = string(80); var str1 : string80; str2 : string(60); procedure foo(s : string); This defines string80 to be an alias for a string type discriminated to have a capacity of 80 characters. Discriminated string variables, like str1 and str2, can be passed to functions and procedures that take undiscriminated strings as arguments, like foo, which thereby work on strings of any capacity. In memory, str1 is laid out as a sequence of 80 chars, followed by a ushort that encodes the length of the string. I say encodes because a shorter string is padded with \0s up to the capacity and the ushort actually contains the length of that padding. This way, when a pointer to the string is passed to a C function and the contents of the string occupy its full capacity, the 0 in the padding length doubles as the terminating \0 of the C string. My first thought was to mimic this data representation with a D template. But that would require procedures like foo to be turned into templates as well, which would escalate horribly into template bloat, a problem with multiple string arguments and argument ordering, and would complicate translation. Besides, schema types can also be discriminated at run time, which does not translate to a template. Could some sort of inheritance scheme be the solution? Not really, because instances of D classes live on the heap, so a string embedded in a struct would just be a pointer instead of the char array and ushort. But binary layout is actually only relevant in files, and in a stroke of insight I realized that this must be why user-defined attributes, or UDAs, exist. If I annotate the string with the correct capacity for file I/O, then I can just use native D strings everywhere, which genuinely must be the best possible translation and solves the function argument issue. Annotation can be done with an instance of a struct like struct EPString { ushort capacity; } The above Pascal snippet then translates to D like so: @EPString(80) struct string80 { string _; alias _ this; } string80 str1; @EPString(60) string str2; void foo(string s); Notice how the string80 alias is translated into the slightly convoluted struct instead of a normal D alias, which would have looked like @EPString(80) alias string80 = string; </code> Although that compiles, there is no way to retrieve the UDA in that case because plain alias does not introduce a symbol. Then hasUDA!(typeof(str1), EPString) would have been equivalent to hasUDA!(string, EPString) which evaluates to false. By using the struct, string80 is a symbol so typeof(str1) gives string80, and hasUDA!(string80, EPString) evaluates to true in this example. There is one side effect that we will have to learn to accept, and that is that taking a slice of a string does not produce the same result in D as it does in Extended Pascal. That is because string indices start at 1 in Extended Pascal and at 0 in D. My strategy is to eliminate slices from the source and replace them with a call to the standard substr function, which I can implement with index correction. Finding all string slices can be accomplished with a switch in the transpiler that makes it insert a static if to test if the slice is being taken on a string, and abort compilation if it is. (Arrays are transpiled into a custom array type that handles slices and indices compatibly with Extended Pascal.) ## Binary compatible file I/O Now, to write structs to file and handle any embedded @EPString()-annotated strings specially, we can use compile-time introspection in an overload to toFile that acts on structs as shown below. I have left out handling of aliased strings for clarity, as well as shortstring, which is a legacy string type with yet a different binary format. void toFile(S)(S s, File f) if (is(S == struct)) { import std.traits; static if (!hasIndirections!S) f.lockingBinaryWriter.put(s); else // TODO unions foreach(field; FieldNameTuple!S) { // If the member has itself a toFile method, call it. static if (hasMember!(typeof(__traits(getMember, s, field)), "toFile") && __traits(compiles, __traits(getMember, s, field).toFile(f))) __traits(getMember, s, field).toFile(f); // If the member is a struct, recurse. else static if (is(typeof(__traits(getMember, s, field)) == struct)) toFile(__traits(getMember, s, field), f); // Treat strings specially. else static if (is(typeof(__traits(getMember, s, field)) == string)) { // Look for a UDA on the member string. static if (hasUDA!(__traits(getMember, s, field), EPString)) { enum capacity = getUDAs!(__traits(getMember, s, field), EPString)[0].capacity; static assert(capacity > 0); writeAsEPString(__traits(getMember, s, field), capacity, f); } else static assert(false, Need an @EPString(n) in front of  ~ fullyQualifiedName!S ~ . ~ field ); } // Just write other data members. else static if(!isFunction!(__traits(getMember, s, field))) f.lockingBinaryWriter.put(__traits(getMember, s, field)); } }  At the time of writing, I still have work to do for unions, which are used in the translation of variant records (including considering the use of one of the seven existing library solutions 1, 2, 3, 4, 5, 6, 7). Currently, detecting unions is a bit involved . Also, there is a complication in the determination of the size of a union when the largest variant contains strings: the D version of that variant may not be the largest because D strings are just slices. I’ll probably work around this by adding a dummy variant that is a fixed size array of bytes to force the size of the union to be compatible with Extended Pascal. This is the reason why D scored a mere 0 in file format compatibility. It is amazing what D allows you to do though, so I may be able to do all of that automatically and award D a perfect score retroactively. On the other hand, it is probably easiest to just add the dummy variant in the Pascal source at the few places where it matters and be done with it. # The way forward Obviously, this is long term planning. It has taken years to grow into D; it will possibly take a year, and probably longer, to migrate to D. Unless others turn up who are in the same boat as us (please contribute!) it’ll be me who has to row this ship to D-land and I still have my regular duties to attend to. My colleagues will continue to develop in Extended Pascal as usual, and once my transpiler is able to translate all or almost all of it, we will make the switch to D overnight. From then on, we’ll be in it for the long run. We trust to be with D and D to be with us for decades to come! # User Stories: Funkwerk The deadline for the early-bird registration for DConf 2018 in Munich is coming up on March 17th. The price will go up from$340 to $400. If you’d like to go, hurry and sign up to save yourself$60. And remember, the NH Munich Messe hotel, the conference venue, is offering a special deal on single rooms plus breakfast for attendees.

A few of the DConf attendees are coming from a local company called Funkwerk. They’re a D shop that we’ve highlighted here on this blog in a series of posts about their projects (you’ll see one of their products in action if you take the subway or local train service in Munich).

In this post, we cap off the Funkwerk series with the launch of a new feature we creatively call “User Stories”. Now and again, we’ll publish a post in which D users talk of their experiences with D, not about specific projects, but about the language itself. They’ll tell of things like their favorite features, why they use it, how it has changed the way they write code, or anything they’d like to say that expresses how they feel about programming in D.

For this inaugural post, we’ve got three programmers from Funkwerk. First up, Michael Schnelle talks about the power of ranges. Next, Ronny Spiegel tells why generated code is better code. Finally, Stefan Rohe enlightens us on Funkwerk’s community outreach.

## The power of ranges

Michael Schnelle has been working as a software developer for about 5 years. Before starting with D 3 years ago, he worked in (Web)Application Development, mostly with Java, Ruby on Rails, and C++, and did Thread Modeling for Applications. He enjoys coding in D and likes how it helps programmers write clean code.

In my experience, no matter what I am programming, I always end up applying functions to a set of data and filter this set of data. Occasionally I also execute something with side effects in between. Let’s look at a simplified use case: the transformation of a given set of data and filtering for a condition afterwards. I could simply write:

foreach(element; elements) {
auto transformed = transform(element);
if (metCondition(transformed) {
results ~= transformed
}
}

Using the power from std.algorithm, I can instead write:

filter!(element => metCondition(element))
(map!(element => transform(element))(elements));

At this point, we have a mixture of functional and object-oriented code, which is quite nice, but still not quite as readable or easy to understand as it could be. Let’s combine it with UFCS (Uniform Function Call Syntax):

elements.map!(element => element.transform)
.filter!(element => element.metCondition);

I really like this kind of code, because it is clearly self-explanatory. The foreach loop, on the other hand, only tells me how it is being done. If I look through our code at Funkwerk, it is almost impossible to find traditional loops.

But this only takes you one step further. In many cases, there happen to be side effects which need to be executed during the workflow of the program. For this kind of thing, the library provides functions like std.range.tee. Let’s say I want to execute something external with the transformed value before filtering:

elements
.map!(element => element.transform)
.tee!(element => operation(element))
.filter!(element => element.metCondition)
.array;

It is crucial that operations with side effects are only executed with higher-order functions that are built for that purpose.

int square(int a) { writefln("square value"); return a*a; }

[4, 5, 8]
.map!(a => square(a))
.tee!(a => writeln(a))
.array;

The code above would print out the square value six times, because tee calls range.front twice. It is possible to avoid this by using functions like std.algorithm.iteration.cache, but in my opinion, the nice way would be to avoid side effects in functions that are not meant for that.

In the end, D gives you the possibility to combine the advantages of object-oriented and functional programming, resulting in more readable and maintainable code.

## Generated code is better code

Ronny Spiegel has worked as a professional software developer for almost 20 years. He started out using C and C++, but when he joined Funkwerk he really started to love the D language and the tools it provides to introspect code and to automate things at compile time.

In a previous blog post, I gave a short overview of the evolution of the accessors library. As you might imagine, I really like the idea of using the compiler to generate code; in the end this usually results in less work for me and, as a direct result, causes fewer errors.

The establishment of coding guidelines is crucial for a team in order to create maintainable software, and so we have them here at Funkwerk. There is a rule that every value object (or entity) has to implement the toString method in order to provide diagnostic output. The provided string shall be unambiguous so that it’s more like Python’s __repr__ than __str__.

Example:

StationMessage(GeneralMessage(4711, 2017-12-12T10:00:00Z), station="BAR", …)

The generated string should follow some conventions:

• provide a way to uniquely reconstruct data from a string
• continue with any potential superclasses
• list all fields providing their name and value separated by a comma
• be compact but still human readable (for developers)
• skip the name where it matches the type (e.g. a field of type SysTime is called time)
• skip the name if the field is called id (usually there’s an IdType used for type safety)
• there’s some special output format defined for types like Date and SysTime
• Nullable!T’s will be skipped if null etc.

To format output in a consistent manner, we implemented a SinkWriter wrapping formattedWrite in a way that follows the listed conventions. If this SinkWriter is used everywhere, this is the first step to fully generate the toString method.

Unfortunately that’s not enough; it’s very common to forget something when adding a new field to a class. Today I stumbled across some code where a field was missing in the diagnostics output and that led to some confusion.

Using (template) mixins together with CTFE (Compile Time Function Execution) and the provided type traits, D provides a powerful toolset which enables us to generate such functions automatically.

We usually implement an alternative toString method which uses a sink delegate as described in https://wiki.dlang.org/Defining_custom_print_format_specifiers. The implementation is a no-brainer and looks like this:

public void toString(scope void delegate(const(char)[]) sink) const
{
alias MySelf = Unqual!(typeof(this));

sink(MySelf.stringof);
sink("(");

with (SinkWriter(sink))
{
write("%s", this.id_);
write("station=%s", this.station_);
// ...
}

sink(")");
}

This code seems to be so easy that it might be generalized like this:

public void toString(scope void delegate(const(char)[]) sink) const
{
import std.traits : FieldNameTuple, Unqual;

alias MySelf = Unqual!(typeof(this));

sink(MySelf.stringof);
sink("(");

with (SinkWriter(sink))
{
static foreach (fieldName; FieldNameTuple!MySelf)
{{
mixin("const value = this." ~ fieldName ~ ";");
write!"%s=%s"(fieldName, value);
}}
}

sink(")");
}

The above is just a rough sketch of how such a generic function might look. For a class to use this generation approach, simply call something like

mixin(GenerateToString);

inside the class declaration, and that’s it. Never again will a field be missing in the class’s toString output.

Generating the toString method automatically might also help us to switch from the common toString method to an alternative implementation. If there will be more conventions over time, we will only have to extend the SinkWriter and/or the toString-template, and that’s it.

As a summary: Try to generate code if possible – it is less error prone and D supports you with a great set of tools!

## Funkwerk and the D-Community

Stefan Rohe started the D-train at Funkwerk back in 2008. They have loved DLang since then and replaced D1-Tango with D2-Phobos in 2013. They are strong believers in open source and local communities, and are thrilled to see you all in Munich at DConf 2018.

Funkwerk is the largest D shop in south Germany, so we hire D-velopers, mainly just through being known for programming in D. In order to give a little bit back to the D community at large and help the local community grow, Funkwerk hosted the foundational edition of the Munich D Meetup.

### The local community is important …

The meetup was founded in August 2016, 8 years after the first line of D code at Funkwerk was written. Since then, the Meetup has grown steadily to ~350 members. At that number, it is still not the biggest D Meetup, but it is the most visited and the most active. It provides a chance for locals in Munich to interact with like-minded D-interested people each month. And with an alternating level of detail and a different location each month, it stays interesting and attracts different participants.

### … and so is the global community

To engage with the global community, Funkwerk is willing to open source some of its general-purpose D libraries. They can all be found under github.com/funkwerk, and some are registered in the DUB registry.

To mention are:

• accessors – a library to auto generate getters and setters with UDAs
• depend – a tool that checks actual import dependencies against a UML model of target dependencies
• d2uml – reverse engineering of D source code into PlantUML class outlines

Feel free to use these and let us know how you like them.

# Unit Testing In Action

Mario Kröplin is a developer at Funkwerk AG, a German company whose passenger information system is developed in D and was recently highlighted on this blog. That post describes Funkwerk’s use of third-party unit testing frameworks and says, “the team recently discovered a way to combine xUnit testing with D’s built-in unittest, which may lead to another transition in their unit testing.” That’s Mario’s subject in this post.

### There and Back Again

Ten years ago, programming in D was like starting over in our company. And, of course, unit testing was part of it right from the beginning. D’s built-in simple support made it easy to quickly write lots of unit tests. Until some of them failed. And soon, the failure became the rule. There’s always someone else to blame: D’s simple unit-test support is too simple. A look at Python reveals that the modules doctest and unittest live side by side in the standard library. We concluded that D’s unit test support corresponds to Python’s doctest, which means that there must be something else for the real unit testing.

Even back then, we immediately found such a unit testing framework in DUnit [An old D1 unit testing framework that you can read about at the old dsource.org – Ed.]. Thanks to good advice for xUnit testing, we were happy and content with this approach. At the end of life of D1, a replacement library for D2 was soon found. After a bumpy start, I found myself in the role of the maintainer of dunit [A D2 unit-testing framework that is separate from DUnit – Ed].

During DConf 2013, I copied a first example use of user-defined attributes to dunit. This allowed imitating JUnit 4, where, for example, test methods are annotated with @Test. By now, dunit imitates JUnit 5. So if you want to write unit tests in Java style, dunit is a good choice. But which D programmers would want to do that?

Recently, we reconsidered the weaknesses of D’s unit test support. Various solutions have been found to bypass the blockers (described in the following). On the other hand, good guidelines are added, for example, to use attributes even for unittest functions. So we decided to return to making use of D’s built-in unit test support. From our detour we retain some ideas to keep the test implementation maintainable.

### Expectations

Whenever a unit test fails at run time, the question is, why? The error message refers to the line number, where you find something like assert(answer == 42). But what is the value of answer if it isn’t 42? The irony is that this need is well understood. If you use a static assert instead, the error message reads like: static assert 54 == 42 is false. The fear of code bloat is the reason why you don’t get this automatically at run time.

If you look at the Language Reference, you will notice that the chapter Unit Tests covers primarily the special unittest function. It is assumed that assert is used for test verification, which is introduced in the chapter Contract Programming. In theory, it’s completely OK to reuse assert for test verification. Any failure reveals a programming error that must be fixed. In practice, however, test expectations are quite different from preconditions, postconditions, and invariants. While the expectations are usually specific (actual == expected) the contracts rather exclude specific values ​​(value != 0 or value !is null).

So there are lots of implementations of templates like assertEquals or test!"==". The problem shows up if you want to have the most helpful error messages: expected 42 but got 54. For this, assertEquals is too symmetrical. In fact, JUnit’s assertEquals(expected, actual) was turned into TestNG’s assertEquals(actual, expected). Even with UFCS (Uniform Function Call Syntax), it is not clear how a.assertEquals(b) should be used. From time to time, programmers don’t write the arguments in the intended order. Then the error messages are the opposite of helpful. They are misleading: expected 54 but got 42.

Fluent assertions avoid this symmetry problem: actual.should.eq(expected) or expect(actual).to.eq(expected) are harder to use incorrectly. Thanks to UFCS and lazy parameters, the implementation in D is no problem. The common criticism is “the natural language formulation is too verbose”, or just “too many dots”. Currently, however, this seems to be the only way to get the most helpful error messages.

The next problem is that string comparisons are seldom as simple as: expected foo but got bar. Non-printable characters or lengthy texts, such as XML or JSON representations, sabotage error messages that were meant to be helpful. This can be avoided by escaping special characters and by showing differences. Finally, this is what the fluent-asserts library does.

### Test Execution

At large, we want to get as much information as possible from a failed test run. How many test cases fail? Which test cases fail? Does the happy path fail or rather edge cases? Is it worth addressing the failures, or is it better to undo the change? The approach of stopping on the first error is contrary to these needs. The original idea was to run the unit tests before the start of the actual program. By now, however, separate test runners are often used, which continue in case of a failure. To emphasize this, test expectations usually throw their own exceptions, instead of the unrecoverable AssertError. This change already shows how many test cases fail.

Finding out what’s tested in the failing test cases is more difficult. At best, there are corresponding comments for documented unit tests. But an empty DDoc comment, ///, is all that’s needed to include the body of the unittest function as an example in the documentation. In the worst case, the unit test goes on and on verifying this and that.

The idea of the Sentence Style For Naming Unit Tests is that the name of the test function describes the test case. In D, however, the unittest functions are anonymous. On the other hand, D has user-defined attributes so that you can even use strings for the test description instead of CamelCase names. unit-threaded, for example, shows these string attributes so that you get a good impression of the extent of the problem in case of a failure. In addition, unit-threaded satisfies the requirement to execute test cases selectively. For example, only the one problematic test case or all tests except those tagged as “slow”. It’s promising to use unit-threaded as needed. You let D run the unittest functions as long as they pass. Only for troubleshooting should you switch to unit-threaded. You have to be careful, however, to only use compatible features.

By the way: the parallel test execution (from it’s name, the main goal of unit-threaded) was quite problematic with the first test suite we converted. On the other hand, the speedup was just 10%.

### Coverage

The D compiler has built-in code-coverage analysis. The ratio of the lines executed in the test is often used as an indicator for the quality of the tests. (See: Testing in the D Standard Library) A coverage of 100% cannot be achieved, for example, if you have an assert(0). Lower thresholds for the coverage can always be achieved by cheating. The fact that the unittest functions are also incorporated in the coverage is questionable. Imagine that a single line that has not yet been executed requires a lengthy unit test. As a consequence, this new unit test could significantly raise the coverage.

In order to avoid such measurement errors, we decided from the beginning to extract non-trivial unit tests to separate modules. We place these in parallel to the src tree in a unittest directory. Test utilities are also placed in the unittest directory, so that reading the actual code is not encumbered by large version (unittest) sections. (We also have test directories for customer tests.) For the coverage, we only count the modules under src. Code-coverage analysis creates a report file for each module. For a summary, which we output at the end of each successful test run, we have written a simple script. By now, covered is a ready-made solution.

In order to fully exploit the code-coverage analysis, an unusual formatting is required, for example, for the short-circuit evaluation of expressions with &&, ||, and ?:. We hope that dfmt can be changed to reformat the code temporarily.

### Fixtures

What can you do to prevent the test implementation from getting out of control? After all, test code is also code that needs to be maintained. Sometimes the test implementation is more obscure than the code being tested.

As a solution the xUnit patterns suggest a structuring of the test implementation as a Four-Phase Test: fixture setup, exercise system under test, result verification, fixture teardown. The term fixture refers to the test context. For JUnit, this is the test class with attributes that are available to all test methods. A method with the annotation @BeforeEach initializes the attributes. This is the fixture setup. Another method with the annotation @AfterEach implements the fixture teardown. All methods annotated with @Test focus on exercise and verification.

At first glance, this approach seems to be incompatible with D’s unittest functions. The unittest functions do not get automatic access to the attributes of a class, even if they are defined in the context of a class. On the other hand, one can mimic the approach, for example, by implementing the fixtures next to the unittest functions as a struct:

unittest
{
Fixture fixture;
fixture.setup;
scope (exit) fixture.teardown;
(fixture.x * fixture.y).should.eq(42);
}

The test implementation can be improved by executing the fixture setup in the constructor (or in opCall(), since default constructors are disallowed in structs) and the fixture teardown in the destructor:

unittest
{
with (Fixture())
(x * y).should.eq(42);
}

The with (Fixture()) pulls the context, in which test methods are executed implicitly in JUnit, explicitly into the unittest function. With this simple pattern you can structure unit tests in a tried and trusted way without having to use a framework for test classes ever again.

### Parameterized Tests

A parameterized test is a means to reuse a test implementation with different values ​​or with different types. Within a unittest function this would be no problem. Our goal, however, is to get as much information as possible from a failing test run. For which values ​​or which types does the test fail? unit-threaded provides support for parameterized tests with @Values ​​and @Types. If unit-threaded is not used to run the unittest functions, these test cases do not work at all.

With the new static foreach feature however, it is easy to implement parameterized tests without the support of a framework:

static foreach (i; 0 .. 2)
static foreach (j; 0 .. 2)
@(format!"%s + %s == 1"(i, j))
unittest
{
(i + j).should.eq(1);
}

And if you run the failing test with unit-threaded, the descriptions of the failing test cases reveal the problem without the need to take a look at the test implementation:

0 + 0 == 1: expected 1 but got 0
1 + 1 == 1: expected 1 but got 2

### Conclusion

D’s built-in unit test support works best when there are no failures. As shown, however, you do not need to change too much to be able to work properly in situations where you rely on helpful error messages. The imitation of a solution from another programming language is often easy in D. Nevertheless, one should reconsider such solutions from time to time.

If we had a wish, we would want separate libraries for expectations and for test execution. Currently, you get frameworks where not all features are great, or they are overloaded with alternative solutions. Such a separation should probably be supported by the Phobos runtime library. Currently, each framework defines expectations with its own unit test exceptions. In order to combine them, ugly interdependencies are required to match the exceptions thrown in one library to the exceptions caught in another library. A unit test exception in Phobos could avoid this problem.

# The Evolution of the accessors Library

Ronny Spiegel is a developer at Funkwerk AG, a German company whose passenger information system is developed in D and was recently highlighted on this blog. In this post, Ronny tells the story of the company’s open source accessors library, which provides a mechanism for users to automatically generate property getters and setters using D’s robust compile-time features.

### A little bit of history.

We’ve always used UML tools to visualize the internal structure and document details of software. That’s true for me not only at Funkwerk, but also in the companies I worked before I joined the team here in Karlsfeld. One of the major issues of documentation is that at some point in time it will diverge from the actual implementation and become outdated. Additionally, if you have to support old versions of your components you will have to take care of old versions of your documentation as well.

The first approach to connecting code and model is to generate code from the model, which requires the model to reflect the current implementation. When I joined Funkwerk we were using ArgoUML to manage class diagrams which were used as input to generate code. Not only class or struct skeletons were generated (existing code was kept), but also methods to access members which were not even part of the model. In order to control whether a member should be accessible, annotations, similar to UDAs (User-Defined Attributes), were used as part of the member documentation. So it was very common for us to annotate a member with @Read or @Write even though it was only in the documentation. The tool which we used to generate code was powerful enough to create the implementation of these field accessor methods supported by annotations to synchronize access, or to automatically use invariants for pre- and post-conditions as well.

Anyway, the approach of using the model as a base for code generation always suffers from the same problem: it is very hard to merge models!

So we reversed the whole thing and decided to create documentation from code. We could still use code which had been generated before, but all the new classes had to be supplied with accessor functions. You can imagine that this was very annoying.

public class Journey
{
private Leg[] legs_;

public Leg[] legs()
{
return this.legs_.dup;
}

...
}


(Yes, we’ve been writing Java and compiling as D.)

Code which was generated before still had these @Read and @Write annotations next to the fields. So I thought, “These look like UDAs. Why not just use those to generate the methods automatically?” I’d always wanted to use mixins and compile-time introspection in order to move forward with a more D-like development approach.

### A first draft…

The very first version of the accessors library was able to generate basic read- and write-accessor methods using the allMembers trait, filtering by UDAs, and generating some basic code like:

public final Leg[] legs() { return this.legs_.dup; }

It works… Yes, it does.

We did not replace all existing accessor methods at once, but working on a large project at that time we introduced many of them. The automated generation of accessor methods was really a simplification for us.

### …always has some issues.

The first implementation looked so simple – there must have been issues. And yes, there were. I cannot list all of them because I do not remember anymore, but some of these issues were:

#### Explicitly defined properties suppressed generated ones

We ran into a situation where we explicitly defined a setter method (e.g. because it had to notify an observer) but wanted to use the generated getter method. The result was that the defined setter method could be used but accessing the generated getter method (with the same name) was impossible.

According to the specification, the compiler places mixins in a nested scope and then imports them into the surrounding scope. If a function with the same name already exists in the surrounding scope, then this function overwrites the function from the mixin. So if there is a field with a @Read annotation and another explicitly defined mutating field accessor, then the @Read accessor is overwritten by the defined one.

The solution to this issue was rather simple. We had to use a string mixin to import the generated code into the class where it shall be used.

#### Flags

We have a guideline to avoid magic bools wherever possible and use much more verbose flags instead. So a simple attribute like:

private bool isExtraJourney_;

Becomes:

private Flag!”isExtraJourney” isExtraJourney_;

This approach has two advantages. Providing a value with Yes.isExtraJourney is much more verbose than just a true, and it creates a type. When there are two or more flags as part of a method signature, you cannot change the order of the flags (by accident) as you could with bools.

To generate the type of the return value (or in case of mutable access of the parameter) we used T.stringof, where T is the type of the field. Unfortunately, this does not work as expected for Flags.

Flag!”foo” fooFlag;

static assert(Flag!”foo”, typeof(fooFlag).stringof); // Fails!
static assert(Flag, typeof(fooFlag).stringof); // Succeeds!

#### Unit Tests

When using the mixin in private types defined in unit tests, a similar issue arose. Classes defined in unittest blocks have a prefix like __unittestL526_8. It was necessary to strip this prefix from the used type string.

#### Private Classes

While iterating over members of private classes, we stumbled across the issue that the allMembers (or derivedMembers) trait returns, in addition to __ctor, an unaccessible member called this. This issue remains unsolved.

### The current implementation…

The currently released version covers the aforementioned issues, although there is still room for new features.

An example might be to provide a predicate which is then used for synchronizing access to the field. That was possible using the old version of the code generator by annotating it with @GuardedBy(“this”). Fortunately, we’ve advanced in our D coding skills and have moved away from Java code compiled with DMD to a more D-like style by using structs wherever we need value semantics (and we don’t have to deal with thousands of copies of that value). So at the moment, this doesn’t really hurt that much.

Another interesting (and still open issue) is to create accessors for aliased imported types. The generated code still refers to the real name of the type, which is then unknown to the compile unit where the code is mixed in.

### …has room for improvement!

If you’re interested in dealing with this kind of problem and want to dive into CTFE and compile-time introspection, we welcome contributions!

# Project Highlight: Funkwerk

Funkwerk is a German company that develops intelligent communication technology. One of their projects is a passenger information system for long-distance and local transport that is deployed by long-distance rail networks in Germany, Austria, Switzerland, Finland, Norway and Luxembourg, as well as city railways in Berlin and Munich. The system is developed at the company’s Munich location and, some time ago, they came to the conclusion that it needed a rewrite. According to Funkwerk’s Mario Kröplin:

From a bird’s-eye view, the long-term replacement of our aged passenger information system is our primary project. At some point, it became obvious that the system was getting hard to maintain and hard to change. In 2008, a new head of the development department was hired. It was time for a change.

They wanted to do more than just rewrite the system. They also wanted to make changes in their development process, and that led to questions of how best to approach the changes so that they didn’t negatively impact productivity.

There is an inertia when experienced programmers are asked to suddenly start with unit testing and code reviews. The motivation for unit testing and code reviews is higher when you learn a new programming language. Especially one where unit testing is a built-in feature. It takes a new language to break old habits.

The decision was made to select a different language for the project, preferably one with built-in support for unit testing. They also allowed themselves a great deal of leeway in making that choice.

Compared with other monolithic legacy systems, we were in the advantageous position that the passenger information system was constructed as a set of services. There is some interprocess communication between the services, but otherwise the services are largely independent. The service architecture made such a decision less serious: just make an experiment with any one of the services in the new language. Should the experiment fail, then it’s not a heavy loss.

So they cast out a net and found D. One thing about the language that struck them immediately was that it fit in a comfort zone somewhere between the languages with which their team was already familiar.

The languages in use were C++ (which rather feels like being part of the problem than part of the solution) and Java. D was presented as a better C++. With garbage collection, changing to D is easy enough for both C++ and Java programmers.

They took D for a spin with some experimentation. The first experiment was a bit of a failure, but the second was a success. In fact, it went well enough that it convinced them to move forward with D.

In retrospect, the change of programming language was one aspect that led to an agile transition. One early example of a primary D project was the replacement of the announcement service. This one gets all information about train journeys – especially about the disruptions – and decides when to play which announcements. The business rules make this quite complicated. The new service uses object-oriented programming to handle the complicated business rules nicely. So you could say it’s a Java program written in D. However, as the business rules change from customer to customer, the announcement service is constantly changing. And with the development of D and with our learning, we are now in a better position to evolve the software further.

When they first started the changeover, D2 was still in its early stages of development and the infamous Phobos/Tango divide was still a thing in the D community (an issue that, for those of you out of the loop the last several years, was put to rest long ago). Since D1 was considered stable, that was an obvious choice for them. They also chose Tango over Phobos, primarily because Tango provided logging and HTTP facilities, while Phobos did not. In 2012, as D2 was stablizing and official support for D1 was being phased out, it became apparent that they would have to make a transition from D1 and Tango to D2 and Phobos, which is what they use today.

One of the D features that hooked Mario early on was its built-in support for contracts.

Contracts won me over from the beginning. No joke! When we had hard times to hunt down segmentation faults, a failed assertion boosted the diagnostics. I used the assert macro a lot in C++, but with contracts in D there is no question about whom to blame. A failed precondition: caller’s fault. A failed postcondition or invariant: my fault. We have contracts everywhere. And we don’t use the -release switch. We would rather give up performance than information that helps us fixing bugs.

As with many other active D users, D’s dynamic arraysslices and the built-in associative arrays stand out for the Funkwerk programmers

They work out of the box. It’s not like in Java, where you’d better not use the arrays of the language, but the ArrayList of the library.

They also have put D’s ranges and Uniform Function Call Syntax to heavy use.

With ranges and UFCS, it became possible to replace loops with intention-revealing chains of map and filter and whatever. While the loop describes how something is done, the ranges describe what is done. We enjoy setting the delay in our tests to 5.minutes. And we use UDAs in accessors and dunit. It’s hard to imagine what pieces of our clean code would look like in any other language.

accessors is a library that uses templatesmixins, and UDAs to automatically generate property getters and setters. dunit is a unit testing library that, in an ironic twist, Mario maintains. As it turned out, their intuition that their developers would be more motivated to write code when a language has built-in unit testing was correct (that’s been cited by Walter Bright as a reason it was included in the language in the first place). However, that first failed experiment in D came down to the simplistic capabilities the built-in unit testing provides.

One of the failures of our first experiment was an inappropriate use of the unittest functions. You’re on your own when you have to write more code per test case, for example for testing interactions of objects. With xUnit testing frameworks like JUnit, you get a toolbox and lots of instructions, like the exhaustive http://xunitpatterns.com/. With D, you get the single tool unittest and lots of examples of test cases that can be expressed as one-liners. This does not help in scenarios where such test cases are the exception. Even Phobos has examples where the unittest blocks are lengthy and obscure. Often, such unittest blocks are not clean code but a collection of “Test Smells”. We soon replaced the built-in unit testing (one of the benefits we’ve seen) with DUnit.

The transition to D2 and Phobos necessitated a move away from the Tango-based DUnit. Mario found the open source (and lowercase) dunit (the original project is here) a promising replacement, and ultimately found himself maintaining a fork that has evolved considerably from the original. However, the team recently discovered a way to combine xUnit testing with D’s built-in unittest, which may lead to another transition in their unit testing.

Additionally, Phobos still had no logging package in 2012, so they needed a replacement for the Tango logger API. Mario rolled up his sleeves and implemented log, which, like acessors and dunit, is available under the Boost Software License.

It was meant as an improvement proposal for an early stage of  std.experimental.logger. I was excited that it was possible to implement the logging framework together with support for rolling log files, for logrotate, and for syslog in just 500 lines. We’re still using it.

Funkwerk has also contributed code to Phobos.

I was looking for a way to implement a delay calculation. The problem is that you have an initial delay and you assume that the driver runs faster and makes shorter stops in order to catch up. But the algorithm was missing. So we made a pull request for std.algorithm.cumulativeFold. It’s there because we strived for clean code in a forecast service.

Garbage collection has been a topic of interest on this blog lately. Mario has an interesting anecdote from Funkwerk’s usage of D’s GC.

Our service was losing connections from time to time. It took a while to find that the cause was the interaction between garbage collection and networking. The signals used by the garbage collector interrupt the blocking read. The blocking read is hidden in a library, for example, in std.socketstream, where the retry is missing. Currently, we abuse inheritance to patch the SocketStream class.

The technical description of the derived class from the Ddoc comments read:

/**
* When a timeout has been set on the socket, the interface functions "recv"
* and "send" are not restarted after being interrupted by a signal handler.
* They fail with the error "EINTR" when interrupted, especially
* by the signal used to "stop the world" during garbage collection.
*
* In this case, retries avoid that the stream is mistaken to be exhausted.
* Note, however, that these retries will likely exceed the specified timeout.
*
* See also: Linux manual page signal(7)
*/

After nearly a decade of active development with D, Mario and his team have been pleased with their choice. They plan to continue using the language in new projects.

Over the years, we’ve replaced services in the periphery of the passenger information system whenever this was justified by customer needs. We chose D for the implementation of all of the backend services. We are now working on a small greenfield project (it’s a few buses to be monitored instead of lots of trains), but we intend to grow this project into the next generation of the passenger information system.

In the coming months, we’ll hear more from Mario and the Funkwerk developers about how they use D to develop their system and tooling, and what it’s like to be a professional D programmer.