DMD 2.089.0 Released

Digital Mars logoThe latest release of DMD, the D reference compiler, is ready for download. It’s a relatively light release in terms of changes and features, with 11 major changes and 66 closed Bugzilla issues. Most of the changes cover narrow use cases. To highlight a few: proper non-D mangling in template mixins, a renamed default linker, and expanded support in DUB for LDC.

Proper non-D mangling in template mixins

Maintainers of C bindings, or anyone writing D programs that need to interface with a custom C codebase, may find this change particularly useful. Previously, symbols declared as extern(C), extern(Windows)(which is stdcall), and extern(Objective-C) in template mixins were improperly mangled as D symbols when the templates were mixed in at global scope. It was easy to work around if you didn’t need parameterization (just use string mixins), but if you did need it, then you were either stuck or had to jump through hoops.

The root of the issue was that a template mixin introduces a new scope at the point it’s mixed in and creates aliases to the declarations inside that scope so that they may be accessed externally without the need for dot notation. This is a major convenience when only one instance of the template is mixed into a scope (as is the usual case) and allows for code to be written as if the template mixin doesn’t exist. In other words:

mixin template Foo(T)
{
    T x;
}

mixin Foo;

// Not required
// mixin Foo f;

// Allows this:
int main()
{
    x = 10;

    // Not required
    // f.x = 10;
}

The additional scope meant that, e.g., an extern(C) symbol in a template mixin was never in global scope, so it was mangled as a D symbol rather than a C symbol. With the change, such symbols are promoted from the mixin scope to the global scope and are properly mangled. Now you can choose either string mixins or template mixins for your C/stdcall /Objective-C boilerplate.

No more link.exe conflict

From the first alpha release of DMD, OPTLINK has been the default linker that ships with the compiler on Windows. When Walter Bright first started working on DMD, he used the existing C compiler backend he had been maintaining for 20 years. Since the backend already output object files in the Intel OMF format, he also decided to make use of OPTLINK; it had become a part of the C and C++ compiler package while he was at Borland. It was several years later that he added support Microsoft’s MSCOFF (PE) and the Microsoft linker, cl, to DMD. The fact that both the OPTLINK and cl executables were named link.exe wasn’t an issue early on, but over time it has begun to pop up more often, particularly when performing the compile and link steps separately.

With DMD 2.089, it will never be an issue again. The version of OPTLINK that ships with DMD has been renamed to optlink.exe.

DUB and LDC

DUB, the D build tool and package manager, has shipped with DMD for several releases. When executing dub on a properly configured source package, DMD is the default compiler. It also supports LDC (the LLVM-based D compiler) and GDC (the GCC-based D compiler, which is part of GCC 9) via command-line options, but the full range of DUB features haven’t been available for those compilers.

The latest version improves support for LDC. DUB command-line options to enable code coverage, profiling, keeping the stack frame, and separate linking now work with LDC. Target triples can be given to the --architecture (-a) switch to enable cross-compilation with LDC.

My Vision of D’s Future

When Andrei Alexandrescu stepped down as deputy leader of the D programming language, I was asked to take over the role going forward. It’s needless to say, but I’ll say it anyway, that those are some pretty big shoes to fill.

I’m still settling into my new role in the community and figuring out how I want to do things and what those things even are. None of this happens in a vacuum either, since Walter needs to be on board as well.

I was asked on the D forums to write a blog post on my “dreams and way forward for D”, so here it is. What I’d like to happen with D in the near future:

Memory safety

“But D has a GC!”, I hear you exclaim. Yes, but it’s also a systems programming language with value types and pointers, meaning that today, D isn’t memory safe. DIP1000 was a step in the right direction, but we have to be memory safe unless programmers opt-out via a “I know what I’m doing” @trusted block or function. This includes transitioning to @safe by default.

Safe and easy concurrency

We’re mostly there—using the actor model eliminates a lot of problems that would otherwise naturally occur. We need to finalize shared and make everything @safe as well.

Make D the default implementation language

D’s static reflection and code generation capabilities make it an ideal candidate to implement a codebase that needs to be called from several different languages and environments (e.g. Python, Excel, R, …). Traditionally this is done by specifying data structures and RPC calls in an Interface Definition Language (IDL) then translating that to the supported languages, with a wire protocol to go along with it.

With D, none of that is necessary. One can write the production code in D and have libraries automagically make that code callable from other languages. Add to all of this that it’s possible and easy to write D code that runs as fast or faster than the alternatives, and it’s a win on all fronts.

Second to none reflection.

Instead of disparate ways of getting things done with fragmented APIs (__traits, std.traits, custom code), I’d like for there to be a library that centralizes all reflection needs with a great API. I’m currently working on it.

Easier interop with C++.

As I mentioned in my DConf 2019 talk, C++ has had the success it’s enjoyed so far by making the transition from C virtually seamless. I want current C++ programmers with legacy codebases to just as easily be able to start writing D code. That’s why I wrote dpp, but it’s not quite there yet and we might have to make language changes to accommodate this going forward.

Fast development times.

I think we need a ridiculously fast interpreter so that we can skip machine code generation and linking. To me, this should be the default way of running unittest blocks for faster feedback, with programmers only compiling their code for runtime performance and/or to ship binaries to final users. This would also enable a REPL.

String interpolation

I was initially against this, but the more I think about it the more it seems to make sense for D. Why? String mixins. Code generation is one of D’s greatest strengths, and token strings enable visually pleasing blocks of code that are actually “just strings”. String interpolation would make them vastly easier to use. As it happens, there’s a draft DIP for it in the pipeline.

That’s what I came up with after a long walk by Lake Geneva. I’d love to know what the community thinks of this, what their pet peeves and pet features would be, and how they think this would help or hinder D going forward.

D Language Foundation Funding: New Platforms, New Bounties

Digital Mars logoWhen I first announced the HR Fund here on the blog back in April, there was talk among the D Language Foundation team of hiring one or more people to flesh out the specification and implementation of shared. That sort of work requires a very specific skillset that only a few people in the orbit of D possess. So far, we’ve been unable to find any of them with the time to spare. Meanwhile, the HR Fund is sitting there, waiting to be used.

Mobile Support for LDC

A few weeks ago, Ethan Watson wrote a post in the D forums titled, DMD or LDC on mobile. That thread, followed up by emails with Ethan and a few other people, presented a great opportunity to start putting the HR fund to use. Given that LDC already has support for ARM and DMD does not, it’s more practical to fund efforts on LDC than on DMD

As Adam Ruppe has suggested in the forums, he is currently working under contract to complete the existing work on Android support for LDC. By the time he’s finished, it should be possible for anyone to build a D application for Android and distribute it through the Play Store.

The iOS story, unfortunately, hasn’t yet moved forward. We had the ideal candidate on board and eager to get started, but he was sadly unable to get the time off from work that he would need to get the job done. We’ve asked around, looking for someone else with the same skillset to take on the task, but have come up empty. So now we’re reaching to the community at large. But with a twist…

New Task Bounties

Back in August, I announced that we had launched a new Bug Bounty system. The term “Bug” was perhaps too restrictive, so I’ve renamed the menu to Task Bounties. And as of today the D Language Foundation has seeded three new bounties: two for Bugzilla issues and one for the aforementioned LDC project.

Donate to the campaign for adding iOS/iPadOS support to LDC.

The D Language Foundation has put forward $3000 to seed the bounty to add iOS and iPadOS support to LDC. We encourage anyone interested in seeing this task complete to donate to increase the bounty.

This isn’t a typical bounty, as the money will only be paid as the result of contract work. As such, the money to seed it comes from the HR fund. So if you’re interested in taking the bounty home, click the image above and read the bounty description. We want to get this completed as soon as possible, else we’d wait for our original candidate to become available. So if you have the requisite knowledge, skills, and abilities to get the job done, please don’t hesitate to reach out.

The Foundation seeded two Bugzilla bounties (from the General Fund) at $50 each: one for issue #18472 (betterC: cannot use format at compile time) and the other for issue #18062 (ddoc: Generated .html files should retain the package hierarchy). Click through those links to increase the bounties, or visit Bugzilla #18472 or Bugzilla #18062 for the bug details to get started on fixing one.

I’d like to thank the members of the dlang-jp community for bringing these bugs to our attention. I recently met three of them in Tokyo along with Átila Neves. Aside from having a great time hanging out and touring part of Asakusa, we had a good chat about D and the Japanese D community. I look forward to the next opportunity to see them.

We’ll be seeding more Bugzilla bounties in the coming weeks. I’ll be digging into some of the old #dbugfix issues that are still open. If you have a bug that’s particularly troubling you, please consider seeding a bounty for it yourself. Alternatively, post a link to it on Twitter with the #dbugfix hashtag and we’ll consider the possibility of seeding a bounty with Foundation money.

Please visit the Task Bounties page to see if anything else there strikes your fancy!

The HR Fund Status

The HR Fund currently sits at $16,345. We’re about to lose some of it to Adam for his work on Android and (hopefully) more to someone who takes on the iOS task. Currently, we’re looking into other opportunities to put some of it to use. We still have dreams of funding major work, so we need to continue to make the HR Fund grow.

You can help us by donating to the HR Fund campaign directly, or by using our special $60 campaign: donate $60 to the HR Fund and get a DConf 2019 t-shirt. We still have several shirts available, scattered throughout the world, so please take one off of someone’s hands!

AmazonSmile

In a previous post, I mentioned the AmazonSmile plugins Smile Always for Chrome and Smart Amazon Smile for Firefox as easy ways to support the D Language Foundation. These plugins ensure that every time you visit amazon.com you will be sent to smile.amazon.com instead to support your selected charity. If its the D Language Foundation, we get 0.5% of every eligible purchase you make (and sorry to the international folks, but the D Language Foundation is only available as a charity through the .com domain).

Now, you can also support the D Language Foundation through the Amazon Shopping App for Android. Visit the AmazonSmile Mobile page to see how.

Saving Money by Switching from PHP to D

2night was born in 2000 as an online magazine focused on nightlife and restaurants in Italy. Over the years, we have evolved into a full-blown experiential marketing agency, keeping up our vocation of spreading what’s cool to do when you go out, but specialized in producing brand events and below-the-line unconventional marketing campaigns.

We started using D at 2night in 2012 when we developed a webservice used by our Android and iOS apps. It has worked fine since then, but it was just a small experiment. In 2019, after many other experiments, we decided to take the big step: we switched the complete website from PHP to D. The time was right; we had been planning to give our website a new look and we took this opportunity to rewrite the entire infrastructure.

Development

The job turned out to be easier than we had imagined. We implemented a small D backend over our Mongo database in a few hundred lines. We created a Simple Common Gateway Interface (SCGI) library to interface with the NGINX server and another library to work with the DOM. Using the HTML DOM instead of an obscure HTML template language helped us speed up development a lot. In this way, someone who works on HTML or JavaScript is not required to know D or any template language and can deploy plain HTML and CSS files. On the other hand, someone who works on the backend does not care so much about HTML tags since he can simply access elements by ID, class, etc.; if some HTML tags are moved around the page the whole thing still works. HTML+CSS+JavaScript on the frontend and D on the backend are totally independent.

Writing code in this way is quite simple. Let’s say we want to build a blog page. We start from a simple HTML file like this:

<!DOCTYPE html>
<html lang="en">
  <head><title>Test page</title></head>
  <body>

    <!-- Main post -->
    <h1>Post title</h1>
    <h2>The optional subheading</h2>
    <p>
      Lorem ipsum dolor sit amet, consectetur adipiscing elit.
      Proin a velit tempus, eleifend ex non, aliquam ipsum.
      Nullam molestie enim leo, viverra finibus diam faucibus a.
      Ut dapibus a orci in eleifend.
    </p>

    <!-- Two more posts -->
    <div id="others">
      <h3>Other posts</h3>

      <div>
        <h4>Post#2</h4>
        <p>
          Morbi tempus pretium metus, et aliquet dolor.
          Duis venenatis convallis nisi, auctor elementum augue rutrum in.
          Quisque euismod vestibulum velit id pharetra.
          Morbi hendrerit faucibus sem, ac tristique libero...
        </p>
      </div>

      <div>
        <h4>Post #3</h4>
        <p>Sed sit amet vehicula nisl. Nulla in mi est.
          Vivamus mollis purus eu magna ullamcorper, eget posuere metus sodales.
          Vestibulum ipsum ligula, vehicula sit amet libero at, elementum vestibulum mi.
        </p>
      </div>
    </div>

  </body>
</html>

This is a valid HTML5 file that can be edited by anyone who knows HTML. Now we have to fill this template with real data from a database, which we can represent as an array in this example for the sake of simplicity:

// A blog post
struct SimplePost
{
  string heading;
  string subheading;
  string text;
  string uri;
}

SimplePost[] posts = [
  SimplePost("D is awesome!", "This is a real subheading", "Original content was replaced", "http://dlang.org"),
  SimplePost("Example post #1", "Example subheading #1", "Random text #1"),
  SimplePost("Example post #2", "Example subheading #2", "Random text #2"),
  SimplePost("Example post #3", "Example subheading #3", "This will never be shown")
];

First, we must read our HTML template just as it is and parse it using our html5 library:

  auto page = readText("html/test.html");

  // Parse the source
  auto dom = parser.parse(page);

Then we replace the content of the main article with data from the first element of our array. We use the tag name in order to select the correct HTML element:

  // Take the first element from our data source
  auto mainPost = posts.front;

  // Update rendered data of main post
  dom.byTagName("h1").front.innerText = mainPost.heading;
  dom.byTagName("p").front.innerText = mainPost.text;
  dom.byTagName("a").front["href"] = mainPost.uri;

We want to check if our article has a subtitle. If it doesn’t we’re going to remove the related tag.

  // If we have a subtitle we show it. If not, we remove the node from our page
  if (mainPost.subheading.empty) dom.byTagName("h2").front.detach();
  else dom.byTagName("h2").front.innerText = mainPost.subheading;

If you wanted to get the same result with a template language, you’d probably need to mess up the HTML with something like this:

<!-- We don't like this! -->
<? if(!post.subheading.isEmpty) ?>
<h2><?= post.subheading ?></h2>
<? endif ?>

This mixes logic inside the view and it disrupts the whole HTML file. Anyone who works on the HTML frontend is supposed to know what post is, the logic behind this object, and the template language itself. Last but not least, many HTML editors would probably be driven crazy by any custom syntax. And this is still a simple case!

Going back to our example, to fill the last part of our page we must get the container from the DOM. All we need is to perform a search by ID on the DOM:

auto container = dom.byId("others").front;

Now we use the first element inside the container as a template. So we clone it and we empty the container itself:

  // Use the first children as template
  auto containerItems = container.byCssSelector(`div[id="others"] > div`);
  auto otherPostTemplate = containerItems.front.clone();

  // Remove all existing children from container
  containerItems.each!(item => item.detach);

Finally we add a new child to the container for each post in our data source:

  // Take 2 more posts from list. We drop the first, it's the main one.
  foreach(post; posts.drop(1).take(2))
  {
    // Clone our html template
    auto newOtherPost = otherPostTemplate.clone();

    // Update it with our data
    newOtherPost.byTagName("h4").front.innerText = post.heading;
    newOtherPost.byTagName("p").front.innerText = post.text;

    // Add it to html container
    container.appendChild(newOtherPost);
  }

Putting it all together:

import std;
import arrogant;

// Init
auto parser = Arrogant();

// A blog post
struct SimplePost
{
  string heading;
  string subheading;
  string text;
  string uri;
}

/*
  Of course real data should come from a db query.
  We're using an array for simplicity
*/
SimplePost[] posts = [
  SimplePost("D is awesome!", "This is a real subheading", "Original content was replaced", "http://dlang.org"),
  SimplePost("Example post #1", "Example subheading #1", "Random text #1"),
  SimplePost("Example post #2", "Example subheading #2", "Random text #2"),
  SimplePost("Example post #3", "Example subheading #3", "This will never be shown")
];

void main()
{
  // Our template from disk
  auto page = readText("html/test.html");

  // Parse the source
    auto dom = parser.parse(page);

  // Take the first element from our data source
  auto mainPost = posts.front;

  // Update rendered data of main post
  dom.byTagName("h1").front.innerText = mainPost.heading;
  dom.byTagName("p").front.innerText = mainPost.text;
  dom.byTagName("a").front["href"] = mainPost.uri;

  // If we have a subtitle we show it. If not, we remove the node from our page
  if (mainPost.subheading.empty) dom.byTagName("h2").front.detach();
  else dom.byTagName("h2").front.innerText = mainPost.subheading;

  // -----
  // Other articles
  // -----

  // Get the container
  auto container = dom.byId("others").front;

  // Use the first children as template
  auto containerItems = container.byCssSelector(`div[id="others"] > div`);
  auto otherPostTemplate = containerItems.front.clone();

  containerItems.each!(item => item.detach);

  // Take 2 more posts from list. We drop the first, it's the main one.
  foreach(post; posts.drop(1).take(2))
  {
    // Clone our html template
    auto newOtherPost = otherPostTemplate.clone();

    // Update it with our data
    newOtherPost.byTagName("h4").front.innerText = post.heading;
    newOtherPost.byTagName("p").front.innerText = post.text;

    // Add it to html container
    container.appendChild(newOtherPost);
  }

  writeln(dom.document);

}

This program will output a new valid HTML5 page like this:

<!DOCTYPE html>
<html lang="en">
  <head><title>Test page</title></head>
  <body>
    <h1>D is awesome!</h1>
    <h2>This is a real subheading</h2>
    <p>Original content was replaced</p>
    <a href="http://dlang.org">More...</a>
    <h3>Other posts</h3>
    <div id="others">
      <div>
        <h4>Example post #1</h4>
        <p>Random text #1</p>
      </div>
      <div>
        <h4>Example post #2</h4>
        <p>Random text #2</p>
      </div>
    </div>
  </body>
</html>

Of course, the same results could be achieved in many other ways and in other languages too. Our library is just a wrapper over a plain C library named Modest. But what really makes the difference is how easy it is to write and read code thanks to D’s powerful and easy-to-understand syntax. The code shown above can be easily understood by anyone has some programming experience. I’ve received pull requests for our project from colleagues who had never heard of D at all.

That’s only one part of the big picture since we’re using many different libraries for different purposes.

Performance

Obviously, performance was a big win. The website felt like it was running on local machines, bringing a dramatic increase to speed and lower latency across the board. After the switch, at first the load on our cloud servers was so low that we thought the website was down! Switching from PHP to D meant we could cut in half the instance size of each Amazon AWS machine in our cloud. And these machines are still underloaded. Our database cloud was highly affected by this too. We now use one quarter of its original computational power. All of this brought an instantaneous and dramatic cost savings, down to more than half of what our costs used to be.

One more thing…

A few days after launch we realized that some of our costs were rising anyway. We were relying on a third-party service to host and cut the pictures we display on the website. This is not a simple task; in order to crop a picture correctly, you need to know where the subjects of the picture are located and you must try to keep them inside the trimmed frame. On the legacy website we mostly used a fixed proportion for images and we used a third-party service for some special cases. The new version of 2night.it has several different possible cuts for each “master” picture, and this raised the costs by 15x! Luckily, we found that a D binding to the OpenCV API is available. We used this to develop a smart algorithm that can cut any photo while preserving the subject of the picture. And again, the performance of our service is so impressive that we do not need a new machine to host it. In a week or so the costs for pictures dropped from some thousands of euros per month to almost 0.

DMD 2.088.0 Released

Digital Mars logoThe newest DMD has rolled off the assembly line and is ready for download. A total of 58 contributors fixed 58 bugs and introduced 27 major changes to version 2.088.0 of the compiler.

I’m always looking for the big ticket items in a new DMD release to highlight on the blog, but this is a workaday release that isn’t showing off anything too shiny in the changleog. Much of it is run-of-the mill maintenance: deprecations, removals, and behavior adjustments. All of that is important, and we all welcome it, but it doesn’t make for great reading on the blog. That said, there are a handful of useful additions that I can point to, one of which actually is a big deal when it comes to C++ interop.

std::string and std::vector

Thanks to the work Manu Evans has been performing and advocating, C++ interoperability gets a big boost in this release with bindings to std::string and std::vector in the DRuntime modules core.stdcpp.string and core.stdcpp.vector, respectively.  There’s one caveat with the std::string binding that anyone intending to use it must be aware of.

When compiling on Linux, where DMD makes use of the GCC libraries and linker, there’s a compatibility issue when using the modern version of std::string which is compliant with C++11. It contains an interior pointer, which in D is both illegal and incompatible with move semantics. The work around is to pass -D_GLIBCXX_USE_CXX11_ABI=0 to g++ and compile your D application with -version=_GLIBCXX_USE_CXX98_ABI. This will be resolved in the future when work on move constructors in D is complete.

New Utilities

The language gets an interesting new compile-time trait in the form of getLocation. Given a symbol, this trait will return a tuple containing the file name, line number, and column number at which the symbol appears in the source code. This opens the door to more informative debug logging and error reporting beyond the functionality already available via __FILE__ and __LINE__. And I’m sure folks will find other uses for it.

The standard library utility module std.file, which provides a lot of convenience functions for working with files as a unit, now has the new function getAvailableDiskSpace. Give it a directory path on Windows, or the path to a directory or file on Posix, and it will give you the number of bytes available on that path.

Other News

The Symmetry Autumn of Code 2019 participants all have mentors now and they are hard at work laying out their milestones. Milestone 1 officially kicks off on September 15, after which we can expect to see weekly updates from the participants in the General forum.

Google Summer of Code 2019 has come to an end. Five of our students submitted their work at the end of August. You can find information about their projects and view their code submissions from our GSOC projects page. Congratulations to all who participated!

The D Language Foundation is currently in discussions to put some of the Human Resource Fund to use in finalizing LDC support for iOS and Android. Hopefully, I’ll have details to report on that front in the very near future. In the meantime, please help us raise the HR Fund even higher than it is now. There’s some important work waiting to be done that will require as much money as we can throw at it. You can donate any amount directly to the HR Fund Campaign or use the special campaign we set up to send $60 to the HR Fund and get a DConf 2019 t-shirt in return.

Speaking of t-shirts, thanks to everyone who has made a purchase in our DLang Swag Emporium. You’ve helped us raise over $77 so far, all of which will go to the General Fund. If you haven’t yet dropped in, what are you waiting for? We’ve got t-shirts, stickers, and coffee mugs, with updates coming soon. It’s an easy way to support our favorite programming language!

 

SAOC 2019 Projects and Participants

Symmetry Investments logoLast Sunday, August 18, was the deadline for Symmetry Autumn of Code 2019 applications. We received a total of eight applications, which is the same number we saw last year. This time around we were able to accept more than three: five of the applicants will be participating.

The applications were reviewed by the five members of the SAOC 2019 Committee. Each member independently ranked the applications in order of preference. Points were assigned based on the rankings and the top five applications were accepted.

Before we get into the details of the projects, on behalf of the D Language Foundation and the SAOC team, I’d like to publicly thank all eight applicants for taking the time to submit an application. I’d also like to thank Laeeth Isharc and Symmetry Investments for sponsoring the event again this year, and our five SAOC Committee members for volunteering their time throughout the event:  John Colvin, Mathias Lang, Átila Neves,  Robert Schadek, and Ethan Watson. They will be monitoring the progress of each project through the milestone reports and ultimately will select one participant to receive an extra $1000 payment and an all-expense paid trip to DConf 2020.

The Projects

  • Multi-Level Intermediate Representation Support for LDC – According to the MLIR project README, it’s “a common intermediate representation” intended to “unify the infrastructure required to execute high performance machine learning models in TensorFlow and similar ML frameworks”. Roberto Rosmaninho’s primary project goal is to provide the LDC D compiler with “a new level of abstraction to support the integration of MLIR into the D ecosystem”. Roberto is working on a Computer Science major and is an undergraduate research assistant at Federal University of Minas Gerais in Brazil. His mentor for the project is Nicholas Wilson.
  • Implement DIP 1014 and expand support for C++ STL containers – Suleyman Sahmi’s main goal is to “advance the existing work on [the] D interface to C++ STL containers”. There is a project at GitHub geared toward that end which Manu Evans and Laeeth Isharc have been working on and which is blocked on the lack of an implementation for DIP 1014, along with a few issues with the D ABI and name mangling. He first intends to implement DIP 1014, then he’ll resolve several of the related DMD bugs and will use the remaining time to expand support for the C++ STL. Suleyman is a self-taught programmer from Morocco and already has become a contributor to DMD.
  • DPP with Linux kernel headers – The DPP tool, which allows D modules to directly #include C and C++ headers, currently is unable to work with the Linux kernel headers. Cristian Becerescu aims to fix that. If he is able to do so with time remaining, he will work on further improvements and refinements to DPP, including ironing out issues it might have with other C library headers the community brings to his attention. Cristian is a 4th-year Computer Science and Engineering student at University Politehnica of Bucharest. He is fortunate to have two mentors for this project in the form of Edi Staniloiu and Razvan Nitu.
  • Create a CI or other infrastructure for measuring D’s progress and performance – Max Haughton, an 18-year-old British physics student, will be taking on the task of “creating a mechanism by which we can measure various properties of the D ecosystem in a deterministic manner”. This includes properties such as compilation time, compile-time memory usage, and profiling the compiler to determine “why performance is what it is”. He also intends to extend it to run-time performance by “forming a set of benchmarks by which we can profile Phobos and druntime both against their versions…and the version of the compiler”.
  • Solve Dependency Hell: link with more than one version of the same project – When a project has dependencies that in turn rely on different versions of the same library, steps must be taken to reconcile the version difference in order to successfully compile. If it’s even possible, it’s cumbersome and introduces new difficulties. Tiberiu Lepadatu aims to solve this problem of Dependency Hell by making it possible to compile a project with multiple versions of the same library. This is considered as a crucial first step in making Phobos available via the DUB registry. Tiberiu is no stranger to D and has contributed to the core projects in the past. He will likely be working with Sebastian Wilzbach as his mentor.

Getting Under Way

The SAOC participants will spend the next three weeks preparing to get their projects started. They’ll be compiling their Milestones, doing research, and those without a mentor will be searching for one. Things officially kick off on September 15, with Milestone deadlines falling on the 15th of each month through January.

This year, we’ll be expecting the participants to make weekly updates in the forums. We will also encourage them to spend time on IRC, Slack, or Discourse to get to know the community, discuss their projects, and find inspiration in solving the challenges they’ll face. We encourage all members of the D community to show their support and help keep up motivation.

Each of these projects will improve the D ecosystem.We’re fortunate to have this opportunity, along with out participation in the currently ongoing Google Summer of Code, to get so much done without the need to raise more money or dig into our Human Resource Fund. We should all be willing to do what we can to help these projects succeed.

On a personal level, I’m looking forward to working with these five programmers in the coming months and to seeing all of them make it through to the end of a successful Symmetry Autumn of Code!

 

Task Bounties Have Arrived

In 2013, Facebook launched a page for D bounties at BountySource. It saw a burst of excitement, a few bugs fixed, and then went quiet. By 2017, it had been “deemed unsuccessful”. In internal discussions on fundraising options and how to increase community participation in fixing bugs, the topic of bug bounties has often come up, but the failure of the BountySource page has led us to different alternatives and we’ve never tried to revive it. Last year, before we opened our OpenCollective page, we requested that BountySource refund all unpaid bounties and received confirmation that they had done so. Still, we’re always open to new approaches.

In June of this year, Mathias Lang, the CTO of BPF Korea (BOS Platform Foundation), signaled interest in offering bounties on a few issues and asked the D Language Foundation for suggestions on how to go about it. Given our increasing usage of Flipcause to handle our donations, and the benefits we get in terms of processing fees when more of our donation money is going through Flipcause, it was an easy decision to make. We could set up a menu with multiple campaigns, one for each bounty. Community members would be free to add to money to specific bounties, try their hand at fixing them, or request new bounties be added to the menu.

As of today, the D Task Bounties menu is live.

The Task Bounty System

There are several bounties available, some of which are tied to a Bugzilla issue. The remainder provide a description of the task and the conditions for receiving the bounty. Click a card on the menu to see the details, including the amount of the bounty. You can also increase the bounty of any of the issues by clicking on the card and submitting a donation.

There’s also a Task Bounty Catch-All campaign. This is where you can add to the bounties for multiple tasks with one credit card transaction; click the Catch-All card, enter the total amount you’d like to donate, and specify in a comment how the money is to be dispersed. You can also seed a new bounty through the Catch-All campaign; donate the amount you’d like to seed and either provide a Bugzilla issue number in the comment field or go to the menu at the top of the page, click Contact, and provide a detailed description of the issue and the conditions under which the bounty will be payable.

Bounties on any D ecosystem project are acceptable. Where possible, when seeding a new one, please try to connect it to a Bugzilla issue, a GitHub issue, or equivalent in the project’s issue tracker. We prefer bounties that are not tied to any issue to be the exception rather than the rule.

The conditions for payment of any bounty tied to a core DLang Bugzilla issue is always the same by default: the issue must be closed through a pull request being merged and subsequently shipped in a release of the affected program (dmd, dub, rdmd, etc..). Conditions may be altered by the person who seeds the bounty (e.g., if you decide to seed a bounty for a Bugzilla issue, you can request that payment be made when the PR is merged). Conditions for payment of bounties on issues for other projects must be stated in the description.

I want to reiterate: the conditions for payment of the bounty are entirely up to the person who seeds it, but they must be set when seeding the bounty (or left to the default in the case of Bugzilla issues).To receive a bounty, please email social@dlang.org when the conditions for payment have been met. You’ll need links to the pull request(s) and any other information we need to verify that you’re eligible. You’ll also need an account with a payment service of some kind (preferably Circle, TransferWise, or PayPal).

We’ll see how it goes and adapt with any rules or restrictions as needed. Our goal is to improve the D ecosystem for everyone, and if some folks can make a little money out of it in the process that’s even better. Ideas, suggestions, and feedback are always welcome.

Other News

I want to thank everyone who has donated $60.00 to the HR Fund in exchange for a DConf 2019 t-shirt. I’d also like to thank everyone who has donated directly to the HR Fund Campaign (for a DMan shirt or not). A big thanks especially to WekaIO, who have provided us with the lion’s share of the total. With the $360 we’ve raised with the DConf t-shirts, the fund is currently at $16,345. We still need to keep it growing, so don’t get complacent!

The application deadline for SAOC 2019 is tomorrow,  midnight AoE. Don’t be late!

Finally, for those of you who would like to support the D Language Foundation through Amazon Smile but never remember to go to smile.amazon.com, there’s a browser plugin for that! You can get Smile Always for Chrome or Smart Amazon Smile for Firefox. That way, you can always be sure to support your selected charity with 0.5% of your purchase price when you shop at Amazon.

D Summer School Postmortem

The first edition of the D summer school, held at University POLITEHNICA of Bucharest, took place from the 17th of June to the 4th of July. It was three weeks of bootcamping bachelor students into the basics of D during eight sessions of hands-on workshops, a homework project, and ending with a hackathon. We will describe our experience in organizing the program, teaching the students, and trying to integrate them into the D community.

Who we are

We are Edi Staniloiu and Razvan Nitu, two PhD students at UPB, doing our theses in close relationship with the D programming language. For the past three years been recipients of scholarships from the D Language Foundation for contributing to the D ecosystem—you might also know us under the pseudonym of “Andrei’s students”.

During our first two years of contributing to D our focus was entirely on technical aspects, but last year we started thinking on how we can raise the popularity of the language both in our community and in the local industry. The summer school represents our first step in this direction.

Inception

The idea of organizing a summer school first occurred to us during DConf 2018 in Munich. We presented our thoughts to Andrei and he reacted with enthusiasm, but unfortunately there was too little time to organize something before the start of July. Why so early? Because that’s when students finish with their finals and prepare to leave for their internships. In UPB we have a wide range of summer schools that all start in that period and it doesn’t play well to go against tradition. So we decided to postpone it.

Even though we had to put off the summer school, we still wanted to introduce D to bachelor students one way or another. We thought that the best way to do so would be through a bachelor thesis project. That’s how we ended up working with Alex Militaru on his bachelor project, “D for a @safer Linux Kernel”, which he presented at DConf 2019 in London.

Alex is a top student, but he had never even heard of D. It’s not a difficult language to learn for anyone with programming experience, but it does take time to adapt to the details and subtleties of the language. This made for a rough start, as Alex had a very small time frame to learn it. The potential benefits of an introductory D school came to mind again, making us even more motivated to turn it into a reality. That’s when we committed to making the D summer school happen in 2019.

First steps

The first thing that we had to agree on was the set of topics we were going to cover. D has a lot of interesting features. Books of 300 to 500 pages have been written describing them in various levels of detail. We would not have the time to delve into all the details, so our goal was to touch all the basic concepts without being boring, but also highlight some of the most interesting aspects without being too complex. After a polite debate (which was neither polite, nor a debate)
we agreed on the following topics:

  1. Introduction to D: builtin types, arrays (static, dynamic, associative), slices, imports, functions
    (UFCS), unit tests, contract programming, user-defined types.
  2. Introduction to meta-programming: enum, static if, static foreach, templates, template constraints.
  3. Memory safety: @safe, type qualifiers, template inference, template this parameters.
  4. Advanced D concepts: operator overloading, alias, alias this, overload sets, function attributes, ranges.
  5. Multi-threading: data sharing, concurrency, synchronized, fibers
  6. GC vs. Manual Memory Management.
  7. Interoperability with C/C++ and tooling.
  8. Design by introspection: __traits, mixin, tuples, CTFE, pragmas.

Each session was expected to last three hours: one hour of theoretical presentation and two hours of hands-on exercises (keyboard bashing). The theoretical structure was inspired (and some times shamelessly copy-pasted) from Ali’s awesome (and freely available) book. On this occasion, we would like to publicly thank him for allowing us to use his material.

The practical hands-on segment was split into two: a tutorial and observation period where the student, typically, had to run a program and understand the outcome; a hands-on period where the student had to write code to solve a given problem or fix some intentionally inserted bug.

In order to apply for the summer school, students had to complete an assignment we devised. The was the basis on which they were selected. If you are interested in details you can check our official page where all the materials are located.

Marketing

Now that we knew what the summer school was going to look like, all we had to do was find the students that would attend it. This may not sound like hard work, but here are a few considerations that will put things into perspective:

  1. Every summer school, naturally, wants to attract the best possible students.
  2. There are at least 6 summer schools taking place in UPB during the same period.
  3. Many students have summer internships.

As you can see, not only did we have to compete with other established summer schools, but we also had to convince students that their free time after work would be well spent and that they would learn something cool, interesting, and, most importantly, useful.

Considering that both of us are complete noobs when it comes to marketing, we can say that this was the most challenging part. Luckily, our mentors Razvan Rughinis and Razvan Deaconescu, well established professors in our university and traditional summer school creators, were available to coach us through the basics of human manipulation, a.k.a. marketing. Their contributions came in many forms:

  • they used their status to promote the summer school on all the university’s social media platforms
  • they highlighted the fact that “Secure and Fast Programming in D” is a catchier title than the
    blunt “D Summer School” that we had originally used
  • they provided the grounds for obtaining funding from a local organization, Tech Lounge

With that, all we had to do was wait for students to apply.

The Actual Summer School

Contrary to our expectations, we weren’t flooded with student applications. Actually, rather disappointingly, we did not fill all of our spots. However, after consulting the history of the other summer schools, we learned that our expectations of gathering 40 students were rather unrealistic. To make a comparison, the star summer school in our university, which is the Security Summer School (currently in its 6th edition), had only 15 student applications and 10 participants in its debut year, and saw 40 student applications this year. By that standard, our total of 11 student applications doesn’t look that bad. After checking the submitted assignments, we decided that we would accept all of them.

Most of the summer school went according to plan, with some minor differences:

  • The theoretical part, usually, took more than the planned timeslot of 1 hour, due to the high interest that students expressed with regard to the presented topics. Although we appreciated the level of interest, the remaining time often wasn’t sufficient to finish the practical part.
  • It happened that Andrei Alexandrescu was in Romania during the summer school, so we thought it would be neat if he would teach the “Design by Introspection” course. As was expected, Andrei nailed it and the students were thrilled. However, after the presentation, the hype did not permit us to continue with the practical part, so we had to postpone it to another day. This put us into the position of dropping the “GC vs MMM” course, because we felt that it was more important to have the students get their hands dirty with some DbI.
  • We had also planned that the students complete homework project during the summer school period. The assignment was to implement a simple peer-to-peer file sharing application using vibe-d. The project was intended for them to exercise their newly developed skills, but given the fact that most of the students were attending the summer school after they had previously gone to work, this left them with little time and energy to put into the project.
  • The end-of-school hackathon was intended to have them finish their projects, but in reality, most of the projects were still in an incipient phase. The consequence was that nobody finished their project.

The funding that we obtained from Tech Lounge was used to buy beverages and snacks that were offered throughout the workshops. In addition, we used the money to buy personalized D T-shirts, pizza, and beer/soda/water for everyone during the hackathon.

Next edition

For the second edition (which will take place in June 2020) we will start marketing early, as soon as the school year starts in October 2019 (this year we started marketing in April). Hopefully, this will raise awareness and will lead to an increased number of participants.

We would also like to expand our team by integrating this years’ participants into the teaching and material development process, thus increasing the quality of the experience for future participants.

As it has proven ineffective, the homework project will be dropped; instead, the students will be encouraged at the hackathon to make a Pull Request in one of the core D projects.

Conclusions

At the end of the summer school, we felt that the students were impressed with the language and its capabilities. Most of them have expressed their desire to get involved in the community. In response, we have encouraged them to apply to the Symmetry Autumn of Code or contribute to the wider D ecosystem. We cannot know which path they will take from here but hope that they will continue to build on their recently acquired D skills.

The summer school did not represent the only way in which students could get a crash course on D, but it did increase our exposure to other departments at UPB. As a result, we’ve had multiple discussions on how we can integrate D in various university projects. To name a few:

All these discussions are now materializing in student project proposals, thus expanding our community.

All in all, we feel that this was a great first edition. Not only did we have great students who will hopefully join our community, but we are now also on the radar of our university peers.

Now we’re looking forward for “Secure and Fast Programming in D” V2!

The DLang Swag Emporium (and more)!

It’s been quite a long time coming, but I’ve finally gotten around to setting up a storefront where folks can purchase D swag and support the D Language Foundation at the same time. You can now head over to the DLang Swag Emporium and throw some money at us.

At the moment, you can buy t-shirts, hoodies, and coffee cups sporting the Digital Mars logo and the D Rocket. If the prices appear arbitrary, they are not. They’re based on the cost of the item and the percentage that the Foundation takes in. I’ve set the percentages from 11.5% to 20%, depending on the base cost of the item, with the D Rocket items sending us more money than the Digital Mars logo. I’ll add more items in regular updates, so please keep an eye out for something you like if there’s nothing there for you now.

At some point, I’ll see about integrating the shop into dlang.org. Until then, the store page as it is will server its purpose just fine.

Please remember, we’ve also got DConf 2019 shirts available for anyone willing to donate $60 to our Human Resource Fund through this link. DMan shirts are available for those who can make a one-time donation of $100, or the shirt and a guaranteed DConf discount to those who can set up an annual donation of $200, through this link. The Human Resource Fund is our top fundraising priority right now. Please help us grow it!

As I mentioned in the forums recently, we’ll be setting up a bug bounty system through our Flipcause account Real Soon Now. This week I got an update from the company that’s seeding the system with a few issues. They’re getting their list and the dollar amounts finalized. But anyone who wants to get us started Even Sooner is welcome to contact me with a Bugzilla issue number and a dollar amount, then I’ll get it set up.

I hope to see some of those D Rocket and Digital Mars t-shirts at DConf 2020!

Symmetry Autumn of Code Experience Report: Porting a fork-based GC

Symmetry Investments logoThe 2018 edition of the Symmetry Autumn of Code was a wonderful opportunity for me and two other students to dive into an interesting programming challenge and contribute to the D community. I am going to describe the process that led to my participation in SAOC and what this four months of work meant to me.

Who I am

I am an MSc student in Computer Science at the University of Turin in Italy. My interests mainly revolve around type systems, language theory, formal languages and compilers, and concurrent programming techniques. I am 23 years old and I have been programming since I was 20.

What happened before SAOC

While browsing Hacker News one day, one comment caught my attention: “the [D] community seems second to none as far as signal to noise ratio goes”. From that day I started lurking on the D forums and later read Andrei Alexandrescu’s book, The D Programming Language. While I think it alone wasn’t sufficient to learn the language, it was worth the read for all the discussions about compiler internals, the compromises about the syntax, and the details on ranges and concurrency by message passing.

After reading the book, I had enough confidence to start exploring some real-world code and I decided to dig into the codebases of the D standard library (Phobos) and vibe.d, a popular web app framework. My colleague Francesco Gallà, who also would participate in SAOC (see his DConf 2019 presentation about his experience), started learning D and we approached the community together by discussing one of the past Google Summer of Code proposals, HTTP2 support, on the vibe.d forum.

At the same time, I bought the Garbage Collection Handbook as a personal reading. At the time it seemed totally unrelated to what I was doing with D.

SAOC

On July 14, 2018, the D Language Foundation announced the Symmetry Autumn of Code. It seemed the right occasion to boost my skills with D and get even more involved with the community. I was also thrilled by the possibility of learning new methodologies for writing code with a remote mentor.

I gathered information about a past project, a concurrent garbage collector that was used in D1 in the Tango library. I decided that my SAOC project proposal would be to port it to D2 with the goal of incorporating it into the D runtime.

The original work was done by Leandro Lucarella as his master thesis, and he documented it extensively on his blog. I contacted him and received confirmation that he could mentor such a project. After that, I wrote my proposal. It consisted of two documents, one about me and the other about the project.

The concurrent GC project

The project document had the objective of showing my efforts in understanding the garbage collector currently shipping with the D runtime—a conservative mark and sweep GC that does the following whenever an allocation is issued and memory needs to be reclaimed:

  1. stop all threads except the current one
  2. “hijack” the work of the current thread in order to run the GC routines
  3. start scanning recursively every root memory pointer in order to find every memory block that has been used or is being used by the program
  4. mark all GC-allocated blocks that are no longer referenced
  5. resume all other threads
  6. run destructors from unreachable memory that has been marked in phase 4 and free the remaining unreachable memory
  7. continue execution of the current thread.

The program is paused during steps 1 to 4 (the mark phase) and memory is reclaimed during step 6, where the GC thread hijacks the flow of the thread that triggered the collection. There are various approaches for reducing the pause time, such as using threads to scan and mark the memory objects in parallel; work is being done in that direction.

My proposal highlighted another strategy. By sacrificing memory consumption on systems that support Posix’s fork() system call, pause time could be drastically reduced. A fork-based concurrent GC would be represented by the following sequence of routines:

  1. a thread triggers the GC collection (such a thread is called the mutator)
  2. the GC clones the address space of the current program with the fork() system call
  3. while the parent process continues execution, the child process starts the mark phase
  4. at the end of the mark phase, before exiting, the child process communicates the marked bits (that can be reused) to the parent
  5. the GC uses the unmarked blocks for future allocations (lazy sweep).

With this schema, the pause time of the marking phase is reduced to the duration of the fork() call only. There are many advantages, such as the fact that the other threads do not need to be stopped and that the reclaim phase could run concurrently without hijacking the mutator. In particular, this strategy shows its strength in potentially long-running programs that have large heaps with a high number of live objects.

The implementation resulted in a bit more than 500 lines of code, given that the calling process (called the parent) generates a duplicate of itself in a different address space. This removes the need for synchronization, which has a high overhead both in terms of runtime performance and code implementation. Moreover, Unix and Linux systems provide very efficient fork() implementations with the use of COW memory.

Methodologies

The first thing that I did with Leandro was to correct the milestones that I had predicted. Based on his experience, we put a bigger focus on defining the tests rather than the programming. After that he specified the workflow to me. We setup a test suite in place, mainly composed of dustmite, the runtime benchmarks and tests, and some D1 programs that I ported.

After discussing and applying every change to the codebase in a different branch, I had to run the tests and then open a pull request on my GitHub repository asking for a review from Leandro. Commits had to be very granular and he always provided a lot of feedback. He was always prompt in replying and we had a number of exchanges by email before applying a change. Many times we discussed benchmarks and regressions, and sometimes I asked for help with debugging. I can confidently say that I spent more time researching, reading, debugging, and discussing code than I spent writing it.

I was alone in managing my time and my commitment. Leandro and the D Foundation were always present in discussing things by email, but they didn’t force any time sheet on me nor did they micromanage my work.

The end of SAOC

At the end of the four months of SAOC, I had a working implementation, but I decided to delay a pull request to the D runtime in order to work on some profiling code that could help developers understand in which cases the fork-based garbage collector brings advantages. After a precise garbage collector was announced for DMD 2.085.0, I decided to adapt my work to it. It was very straightforward given the clarity of the added code and the separation of concerns in place.

Leandro was available for mentoring even after SAOC. We exchanged many emails and he showed me how the D garbage collector is being profiled at Sociomantic.

The pull request

A pull request to the D runtime was my final milestone. I was ready at the beginning of February, but I started to procrastinate. I’d had no previous communication with any of the reviewers and I was timorous about engaging with them. I spent a lot of time refactoring my code back and forth and delaying my pull request. At a certain point, I even considered abandoning the final milestone and providing the GC as a library. In the meantime, Rainer Scheutze published a threaded implementation of the mark phase that reduced the mark time in the GC and I lost faith in my project.

Luckily for me, I had the opportunity to attend DConf 2019 in London. There I found many great people who talked with me and convinced to resume my work. I had a brief discussion with Rainer and I started testing against his implementation (I also found a related bug in the CPU detection code) and on the last day, during the annual Hackathon, I finally opened the pull request.

Since May 11, I have discussed changes to my code, reduced the number of lines of code, refactored and collapsed some functions, and resolved bugs related to program termination. The pull request is still open given that there were many rough edges, but overall I am very satisfied about the feedback received and with the whole process. Reviewers responded to every commit and provided guidance when needed. Whenever something wasn’t clear, I replied on GitHub or asked for help in the #D IRC on freenode.net.

In retrospect, I should have opened the PR much earlier and presented the reviewers with every doubt that I had along the path.

Experience analysis

SAOC helped me in understanding the dynamics behind contributing to a community project such as D. I was already spending a fixed amount of time reading the forum every day and I started to direct some of that effort to GitHub. Checking pull requests, commits, and issues was expensive work, but it was necessary to gain knowledge about the methodologies regarding the development of Phobos and the runtime.

I also learned that there are many changes that go unnoticed if you don’t closely follow the discussions on the PR queue. One example is Manu Evans’s work on core.stdcpp that is very difficult to follow when it’s scattered across different forum threads and lacks a proper announcement.

I think that overall, communication was a weak point of my experience. Regarding SAOC, we defined the work in detail but we didn’t put any effort on communicating the status of my work to the community. That could have triggered helpful suggestions from the community that could have helped me to discover holes in my implementation. Again, I benefited from this great amount of freedom in managing my work, but I think it could have coexisted with some effort on communicating more.

Finally, I had previously used D for higher level code, but SAOC forced me to discover D from a new angle; I dived into low-level code. I was amazed by the flexibility of the language and found myself becoming familiar with this different style of writing code very quickly.

Conclusion

The Symmetry Autumn of Code was a very rewarding experience. It was a period of discovery, self-growth, and involvement in a new community.

I am very grateful to Symmetry Investments and the D Language Foundation for this opportunity. Moreover, I want to thank my mentor, Leandro, for all of his help and for all the positive exchanges that we had.