DConf Online 2020: How to Participate

DConf Online 2020
As I write, we are a little over 24 hours away from the start of DConf Online 2020, our first online version of DConf. All of the talks for Day One are uploaded, the livestreams are scheduled, and #BeerConf is almost ready to launch.

The details

All of the prerecorded talks will be accessible on our YouTube channel via the DConf Online 2020 playlist (look under the live chat box for the full playlist; you may have to scroll down). Use the live chat to ask questions during the talk. The speaker will be available to provide short answers in the chat box. Longer, more complex answers, and/or additional context, will be provided in the Q & A livestream. The speaker will let you know if he is providing more detail in the livestream. If you don’t want to tab over to the livestream and miss part of the talk, the livestream will be saved to our channel once it ends and you will be able to go back and watch any part of it you may be interested in.

Each day, the Q & A livestream will begin at 13:50 UTC. Each speaker will be in the livestream 5 minutes before his talk begins and will be available to answer questions for the duration of the talk and for up to 15 minutes after. As I said above, you may ask questions in the live chat of the talk, but you may also ask them in the live chat of the livestream (and will likely have to if you have questions after the talk ends). Depending on the amount of time available, the number of questions, and the speaker’s schedule, each speaker may stay longer than 15 minutes after the talk, but is not required to.

Please note that speakers are not expected to answer off-topic questions. It’s entirely up to them if they do so.

I’ll be hosting the livestream throughout each day. I’ll be chatting with the speakers about their talks and D in general to fill in the dead time when no one is asking questions. After the conference is over, I intend to chop up the livestream and upload the Q & A session for each talk as separate videos.

On Day One, we have an Ask us Anything session scheduled with Walter and Átila. This will take place in the Q & A livestream for that day. We also have a livecoding session by Adam Ruppe scheduled. That will take place in a separate livestream when the Day One Q & A livestream ends (links below). Adam will be monitoring the chat as he codes, so he will answer any questions you have.

The livestream links:

BeerConf

From 18:00 UTC November 20, we’ll be running a Jitsi Meet instance for our online version of BeerConf. Everyone is welcome to join, no alcohol required. If you aren’t familiar with BeerConf, you can read a brief description of it on the DConf Online 2020 website. You can also read about it here on the blog.

BeerConf will run all weekend long. You can come and go as you please, during talks, in between talks, day time, night time, anytime!

See this D forum thread for details on how to join.

The prizes

Throughout the event, I’ll be announcing different ways for viewers to win various prizes. We’ll be handing out t-shirts, coffee mugs, and other items from the DLang Swag Emporium (and maybe a DMan shirt or two). I’ll announce the details in the Q & A livestream and, if a talk is ongoing, in the talk’s live chat. Sometimes, winning the prize may involve tweeting, in which case I’ll announce the details on Twitter, so be sure to follow us if you aren’t already.

Additionally, everyone who asks a question to which a speaker provides an answer will be entered into at least two random drawings. There will be one random drawing at the end of each day which includes those eligible on that day. The winners of these drawings will receive a $50 Amazon eGift card. The winner of the two-day drawing will receive a $100 Amazon eGift card. If you win on Day One, you will not be eligible to win on Day Two, but both winners will be eligible to win the two-day drawing.

Funding for all prizes comes from the D Language Foundation General Fund. You can contribute by buying DConf Online 2020 swag or other items from the DLang Swag Emporium, by selecting the D Language Foundation as your preferred AmazonSmile charity and shopping through smile.amazon.com, or by donating directly to the General Fund.

Swag prize winners will be announced in a talk’s live chat and/or the Q & A livestream, depending on the nature of the prize task. For prize tasks that take place on Twitter, winners will not be announced, but will be notified through private message. Amazon eGift card winners will be announced in the livestream. Since YouTube apparently no longer allows private messages, winners on YouTube will be instructed on how to claim their prize when they are announced in the livestream.

Enjoy!

We want to thank all of our speakers for volunteering their time to put together these presentations and making themselves available for Q & A. Without them, this event would not be possible. We hope you enjoy DConf Online 2020!

D 2.094.0, DConf Online Schedule, and SAOC 2020

Digital Mars D logo

The end of September saw a new release of the reference D compiler, DMD 2.094.0, sporting the latest language features. That was followed not long after by a beta release of LDC, the LLVM-based D compiler, based on the same frontend version. The DMD 2.094.1 patch release entered into beta a few days before this post was published. Meanwhile, the first Milestone of the Symmetry Autumn of Code has come to an end, and the DConf Online 2020 schedule has been published.

DMD 2.094.0

This release of DMD incorporates 21 major changes and 119 fixed Bugzilla issues, thanks to the efforts of 49 contributors. Here are some highlights.

This ain’t your grandpa’s in parameter

Back in the days of yore, when DMD was still a pre-1.000 alpha, the D language supported in, out, and inout parameter storage classes. They had the following meanings:

  • in (input), the default, was the bog standard function parameter which is a mutable copy of its argument, i.e., the normal passed-by-value parameter.
  • out (output) parameters were passed by reference and, upon function entry, initialized to the default initializer value for the parameter type (e.g., 0 for int, float.nan for float, etc).
  • inout (input/output) parameters were passed unmodified by reference.

When D2 came along, there were some changes. inout was replaced by the ref keyword and out kept the same meaning, but now there was an explicit restriction that these parameters could only take lvalue references; rvalue references, commonly used in C++, were forbidden as arguments. With in, things became a little muddy. And that brings us to scope parameters, a D2 feature that has evolved over time.

For quite some time, it was not fully implemented and only affected parameters that were delegates: the compiler would only allocate a closure for a scope delegate if it absolutely needed to. The D2 version of in was intended to be equivalent to const scope, but it was never fully implemented and was effectively equivalent to const. Today, scope is intended to be applied to ref or out parameters to prevent them from escaping the function scope, and with DMD 2.092.0, in finally became equivalent to const scope. In DMD 2.094.0, in has been reimagined and extended to solve the rvalue reference issue.

The first thing to know about the new in is that it’s still equivalent to const scope, but now the compiler is free to choose whether to pass an in parameter’s argument by reference or by value. The second thing to know is that in parameters can now take rvalue references. All of this is implemented behind the -preview=in command line switch first introduced in 2.092.0.

Like any preview feature, the new in may or may not make it into the language proper, and if it does it might not be without changes. But for now, it’s there and waiting to be put through its paces. The more people using it, pushing it, and looking for holes, the sooner we can know if this is the in we’re looking for.

Ddoc Markdown support

Quite a while ago, Ddoc, D’s built-in documentation syntax, was enhanced to support some Markdown features. It was hidden beind a -preview switch. Now, that switch is no longer necessary—Ddoc supports Markdown out of the box.

Note that this is not full-on Markdown. For example, although asterisks are supported for italic and bold text, underscores are not. But Markdown-style links, code blocks, inline code, and images are supported. For the details, see the Documentation Generator documentation.

More speed please

Since the release of DMD 2.091.0, the DMD binaries in the Windows release packages are being compiled with LDC. This is a good thing because LDC has a better optimizer than DMD, which makes DMD’s fast compile times even faster. Now, LDC is used to compile binary releases on Linux, OSX, and FreeBSD. As a side effect, there are now no more 32-bit releases for FreeBSD, and additional binary tools are no longer included. If you need them, you can still pick them up from https://digitalmars.com/ or from older DMD releases.

Download

The latest release of DMD is always available for download at https://dlang.org/download.html. The latest Beta or Release Candidate can always be found there as well. You can also find links to download LDC and GDC, the GCC-based D compiler (which is now an official component of GCC). While you’re there, if you enjoy the D programming language, consider leaving a tip to the D Language Foundation.

DConf Online 2020 Schedule

DConf Online 2020 is coming together nicely. Over the two days of November 21 and 22, we have nine prerecorded talks, a livestream Q & A with the language maintainers, and a livecoding session. We’ll also be bringing our annual real-world BeerConf to the virtual world.

The talks

The prerecorded talks will be scheduled to premiere on our YouTube channel at the UTC times listed on the schedule. For the duration of each talk and for 15 minutes after, each speaker will be avalailable in a separate livestream for questions and answers related to the talk. We want to record the questions and answers verbally for posterity. The idea is that viewers of the prerecorded talk can ask questions in the video’s chat, or ask in the livestream chat during or up to 15 minutes after the talk. The speaker will read the questions out loud. Short answers will be provided both verbally and in the chat. Longer answers will be provided verbally only. Commenters asking questions during the talk will be notified in the chat if their questions were selected so that they don’t have to tab out to the Q & A and miss a portion of the talk. They can go back and watch the Q & A video later on our YouTube channel.

The livestream Q & A with the language maintainers will run on our YouTube channel. We’ll be streaming a video conference call and questions will be taken from the livestream chat. During the livestream, some viewers will be invited to join in on the conference call and ask their question directly in order to provide more opportunity for follow up and feedback. Details on how to participate will be released on the day of the livestream.

Throughout the weekend, we’ll be handing out prizes to random viewers. Eligibility details will be provided during the course of the event, so pay attention!

BeerConf

BeerConf is a real-world DConf tradition dating back to the first edition of the conference, though the name didn’t come around until Ethan Watson coined it a few years later. Every year, we designate a gathering spot where DConf attendees can mingle every evening to unwind. The DConf days are where we all wear our D programmer hats and spend our time talking about our favorite programming language, but BeerConf is our chance to be human. We still talk about D, but we also have the opportunity to go beyond the code and get to know each other on a more personal level.

So for DConf Online, we’re taking BeerConf online. On the evening (UTC) of Friday, November 20, we’ll open the BeerConf video conference to any and all, and we’ll leave it open all weekend. Despite the name, no alcohol is required to participate. All you need is an internet connection and a web browser, and you can come and go as you please. We’ve been running monthly BeerConf events since June of this year, so we know that, though it’s not quite the same as being in the same place, it’s still a lot of fun.

We hope to see you November 20–22 in BeerConf and DConf!

Symmetry Autumn of Code

We are currently running our third annual Symmetry Autumn of Code (SAOC). Sponsored by Symmetry Investments, the event provides an opportunity for D programmers to make a little money working on projects aimed at improving the D ecosystem. Particpants each get paid $1000 for the successful completion of each of three milestones. At the end of a fourth milestone, the progress of each participant will be evaluated by the SAOC committee, then one participant will be awarded a final $1000 payment, and receive free registration and reimbursement for transportation and lodging for the next real-world DConf.

We currently have four programmers coding away toward their goals. Milestone 1 has just come to an end and Milestone 2 is set to begin. The participants will soon be sending in their milestone reports, their mentors will send in progress evaluations, and the SAOC Committee will review it all to determine if everyone has put forth the effort required to continue through the event (we expect no issues on that front!). You can follow the progress of each participant, and perhaps provide them with some timely advice, through their weekly updates in the D General Forum. Search for “SAOC2020”.

Function Generation in D: The Good, the Bad, the Ugly, and the Bolt

Introduction

Digital Mars D logo

A while ago, Andrei Alexandrescu started a thread in the D Programming Language forums, titled “Perfect forwarding”, about a challenge which came up during the July 2020 beerconf:

Write an idiomatic template forward that takes an alias fun and defines (generates) one overload for each overload of fun.

Several people proposed solutions. In the course of the discussion, it arose that there is sometimes a need to alter a function’s properties, like adding, removing, or hanging user-defined attributes or parameter storage classes. This was exactly the problem I struggled with while trying to support all the bells and whistles of D functions in my openmethods library.

Eventually, I created a module that helps with this problem and contributed it to the bolts meta-programming library. But before we get to that, let’s first take a closer look at function generation in D.

The Good

I speculate that many a programmer who is a moderately advanced beginner in D would quickly come up with a mostly correct solution to the “Perfect forwarding” challenge, especially those with a C++ background who have an interest in performing all sorts of magic tricks by means of template meta-programming. The solution will probably look like this:

template forward(alias fun)
{
  import std.traits: Parameters;
  static foreach (
    ovl; __traits(getOverloads, __traits(parent, fun), __traits(identifier, fun))) {
    auto forward(Parameters!ovl args)
    {
      return ovl(args);
    }
  }
}

...

int plus(int a, int b) { return a + b; }
string plus(string a, string b) { return a ~ b; }

assert(forward!plus(1, 2) == 3);        // pass
assert(forward!plus("a", "b") == "ab"); // pass

This solution is not perfect, as we shall see, but it is not far off either. It covers many cases, including some that a beginner may not even be aware of. For example, forward handles the following function without dropping function attributes or parameter storage classes:

class Matrix { ... }

Matrix times(scope const Matrix a, scope const Matrix b) pure @safe
{
  return ...;
}

pragma(msg, typeof(times));
// pure @safe Matrix(scope const(Matrix) a, scope const(Matrix) b)

pragma(msg, typeof(forward!times));
// pure @safe Matrix(scope const(Matrix) _param_0, scope const(Matrix) _param_1)

It even handles user-defined attributes (UDAs) on parameters:

struct testParameter;

void testPlus(@testParameter int a, @testParameter int b);

pragma(msg, typeof(testPlus));
// void(@(testParameter) int a, @(testParameter) int b)

pragma(msg, typeof(forward!testPlus));
// void(@(testParameter) int a, @(testParameter) int b)

Speaking of UDAs, that’s one of the issues with the solution above: it doesn’t carry function UDAs. It also doesn’t work with functions that return a reference. Both issues are easy to fix:

template forward(alias fun)
{
  import std.traits: Parameters;
  static foreach (ovl; __traits(getOverloads, __traits(parent, fun), __traits(identifier, fun)))
  {
    @(__traits(getAttributes, fun)) // copy function UDAs
    auto ref forward(Parameters!ovl args)
    {
      return ovl(args);
    }
  }
}

This solution is still not 100% correct though. If the forwardee is @trusted, the forwarder will be @safe:

@trusted void useSysCall() { ... }

pragma(msg, typeof(&useSysCall));         // void function() @trusted
pragma(msg, typeof(&forward!useSysCall)); // void function() @safe

This happens because the body of the forwarder consists of a single statement calling the useSysCall function. Since calling a trusted function is safe, the forwarder is automatically deemed safe by the compiler.

The Bad

However, Andrei’s challenge was not exactly what we discussed in the previous section. It came with a bit of pseudocode that suggested the template should not be eponymous. In other words, I believe that the exact task was to write a template that would be used like this: forward!fun.fun(...). Here is the pseudocode:

// the instantiation of forward!myfun would be (stylized):

template forward!myfun
{
    void myfun(int a, ref double b, out string c)
    {
        return myfun(a, b, c);
    }
    int myfun(in string a, inout double b)
    {
        return myfun(a, b);
    }
}

Though this looks like a small difference, if we want to implement exactly this, a complication arises. In the eponymous forward, we did not need to create a new identifier; we simply used the template name as the function name. Thus, the function name was fixed. Now we need to create a function with a name that depends on the forwardee’s name. And the only way to do this is with a string mixin.

The first time I had to do this, I tried the following:

template forward(alias fun)
{
  import std.format : format;
  import std.traits: Parameters;
  enum name = __traits(identifier, fun);
  static foreach (ovl; __traits(getOverloads, __traits(parent, fun), name)) {
    @(__traits(getAttributes, fun))
    auto ref mixin(name)(Parameters!ovl args)
    {
      return ovl(args);
    }
  }
}

This doesn’t work because a string mixin can only be used to create expressions or statements. Therefore, the solution is to simply expand the mixin to encompass the entire function definition. The token-quote operator q{} is very handy for this:

template forward(alias fun)
{
  import std.format : format;
  import std.traits: Parameters;
  enum name = __traits(identifier, fun);
  static foreach (ovl; __traits(getOverloads, __traits(parent, fun), name)) {
    mixin(q{
        @(__traits(getAttributes, fun))
          auto ref %s(Parameters!ovl args)
        {
          return ovl(args);
        }
      }.format(name));
  }
}

Though string mixins are powerful, they are essentially C macros. For many D programmers, resorting to a string mixin can feel like a defeat.

Let us now move on to a similar, yet significantly more difficult, challenge:

Write a class template that mocks an interface.

For example:

interface JsonSerializable
{
  string asJson() const;
}

void main()
{
  auto mock = new Mock!JsonSerializable();
}

Extrapolating the techniques acquired during the previous challenge, a beginner would probably try this first:

class Mock(alias Interface) : Interface
{
  import std.format : format;
  import std.traits: Parameters;
  static foreach (member; __traits(allMembers, Interface)) {
    static foreach (fun; __traits(getOverloads, Interface, member)) {
      mixin(q{
          @(__traits(getAttributes, fun))
          auto ref %s(Parameters!fun args)
          {
            // record call
            static if (!is(ReturnType!fun == void)) {
              return ReturnType!fun.init;
            }
          }
        }.format(member));
    }
  }
}

Alas, this fails to compile, throwing errors like:

Error: function `challenge.Mock!(JsonSerializable).Mock.asJson` return type
inference is not supported if may override base class function

In other words, auto cannot be used here. We have to fall back to explicitly specifying the return type:

class Mock(alias Interface) : Interface
{
  import std.format : format;
  import std.traits: Parameters, ReturnType;
  static foreach (member; __traits(allMembers, Interface)) {
    static foreach (fun; __traits(getOverloads, Interface, member)) {
      mixin(q{
          @(__traits(getAttributes, fun))
          ReturnType!fun %s(Parameters!fun args)
          {
            // record call
            static if (!is(ReturnType!fun == void)) {
              return ReturnType!fun.init;
            }
          }
        }.format(member));
    }
  }
}

This will not handle ref functions though. What about adding a ref in front of the return type, like we did in the first challenge?

// as before
          ref ReturnType!fun %s(Parameters!fun args) ...

This will fail with all the functions in the interface that do not return a reference.

The reason why everything worked almost magically in the first challenge is that we called the wrapped function inside the template. It enabled the compiler to deduce almost all of the characteristics of the original function and copy them to the forwarder function. But we have no model to copy from here. The compiler will copy some of the aspects of the function (pure, @safe, etc.) to match those of the overriden function, but not some others (ref, const, and the other modifiers).

Then, there is the issue of the function modifiers: const, immutable, shared, and static. These are yet another category of function “aspects”.

At this point, there is no other option than to analyze some of the function attributes by means of traits, and convert them to a string to be injected in the string mixin:

      mixin(q{
          @(__traits(getAttributes, fun))
          %sReturnType!fun %s(Parameters!fun args)
          {
            // record call
            static if (!is(ReturnType!fun == void)) {
              return ReturnType!fun.init;
            }
          }
        }.format(
            (functionAttributes!fun & FunctionAttribute.const_ ? "const " : "")
          ~ (functionAttributes!fun & FunctionAttribute.ref_ ? "ref " : "")
          ~ ...,
          member));
    }

If you look at the implementation of std.typecons.wrap, you will see that part of the code deals with synthesizing bits of a string mixin for the storage classes and modifiers.

The Ugly

So far, we have looked at the function storage classes, modifiers, and UDAs, but we have merely passed the parameter list as a single, monolithic block. However, sometimes we need to perform adjustments to the parameter list of the generated function. This may seem far-fetched, but it does happen. I encountered this problem in my openmethods library. During the “Perfect forwarding” discussion, it appeared that I was not the only one who wanted to do this.

I won’t delve into the details of openmethods here (see an older blog post for an overview of the module); for the purpose of this article, it suffices to say that, given a function declaration like this one:

Matrix times(virtual!Matrix a, double b);

openmethods generates this function:

Matrix dispatcher(Matrix a, double b)
{
  return resolve(a)(a, b);
}

The virtual template is a marker; it indicates which parameters should be taken into account (i.e., passed to resolve) when picking the appropriate specialization of times. Note that only a is passed to the resolve function—that is because the first parameter uses the virtual! marker and the second does not.

Bear in mind, though, that dispatcher is not allowed to use the type of the parameters directly. Inside the openmethods module, there is no Matrix type. Thus, when openmethods is handed a function declaration, it needs to synthesize a dispatcher function that refers to the declaration’s parameter types exclusively via the declaration. In other words, it needs to use the ReturnType and Parameters templates from std.traits to extract the types involved in the declaration – just like we did in the examples above.

Let’s put aside function attributes and UDAs – we already discussed those in the previous section. The obvious solution then seems to be:

ReturnType!times dispatcher(
  RemoveVirtual!(Parameters!times[0]) a, Parameters!times[1] b)
{
  return resolve(a)(a, b);
}

pragma(msg, typeof(&dispatcher)); // Matrix function(Matrix, double)

where RemoveVirtual is a simple template that peels off the virtual! marker from the type.

Does this preserve parameter storage classes and UDAs? Unfortunately, it does not:

@nogc void scale(ref virtual!Matrix m, lazy double by);

@nogc ReturnType!scale dispatcher(RemoveVirtual!(Parameters!scale[0]) a, Parameters!scale[1] b)
{
  return resolve(a)(a, b);
}

pragma(msg, typeof(&dispatcher)); // void function(Matrix a, double b)

We lost the ref on the first parameter and the lazy on the second. What happened to them?

The culprit is Parameters. This template is a wrapper around an obscure feature of the is operator used in conjunction with the __parameters type specialization. And it is quite a strange beast. We used it above to copy the parameter list of a function, as a whole, to another function, and it worked perfectly. The problem is what happens when you try to process the parameters separately. Let’s look at a few examples:

pragma(msg, Parameters!scale.stringof); // (ref virtual!(Matrix), lazy double)
pragma(msg, Parameters!scale[0].stringof); // virtual!(Matrix)
pragma(msg, Parameters!scale[1].stringof); // double

We see that accessing a parameter individually returns the type… and discards everything else!

There is actually a way to extract everything about a single parameter: use a slice instead of an element of the paramneter pack (yes, this is getting strange):

pragma(msg, Parameters!scale[0..1].stringof); // (ref virtual!(Matrix))
pragma(msg, Parameters!scale[1..2].stringof); // (lazy double)

So this gives us a solution for handling the second parameter of scale:

ReturnType!scale dispatcher(???, Parameters!scale[1..2]) { ... }

But what can we put in place of ???. RemoveVirtual!(Parameters!scale[0..1]) would not work. RemoveVirtual expects a type, and Parameters!scale[1..2] is not a type—it is a sort of conglomerate that contains a type, and perhaps storage classes, type constructors, and UDAs.

At this point, we have no other choice but to construct a string mixin once again. Something like this:

mixin(q{
    %s ReturnType!(scale) dispatcher(
      %s RemoveVirtual!(Parameters!(scale)[1]) a,
      Parameters!(scale)[1..2] b)
    {
        resolve(a)(a, b);
    }
  }.format(
    functionAttributes!scale & FunctionAttribute.nogc ? "@nogc " : ""
    /* also handle other function attributes */,
    __traits(getParameterStorageClasses, scale, 0)));

pragma(msg, typeof(dispatcher)); // @nogc void(ref double a, lazy double)

This is not quite sufficient though, because it still doesn’t take care of parameter UDAs.

To Boltly Refract…

openmethods once contained kilometers of mixin code like the above. Such heavy use of string mixins was too ugly and messy, so much so that the project began to feel less like fun and more like work. So I decided to sweep all the ugliness under a neat interface, once and for all. The result was a “refraction” module, which I later carved out of openmethods and donated to Ali Akhtarzada’s excellent bolts library. bolts attempts to fill in the gaps and bring some regularity to D’s motley set of meta-programming features.

refraction’s entry point is the refract function template. It takes a function and an “anchor” string, and returns an immutable Function object that captures all the aspects of a function. Function objects can be used at compile-time. It is, actually, their raison d’être.

Function has a mixture property that returns a declaration for the original function. For example:

Matrix times(virtual!Matrix a, double b);
pragma(msg, refract!(times, "times").mixture);
// @system ReturnType!(times) times(Parameters!(times) _0);

Why does refract need the anchor string? Can’t the string "times" be inferred from the function by means of __traits(identifier...)? Yes, it can, but in real applications we don’t want to use this. The whole point of the library is to be used in templates, where the function is typically passed to refract via an alias. In general, the function’s name has no meaning in the template’s scope—or if, by chance, the name exists, it does not name the function. All the meta-expressions used to dissect the function must work in terms of the local symbol that identifies the alias.

Consider:

module matrix;

Matrix times(virtual!Matrix a, double b);

Method!times timesMethod; // openmethods creates a `Method` object for each
                          // declared method

module openmethods;

struct Method(alias fun)
{
    enum returnTypeMixture = refract!(fun, "fun").returnType;
    pragma(msg, returnTypeMixture);              // ReturnType!(fun)
    mixin("alias R = ", returnTypeMixture, ";"); // ok
    pragma(msg, R.stringof);                     // Matrix
}

There is no times and no Matrix in module openmethods. Even if they existed, they could not be the times function and the Matrix class from module matrix, as this would require a circular dependency between the two modules, something that D forbids by default. However, there is a fun symbol, and it aliases to the function; thus, the return type can be expressed as ReturnType!(fun).

All aspects of the function are available piecemeal. For example:

@nogc void scale(ref virtual!Matrix m, lazy double by);
pragma(msg, refract!(scale, "scale").parameters[0].storageClasses); // ["ref"]

Function also has methods that return a new Function object, with an alteration to one of the aspects. They can be used to create a variation of a function. For example:

pragma(msg,
  refract!(scale, "scale")
  .withName("dispatcher")
  .withBody(q{{ resolve(_0[0])(_0); }})
  .mixture
);

@nogc @system ReturnType!(scale) dispatcher(ref Parameters!(scale)[0] _0, lazy Parameters!(scale)[1] _1)
{
  resolve(_0[0])(_0);
}

This is the reason behind the name “refraction”: the module creates a blueprint of a function, performs some alterations on it, and returns a string—called a mixture—which, when passed to mixin, will create a new function.

openmethods needs to change the type of the first parameter while preserving storage classes. With bolts.experimental.refraction, this becomes easy:

original = refract!(scale, "scale");

pragma(msg,
  original
  .withName("dispatcher")
  .withParameters(
    [original.parameters[0].withType(
        "RemoveVirtual!(%s)".format(original.parameters[0].type)),
     original.parameters[1],
    ])
  .withBody(q{{
      return resolve(_0)(%s);
   }}.format(original.argumentMixture))
);

This time, the generated code splits the parameter pack into individual components:

@nogc @system ReturnType!(scale) dispatcher(
  ref RemoveVirtual!(Parameters!(scale)[0]) _0, Parameters!(scale)[1..2] _1)
{
  return resolve(_0)(_0);
}

Note how the first and second parameters are handled differently. The first parameter is cracked open because we need to replace the type. That forces us to access the first Parameters value via indexing, and that loses the storage classes, UDAs, etc. So they need to be re-applied explicitly.

On the other hand, the second parameter does not have this problem. It is not edited; thus, the Parameters slice trick can be used. The lazy is indeed there, but it is inside the parameter conglomerate.

Conclusion

Initially, D looked almost as good as Lisp for generating functions. As we tried to gain finer control of the generated function, our code started to look a lot more like C macros; in fact, in some respects, it was even worse: we had to put an entire function definition in a string mixin just to set its name.

This is due to the fact that D is not as “regular” a language as Lisp. Some of the people helming the evolution of D are working on changing this, and it is my hope that an improved D will emerge in the not-too-distant future.

In the meantime, the experimental refraction module from the bolts meta-programming library offers a saner, easier way of generating functions without compromising on the idiosyncrasies that come with them. It allows you to pretend that functions can be disassembled and reassembled at will, while hiding all the gory details of the string mixins that are necessarily involved in that task.


Jean-Louis Leroy is not French, but Belgian. He got his first taste of programming from a HP-25 calculator. His first real programming language was Forth, where CTFE is pervasive. Later he programmed (a little) in Lisp and Smalltalk, and (a lot) in C, C++, and Perl. He now works for Bloomberg LP in New York. His interests include object-relational mapping, open multi-methods, DSLs, and language extensions in general.

Symmetry Investments and the D Language Foundation are Hiring

Digital Mars D logo

The D Language Foundation is hiring! Thanks to generous funding from Symmetry Investments, we are looking to fill two (mostly) non-programming positions geared toward improving the D ecosystem. Symmetry is also offering a bounty for a specific improvement to DUB, the D build tool and package manager. And on top of all of that, they are hiring D programmers.

D Pull Request/Issue Manager

A lot of good work goes into the D Programming Language GitHub repositories. Unfortunately, some of that good work sometimes gets left behind. A similar story can be told for our Bugzilla database, where some issues are fixed almost as soon as they’re reported and others fall victim to a lack of attention. Efforts have been made in the past to tidy things up, but without someone in a position to permanently keep at it, it’s a task that is never complete.

The D Language Foundation is looking for one or two motivated individuals to take on that permanent position, get the work done, and keep things running smoothly. Symmetry Investements is generously funding this role with $50,000 per year for one person, or $25,000 per year for each of two.

The ideal candidate is someone who:

  • is familiar with git, GitHub, and Bugzilla;
  • is familiar enough with D to be able to review simple pull requests;
  • is able to recognize when more specialized reviews are required and
  • is able to proofread English text (for reviewing documentation and web site pull requests).

Examples of the role’s responsibilities include:

  • ensuring all pull requests follow procedure;
  • reviewing simple pull requests;
  • finding appropriate reviewers for more complex pull requests;
  • ensuring that pull requests are reviewed in a timely manner;
  • reviving stale pull requests;
  • coordinating between pull request submitters and reviewers to prevent pull requests from going stale;
  • closing pull requests that are no longer valid;
  • identifying Bugzilla issues that are duplicates or invalid;
  • identifying Bugzilla issues that are candidates for bounties;
  • publicizing Bugzilla issues in need of a champion and
  • other related tasks.

We are hoping to hire from within the D community, though we will accept queries from anyone. If you are interested in taking on the role, please send your resume to social@dlang.org. You should also indicate if you are willing to do the job full time (just you) or part time (share the responsibilities with someone else).

Community Relations Assistant

I’ve been working with the D Language Foundation for the past three years. Much of what I do falls loosely in the category of Community Relations. These days, I’m in need of an assistant. Symmetry Investments is providing $600 per month for the role.

The job will involve a number of different activities as the need arises, such as:

  • seeking out guest authors and projects to highlight for the D Blog;
  • monitoring our social media accounts;
  • sending out messages from the D Language Foundation (such as thank you notes to new donors);
  • assisting with maintenance of pages at dlang.org and dconf.org;
  • assisting with the organization of events like DConf and SAOC and
  • any odd jobs that pop up now and again.

If you have good communication skills, an optimistic disposition, and enthusiasm for the D Programming Language, I’d like to talk to you. I don’t need a resume. Instead, please send an email to social@dlang.org explaining why you’re the right person for the job.

DUB Bounty

Symmetry Investments logoDUB has become a critical component in the D ecosystem. A significant number of projects depend on it and we need it to be able to meet a wide range of project needs. To that end, there are certainly improvements to be made. One such is in how DUB determines which of a project’s source files are in need of recompilation. Currently, DUB follows in the tradition of the venerable make and uses timestamp comparisons to make that determination.

A new generation of version control and build tools (git, buck, bazel, scons, waf, plz, and more) rely on file checksums to assess the need for action. This is a much more robust approach because it detects actual changes in file content. Timestamps can change in any number of irrelevant ways. Robustness is important if one is to depend on a build working properly even when files are moved, copied, and shared across people, machines, and teams. As hashes are fast to compute on modern hardware, the impact on speed is very low.

Symmetry Investments is offering a $2,000 bounty to the programmer who either converts DUB’s use of timestamp-dependent builds to use SHA-1 hashing throughout, or implements it as a global option to preserve the current behavior.

For inspiration, see this clip from Linus Torvald’s Google talk, and the article Build-Systems Should Use Hashes Over Timestamps. Note that shasum $(git ls-files) in Phobos takes 0.05 seconds on a warm SSD drive in a desktop machine.

Anyone interested in taking on this bounty should contact social@dlang.org beforehand. Anyone interested in contributing to the bounty amount can do so via the bounty card Support for Hash-Based Recompilation in DUB at our Task Bounties page.

Symmetry Autumn of Code 2020 Projects and Participants

Symmetry Investments logoThe verdict is in! Five programmers will be participating in the 2020 edition of the Symmetry Autumn of Code. Over the next three weeks, they will be working with their mentors to take the goals they outlined in their applications and turn them into concrete tasks across four milestones. Then, on September 15th, the first milestone gets under way.

Throughout the event, anyone can follow the progress of each project through the participants’ weekly updates in the General forum. Please don’t ignore those posts! You might be able to offer suggestions to help them along the way.

And now a little about the SAOC 2020 participants and their projects.

  • Robert Aron is a fourth-year Computer Science student at University POLITEHNICA of Bucharest. For his project, he’ll be implementing D client libraries for the Google APIs. When it’s complete, we’ll be able to interact with Google service APIs, such as GDrive, Calendar, Search, and GMail, directly from D. The goal is to complete the project by the end of the event.
  • Michael Boston is currently developing a game in D. For his SAOC project, he’ll be taking some custom data structures he’s developed and adapting them to be more generic. The ultimate goal is to get Michael’s modified implementation merged into Phobos. Should that not happen, the library will still be part of the D ecosystem once it’s complete.
  • Mihaela Chirea is a fourth-year Computer Engineering student at University POLITEHNICA of Bucharest. Mihaela will spend SAOC 2020 improving DMD as a library. Part of this project will involve soliciting community feedback regarding proposed changes, so anyone interested in using DMD as a library should keep an eye out for Mihaela’s posts in the D forums.
  • Teona Severin is a first-year master’s degree student at University POLITEHNICA of Bucharest who will be working on a mini DRuntime in order to bring D to low-performance microcontrollers based on ARM Cortex-M CPUs. Currently, D can run on such systems when compiled as -betterC, but the end goal of this project is to get enough of a functional DRuntime to write “a simple application that actively uses a class.“
  • Adela Vais is yet another fourth-year Computer Engineering student at University POLITEHNICA of Bucharest. For her project, she’ll be implementing a new D backend for GNU Bison. Currently, Bison has an experimental LALR1 parser for D. Adela’s implementation will be a GLR parser intended to overcome the limitations of the LALR1 parser.

The strong showing from Bucharest is down to the work of Razvan Nitzu and Eduard Staniloiu. They have introduced a number of students to the D programming language and encouraged them in seeking out projects beneficial both to their education and to the D community. Plus, Razvan and Edi will be participating in SAOC 2020 as mentors.

On behalf of the SAOC 2020 committee, the D Language Foundation, and Symmetry Investments, I want to thank everyone who submitted an application and wish the participants the best of luck in the coming months.

Deadlines and New Swag

SAOC 2020 Application Deadline

Symmetry Investments logoThe deadline for Symmetry Autumn of Code (SAOC) 2020 applications is on August 16th. There’s work to be done and money to be paid (courtesy of Symmetry Investments). If you know of a project that can keep an eager programmer busy for at least 20 hours a week over the course of four months, please advertise it in the forums and add it to the project ideas list if it isn’t already there.

As for potential applicants, remember that experience with D is not necessary. Experience with another language can be transferred to D “on the job”, with a mentor to guide the way. Tell your friends and spread the word to other programming communities. This is a great way to bring new faces to the D community and the new ideas they may bring with them. All the information on how to apply and how to become a mentor is available on the SAOC 2020 page.

DConf Online 2020 Swag

DConf Online 2020 Logo

The DConf Online 2020 submission deadline of August 31 will be here before we know it. If you haven’t put a submission together yet, head over to the DConf Online 2020 home page (and/or the announcement here on the blog) for the details on what we’re looking for and how to go about it. Everyone whose submission is accepted for the event schedule will receive a t-shirt and coffee mug to commemorate the occasion.

For everyone else, those t-shirts and mugs are on sale now at the DLang Swag Emporium along with tote bags and stickers. Remember, all of the money we raise through the store goes straight into the General Fund, which we’ll dip into to provide DConf Online 2020 speakers with their free stuff and a few lucky viewers with prizes. Donations made directly to the General Fund, or to any of our ongoing campaigns, are also greatly appreciated.

The ABC’s of Templates in D

D was not supposed to have templates.

Several months before Walter Bright released the first alpha version of the DMD compiler in December 2001, he completed a draft language specification in which he wrote:

Templates. A great idea in theory, but in practice leads to numbingly complex code to implement trivial things like a “next” pointer in a singly linked list.

The (freely available) HOPL IV paper, Origins of the D Programming Language, expands on this:

[Walter] initially objected to templates on the grounds that they added disproportionate complexity to the front end, they could lead to overly complex user code, and, as he had heard many C++ users complain, they had a syntax that was difficult to understand. He would later be convinced to change his mind.

It did take some convincing. As activity grew in the (then singular) D newsgroup, requests that templates be added to the language became more frequent. Eventually, Walter started mulling over how to approach templates in a way that was less confusing for both the programmer and the compiler. Then he had an epiphany: the difference between template parameters and function parameters is one of compile time vs. run time.

From this perspective, there’s no need to introduce a special template syntax (like the C++ style <T>) when there’s already a syntax for parameter lists in the form of (T). So a template declaration in D could look like this:

template foo(T, U) {
    // template members here
}

From there, the basic features fell into place and were expanded and enhanced over time.

In this article, we’re going to lay the foundation for future articles by introducing the most basic concepts and terminology of D’s template implementation. If you’ve never used templates before in any language, this may be confusing. That’s not unexpected. Even though many find D’s templates easier to understand than other implementations, the concept itself can still be confusing. You’ll find links at the end to some tutorial resources to help build a better understanding.

Template declarations

Inside a template declaration, one can nest any other valid D declaration:

template foo(T, U) {
    int x;
    T y;

    struct Bar {
        U thing;
    }

    void doSomething(T t, U u) {
        ...
    }
}

In the above example, the parameters T and U are template type parameters, meaning they are generic substitutes for concrete types, which might be built-in types like int, float, or any type the programmer might implement with a class or struct declaration. By declaring a template, it’s possible to, for example, write a single implementation of a function like doSomething that can accept multiple types for the same parameters. The compiler will generate as many copies of the function as it needs to accomodate the concrete types used in each unique template instantiation.

Other kinds of parameters are supported: value parameters, alias parameters, sequence (or variadic) parameters, and this parameters, all of which we’ll explore in future blog posts.

One name to rule them all

In practice, it’s not very common to implement templates with multiple members. By far, the most common form of template declaration is the single-member eponymous template. Consider the following:

template max(T) {
    T max(T a, T b) { ... }
}

An eponymous template can have multiple members that share the template name, but when there is only one, D provides us with an alternate template declaration syntax. In this example, we can opt for a normal function declaration that has the template parameter list in front of the function parameter list:

T max(T)(T a, T b) { ... }

The same holds for eponymous templates that declare an aggregate type:

// Instead of the longhand template declaration...
/* 
template MyStruct(T, U) {
    struct MyStruct { 
        T t;
        U u;
    }
}
*/

// ...just declare a struct with a type parameter list
struct MyStruct(T, U) {
    T t;
    U u;
}

Eponymous templates also offer a shortcut for instantiation, as we’ll see in the next section.

Instantiating templates

In relation to templates, the term instantiate means essentially the same as it does in relation to classes and structs: create an instance of something. A template instance is, essentially, a concrete implementation of the template in which the template parameters have been replaced by the arguments provided in the instantiation. For a template function, that means a new copy of the function is generated, just as if the programmer had written it. For a type declaration, a new copy of the declaration is generated, just as if the programmer had written it.

We’ll see an example, but first we need to see the syntax.

Explicit instantiations

An explicit instantiation is a template instance created by the programmer using the template instantiation syntax. To easily disambiguate template instantiations from function calls, D requires the template instantiation operator, !, to be present in explicit instantiations. If the template has multiple members, they can be accessed in the same manner that members of aggregates are accessed: using dot notation.

import std;

template Temp(T, U) {
    T x;
    struct Pair {
        T t;
        U u;
    }
}

void main()
{
    Temp!(int, float).x = 10;
    Temp!(int, float).Pair p;
    p.t = 4;
    p.u = 3.2;
    writeln(Temp!(int, float).x);
    writeln(p);            
}

Run it online at run.dlang.io

There is one template instantiation in this example: Temp!(int, float). Although it appears three times, it refers to the same instance of Temp, one in which T == int and U == float. The compiler will generate declarations of x and the Pair type as if the programmer had written the following:

int x;
struct Pair {
    int t;
    float u;
}

However, we can’t just refer to x and Pair by themselves. We might have other instantiations of the same template, like Temp!(double, long), or Temp(MyStruct, short). To avoid conflict, the template members must be accessed through a namespace unique to each instantiation. In that regard, Temp!(int, float) is like a class or struct with static members; just as you would access a static x member variable in a struct Foo using the struct name, Foo.x, you access a template member using the template name, Temp!(int, float).x.

There is only ever one instance of the variable x for the instantiation Temp!(int float), so no matter where you use it in a code base, you will always be reading and writing the same x. Hence, the first line of main isn’t declaring and initializing the variable x, but is assigning to the already declared variable. Temp!(int, float).Pair is a struct type, so that after the declaration Temp!(int, float).Pair p, we can refer to p by itself. Unlike x, p is not a member of the template. The type Pair is a member, so we can’t refer to it without the prefix.

Aliased instantiations

It’s possible to simplify the syntax by using an alias declaration for the instantiation:

import std;

template Temp(T, U) {
    T x;
    struct Pair {
        T t;
        U u;
    }
}
alias TempIF = Temp!(int, float);

void main()
{
    TempIF.x = 10;
    TempIF.Pair p = TempIF.Pair(4, 3.2);
    writeln(TempIF.x);
    writeln(p);            
}

Run it online at run.dlang.io

Since we no longer need to type the template argument list, using a struct literal to initialize p, in this case TempIF.Pair(3, 3.2), looks cleaner than it would with the template arguments. So I opted to use that here rather than first declare p and then initialize its members. We can trim it down still more using D’s auto attribute, but whether this is cleaner is a matter of preference:

auto p = TempIF.Pair(4, 3.2);

Run it online at run.dlang.io

Instantiating eponymous templates

Not only do eponymous templates have a shorthand declaration syntax, they also allow for a shorthand instantiation syntax. Let’s take the x out of Temp and rename the template to Pair. We’re left with a Pair template that provides a declaration struct Pair. Then we can take advantage of both the shorthand declaration and instantiation syntaxes:

import std;

struct Pair(T, U) {
    T t;
    U u;
}

// We can instantiate Pair without the dot operator, but still use
// the alias to avoid writing the argument list every time
alias PairIF = Pair!(int, float);

void main()
{
    PairIF p = PairIF(4, 3.2);
    writeln(p);            
}

Run it online at run.dlang.io

The shorthand instantiation syntax means we don’t have to use the dot operator to access the Pair type.

Even shorter syntax

When a template instantiation is passed only one argument, and the argument’s symbol is a single token (e.g., int as opposed to int[] or int*), the parentheses can be dropped from the template argument list. Take the standard library template function std.conv.to as an example:

void main() {
    import std.stdio : writeln;
    import std.conv : to;
    writeln(to!(int)("42"));
}

Run it online at run.dlang.io

std.conv.to is an eponymous template, so we can use the shortened instantiation syntax. But the fact that we’ve instantiated it as an argument to the writeln function means we’ve got three pairs of parentheses in close proximity. That sort of thing can impair readability if it pops up too often. We could move it out and store it in a variable if we really care about it, but since we’ve only got one template argument, this is a good place to drop the parentheses from the template argument list.

writeln(to!int("42"));

Whether that looks better is another case where it’s up to preference, but it’s fairly idiomatic these days to drop the parentheses for a single template argument no matter where the instantiation appears.

Not done with the short stuff yet

std.conv.to is an interesting example because it’s an eponymous template with multiple members that share the template name. That means that it must be declared using the longform syntax (as you can see in the source code), but we can still instantiate it without the dot notation. It’s also interesting because, even though it accepts two template arguments, it is generally only instantiated with one. This is because the second template argument can be deduced by the compiler based on the function argument.

For a somewhat simpler example, take a look at std.utf.toUTF8:

void main()
{    
    import std.stdio : writeln;
    import std.utf : toUTF8;
    string s1 = toUTF8("This was a UTF-16 string."w);
    string s2 = toUTF8("This was a UTF-32 string."d);
    writeln(s1);
    writeln(s2);
}

Run it online at run.dlang.io

Unlike std.conv.to, toUTF8 takes exactly one parameter. The signature of the declaration looks like this:

string toUTF8(S)(S s)

But in the example, we aren’t passing a type as a template argument. Just as the compiler was able to deduce to’s second argument, it’s able to deduce toUTF8’s sole argument.

toUTF8 is an eponymous function template with a template parameter S and a function parameter s of type S. There are two things we can say about this: 1) the return type is independent of the template parameter and 2) the template parameter is the type of the function parameter. Because of 2), the compiler has all the information it needs from the function call itself and has no need for the template argument in the instantiation.

Take the first call to the toUTF8 function in the declaration of s1. In long form, it would be toUTF8!(wstring)("blah"w). The w at the end of the string literal indicates it is of type wstring, with UTF-16 encoding, as opposed to string, with UTF-8 encoding (the default for string literals). In this situation, having to specify !(wstring) in the template instantiation is completely redundant. The compiler already knows that the argument to the function is a wstring. It doesn’t need the programmer to tell it that. So we can drop the template instantiation operator and the template argument list, leaving what looks like a simple function call. The compiler knows that toUTF8 is a template, knows that the template is declared with one type parameter, and knows that the type should be wstring.

Similarly, the d suffix on the literal in the initialization of s2 indicates a UTF-32 encoded dstring, and the compiler knows all it needs to know for that instantiation. So in this case also, we drop the template argument and make the instantiation appear as a function call.

It does seem silly to convert a wstring or dstring literal to a string when we could just drop the w and d prefixes and have string literals that we can directly assign to s1 and s2. Contrived examples and all that. But the syntax the examples are demonstrating really shines when we work with variables.

wstring ws = "This is a UTF-16 string"w;
string s = ws.toUTF8;
writeln(s);

Run it online at run.dlang.io

Take a close look at the initialization of s. This combines the shorthand template instantiation syntax with Uniform Function Call Syntax (UFCS) and D’s shorthand function call syntax. We’ve already seen the template syntax in this post. As for the other two:

  • UFCS allows using dot notation on the first argument of any function call so that it appears as if a member function is being called. By itself, it doesn’t offer much benefit aside from, some believe, readability. Generally, it’s a matter of preference. But this feature can seriously simplify the implementation of generic templates that operate on aggregate types and built-in types.
  • The parentheses in a function call can be dropped when the function takes no arguments, so that foo() becomes foo. In this case, the function takes one argument, but we’ve taken it out of the argument list using UFCS, so the argument list is now empty and the parentheses can be dropped. (The compiler will lower this to a normal function call, toUTF8(ws)—it’s purely programmer convenience.) When and whether to do this in the general case is a matter of preference. The big win, again, is in the simplification of template implementations: a template can be implemented to accept a type T with a member variable foo or a member function foo, or a free function foo for which the first argument is of type T.

All of this shorthand syntax is employed to great effect in D’s range API, which allows chained function calls on types that are completely hidden from the public-facing API (aka Voldemort types).

More to come

In future articles, we’ll explore the different kinds of template parameters, introduce template constraints, see inside a template instantiation, and take a look at some of the ways people combine templates with D’s powerful compile-time features in the real world. In the meantime, here are some template tutorial resources to keep you busy:

  • Ali Çehreli’s Programming in D is an excellent introduction to D in general, suitable even for those with little programming experience. The two chapters on templates (the first called ‘Templates’ and the second ‘More Templates’) provide a great introduction. (The online version of the book is free, but if you find it useful, please consider throwing Ali some love by buying the ebook or print version linked at the top of the TOC.)
  • More experienced programmers may find Phillipe Sigaud’s ‘D Template Tutorial’ a good read. It’s almost a decade old, but still relevant (and still free!). This tutorial goes beyond the basics into some of the more advanced template features. It includes a look at D’s compile-time features, provides a number of examples, and sports an appendix detailing the is expression (a key component of template constraints). It can also serve as a great reference when reading open source D code until you learn your way around D templates.

There are other resources, though other than my book ‘Learning D’ (this is a referral link that will benefit the D Language Foundation), I’m not aware of any that provide as much detail as the above. (And my book is currently not freely available). Eventually, we’ll be able to add this blog series to the list.

Thanks to Stefan Koch for reviewing this article.

DConf Online 2020: Call For Submissions

DConf Online 2020 LogoDConf Online 2020 is happening November 21 & 22, 2020 in your local web browser! We are currently taking submissions for pre-recorded talks, livstreamed panels, and livecoding events. See the DConf Online 2020 web site for details on how you can participate. Keep reading here for more info on how it came together and what we hope to achieve, as well as for a reminder about the 2020 edition of the Symmetry Autumn of Code (the SAOC 2020 registration deadline is just over three weeks away!).

Maybe Next Time, London!

Due to the onset of COVID-19, the D Language Foundation and Symmetry Investments decided in early March to cancel DConf 2020, which had been scheduled to take place June 17–20 in London. DConf has been the premiere D programming language event every year since 2013, the one chance for members of the D community from around the world to gather face-to-face outside of their local meetups for four days of knowledge sharing and comradery. It was a painful decision, but the right one. As of now, we can’t say for sure there will be a DConf 2021, but it’s looking increasingly unlikely.

Immediately upon reaching the decision to cancel DConf, the obvious question arose of whether we should take the conference online. It was something none of the DConf organizers had any experience with, so we were unwilling to commit to anything until we could figure out a way to go about it that makes sense for our community. As time progressed and we explored our options, the idea became more attractive. Finally, we settled on an approach that we think will work for our community while still allowing outsiders to easily drop by to get a look at our favorite programming language.

We also decided that this is not going to be an online substitute for the real-world DConf. That’s why we’ve named it DConf Online 2020 and not DConf 2020 Online. We’re planning to make this an annual event. The real-world DConf will still take place in spring or summer (barring pandemics or other unforeseen circumstances), and DConf Online six months or so later. Without the DConf cancellation, we never would have reached this point, so for us that’s a bit of a bright side to these dark days.

DConf on YouTube

DConf Online will take place on the D Language Foundation’s YouTube Channel. The event will kick off with a pre-recorded keynote from Walter Bright, the creator and co-maintainer of D, on November 21, scheduled to premiere at a yet-to-be-determined time. Other pre-recorded talks will be scheduled to premiere throughout the weekend, including a Day Two keynote on November 22 from co-maintainer Átila Neves. Presenters from the pre-recorded talks will be available for livestreamed question and answer sessions just as they would be in the real-world DConf.

We’ll also be livestreaming an Ask Us Anything session, a DConf tradition, with Walter and Átila. We’re looking for other ideas for livestream panels. Anyone submitting a panel proposal should either be willing to moderate the panel or have already found someone to commit to the position.

And we really, really want to have at least two livecoding sessions. Anyone familiar with D who has experience livecoding is welcome to submit a proposal. Ideally, we’re looking for sessions that present a solid demonstration of D in use, preferably a small project designed exclusively for the livestream, something that can be developed from start to finish in no more than 90 minutes. We aren’t looking for tutorial style sessions that go into great detail on a feature or two (though that sort of thing is great for a pre-recorded talk submission!), but something that shows how a D program comes together and what D features look like in action.

Everything you need to know to submit a pre-recorded talk, panel, or livecoding session to the D Language Foundation can be found at the DConf Online 2020 web site. We’ll have more details here and on the web site in the coming weeks as our plans solidify. Oh, and everyone whose submission is accepted will receive some swag from the DLang Swag Emporium (DConf Online 2020 swag is coming soon).

BeerConf

It’s a DConf tradition that a gathering spot is selected where attendees can get together each evening for drinks, food, and conversation. For many attendees, this is a highlight of the conference. The opportunity to engage in conversation with so many smart, like-minded people is not one to be missed. Ethan Watson dubbed these evening soirees “BeerConf”, and the name has stuck.

Recently, Ethan and other D community members have been gathering for a monthly online #BeerConf. Given that it’s such an integral part of the DConf experience, we hope to make use of the lessons they’re learning to run a BeerConf in parallel to DConf Online, starting on the 20th. Despite the name, no one will be expected to drink alcohol of any kind. It’s all about getting together to socialize as close to face-to-face as we can get online.

More details regarding BeerConf will be announced closer to the conference dates, so keep an eye on the blog!

SAOC 2020

Symmetry Autumn of Code is an annual event where a handful of lucky programmers get paid to write some D code. Sponsored by Symmetry Investments, SAOC 2020 is the third edition of the event. Although priority is given to university students, SAOC is open to anyone over 18.

Applicants send a project proposal and a short bio to the D Language Foundation. Those who are selected will be required to work on their project at least 20 hours per week from September 15, 2020, until January 15, 2021. The event consists of four milestones. Participants who meet their goals for the first three milestones will each receive a payment of $1000. For the fourth milestone, the SAOC Committee will evaluate each participant’s progress for the entire event. On that basis, one will be selected to receive a final $1000 payment and a free trip to the next real-world DConf (no registration fee; travel and lodging expenses reimbursed on the same terms as offered to DConf speakers). The lucky participant will be asked to submit a proposal for the same DConf they attend, but their proposal will be evaluated in the same manner as all proposals (i.e., acceptance is not guaranteed), but they are guaranteed free registration and reimbursement regardless.

Although Roberto Romaninho, our SAOC 2019 selectee, was robbed of the opportunity to attend DConf 2020, he will still be eligible to make use of his reward at our next real-world event along with the 2020 selectee. Francesco Gallà, who was selected in the inaugural SAOC 2018, gave a presentation about his project and the SAOC experience at DConf 2019. The runner up, Franceso Mecca, wrote about his own project for the D Blog.

SAOC 2020 applications are open until August 16. See the SAOC 2020 page for all the details on how to apply.

A Pattern for Head-mutable Structures

When Andrei Alexandrescu introduced ranges to the D programming language, the gap between built-in and user-defined types (UDTs) narrowed, enabling new abstractions and greater composability. Even today, though, UDTs are still second-class citizens in D. One example of this is support for head mutability—the ability to manipulate a reference without changing the referenced value(s). This document details a pattern that will further narrow the UDT gap by introducing functions for defining and working with head-mutable user-defined types.

Introduction

D is neither Kernel nor Scheme—it has first-class and second-class citizens. Among its first-class citizens are arrays and pointers. One of the benefits these types enjoy is implicit conversion to head-mutable. For instance, const(T[]) is implicitly convertible to const(T)[]. Partly to address this difference, D has many ways to define how one type may convert to or behave like another – alias this, constructors, opDispatch, opCast, and, of course, subclassing. The way pointers and dynamic arrays decay into their head-mutable variants is different from the semantics of any of these features, so we would need to define a new type of conversion if we were to mimic this behavior.

Changing the compiler and the language to permit yet another way of converting one type into another is not desirable: it makes the job harder for compiler writers, makes an already complex language even harder to learn, and any implicit conversion can make code harder to read and maintain. If we can define conversions to head-mutable data structures without introducing compiler or language changes, this will also make the feature available to users sooner, since such a mechanism would not necessarily require changes in the standard library, and users could gradually implement it in their own code and benefit from the code in the standard library catching up at a later point.

Unqual

The tool used today to get a head-mutable version of a type is std.traits.Unqual. In some cases, this is the right tool—it strips away one layer of const, immutable, inout, and shared. For some types though, it either does not give a head-mutable result, or it gives a head-mutable result with mutable indirections:

struct S(T) {
    T[] arr;
}

With Unqual, this code fails to compile:

void foo(T)(T a) {
    Unqual!T b = a; // cannot implicitly convert immutable(S!int) to S!int
}

unittest {
    immutable s = S!int([1,2,3]);
    foo(s);
}

A programmer who sees that message hopefully finds a different way to achieve the same goal. However, the error message says that the conversion failed, indicating that a conversion is possible, perhaps even without issue. An inexperienced programmer, or one who knows that doing so is safe right now, could use a cast to shut the compiler up:

void bar(T)(T a) {
    Unqual!T b = cast(Unqual!T)a;
    b.arr[0] = 4;
}

unittest {
    immutable s = S!int([1,2,3]);
    bar(s);
    assert(s.arr[0] == 1); // Fails, since bar() changed it.
}

If, instead of S!int, the programmer had used int[], the first example would have compiled, and the cast in the second example would have never seen the light of day. However, since S!int is a user-defined type, we are forced to write a templated function that either fails to compile for some types it really should support or gives undesirable behavior at run time.

headMutable()

Clearly, we should be able to do better than Unqual, and in fact we can. D has template this parameters which pick up on the dynamic type of the this reference, and with that, its const or immutable status:

struct S {
    void foo(this T)() {
        import std.stdio : writeln;
        writeln(T.stringof);
    }
}
unittest {
    S s1;
    const S s2;
    s1.foo(); // Prints "S".
    s2.foo(); // Prints "const(S)".
}

This way, the type has the necessary knowledge of which type qualifiers a head-mutable version needs. We can now define a method that uses this information to create the correct head-mutable type:

struct S(T) {
    T[] arr;
    auto headMutable(this This)() const {
        import std.traits : CopyTypeQualifiers;
        return S!(CopyTypeQualifiers!(This, T))(arr);
    }
}
unittest {
    const a = S!int([1,2,3]);
    auto b = a.headMutable();
    assert(is(typeof(b) == S!(const int))); // The correct part of the type is now const.
    assert(a.arr is b.arr); // It's the same array, no copying has taken place.
    b.arr[0] = 3; // Correctly fails to compile: cannot modify const expression.
}

Thanks to the magic of Uniform Function Call Syntax, we can also define headMutable() for built-in types:

auto headMutable(T)(T value) {
    import std.traits;
    import std.typecons : rebindable;
    static if (isPointer!T) {
        // T is a pointer and decays naturally.
        return value;
    } else static if (isDynamicArray!T) {
        // T is a dynamic array and decays naturally.
        return value;
    } else static if (!hasAliasing!(Unqual!T)) {
        // T is a POD datatype - either a built-in type, or a struct with only POD members.
        return cast(Unqual!T)value;
    } else static if (is(T == class)) {
        // Classes are reference types, so only the reference may be made head-mutable.
        return rebindable(value);
    } else static if (isAssociativeArray!T) {
        // AAs are reference types, so only the reference may be made head-mutable.
        return rebindable(value);
    } else {
        static assert(false, "Type "~T.stringof~" cannot be made head-mutable.");
    }
}
unittest {
    const(int*[3]) a = [null, null, null];
    auto b = a.headMutable();
    assert(is(typeof(b) == const(int)*[3]));
}

Now, whenever we need a head-mutable variable to point to tail-const data, we can simply call headMutable() on the value we need to store. Unlike the ham-fisted approach of casting to Unqual!T, which may throw away important type information and also silences any error messages that may inform you of the foolishness of your actions, attempting to call headMutable() on a type that doesn’t support it will give an error message explaining what you tried to do and why it didn’t work (“Type T cannot be made head-mutable.”). The only thing missing now is a way to get the head-mutable type. Since headMutable() returns a value of that type, and is defined for all types we can convert to head-mutable, that’s a template one-liner:

import std.traits : ReturnType;
alias HeadMutable(T) = ReturnType!((T t) => t.headMutable());

Where Unqual returns a type with potentially the wrong semantics and only gives an error once you try assigning to it, HeadMutable disallows creating the type in the first place. The programmer will have to deal with that before casting or otherwise coercing a value into the variable. Since HeadMutable uses headMutable() to figure out the type, it also gives the same informative error message when it fails.

Lastly, since one common use case requires us to preserve the tail-const or tail-immutable properties of a type, it is beneficial to define a template that converts to head-mutable while propagating const or immutable using std.traits.CopyTypeQualifiers:

import std.traits : CopyTypeQualifiers;
alias HeadMutable(T, ConstSource) = HeadMutable!(CopyTypeQualifiers!(ConstSource, T));

This way, immutable(MyStruct!int) can become MyStruct!(immutable int), while the const version would propagate constness instead of immutability.

Example Code

Since the pattern for range functions in Phobos is to have a constructor function (e.g. map) that forwards its arguments to a range type (e.g. MapResult), the code changes required to use headMutable() are rather limited. Likewise, user code should generally not need to change at all in order to use headMutable(). To give an impression of the code changes needed, I have implemented map and equal:

import std.range;

// Note that we check not if R is a range, but if HeadMutable!R is
auto map(alias Fn, R)(R range) if (isInputRange!(HeadMutable!R)) {
    // Using HeadMutable!R and range.headMutable() here.
    // This is basically the extent to which code that uses head-mutable data types will need to change.
    return MapResult!(Fn, HeadMutable!R)(range.headMutable());
}

struct MapResult(alias Fn, R) {
    R range;
    
    this(R _range) {
        range = _range;
    }
    
    void popFront() {
        range.popFront();
    }
    
    @property
    auto ref front() {
        return Fn(range.front);
    }
    
    @property
    bool empty() {
        return range.empty;
    }
    
    static if (isBidirectionalRange!R) {
        @property
        auto ref back() {
            return Fn(range.back);
        }

        void popBack() {
            range.popBack();
        }
    }

    static if (hasLength!R) {
        @property
        auto length() {
            return range.length;
        }
        alias opDollar = length;
    }

    static if (isRandomAccessRange!R) {
        auto ref opIndex(size_t idx) {
            return Fn(range[idx]);
        }
    }

    static if (isForwardRange!R) {
        @property
        auto save() {
            return MapResult(range.save);
        }
    }
    
    static if (hasSlicing!R) {
        auto opSlice(size_t from, size_t to) {
            return MapResult(range[from..to]);
        }
    }
    
    // All the above is as you would normally write it.
    // We also need to implement headMutable().
    // Generally, headMutable() will look very much like this - instantiate the same
    // type template that defines typeof(this), use HeadMutable!(T, ConstSource) to make
    // the right parts const or immutable, and call headMutable() on fields as we pass
    // them to the head-mutable type.
    auto headMutable(this This)() const {
        alias HeadMutableMapResult = MapResult!(Fn, HeadMutable!(R, This));
        return HeadMutableMapResult(range.headMutable());
    }
}

auto equal(R1, R2)(R1 r1, R2 r2) if (isInputRange!(HeadMutable!R1) && isInputRange!(HeadMutable!R2)) {
    // Need to get head-mutable version of the parameters to iterate over them.
    auto _r1 = r1.headMutable();
    auto _r2 = r2.headMutable();
    while (!_r1.empty && !_r2.empty) {
        if (_r1.front != _r2.front) return false;
        _r1.popFront();
        _r2.popFront();
    }
    return _r1.empty && _r2.empty;
}

unittest {
    // User code does not use headMutable at all:
    const arr = [1,2,3];
    const squares = arr.map!(a => a*a);
    const squaresPlusTwo = squares.map!(a => a+2);
    assert(equal(squaresPlusTwo, [3, 6, 11]));
}

(Note that these implementations are simplified slightly from Phobos code to better showcase the use of headMutable)

The unittest block shows a use case where the current Phobos map would fail—it is perfectly possible to create a const MapResult, but there is no way of iterating over it. Note that only two functions are impacted by the addition of headMutable(): map tests if HeadMutable!R is an input range and converts its arguments to head-mutable when passing them to MapResult, and MapResult needs to implement headMutable(). The rest of the code is exactly as you would otherwise write it.

The implementation of equal() shows a situation where implicit conversions would be beneficial. For const(int[]) the call to headMutable() is superfluous—it is implicitly converted to const(int)[] when passed to the function. For user-defined types however, this is not the case, so the call is necessary in the general case.

While I have chosen to implement a range here, ranges are merely the most common example of a place where headmutable would be useful; the idea has merits beyond ranges. Another type in the standard library that would benefit from headmutable is RefCounted!T: const(RefCounted!(T)) should convert to RefCounted!(const(T)).

Why not Tail-Const?

In previous discussions of this problem, the solution has been described as tail-const, and a function tailConst() has been proposed. While this idea might at first seem the most intuitive solution, it has some problems, which together make headMutable() far superior.

The main problem with tailConst() is that it does not play well with D’s existing const system. It needs to be called on a mutable value, and there is no way to convert a const(Foo!T) to Foo!(const(T)). It thus requires that the programmer explicitly call tailConst() on any value that is to be passed to a function expecting a non-mutable value and, abstain from using const or immutable to convey the same information. This creates a separate world of tail-constness and plays havoc with generic code, which consequently has no way to guarantee that it won’t mutate its arguments.

Secondly, the onus is placed on library users to call tailConst() whenever they pass an argument anywhere, causing an inversion of responsibility: the user has to tell the library that it is not allowed to edit the data instead of the other way around. In the best case, this merely causes unnecessary verbiage. In other cases, the omission of const will lead to mutation of data expected to be immutable.

A minor quibble in comparison is that the tail-const solution also requires the existence of tailImmutable to cover the cases where the values are immutable.

Issues

The ideas outlined in this document concern only conversion to head-mutable. A related issue is conversion to tail const, e.g. from RefCounted!T or RefCounted!(immutable T) to RefCounted!(const T), a conversion that, again, is implicit for arrays and pointers in D today.

One issue that may be serious is the fact that headMutable often cannot be @safe and may, in fact, need to rely on undefined behavior in some places. For instance, RefCounted!T contains a pointer to the actual ref count. For immutable(RefCounted!T), headMutable() would need to cast away immutable, which is undefined behavior per the spec.

The Compiler Solution

It is logical to think that, as with built-in types, headMutable() could be elided in its entirety, and the compiler could handle the conversions for us. In many cases, this would be possible, and in fact the compiler already does so for POD types like struct S { int n; }—a const or immutable S may be assigned to a mutable variable of type S. This breaks down, however, when the type includes some level of mutable indirection. For templated types it would be possible to wiggle the template parameters to see if the resulting type compiles and has fields with the same offsets and similar types, but even such an intelligent solution breaks down in the presence of D’s Turing-complete template system, and some cases will always need to be handled by the implementer of a type.

It is also a virtue that the logic behind such an implementation be understandable to the average D programmer. The best case result of that not being true is that the forums would be inundated with a flood of posts about why types don’t convert the way users expect them to.

For these reasons, headMutable() will be necessary even with compiler support. But what would that support look like? Implicit casting to head-mutable happens in the language today in two situations:

  • Assignment to head-mutable variables: const(int)[] a = create!(const(int[]))(); (all POD types, pointers and arrays)
  • Function calls: fun(create!(const(int[]))(); (only pointers and arrays)

The first is covered by existing language features (alias headMutable this; fits the bill perfectly). The second is not but is equivalent to calling .headMutable whenever a const or immutable value is passed to a function that does not explicitly expect a const or immutable argument. This would change the behavior of existing code, in that templated functions would prefer a.headMutable over a, but would greatly improve the experience of working with const types that do define headMutable(). If headMutable is correctly implemented, the different choice of template instantiations should not cause any actual breakage.

Future Work

While this document proposes to implement the described feature without any changes to the compiler or language, it would be possible for the compiler in the future to recognize headMutable() and call it whenever a type that defines that method is passed to a function that doesn’t explicitly take exactly that type, or upon assignment to a variable that matches headMutable()’s return value. This behavior would mirror the current behavior of pointers and arrays.

Conclusion

It is possible to create a framework for defining head-mutable types in D today without compiler or language changes. It requires a little more code in the methods that use head-mutable types but offers a solution to a problem that has bothered the D community for a long time.

While this document deals mostly with ranges, other types will also benefit from this pattern: smart pointers and mutable graphs with immutable nodes are but two possible examples.

Definitions

Head-mutable

A type is head-mutable if some or all of its members without indirections are mutable. Note that a head-mutable datatype may also have const or immutable members without indirections; the requirement is merely that some subset of its members may be mutated. A head-mutable datatype may be tail-const, tail-immutable or tail-mutable—head-mutable only refers to its non-indirected members. Examples of head-mutable types include const(int)[], int*, string, and Rebindable!MyClass. Types without indirections (like int, float and struct S { int n; }) are trivially head-mutable.

Tail-const

A type is tail-const if some of its members with indirections have the const type qualifier. A tail-const type may be head-mutable or head-const. Examples of tail-const types are const(int)*, const(int[]), const(immutable(int)[])* and string.

Source

The source code for HeadMutable and headMutable is available here.

SAOC 2020 and Other News

Symmetry Autumn of Code 2020

Symmetry Investments logo

The 3rd annual Symmetry Autumn of Code (SAoC) is on!

From now until August 16th, we’re accepting applications from motivated coders interested in getting paid to improve the D ecosystem. The SAoC committee will review all submissions and, based on the quality of the applications received, select a number of applicants to complete four milestones from September 15th to January 15th. Each participant will receive $1000 for the successful completion of each of the first three milestones, and one of them will receive an additional $1000 and a free trip (reimbursement for transportation and accommodation, and free registration) to the next real-world DConf (given the ongoing pandemic, we can’t yet be sure when that will be).

Anyone interested in programming D is welcome to apply, but preference will be given to those who can provide proof of enrollment in undergraduate or postgraduate university programs. For details on how to apply, see the SAoC 2020 page here at the D Blog.

The participants will need mentors, so we invite experienced D programmers interested in lending a hand to get in touch and to keep an eye out in the forums for any SAoC applicants in search of a mentor. As with the previous edition of SAoC, all mentors whose mentee completes the event will be guaranteed a one-time payment of $500 after the final milestone (mentors of unsuccessful mentees may still be eligible for the payment at the discretion of the SAoC committee). Potential mentors can follow the same link for details on their responsibilities and how to make themselves available.

We’re also looking for community input on potential SAoC projects. If there’s any work you’re aware of that needs doing in the D ecosystem and which may keep a lone coder occupied for 20 hours per week over four months, please let us know! Once again, details on how submit your suggestions and what sort of information we’re looking for can be found on the SAoC 2020 page.

Our SAoC 2019 selectee, Roberto Rosmaninho, was all set to attend DConf 2020 and we were all looking forward to meeting him. He’ll still be eligible to claim his free DConf trip at the next available opportunity.

SAoC would not be possible without the generosity of Symmetry Investments. A big thanks to them for once again funding this event and for the other ways, both financial and otherwise, they contribute back to the D programming language community.

Finances

Thanks to everyone who has shopped in the DLang Swag Emporium! To date, the D Language Foundation has received over $177 in royalties and referral fees. Thanks are also in order to those who have supported the foundation through smile.amazon.com. Your purchases have brought over $288 into the General Fund. Amazon Smile is perhaps the easiest way to support D financially if you shop through Amazon’s .com domain (the D Language Foundation is unavailable in other Amazon domains). If you’ve never done so, you can select a charitable foundation (the D Language Foundation, of course) on your first visit to smile.amazon.com. Then, every time you shop through that link, the foundation will receive a small percentage of your total purchase. Check your browser’s extension market for plugins that convert every amazon.com link to a smile.amazon.com link!

On the Task Bounties front, we may have closed out a big bounty for bringing D to iOS and iPadOS, but there are still several other bounties waiting to be claimed. The latest, currently at $220, is a bounty to improve DLL support on Windows by closing two related Bugzilla issues; 50% of the total bounty will be paid for the successful closure (merged PR and DMD release) of each issue. We welcome anyone interested in fixing these issues to either up the bounty or roll up their sleeves and start working toward claiming it. If you’d like to contribute to multiple bounties with a single credit card payment, or seed one or more new bounties with a specific amount, visit the Task Bounty Catch-All and follow the instructions there.

Finally, the question was recently raised in the forums about how to view the D Language Foundation’s finances. Because the foundation is a 501(3)(c) non-profit public charity, the Form 990 that the organization is required to submit to the IRS every year is publicly available. There are different ways you can obtain the documents for multiple years, such as searching online databases or contacting the IRS directly. Several websites, such as grantspace.org, provide details on how to do so. The Form 990 does not break down specific expenditures or sources of income except for special circumstances (like scholarship payments). With Andrei’s help, I’m currently working on gathering up more information on the past five years of the foundation’s finances so that we can put up an overview page at dlang.org. It won’t be at line-item detail, but we hope to provide a little more detail than the Form 990. I can’t provide a timeline on when it will be available (I don’t consider it a high priority task, so I’m working on it sporadically), but expect it sometime in the next few months.

DConf Online?

Rumor has it that online conferences are actually a thing. Voices in the wind speak of the potential for an annual event related to D. I don’t usually listen to voices I hear in the wind, but this time I’m intrigued…