Category Archives: Guest Posts

Memoization in the D Programming Language

The D programming language provides advanced facilities for structuring programs logically, almost like Python or Ruby, but with high performance and the higher reliability of static typing and contract programming.

In this article, I will describe how to use D templates and mixins for memoization, that is, to automatically remember a function (or property) result.

std.functional.memoize from the standard library

The first way is built straight into Phobos, the D standard library, and is very easy to use:

import std.functional;
import std.stdio;

float doCalculations() {
    writeln("Doing calculations.");
    return -1.7; // the value of the calculations
}

// Apply the template “memoize” to the function doCalculations():
alias doCalculationsOnce = memoize!doCalculations;

Now the alias doCalculationsOnce() does the same as doCalculations(), but the calculations are done only once (if we call doCalculationsOnce() then “Doing calculations.” would be printed only once). This is useful for long slow calculations and in some other situations (like to create only a single window on the screen). That is memoization.

It’s even possible to memoize a function with arguments:

float doCalculations2(float arg1, float arg2) {
    writeln("Doing calculations.");
    return arg1 + arg2; // the value of the calculations
}

// Apply the template “memoize” to the function  doCalculations2():
alias doCalculationsOnce2 = memoize!doCalculations2;

void main(string[] args)
{
    writeln(doCalculationsOnce2(1.0, 1.2));
    writeln(doCalculationsOnce2(1.0, 1.3));
    writeln(doCalculationsOnce2(1.0, 1.3));
}

This outputs:

Doing calculations.
2.2
Doing calculations.
2.3
2.3

You see that the calculations are not repeated again when the argument values are the same.

Memoizing struct or class properties

I’ve found another way to memoize in D. It involves caching a property of a struct or class. Properties are zero-argument (for reading) or one-argument (for writing) member functions (or sometimes two-argument non-member functions) and differ only in syntax. I deemed it more elegant to cache a property of a struct or class rather than to cache a member function’s return value. My code can easily be changed to memoize a member function instead of a property, but you can always convert zero-argument member functions into a property, so why bother?

Here is the source (you can also install it from https://github.com/vporton/memoize-dlang for use in your programs):

module memoize;

mixin template CachedProperty(string name, string baseName = '_' ~ name) {
    mixin("private typeof(" ~ baseName ~ ") " ~ name ~ "Cache;");
    mixin("private bool " ~ name ~ "IsCached = false;");
    mixin("@property typeof(" ~ baseName ~ ") " ~ name ~ "() {\n" ~
          "if (" ~ name ~ "IsCached" ~ ") return " ~ name ~ "Cache;\n" ~
          name ~ "IsCached = true;\n" ~
          "return " ~ name ~ "Cache = " ~ baseName ~ ";\n" ~
          '}');
}

It is used like this (with either structs or classes at your choice):

import memoize;

struct S {
    @property float _x() { return 1.5; }
    mixin CachedProperty!"x";
}

Then just:

S s;
assert(s.x == 1.5);

Or you can specify the explicit name of the cached property:

import memoize;

struct S {
    @property float _x() { return 1.5; }
    mixin CachedProperty!("x", "_x");
}

CachedProperty is a template mixin. Template mixins insert the declarations of the template body directly into the current context. In this case, the template body is composed of string mixins. As you can guess, a string mixin generates code at compile time from strings. So,

struct S {
    @property float _x() { return 1.5; }
    mixin CachedProperty!"x";
}

turns into

struct S {
    @property float _x() { return 1.5; }
    private typeof(_x) xCache;
    private bool xIsCached = false;
    @property typeof(_x) x() {
        if (xIsCached) return xCache;
        xIsCached = true;
        return xCache = _x;
    }
}

That is, it sets xCache to _x unless xIsCached and also sets xIsCached to true when retrieving x.

The idea originates from the following Python code:

class cached_property(object):
    """A version of @property which caches the value.  On access, it calls the
    underlying function and sets the value in `__dict__` so future accesses
    will not re-call the property.
    """
    def __init__(self, f):
        self._fname = f.__name__
        self._f = f

    def __get__(self, obj, owner):
        assert obj is not None, 'call {} on an instance'.format(self._fname)
        ret = obj.__dict__[self._fname] = self._f(obj)
        return ret

My way of memoization does not (yet) involve caching return values of functions with arguments.

Victor Porton is an open source developer, a math researcher, and a Christian writer. He earns his living as a programmer.

Using const to Enforce Design Decisions

The saying goes that the best code is no code. As soon as a project starts to grow, technical debt is introduced. When a team is forced to adapt to a new company guideline inconsistent with their previous vision, the debt results from a business decision. This could be tackled at the company level. Sometimes technical debt can arise simply due to the passage of time, when new versions of dependencies or of the compiler introduce breaking changes. You can try to tackle this by letting your local physicist stop the flow of time. More often however, technical debt is caused when issues are fixed by a quick hack, due to time pressure or a lack of knowledge of the code base. Design strategies that were carefully crafted are temporarily neglected. This blog post will focus on using the const modifier. It is one of the convenient tools D offers to minimize the increase of technical debt and enforce design decisions.

To keep a code base consistent, often design guidelines, either explicit or implicit, are put in place. Every developer on the team is expected to adhere to the guidelines as a gentleman’s agreement. This effectively results in a policy that is only enforced if both the programmer and the reviewer have had enough coffee. Simple changes, like adding a call to an object method, might seem innocent, but can reduce the consistency and the traceability of errors. To detect this in a code review requires in-depth knowledge of the method’s implementation.

An example from a real world project I’ve worked on is generating financial transactions for a read-only view, the display function in the following code fragment. Nothing seemed wrong with it, until I realized that those transactions were persisted and eventually used for actual payments without being calculated again, as seen in the process method. Potentially different payments occurred depending on whether the user decided to glance at the summary, thereby triggering the generation with new currency exchange rates, or just blindly clicked the OK button. That’s not what an innocent bystander like myself expects and has caused many frowns already.

public class Order
{
    private Transaction[] _transactions;

    public Transaction[] getTransactions()
    {
        _transactions = calculate();
        return _transactions;
    }

    public void process()
    {
        foreach(t; _transactions){
            // ...
        }
    }
}

void display(Order order)
{
    auto t = order.getTransactions();
    show(t);
}

The internet has taught me that if it is possible, it will one day happen. Therefore, we should make an attempt to make the undesired impossible.

A constant feature

By default, variables and object instances in D are mutable, just like in many other programming languages. If we want to prevent objects from ever changing, we can mark them immutable, i.e. immutable MyClass obj = new MyClass();. immutable means that we can modify neither the object reference (head constant) nor the object’s properties (tail constant). The first case corresponds to final in Java and readonly in C#, both of which signify head constant only. D’s implementation means that nobody can ever modify an object marked immutable. What if an object needs to be mutable in one place, but immutable in another? That’s where D’s const pops in.

Unlike immutable, whose contract states that an object cannot be mutated through any reference, const allows an object to be modified through another, non-const reference. This means it’s illegal to initialize an immutable reference with a mutable one, but a const reference can be initialized with a mutable, const, or immutable reference. In a function parameter list, const is preferred over immutable because it can accept arguments with either qualifier or none. Schematically, it can be visualized as in the following figure.

const relationships

A constant detour

D’s const differs from C++ const in a significant way: it’s transitive (see the const(FAQ) for more details). In other words, it’s not possible to declare any object in D as head constant. This isn’t obvious from examples with classes, since classes in D are reference types, but is more apparent with pointers and arrays. Consider these C++ declarations:

const int *cp0;         // mutable pointer to const data
int const *cp1;         // alternative syntax for the same

Variable declarations in C++ are best read from right to left. Although const int is likely the more common syntax, int const matches the way the declaration should be read: cp0 is a mutable pointer to a const int. In D, the equivalent of cp0 and cp1 is:

const(int)* dp0;

The next example shows what head constant looks like in C++.

int *const cp2;         // const pointer to mutable data

We can read the declaration of cp2 as: cp2 is a const pointer to a mutable int. There is no equivalent in D. It’s possible in C++ to have any number and combination of const and mutable pointers to const or mutable data. But in D, if const is applied to the outermost pointer, then it applies to all the inner pointers and the data as well. Or, as they say, it’s turtles all the way down.

The equivalent in C++ looks like this:

int const *const cp3;         // const pointer to const data

This declaration says cp3 is a const pointer to const data, and is possible in D like so:

const(int*) dp1;
const int* dp2;     // same as dp1

All of the above syntax holds true for D arrays:

const(int)[] a1;    // mutable reference to const data
const(int[]) a2;    // const reference to const data
const int[] a3;     // same as a2

const in the examples can be replaced with immutable, with the caveat that initializers must match the declaration, e.g. immutable(int)* can only be initialized with immutable(int)*.

Finally, note that classes in D are reference types; the reference is baked in, so applying const or immutable to a class reference is always equivalent to const(p*) and there is no equivalent to const(p)* for classes. Structs in D, on the other hand, are value types, so pointers to structs can be declared like the int pointers in the examples above.

A constant example

For the sake of argument, we assume that updating the currency exchange rates is a function that, by definition, needs to mutate the order. After updating the order, we want to show the updated prices to our user. Conceptually, the display function should not modify the order. We can prevent mutation by adding the const modifier to our function parameter. The implicit rule is now made explicit: the function takes an Order as input, and treats the object as a constant. We no longer have a gentleman’s agreement, but a formal contract. Changing the contract will, hopefully, require thorough negotiation with your peer reviewer.

void updateExchangeRates(Order order);
void display(const Order order);

void updateAndDisplay()
{
    Order order = //…
    updateExchangeRates(order);
    display(order); // Implicitly converted to const.
}

As with all contracts, defining it is the easiest part. The hard part is enforcing it. The D compiler is our friend in this problem. If we try to compile the code, we will get a compiler error.

void display(const Order order)
{
    // ERROR: cannot call mutable method
    auto t = order.getTransactions();
    show(t);
}

We never explicitly stated that getTransactions doesn’t modify the object. As the method is virtual by default, the compiler cannot derive the behavior either way. Without that knowledge, the compiler is required to assume that the method might modify the object. In other words, in the D justice system one is guilty until proven innocent. Let’s prove our innocence by marking the method itself const, telling the compiler that we do not intend to modify our data.

public class Order
{
    private Transaction[] _transactions;

    public Transaction[] getTransactions() const
    {
        _transactions = calculate(); // ERROR: cannot mutate field
        return _transactions;
    }
}

void display(const Order order)
{
    auto t = order.getTransactions(); // Now compiles :)
    show(t);
}

By marking the method const, the original compile error has moved away. The promise that we do not modify any object state is part of the method signature. The compiler is now satisfied with the method call in the display function, but finds another problem. Our getter, which we stated should not modify data, actually does modify it. We found our code smell by formalizing our guidelines and letting the compiler figure out the rest.

It seems promising enough to try it on a real project.

A constant application

I had a pet project lying around and decided to put the effort into enforcing the constraint. This is what inspired me to write this post. The project is a four-player mahjong game. The relevant part, in abstraction, is highlighted in the image.

Mahjong abstraction

The main engine behind the game is the white box in the center. A player or AI is sent a message with a const view of the game data for display purposes and to determine their next move. A message is sent back to the engine, which then determines the mutation on the internally mutable game data. The most obvious win is that I cannot accidentally modify the game data when drawing my UI. Which, of course, appeared to be the case before I refactored in the const-ness of the game data.

Upon closer inspection, coming from the UI there is only one way to manipulate the state of the game. The UI sends a message to the engine and remains oblivious of the state changes that need to be applied. This also encourages layered development and improves testability of the code. So far, so good. But during the refactoring, a problem arose. Recall that marking an object with const means that only member functions that promise not to modify the object, those marked with const themselves, can be called. Some of these could be trivially fixed by applying const or, at worst, inout (a sort of wildcard). However, the more persistent issues, like in the Order example, required me to go back to the drawing board and rethink my domain. In the end, being forced to think about mutability versus immutability improved my understanding of my own code base.

A constant verdict

Is const all good? It’s not a universal answer and certainly has downsides. The most notable one is that this kills lazy initialization where a property is computed only when first requested and then the result is cached. Sometimes, like in the earlier example, this is a code smell, but there are legit use cases. In my game, I have a class that composes the dashboard per player. Updating it is expensive and rarely required. The screen, however, gets rendered sixty times a second. It makes sense to cache the dashboards and only update them when the player objects change. I could split the method in two, but then my abstraction would leak its optimization. I settled for not using const here, as it was a module-private class and didn’t have a large impact on my codebase.

A complaint that is sometimes heard regarding const is that it does not work well with one of D’s main selling points, ranges. A range is D’s implementation of a lazy iterator, usable in foreach loops and heavily used in std.algorithm. The functions in std.algorithm can handle ranges with const elements perfectly fine. However, iterating a range changes the internal values and ultimately consumes the range. Therefore, when a range itself is const, it cannot be iterated over. I think this makes sense by design, but I can imagine that this behavior can be surprising in edge cases. I haven’t encountered this as I generate new ranges on every query, both on mutable and const objects.

A guideline for when to use const would be to separate the queries from the command, a.k.a. ye olde Command-Query Separation (CQS). All queries should, in principle, be const. To perform a mutation, even on nested objects, one should call a method on the object. I’d double down on this and state that member functions should be commands, with logic that can be overridden, and therefore never be marked constant. Queries basically serve as a means to peek at encapsulated data and don’t need to be overridden. This should be a final, non-virtual, function that simply exposes a read-only view on the inner field. For example, using D’s module-private modifier on the field in conjunction with an acquainted function in the same module, we can put the logic inside the class definition and the queries outside.

// in order.d
public class Order
{
    private Transaction[] _transactions; // Accessible in order.d

    public void process(); // Virtual and not restricted
}

public const(Transaction)[] getTransactions(const Order order)
{
    // Function, not virtual, operates on a read-only view of Order
    return order._transactions;
}

// in view.d
void display(const Order order)
{
    auto t = order.getTransactions();
    show(t);
}

We should take care, however, not to overapply the modifier. The question that we need to answer is not “Do I modify this object here?”, but rather, “Does it make sense that the object is modified here?” Wrongly assuming constant objects will result in trouble when you finally need to change the instance due to a new feature. For example, in my game a central Game class contains the players’ hands, but doesn’t explicitly modify them. However, given the structure of my code, it does not make sense to make the player objects constant, as free functions in the engine do use mutable player instances of the game object.

Reflecting on my design, even when writing this blog post, gave me valuable insights. Taking the effort to properly use the tool called const has paid off for me. It improved the structure of my code and improved my understanding of the ramblings I write. It is like any other tool not a silver bullet. It serves to formalize our gentleman’s agreement and is therefore just as fragile as one.

Marco graduated in physics, were he used Fortran and Matlab. He used Programming in D to learn application programming. After 3 years of being a C# developer, he is now trainer-coach of mainly Java and C# developers at Sogyo. Marco uses D for programming experiments and side projects including his darling mahjong game.

Containerize Your D Server Application

A container consists of an application packed together with all of its required dependencies. The container is run as an isolated process on Linux or Windows. The Docker tool has made the handling of containers very popular and is now the de-facto standard for deploying containers to a cloud environment. In this blog post I discuss how to create a simple vibe.d application and ship it as a Docker container.

Setting up the build environment

I use Ubuntu 18.10 as my development environment for this application. Additionally, I installed the packages ldc (the LLVM-based D compiler), dub (the D package manager and build tool), gcc, zlib1g-dev and libssl-dev (all required for compiling my vibe.d application). To build and run my container I use Docker CE. I installed it following the instructions at https://docs.docker.com/install/linux/docker-ce/ubuntu/. As the last step, I added my user to the docker group (sudo adduser kai docker).

A sample REST application

My vibe.d application is a very simple REST server. You can call the /hello endpoint (with an optional name parameter) and you get back a friendly message in JSON format. The second endpoint, /healthz, is intended as a health check and simply returns the string "OK". You can clone my source repository at https://github.com/redstar/vibed-docker/ to get the source code. Here is the application:

import vibe.d;
import std.conv : to;
import std.process : environment;
import std.typecons : Nullable;

shared static this()
{
    logInfo("Environment dump");
    auto env = environment.toAA;
    foreach(k, v; env)
        logInfo("%s = %s", k, v);

    auto host = environment.get("HELLO_HOST", "0.0.0.0");
    auto port = to!ushort(environment.get("HELLO_PORT", "17890"));

    auto router = new URLRouter;
    router.registerRestInterface(new HelloImpl());

    auto settings = new HTTPServerSettings;
    settings.port = port;
    settings.bindAddresses = [host];

    listenHTTP(settings, router);

    logInfo("Please open http://%s:%d/hello in your browser.", host, port);
}

interface Hello
{
    @method(HTTPMethod.GET)
    @path("hello")
    @queryParam("name", "name")
    Msg hello(Nullable!string name);

    @method(HTTPMethod.GET)
    @path("healthz")
    string healthz();
}

class HelloImpl : Hello
{
    Msg hello(Nullable!string name) @safe
    {
        logInfo("hello called");
        return Msg(format("Hello %s", name.isNull ? "visitor" : name));
    }

    string healthz() @safe
    {
        logInfo("healthz called");
        return "OK";
    }
}

struct Msg
{
    string msg;
}

And this is the dub.sdl file to compile the application:

name "hellorest"
description "A minimal REST server."
authors "Kai Nacke"
copyright "Copyright © 2018, Kai Nacke"
license "BSD 2-clause"
dependency "vibe-d" version="~>0.8.4"
dependency "vibe-d:tls" version="*"
subConfiguration "vibe-d:tls" "openssl-1.1"
versions "VibeDefaultMain"

Compile and run the application with dub. Then open the URL http://127.0.0.1:17890/hello to check that you get a JSON result.

A cloud-native application should follow the twelve-factor app methodology. You can read about the twelve-factor app at https://12factor.net/. In this post I only highlight two of the factors: III. Config and XI. Logs.

Ideally, you build an application only once and then deploy it into different environments, e.g. first to your quality testing environment and then to production. When you ship your application as a container, it comes with all of its required dependencies. This solves the problem that different versions of a library might be installed in different environments, possibly causing hard-to-find errors. You still need to find a solution for how to deal with different configuration settings. Port numbers, passwords or the location of databases are all configuration settings which typically differ from environment to environment. The factor III. Config recommends that the configuration be stored in environment variables. This has the advantage that you can change the configuration without touching a single file. My application follows this recommendation. It uses the environment variable HELLO_HOST for the configuration of the host IP and the variable HELLO_PORT for the port number. For easy testing, the application uses the default values 0.0.0.0 and 17890 in case the variables do not exist. (To be sure that every configuration is complete, it would be safer to stop the application with an error message in case an environment variable is not found.)

The application writes log entries on startup and when a url endpoint is called. The log is written to stdout. This is exactly the point of factor XI. Logs: an application should not bother to handle logs at all. Instead, it should treat logs as an event stream and write everything to stdout. The cloud environment is then responsible for collecting, storing and analyzing the logs.

Building the container

A Docker container is specified with a Dockerfile. Here is the Dockerfile for the application:

FROM ubuntu:cosmic

RUN \
  apt-get update && \
  apt-get install -y libphobos2-ldc-shared81 zlib1g libssl1.1 && \
  rm -rf /var/lib/apt/lists/*

COPY hellorest /

USER nobody

ENTRYPOINT ["/hellorest"]

A Docker container is a stack of read-only layers. With the first line, FROM ubuntu:cosmic, I specify that I want to use this specific Ubuntu version as the base layer of my container. During the first build, this layer is downloaded from Docker Hub. Every other line in the Dockerfile creates a new layer. The RUN line is executed at build time. I use it to install dependent libraries which are needed for the application. The COPY command copies the executable into the root directory inside the container. And last, CMD specifies the command which the container will run.

Run the Docker command

docker build -t vibed-docker/hello:v1 .

to build the Docker container. After the container is built successfully, you can run it with

docker run -p 17890:17890 vibed-docker/hello:v1

Now open again the URL http://127.0.0.1:17890/hello. You should get the same result as before. Congratulations! Your vibe.d application is now running in a container!

Using a multi-stage build for the container

The binary hellorest was compiled outside the container. This creates difficulties as soon as dependencies in your development environment change. It is easy to integrate compiliation into the Dockerfile, but this creates another issue. The requirements for compiling and running the application are different, e.g. the compiler is not required to run the application.

The solution is to use a multi-stage build. In the first stage, the application is build. The second stage contains only the runtime dependencies and application binary built in the first stage. This is possible because Docker allows the copying of files between stages. Here is the multi-stage Dockerfile:

FROM ubuntu:cosmic AS build

RUN \
  apt-get update && \
  apt-get install -y ldc gcc dub zlib1g-dev libssl-dev && \
  rm -rf /var/lib/apt/lists/*

COPY . /tmp

WORKDIR /tmp

RUN dub -v build

FROM ubuntu:cosmic

RUN \
  apt-get update && \
  apt-get install -y libphobos2-ldc-shared81 zlib1g libssl1.1 && \
  rm -rf /var/lib/apt/lists/*

COPY --from=build /tmp/hellorest /

USER nobody

ENTRYPOINT ["/hellorest"]

In my repository I called this file Dockerfile.multi. Therefore, you have to specify the file on the command line:

docker build -f Dockerfile.multi -t vibed-docker/hello:v1 .

Building the container now requires much more time because a clean build of the application is included. The advantage is that your build environment is now independent of your host environment.

Where to go from here?

Using containers is fun. But the fun diminishes as soon as the containers get larger. Using Ubuntu as the base image is comfortable but not the best solution. To reduce the size of your container you may want to try Alpine Linux as the base image, or use no base image as all.

If your application is split over several containers then you can use Docker Compose to manage your containers. For real container orchestration in the cloud you will want to learn about Kubernetes.

A long-time contributor to the D community, Kai Nacke is the author of ‘D Web Development‘ and a maintainer of LDC, the LLVM D Compiler.

Writing a D Wrapper for a C Library

In porting to D a program I created for a research project, I wrote a D wrapper of a C library in an object-oriented manner. I want to share my experience with other programmers. This article provides some D tips and tricks for writers of D wrappers around C libraries.

I initially started my research project using the Ada 2012 programming language (see my article “Experiences on Writing Ada Bindings for a C Library” in Ada User Journal, Volume 39, Number 1, March 2018). Due to a number of bugs that I was unable to overcome, I started looking for another programming language. After some unsatisfying experiments with Java and Python, I settled on the D programming language.

The C Library

We have a C library, written in an object-oriented style (C structure pointers serve as objects, and C functions taking such structure pointers serve as methods). Fortunately for us, there is no inheritance in that C library.

The particular libraries we will deal with are the Redland RDF Libraries, a set of libraries which parse Resource Description Framework (RDF) files or other RDF resources, manages them, enables RDF queries, etc. Don’t worry if you don’t know what RDF is, it is not really relevant for this article.

The first stage of this project was to write a D wrapper over librdf. I modeled it on the Ada wrapper I had already written. One advantage I found in D over Ada is that template instantiation is easier—there’s no need in D to instantiate every single template invocation with a separate declaration. I expect this to substantially simplify the code of XML Boiler, my program which uses this library.

I wrote both raw bindings and a wrapper. The bindings translate the C declarations directly into D, and the wrapper is a new API which is a full-fledged D interface. For example, it uses D types with constructors and destructors to represent objects. It also uses some other D features which are not available in C. This is a work in progress and your comments are welcome.

The source code of my library (forked from Dave Beckett’s original multi-language bindings of his libraries) is available at GitHub (currently only in the dlang branch). Initially, I tried some automatic parsers of C headers which generate D code. I found these unsatisfactory, so I wrote the necessary bindings myself.

Package structure

I put my entire API into the rdf.* package hierarchy. I also have the rdf.auxiliary package and its subpackages for things used by or with my bindings. I will discuss some particular rdf.auxiliary.* packages below.

My mixins

In Ada I used tagged types, which are a rough equivalent of D classes, and derived _With_Finalization types from _Without_Finalization types (see below). However, tagged types increase variable sizes and execution time.

In D I use structs instead of classes, mainly for efficiency reasons. D structs do not support inheritance, and therefore have no virtual method table (vtable), but do provide constructors and destructors, making classes unnecessary for my use case (however, see below). To simulate inheritance, I use template mixins (defined in the rdf.auxiliary.handled_record module) and the alias this construct.

As I’ve said above, C objects are pointers to structures. All C pointers to structures have the same format and alignment (ISO/IEC 9899:2011 section 6.2.5 paragraph 28). This allows the representation of any pointer to a C structure as a pointer to an opaque struct (in the below example, URIHandle is an opaque struct declared as struct URIHandle;).

Using the mixins shown below, we can declare the public structs of our API this way (you should look into the actual source for real examples):

struct URIWithoutFinalize {
    mixin WithoutFinalize!(URIHandle,
                           URIWithoutFinalize,
                           URI,
                           raptor_uri_copy);
    // …
}
struct URI {
    mixin WithFinalize!(URIHandle,
                        URIWithoutFinalize,
                        URI,
                        raptor_free_uri);
}

The difference between the WithoutFinalize and WithFinalize mixins is explained below.

About finalization and related stuff

The main challenge in writing object-oriented bindings for a C library is finalization.

In the C library in consideration (as well as in many other C libraries), every object is represented as a pointer to a dynamically allocated C structure. The corresponding D object can be a struct holding the pointer (aka handle), but oftentimes a C function returns a so-called “shared handle”—a pointer to a C struct which we should not free because it is a part of a larger C object and shall be freed by the C library only when that larger C object goes away.

As such, I first define both (for example) URIWithoutFinalize and URI. Only URI has a destructor. For URIWithoutFinalize, a shared handle is not finalized. As D does not support inheritance for structs, I do it with template mixins instead. Below is a partial listing. See the above URI example on how to use them:

mixin template WithoutFinalize(alias Dummy,
                               alias _WithoutFinalize,
                               alias _WithFinalize,
                               alias copier = null)
{
    private Dummy* ptr;
    private this(Dummy* ptr) {
        this.ptr = ptr;
    }
    @property Dummy* handle() const {
        return cast(Dummy*)ptr;
    }
    static _WithoutFinalize fromHandle(const Dummy* ptr) {
        return _WithoutFinalize(cast(Dummy*)ptr);
    }
    static if(isCallable!copier) {
        _WithFinalize dup() {
            return _WithFinalize(copier(ptr));
        }
    }
    // ...
}


mixin template WithFinalize(alias Dummy,
                            alias _WithoutFinalize,
                            alias _WithFinalize,
                            alias destructor,
                            alias constructor = null)
{
    private Dummy* ptr;
    @disable this();
    @disable this(this);
    // Use fromHandle() instead
    private this(Dummy* ptr) {
        this.ptr = ptr;
    }
    ~this() {
        destructor(ptr);
    }
    /*private*/ @property _WithoutFinalize base() { // private does not work in v2.081.2
        return _WithoutFinalize(ptr);
    }
    alias base this;
    @property Dummy* handle() const {
        return cast(Dummy*)ptr;
    }
    static _WithFinalize fromHandle(const Dummy* ptr) {
        return _WithFinalize(cast(Dummy*)ptr);
    }
    // ...
}

I’ve used template alias parameters here, which allow a template to be parameterized with more than just types. The Dummy argument is the type of the handle instance (usually an opaque struct). The destructor and copier arguments are self-explanatory. For the usage of the constructor argument, see the real source (here it is omitted).

The _WithoutFinalize and _WithFinalize template arguments should specify the structs we define, allowing them to reference each other. Note that the alias this construct makes _WithoutFinalize essentially a base of _WithFinalize, allowing us to use all methods and properties of _WithoutFinalize in _WithFinalize.

Also note that instances of the _WithoutFinalize type may become invalid, i.e. it may contain dangling access values. It seems that there is no easy way to deal with this problem because of the way the C library works. We may not know when an object is destroyed by the C library. Or we may know but be unable to appropriately “explain” it to the D compiler. Just be careful when using this library not to use objects which are already destroyed.

Dealing with callbacks

To deal with C callbacks (particularly when accepting a void* argument for additional data) in an object-oriented way, we need a way to convert between C void pointers and D class objects (we pass D objects as C “user data” pointers). D structs are enough (and are very efficient) to represent C objects like librdf library objects, but for conveniently working with callbacks, classes are more useful because they provide good callback machinery in the form of virtual functions.

First, the D object, which is passed as a callback parameter to C, should not unexpectedly be moved in memory by the D garbage collector. So I make them descendants of this class:

class UnmovableObject {
    this() {
        GC.setAttr(cast(void*)this, GC.BlkAttr.NO_MOVE);
    }
}

Moreover, I add the property context() to pass it as a void* pointer to C functions which register callbacks:

abstract class UserObject : UnmovableObject {
    final @property void* context() const { return cast(void*)this; }
}

When we create a callback we need to pass a D object as a C pointer and an extern(C) function defined by us as the callback. The callback receives the pointer previously passed by us and in the callback code we should (if we want to stay object-oriented) convert this pointer into a D object pointer.

What we need is a bijective (“back and forth”) mapping between D pointers and C void* pointers. This is trivial in D: just use the cast() operator.

How to do this in practice? The best way to explain is with an example. We will consider how to create an I/O stream class which uses the C library callbacks to implement it. For example, when the user of our wrapper requests to write some information to a file, our class receives write message. To handle this message, our implementation calls our virtual function doWriteBytes(), which actually handles the user’s request.

private immutable DispatcherType Dispatch =
    { version_: 2,
      init: null,
      finish: null,
      write_byte : &raptor_iostream_write_byte_impl,
      write_bytes: &raptor_iostream_write_bytes_impl,
      write_end  : &raptor_iostream_write_end_impl,
      read_bytes : &raptor_iostream_read_bytes_impl,
      read_eof   : &raptor_iostream_read_eof_impl };


class UserIOStream : UserObject {
    IOStream record;
    this(RaptorWorldWithoutFinalize world) {
        IOStreamHandle* handle = raptor_new_iostream_from_handler(world.handle,
                                                                  context,
                                                                  &Dispatch);
        record = IOStream.fromNonnullHandle(handle);
    }
    void doWriteByte(char byte_) {
        if(doWriteBytes(&byte_, 1, 1) != 1)
            throw new IOStreamException();
    }
    abstract int doWriteBytes(char* data, size_t size, size_t count);
    abstract void doWriteEnd();
    abstract size_t doReadBytes(char* data, size_t size, size_t count);
    abstract bool doReadEof();
}

And for example:

int raptor_iostream_write_bytes_impl(void* context, const void* ptr, size_t size, size_t nmemb) {
    try {
        return (cast(UserIOStream)context).doWriteBytes(cast(char*)ptr, size, nmemb);
    }
    catch(Exception) {
        return -1;
    }
}

More little things

I “encode” C strings (which can be null) as a D template instance, Nullable!string. If the string is null, the holder is empty. However, it is often enough to transform an empty D string into a null C string (this can work only if we don’t differentiate between empty and null strings). See rdf.auxiliary.nullable_string for an actually useful code.

I would write a lot more advice on how to write D bindings for a C library, but you can just follow my source, which can serve as an example.

Static if

One thing which can be done in D but not in Ada is compile-time comparison via static if. This is a D construct (similar to but more advanced than C conditional preprocessor directives) which allows conditional compilation based on compile-time values. I use static if with my custom Version type to enable/disable features of my library depending on the available features of the version of the base C library in use. In the following example, rasqalVersionFeatures is a D constant defined in my rdf.config package, created by the GNU configure script from the config.d.in file.

static if(Version(rasqalVersionFeatures) >= Version("0.9.33")) {
    private extern extern(C)
    QueryResultsHandle* rasqal_new_query_results_from_string(RasqalWorldHandle* world,
                                                             QueryResultsType type,
                                                             URIHandle* base_uri,
                                                             const char* string,
                                                             size_t string_len);
    static create(RasqalWorldWithoutFinalize world,
                  QueryResultsType type,
                  URITypeWithoutFinalize baseURI,
                  string value)
    {
        return QueryResults.fromNonnullHandle(
            rasqal_new_query_results_from_string(world.handle,
                                                 type,
                                                 baseURI.handle,
                                                 value.ptr, value.length));
    }
}

Comparisons

Order comparisons between structs can be easily done with this mixin:

mixin template CompareHandles(alias equal, alias compare) {
    import std.traits;
    bool opEquals(const ref typeof(this) s) const {
        static if(isCallable!equal) {
          return equal(handle, s.handle) != 0;
        } else {
          return compare(handle, s.handle) == 0;
        }
    }
    int opCmp(const ref typeof(this) s) const {
      return compare(handle, s.handle);
    }
}

Sadly, this mixin has to be called in both the _WithoutFinalization and the _WithFinalization structs. I found no solution to write it once.

Conclusion

I’ve found that D is a great language for writing object-oriented wrappers around C libraries. There are some small annoyances like using class wrappers around structs for callbacks, but generally, D wraps up around C well.

Victor Porton is an open source developer, a math researcher, and a Christian writer. He earns his living as a programmer.

Last Year In D

2018 was one of the biggest years in D we’ve had for a while. It was the first complete year that DMD stuck to a steady release schedule. Over the course of the year, the language got new features and cruft cleanup. The compiler even got some new user-facing features. At the same time, the community-at-large continued to evolve the surrounding ecosystem.

Release Rundown

The first DMD release of the year was version 2.078 on January 1. It set the tone for much of the year, including more support for a stripped-down runtime for -betterC mode, and a deprecation change in arithmetic rules.

The next major release came out in March, and I personally found it to be the most exciting release in a couple of years because it came with improved error messages – notably, calling out the specific argument to a function that mismatched a type (though I will note there is still a lot of room for further improvement!). Moreover, this release introduced DMD’s -i flag, which automatically includes imported modules and has proven to be a major convenience to me over the year.

Meanwhile, more stripped-down runtime support and general language cleanup was included in the release. March also introduced experimental @nogc exception throwing (as proposed in DIP1008). This release also changed the garbage collector to be lazily initialized, part of the effort to make things cheaper and pay-as-you-go.

March was also the first time DMD on Windows could do a total build of a 64-bit program without Visual Studio installed because it bundled the LLVM linker.

Keeping to the release schedule, May and July brought us new major versions, which focused on language cleanup, with a lot of deprecations. There was a #dbugfix campaign over the year, and it brought a fix in May’s release: allowing the pow operator (^^) and several std.math functions to be used at compile time. July’s release brought a change in contract syntax, based on the proposal in DIP 1009, allowing small expressions instead of requiring whole blocks.

Version 2.082, in September, finally brought me something I have wanted for a long time: User Defined Attributes on enum members and function parameters, as well as the ability to disable DRuntime’s exception trapping via the --DRT-trapExceptions=0 runtime command-line switch, which allows for easier debugging of uncaught exceptions. Moreover, with this release the D Language Foundation began digitally signing the Windows binary releases, which has helped smooth out the end-user experience when installing and running DMD. It should continue to help solve the false-positive problem we have seen with some virus scanners.

At the end of the year, version 2.083 saw extern(C++) get a big improvement its users have been waiting for: namespaces without scopes, which makes C++ interoperability and code organization a lot easier. Work on making the C++ standard library accessible from D has been progressing throughout the year.

Overall, the year brought a lot of awaited improvements. D is more usable in various runtime library situations (even Phobos has some -betterC support, like RefCounted and Tuple!). The language had over ten deprecations of old cruft like comma expressions, read-modify-write on shared, comma expressions, class allocators (did you even know D still had class allocators? To be honest, I thought they were formally deprecated years ago, but it was actually 2018 when the change officially happened!), and more.

Debugging got better, with the uncaught exception switch, but also allowing the debug statement to escape more attribute restrictions, better error messages out of the compiler, and a -verrors=context switch to show the line right in the message. -betterC debugging aids also improved, with the C assert function being a possibility. I’ll also note that 2019 is already moving further—new assert printing code was merged just recently; we can choose how to control them and whether to use D or C facilities!

Of course, the library continued to get better range support and its support for @nogc and -betterC has grown as well.

And as a fun fact, DMD is now 98% ported to D, with another major part converted over the year.

The D Community

More companies invested in D in 2018. Hosted by QA Systems, DConf 2018 was held in Munich, Germany, the first time DConf has been to that city (and the third time in Germany, after taking place in Berlin in 2016 and 2017). Symmetry Investments sponsored the successful Symmetry Autumn of Code, funding three students to work on D-related projects.

run.dlang.io came out at the end of 2017 and grew in popularity throughout 2018, becoming the new standard for running D code online, including from the dlang.org homepage. It even supports a whitelist of third-party libraries.

The dlang-tour website gained a few new translations from the community, including Vietnamese, Portugese, French, Turkish, German, and Ukrainian.

The D Language Foundation was busy in 2018 as well. They received over $5000 on Open Collective, $3000 of which was earmarked to support development of the code-d VS Code plugin and the language server that drives it. With a successful campaign launched through Flipcause, the DLF was able to hire a pull request manager for three months. Donations through all platforms allowed them to fund some outreach efforts as well as student work via scholarships. To increase visibility, they submitted a history of D paper to the Fourth ACM SIGPLAN History of Programming Languages Conference.

Over 2,500 pull results have now been merged by the dlang-bot on GitHub, and the Project Tester has now gained 50 projects that it tests on every PR to give real-world data on compiler regressions and the impact of breakages.

DIP 1014, “Hooking D’s struct move semantics”, was also accepted. It will open some doors that were closed by design in old D, but that many developers had found limiting. Before, the compiler could move your structs whenever it wanted. Now, it will be hookable to give more control to the programmer and avoid a nasty case of bugs.

I opened up my documentation generation website at dpldocs.info to all DUB projects this year, and upstream linked to it, encouraging D library authors to better document their libraries and allowing users to better evaluate the libraries before downloading them.

On Twitter, @dlang_ng was started and has gained about 200 followers. All announcement topics from the Announce forum (aka the digitalmars.D.announce newsgroup) are tweeted out here.

2018 also saw community announcements of the autowrap and dpp projects. autowrap automatically wraps existing D code to be used from other environments, while dpp runs a C pre-processor over D code to make it possible to use C headers, unmodified, macros and all, directly in D. These two projects show the community’s desire to integrate D more fully into existing codebases and projects.

Of course, not all is perfect in the D development community, including a few areas where this author would like to see improvement.

The most +1-ed PR on DMD, a string interpolation implementation (and a very elegant approach in this writer’s opinion), remains open and of uncertain status, despite a renewed effort to get it merged in December. Similarly, the State of D 2018 Survey identified several areas for improvement. Very few of these came to pass in 2018, though some progress was made. We deprecated some cruft, but auto-decoding, one of the top-five disliked Phobos features in the survey and one often derided in the forums, remains in place. We are still using Bugzilla, which got an average of 3.27 satisfaction rating out of a possible 5 stars in the survey. We achieved fewer regressions, but added even more to the attribute bloat. We have an official blog and the annual DConf, but the D Language Foundation’s inner workings are still opaque.

It would be great if some of these weak points are addressed among the improvements and changes in 2019.

The DUB Package Manager

The code.dlang.org website received some criticism in 2017, and it was addressed in 2018. The site got a new front page with usage statistics and project ranking based on GitHub info. The DUB program itself got faster and more stable, no longer going online on every run to check for updates, and achieving better online uptime.

The Mir library, which has algorithms, collections, and more written to be 100% -betterC compatible (which, of course, means it also works in all other D environments, too), is now the fifth-most popular package on the DUB registry, while the unit testing library unit-threaded takes the third-place slot. First, second, and fourth are all related to vibe.d, which is the project DUB was originally created to support.

Conclusion

2018 was a really good year for D. There is still much work to do, but worthwhile developments in all facets of the language and ecosystem gave me renewed excitement going into 2019. I expect that March 2019 release of D is going to be another big step toward improving the best programming language we have on Earth today!

Adam Ruppe is the author of D Cookbook and maintainer of This Week in D. Modules from his freely available arsd package are used throughout the D community. He is also known for his legendary DConf 2014 presentation.

Addendum

There were a total of 166 contributors listed on the DMD changelog in 2018. Special thanks to them and to all the others in the D community who flesh out the ecosystem and make our favorite programming language!

  • 0xEAB
  • Adam D. Ruppe
  • Alexandru Caciulescu
  • Alexandru Jercaianu
  • Alexandru ermicioi
  • Alexibu
  • Ali Akhtarzada
  • Ali Çehreli
  • Andrei Alexandrescu
  • Andrei-Cristian VASILE (87585)
  • Andrey Penechko
  • Andy Smith
  • Aravinda VK
  • Arun Chandrasekaran
  • Atila Neves
  • BBasile
  • Basile Burg
  • Bastiaan Veelo
  • Benoit Rostykus
  • Brad Roberts
  • Brian Schott
  • Carsten Schlote
  • Chris Coutinho
  • Daniel Kozak
  • Dashster
  • David Bennett
  • David Gileadi
  • David Nadlinger
  • Denis Feklushkin
  • Diederik de Groot
  • Dmitry Olshansky
  • Dragos Carp
  • Duncan Paterson
  • Eduard Staniloiu
  • Elias Batek
  • Erik van Velzen
  • Eugen Wissner
  • FeepingCreature
  • GabyForceQ
  • Giles Bathgate
  • GoaLitiuM
  • Greg V
  • H. S. Teoh
  • Harry T. Vennik
  • Hiroki Noda
  • Héctor Barreras Almarcha [Dechcaudron]
  • Iain Buclaw
  • Ilya Yaroshenko
  • Ionut
  • Jack Stouffer
  • Jacob Carlborg
  • Jan Jurzitza
  • Jean-Louis Leroy
  • JinShil
  • Joakim Noah
  • Johan Engelen
  • Johannes Loher
  • Johannes Pfau
  • John Belmonte
  • John Colvin
  • Jon Degenhardt
  • Jonathan M Davis
  • Jonathan Marler
  • Jordi Sayol
  • Joseph Rushton Wakeling
  • Kai Nacke
  • Kevin De Keyser
  • Kotet
  • Laeeth Isharc
  • Lance Bachmeier
  • Leandro Lucarella
  • LemonBoy
  • Lucia Mcojocaru
  • Luís Marques
  • Manu Evans
  • Manuel Maier
  • Markus F.X.J. Oberhumer
  • Martin Kinkelin
  • Martin Krejcirik
  • Martin Nowak
  • Mathias Baumann
  • Mathias Lang
  • Mathis Beer
  • MetaLang
  • Michael Parker
  • Mihails Strasuns
  • Mike Franklin
  • Mike Parker
  • Márcio Martins
  • Nathan Sashihara
  • Nemanja Boric
  • Nicholas Lindsay Wilson
  • Nicholas Wilson
  • Nick Treleaven
  • Oleg Nykytenko
  • Patrick Schlüter
  • Paul Backus
  • Per Nordlöw
  • Petar Kirov
  • Pjotr Prins
  • Pradeep Gowda
  • Quirin F. Schroll
  • Radosław Rusiniak
  • Radu Racariu
  • Rainer Schuetze
  • Razvan Nitu
  • Remi Thebault
  • Robert burner Schadek
  • Roman Chistokhodov
  • Ryan David Sheasby
  • Ryan Frame
  • Sebastian Wilzbach
  • Simen Kjærås
  • Simon Naarmann
  • Spoov
  • Stanislav Blinov
  • Stefan Koch
  • Steven Schveighoffer
  • Superstar64
  • Temtaime
  • Tero Hänninen
  • Thibaut CHARLES
  • Thomas Mader
  • Timon Gehr
  • Timoses
  • Timothee Cour
  • Tomáš Chaloupka
  • Tyler Knott
  • Unknown
  • Vlad Vitan
  • Vladimir Panteleev
  • Walter Bright
  • Yannick Koechlin
  • Yuxuan Shui
  • Zach Tollen
  • Zevenberge
  • aG0aep6G
  • abaga129
  • byebye
  • carblue
  • cedretaber
  • crimaniak
  • devel
  • deviator
  • dhasenan
  • dmdw64
  • drug007
  • dukc
  • gapdan
  • glitchbunny
  • growlercab
  • jercaianu
  • jmh530
  • kinke
  • leitimmel
  • look-at-me
  • n8sh
  • rracariu
  • shoo
  • shove70
  • skl131313
  • thaven
  • viktor
  • wazar
  • olframw
  • yashikno
  • Ľudovít Lučenič

D-lighted, I’m Sure

For me, finding D is the most recent step along a road winding back at least a dozen years. I’d been searching for a cross-platform development language/environment (POSIX and Windows, but not so much mobile since my search began before mobile was really a thing) and at this point, D fits better than anything else I’ve tried. I won’t go out on a limb and say it’s the Holy Grail of X-Plat, but at the very least, it’s brought some fun back into coding for me. And whenever I massage a hunk of code until it finally works… well, I’m addicted to those little victories.

The Road to D

The first language I tackled back in the mid-oughts to meet this end was PHP. I’d been a web developer for a few years when I found out that PHP had a standalone desktop version. When I then stumbled across Andrei Zmievski’s PHP-GTK, lightbulbs went on and fireworks went off. For a while. The big drawback I found with PHP-GTK was that no PHP compiler could handle the GTK end. So after a few years of patiently hoping someone would tackle this while I wrote nearly 40 blog posts on its use, I started looking elsewhere.

Back in the 1990’s, I’d been steeped in Javascript and HTML, writing simple online apps for banking clients out of Vancouver, BC and later, Bancroft, Ontario. With such a background, when I ran across Electron a few years ago, it seemed like a good fit. I assumed learning it would be easy-peasy-lemon-squeezy, but a lot about the nature of web development languages had changed. To boot, several more languages, standards, and paradigms had been thrown into the mix, so to say the least, I found it all confusing and more than a little intimidating. What I really wanted was one language, an easy distribution system, and a GUI toolkit that didn’t necessitate balancing style sheets with front- and backend code as well as JSON files. And Electron, unfortunately, needs to drag Chromium along for the ride in a little metaphorical red wagon. It’s a solution, but not the one I was looking for.

Then last year, I stumbled onto D. I’d been hearing about it for years, but I hadn’t read much about it. I didn’t realize it was available across so many POSIX platforms. I also didn’t realize it embraced the OOP paradigm and so hadn’t given it much thought. I liked the subtle humor of D being next in line after B, C and A (which oddly enough came along later than B and C), but after a brief smile, I paid no more attention until last October.

When I finally took a good look, I realized that with its curly braces and OOP propensity, D runs right up my street. But before rolling up my sleeves, I made sure there was a GUI toolkit I could use, something that didn’t necessitate balancing three differing paradigms at once. I found a list of GUI toolkits on the D Wiki and was gratified to see GtkD among them. So for the last two plus months, I’ve been putting most of my effort into learning how D and GtkD work together.

Perspiration and Refreshments

It may or may not be important to know this, but I don’t have a CS degree. I’m completely self-taught, a process that started while stuck in the middle of a frozen nowhere for three weeks more than 30 years ago. But that’s a whole other story. My point is, there are holes in my education. That’s what happens when you follow your nose instead of a syllabus.

Because of that, some of the intricacies of D elude me and may always do so. Although I’ve read Ali Çehreli’s chapter on the subject (from Programming in D) I have no idea what mixins are or why I’d want to use them. And templates seem like a good idea, but I don’t know why. I blame my lack of formal CS education for this, but I’m quite comfortable with classes and objects, so I’m not sweating it.

I was first introduced to OOP and the Gang of Four when I was learning PHP, so D covers familiar territory in that regard. Curly braces are another thing I feel quite at home with, having used them for most of my programming life.

But I’m finding the familiarity of D to be a bit of a stumbling block as well. It’s just different enough from C and PHP to mean I have to work hard at pounding those differences into my brain. I deal with it through rote typing of example code. I figure if I copy out enough D code instead of lazing along with copy-n-paste, eventually I’ll push C and PHP far enough into the background that I can see past them. And I keep Mr. Çehreli’s book handy so I don’t go completely off the rails. It’s been quite helpful. And speaking of helpful…

The Ecology of D

Forums

So far, I’ve signed up for two D-oriented forums, one at dlang.org and the other at gtkd.org. I have yet to find an unfriendly avatar. Everyone I’ve encountered seems willing to jump in and help. To contrast these forums with one I’ve been active on for a few years (not mentioning names, but this other forum is related to art and graphics, not programming) on the D forums I haven’t been insulted, nor have I been questioned for looking at things from a (warning: film term) Dutch angle—which is one of the things I do best. That comes from my art background, I suspect. I was quite the rabble rouser in art college. (Just ask Alan Wood the installation artist about our knock-down-drag-out shouting match in the cafeteria of Emily Carr College if you don’t believe me. But again, that’s a whole other story.)

On dlang.org, I mostly hang in the Learn sub-forum, which I suppose is only natural at this stage. It’s probably the most polite bunch I’ve run across on a forum ever, and I’ve been frequenting forums since the hoary days of the BBS when 1200 BAUD was lauded as the fastest thing since the 427 hemi.

Mike Wey seems quite patient for someone who answers more-or-less the same question over and over again on the GtkD forum. The only negative I found with that forum was technical. I signed up, made some posts, and when I went back to sign in a second time, I had to reregister. But I was still identified as the same Ron Tarrant who signed up the first time (I think) so perhaps that’s how things are supposed to be. It’s unusual, but it works.

I will also mention that the GtkD documentation pages are a mind-bender, but because this is where I’m planning to spend a lot of time over the next while, I’ve decided to pitch in and help make things more accessible.

I’m drawing on experiences writing my PHP-GTK blog back in 2006 and porting a bunch of code examples and tutorials into GtkD. I went so far as to buy a domain name (gtkDcoding.com) in preparation for launching a site and blog covering all this. That’s how I deal with learning curves and have done since I wrote that series of BASIC tutorials for my sister while freezing in a Newfoundland outport back in 1985.

Tools

I got up and running with dmd quite quickly. Installation on Windows 10 and FreeBSD was straightforward. A few quick questions on the D forum, and I had everything I needed to do single- or multi-file projects. A few more questions answered on the GtkD forum and I was comfortable enough to start porting my PHP-GTK code.

But I have to say, DUB eludes me. I don’t know if it’s because Electron left me feeling like JSON files are a disgruntled engineer’s revenge plot or if it’s just the way my brain works. Perhaps if I put my mind to it, I could learn it, but since I’m getting the results I want from plain ole dmd, I’ve done no more than skim DUB’s docs up until now. Just for the record, back in my C days, make files were a caution for me, too. I eventually licked them, and if it ever becomes important enough to me to figure out DUB, I guess I will. But for now, my heart’s not in it.

Finding a text editor that supports D (and especially GtkD) syntax highlighting rather than—as a few people on the forum stated—supporting C and getting ‘good enough’ support for D, led me to abandon the search and roll my own. So far, I’ve done more-or-less complete highlighters for PSPad and CodeBlocks. To be fair, they both support D out of the box, but not GtkD which is important to me, mostly as an aid to remembering module names.

I haven’t even looked at other tools. I’m dimly aware of some other sub-forums for what look like other tools, but to be honest I haven’t read them. As I said, dmd does the job, so I’m satisfied for now.

Conclusion

And that’s a quick summary of my first two months with D. On the one hand, I don’t really understand some of D’s paradigms, but on the other, the ones I do relate to are meeting my expectations. The only two things I haven’t found yet are:

  • information about how I would go about packaging a D app (with GtkD) for distribution, and
  • how to build on one platform for distribution on another (if that’s even possible).

I’m willing to put in some time on this and eventually get gtkDcoding.com off the ground as my way of giving back. It’s been a long time since those frozen weeks with a Coleco Adam when I tried to explain BASIC to my thirteen-year-old sister (who is now a paramedic and doesn’t care any more). And I must say, I’m as excited now about D and GtkD as I was about BASIC and general computing way back then. Being retired, I have the time to pursue it and I’m looking forward to becoming a regular member of the D community.

And in case you don’t know what a Dutch angle is, I urge you to watch some episodes of Batman from the 1960’s. See how the camera tilts when things go wrong for our heroes? That’s a Dutch angle.

Bio

I was inspired to learn programming while vacationing in a frozen Newfoundland outport in April of 1985. It started as a desperate attempt to keep from stripping down to my shorts and disappearing into a blizzard, but became a lifelong passion along with acting, writing, and music. In keeping with this non-typical start, it was in art college where I learned my first serious programming language (6502 assembly) and later it was on a job as a technical writer and artist that I wrote my first serious code for a client, an online mortgage calculator for a credit union in British Columbia. The culmination of my programming career was finishing the PHP-GTK app, Corkboard, a writer’s tool for story planning. (Don’t bother looking. I couldn’t come up with a distribution system, so it languishes here on a back-up drive.)

Since dropping out of high school in 1972, I’ve made a living as a taxi driver, musician, screenwriter, technical writer, artist, sound reinforcement equipment salesman, and biology lab technician among other things. I’ve also made money acting, programming, and promoting concerts. I retired from Statistics Canada in 2010 and have since divided my time between acting, writing lame novels, pursuing the elusive X-Plat beast, and keeping house for my wife of 33 years.

How an Engineering Company Chose to Migrate to D

Bastiaan Veelo is the lead developer of a specialised program for the
computer aided geometric design of ship hulls called Fairway, for the
company SARC in the Netherlands.

Imagine there is this little-known programming language in which you enjoy programming in your free time. You know it is ready for prime time and you dream about using it at work everyday. This is the story about how I made a dream like that come true.

My early acquaintance with D

Back when “google” was not yet a common verb, I was doing a web search for “parsing C++”. The reason was that writing a report for an assignment had derailed into writing a syntax highlighter for noweb using bison and flex, and I found out firsthand that C++ is not easy to parse. That web search brought up this page (present version) with an overview of the D Programming Language, and the following statement has had me hooked ever since:

D’s lexical analyzer and parser are totally independent of each other and of the semantic analyzer. This means it is easy to write simple tools to manipulate D source perfectly without having to build a full compiler. It also means that source code can be transmitted in tokenized form for specialized applications.

“Genius,” I thought, “here we have someone who knows what he’s doing.” This is representative of the pragmatic professionalism that still radiates from the D community, and it combines with an unpretentious flair that makes it pleasant to be around. This funny quote decorated its homepage for many years:

“Great, just what I need.. another D in programming.” – Segfault

Nevertheless, I didn’t have many opportunities to use the language and I largely remained sitting on the fence, observing its development.

Programming professionally

With mostly academic programming experience, I started programming professionally in 2006 for SARC, a Dutch engineering company serving the maritime industry. Since the early ’80s they have been developing software for ship design and onboard loading calculations, which today amounts to roughly half a million lines of code. I think their success can partly be attributed to their choice of programming language: Extended Pascal (the ISO 10206 standard, not one of the many proprietary extensions of Pascal).

Extended Pascal was a great improvement over ISO 7185 Pascal. Its compiler, by Prospero Software from England, was fast and well documented. The language is small enough and its syntax appropriately verbose to make engineering professionals quickly productive in programming. Personally though, I spent most of my time programming in C++, modernizing their system for computer aided design of ship hulls using Qt and Coin3D.

When your company outlives a programming language

Although selecting an ISO standard in favor of a proprietary Pascal dialect seemed wise at the time, it is apparent now that the company has outlived the language. Prospero Development Software Ltd was officially dissolved 15 years ago. Still, its former director, Tony Hetherington, continued giving support many years after, but he’d be close to 86 years old now and can no longer be reached. Its website is gone, last archived in 2013. There’s GNU Pascal, which also supports ISO 10206, but that project has stopped moving and long ago lost synchrony with gcc. Although there is no immediate crisis, it is clear that something needs to happen sometime if the company wants to continue its activities in the coming decades.

Changing the odds

A couple of years ago, I secretly started playing with the fantasy of replacing Extended Pascal with D. Even though D’s syntax is somewhat different from Pascal, it shares at least four important similarities: support for nested functions, boundary checking, modules, and compilation speed. In addition, it has many traits that make the language attractive to engineers: good focus on performance and numerics, garbage collection, dynamic arrays, easy parallelization, understandable templates, contract programming, memory safety, unit tests, and even wysiwyg strings and formatted numerals. D’s language features encourage experimentation, which resonates well with engineers.

So I wondered what I could do to highlight D’s significance to my employer and show it’s an attractive language to switch to. I thought I could make a compelling case if I could write a parser in D that would take Extended Pascal source and transpile it to D source. At least I would have fun trying!

So I went over to code.dlang.org to see if there were any D alternatives to flex and bison. There, I found Pegged, and instantly the fun began. Pegged combines the functionality of flex and bison in one incredibly easy to use package, for which its creator Philippe Sigaud obviously enjoyed writing excellent documentation. Nowadays, Pegged is part of the D language tour and you can try it out on-line without having to install a thing. The beauty is that the grammar from the Extended Pascal language specification maps almost linearly to the PEG from which Pegged generates the parser. For this it makes heavy use of D’s generic programming capabilities and compile-time function evaluation — it can generate a parser at compile time if you want it to!

However, it wasn’t smooth sailing all along. As I was testing D, I suddenly found myself being tested as well. I learned the hard way that there is a phenomenon called left-recursion, from which a PEG parser typically cannot break out of. And the Extended Pascal grammar is left-recursive in several ways. Consequently, I spent many evenings and weekends researching parsing theory, until eventually I managed to extend Pegged with support for all kinds of left-recursion! From one thing came another, and I added longest match alternation, case insensitive literals, the toHTML() method for dynamically browsing the syntax tree, and a tracer for logging the parsing process.

Obviously, I was having fun. But more importantly, I was demonstrating that the D programming language is accessible enough that a naval architect can understand other people’s code and expand it in non-trivial ways. The icing on the cake came when I was asked to present my experiences at DConf 2017 in Berlin, which you can watch here (and here’s the extra bit I presented at lunch time for the livestream audience).

At this time, I was able to automatically translate the following trivial example:

program hello(output);

begin
    writeln('Hello D''s "World"!');
end.

into D:

import std.stdio;

// Program name: hello
void main(string[] args)
{
    writeln("Hello D's \"World\"!");
}

Language competition

Having come this far, the founder of SARC agreed that it was time to investigate the merits of various alternative programming languages. We would do a thorough and objective comparison based on trial translations of a comprehensive set of language features. Due to the amount of manual labor that this requires, we had to drastically prune the space of programming languages in an initial review round. Note that what I am about to present does not declare which programming language is the best in our industry. What we are looking for is a language that allows an efficient transition from Extended Pascal without interrupting our business, and which enables us to take advantage of modern insights and tools.

In the initial review round we looked at general language characteristics. Here I’ll just highlight what fell through the sieve and why.

Performance is important to us, which is why we did not consider interpreted languages. C++ is in use for one component of our software, but that was written from the ground up. We feel that the options for translation are not favorable, that its long compile times are a serious hindrance to productivity, and that there are too many ways in which one can shoot one’s self in the foot. We cannot require our expert naval architects to also become experts in C++.

Nowadays, whenever D is publicly evaluated, the younger languages Go and Rust are often brought up as alternatives. Here, we need not go into an in-depth comparison of these languages because both Rust and Go lack one feature that we rely on heavily: nested functions with access to variables in their enclosing scope. Solutions for eliminating nested functions, like bringing them into global scope and passing extra variables, or breaking files up into smaller modules, we find unattractive because it would complicate automated translation, and we’d like to preserve the structure and style of our current code. GNU C does offer nested functions, but it is a non-standard extension and it has been predicted that many will move away from C due to its unsafe features. After this initial pruning, three languages remained on our shortlist: Free Pascal, Ada and D.

As a basis for our detailed comparison, we wrote fifteen small programs that each used a specific feature of Extended Pascal that is important in our current code base. We then translated those programs into each language on our shortlist. We kept a simple score board on how well these features were represented in each language: +1 if the feature is supported or can be implemented, 0 if the lack of the feature can be worked around, and -1 if it can’t. This is what came out of that evaluation:

Test Free Pascal Ada D
Arrays beginning at arbitrary indices +1 +1 +1
Sets 0 0 +1
Schema types 0 0 +1
Types with custom initial values -1 0 +1
Classes +1 +1 +1
Casts +1 +1 +1
Protection against use of dangling pointers -1 +1 +1
Thread safe memory [de]allocation +1 +1 +1
Calling into Windows API +1 +1 +1
Forwarding Windows callbacks to nested functions +1 +1 +1
Speed of calculations +1 +1 +1
Calling procedures written in assembly +1 0 +1
Calling procedures in a DLL +1 +1 +1
Binary compatibility of strings 0 +1 +1
Binary compatible file i/o -1 0 0
Score 6 10 14

So, Free Pascal is the only candidate with negative scores, Ada positions itself in the middle, and D achieves an almost perfect score. Not effortlessly, though; we’ll talk about some of the technical challenges later. Because Free Pascal is, like D, fully Open Source and written in itself, extending the language and filling in the gaps is theoretically possible. Although some of its deficiencies could certainly be resolved that way, others would be quite complicated and/or unlikely to be accepted upstream.

We also estimated the productivity of the languages. Free Pascal scored high because it is closest to what we are used to. Despite its dissimilar syntax, D scored high because of its expressiveness and flexibility. Ada scored lowest because of its rigidity and because of the extra work the programmer has to put in (most importantly casts and conversions). Ada is more verbose than Pascal which was disliked by some of us because it can somewhat obscure the essence of what a piece of code tries to express, and frequently the code became not only verbose but cryptic, which was unanimously disliked.

Third, we estimated the future prospects and the advantages each language could bring to the table. Although Free Pascal has a more active community than we expected it to have, we do not see great potential for growth. Ada is renowned for its support for writing reliable code (although it has no monopoly in that field) but it does come at a cost and requires real effort. D has a dynamic and open community, supports both script-like productivity and high performance, includes various features for writing reliable software (approaching Ada but at a much lower cost), and offers some unique advanced features with which wonders can be accomplished.

Finally, we estimated the effort of translation. Although Free Pascal is very similar to Extended Pascal, missing features pose a real problem and would require a high degree of manual translation and rewriting. Although p2ada exists, it only works partially in our case and does not fully support Extended Pascal. Because Ada frequently requires additional code (casting to the correct type, pulling in a package, instantiating a generic type, adding a pragma, splitting up Put_Lines etc.), writing or extending a reliable transpiler into Ada would be more difficult than doing the same into D.

Selecting a winner

I gave away the winner in the title, but we landed at that conclusion as follows. Ada was the first language to be dropped. We really felt that the extra work that the programmer has to put in is a brake on productivity and creativity. Although it barely played a role in our evaluation, illustrative is the difference between the Ada and D equivalents to the Expressive C++17 Challenge. The D solution is both concise and expressive, the Ada solution is hardly expressive and consists of more lines than I want to write or read. Also of secondary importance, but difficult to ignore, is the difference between the communities surrounding the languages, which in Ada’s case is AdaCore Support, who has no problems demanding annual five-figure subscription fees.

Although akin to our current language, Free Pascal was mainly dropped due to its porting challenges and our estimation that its potential is lower and its future outlook is less optimistic than that of D. If we were to choose Free Pascal, we would basically invest a lot of effort only to arrive at a technological solution that we felt would be of lower quality than we currently have.

And that’s were I saw a dream come true: A clap on the table by the company founder and it was decided to commit to the effort of bringing twenty-five years worth of Extended Pascal code to D!

What makes a difference

In short, my experience is that if a feature is not present in the language, D is powerful enough that the feature can be implemented in a library. Translating each sample program by hand has really helped to focus on replicating functionality, leaving the translation process for later concern. This has led to writing a compatibility library with types and functions that are vital for the conversion. Now that equivalents are known and the parser is done, I just have to implement code generation.

Below follows another example that currently translates automatically and executes identically. It iterates over a fixed length array running from 2 to 20 inclusive, fills it with values, prints the memory footprint and writes it to binary file:

program arraybase(input,output);

type t = array[2..20] of integer;
var a : t;
    n : integer;
    f : bindable file of t;

begin
  for n := 2 to 20 do
    a[n] := n;
  writeln('Size of t in bytes is ',sizeof(a):1); { 76 }
  if openwrite(f,'array.dat') then
    begin
      write(f,a);
      close(f);
    end;
end.

Transpiled to D (or should I say Dascal?) and post-processed by dfmt to fix up formatting:

import epcompat;
import std.stdio;

// Program name: arraybase
alias t = StaticArray!(int, 2, 20);

t a;
int n;
Bindable!t f;

void main(string[] args)
{
    for (n = 2; n <= 20; n++)
        a[n] = n;
    writeln("Size of t in bytes is ", a.sizeof); // 76
    if (openwrite(f, "array.dat"))
    {
        epcompat.write(f, a);
        close(f);
    }
}

Of course this is by no means idiomatic D, but the fact that it is recognizable and readable is nice, especially for my colleagues who will have to go through an unusual transition. By the way, did you notice that code comments are preserved?

One very-nice-to-have feature is binary file compatibility; In fact it may have been the killer feature, without which D might not have been so victorious. The case is that whenever a persistent data structure is extended in our software, we make sure that we can still read and convert that structure from its prior format. That way, if a client pulls out an old design from its archives and runs it through our current software, it will still work without the user even being aware that conversion occurs, possibly in multiple steps. Not having to give up that ability is very attractive.

But it wasn’t easy to get there. The main difficulty is the difference in how strings are represented in D and the Prospero implementation of Extended Pascal, in memory and on file. This presented the challenge of how to preserve binary compatibility in file I/O with data structures that contain string members.

Strings

In Prospero Extended Pascal, strings are implemented as a schema type, which is a parameterized type that can be used in the following ways:

type string80 = string(80);
var str1 : string80;
    str2 : string(60);
procedure foo(s : string);

This defines string80 to be an alias for a string type discriminated to have a capacity of 80 characters. Discriminated string variables, like str1 and str2, can be passed to functions and procedures that take undiscriminated strings as arguments, like foo, which thereby work on strings of any capacity. In memory, str1 is laid out as a sequence of 80 chars, followed by a ushort that encodes the length of the string. I say encodes because a shorter string is padded with \0s up to the capacity and the ushort actually contains the length of that padding. This way, when a pointer to the string is passed to a C function and the contents of the string occupy its full capacity, the 0 in the padding length doubles as the terminating \0 of the C string.

My first thought was to mimic this data representation with a D template. But that would require procedures like foo to be turned into templates as well, which would escalate horribly into template bloat, a problem with multiple string arguments and argument ordering, and would complicate translation. Besides, schema types can also be discriminated at run time, which does not translate to a template.

Could some sort of inheritance scheme be the solution? Not really, because instances of D classes live on the heap, so a string embedded in a struct would just be a pointer instead of the char array and ushort.

But binary layout is actually only relevant in files, and in a stroke of insight I realized that this must be why user-defined attributes, or UDAs, exist. If I annotate the string with the correct capacity for file I/O, then I can just use native D strings everywhere, which genuinely must be the best possible translation and solves the function argument issue. Annotation can be done with an instance of a struct like

struct EPString
{
    ushort capacity;
}

The above Pascal snippet then translates to D like so:

@EPString(80) struct string80 { string _; alias _ this; }
string80 str1;
@EPString(60) string str2;
void foo(string s);

Notice how the string80 alias is translated into the slightly convoluted struct instead of a normal D alias, which would have looked like

@EPString(80) alias string80 = string;
</code>

Although that compiles, there is no way to retrieve the UDA in that case because plain alias does not introduce a symbol. Then hasUDA!(typeof(str1), EPString) would have been equivalent to hasUDA!(string, EPString) which evaluates to false. By using the struct, string80 is a symbol so typeof(str1) gives string80, and hasUDA!(string80, EPString) evaluates to true in this example.

There is one side effect that we will have to learn to accept, and that is that taking a slice of a string does not produce the same result in D as it does in Extended Pascal. That is because string indices start at 1 in Extended Pascal and at 0 in D. My strategy is to eliminate slices from the source and replace them with a call to the standard substr function, which I can implement with index correction. Finding all string slices can be accomplished with a switch in the transpiler that makes it insert a static if to test if the slice is being taken on a string, and abort compilation if it is. (Arrays are transpiled into a custom array type that handles slices and indices compatibly with Extended Pascal.)

Binary compatible file I/O

Now, to write structs to file and handle any embedded @EPString()-annotated strings specially, we can use compile-time introspection in an overload to toFile that acts on structs as shown below. I have left out handling of aliased strings for clarity, as well as shortstring, which is a legacy string type with yet a different binary format.

void toFile(S)(S s, File f) if (is(S == struct))
{
    import std.traits;
    static if (!hasIndirections!S)
        f.lockingBinaryWriter.put(s);
    else
        // TODO unions
        foreach(field; FieldNameTuple!S)
        {
            // If the member has itself a toFile method, call it.
            static if (hasMember!(typeof(__traits(getMember, s, field)), "toFile") &&
                       __traits(compiles, __traits(getMember, s, field).toFile(f)))
                __traits(getMember, s, field).toFile(f);
            // If the member is a struct, recurse.
            else static if (is(typeof(__traits(getMember, s, field)) == struct))
                toFile(__traits(getMember, s, field), f);
            // Treat strings specially.
            else static if (is(typeof(__traits(getMember, s, field)) == string))
            {
                // Look for a UDA on the member string.
                static if (hasUDA!(__traits(getMember, s, field), EPString))
                {
                    enum capacity = getUDAs!(__traits(getMember, s, field), EPString)[0].capacity;
                    static assert(capacity > 0);
                    writeAsEPString(__traits(getMember, s, field), capacity, f);
                }
                else static assert(false, `Need an @EPString(n) in front of ` ~ fullyQualifiedName!S ~ `.` ~ field );
            }
            // Just write other data members.
            else static if(!isFunction!(__traits(getMember, s, field)))
                f.lockingBinaryWriter.put(__traits(getMember, s, field));
        }
}

At the time of writing, I still have work to do for unions, which are used in the translation of variant records (including considering the use of one of the seven existing library solutions 1, 2, 3, 4, 5, 6, 7).

Currently, detecting unions is a bit involved . Also, there is a complication in the determination of the size of a union when the largest variant contains strings: the D version of that variant may not be the largest because D strings are just slices. I’ll probably work around this by adding a dummy variant that is a fixed size array of bytes to force the size of the union to be compatible with Extended Pascal. This is the reason why D scored a mere 0 in file format compatibility. It is amazing what D allows you to do though, so I may be able to do all of that automatically and award D a perfect score retroactively. On the other hand, it is probably easiest to just add the dummy variant in the Pascal source at the few places where it matters and be done with it.

The way forward

Obviously, this is long term planning. It has taken years to grow into D; it will possibly take a year, and probably longer, to migrate to D. Unless others turn up who are in the same boat as us (please contribute!) it’ll be me who has to row this ship to D-land and I still have my regular duties to attend to. My colleagues will continue to develop in Extended Pascal as usual, and once my transpiler is able to translate all or almost all of it, we will make the switch to D overnight. From then on, we’ll be in it for the long run. We trust to be with D and D to be with us for decades to come!

D Goes Business

A long-time contributor to the D community, Kai Nacke is the author of ‘D Web Development‘ and the maintainer of LDC, the LLVM D Compiler. Be sure to catch Kai at DConf 2018 in Munich, May 2 – 5, where he’ll be speaking about “D for the Blockchain“.

Many companies run their business processes on an SAP system. Often other applications need to access the SAP system because they provide data or request some calculation. There are many ways to achieve this… Let’s use D for it!

 

The SDK and the D binding

As an SAP customer you can download the SAP NetWeaver Remote Function Call SDK. You can call RFC-enabled functions in the SAP system from a native application. Conversely, it is possible from an ABAP program to call a function in a native program. The API documentation for the library is available as a separate download. I recommend downloading it together with the library.

The C/C++ interface is well structured but can be very tedious to use. That’s why I not only created a D binding for the library but also used some D magic to make the developer’s life much easier. I introduced the following additions:

  • Most functions have a parameter of type RFC_ERROR_INFO. As the name implies, this is only needed in case of an error. For each of these functions a new function is generated without the RFC_ERROR_INFO parameter and the return type RFC_RC changed to void. Instead, a SAPException is thrown in case of error.
  • Functions named RfcGet*Count now return a size_t value instead of using an out parameter. This is possible because of the above step which changed the return type to void.
  • Functions named Rfc*ByIndex use a size_t parameter for the index, thus better integrating with D.
  • Functions which have a pointer and length value now accept a D array instead.
  • Use of pointers to zero-terminated UTF–16 strings is replaced with wstring type parameters.

Using the library is easy, just add

dependency "sapnwrfc-d"

to your dub.sdl file. One caveat: the libraries provided by the SAP package must be installed in such a way that the linker can find them. On Windows, you can add the path to the lib folder of the SDK to the LIB and PATH environment variables.

An example application

Let’s create an application calling a remote-enabled function on the SAP system. I use the DATE_COMPUTE_DAY function because it is simple but still has import and export parameters. This function takes a date (a string of format “YYYYMMDD”) and returns the weekday as a number (1 = Monday, 2 = Tuesday and so on).

The application needs two parameters: the system identifier of the SAP system and the date. The system identifier denotes the destination for the RFC call. The parameters for the connection must be in the sapnwrfc.ini file, which must be located in the same folder as the application executable. An invocation of the application using the SAP system X01 looks like this:

D:\OpenSource\sapnwrfc-example>sapnwrfc-example.exe X01 20180316
Date 20180316 is day 5

First, create the application with DUB:

D:\OpenSource>dub init sapnwrfc-example
Package recipe format (sdl/json) [json]: sdl
Name [sapnwrfc-example]:
Description [A minimal D application.]: An example rfc application
Author name [Kai]: Kai Nacke
License [proprietary]:
Copyright string [Copyright © 2018, Kai Nacke]:
Add dependency (leave empty to skip) []: sapnwrfc-d
Added dependency sapnwrfc-d ~>0.0.5
Add dependency (leave empty to skip) []:
Successfully created an empty project in 'D:\OpenSource\sapnwrfc-example'.
Package successfully created in sapnwrfc-example

D:\OpenSource>

Let’s edit the application in source\app.d. Since this is only an example application, I’ve put all the code into the main() function. In order to use the library you simply import the sapnwrfc module. Most functions can throw a SAPException, so you want to wrap them in a try block.

import std.stdio;
import sapnwrfc;

int main(string[] args)
{
    try
    {
        // Put your code here
    }
    catch (SAPException e)
    {
        writefln("Error occured %d %s", e.code, e.codeAsString);
        writefln("'%s'", e.message);
        return 100;
    }
    return 0;
}

The library uses only UTF–16. Like the C/C++ version, the alias cU() can be used to create a zero-terminated UTF–16 string from a D string. I convert the command line parameters first:

    auto dest = cU(args[1]);
    auto date = cU(args[2]);

Now initialize the library. Most important, this function loads the sapnwrfc.ini file and initializes the environment. If this call is missing then it is implicitly done inside the library. Nevertheless, I recommend calling the function. It is possible that I will add more functionality to this function.

    RfcInit();

The next step is to open a connection to the SAP system. Since the connection parameters are in the sapnwrf.ini file, it is only necessary to provide the destination. In your own application you do not need to use the sapnwrf.ini file. You can provide all parameters in the RFC_CONNECTION_PARAMETER[] array which is passed to the RfcOpenConnection() function.

    RFC_CONNECTION_PARAMETER[1] conParams = [ { "DEST"w.ptr, dest } ];
    auto connection = RfcOpenConnection(conParams);
    scope(exit) RfcCloseConnection(connection);

Please note the D features used here. A string literal is always zero-terminated, therefore there is no need to use cU() on the "DEST"w literal. With the scope guard, I make sure that the connection is closed at the end of the block.

Before you can make an RFC call, you have to retrieve the function meta data (function description) and create the function from it.

    auto desc = RfcGetFunctionDesc(connection, "DATE_COMPUTE_DAY"w);
    auto func = RfcCreateFunction(desc);
    scope(exit) RfcDestroyFunction(func);

The RfcGetFunctionDesc() calls the SAP system to look up the meta data. The result is cached to avoid a network round trip each time you invoke this function. The implication is that the remote user needs the right to perform the look up. If this step fails with a security-related error, you should talk to your SAP administrator and review the rights of the remote user.

The DATE_COMPUTE_DAY function has one import parameter, DATE. To pass a parameter you call one of the RfcSetXXX() or RfcSetXXXByIndex() functions. The difference is that the first variant uses the parameter name (here: "DATE") or the index of the parameter in the signature (here: 1). I often use the named parameter because the resulting code is much more readable. The date data type expects an 8 character UTF–16 array.

    RfcSetDate(func, "DATE", date[0..8]);

Now the function can be called:

    RfcInvoke(connection, func);

The computed weekday is returned in the export parameter DAY. There is a set of RfcGetXXX() and RfcGetXXXByIndex() functions to retrieve the value.

    wchar[1] day;
    RfcGetChars(func, "DAY", day);

Let’s print the result:

    writefln("Date %s is weekday %s", date[0..8], day);

Congratulations! You’ve finished your first RFC call.

Build the application with dub build. Before you can run the application you still need to create the sapnwrfc.ini file. This looks like:

DEST=X01
MSHOST=x01.your.domain
GROUP=PUBLIC
CLIENT=001
USER=kai
PASSWD=secret
LANG=EN

The SDK contains a commented sapnwrfc.ini file in the demo folder. If you are on Windows and your system still uses SAP GUI with the (deprecated) saplogon.ini file, then you can use the createini example application from my bindings library to convert the saplogon.ini file into the sapnwrfc.ini file.

Summary

Calling an RFC function of an SAP system with D is very easy. D features like support for UTF–16 strings, scope guards, and exceptions make the source quite readable. The presented example application is part of the D bindings library and can be downloaded from GitHub at https://github.com/redstar/sapnwrfc-d.

User Stories: Funkwerk

The deadline for the early-bird registration for DConf 2018 in Munich is coming up on March 17th. The price will go up from $340 to $400. If you’d like to go, hurry and sign up to save yourself $60. And remember, the NH Munich Messe hotel, the conference venue, is offering a special deal on single rooms plus breakfast for attendees.

A few of the DConf attendees are coming from a local company called Funkwerk. They’re a D shop that we’ve highlighted here on this blog in a series of posts about their projects (you’ll see one of their products in action if you take the subway or local train service in Munich).

In this post, we cap off the Funkwerk series with the launch of a new feature we creatively call “User Stories”. Now and again, we’ll publish a post in which D users talk of their experiences with D, not about specific projects, but about the language itself. They’ll tell of things like their favorite features, why they use it, how it has changed the way they write code, or anything they’d like to say that expresses how they feel about programming in D.

For this inaugural post, we’ve got three programmers from Funkwerk. First up, Michael Schnelle talks about the power of ranges. Next, Ronny Spiegel tells why generated code is better code. Finally, Stefan Rohe enlightens us on Funkwerk’s community outreach.

The power of ranges

Michael Schnelle has been working as a software developer for about 5 years. Before starting with D 3 years ago, he worked in (Web)Application Development, mostly with Java, Ruby on Rails, and C++, and did Thread Modeling for Applications. He enjoys coding in D and likes how it helps programmers write clean code.

In my experience, no matter what I am programming, I always end up applying functions to a set of data and filter this set of data. Occasionally I also execute something with side effects in between. Let’s look at a simplified use case: the transformation of a given set of data and filtering for a condition afterwards. I could simply write:

foreach(element; elements) {
  auto transformed = transform(element);
  if (metCondition(transformed) {
     results ~= transformed
  } 
}

Using the power from std.algorithm, I can instead write:

filter!(element => metCondition(element))
       (map!(element => transform(element))(elements));

At this point, we have a mixture of functional and object-oriented code, which is quite nice, but still not quite as readable or easy to understand as it could be. Let’s combine it with UFCS (Uniform Function Call Syntax):

elements.map!(element => element.transform)
        .filter!(element => element.metCondition);

I really like this kind of code, because it is clearly self-explanatory. The foreach loop, on the other hand, only tells me how it is being done. If I look through our code at Funkwerk, it is almost impossible to find traditional loops.

But this only takes you one step further. In many cases, there happen to be side effects which need to be executed during the workflow of the program. For this kind of thing, the library provides functions like std.range.tee. Let’s say I want to execute something external with the transformed value before filtering:

elements
  .map!(element => element.transform)
  .tee!(element => operation(element))
  .filter!(element => element.metCondition)
  .array;

It is crucial that operations with side effects are only executed with higher-order functions that are built for that purpose.

int square(int a) { writefln("square value"); return a*a; }

[4, 5, 8]
  .map!(a => square(a))
  .tee!(a => writeln(a))
  .array;

The code above would print out the square value six times, because tee calls range.front twice. It is possible to avoid this by using functions like std.algorithm.iteration.cache, but in my opinion, the nice way would be to avoid side effects in functions that are not meant for that.

In the end, D gives you the possibility to combine the advantages of object-oriented and functional programming, resulting in more readable and maintainable code.

Generated code is better code

Ronny Spiegel has worked as a professional software developer for almost 20 years. He started out using C and C++, but when he joined Funkwerk he really started to love the D language and the tools it provides to introspect code and to automate things at compile time.

In a previous blog post, I gave a short overview of the evolution of the accessors library. As you might imagine, I really like the idea of using the compiler to generate code; in the end this usually results in less work for me and, as a direct result, causes fewer errors.

The establishment of coding guidelines is crucial for a team in order to create maintainable software, and so we have them here at Funkwerk. There is a rule that every value object (or entity) has to implement the toString method in order to provide diagnostic output. The provided string shall be unambiguous so that it’s more like Python’s __repr__ than __str__.

Example:

StationMessage(GeneralMessage(4711, 2017-12-12T10:00:00Z), station="BAR", …)

The generated string should follow some conventions:

  • provide a way to uniquely reconstruct data from a string
    • start with the class name
    • continue with any potential superclasses
    • list all fields providing their name and value separated by a comma
  • be compact but still human readable (for developers)
    • skip the name where it matches the type (e.g. a field of type SysTime is called time)
    • skip the name if the field is called id (usually there’s an IdType used for type safety)
    • there’s some special output format defined for types like Date and SysTime
    • Nullable!T’s will be skipped if null etc.

To format output in a consistent manner, we implemented a SinkWriter wrapping formattedWrite in a way that follows the listed conventions. If this SinkWriter is used everywhere, this is the first step to fully generate the toString method.

Unfortunately that’s not enough; it’s very common to forget something when adding a new field to a class. Today I stumbled across some code where a field was missing in the diagnostics output and that led to some confusion.

Using (template) mixins together with CTFE (Compile Time Function Execution) and the provided type traits, D provides a powerful toolset which enables us to generate such functions automatically.

We usually implement an alternative toString method which uses a sink delegate as described in https://wiki.dlang.org/Defining_custom_print_format_specifiers. The implementation is a no-brainer and looks like this:

public void toString(scope void delegate(const(char)[]) sink) const
{
    alias MySelf = Unqual!(typeof(this));

    sink(MySelf.stringof);
    sink("(");

    with (SinkWriter(sink))
    {
        write("%s", this.id_);
        write("station=%s", this.station_);
        // ...
    }

    sink(")");
}

This code seems to be so easy that it might be generalized like this:

public void toString(scope void delegate(const(char)[]) sink) const
{
    import std.traits : FieldNameTuple, Unqual;

    alias MySelf = Unqual!(typeof(this));

    sink(MySelf.stringof);
    sink("(");

    with (SinkWriter(sink))
    {
        static foreach (fieldName; FieldNameTuple!MySelf)
        {{
            mixin("const value = this." ~ fieldName ~ ";");
            write!"%s=%s"(fieldName, value);
        }}
    }

    sink(")");
}

The above is just a rough sketch of how such a generic function might look. For a class to use this generation approach, simply call something like

mixin(GenerateToString);

inside the class declaration, and that’s it. Never again will a field be missing in the class’s toString output.

Generating the toString method automatically might also help us to switch from the common toString method to an alternative implementation. If there will be more conventions over time, we will only have to extend the SinkWriter and/or the toString-template, and that’s it.

As a summary: Try to generate code if possible – it is less error prone and D supports you with a great set of tools!

Funkwerk and the D-Community

Stefan Rohe started the D-train at Funkwerk back in 2008. They have loved DLang since then and replaced D1-Tango with D2-Phobos in 2013. They are strong believers in open source and local communities, and are thrilled to see you all in Munich at DConf 2018.

Funkwerk is the largest D shop in south Germany, so we hire D-velopers, mainly just through being known for programming in D. In order to give a little bit back to the D community at large and help the local community grow, Funkwerk hosted the foundational edition of the Munich D Meetup.

The local community is important …

Munich Meetup at Brainlab

The meetup was founded in August 2016, 8 years after the first line of D code at Funkwerk was written. Since then, the Meetup has grown steadily to ~350 members. At that number, it is still not the biggest D Meetup, but it is the most visited and the most active. It provides a chance for locals in Munich to interact with like-minded D-interested people each month. And with an alternating level of detail and a different location each month, it stays interesting and attracts different participants.

… and so is the global community

To engage with the global community, Funkwerk is willing to open source some of its general-purpose D libraries. They can all be found under github.com/funkwerk, and some are registered in the DUB registry.

To mention are:

  • accessors – a library to auto generate getters and setters with UDAs
  • depend – a tool that checks actual import dependencies against a UML model of target dependencies
  • d2uml – reverse engineering of D source code into PlantUML class outlines

Feel free to use these and let us know how you like them.

LDC 1.8.0 Released

LDC, the D compiler using the LLVM backend, has been actively developed for going on a dozen years, as laid out by co-maintainer David Nadlinger in his DConf 2013 talk. It is considered one of two production compilers for D, along with GDC, which uses the gcc backend and has been accepted for inclusion into the gcc tree.

The LDC developers are proud to announce the release of version 1.8.0, following a short year and a half from the 1.0 release. This version integrates version 2.078.3 of the D front-end (see the DMD 2.078.0 changelog for the important front-end changes), which is itself written in D, making LDC one of the most prominent mixed D/C++ codebases. You can download LDC 1.8.0 and read about the major changes and bug fixes for this release at GitHub.

More platforms

Kai Nacke, the other LDC co-maintainer, talked at DConf 2016 about taking D everywhere, to every CPU architecture that LLVM supports and as many OS platforms as we can. LDC is a cross-compiler: the same program can compile code for different platforms, in contrast to DMD and GDC, where a different DMD/GDC binary is needed for each platform. Towards that end, this release continues the existing Android cross-compilation support, which was introduced with LDC 1.4. A native LDC compiler to both build and run D apps on your Android phone is also available for the Termux Android app. See the wiki page for instructions on how to use one of the desktop compilers or the native compiler to compile D for Android.

The LDC team has also been putting out LDC builds for ARM boards, such as the Raspberry Pi 3. Download the armhf build if you want to try it out. Finally, some developers have expressed interest in using D for microservices, which usually means running in an Alpine container. This release also comes with an Alpine build of LDC, using the just-merged Musl port by @yshui. This port is brand new. Please try it out and let us know how well it works.

Linking options – shared default libraries

LDC now makes it easier to link with the shared version of the default libraries (DRuntime and the standard library, called Phobos) through the -link-defaultlib-shared compiler flag. The change was paired with a rework of linking-related options. See the new help output:

Linking options:

-L= - Pass to the linker
-Xcc= - Pass to GCC/Clang for linking
-defaultlib=<lib1,lib2,…> - Default libraries to link with (overrides previous)
-disable-linker-strip-dead - Do not try to remove unused symbols during linking
-link-defaultlib-debug - Link with debug versions of default libraries
-link-defaultlib-shared - Link with shared versions of default libraries
-linker=<lld-link|lld|gold|bfd|…> - Linker to use
-mscrtlib=<libcmt[d]|msvcrt[d]> - MS C runtime library to link with
-static

Other new options

  • -plugin=... for compiling with LLVM-IR pass plugins, such as the AFLfuzz LLVM-mode plugin
  • -fprofile-{generate,use} for Profile-Guided Optimization (PGO) based on the LLVM IR code (instead of PGO based on the D abstract syntax tree)
  • -fxray-{instrument,instruction-threshold} for generating code for XRay instrumentation
  • -profile (LDMD2) and -fdmd-trace-functions (LDC2) to support DMD-style profiling of programs

Vanilla compiler-rt libraries

LDC uses LLVM’s compiler-rt runtime library for Profile-Guided Optimization (PGO), Address Sanitizer, and fuzzing. When PGO was first added to LDC 1.1.0, the required portion of compiler-rt was copied to LDC’s source repository. This made it easy to ship the correct version of the library with LDC, and make changes for LDC specifically. However, a copy was needed for each LLVM version that LDC supports (compiler-rt is compatible with only one LLVM version): the source of LDC 1.7.0 has 6 (!) copies of compiler-rt’s profile library.

For the introduction of ASan and libFuzzer in the official LDC binary packages, a different mechanism was used: when building LDC, we check whether the compiler-rt libraries are available in LLVM’s installation and, if so, copy them into LDC’s lib/ directory. To use the same mechanism for the PGO runtime library, we had to remove our additions to that library. Although the added functionality is rarely used, we didn’t want to just drop it. Instead, the functionality was turned into template-only code, such that it does not need to be compiled into the library (if the templated functionality is called by the user, the template’s code will be generated in the caller’s object file).

With this change, LDC no longer needs its own copy of compiler-rt’s profile library and all copies were removed from LDC’s source repository. LDC 1.8.0 ships with vanilla compiler-rt libraries. LDC’s users shouldn’t notice any difference, but for the LDC team it means less maintenance burden.

Onward

A compiler developer’s work is never done. With this release out the door, we march onward toward 1.9. Until then, start optimizing your D programs by downloading the pre-compiled LDC 1.8.0 binaries for Linux, Mac, Windows, Alpine, ARM, and Android, or build the compiler from source from our GitHub repository.

Thanks to LDC contributors Johan Engelen and Joakim for coauthoring this post.