Category Archives: Community

Symmetry Autumn of Code Experience Report: Porting a fork-based GC

Symmetry Investments logoThe 2018 edition of the Symmetry Autumn of Code was a wonderful opportunity for me and two other students to dive into an interesting programming challenge and contribute to the D community. I am going to describe the process that led to my participation in SAOC and what this four months of work meant to me.

Who I am

I am an MSc student in Computer Science at the University of Turin in Italy. My interests mainly revolve around type systems, language theory, formal languages and compilers, and concurrent programming techniques. I am 23 years old and I have been programming since I was 20.

What happened before SAOC

While browsing Hacker News one day, one comment caught my attention: “the [D] community seems second to none as far as signal to noise ratio goes”. From that day I started lurking on the D forums and later read Andrei Alexandrescu’s book, The D Programming Language. While I think it alone wasn’t sufficient to learn the language, it was worth the read for all the discussions about compiler internals, the compromises about the syntax, and the details on ranges and concurrency by message passing.

After reading the book, I had enough confidence to start exploring some real-world code and I decided to dig into the codebases of the D standard library (Phobos) and vibe.d, a popular web app framework. My colleague Francesco Gallà, who also would participate in SAOC (see his DConf 2019 presentation about his experience), started learning D and we approached the community together by discussing one of the past Google Summer of Code proposals, HTTP2 support, on the vibe.d forum.

At the same time, I bought the Garbage Collection Handbook as a personal reading. At the time it seemed totally unrelated to what I was doing with D.

SAOC

On July 14, 2018, the D Language Foundation announced the Symmetry Autumn of Code. It seemed the right occasion to boost my skills with D and get even more involved with the community. I was also thrilled by the possibility of learning new methodologies for writing code with a remote mentor.

I gathered information about a past project, a concurrent garbage collector that was used in D1 in the Tango library. I decided that my SAOC project proposal would be to port it to D2 with the goal of incorporating it into the D runtime.

The original work was done by Leandro Lucarella as his master thesis, and he documented it extensively on his blog. I contacted him and received confirmation that he could mentor such a project. After that, I wrote my proposal. It consisted of two documents, one about me and the other about the project.

The concurrent GC project

The project document had the objective of showing my efforts in understanding the garbage collector currently shipping with the D runtime—a conservative mark and sweep GC that does the following whenever an allocation is issued and memory needs to be reclaimed:

  1. stop all threads except the current one
  2. “hijack” the work of the current thread in order to run the GC routines
  3. start scanning recursively every root memory pointer in order to find every memory block that has been used or is being used by the program
  4. mark all GC-allocated blocks that are no longer referenced
  5. resume all other threads
  6. run destructors from unreachable memory that has been marked in phase 4 and free the remaining unreachable memory
  7. continue execution of the current thread.

The program is paused during steps 1 to 4 (the mark phase) and memory is reclaimed during step 6, where the GC thread hijacks the flow of the thread that triggered the collection. There are various approaches for reducing the pause time, such as using threads to scan and mark the memory objects in parallel; work is being done in that direction.

My proposal highlighted another strategy. By sacrificing memory consumption on systems that support Posix’s fork() system call, pause time could be drastically reduced. A fork-based concurrent GC would be represented by the following sequence of routines:

  1. a thread triggers the GC collection (such a thread is called the mutator)
  2. the GC clones the address space of the current program with the fork() system call
  3. while the parent process continues execution, the child process starts the mark phase
  4. at the end of the mark phase, before exiting, the child process communicates the marked bits (that can be reused) to the parent
  5. the GC uses the unmarked blocks for future allocations (lazy sweep).

With this schema, the pause time of the marking phase is reduced to the duration of the fork() call only. There are many advantages, such as the fact that the other threads do not need to be stopped and that the reclaim phase could run concurrently without hijacking the mutator. In particular, this strategy shows its strength in potentially long-running programs that have large heaps with a high number of live objects.

The implementation resulted in a bit more than 500 lines of code, given that the calling process (called the parent) generates a duplicate of itself in a different address space. This removes the need for synchronization, which has a high overhead both in terms of runtime performance and code implementation. Moreover, Unix and Linux systems provide very efficient fork() implementations with the use of COW memory.

Methodologies

The first thing that I did with Leandro was to correct the milestones that I had predicted. Based on his experience, we put a bigger focus on defining the tests rather than the programming. After that he specified the workflow to me. We setup a test suite in place, mainly composed of dustmite, the runtime benchmarks and tests, and some D1 programs that I ported.

After discussing and applying every change to the codebase in a different branch, I had to run the tests and then open a pull request on my GitHub repository asking for a review from Leandro. Commits had to be very granular and he always provided a lot of feedback. He was always prompt in replying and we had a number of exchanges by email before applying a change. Many times we discussed benchmarks and regressions, and sometimes I asked for help with debugging. I can confidently say that I spent more time researching, reading, debugging, and discussing code than I spent writing it.

I was alone in managing my time and my commitment. Leandro and the D Foundation were always present in discussing things by email, but they didn’t force any time sheet on me nor did they micromanage my work.

The end of SAOC

At the end of the four months of SAOC, I had a working implementation, but I decided to delay a pull request to the D runtime in order to work on some profiling code that could help developers understand in which cases the fork-based garbage collector brings advantages. After a precise garbage collector was announced for DMD 2.085.0, I decided to adapt my work to it. It was very straightforward given the clarity of the added code and the separation of concerns in place.

Leandro was available for mentoring even after SAOC. We exchanged many emails and he showed me how the D garbage collector is being profiled at Sociomantic.

The pull request

A pull request to the D runtime was my final milestone. I was ready at the beginning of February, but I started to procrastinate. I’d had no previous communication with any of the reviewers and I was timorous about engaging with them. I spent a lot of time refactoring my code back and forth and delaying my pull request. At a certain point, I even considered abandoning the final milestone and providing the GC as a library. In the meantime, Rainer Scheutze published a threaded implementation of the mark phase that reduced the mark time in the GC and I lost faith in my project.

Luckily for me, I had the opportunity to attend DConf 2019 in London. There I found many great people who talked with me and convinced to resume my work. I had a brief discussion with Rainer and I started testing against his implementation (I also found a related bug in the CPU detection code) and on the last day, during the annual Hackathon, I finally opened the pull request.

Since May 11, I have discussed changes to my code, reduced the number of lines of code, refactored and collapsed some functions, and resolved bugs related to program termination. The pull request is still open given that there were many rough edges, but overall I am very satisfied about the feedback received and with the whole process. Reviewers responded to every commit and provided guidance when needed. Whenever something wasn’t clear, I replied on GitHub or asked for help in the #D IRC on freenode.net.

In retrospect, I should have opened the PR much earlier and presented the reviewers with every doubt that I had along the path.

Experience analysis

SAOC helped me in understanding the dynamics behind contributing to a community project such as D. I was already spending a fixed amount of time reading the forum every day and I started to direct some of that effort to GitHub. Checking pull requests, commits, and issues was expensive work, but it was necessary to gain knowledge about the methodologies regarding the development of Phobos and the runtime.

I also learned that there are many changes that go unnoticed if you don’t closely follow the discussions on the PR queue. One example is Manu Evans’s work on core.stdcpp that is very difficult to follow when it’s scattered across different forum threads and lacks a proper announcement.

I think that overall, communication was a weak point of my experience. Regarding SAOC, we defined the work in detail but we didn’t put any effort on communicating the status of my work to the community. That could have triggered helpful suggestions from the community that could have helped me to discover holes in my implementation. Again, I benefited from this great amount of freedom in managing my work, but I think it could have coexisted with some effort on communicating more.

Finally, I had previously used D for higher level code, but SAOC forced me to discover D from a new angle; I dived into low-level code. I was amazed by the flexibility of the language and found myself becoming familiar with this different style of writing code very quickly.

Conclusion

The Symmetry Autumn of Code was a very rewarding experience. It was a period of discovery, self-growth, and involvement in a new community.

I am very grateful to Symmetry Investments and the D Language Foundation for this opportunity. Moreover, I want to thank my mentor, Leandro, for all of his help and for all the positive exchanges that we had.

DConf T-Shirts and the Human Resource Fund

I announced the Human Resource Fund back in April as part of an initiative we’re working on with companies using D in production. It was seeded by WekaIO. Now we need to see it grow.

We always have an excess of t-shirts after DConf. Often, they are given away. This year, we intentionally ordered many more than necessary. Some were given away to “Friends of D” (such as the venue staff), but the rest were intended to help us raise money.

Some of the shirts remained in London. The remainder were given to several volunteers to carry with them back home, the goal being to position them around the world so that we can minimize shipping costs. Now, we’re ready to start sending the shirts through the mail.

Help us by donating $60 to the Human Resource Fund and one of our volunteers will send you a DConf 2019 t-shirt. To take advantage of this, you have to donate through the following campaign:

Donate and get a DConf 2019 t-shirt!

The D Language Foundation will reimburse each volunteer for the shipping costs. In order to help us minimize those costs and maximize the amount we get to keep, we’ve set up a menu organized by shirt size and region, as shown in the following screenshot.

Screenshot of donation form.This campaign will help us minimize shipping costs and keep track in real time of the number of shirts remaining. Once the shirts are gone, the campaign is closed. So please, when you make your donation, help us out by selecting the region in which you live if there are still shirts available.

Those of you who have seen our Open Collective page may be aware that Gold Donors, those who make a one-time donation of $100 or more, receive a DMan t-shirt. Personal Sponsors, those who commit to an annually recurring donation of $200 or more, receive a DMan t-shirt and a guaranteed Early-Bird discount to the next DConf. We’re extending both of those packages to the Human Resource Fund campaign.

To take advantage of this, please donate through the HR Fund campaign page:

Donate to the Human Resource Fund.

After I receive notification from Flipcause of your donation, I’ll contact you to verify your shipping address and determine the size and color you’d prefer for your new DMan t-shirt.

Dman t-shirts.

Please help us grow this fund! The General Fund is also important and we still welcome donations there, but the Human Resource Fund will be our primary focus for the next few months. The sooner we reach the levels we need, the sooner we can hire people to work on important and difficult tasks in the D ecosystem.

DConf 2019 Recap

In late November of last year, Laeeth Isharc of Symmetry Investments expressed interest in hosting DConf 2019 in London. On a personal note, I had been looking for an excuse to get back to London since my brief visit at the end of the first Berlin DConf in 2016, so as my inbox filled with emails discussing possible venues, my excitement started to build. At one point, the Royal Institution was among the list of candidates.

I couldn’t make an announcement yet as nothing was certain, but I did start teasing it on Twitter and here on the blog. At our first D Language Foundation quarterly meeting on December 1, there was unanimous agreement that London was the place to be. As the days passed and it seemed to be a near certainty, I was eager to make the announcement, but near certainty is not certainty. I had to wait until Symmetry had selected a venue. That news reached my inbox on December 21. I announced it on the blog the next day.

After that, it was time to get into the details.

Planning a DConf

Past editions of DConf were organized either by hosts with employees who regularly organize conferences as part of their job descriptions, or, in the case of DConf 2018, an event planner hired by the host. This year there were no event planners and no dedicated conference organizers. It was a very different experience compared to my first peek behind the DConf curtain last year. Most of the details were hashed out in numerous emails and phone calls with Belinda Liao. Though we can thank Laeeth and Symmetry for making DConf 2019 happen, we owe a big thanks to Belinda for making it work.

I first became acquainted with Belinda–who Laeeth introduced as his “chief of staff in London”, who Symmetry’s tech team affectionately refer to as “the official nag”, and whose official title is Business Manager for Technology at Symmetry–during last year’s Symmetry Autumn of Code. Throughout the planning for DConf, she was the one doing all of the legwork. She also made sure we covered all of the bases, querying me for our requirements, pointing out anything I overlooked, and bringing new ideas.

The venue told us they would handle the live stream, but we also wanted a separate solution for recording and producing the individual talk videos. Belinda hired Stage Engage, who sent a single technician, Rowan While, to get the job done. He set up multiple cameras and sat at his primary camera in a back corner for the entirety of the three days of talks. He and his colleagues did an excellent job and three weeks after DConf the link to download the videos was sitting in my inbox. They’re all available on our YouTube channel and are accessible, along with the slides for each talk, through the DConf 2019 schedule.

During the talk submission period, Ethan Watson reached out to tell us he could submit a talk, help us in reviewing drafts of the speakers’ slides, or volunteer to be the emcee. Andrei suggested he do all three. So he did!

This year was the first time we asked the speakers to submit drafts of their slides. Last year, Andrei participated in a conference where the speakers were required to present their talks via Skype for review prior to the conference. He suggested to me at the end of DConf 2018 that we might want to consider that this year. When Ethan came onboard, he suggested instead simply reviewing drafts of the slides, which is standard procedure at the Game Developer’s Conference where he had previously presented. So we set up a deadline for the speakers to send us their drafts. Ethan reviewed them and provided feedback.

Planning the peripherals

When I first heard we might be going to London, I wanted to find some places to see other than the well-known tourist spots. On my first visit, I’d only had a day to be a tourist. This time, my wife was coming along and we were making a two-week trip out of it. So I hit YouTube to search for some video guides. That’s where I found Joolz Guides.

Julian McDonnell uses the channel to post London history walks and travel films. A filmmaker and actor, he also makes himself available for private guided walks through his web site. The idea came to me later than it should have, but in late February I contacted Julian to see if he would be available for a couple of pre-DConf walks. He doesn’t generally work with large groups, though he told me he had scheduled a pub crawl with 18 employees of a company. So we set 18 as the maximum size of a group, worked out a payment structure based on the total number of people, and I got Andrei’s approval for the foundation to cover the cost.

Around the same time, I got in touch with a pub near the venue. Finding a nightly gathering spot was a concern from the beginning. At past conferences, it was either the “official” hotel or, last year, the venue itself. There were several hotel options around the venue, many of them rather pricey. The budget hotels didn’t strike me as places where we could be holding our nightly “BeerConf”. I wanted to avoid the situation that happened in 2017 in Berlin when the hotel staff kicked us out of the lobby and relegated us to a back room. Belinda offered suggestions and I also sought advice from Russel Winder. Ultimately, I stumbled upon the Prince Arthur Pub while poking around Google Maps one night.

The pub has a second-floor space available for private hire. Getting it booked for three nights was a simple process. It also presented an opportunity for sponsorship. Ali Çehreli had been hoping to get his employer, Mercedes Benz Research and Development North America, to sponsor us in one form or another. We had been looking at potential swag, but now that we had the pub, he got approval for the company to cover the booking fee and a couple of rounds of drinks for each person who joined us each night.

Later on, Symmetry rented a different space nearby for the third night, including beer and food. So we had a cozy neighborhood pub for the first two nights of BeerConf and a more upscale bar for the third. Two different atmospheres that both allowed us all to have a good time.

The tours

The first of the two tours we booked with Julian took place two days before the conference, starting outside the Ritz at Green Park station (which, as it turns out, is right up the street from Symmetry) and ended up at the Strand near Charring Cross. The next day we met at Temple station and went through Temple (the legal district named for Temple Church, which was built by the Knights Templar) and on a winding route through the City of London.

Julian pointed out a number of sites we most likely would not have picked out on our own, giving us little nuggets of history for each. For me, some of the highlights were the building where the Beatles did their famous rooftop concert, the tailor shop that served as a front in the movie Kingsman: The Secret Service, the story of Temple Church (which we decided at the time not to go inside, but which I will visit on my next London trip to see the tomb of William Marshal), St. Etheldreda’s Church in Ely Place (dating back to the reign of Edward I), and the Charterhouse. Oh, and an interesting bit of trivia about the origin of the Japanese word for suit.

Julian is an entertaining guide and I believe everyone enjoyed the tours. If you’re ever in London with time to spare, I recommend you contact him about a private walk or a historical pub crawl.

99 City Road

The conference took place on the second floor of Inmarsat’s Old Street office building. Some of us arrived before 8:00 am on the first day and were directed by the security staff to a cozy little waiting area on the first floor. When the time came, we were guided to a side entrance and issued ID cards that would allow us access through the main entrance through Saturday.

As far as I’m aware, the conference went smoothly for just about everyone on site. There were a few hiccups along the way, most of which were noticed by few, if any, of the attendees. For example, those of us early arrivals on the first day found that the power outlets were located under trapdoors scattered throughout the room, but they were out of range of many of the seats. Before it became an issue, Belinda appeared with two of the venue staff, all bearing power strips. Belinda put out most fires before anyone smelled smoke.

Speaking of fires, we started the third day with a test of the building’s fire alarm system. It wasn’t a drill, just a test, so we didn’t have to go anywhere. All was well. Until Steven Schveighoffer got about 75% through his talk. The Stage Engage team edited it out of their recording, but in the live stream, you can see the point where Steve was interrupted by the fire alarm. This time, it was no test. We had to evacuate the building. Several folks got outside and were told to move down the street before abruptly being called back. Five minutes later, everyone filed back into the conference hall and Steve was able to finish his talk.

The venue staff encountered their own minor issue on the first day. At every DConf, we have mid-morning and mid-afternoon snacks, but the coffee is generally available all day. At 99 City Road, they’re used to events with “coffee breaks” in the mid-morning and mid-afternoon, where the coffee is set up and taken down along with the snacks. They soon learned that many DConf attendees are powered by caffeine, so they adapted and left the coffee out all day for the duration of the conference.

The food provided for our lunch each day was fantastic. Speaking for myself, it’s the best food I’ve ever had at a DConf (I did not attend the 2014 and 2015 editions, but I doubt that they compare). It was so good that I went back for seconds each day, which was possible because we had an abundance of food. If we find ourselves at this venue again, the quality of lunch is something we know we don’t have to worry about.

We did encounter one major issue this year, though it did not affect the conference attendees. A number of remote viewers on the first day encountered issues with the live stream, with some unable to see it and others having audio trouble. The venue was using Webex to handle the live stream. Sinisa Poznanovic, the venue’s A/V tech, attempted to switch to YouTube during the lunch break, but the video was oddly being flipped horizontally. He was unable to resolve the issue before the afternoon session, but he promised to stay after we left in the evening until he got it working. When we came in the next morning, the YouTube live stream was set up and working properly. I have to say it was a pleasure working with Sinisa and he has our gratitude for the great work he did throughout the conference.

The Webex issue is something none of us foresaw, but it’s possible we could have. Belinda had sent me a PDF with the links and login information several days before the conference. At the time, I was in Canterbury hanging out with a couple of old friends. I had never heard of Webex, but a cursory search on my phone showed that it’s owned by Cisco. That and the fact that it’s what the venue crew always use were enough to satisfy me, so I searched no further and went back to my vacation. In hindsight, had I dug more deeply into the search results (which I have since done), I would have learned that 64-bit Linux is not officially supported. Had I posted the login details to the forums as soon as I got them from Belinda, those with negative Webex experiences could have spoken up prior to the conference. Such red flags might have motivated us to insist on using YouTube instead.

In the future, we’ll require YouTube for all of our live streaming and, if we encounter anything new, I’ll enlist some help to do more thorough vetting in an effort to uncover potential problems.

The AGM

One of the proposals that came in during the talk submission period was from Nicholas Wilson outlining an Annual General Meeting. When the selection committee met to select this year’s speakers, we decided it would not be feasible to have an AGM as part of the regular schedule. We agreed instead to hold it before the Hackathon.

Just as the DConf Hackathon isn’t the sort of event most people think of when they hear the term, we didn’t envision the AGM as the sort of meeting corporate shareholders would be familiar with. We wanted to limit it to two hours so that we would have time for people to discuss their Hackathon plans before lunch. Nicholas had the idea and put together the agenda, so he would be the moderator. Ethan and I would roam the room with mics so attendees could ask questions. Initially, we had no plans to live stream the event, but in the end, we decided to do it anyway.

The meeting began with an announcement from Andrei. For those who haven’t heard, he is stepping away from his leadership role in the D Language Foundation. He’s still involved in the D community and still manages the foundation’s finances, but for personal reasons, he can no longer devote the time and attention a leadership role requires. Átila Neves was invited to join the team and take over that role. To what I’m certain will be the benefit of the D community, he accepted. It was made possible because Laeeth, his employer, agreed to allow him to do foundation work on Symmetry time.

One of the benefits of DConf is face-to-face communication. Some of the conversations that take place lead to new ideas, collaborations, and projects, but the majority of them are lost to time and memory. In our first AGM, we have not only the benefit of face-to-face communication but also a video record. We covered a lot of ground in the meeting: DUB, DIP 1000, the PR queue, shared, @property, std.experimental, DMD as a library, the DIP process, and more. The ideas put forward are there on video so they won’t be lost. And, thanks to the note-taking skills of Johannes Loher, we have a nice list of action items to work with.

One direct result of the AGM is that I recently revised the DIP process to address some concerns that have been raised in recent times. More items will be ticked off the action list over time.

The quarterly D Language Foundation meeting

The first two quarterly meetings took place over Skype. This time, we were able to hold it face-to-face. Andrei, Walter, Ali, and I were joined on site by representatives from a handful of D shops. We had scheduled the meeting during one of the talks so that we could ensure we’d have a quiet spot for those who were participating remotely. Unfortunately, we had issues with Google Talk for one remote participant while others who were to participate via Skype were too busy to attend.

Skyping these meetings is better than nothing, but meeting face-to-face was a tremendously more efficient and enjoyable experience. We had productive discussions on several topics that garnered more participation than the Skype meetings do, where the discussions tend to be less animated. The company reps aired their issues, we talked about some future plans, and all went well. Most of the items discussed will benefit the community at large when they are finally realized, e.g. Bugzilla issues and new tooling.

I expect our quarterly meetings will become a regular sideline event at future DConfs.

Until next year…

I thoroughly enjoyed myself at DConf this year. Last year, I was too busy emceeing to have much fun. As Ethan discovered this year, there’s more to the emcee job than one might expect (and I would say he’s much more suited to it than I am). When I did it, I was worried about drinking in the evening and wanted to get in bed early each night, so I only fully participated in BeerConf the final night. This year, I had no such concerns, though I did leave early the second night to surprise my wife for dinner.

I hope that everyone at DConf 2019 enjoyed it as much as I did. I also hope that those who were unable to attend this year, especially those who have never attended a DConf, can make the trip next year no matter where in the world we end up. Just think, it’s only been a few weeks since the conference, but we should be talking about DConf 2020 in just five more months.

The countdown is on!

Human Resources in the D Ecosystem (or Resources, Resources, Resources)

In November of last year, I announced here that we were launching a Pull Request Manager campaign. We wanted to raise $3000 in three months as compensation for Nicholas Wilson to make a dent in the pull request queues of the core D projects. The community answered the call and Nicholas got to work. It was a successful run, so we decided to go another round.

About a month remains in the second round campaign, but donations have come in at a slower pace. Still, we’re confident the community will once again help us reach the finish line so we can compensate Nicholas for his time. We intend to launch a third round at some point in the not-too-distant future, but before we do that we’ve got some different fish to fry.

The Human Resource Share

In December 2018, the first Quarterly D Language Foundation meeting was held online. It came together at the prodding of the aforementioned Nicholas Wilson. Some well-known D shops—Dunhumby (Sociomantic), Funkwerk, WekaIO, and Symmetry Investments—were invited to send representatives to join Walter, Andrei, Ali Çehreli, Nicholas, and myself in a Skype call. Given that it was the inaugural meeting, the agenda was light. We primarily wanted to hear what the companies’ biggest D issues were at the time so that we could prioritize them in the work pipeline. However, we did raise one idea that we wanted the companies to consider for our benefit.

A persistent problem in the D ecosystem is a lack of available human resources to work on the issues that don’t fall into the realm of personal or corporate interests. By that I mean that volunteers tend to contribute where they have a personal interest and contributions from the companies tend to be aimed almost exclusively at areas that have a direct benefit to their projects. The result is that a large number of issues that do not fit into either category fall by the wayside. This is only to be expected and we aren’t complaining. What we are doing is trying to determine how to direct energy toward those neglected issues without the need for raising money.

So we asked the companies to consider a form of “human resource sharing”, the idea being that each company would periodically designate one employee to spend a day on the company dime working on D ecosystem tasks that don’t necessarily have a direct impact on the company’s interests. The representatives promised to take it back to their bosses and give us an answer at the next meeting.

In the interim, I wrote up a more concrete proposal that outlined two options for approaching it: a monthly rotation where each company takes turns doing the work, and a quarterly system where each company commits to completing at least one item on the task list per quarter. I asked the companies to provide us with their preference at our second quarterly meeting in March.

At the March meeting, we invited a few more companies to join us. Given that these are either in the startup phase or aren’t using D exclusively, I’ll play it safe and keep quiet about who they are for now. Overall, the response to the Human Resource Share was positive. Unfortunately, most of the companies are already short on human resources as it is and cannot commit to our quarterly scheme. However, one company did commit to starting immediately and another committed to providing work as they can. All of them committed to helping in other ways, which includes the provision of funds (see below), and hope to have the human resources to spare in the future.

To that end, we’ve set up the ecotasks repository to house our Ecosystem Task List. The list was initially envisioned as a collection of specific tasks, e.g. specific Bugzilla issues, but that makes it more difficult for each tasked worker to decide what to do. Instead, we’ve cobbled together a set of task groups. For now, that consists primarily of links to the GitHub Issues page for different projects and a request to “close as many issues as possible”. The idea is that workers can go to an issues page and work on solving those they can squeeze into their allotted time.

We’ve put this on GitHub not just as a means of transparency, but also because we would like to invite the entire community to participate. The list is loosely sorted by priority in that items higher on the list are considered higher priority than those lower on the list, but there’s no relative priority between specific tasks. The ordering is sure to change over time.

We ask that anyone working through the list to, at the start of the work session, open an issue and leave a comment indicating which item is being worked on. For example:

I’m working on the dub registry issues right now. Specifically, I plan to tackle issues #I, #J, and #K.

Then, at the end of the work session, close the issue with a note indicating what was accomplished.

If you have a little time to spare one weekend, please consider visiting the ecotasks repository and taking on one or two issues. Even better, challenge yourself to go through it once a month and see what you can accomplish. D depends on volunteer effort to thrive. We have a lot of it already, but we always need more. This is one of many ways to make an impact even if it isn’t an enjoyable or very visible one.

The Human Resource Fund

There are some tasks in the D ecosystem that no amount of cajoling and begging will get done because they’re too complex, too time consuming, require a specific skill set to properly complete, or all of the above. When the companies offered to throw money at us in place of human resources, that led us to a new idea.

We are now running a permanent fundraising campaign specifically aimed at solving the bigger issues. The Human Resource Fund for D Ecosystem Tasks is intended to grow and grow and grow. We’re currently in talks with some of the companies about how often and how much they can contribute toward it and in what amounts. We also invite the community at large to donate to it now and again.

Recently, Andrei mentioned in the forums that we need to put together a qualified team to complete the spec and implementation of shared. That’s an example of the sort of big issue we want to use this fund to solve. Donations small and large are equally welcome. The sooner we can get a nice pile built up, the sooner we can start prioritizing issues and finding the people to solve them.

For now, we want to focus on beefing this fund up a bit so we’re going to hold off temporarily on the next Pull Request Manager campaign, but we’ll definitely come back to that again before too long. Anyway, the current campaign is still in need of your attention!

Effecting Change

Every D user has different priorities and goals, different needs and desires. A full-featured IDE is important to one person but not even worth mentioning for another. One programmer expects to see a native D GUI, another is happy with bindings to an existing C or C++ library, and yet another has no need of a GUI at all. One person contributes to a certain D project, but never to any others, while another person has a different set they contribute to, or starts their own. Somewhere in the middle are the issues that are too boring or too complex, the issues that never rise to anyone’s attention or are considered undoable for whatever reason.

In the time I’ve been following and involved with D, I’ve seen the leadership try a number of different ways to drive energy toward some of these unsolved issues. What every approach they’ve tried has had in common is that they depend on the community. When you don’t have the human resources to do the job, you need the money to hire the human resources. When you don’t have the money, you need to ask for and rely upon the charity and goodwill of others. And if there’s no one with the bandwidth to continuously push the issue, fewer people step up. When the community doesn’t step up, then either someone on the core team has to (at the expense of time taken from their normal workload) or the issue languishes.

The two initiatives I’ve described above, the Human Resource Share and the Human Resource Fund, are the latest attempts to make things happen. Again, all contributions are welcome and appreciated! I don’t want this to come off as a complaint, because that’s not what is intended at all.

This is a call to arms! We’re asking members of the community to roll up their sleeves and do the dirty work they normally wouldn’t think to do, or even would prefer not to do. We’re asking for crowdsourced effort in solving problems that will make the D ecosystem better for all of us. A few dozen people spending an hour here or a weekend there will mean more issues closed and more members of the core team can stay focused more often on the work in their purview, which is another big win.

If you can’t help us out with your time, help us out with your money! The Human Resource fund will always need boosting. So, too, the General Fund, the current PR Manager campaign, and any other campaigns we launch in the future. You can also support us when you shop at Amazon by doing so via smile.amazon.com and selecting “D Language Foundation” as your supported charity. When you buy products marked “Eligible for AmazonSmile donation” through smile.amazon.com, the foundation will receive 0.5% of the purchase price. Finally, you’ll soon be able to support D through the DLang Swag Emporium, where you’ll be able to purchase D-themed t-shirts, coffee mugs, and more.

We’re always open to ideas on how to get more things done. If you have anything you’d like to suggest, bring it to the forums for community discussion or email me directly at aldacron@gmail.com.

Now, let’s make things happen!

DConf 2019 Early-Bird Registration Opens (and Other News)

I’ve got a few big news items for you today, the biggest being one I’ve been eagerly awaiting: early-bird registration for DConf 2019 London is now open!

Early-bird Registrations…

are now open!

We’re very fortunate, thanks to the generosity of Symmetry Investments, to be able to keep the normal registration fee to our standard $400 this year. Like last year, the early-bird discount is once again 15%, so register before March 17, 24:00 AOE, and you’ll only pay $340. Unfortunately, unlike past editions of DConf, we’re required to charge a VAT of 20% this year, so the early-bird rate with tax is $408 and the regular registration will be $480 with tax. The GBP value of the VAT is listed alongside the price on the registration page and will be updated when the average monthly exchange rate changes.

Currently, you’ll find two options for payment on the registration page: Flipcause and PayPal. Those of you who have been following the latest goings on will know that we’re using Flipcause to create donation campaigns. They also provide support to configure campaigns for events like DConf, allowing us to keep as much as possible coming into one place. In the long run, this will be more efficient for us than accepting money through other services, so if you aren’t paying with a PayPal balance for your DConf registration, we ask that you please choose the Flipcause option.

You’ll also find the DConf campaign listed in our Campaign Menu, which is accessible from the big Donate Now button in this blog’s sidebar as well as from the D Language Foundation’s donation page. In the past, we have supported Eventbrite as a payment option, but have not yet decided if we will do so this year.

Invited Keynote Speaker

I’m also happy to announce that Laeeth Isharc has accepted our invitation to be a keynote speaker at DConf 2019.

Every year, Walter and Andrei are joined by an invited keynote speaker to open each of the three presentation days of the conference. Sometimes, the speaker is from outside of the D community for a different perspective (Scott Meyers was the invited keynote speaker at DConfs 2014 and 2017, and Martin Odersky was the invited keynote speaker last year). This year, offering the invitation to Laeeth was a no-brainer.

Not only is Laeeth responsible for bringing DConf 2019 to London under the sponsorship of Symmetry Investments, he’s also an enthusiastic supporter of the D programming language. He hires D programmers, sponsors open-source D projects, initiated the Symmetry Autumn of Code (SAoC), and can be found in various forums around the internet answering questions about D. Despite all of that, he’s never spoken at a DConf. We can’t wait to hear what he has to say!

The SAoC Finalist

Three programmers started the Symmetry Autumn of Code. Each participant was to complete three milestones, each of one-month duration, and would receive $1000 upon the successful completion of each. After a final month of clean-up work (whatever was required by each project), one participant was to be selected for a final $1000 payment and a free pass to DConf 2019, including travel and lodging.

One participant was unable to continue after the first milestone. The other two—Francesco Mecca, whose project was porting Leandro Lucarella’s old D1 forking GC to DRuntime, and Francesco Galla, whose project was adding HTTP/2 support to vibe-http—were able to see the event through to the end. Both did some excellent work on their chosen projects, but only one would be selected for the final prize.

I can now announce that congratulations are in order for Francesco Galla! He’ll be receiving the final payment and the trip to DConf 2019. As it turns out, he and the other Francesco happen to be friends. They had an agreement beforehand that the finalist would use the extra $1000 to pay for the other to attend DConf. And we’ve been informed that we’ll be fortunate enough to meet both of them in London!

We’ll also be hearing from Francesco Mecca before then, as he has agreed to write about his project for this blog. Francesco Galla will either write a blog post or, depending on how the conference schedule comes together, give a presentation about his project at DConf (possibly both!). Keep an ear open for the announcements.

The New Fundraiser

The PR Manager Campaign was a tremendous success. Not only did we meet our goal well before the deadline, but Nicholas Wilson has done a great job cleaning up the pull request queues. We will continue that campaign for another quarter, starting next month.

In the meantime, we’re raising $2000 for a new server for the D Forums. There are two reasons we’ve selected this as our next target.

First, Vladimir Panteleev has been paying for the server for the D Forums (and his other D services) out of his pocket for years. It’s time we put a stop to that. The forums are a crucial part of the D programming language community and it shouldn’t be up to one person to pay the bills.

Second, the forums have been experiencing performance issues at an increasing frequency over the past several months. Among the possible solutions that Vladimir has identified to improve this state of things, moving to better hardware is a big one. If ever there was a time for the community to take over the server bills, it’s now.

So we encourage you to donate to the cause! Helping us meet our $2000 goal will cover a new server for the forums and provide a cushion for any incidental expenses. Vladimir has graciously declined to accept any money from the D Language Foundation for the work he does in actually maintaining and improving the forums, so we’d like to draw your attention to his Patreon account, where you can more broadly support the open-source work he does.

We thank Vladimir for all the time and money he’s put into this over the years, and thank you in advance for your donations!

Last Year In D

2018 was one of the biggest years in D we’ve had for a while. It was the first complete year that DMD stuck to a steady release schedule. Over the course of the year, the language got new features and cruft cleanup. The compiler even got some new user-facing features. At the same time, the community-at-large continued to evolve the surrounding ecosystem.

Release Rundown

The first DMD release of the year was version 2.078 on January 1. It set the tone for much of the year, including more support for a stripped-down runtime for -betterC mode, and a deprecation change in arithmetic rules.

The next major release came out in March, and I personally found it to be the most exciting release in a couple of years because it came with improved error messages – notably, calling out the specific argument to a function that mismatched a type (though I will note there is still a lot of room for further improvement!). Moreover, this release introduced DMD’s -i flag, which automatically includes imported modules and has proven to be a major convenience to me over the year.

Meanwhile, more stripped-down runtime support and general language cleanup was included in the release. March also introduced experimental @nogc exception throwing (as proposed in DIP1008). This release also changed the garbage collector to be lazily initialized, part of the effort to make things cheaper and pay-as-you-go.

March was also the first time DMD on Windows could do a total build of a 64-bit program without Visual Studio installed because it bundled the LLVM linker.

Keeping to the release schedule, May and July brought us new major versions, which focused on language cleanup, with a lot of deprecations. There was a #dbugfix campaign over the year, and it brought a fix in May’s release: allowing the pow operator (^^) and several std.math functions to be used at compile time. July’s release brought a change in contract syntax, based on the proposal in DIP 1009, allowing small expressions instead of requiring whole blocks.

Version 2.082, in September, finally brought me something I have wanted for a long time: User Defined Attributes on enum members and function parameters, as well as the ability to disable DRuntime’s exception trapping via the --DRT-trapExceptions=0 runtime command-line switch, which allows for easier debugging of uncaught exceptions. Moreover, with this release the D Language Foundation began digitally signing the Windows binary releases, which has helped smooth out the end-user experience when installing and running DMD. It should continue to help solve the false-positive problem we have seen with some virus scanners.

At the end of the year, version 2.083 saw extern(C++) get a big improvement its users have been waiting for: namespaces without scopes, which makes C++ interoperability and code organization a lot easier. Work on making the C++ standard library accessible from D has been progressing throughout the year.

Overall, the year brought a lot of awaited improvements. D is more usable in various runtime library situations (even Phobos has some -betterC support, like RefCounted and Tuple!). The language had over ten deprecations of old cruft like comma expressions, read-modify-write on shared, comma expressions, class allocators (did you even know D still had class allocators? To be honest, I thought they were formally deprecated years ago, but it was actually 2018 when the change officially happened!), and more.

Debugging got better, with the uncaught exception switch, but also allowing the debug statement to escape more attribute restrictions, better error messages out of the compiler, and a -verrors=context switch to show the line right in the message. -betterC debugging aids also improved, with the C assert function being a possibility. I’ll also note that 2019 is already moving further—new assert printing code was merged just recently; we can choose how to control them and whether to use D or C facilities!

Of course, the library continued to get better range support and its support for @nogc and -betterC has grown as well.

And as a fun fact, DMD is now 98% ported to D, with another major part converted over the year.

The D Community

More companies invested in D in 2018. Hosted by QA Systems, DConf 2018 was held in Munich, Germany, the first time DConf has been to that city (and the third time in Germany, after taking place in Berlin in 2016 and 2017). Symmetry Investments sponsored the successful Symmetry Autumn of Code, funding three students to work on D-related projects.

run.dlang.io came out at the end of 2017 and grew in popularity throughout 2018, becoming the new standard for running D code online, including from the dlang.org homepage. It even supports a whitelist of third-party libraries.

The dlang-tour website gained a few new translations from the community, including Vietnamese, Portugese, French, Turkish, German, and Ukrainian.

The D Language Foundation was busy in 2018 as well. They received over $5000 on Open Collective, $3000 of which was earmarked to support development of the code-d VS Code plugin and the language server that drives it. With a successful campaign launched through Flipcause, the DLF was able to hire a pull request manager for three months. Donations through all platforms allowed them to fund some outreach efforts as well as student work via scholarships. To increase visibility, they submitted a history of D paper to the Fourth ACM SIGPLAN History of Programming Languages Conference.

Over 2,500 pull results have now been merged by the dlang-bot on GitHub, and the Project Tester has now gained 50 projects that it tests on every PR to give real-world data on compiler regressions and the impact of breakages.

DIP 1014, “Hooking D’s struct move semantics”, was also accepted. It will open some doors that were closed by design in old D, but that many developers had found limiting. Before, the compiler could move your structs whenever it wanted. Now, it will be hookable to give more control to the programmer and avoid a nasty case of bugs.

I opened up my documentation generation website at dpldocs.info to all DUB projects this year, and upstream linked to it, encouraging D library authors to better document their libraries and allowing users to better evaluate the libraries before downloading them.

On Twitter, @dlang_ng was started and has gained about 200 followers. All announcement topics from the Announce forum (aka the digitalmars.D.announce newsgroup) are tweeted out here.

2018 also saw community announcements of the autowrap and dpp projects. autowrap automatically wraps existing D code to be used from other environments, while dpp runs a C pre-processor over D code to make it possible to use C headers, unmodified, macros and all, directly in D. These two projects show the community’s desire to integrate D more fully into existing codebases and projects.

Of course, not all is perfect in the D development community, including a few areas where this author would like to see improvement.

The most +1-ed PR on DMD, a string interpolation implementation (and a very elegant approach in this writer’s opinion), remains open and of uncertain status, despite a renewed effort to get it merged in December. Similarly, the State of D 2018 Survey identified several areas for improvement. Very few of these came to pass in 2018, though some progress was made. We deprecated some cruft, but auto-decoding, one of the top-five disliked Phobos features in the survey and one often derided in the forums, remains in place. We are still using Bugzilla, which got an average of 3.27 satisfaction rating out of a possible 5 stars in the survey. We achieved fewer regressions, but added even more to the attribute bloat. We have an official blog and the annual DConf, but the D Language Foundation’s inner workings are still opaque.

It would be great if some of these weak points are addressed among the improvements and changes in 2019.

The DUB Package Manager

The code.dlang.org website received some criticism in 2017, and it was addressed in 2018. The site got a new front page with usage statistics and project ranking based on GitHub info. The DUB program itself got faster and more stable, no longer going online on every run to check for updates, and achieving better online uptime.

The Mir library, which has algorithms, collections, and more written to be 100% -betterC compatible (which, of course, means it also works in all other D environments, too), is now the fifth-most popular package on the DUB registry, while the unit testing library unit-threaded takes the third-place slot. First, second, and fourth are all related to vibe.d, which is the project DUB was originally created to support.

Conclusion

2018 was a really good year for D. There is still much work to do, but worthwhile developments in all facets of the language and ecosystem gave me renewed excitement going into 2019. I expect that March 2019 release of D is going to be another big step toward improving the best programming language we have on Earth today!


Adam Ruppe is the author of D Cookbook and maintainer of This Week in D. Modules from his freely available arsd package are used throughout the D community. He is also known for his legendary DConf 2014 presentation.


Addendum

There were a total of 166 contributors listed on the DMD changelog in 2018. Special thanks to them and to all the others in the D community who flesh out the ecosystem and make our favorite programming language!

  • 0xEAB
  • Adam D. Ruppe
  • Alexandru Caciulescu
  • Alexandru Jercaianu
  • Alexandru ermicioi
  • Alexibu
  • Ali Akhtarzada
  • Ali Çehreli
  • Andrei Alexandrescu
  • Andrei-Cristian VASILE (87585)
  • Andrey Penechko
  • Andy Smith
  • Aravinda VK
  • Arun Chandrasekaran
  • Atila Neves
  • BBasile
  • Basile Burg
  • Bastiaan Veelo
  • Benoit Rostykus
  • Brad Roberts
  • Brian Schott
  • Carsten Schlote
  • Chris Coutinho
  • Daniel Kozak
  • Dashster
  • David Bennett
  • David Gileadi
  • David Nadlinger
  • Denis Feklushkin
  • Diederik de Groot
  • Dmitry Olshansky
  • Dragos Carp
  • Duncan Paterson
  • Eduard Staniloiu
  • Elias Batek
  • Erik van Velzen
  • Eugen Wissner
  • FeepingCreature
  • GabyForceQ
  • Giles Bathgate
  • GoaLitiuM
  • Greg V
  • H. S. Teoh
  • Harry T. Vennik
  • Hiroki Noda
  • Héctor Barreras Almarcha [Dechcaudron]
  • Iain Buclaw
  • Ilya Yaroshenko
  • Ionut
  • Jack Stouffer
  • Jacob Carlborg
  • Jan Jurzitza
  • Jean-Louis Leroy
  • JinShil
  • Joakim Noah
  • Johan Engelen
  • Johannes Loher
  • Johannes Pfau
  • John Belmonte
  • John Colvin
  • Jon Degenhardt
  • Jonathan M Davis
  • Jonathan Marler
  • Jordi Sayol
  • Joseph Rushton Wakeling
  • Kai Nacke
  • Kevin De Keyser
  • Kotet
  • Laeeth Isharc
  • Lance Bachmeier
  • Leandro Lucarella
  • LemonBoy
  • Lucia Mcojocaru
  • Luís Marques
  • Manu Evans
  • Manuel Maier
  • Markus F.X.J. Oberhumer
  • Martin Kinkelin
  • Martin Krejcirik
  • Martin Nowak
  • Mathias Baumann
  • Mathias Lang
  • Mathis Beer
  • MetaLang
  • Michael Parker
  • Mihails Strasuns
  • Mike Franklin
  • Mike Parker
  • Márcio Martins
  • Nathan Sashihara
  • Nemanja Boric
  • Nicholas Lindsay Wilson
  • Nicholas Wilson
  • Nick Treleaven
  • Oleg Nykytenko
  • Patrick Schlüter
  • Paul Backus
  • Per Nordlöw
  • Petar Kirov
  • Pjotr Prins
  • Pradeep Gowda
  • Quirin F. Schroll
  • Radosław Rusiniak
  • Radu Racariu
  • Rainer Schuetze
  • Razvan Nitu
  • Remi Thebault
  • Robert burner Schadek
  • Roman Chistokhodov
  • Ryan David Sheasby
  • Ryan Frame
  • Sebastian Wilzbach
  • Simen Kjærås
  • Simon Naarmann
  • Spoov
  • Stanislav Blinov
  • Stefan Koch
  • Steven Schveighoffer
  • Superstar64
  • Temtaime
  • Tero Hänninen
  • Thibaut CHARLES
  • Thomas Mader
  • Timon Gehr
  • Timoses
  • Timothee Cour
  • Tomáš Chaloupka
  • Tyler Knott
  • Unknown
  • Vlad Vitan
  • Vladimir Panteleev
  • Walter Bright
  • Yannick Koechlin
  • Yuxuan Shui
  • Zach Tollen
  • Zevenberge
  • aG0aep6G
  • abaga129
  • byebye
  • carblue
  • cedretaber
  • crimaniak
  • devel
  • deviator
  • dhasenan
  • dmdw64
  • drug007
  • dukc
  • gapdan
  • glitchbunny
  • growlercab
  • jercaianu
  • jmh530
  • kinke
  • leitimmel
  • look-at-me
  • n8sh
  • rracariu
  • shoo
  • shove70
  • skl131313
  • thaven
  • viktor
  • wazar
  • olframw
  • yashikno
  • Ľudovít Lučenič

D-lighted, I’m Sure

For me, finding D is the most recent step along a road winding back at least a dozen years. I’d been searching for a cross-platform development language/environment (POSIX and Windows, but not so much mobile since my search began before mobile was really a thing) and at this point, D fits better than anything else I’ve tried. I won’t go out on a limb and say it’s the Holy Grail of X-Plat, but at the very least, it’s brought some fun back into coding for me. And whenever I massage a hunk of code until it finally works… well, I’m addicted to those little victories.

The Road to D

The first language I tackled back in the mid-oughts to meet this end was PHP. I’d been a web developer for a few years when I found out that PHP had a standalone desktop version. When I then stumbled across Andrei Zmievski’s PHP-GTK, lightbulbs went on and fireworks went off. For a while. The big drawback I found with PHP-GTK was that no PHP compiler could handle the GTK end. So after a few years of patiently hoping someone would tackle this while I wrote nearly 40 blog posts on its use, I started looking elsewhere.

Back in the 1990’s, I’d been steeped in Javascript and HTML, writing simple online apps for banking clients out of Vancouver, BC and later, Bancroft, Ontario. With such a background, when I ran across Electron a few years ago, it seemed like a good fit. I assumed learning it would be easy-peasy-lemon-squeezy, but a lot about the nature of web development languages had changed. To boot, several more languages, standards, and paradigms had been thrown into the mix, so to say the least, I found it all confusing and more than a little intimidating. What I really wanted was one language, an easy distribution system, and a GUI toolkit that didn’t necessitate balancing style sheets with front- and backend code as well as JSON files. And Electron, unfortunately, needs to drag Chromium along for the ride in a little metaphorical red wagon. It’s a solution, but not the one I was looking for.

Then last year, I stumbled onto D. I’d been hearing about it for years, but I hadn’t read much about it. I didn’t realize it was available across so many POSIX platforms. I also didn’t realize it embraced the OOP paradigm and so hadn’t given it much thought. I liked the subtle humor of D being next in line after B, C and A (which oddly enough came along later than B and C), but after a brief smile, I paid no more attention until last October.

When I finally took a good look, I realized that with its curly braces and OOP propensity, D runs right up my street. But before rolling up my sleeves, I made sure there was a GUI toolkit I could use, something that didn’t necessitate balancing three differing paradigms at once. I found a list of GUI toolkits on the D Wiki and was gratified to see GtkD among them. So for the last two plus months, I’ve been putting most of my effort into learning how D and GtkD work together.

Perspiration and Refreshments

It may or may not be important to know this, but I don’t have a CS degree. I’m completely self-taught, a process that started while stuck in the middle of a frozen nowhere for three weeks more than 30 years ago. But that’s a whole other story. My point is, there are holes in my education. That’s what happens when you follow your nose instead of a syllabus.

Because of that, some of the intricacies of D elude me and may always do so. Although I’ve read Ali Çehreli’s chapter on the subject (from Programming in D) I have no idea what mixins are or why I’d want to use them. And templates seem like a good idea, but I don’t know why. I blame my lack of formal CS education for this, but I’m quite comfortable with classes and objects, so I’m not sweating it.

I was first introduced to OOP and the Gang of Four when I was learning PHP, so D covers familiar territory in that regard. Curly braces are another thing I feel quite at home with, having used them for most of my programming life.

But I’m finding the familiarity of D to be a bit of a stumbling block as well. It’s just different enough from C and PHP to mean I have to work hard at pounding those differences into my brain. I deal with it through rote typing of example code. I figure if I copy out enough D code instead of lazing along with copy-n-paste, eventually I’ll push C and PHP far enough into the background that I can see past them. And I keep Mr. Çehreli’s book handy so I don’t go completely off the rails. It’s been quite helpful. And speaking of helpful…

The Ecology of D

Forums

So far, I’ve signed up for two D-oriented forums, one at dlang.org and the other at gtkd.org. I have yet to find an unfriendly avatar. Everyone I’ve encountered seems willing to jump in and help. To contrast these forums with one I’ve been active on for a few years (not mentioning names, but this other forum is related to art and graphics, not programming) on the D forums I haven’t been insulted, nor have I been questioned for looking at things from a (warning: film term) Dutch angle—which is one of the things I do best. That comes from my art background, I suspect. I was quite the rabble rouser in art college. (Just ask Alan Wood the installation artist about our knock-down-drag-out shouting match in the cafeteria of Emily Carr College if you don’t believe me. But again, that’s a whole other story.)

On dlang.org, I mostly hang in the Learn sub-forum, which I suppose is only natural at this stage. It’s probably the most polite bunch I’ve run across on a forum ever, and I’ve been frequenting forums since the hoary days of the BBS when 1200 BAUD was lauded as the fastest thing since the 427 hemi.

Mike Wey seems quite patient for someone who answers more-or-less the same question over and over again on the GtkD forum. The only negative I found with that forum was technical. I signed up, made some posts, and when I went back to sign in a second time, I had to reregister. But I was still identified as the same Ron Tarrant who signed up the first time (I think) so perhaps that’s how things are supposed to be. It’s unusual, but it works.

I will also mention that the GtkD documentation pages are a mind-bender, but because this is where I’m planning to spend a lot of time over the next while, I’ve decided to pitch in and help make things more accessible.

I’m drawing on experiences writing my PHP-GTK blog back in 2006 and porting a bunch of code examples and tutorials into GtkD. I went so far as to buy a domain name (gtkDcoding.com) in preparation for launching a site and blog covering all this. That’s how I deal with learning curves and have done since I wrote that series of BASIC tutorials for my sister while freezing in a Newfoundland outport back in 1985.

Tools

I got up and running with dmd quite quickly. Installation on Windows 10 and FreeBSD was straightforward. A few quick questions on the D forum, and I had everything I needed to do single- or multi-file projects. A few more questions answered on the GtkD forum and I was comfortable enough to start porting my PHP-GTK code.

But I have to say, DUB eludes me. I don’t know if it’s because Electron left me feeling like JSON files are a disgruntled engineer’s revenge plot or if it’s just the way my brain works. Perhaps if I put my mind to it, I could learn it, but since I’m getting the results I want from plain ole dmd, I’ve done no more than skim DUB’s docs up until now. Just for the record, back in my C days, make files were a caution for me, too. I eventually licked them, and if it ever becomes important enough to me to figure out DUB, I guess I will. But for now, my heart’s not in it.

Finding a text editor that supports D (and especially GtkD) syntax highlighting rather than—as a few people on the forum stated—supporting C and getting ‘good enough’ support for D, led me to abandon the search and roll my own. So far, I’ve done more-or-less complete highlighters for PSPad and CodeBlocks. To be fair, they both support D out of the box, but not GtkD which is important to me, mostly as an aid to remembering module names.

I haven’t even looked at other tools. I’m dimly aware of some other sub-forums for what look like other tools, but to be honest I haven’t read them. As I said, dmd does the job, so I’m satisfied for now.

Conclusion

And that’s a quick summary of my first two months with D. On the one hand, I don’t really understand some of D’s paradigms, but on the other, the ones I do relate to are meeting my expectations. The only two things I haven’t found yet are:

  • information about how I would go about packaging a D app (with GtkD) for distribution, and
  • how to build on one platform for distribution on another (if that’s even possible).

I’m willing to put in some time on this and eventually get gtkDcoding.com off the ground as my way of giving back. It’s been a long time since those frozen weeks with a Coleco Adam when I tried to explain BASIC to my thirteen-year-old sister (who is now a paramedic and doesn’t care any more). And I must say, I’m as excited now about D and GtkD as I was about BASIC and general computing way back then. Being retired, I have the time to pursue it and I’m looking forward to becoming a regular member of the D community.

And in case you don’t know what a Dutch angle is, I urge you to watch some episodes of Batman from the 1960’s. See how the camera tilts when things go wrong for our heroes? That’s a Dutch angle.

Bio

I was inspired to learn programming while vacationing in a frozen Newfoundland outport in April of 1985. It started as a desperate attempt to keep from stripping down to my shorts and disappearing into a blizzard, but became a lifelong passion along with acting, writing, and music. In keeping with this non-typical start, it was in art college where I learned my first serious programming language (6502 assembly) and later it was on a job as a technical writer and artist that I wrote my first serious code for a client, an online mortgage calculator for a credit union in British Columbia. The culmination of my programming career was finishing the PHP-GTK app, Corkboard, a writer’s tool for story planning. (Don’t bother looking. I couldn’t come up with a distribution system, so it languishes here on a back-up drive.)

Since dropping out of high school in 1972, I’ve made a living as a taxi driver, musician, screenwriter, technical writer, artist, sound reinforcement equipment salesman, and biology lab technician among other things. I’ve also made money acting, programming, and promoting concerts. I retired from Statistics Canada in 2010 and have since divided my time between acting, writing lame novels, pursuing the elusive X-Plat beast, and keeping house for my wife of 33 years.

Symmetry Autumn of Code is Underway

Earlier this year, Laeeth Isharc brought an idea to the D Foundation for Symmetry Investments to sponsor a summer of code. He was eager to provide a few motivated individuals the incentive to get some great work done for D and the D community. Sporadic email discussions preceded some chatting at DConf 2018 Munich and the idea subsequently began to pick up steam. By the time the details were sketched out, it had transformed into the Symmetry Autumn of Code.

The SAoC projects

Eight applicants submitted their resumes and project proposals for three slots. Nearly all of the proposals were taken from the SAoC suggestions page at the D Wiki. Given the limited window, the selection process went fairly quick. Of the three selected, two had mentors attached. It took a little while to find a mentor for the third selection, so we extended the milestone deadline for that participant. Now I can happily say that all three are well underway.

A fork-based concurrent GC for DRuntime

Francesco Mecca proposed this project. The goal is to take Leandro Lucarella’s original D1 fork-based GC, port it to D2, adapt it to DRuntime’s GC interface, and culminate with a pull request for DRuntime to present the work and open discussion. Leandro agreed to mentor this project and is working with Francesco to develop a test suite that the port must pass as part of Milestone 2. They have also included documentation in their milestone list, which is good news.

vibe.d HTTP/2 implementation

This one came from Francesco Galla, who is currently pursuing a MSc in Network and Security. The goal here is to enhance the vibe-http library to support HTTP/2. Fittingly, Sönke Ludwig, the maintainer of vibe.d, agreed to mentor the project, but requested someone to share the load due to his schedule. Sebastian Wilzbach stepped up as co-mentor.

This project involves rewriting the current HTTP/1 API, ensuring it works as expected, then incrementally adding support for HTTP/2. Portions of the rewrite were already completed before SAoC came along, but had not yet been tested. As such, testing and bug fixing will be a significant portion of the first milestone.

Porting the Mago debugger to D

This project was proposed by László Szerémi. He’s a heavy user of debuggers in D and wants to see the situation improved. He believes that porting the Mago debugger to D is a major step in that direction. His first two milestones are concerned with translating, testing, and bug fixing. To cap off the SAoC event, he intends to get a GUI frontend up and running with some basic features.

László had no mentor when he applied, and no one had specifically volunteered to mentor any debugger projects, so we put out a call in the forums and I reached out directly to a few people. In the end, Stefan Koch agreed to take it on.

SAoC is not the end

I know that one of the goals Laeeth had behind his initial suggestion for this event was to enhance the D ecosystem. None of the selected projects are simple, one-shot tasks. These are projects which will all require attention, care, and effort beyond SAoC. The participants are getting them started and we hope they’ll continue to maintain them for some time to come, but in the end, these are projects for the community. When SAoC 2018 is behind us, it will ultimately be up to the community to determine if the projects live long and prosper or die young.

I’ll post more about SAoC and the participants as the event goes on. We wish them the best in meeting their milestones!

Funding code-d

At the DConf 2018 Hackathon, I announced that the D Language Foundation would be raising money to further the development of the suite of tools comprising code-d, the Visual Studio Code plugin. The project is developed and maintained by Jan Jurzitza, a.k.a. Webfreak001.

The plan was that if we reached $3,000 in donations on our Open Collective page, we’d make the money available to Jan. When we finally reached the goal, I was quite happy to tweet about it:

Jan and I still hadn’t finalized the details at that point, but we finally got it all sorted a few days later. Before I show how we’ll be using the money, I’d first like to talk about why we’re doing this.

The motivation

As the D programming language community has grown, so has the amount of work that needs to be done. Since D is a community-driven language, much of the work that gets completed is done by volunteers. Unfortunately, the priorities for those who actively contribute aren’t necessarily aligned with those of everyone else, and this can lead to some users feeling that there is little progress at all, or that their concerns are being deliberately ignored. That’s just the nature of the beast. More potential pain points and unfulfilled wishlists means there’s a continual need for more manpower.

Therein lies one of the most fundamental issues that has plagued D’s development for years: how can more people be motivated to directly participate so we can keep pace with the consequences of progress?

To be clear, everyone involved in the core team is aware that lack of manpower is not the only negative side effect we suffer from growth, but it is certainly a big one. Part of my role these days is to figure out, along with Sebastian Wilzbach, ways to mitigate it. My inbox is littered with conversations exploring this topic, along with recommendations from Andrei to research how other projects handle one thing or another for inspiration.

To that end, in early February I launched the #dbugfix campaign with the goal of increasing community participation in the bug fixing process by providing a simple way for folks to bring attention to their pet peeve issues (keep it going people – participation has slowed and we need to pick it up!). We also announced the State of D Survey, initiated by Seb, at the end of February to get an idea of what some of the personal pain points are and where people think things are going swell. We’ve got other ideas in development, one of which I’ll be telling you about in the coming days.

Among the data that jumped out at us from the survey is that Visual Studio and Visual Studio Code were the most popular IDEs/editors among participants. The plugins for both Visual Studio (Visual D) and VS Code (code-d/serve-d/workspace-d) are each maintained by solo developers.

That’s a heavy workload for each of them. Look at the issues list for code-d and you’ll find several there (39 open/132 closed as I write), but the PR list is much shorter (1 open/21 closed as I write). What I infer from that is that the demand for improvements is higher than the supply for implementing them. And from personal experience, I know there are projects like code-d whose maintainers would love to add new features, or improve existing ones, but simply can’t (or aren’t motivated to) make the time for it.

Here then is a good place for a little experiment. What if the D Language Foundation started raising money to help fund the development of specific projects like code-d? By creating the opportunity for those with extra money but little time to directly contribute toward fixing their personal pain points and shrinking their wishlists, can development be focused toward getting bugs fixed and improvements implemented more quickly?

Long story short, I picked the VS Code plugin for our trial run because it was prominent in the survey data, it’s cross-platform, and the serve-d component can be beneficial to other IDE/editor plugins. Open Collective launched their goal system just as I was trying to determine how best to go about it, so it seemed the obvious choice (though we didn’t learn until after our announcement that it wasn’t the system we thought it was, i.e. it’s not a system of multiple, targeted fund raising goals but instead a milestone on the global donation progress bar at the top of our Open Collective page). I contacted Jan just before DConf to gauge his interest, then finally had a meeting with the Foundation board the night before the Hackathon to get approval and decide how to go about doling out the money (we decided the money should be paid out in increments based on completion of milestones, with the option to provide money as needed for targeted development costs, e.g. a Windows license, bug bounties, etc). It all came together fairly quickly.

We plan to continue this initiative and raise money for more work in the future. Some will be like this trial run, with the goal of motivating project maintainers to focus development and spend that extra hour or two a day to get things done. Some will be aimed at specific development tasks, as bounties or as payment for a single part-time or full-time developer. We’ll experiment and see what works.

We won’t be using Open Collective’s goals for this (or any other service) going forward. Instead, we’re going to add a page to the web site where we can define targets, allow donations through Open Collective or PayPal, and track donation progress. Each target will allow us to lay out exactly what the donations are being used for, so potential donors can see in advance where their money is going. We’ll be using the State of D Survey as a guide to begin with, but we’ll always be open to suggestions, and we’ll adapt to what works over what doesn’t as we go along.

As long as there are people willing to put either time or money into the D language, community, and ecosystem, this will surely pay off in the long run with more visible progress, hopefully solving a wider range of pain points than currently happens in our “what-I-want-to-work-on” system of community development. We’re looking forward to see the synergy that results from this and other initiatives going forward.

The milestones

Jan has some specific ideas about how he wants to improve his plugin tools and was happy to take this opportunity to put them all front and center. I asked him to make a list, then estimate how much time it would take to complete each task if he were working on it full time, then divide that list into roughly equal milestones of 3–4 weeks each. That does not mean he will be working full time, nor that each milestone will take 3–4 weeks; payment is not dependent on deadlines. It was simply a means of organizing his task list.

We ended up with two sets of tasks as a starting point and agreed to a $500 payout upon completion of both milestones. Once all the tasks in a milestone are completed, the $500 will be paid. At the end of the second milestone, we’ll determine what to do next – another milestone of feature improvements, a heavy round of bug fixing, or something else – and how to make use of the remaining $2000.

Following is a list of the tasks in both of the initial milestones, with a brief description of each directly from Jan (lightly edited).

Improvements & maintainence

  • add automated tests for windows 7, 10, archlinux, ubuntu, fedora, osx for startup, from scratch project initialization, reloading, adding dependencies
    There are automated tests in the workspace-d repo which test different scenarios and combinations with dub, DCD, and d-scanner, ranging from general usage to weird setups. Identifying more issues (especially on windows), creating more tests and especially running all the tests on several platforms would be a good first step towards a less buggy version of serve-d which should work on all platforms equally well.
  • Single file mode
    A requested feature and one which makes the usage of the plugin a lot easier would be single file usage. Currently, single file usage is very limited and without an open workspace it doesn’t work at all. Adding this (with quick startup time and low resource usage) will make code-d a good alternative to vim or notepad for quickly editing a D file.
  • fix unknown DDoc macro parsing
    Currently, if any unknown DDoc macros are used, they are omitted from the documentation renderer, which makes some documentation unreadable and weird. For this, libddoc needs to be worked on by identifying where exactly the macros are discarded, how they could be kept track of, and how to make it more obvious in the API without breaking backwards compatibility.
  • project template creator as separate plugin (with standardized template format if possible)
    Currently the “Create Project” command is a very static command reading from a directory in the plugin. Making this a separate VS Code plugin with an API for other plugins to extend with more functionality would make it more flexible and open it to new platforms and APIs. Also, research on if there are already some standards for this and if we can parse other template formats.
  • detect used imports & add command to remove unused ones & add Remove unused imports + sort command
    Add a way to find out if imports are used and add a command to remove unused ones. DCD could probably be extended for this.
  • code folding on grammar
    The expand/collapse feature on arrays & functions (bracket based) and lambdas. (Instead of indentation based)
  • parse dub commandline arguments
    Dub command line arguments should be able to be passed into the build buttons and future tasks. This will require parsing them because dub is used as an API and not as a program.
  • experiment with adding a mode to DCD to make all symbols available
    A mode to always give all public symbols regardless of imports and additionally provide the import where the symbol comes from (so when completing it, the IDE adds an import)

UX & Tools for beginners

  • improve UI/UX for building and running projects (implement tasks api) 
    As the tasks API is now actually released, I can incorporate the different build types into the task system of VS Code. This would include having build (debug, release), run, test for each detected dub project and submodule.
  • make workspace symbol search show more symbols
    The workspace symbol search (Ctrl-P -> #) currently is using just DCD to find exact matches for types. I would include DScanner ctag/etag results in this response to make the search better.
  • code action: improve implement interface (parse existing functions and not include them)
    Test cases need to be written, existing interfaces must be parsed, cross-file it should all still work, inherit documentation, etc. Need to check in real world applications where interfaces and classes are used. Also add support for abstract classes.
  • auto complete in diet templates (both HTML tags and D code)
    Add static HTML auto completion for tags & attributes and create a virtual D file for D code for DCD for autocompletion.
  • code action: create getter/setter for variable
    For code like int _foo;, create the property functions int foo() @property const { return _foo; } int foo(int value) @property { return _foo = value; } with automatic detection if const can be used. This makes writing classes easier (snippets already exist, but this is more intuitive) and shows beginners how properties are created. The property should be added where other properties are. It could check the comments for “Getters”, “Setters” and “Properties” if possible, otherwise just put it after a block of members.
  • code action: convert boolean parameter to Flag (+import)
    This should change the definition of a function void foo(bool createDirectory) to alias CreateDirectory = Flag!"createDirectory"; void foo(CreateDirectory createDirectory) and add the appropriate import if it doesn’t exist yet.
  • code action: make foreach parallel (with import if neccessary)
    Replace a foreach (a; array) with foreach (a; parallel(array)) and import std.parallelism.
  • code action: convert enum to bitflag enum (isBitFlagEnum assert + modify values)
    Convert any enum enum Foo { a, b, c } to enum Foo { a = 1 << 0, b = 1 << 1, c = 1 << 2 }, add static assert(isBitFlagEnum!Foo); and import std.typecons if neccessary.
  • search for unknown symbol or import online
    Use dpldocs.info. Example: import mongoschema; -> Error importing -> Search online code fix -> Add dub package mongoschemad, etc. For undefined symbols the same: check local projects for symbols, otherwise see if online providers know anything.
  • show local references from DCD & local rename
    Highlight a symbol in the document when hovering on a member and provide the ability to rename it (not cross-file, because DCD doesn’t support that).

Thank you

Thanks again to all of the donors who pushed us over the goal line for this. Though we realize not everyone will be entirely pleased with the milestone list, and that it would have been better if we had finalized everything before asking for donations rather than coming at it haphazardly like we did, we hope that most of you find more here to like than not.

It will be a little while yet before we set up the new system and announce the next goal, as we’ve got other initiatives taking priority at the moment. So please be patient. We haven’t forgotten about the State of D Survey, nor are we going to let the results sit and bitrot. This train may be slow sometimes, but it’s rolling on.

Communal Benevolence Required

DConf 2018 is coming up fast. The final pieces of preparation are soon to fall into place and then the day will be upon us. Already, we’re looking at what comes next.

We recently put out a survey to gather a good deal of feedback from the community. At that time, we promised that the survey results would be used to guide some initiatives going forward. Though we’ve been silent on that front, it’s still in the works. You should be hearing an announcement at DConf about the first such initiative, and a subsequent blog post will lay out the framework we’ll be following going forward.

One thing all of our initiatives will have in common is that they will require resources. That, unfortunately, is not something the D Language Foundation has in large quantities. Whether we’re talking about time, money, or manpower, D’s development is constrained by the limited resources available. Fortunately for us, one can be used to drum up the other two, so in the absence of enough folks able to donate their time and manpower, we’ll be relying on donations of money to buy what we need to meet the community’s goals.

The same holds for DConf. In order to keep the registration fees relatively affordable, reimburse the speakers, buy the T-shirts, and fund all of the costs associated with the event, the Foundation uses money from the donation pool to cover what the registrations don’t. That’s exactly the sort of thing the donations are for, of course, but the more we dip into the pool for the conference, the longer it takes to replenish after.

So if you’re looking to help the D community, this is an excellent way to do it. You can head over to our OpenCollective page, or choose one of the alternatives on our donation page at dlang.org, and throw a few dollars at us. Leave a note on the donation form letting us know you’re doing this to support DConf 2018.

As a reminder, if you haven’t registered for DConf and aren’t planning to, the Hackathon on May 5th is open to all. So if you’re in the area that day, come on in and see us!