Monthly Archives: October 2016

Project Highlight: libasync

d6libasync is a cross-platform event loop library written completely in D.  It was created, and continues to be maintained, by Etienne Cimon, who started it as a native driver for vibe.d, a modular asynchronous I/O framework most often used for web app development in D.

In 2014 or so, I was looking for a framework to power my future web development projects. I wasn’t going to use an interpreted language, as binary executables were too attractive. I found vibe.d appealing because, coming from C++, it was relatively simple and featureful. So I studied it, along with the D programming language and the Phobos standard library.

vibe.d has always used libevent under the hood by default. This is where Etienne ran into a problem that bothered him.

I stumbled on some workflow issues when deploying vibe.d apps to other operating systems which may or may not have the right version of libevent in the package repository. I didn’t want to package a DLL with my server, or have to go through dependency hell with my software, and I wanted everything to be consistently written in D to reduce the mental complexity of switching programming languages or to debug other issues.

So he decided to study up on the system APIs across the platforms supported by DMD (Windows, Linux, *BSD and OS X) and create his own event loop library in D. Now he, and anyone using libasync, can issue a single command with DUB to compile and execute a web application without needing to worry about external event loop dependencies.

libasync takes advantage of D’s delegates to provide a very intuitive interface.

void testDNS() {
	auto dns = new shared AsyncDNS(g_evl);
	dns.handler((NetworkAddress addr) {
		writeln("Resolved to: ", addr.toString(), ", it took: ", g_swDns.peek().usecs, " usecs");
	}).resolveHost("127.0.0.1");
}

Etienne says of the code snippet above:

The D garbage collector will keep the AsyncDNS object in dns alive for as long as the delegate used in the parameter of dns.handler is alive in the heap, which is in this object. The delegate syntax is more simple to declare than Javascript, and it is also type-safe. This DNS resolver will work on any platform thrown at it, thanks to D’s compile-time version conditions.

libasync makes use of the asynchronous I/O facilities available on each supported platform and provides a number of event-handlers out of the box.

Cross-platform event handlers have been defined for DNS resolution, UDP Messages, (Buffered/Unbuffered) TCP Connections, TCP Listeners, File Operations, Thread-local (Notifiers) and Cross-thread Signals, Timers and File Watchers. The intrinsics involve EPoll for Linux, KQueue for OS X and BSD, and overlapped I/O for Windows. With all of these features thoroughly tested through a vibe.d driver, libasync has become a very fast and reliable library which I use in all of my projects. My benchmarks show it as being a little slower than the libevent driver in vibe.d, though its self-explanatory code base makes it seamless to understand, maintain, and deploy.

A libasync driver has been added to vibe.d and work is going on to improve the library’s performance.

The stability of the underlying OS features makes for very little need for changes, although there is a big improvement involving the proactor pattern in the works for libasync and a new architecture for vibe.d. Together, those two developments are likely to increase the library’s performance significantly.

If you find yourself needing an event loop in D and want to give libasync a spin, you can visit the library’s page at the DUB repository for information on how to add it as a dependency to your own DUB-managed projects. libasync, in turn, has only one dependency itself, another library maintained by Etienne that provides a set of allocators and allocator-friendly containers called memutils.

It wasn’t so long ago that anyone using D who wanted something like libasync or memutils would need to either roll their own or bind to a C library. The ever-expanding list of libraries in the DUB repository, created and made available by members of the D community like Etienne, make it much easier to jump into D today than ever before.

GSoC Report: std.experimental.xml

Lodovico Giaretta is currently pursuing a Bachelor Degree in Computer Science at the University of Trento, Italy. He participated in Google Summer of Code 2016, working on a new XML module for D’s standard library, Phobos.


GSoC-icon-192I started coding in high school with Pascal. I immediately fell in love with programming, so I started studying it by myself and learned both Java and C++. But when I was using Java, I was missing the powerful metaprogramming facilities and the low level features of C++. When I was using C++, I was missing the simplicity and usability of Java. So I started looking for a language that “filled the gap” between these two worlds. After looking into many languages, I finally found D. Despite being more geared towards C++, D provides a very high level of productivity, as correct code is easier to read and write. As an example, I was programming in D for several months before I was bitten by a segfault for the first time. It easily became one of my favorite languages.

The apparent lack of libraries, my lack of time, and the need to use other languages for university projects made me forget D for some time, at least until someone told me about Google Summer of Code. When I discovered that the D Foundation was participating, I immediately decided to take part and found that there was the need for a new XML library. So I contacted Craig Dillabaugh and Robert Schadek and started to plan my adventure. I want to take this occasion to thank them for their great continuous support, and the entire community for their feedback and help.

This was my first public codebase and my first contribution to a big open source project, so I didn’t really know anything about project management. The advice about this field from my mentor Robert has been fundamental for my success; he helped me improve my workflow, keep my efforts focused towards the goal, and set up correctness tests and performance benchmarks. Without his help, I would never have been able to reach this point.

The first thing to do when writing a library is to pick a set of principles that will guide development. This choice is what will give the library its peculiar shape, and by having a look around one finds that there are XML libraries that want to be minimal in terms of codebase size, or very small in terms of binary size, or fully featured and 101% adherent to the specification. For std.experimental.xml, I decided to focus on genericity and extensibility. The processing is divided in many small, quite simple stages with well-defined interfaces implemented by templated components. The result is a pipeline that is fully customizable; you can add or substitute components anywhere, and add custom validation steps and custom error handlers.

From an XML library, a programmer expects different high level constructs: a SAX parser, a DOM parser, a DOM writer and maybe some extensions like XPath. He also expects to be able to process different kinds of input and, for std.experimental.xml, to “hack in” his own logic in the process. This requires a simple, yet very flexible, intermediate representation, which is produced by the parsing stage and can be easily manipulated, validated, and transformed into whatever high-level construct is needed. For this, I chose a concept called Cursor, a pointer inside an XML document, which can be queried for properties of a given XML node or advanced to a subsequent one. It’s akin to Java’s StAX (Streaming API for XML), from which I took inspiration. In std.experimental.xml, all validations and transformations are implemented as chains of Cursors, which are then usually processed by a SAX parser or a DOM builder, but can also be used directly in user code, providing more control and speed.

Talking about speed, which in XML processing can be very important, I have to admit that I didn’t spend much time on optimization, leaving a lot of space for future performance improvements. Yet, the library is fast enough to guarantee that, for big files (where performance matters), an SSD (Solid State Drive) is needed to move the bottleneck from the fetching to the processing of the data. Being this is an extensible and configurable library, the user can choose his tradeoffs with fine granularity, trading input validation and higher level constructs for speed at will.

To conclude, the GSoC is finished, but the library is not. Although most parts are there, some bits are still missing. As a new university semester has started, time is becoming a rare and valuable resource, but I’ll do my best to finish the work in a short time so that Phobos can finally have a modern XML library to be proud of. I also have a plan to add more advanced functionality, like XML Schemas and XPath, but I don’t know when I’ll manage to work on that, as it is quite a lot to do.

Project Highlight: DlangUI

Vadim Lopatin is an active D user who, like many in the D community, comes from a Java and C++ background.

My current job is writing a Java backend for a virtual call center . I’ve also worked as a C++ developer on IP PBX devices. Programming is my hobby as well. My biggest hobby project, which I’ve rewritten from scratch twice in the last 15 years, is CoolReader, a cross platform e-book reader written in C++.

He kept hearing news about D and, over time, became more interested in its “cool features”, like CTFE and code generation. So, three years ago, he decided to initiate a couple of projects to learn its features.

DDBC is a database connector similar to Java’s JDBC, with an API close to the original. HibernateD is an ORM library, similar to the Java-based Hibernate. Unlike Java, D allows the use of compile-time code introspection and code generation. It was interesting work, and I was impressed by the power of D.

Both projects proved to be no more than learning exercises, however, as he never used either himself and neither became popular in the community. Now they are largely abandoned, but he has since found another area where he could apply his talents and, as it turns out, where community interest has been much higher.

The new idea came about as he surveyed the state of available GUI libraries in D. While there were several options to choose from, he wasn’t satisfied by the fact that they were all either non-native wrappers or not cross-platform. He had already written a cross-platform GUI in C++ for CoolReader GL, a version of his ebook reader that uses the same GUI on all supported platforms. Why not implement another one in D?

He has a long list of items he thinks are important for a GUI library to check off. A few of them are:

  • Cross-platform — the same code should work on all platforms with simple recompilation.
  • Internationalization — it should be easy to write multilingual apps. Unicode everywhere. Strings externalized to resources.
  • Hardware acceleration — take advantage of DirectX or OpenGL where available, but it should be possible to use software rendering where they aren’t.
  • Resolution independence — flexible layouts must be used instead of fixed pixel-by-pixel positioning of controls.

A markup language for describing layouts, touch screen support, 3D rendering, customizable look-and-feel, easy event handling, and several other items complete the list. A big set of requirements for one person to work on alone, but he already had a good deal of experience with the GUI he wrote for CoolReader. So when he got going with his DlangUI project, his previous work is where he started.

Part of DlangUI is a direct port of the CoolReader GL GUI. It was easy to reuse big parts of C++ code thanks to the similarity of D and C++ syntax.

So he set about checking items off of his list. Such as support for hardware acceleration via an interface that easily supports different rendering backends, one of which is implemented using OpenGL. But as things got under way, he discovered that there is one particular issue with porting C++ to D that arises in the parts that can’t be directly reused.

The D GC does not bring any help for resource management, since object destructors may be called in any thread, in any order, or never at all. If an object owns some resources, it ought to be destroyed in a predictable way. Therefore, widgets and other objects holding resources must be destroyed manually by their owners

DlangUI uses reference counting for easy freeing of owned objects. Widgets remove their children on destroy. Windows remove their widgets when closing. I had to add debug mode instance counts for various objects, and corresponding messages in the log, to make sure all resources are freed gracefully.

Some resources (e.g. images) are cached. Their references may be taken from the cache, used, and then released often. To allow cleanup of caches, all such resources have usage flags. The cache provides a checkpoint method which removes the usage flag from all items, and a cleanup method which frees all cache items which have not been used since the last checkpoint.

He has worked on a number of items from his list, such as theme customization.

DlangUI themes are inspired by the Android API. It borrows Android’s state drawables (they may be even used as is), nine-patch PNGs, and resource versions for different screen sizes or resolutions. Usually widgets don’t use a hardcoded look and feel or layout properties. Instead, they use a style ID referencing to currently selected theme. If the theme is changed in runtime, all widgets receive a corresponding notification so that they can reload any cached values from the new theme. Simply providing a new theme changes the  look and feel significally.

Currently, two standard themes are provided in DlangUI: default (light) and dark. Applications may specify a standard theme as a parent, and override only the styles it needs. Standard theme resources are usually embedded into the application executable using the cool D feature import("filename"). Applications may embed their own resources as well. This allows creating a single file app withoug any additional resource files needing to be shipped with the executable.

Another check mark can  be place next to layouts. Here, he again looked to Android.

To support multiple screen resolutions and sizes, widgets must be placed and resized using layouts instead of direct pixel-based positioning. DlangUI uses Android API-like layouts for grouping, placing and resizing widgets, based on a two-phase measure/layout scheme.

And, while a GUI can be assembled entirely in code, he took inspiration from elsewhere for ideas to knock the markup item off his list.

Manually writing code to create a widget hierarchy and setting their properties is a bit boring. DlangUI offers the possibility to create widgets using a JSON-based description similar to Qt QML. I call it DML. Currently, only the creation of widgets and the setting of their properties are supported. In future, I hope to add the ability to describe signal handlers in DML, and automatically assign signals to handlers, and widget instances to variables. There is a GUI app, dlangui:dmledit, which helps to write DML. It combines a text editor for DML and a preview window to see the results.

When it comes to being cross-platform, a lot has been done so far, thanks to different backend implementations: Win32, SDL2, DSFML, X11, and Android. Not long ago, Vadim even announced a text-based interface which works in the Linux terminal or Windows console.

It was a real surprise for me how few changes were required to implement text-mode support. Besides the backend code and the text-mode drawing buffer implementation, most of the changes came in the form of a Console theme. Only a few fixes were required in the widgets, removing several hardcoded margins and sizes. Even DlangIDE, a DlangUI-based IDE for the D programming language, is now usable in terminals.

Here’s what DlangUI’s components normally look like on Windows.
screenshot-example1-windows

And this is what DlangIDE looks like running in the Windows console.

dlangide

When compared to screenshots of programs running with different DlangUI backends, seeing it in a terminal like that is pretty darn cool.

DlangUI manages event handling via signals and has built-in support for 3D graphics, including a 3D scene package. And work still continues on making Vadim’s list smaller, as well as addressing the problems with the library.

The most mentioned issue the non-native look and feel of widgets. Although it’s possible to make a theme looking exactly like native one, it would not track system theme changes anyway. There’s no system menu support on OS X and in Gnome (where a common menu is used for all apps). The documentation is poor. There is some DDOX-generated documentation, but it’s not detailed enough and I seldom update. I need more tutorials and examples. And some advanced controls are missing, e.g. an HTML view.

He also says that there are too few developers working on the project. While some users have submitted PRs, the majority of the work has been done by Vadim alone. Given what he has produced so far, that’s a pretty impressive achievement. But, in addition to solving the problems above, he’s got a lot more he wants to implement, such as:

  • An XML+CSS rendering widget to show/edit HTML or rich text
  • Refactoring DlangUI to extract window creation, OpenGL context creation, drawing, font support, input events code from widget set – for cases when no widgets are needed
  •  Mobile platforms support improvements – add iOS backend, improve android support, improve touch mode support
  •  Native system menu support on OS X and Gnome
  •  Support for fallback fonts in font engines, from which to get missing symbols
  •  A native OSX backend based on Cocoa instead of libSDL2
  • Improvements in Scene3D to make it suitable for writing 3D games

If you need a GUI for your D app, DlangUI is a viable option today. More importantly, if you’re able and willing to help out here and there, Vadim sure could use a few more hands on a few more keyboards!