A Decade of Magnum

Let’s try this new thing called C++0x

Back in the 2000s, C++ was this ven­er­able old lan­guage that wasn’t ex­actly fun to use. C++11, ori­gin­ally code­named C++0x, ar­rived to change that. I saw a huge po­ten­tial in the new fea­tures, es­pe­cially scoped enums, move se­mantics and the short­hand {}-ini­tial­iz­a­tion — “clas­sic” C++, with non-copy­able classes al­loc­ated with new, enums with ugly pre­fixes to avoid name clashes and foo(const Vector3&) / foo(float, float, float) over­loads for everything sud­denly felt like an ugly and in­ef­fi­cient main­ten­ance night­mare — and so I de­cided to design the lib­rary with C++11 in mind.

It was a risky de­cision and hard at first be­cause I was com­monly hit­ting ex­cit­ing yet-un­dis­covered com­piler bugs. In­dustry vet­er­ans re­peatedly told me us­ing C++11 was not a good idea, nev­er­the­less I’m happy that I didn’t back off on that. (Even though back­port­ing to GCC 4.4 to run on a Beagle­Board and to MS­VC 2013 to fi­nally run on Win­dows was truly hellish.) The us­ab­il­ity and clean­ness of the design simply wouldn’t be there without C++11, though to this day I feel un­easy about us­ing lambdas be­cause I re­mem­ber how bad the sup­port was in the early days, and I can still enu­mer­ate all un­fin­ished corners in the GCC 4.7 STL im­ple­ment­a­tion.

Today, Mag­num is still C++11 and I’m not really plan­ning to re­quire a new­er stand­ard ver­sion any­time soon. While Mag­num users are com­monly on C++17, there isn’t any­thing nearly as ground­break­ing as was in C++11 to be use­ful on the lib­rary level. I’m now hap­pily on the “cave­man” side of the spec­trum and users that chose Mag­num be­cause it was the only thing still com­pil­ing on a Cen­tOS with GCC 4.8 con­firm that I made the right de­cision.

Make up­grades pain­less and APIs easy to de­prec­ate

When evolving a lib­rary, there’s al­ways a tradeoff in how much you al­low your­self to in­nov­ate without mak­ing users too mad. If up­grades are pain­ful, users will hes­it­ate to up­grade, which means there will of­ten be nobody to dis­cov­er bugs in new code. In turn, even less users want to up­grade, res­ult­ing in a down­wards spir­al of the lib­rary in­creas­ingly los­ing touch with user ex­pect­a­tions.

I de­cided to not both­er with ABI com­pat­ib­il­ity be­cause it makes it ba­sic­ally im­possible to ex­per­i­ment, and that’s no good in the fast-chan­ging land­scape of GPU de­vel­op­ment. On the oth­er hand, caus­ing com­piler fail­ures every time the users up­grade would mean they even­tu­ally lose pa­tience and leave.

Over time, I real­ized what were the pos­sib­il­it­ies for API-com­pat­ible fea­ture de­prec­a­tion:

• A func­tion can have an inline ali­as that matches the pre­vi­ous sig­na­ture as long as the two over­loads are un­am­bigu­ous
• A class can be turned in­to a typedef and vice versa (which is use­ful when mak­ing a class tem­plated or turn­ing a tem­plate in­to a con­crete class)
• A namespace can be re­named and its mem­bers ex­posed un­der the old name via using, typedef or namespace Old = New
• Head­ers can be re­named, split or joined and de­prec­ated “prox­ies” kept that #include from the new loc­a­tions
• A CMake tar­get can hide a lot of back­wards com­pat­ib­il­ity — link­ing to lib­rar­ies that were split out of the ori­gin­al tar­get, adding back ob­sol­ete in­clude paths, modi­fy­ing com­piler flags …
• Plu­gin in­ter­faces can be changed freely as long as all the virtual func­tions are kept private, out of users’ reach. That way plu­gin im­ple­ment­a­tions can be switched to use the new in­ter­faces and back­wards com­pat­ib­il­ity provided only in the non-vir­tu­al pub­lic API in­stead of every plu­gin.

And what were the lim­its:

• Plain struc­ture data mem­bers are ba­sic­ally im­possible to de­prec­ate, so if you de­cide to have giv­en mem­ber cal­cu­lated on-the-fly or change its name / type, you have no way to provide back­wards com­pat­ib­il­ity. This is why all Mag­num get­ters are inline func­tions, with data mem­bers nev­er ex­posed dir­ectly.
• It’s not gen­er­ally pos­sible to re­order func­tion ar­gu­ments, un­less the types are dis­tinct and not im­pli­citly con­vert­ible to each oth­er. I’m stuck on this in a few places (Util­ity::Dir­ect­ory::write() is one).
• Func­tion re­turn types are hard to change, un­less the types are im­pli­citly con­vert­ible to each oth­er or oth­er­wise com­pat­ible. This is also why I dis­cour­age users from auto — if they use a con­crete type, back­wards-com­pat­ib­il­ity meas­ures en­sure what’s re­turned is im­pli­citly con­ver­ted to what’s ex­pec­ted. Thanks to that the change from std::unique_ptr to Con­tain­ers::Point­er every­where went so smooth for most of the users.

While I want user code to keep build­ing, I also want to nudge them to up­grade to new­er APIs and drop back­wards-com­pat­ib­il­ity ali­ases and wrap­pers after a year or two after de­prec­a­tion. Apart from de­prec­ated APIs be­ing clearly marked as such in the doc­u­ment­a­tion, this is done us­ing the COR­RADE_DE­PREC­ATED() fam­ily of mac­ros, which on suf­fi­ciently re­cent com­pilers can add de­prec­a­tion warn­ings on everything in­clud­ing func­tions, classes, enums, namespaces and files. One step fur­ther, the users also have an op­tion to dis­able back­wards com­pat­ib­il­ity al­to­geth­er and fix even the spots the com­piler didn’t / couldn’t warn about.

This paid off the most when I made the OpenGL wrap­per op­tion­al in 2018 — while it was ba­sic­ally a com­plete re­write of the most cent­ral parts, the com­pat­ib­il­ity ali­ases made ex­ist­ing code still com­pile, only with a ton of warn­ings that told people what to change and how. The up­grade went sur­pris­ingly well for every­body and I fi­nally re­moved the com­pat­ib­il­ity in­ter­faces earli­er this year.

I’m happy to re­port that I have sev­er­al long-time users run­ning their pro­duc­tion code off Mag­num master — be­cause they trust it that much.

Friendly vendor lock-in

While pro­jects made by in­dustry vet­er­ans with good repu­ta­tion are usu­ally trus­ted im­pli­citly, a pro­ject from an un­known has to be ex­cep­tion­ally per­fect to make a dent. Though … why even lower the bar as the repu­ta­tion builds up over time?

Even though many suc­cess­ful pro­jects can do with only a Git­Hub README just fine, I spent sev­er­al months build­ing a whole CSS lay­out frame­work and site theme from scratch, with a fast doc­u­ment­a­tion search ar­riv­ing shortly after. The time wasn’t wasted and it was great to hear feed­back say­ing that people miss this in oth­er pro­jects, or see people ad­apt­ing Mag­num’s doc­u­ment­a­tion sys­tem for their pro­jects.

Apart from that there’s vari­ous minor Qual­ity-of-Life fea­tures one gets quickly used to like the 90.0_degf or 0x3bd267_rgbf lit­er­als, abil­ity to con­veni­ently print al­most any con­tain­er or enum with De­bug or as­sert mes­sages that show what ex­actly went wrong, not just that some­thing wrong happened. To help ad­op­tion, small re­usable bits of Mag­num were ex­trac­ted to single-head­er lib­rar­ies so users can eas­ily bring their fa­vor­ite APIs to oth­er pro­jects as well.

Find­ing the right amount of NIH

While in­vest­ing time in­to writ­ing my own math lib­rary was worth it as I could design some­thing from the ground up without be­ing tied to how GLSL works or how math was done in the C++03 days, at­tempt­ing to write my own phys­ics lib­rary was a mis­take. It was a use­ful learn­ing ex­per­i­ence tho — next time I made sure that if I go all ar­chi­tec­ture as­tro­naut on some­thing, I’ll im­ple­ment the ac­tu­ally use­ful bits (in case of phys­ics, in­ter­sec­tion al­gorithms) on a lower level first so when it ul­ti­mately doesn’t work out, I don’t need to throw away everything.

The be­ne­fits of a layered ap­proach was some­thing that dawned on me very slowly — ori­gin­ally, to get any­thing on screen, it was man­dat­ory to use a scene graph, a cam­era ab­strac­tion and a bare­bones GLUT-based ap­plic­a­tion. Today, you can use for ex­ample just the math lib­rary and out­put to a SVG, without touch­ing the GPU or open­ing any win­dow, and the lib­rary pieces are largely in­de­pend­ent. As the uses broadened from games to ed­it­ors to re­search demos to heavy­weight data pro­cessing, it be­came clear that one design can’t fit everything and while Mag­num APIs can be bet­ter for a cer­tain use case than a com­mon 3rd party lib, there are also use cases for which the same 3rd party lib is more suited than Mag­num.

• The built­in Math lib­rary is fea­ture­ful and well doc­u­mented, but if you need to do heavy cal­cu­la­tions on large sparse matrices, Ei­gen is a bet­ter fit. And we can do the type con­ver­sion for you. Or maybe the big­ger half of your pro­ject already uses GLM? No wor­ries either.
• Mag­num has a hier­arch­ic­al SceneGraph, but you can simply not use it, if you don’t need. Or, if you’re in­to ECS in­stead, use entt.
• There’s Plat­form::Sdl2Ap­plic­a­tion that cov­ers most plat­forms, or Plat­form::GlfwAp­plic­a­tion if you need just desktop and want to be lean­er. Or Plat­form::Em­scrip­tenAp­plic­a­tion if you want some­thing op­tim­ized for the web. Or go com­pletely cus­tom and em­bed Mag­num in GTK, wx­Wid­gets or Qt — for each there’s a boot­strap pro­ject to get you star­ted.
• Want phys­ics? Feel free to use Mag­num with Bul­let, DART, Box2D or for ex­ample PhysX. In­teg­ra­tions, help­ers and ex­amples provided.
• Want to use some OpenGL or Vulkan API that’s not wrapped by Mag­num, or com­bine with 3rd party ren­der­ing? No prob­lem.
• Don’t like Mag­num’s OpenGL or Vulkan wrap­pers at all, ac­tu­ally? Turn them off and keep us­ing Mag­num as­set man­age­ment APIs in com­bin­a­tion with sokol_g­fx, bg­fx and oth­er 3rd party ren­der­ers.

I have to ad­mit it took quite some ef­fort to swal­low the pride and ac­cept the fact that it’s simply not hu­manly pos­sible to make Mag­num the best op­tion for every use case — but in the end I real­ized that if I give the users con­veni­ent in­teg­ra­tions with 3rd party lib­rar­ies, they will hap­pily stay be­cause the re­main­ing parts of Mag­num are still worth it for them.

It’s not good to use the STL but it’s not good to not use it either

Ori­gin­ally I wanted to up­grade to C++14 as soon as it comes out to make use of std::op­tion­al, how­ever that con­tain­er got over­en­gin­eered bey­ond any reas­on and delayed to C++17; then I con­sidered up­dat­ing to C++17 to get std::string_view and std::array_view, how­ever string views ended up im­mut­able and use­less and the ar­ray view got re­named to std::span and delayed to C++20, and std::mdspan might fi­nally ar­rive in C++23 if things go well …

One of the oth­er long-term is­sues was std::unique_ptr. #include <memory> had a meas­ur­able im­pact on com­pile times from the very be­gin­ning, which is why I hes­it­ated to use it in class in­tern­als, mean­ing a lot of PIMPL’d state was in­stead man­aged (and routinely leaked) us­ing clas­sic new / delete.

Even though it was sug­ges­ted nu­mer­ous times, I frowned upon the thought of “writ­ing my own STL” or switch­ing to STL­port / EASTL, be­cause do­ing so would mean ali­en­at­ing com­mon users — they would not only need to learn a new en­gine, but also write ex­tra code to trans­form their std::vec­tors and std::strings to some­thing the en­gine used. Only re­l­at­ively re­cently I real­ized that I can design con­tain­ers that are both STL-in­de­pend­ent and STL-com­pat­ible, so the en­gine can be­ne­fit from faster com­pile times and ex­tra flex­ib­il­ity like memory own­er­ship trans­fer, but users can still keep us­ing std::vec­tor, std::unique_ptr and the like, of­ten without even real­iz­ing those are not the types the en­gine nat­ively works with.

A friendly com­mu­nity is what keeps the pro­ject tick­ing

And fi­nally, I can’t un­der­es­tim­ate how much this pro­ject owes the com­munity on Git­Hub, Git­ter and else­where for its ex­ist­ence. I had the luck that over the years I only had to deal with a single per­son with of­fens­ive be­ha­vi­or, every­one else is go­ing out of their way to help each oth­er, provide valu­able feed­back and en­cour­age­ment and con­trib­ute back massive amounts of great code.

I’m happy to see that every ef­fort to help users do their first steps or re­solve their is­sues has re­turned back nu­mer­ous times and the com­munity brings a con­tin­ued whole­some ex­per­i­ence.

Thank you, every­body. Cheers for the next ten years.

Cov­er im­age cre­ated with Gource, re­flect­ing the state from Decem­ber 19th, 2020.