The new release brings Python bindings, Basis Universal texture compression, improved STL interoperability, better Unicode experience for Windows users, a more efficient Emscripten application implementation, single-header libraries, new OpenGL driver workarounds and much more.
STL-compatible container classes and further work on reducing compile times
Continuing from Containers::Pointer and Containers::Optional introduced in the previous release, the 2019.10 release adds STL compatibility to its Containers::ArrayView classes as well. In practice this means std::vector or std::array instances can be implicitly converted to Magnum array views, and if you’re on the bleeding edge and use C++2a std::span, the conversion can go both ways:
#include <Corrade/Containers/ArrayViewStl.h> // enables implicit conversions /* Image view backed by a std::vector */ std::vector<char> pixels = …; MutableImageView2D image{PixelFormat::RGBA8Unorm, {32, 24}, pixels}; /* Getting the data back as a span on C++2a */ std::span<char> span = image.data();
The new release also introduces a reworked Containers::StridedArrayView, now supporting multiple dimensions and zero / negative strides, giving it a feature parity with numpy.ndarray in Python. It’s also getting used in a broader set of APIs, including MeshTools::generateSmoothNormals(), range-taking Math::min() or the very useful Image::pixels() that gives raw typed access to pixel data, and even print them on terminal:
All this interoperability however doesn’t mean all headers suddenly need to
#include <vector>
or the like. On the contrary — the conversion is
enabled by including dedicated headers listed below, together with the ability
to forward-declare some STL types when you don’t need the full definition.
Put all together, this means the code both compiles faster (when you don’t need
to use STL types) and can interoperate with STL types better (when you actually
want that).
- Corrade/Containers/ArrayViewStl.h makes array views convertible from std::vector / std::array
- Corrade/Containers/ArrayViewStlSpan.h makes array views convertible from / to C++2a std::span
- Corrade/Containers/OptionalStl.h enables conversion between Containers::Optional and C++17 std::optional
- Corrade/Containers/PointerStl.h enables conversion between Containers::Pointer and std::unique_ptr
- Corrade/Utility/DebugStl.h is now an opt-in header to make Utility::Debug work with std::string or std::tuple
- Corrade/Utility/FormatStl.h is now an opt-in header to make Utility::format() work with std::string
- Corrade/Utility/StlMath.h
is
#include <cmath>
but without dragging in all C++17 additions that make your code compile twice as slow - Corrade/Utility/StlForwardArray.h, StlForwardString.h, StlForwardTuple.h and StlForwardVector.h provide forward declarations for std::array, std::vector, std::tuple and std::vector where STL implementations make it possible (and including the full definitions where not)
New power-efficient application implementation for Emscripten
While all Magnum applications historically defaulted to redrawing and making the CPU busy only when needed in order to save power, this was not really the case on the web. Contributed by @Squareys, there’s a new Platform::EmscriptenApplication that aims to be more efficient and smaller to download. Besides that, the Emscripten SDL “emulation” has a lot of limitations and having an implementation based directly off the HTML5 APIs allows us to be more flexible.
The new implementation was also used for an experiment in how far can Magnum potentially get with executable size optimization. A few of those optimizations already made it to 2019.10 and lots more is in the buffer for next releases — subscribe to mosra/magnum#293 for updates.
Python bindings and Eigen interoperability
By far the largest part of this release are the new Python bindings, made using pybind11 and available through a separate repository at https://github.com/mosra/magnum-bindings. To get a first impressions, check out how the basic C++ tutorials look like when rewritten in Python. Large effort went into making the Python API feel like Python, including GLSL-like vector swizzles:
>>> from magnum import * >>> a = Vector4(1.5, 0.3, -1.0, 1.0) >>> b = Vector4(7.2, 2.3, 1.1, 0.0) >>> a.wxy = b.xwz >>> a Vector(0, 1.1, -1, 7.2)
The bindings are optimized for zero-copy data transfer between C++ and Python using CPython’s Buffer Protocol, which at the very core means the Magnum array view classes got exposed as containers.ArrayView, containers.StridedArrayView1D etc., with support for the full Python slicing syntax and interoperability with numpy.ndarray, memoryview and other views and containers.
While NumPy is used extensively by researchers in the Python world, same could be said about Eigen in the C++ world. In 2019.10, EigenIntegration joins the ranks of GlmIntegration in bringing builtin conversion between foreign and Magnum math types. Goal for both is not needing to worry about whether matrix is row- or column-major or which order quaternion components are stored in:
#include <Magnum/EigenIntegration/Integration.h> Eigen::Vector3f a{1.0f, 2.0f, 3.0f}; Vector3 b(a); auto c = Matrix4::rotation(Vector3(a), 35.0_degf);
Image API improvements
With MutableImageView2D and friends and new overloads to GL::AbstractFramebuffer::read(), GL::Texture::image() etc. it’s now possible to read GPU images into existing memory, without unwanted large memory allocations happening in the background. These new APIs are also exposed to Python, allowing for efficient transfer of rendered images directly into a memory buffer managed by a machine learning framework, for example.
Back in 2018.04, Magnum gained backend-independent pixel formats, however the CompressedPixelFormat enum was quite neglected until now, supporting just basic S3TC. Now it supports all widely-used compression formats — sRGB S3TC variants, one/two-channel BC4 and BC5 formats, BC6h and BC7, ETC2 and EAC formats for mobile platforms, ASTC (including 3D and HDR) and PVRTC. On the GL side, GL::CompressedPixelFormat learned PVRTC formats as well, exposed the (3D) ASTC formats for WebGL, and same was done for the Vk::vkFormat() conversion utility. Besides GL and Vulkan, the PixelFormat / CompressedPixelFormat enum documentation now lists also corresponding D3D and Metal values to make it easier for people using (or coming from) these backends.
These improvements are the initial batch of new features being added, with more following next — improved DDS support (see mosra/magnum-plugins#67), a KTX2 importer or, for example, mip level selection (mosra/magnum#369).
Basis Universal texture compression
The main reason why all the above-listed compression formats were added is Basis Universal. It’s a successor to Crunch, open-sourced a few months ago thanks to funding from Google. What makes it so revolutional is best explained by the following plot. I took the cover.jpg used on top of this article and converted it to cover.basis and a bunch of raw block compression formats for comparison:
Before Basis, you had basically two ways how to optimize your asset size:
- Either optimize storage size by using lossy compression (such as JPEG), but then having to fully uncompress to RGBA8. With the 1536×864 cover image it’s a ~200 kB image inflated to over 5 MB of RGBA data.
- Or optimize GPU memory usage by using various block compression formats (such as BC3 / DXT5), which is only 1.3 MB of data in memory; and with a lossless compression on top you’ll get down to a 270 kB file. However, especially on mobile devices, each GPU vendor supports a different format so you need to ship at least a BCn, ETC and PVRTC variant.
With Basis Universal, you get the best of both worlds — data is internally stored in a subset of the ETC1 block compression format with additional compression on top, making it smaller than an equivalent JPEG, and then you can transcode that single file to BCn, ETC2, ASTC or PVRTC depending on what the GPU needs.
Thanks to work done by @Squareys, Magnum supports both importing (and
transcoding to a desired GPU format) via the BasisImporter
plugin as well as encoding images into the Basis Universal format using the
BasisImageConverter. Compared to the
official basisu
tool, which works only with PNGs, the
magnum-imageconverter utility supports any
format that Magnum can import:
magnum-imageconverter image.jpg image.basis
Of course, all options supported by basisu
are exposed to the plugin
configuration as well:
magnum-imageconverter image.jpg --converter BasisImageConverter \ -c flip_y=false,threads=8 image.basis
The TinyGltfImporter supports Basis files
through the unofficial GOOGLE_texture_basis
extension. There are still some
features we’re waiting on to get merged in order to have a full support. One of
them is an ability to Y-flip images during transcode (instead of only in the
encoder, BinomialLLC/basis_universal#79), another are buildsystem
improvements (BinomialLLC/basis_universal#13) — right now, the software
can’t be built as a library on its own and thus is impossible to package /
distribute without requiring each project to bundle it. Until that’s resolved,
Basis won’t be enabled in any Magnum packages. The only exception is Vcpkg,
where a Basis fork, based off the above PR, is used.
Magnum Player improvements
The Magnum Player utility received quite a few new features. It can now automatically generate smooth normals for models that don’t have them and you can inspect mesh topology by selecting it using a right-click.
Apart from meshes, the player can now also open images of all types that Magnum can import. This includes the above-mentioned Basis Universal — and the web version knows those, too, and transcodes to BCn, ETC, PVRTC, ASTC or plain RGBA depending on what your browser supports.
DART integration and an example
The DartIntegration library, integrating the DART Animation and Robotics Toolkit, contributed by @costashatz over a year ago, now received a well-polished interactive example. As a side dish, Costas wrote a detailed overview post, explaining both the code and the robotics background:
Buildsystem usability improvements
The 2019.10 release irons out the remaining pain points in using Magnum libraries as CMake subprojects. All binaries are now put into a common directory inside the build dir, which means no hassle with DLL paths on Windows anymore — and to help the common use cases even further, SDL and GLFW DLLs are automatically copied there as well.
Plugin usage with CMake subprojects is significantly improved too. Dynamic plugin binaries are put in a central place in the build directory and the plugin managers now look for them relatively to location of given plugin interface library, removing the need to install everything first.
set(WITH_TINYGLTFIMPORTER ON) set(WITH_STBIMAGEIMPORTER ON) add_subdirectory(magnum-plugins)
Note that the above bumped the minimal CMake version requirement from 3.1 to 3.4, although we don’t expect any issues as the versions currently in widespread use is 3.5. In any case, you can always download a prebuilt version for your platform.
Thanks to extensive feedback from @alanjfs, the Getting Started Guide
got rewritten and is now easier to follow by first-time users on Windows, not
requiring anybody to fiddle with %PATH%
or installing things to random
places anymore.
Windows-specific goodies
Compared to 2019.01, there’s an official support for MSVC 2019. The compiler still needs a few workarounds compared to GCC / Clang, but it’s relatively minor things that should not affect usability. Extrapolating further, we expect the next version of MSVC to be fully conforming, and thus not needing any compiler-specific handling. We’re commited to fully supporting all previous versions back to MSVC 2015 for the foreseeable future.
Corrade::Main is a new library that, on Windows, adds a shim
around your main()
function, sets up UTF-8 terminal encoding, enables
ANSI color escape codes and converts Unicode command-line arguments to UTF-8 as
well, enabling you to use the same standards-conforming code on all platforms.
Additionally, it’ll also allow you to hide the terminal window lurking in
background without forcing you to implement WinMain()
. With CMake, this
is all you need to do:
find_package(Corrade REQUIRED Main) add_executable(my-application WIN32 main.cpp) # WIN32 turns it into a GUI app target_link_libraries(my-application PRIVATE Corrade::Main)
Another thing worth mentioning is that Platform::Sdl2Application and Platform::GlfwApplication now have basic DPI awareness on Windows, catching up with other platforms.
Single-header libraries
Starting with this release, a subset of Magnum functionality is being exposed through single-header libraries over at https://github.com/mosra/magnum-singles. These are all generated from multi-file sources and thus contain the best of both worlds — small footprint of the generated files as all documentation, comments and non-essential features are stripped out, but on the other hand they inherit extensive documentation and >95% test coverage of the original source code.
The libraries were gradually introduced in the past posts, here’s the whole list:
- 1.
- ^ lines of code after a preprocessor run, with system includes expanded. Gathered using GCC 9.2 and libstdc++, unless said otherwise.
- 2.
- ^ a b not a total size due to inter-library dependencies
- 3.
- ^ a b gathered using Clang 9.0 and libc++, since libstdc++ doesn’t have a forward declaration for std::array / std::vector
- 4.
- ^ gathered using GCC 9.2, libstdc++ and
-std=c++17
TestSuite improvements, shader testing
If you’re not yet using TestSuite for tests in your Magnum-based project (well, or any other), consider giving it a try. For this release, continued effort was put on render output testing — DebugTools::CompareImage received an ability to save a diagnostic file in case of a comparison failure, and can compare against an arbitrary pixel view in addition to files and ImageView instances. Tests can be now also run with verbose output, showing detailed info even in case the comparison passes:
With these improvements in place, the whole Shaders library has tests for rendering output. So far, thanks to these, we ironed out a bunch of bugs in dusty corner cases, but that’s not all — it makes further modifications, optimizations and improvements easier to make as regressions will now be caught through automatic testing.
Reduced overhead, guaranteed thread safety and uniqueness of globals
While globals are often a source of immense pain, sometimes having a state global is the most pragmatic decision of all. Magnum currently uses globals in these few places:
- Utility::Debug scoped output redirection and coloring
- Each compiled-in Utility::Resource resource registers itself into a global storage
- Similarly, static plugins register themselves into PluginManager
- And because OpenGL (and then OpenAL, which is modelled after it) has a global context, it makes sense to have current GL::Context / Audio::Context globally accessible as well
One other usage of globals was in ResourceManager (and transitively in DebugTools::ObjectRenderer as well), but those APIs are now deprecated in favor of explicitly passed references. And, for the upcoming Vulkan backend, there’s no plan to have a GL-like “global context” at all.
For this release, all global state was rewritten to be completely allocation-free (registration of resources and static plugins is now just building an intrusive linked list), which means there’s no need to run any global destructors for these. Moreover, while already very lightweight, the automatic registration can be completely opted out of, allowing you to get rid of global constructors as well.
All global state that’s read-write is now made thread_local
, meaning
every thread will have its own copy of the global data. This makes more sense
than having the global state access guarded by a lock. Besides being faster,
you might want to redirect your log output to a file in one thread but not in
the other. Apart from these, Magnum doesn’t do anything about threading on its
own — if your app needs to share data across threads, you’re fully
responsible for guarding against data races. Thread-local variables of course
come with some small overhead, and if you don’t need that, you can turn it off
via the CORRADE_BUILD_MULTITHREADED option.
With the introduction of Python bindings, globals posed another problem — if
Magnum is built statically and then linked into two distinct Python modules,
the globals get duplicated, each module having its own copy. On Unix systems
this was easily solved by marking the few globals exported weak symbols,
telling the dynamic linker to always pick only one of them. On Windows there’s
no notion of a weak linking and additionally __declspec(dllexport)
attributes can’t be thread_local
, so this got solved by a
brown magic involving
GetProcAddress().
Full changelog … and what’s next?
This release took almost 9 months to make, much more than initially planned, and a “release cut” had to be made in order to keep it from growing indefinitely. Because of that, there’s a lot of things that didn’t fit into this announcement and the changelogs are larger than you might expect:
- Changes in Corrade 2019.10
- Changes in Magnum 2019.10
- Changes in Magnum Plugins 2019.10
- Changes in Magnum Integration 2019.10
- Changes in Magnum Extras 2019.10
- Changes in Magnum Examples 2019.10
For the next version, apart from image-related improvements hinted above, well underway is a rework of Trade::MeshData3D, with support for more vertex attributes, arbitrary data types and zero-copy data import. This one will likely result also in additions to Corrade container types (growable arrays) and various other things. Subscribe to mosra/magnum#371 for updates.
Having Python bindings out of the way, the Vulkan bindings got a priority as well — expect Vulkan-related changes popping up in the next months.
Updating from previous versions
If you’re using Homebrew, MSYS packages ArchLinux AUR or Vcpkg, 2019.10 is already in the repositories. ArchLinux community packages are scheduled for an update, and Ubuntu packages can be built directly from within the cloned repository as usual.
The library is constantly undergoing a “header hygiene” include cleanup, meaning you can now get compiler errors related to use of incomplete types. The fix is in most cases including corresponding headers — in many cases some of these:
#include <Corrade/Containers/Reference.h> #include <Corrade/Containers/Optional.h> #include <Corrade/Utility/DebugStl.h> #include <Magnum/Math/Matrix4.h>
Since it’s been over a year since the “GL split” in 2018.04, 2019.10 removes all compatibility aliases of GL APIs in the root namespace. If you’re upgrading from older versions, the recommended way is as always jumping over all stable releases (so 2018.04, 2018.10, 2019.01) and fixing up what breaks, instead of directly trying with the latest.
Thank you
A huge part of the work for this release was done by external contributors — sincere thanks to everyone (and apologies to those I forgot):
- Alan Jefferson for extensive usability feedback on tutorials and documentation
- Allie for Emscripten-related usability improvements
- Cameron Egbert for work on the Windows port of the new Python bindings
- Daniel Bloor for setting old code on fire
- Daniel Guzman for ResourceManager improvements
- Florian Goujeon for iOS fixes in the Shaders library
- Guillaume Jacquemin for various Platform::GlfwApplication / Platform::Sdl2Application improvements, Audio::Buffer queuing and loop point support, feature parity between Audio::Context and GL::Context, the DrMp3AudioImporter plugin, ImGuiIntegration improvements, MSYS package maintenance and much more
- Igor Kalevatykh for improvements to AssimpImporter
- Jakob Weiss for initial work on making the Interconnect library accept stateful lambdas
- Jonathan Hale for Platform::EmscriptenApplication, BasisImporter and BasisImageConverter, Vcpkg package maintenance and more
- Konstantinos Chatzilygeroudis for continued maintenance of DartIntegration and the DART example
- Marco Melorio for ImGuiIntegration improvements and help with macOS/iOS testing
- Max Schwarz for all work done on improving asset management, multithreading and general stability
- Nick Skelsey for documentation copy-editing
- Winfried Baumann for example code cleanup
- Cong Xie, Erik Wijmans, Ivan Sanz Carasa, Mandeep Singh Baines, Michael Tao, @pkubaj, @Selot, Thibault Jochem and many others who contributed various fixes to make things work better on a broader range of platforms