May 2015 updates to Magnum

The Mag­num C++11/C++14 and OpenGL graph­ics en­gine gained OpenGEX for­mat sup­port, a bunch of new im­porter plug­ins, cross-plat­form ex­ten­sion load­er, trans­form feed­back sup­port and new fea­tures from OpenGL ES 3.1 and OpenGL 4.5, among oth­er things.

The new re­lease comes al­most a whole year af­ter the pre­vi­ous one, main­ly due to the fact that I’m cur­rent­ly full-time em­ployed and I have less free time than I ex­pect­ed. Nev­er­the­less the project ac­cu­mu­lat­ed quite a lot of changes over the year, so this changel­og may be a bit over­whelm­ing. I plan to do one small­er re­lease in a few months to pol­ish and clean up some more things and then I’ll prob­a­bly start break­ing stuff in nev­er-be­fore-seen ways to make Vulkan sup­port a re­al­i­ty.

The May 2015 snap­shot is avail­able un­der snapshot-2015-05 tag in Cor­rade, Mag­num, Mag­num Plug­ins, Mag­num In­te­gra­tion and Mag­num Ex­am­ples GitHub repos­i­to­ries. This ar­ti­cle will de­scribe the most im­por­tant changes, for all the com­mits that went to mas­ter branch­es since the June 2014 snap­shot (tagged snapshot-2014-06) see logs in Cor­rade, Mag­num, Mag­num Plug­ins and Mag­num In­te­gra­tion, re­spec­tive­ly.

For com­pat­i­bil­i­ty branch there is tag snapshot-2015-05-compatibility in Cor­rade, Mag­num, Mag­num Plug­ins and Mag­num Ex­am­ples repos­i­to­ries.

Plug­in alias­es, smart im­porters and new for­mat sup­port

In an on­go­ing at­tempt to make the en­gine use­ful even with­out a slew of ex­ter­nal li­brary de­pen­den­cies, I added a bunch of de­pen­den­cy-less plug­ins for load­ing com­mon scene and im­age for­mats (see the changel­og be­low for a com­plete list). I had a first­hand ex­pe­ri­ence about how crazy hard it is to get work­ing Freetype, Harf­Buzz or even libP­NG bi­na­ry for Win­dows, so there are more plug­ins planned (es­pe­cial­ly for de­pen­den­cy-less font han­dling), stay tuned.

In ad­di­tion to this, the plug­in load­er now sup­ports alias­es. It means that, to load PNG im­ages, for ex­am­ple, you no longer need to wor­ry about which par­tic­u­lar plug­in might be avail­able on giv­en plat­form and just al­ways re­quest PngIm­porter plug­in:

PluginManager::Manager<Trade::AbstractImporter> manager{MAGNUM_PLUGINS_IMPORTER_DIR};
std::unique_ptr<Trade::AbstractImporter> importer = manager.loadAndInstantiate("PngImporter");
// ...

Now, if the PngImporter plug­in is not avail­able, it tries to load any oth­er plug­in that alias­es with PngImporter. Cur­rent­ly it might be for ex­am­ple StbIm­ageIm­porter, in fu­ture re­leas­es pos­si­bly al­so Sdl2ImageImporter or WebGlImageImporter.

Last­ly, there are now smart im­age and scene load­ers, named Any­Im­ageIm­porter and AnySceneIm­porter. They try to de­tect file type from file­name and then re­di­rect the op­er­a­tion to ded­i­cat­ed im­porter plug­in (so if you load image.tga, for ex­am­ple, it tries to load it with TgaImporter, if you load texture.png, it tries to load it with PngImporter etc.). In com­bi­na­tion with plug­in alias­es this makes a fair­ly pow­er­ful im­porter frame­work.

Ini­tial OpenD­DL/OpenGEX sup­port

Pos­si­bly the big­gest new im­porter plug­in is the OpenGEX for­mat im­porter. If you haven’t heard about it yet, it is a fair­ly new 3D scene file for­mat based on the clean­ly de­signed OpenD­DL lan­guage. It aims to be a bet­ter al­ter­na­tive to the bloat­ed XML-based COL­LA­DA for­mat and cur­rent­ly there are ex­porters from 3Ds Max, Maya and Blender.

Cur­rent OpenGEX im­porter im­ple­men­ta­tion in Mag­num cov­ers the same fea­ture set as the Col­ladaIm­porter plug­in, but with faster and more ro­bust im­ple­men­ta­tion and hope­ful­ly with less bugs and un­cov­ered cor­ner cas­es. The fu­ture of the COL­LA­DA im­porter is un­cer­tain and it will prob­a­bly be dep­re­cat­ed in fa­vor of the new OpenGEX im­porter, be­cause COL­LA­DA sup­port in 3D mod­el­ling soft­ware is far from ide­al and the XML pars­er is hell to im­ple­ment.

The un­der­ly­ing OpenD­DL pars­er that pow­ers the plug­in is cur­rent­ly on­ly a pri­vate part of the plug­in, but if it is seen to be use­ful out­side of the plug­in, I may move it in­to a pub­licly us­able li­brary. Here’s an ex­am­ple how OpenD­DL file pars­ing can look like:

OpenDdl::Document d;

/* Parse the document */
if(!d.parse(data, OpenGex::structures, OpenGex::properties)) {
    // ...
}

/* Validate its structure */
if(!d.validate(OpenGex::rootStructures, OpenGex::structureInfo)) {
    // ...
}

/* Parse all meshes */
for(OpenDdl::Structure geometryObject: d.childrenOf(OpenGex::GeometryObject)) {
    /* Decide about primitive */
    if(std::optional<OpenDdl::Property> primitive = geometryObject.findPropertyOf(OpenGex::primitive)) {
        auto&& str = primitive->as<std::string>();
        if(str == "triangles") {
            // ...
        } else if(str == "lines") {
            // ...
        } // ...
    } else {
        // default primitive ...
    }

    /* Parse vertex array */
    OpenDdl::Structure vertexArray = geometryObject.firstChildOf(OpenGex::VertexArray);
    auto&& attrib = vertexArray.propertyOf(OpenGex::attrib).as<std::string>();
    if(attrib == "position") {
        // ...
    } else if(attrib == "normal") {
        // ...
    }

    /* Parse vertex array data */
    Containers::ArrayReference<const Float> vertexData = vertexArray.firstChild().asArray<Float>();
    // ...
}

The View­er ex­am­ple was al­so up­dat­ed to sup­port load­ing OpenGEX files and it is now al­so avail­able on­line through Em­scripten.

Viewer example

New ex­ten­sion load­er, ex­ten­sion load­ing for OpenGL ES

I was be­ing more and more dis­sat­is­fied with the state of glLoadGen (which was orig­i­nal­ly meant to re­place the even worse GLEW) and switched to flextGL. If you don’t know it yet, it is very flex­i­ble and cus­tom­iz­a­ble ex­ten­sion load­er gen­er­a­tor and con­sists of just a sin­gle very clean Python file. Switch­ing to it re­sult­ed in much short­er gen­er­at­ed code (which means faster build times) and af­ter a few mi­nor changes I man­aged to add OpenGL ES sup­port to it. So it is now fi­nal­ly pos­si­ble to use ex­ten­sions on both desk­top GL and OpenGL ES. The on­ly plat­forms which don’t have any no­tion of tra­di­tion­al ex­ten­sion load­ing are We­bGL and Na­Cl, on which the ex­ten­sion han­dling is not yet ful­ly im­ple­ment­ed.

OpenGL 4.5, OpenGL ES 3.1 sup­port

In Au­gust 2014, OpenGL 4.5 was re­leased, which was un­der­stood as some sort of a cleanup re­lease with no rad­i­cal­ly new func­tion­al­i­ty. The most es­sen­tial ad­di­tion is AR­B_­di­rec­t_s­tate_ac­cess, which fi­nal­ly cleans up some rough cor­ners of the old EX­T_­di­rec­t_s­tate_ac­cess ex­ten­sion and puts it in the core spec­i­fi­ca­tion. Mag­num cur­rent­ly im­ple­ments both ex­ten­sions and prefers the ARB one. Some API changes were need­ed to ac­co­mo­date to the new de­sign, main­ly in con­struc­tion of query ob­jects and cube map tex­ture da­ta up­loads. The oth­er mi­nor fea­tures in­clude abil­i­ty to query on­ly a part of tex­ture im­age, in­vert­ed con­di­tion­al ren­der­ing etc., see be­low for com­plete list.

Thanks to the new ex­ten­sion load­er it was al­so fi­nal­ly pos­si­ble to im­ple­ment OpenGL ES 3.1 sup­port. Un­like the ES 2.0/3.0 switch, there is no com­pile-time op­tion to en­able OpenGL ES 3.1, the li­brary is com­piled for both 3.0 and 3.1 and the new fea­tures are en­abled based on ver­sion re­port­ed by the driv­er.

IRC chan­nel

In ad­di­tion to all oth­er com­mu­ni­ca­tion chan­nels there is now a IRC chan­nel: join the dis­cus­sion at #magnum-engine on Freen­ode.

Cor­rade

New features

Changes

  • Test­Suite::Com­pare::Con­tain­er is now able to com­pare non-copy­able con­tain­ers such as Con­tain­ers::Ar­rayRef­er­ence (see mosra/cor­rade#9).
  • Us­ing const char in­stead of const unsigned char for raw bi­na­ry da­ta.
  • Bet­ter al­go­rithm for com­par­ing float­ing-point val­ues in Test­Suite.

Build system

  • CMake now al­ways in­stalls FindCorrade.cmake to li­brary-spe­cif­ic lo­ca­tion, mak­ing it us­able with­out pro­vid­ing own copy of the file in de­pend­ing projects. The WITH_FIND_MODULE op­tion is no longer need­ed.
  • Dis­play­ing all head­er files, plug­in meta­da­ta files and re­source files in project view to make use of some IDEs less painful (such as QtCre­ator).
  • Gen­too ebuild (see mosra/cor­rade#16).

Bug fixes

  • Re­moved stat­ic ini­tial­iz­ers to avoid mem­o­ry cor­rup­tion in stat­ic builds (see mosra/mag­num#90).
  • Plug­in man­ag­er now cor­rect­ly folows de­pen­den­cy or­der when un­load­ing plug­ins.

Deprecated APIs

No API was dep­re­cat­ed in this re­lease.

Removed APIs and features

  • Re­moved un­used plug­in re­place­ment fea­ture, as it had ques­tion­able ben­e­fits and no re­al use.
  • All func­tion­al­i­ty dep­re­cat­ed in Jan­u­ary 2014 snap­shot has been re­moved, name­ly:
    • Re­moved dep­re­cat­ed abil­i­ty to use rel­a­tive in­cludes (e.g. #include <Utility/Debug.h>), use ab­so­lute paths ( #include <Corrade/Utility/Debug.h>) in­stead.
    • Re­moved dep­re­cat­ed Utility::String::split() over­load, use ei­ther split() or splitWith­out­Emp­ty­Parts() in­stead.

Mag­num

New features

Shader samples

Changes

Build system

  • CMake now al­ways in­stalls FindMagnum.cmake to li­brary-spe­cif­ic lo­ca­tion, mak­ing it us­able with­out pro­vid­ing own copy of the file in de­pend­ing projects. The WITH_FIND_MODULE op­tion is no longer need­ed.
  • CMake now han­dles in­ter-li­brary de­pen­den­cies au­to­mat­i­cal­ly with­out man­u­al­ly spec­i­fy­ing each and ev­ery one (see mosra/mag­num#73).
  • Abil­i­ty to con­trol stat­ic build of plug­ins sep­a­rate­ly us­ing BUILD_PLUGINS_STATIC.
  • Dis­play­ing all head­er files, plug­in meta­da­ta files and re­source files in project view to make use of some IDEs less painful (such as QtCre­ator).
  • Gen­too ebuild (see mosra/mag­num#69).

Bug fixes

Deprecated APIs

Removed APIs and functionality

  • All func­tion­al­i­ty dep­re­cat­ed in Jan­u­ary 2014 snap­shot has been re­moved, name­ly:
    • Re­moved dep­re­cat­ed abil­i­ty to use rel­a­tive in­cludes (e.g. #include <Mesh.h>), use ab­so­lute paths (#include <Magnum/Mesh.h>) in­stead.
    • Re­moved dep­re­cat­ed Mesh::Primitive enum, use Mesh­Prim­i­tive in­stead.
    • Re­moved dep­re­cat­ed Sampler::maxAnisotropy() func­tion, use Sam­pler::max­Max­An­isot­ropy() in­stead.
    • Re­moved dep­re­cat­ed Math::Geometry::Rectangle class, use Math::Range in­stead.
    • Re­moved dep­re­cat­ed SceneGraph::Animable::group() func­tion, use Scene­Graph::An­i­mable::an­i­mables() in­stead.
    • Re­moved dep­re­cat­ed Shaders/magnumShadersResourceImport.hpp file, use Shaders/resourceImport.hpp in­stead.
    • Re­moved dep­re­cat­ed Text::TextRenderer alias, use Text::Ren­der­er in­stead.
    • Re­moved long-dep­re­cat­ed Buf­fer­Im­age::set­Da­ta() over­load, use the oth­er one in­stead.

Performance improvements

  • Faster vec­tor mul­ti­pli­ca­tion in Math::Quater­nion.
  • Saved one swiz­zle in Math::cross().
  • Al­lo­ca­tion-free over­loads of func­tions that took std::string just to pass its con­tents to OpenGL.

Mag­num Plug­ins

New features

Removed APIs and functionality

  • All func­tion­al­i­ty dep­re­cat­ed in Jan­u­ary 2014 snap­shot has been re­moved, name­ly re­moved dep­re­cat­ed abil­i­ty to use rel­a­tive in­cludes (e.g. #include <PngImporter/PngImporter.h>), use ab­so­lute paths (#include <MagnumPlugins/PngImporter/PngImporter.h>) in­stead.

Build system

  • CMake now al­ways in­stalls FindMagnumPlugins.cmake to li­brary-spe­cif­ic lo­ca­tion, mak­ing it us­able with­out pro­vid­ing own copy of the file in de­pend­ing projects. The WITH_FIND_MODULE op­tion is no longer need­ed.
  • Dis­play­ing all head­er files, plug­in meta­da­ta files and re­source files in project view to make use of some IDEs less painful (such as QtCre­ator).
  • Gen­too ebuild (see mosra/mag­num-plug­ins#5).

Magnum Integration

Build system

  • CMake now al­ways in­stalls FindMagnumIntegration.cmake to li­brary-spe­cif­ic lo­ca­tion, mak­ing it us­able with­out pro­vid­ing own copy of the file in de­pend­ing projects. The WITH_FIND_MODULE op­tion is no longer need­ed.
  • Dis­play­ing all head­er files, plug­in meta­da­ta files and re­source files in project view to make use of some IDEs less painful (such as QtCre­ator).
  • Gen­too ebuild (see mosra/mag­num-in­te­gra­tion#2).

Removed APIs and functionality

All func­tion­al­i­ty dep­re­cat­ed in Jan­u­ary 2014 snap­shot has been re­moved, name­ly re­moved dep­re­cat­ed abil­i­ty to use rel­a­tive in­cludes (e.g. #include <BulletIntegration/Integration.h>), use ab­so­lute paths (#include <Magnum/BulletIntegration/Integration.h>) in­stead.

Deprecated APIs

No API was dep­re­cat­ed in this re­lease.

Removed APIs

No dep­re­cat­ed API was re­moved in this re­lease.

Mag­num Ex­am­ples

Changes

  • Re­worked and sim­pli­fied the View­er ex­am­ple, up­dat­ed it to work with vir­tu­al­ly any file for­mat.

Bug fixes

Build system

  • All ex­am­ples are now in­stal­lable in a way sim­i­lar to Mag­num it­self.
  • Dis­play­ing all head­er files, plug­in meta­da­ta files and re­source files in project view to make use of some IDEs less painful (such as QtCre­ator).
  • Added Arch­Lin­ux, De­bian and Gen­too pack­ages (see mosra/mag­num-ex­am­ples#8).

Magnum Bootstrap

No us­er-vis­i­ble changes in this re­lease, ex­cept for up­dates of CMake mod­ules.