Forward-declaring STL container types

Some time ago, when look­ing in­to dif­fer­ent ways how to re­duce STL head­er bloat, I dis­cov­ered that libc++’s <iosfwd> has a for­ward dec­la­ra­tion for std::vec­tor. Turns out, there are more of them — and not just on one par­tic­u­lar stan­dard li­brary im­ple­men­ta­tion.

Us­ing for­ward dec­la­ra­tions to avoid the need to #include the whole type def­i­ni­tion is one of the es­sen­tial tech­niques to counter the ev­er-in­creas­ing com­pile times. While it’s very com­mon to do for some li­braries (such as Qt), oth­er li­braries make it hard­er and some straight out im­pos­si­ble. If you want to learn more about for­ward dec­la­ra­tions, check out this archived post from 2013.

The C++ stan­dard ex­plic­it­ly for­bids for­ward-declar­ing types from the stan­dard li­brary on the us­er side, turn­ing it in­to an un­de­fined be­hav­ior. More­over, the STL names­pace usu­al­ly wraps an­oth­er inline names­pace, which is de­fined to some plat­form-spe­cif­ic string like std::__1 on libc++ or std::__ndk1 on An­droid, which makes it hard­er (but not im­pos­si­ble) to cor­rect­ly match the type dec­la­ra­tion on the oth­er side.

The nu­cle­ar so­lu­tion to this im­poss­bil­i­ty-to-for­ward-de­clare is to stop us­ing STL al­to­geth­er. Many per­for­mance-ori­ent­ed projects end­ed up go­ing that way, but, even though Mag­num is grad­u­al­ly pro­vid­ing al­ter­na­tives to more and more STL types, I don’t want to com­plete­ly alien­ate the users but rather give them a choice and a chance to use Mag­num with STL seam­less­ly.

A po­ten­tial so­lu­tion that isn’t an un­de­fined be­hav­ior could be ex­tend­ing the stan­dard li­brary im­ple­men­ta­tion it­self with for­ward dec­la­ra­tions for ad­di­tion­al types. For­ward dec­la­ra­tions for stream types are in the clas­sic <ios­fwd> head­er, but that’s about it for what the stan­dard guar­an­tees. And adding for­ward dec­la­ra­tions for oth­er types isn’t as straight­for­ward as it may seem.

The prob­lem with tem­plates

Most con­tain­er types in the STL take op­tion­al tem­plate pa­ram­e­ters for an al­lo­ca­tor and oth­er things. For ex­am­ple, the full set of things need­ed for defin­ing the std::string type looks like this:

// <string>

namespace std {
    template<class> class allocator;
    template<class> class char_traits;

    template<class CharT,
        class Traits = char_traits<CharT>,
        class Allocator = allocator<CharT>
    > class basic_string {
        
    };

    typedef basic_string<char> string;
    
}

As with func­tions, the stan­dard man­dates that the de­fault val­ue is spec­i­fied on­ly once — ei­ther on a (for­ward) dec­la­ra­tion or on the def­i­ni­tion. So a way to for­ward-de­clare these is putting the de­fault tem­plate pa­ram­e­ter on a for­ward dec­la­ra­tion and then have the def­i­ni­tion with­out. This is by the way the main rea­son Mag­num pro­vides for­ward dec­la­ra­tion head­ers such as Cor­rade/Con­tain­ers/Con­tain­ers.h in­stead of sug­gest­ing peo­ple to write the for­ward dec­la­ra­tions on their own (it’s al­so much eas­i­er and less er­ror-prone when the type is a long typedef chain).

For the std::string case above, it would mean that the def­i­ni­tion has to be split in­to two parts — for ex­am­ple, with <iosfwd> con­tain­ing the for­ward dec­la­ra­tion and all re­lat­ed typedefs, and <string> just the def­i­ni­tion. For the ac­tu­al def­i­ni­tion we have to in­clude the for­ward dec­la­ra­tion as well in or­der to get val­ues of the de­fault pa­ram­e­ters.

// <iosfwd>

namespace std {
    template<class> class allocator;
    template<class> class char_traits;

    template<class CharT,
        class = char_traits<CharT>,
        class = allocator<CharT>
    > class basic_string;

    typedef basic_string<char> string;
    
}
// <string>

#include <iosfwd>

namespace std {
    template<class CharT,
        class Traits,
        class Allocator
    > class basic_string {
        
    };

    
}

So, ba­si­cal­ly, if we’re able to find a for­ward dec­la­ra­tion of a STL con­tain­er type in­clud­ing the de­fault ar­gu­ments in some (even in­ter­nal) STL head­er, the head­er is her­met­ic and sig­nif­i­cant­ly small­er than the cor­re­spond­ing stan­dard head­er, we won. Con­verse­ly, if the type def­i­ni­tion con­tains the tem­plate de­fault pa­ram­e­ters, then we can be sure that no for­ward dec­la­ra­tion is pos­si­ble.

De­tect­ing what STL we’re on

Be­cause we now wan­dered in­to the im­ple­men­ta­tion-spe­cif­ic ter­ri­to­ry, we need a way to de­tect what STL fla­vor is the code be­ing com­piled on. Then, for known STL im­ple­men­ta­tions with known for­ward dec­la­ra­tion head­ers we in­clude the par­tic­u­lar head­er, and use the full type def­i­ni­tion in the stan­dard head­er oth­er­wise. That means our for­ward dec­la­ra­tion wrap­per will al­ways work, but giv­ing us com­pi­la­tion time ad­van­tages in some cas­es.

The clas­sic way to de­tect a STL ven­dor is to in­clude the <ciso646> head­er (which is de­fined to be emp­ty on C++) and then check for ei­ther _LIBCPP_VERSION (de­fined on libc++, used by Clang main­ly on mac­OS and iOS), _CPPLIB_VER (de­fined on MSVC STL, for­mer­ly Dinkumware) or __GLIBCXX__ (de­fined on GCC’s lib­st­dc++). One thing to note is that on GCC be­fore ver­sion 6.1 the <ciso646> head­er doesn’t de­fine the _LIBCPP_VERSION macro, so it’s need­ed to get it via some oth­er means. Be­gin­ning with C++20, there will be a new head­er, <ver­sion>, stan­dard­iz­ing this process.

If you use Cor­rade and in­clude any of its head­ers, you’ll get the de­tect­ed li­brary ex­posed through one of the COR­RADE_­TAR­GET_LIBCXX, COR­RADE_­TAR­GET_LIB­ST­D­CXX or COR­RADE_­TAR­GET_DINKUMWARE macros.

What you can ex­pect

The fol­low­ing ta­ble sum­ma­rizes what com­mon STL con­tain­er types can be for­ward-de­clared on which im­ple­men­ta­tion. The left col­umn shows pre­pro­cessed line count with GNU lib­st­dc++ and C++11 (un­less said oth­er­wise), gath­ered us­ing the fol­low­ing com­mand line:

echo "#include <utility>" | gcc -std=c++11 -P -E -x c++ - | wc -l

The oth­er col­umns then show how many pre­pro­cessed lines is the cor­re­spond­ing for­ward dec­la­ra­tion on a par­tic­u­lar im­ple­men­ta­tion, if ap­pli­ca­ble.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
libstdc++ 8.2 libc++ 7.0.1 MSVC STL 2017
std::pair 1
<utility>

2 197 LoC

4 265 LoC

std::initializer_list 1
<initializer_list>

53 LoC

101 LoC

std::list 2
<list>, 6 749 LoC



std::forward_list 2
<forward_list>, 7 036 LoC



std::string
<string>, 11 576 LoC
3
<bits/stringfwd.h>, 47 LoC
4
<iosfwd>, 741 LoC
5
std::string_view 6
<string_view>, 7 876 LoC



std::array
<array>, 15 003 LoC
7
8
<__tuple>, 2 455 LoC
9
<utility>
std::tuple
<tuple>, 13 444 LoC
10
<type_traits>, 1 615 LoC
8
<__tuple>, 2 455 LoC
9
<utility>
std::vector
<vector>, 8 613 LoC
11
12
<iosfwd>, 741 LoC
11
std::span
<span>, 21 696 LoC
13
14
13
std::[multi]set 15
<set>, 8 170 LoC
std::[multi]map 15
<map>, 15 796 LoC
std::unordered_[multi]set 15
<unordered_set>, 17 114 LoC
std::unordered_[multi]map 15
<unordered_map>, 17 180 LoC
1.
^ Some STL im­ple­men­ta­tions have a for­ward dec­la­ra­tion for std::pair, but you’ll need <utility> in most cas­es any­way for the std::move(), std::for­ward() and oth­er utils there’s no point in both­er­ing with a for­ward dec­la­ra­tion — which wouldn’t be much small­er than the full def­i­ni­tion any­way. Sim­i­lar­ly it goes for std::ini­tial­iz­er_list, the full def­i­ni­tion is al­so very tiny. Both these types don’t have any stan­dard de­fault tem­plate ar­gu­ment so these could be the­o­ret­i­cal­ly safe to for­ward-de­clare how­ev­er noth­ing pre­vents STL im­ple­men­ta­tions from adding their own de­fault tem­plate ar­gu­ments.
2.
^ Both std::list and std::for­ward_list have a full def­i­ni­tion in the stan­dard <list> / <forward_list> head­ers on all three im­ple­men­ta­tions. The libc++ im­ple­men­ta­tion has a for­ward dec­la­ra­tion in the same file, but it would first need to be ex­tract­ed out­side to make it use­ful.
3.
^ lib­st­dc++ has std::string, std::wstring, std::u16string and std::u32string for­ward-de­clared in <bits/stringfwd.h>.
4.
^ libc++ has std::string and std::wstring for­ward-de­clared in the stan­dard <ios­fwd>, un­for­tu­nate­ly the std::u16string and std::u32string type­defs are miss­ing.
5.
^ MSVC STL has the full def­i­ni­tion of std::ba­sic_string in­clud­ing de­fault pa­ram­e­ters in <xstring>, which makes it im­pos­si­ble to for­ward-de­clare.
6.
^ std::ba­sic_string_view has the full def­i­ni­tion di­rect­ly in the <string_view> head­er on both libc++ and lib­st­dc++, MSVC 2017 has both a for­ward dec­la­ra­tion and the full def­i­ni­tion in <xstring> — the dec­la­ra­tion could be ex­tract­ed to a sep­a­rate head­er to make this work.
7.
^ I couldn’t find any for­ward dec­la­ra­tion for std::ar­ray in lib­st­dc++. It how­ev­er doesn’t mean it hasn’t any — the type has no de­fault tem­plate pa­ram­e­ters so it should be pos­si­ble.
8.
^ libc++ has a for­ward dec­la­ra­tion for std::ar­ray and std::tu­ple in <__­tu­ple>.
9.
^ MSVC STL has a for­ward dec­la­ra­tion for std::ar­ray and std::tu­ple de­fined in the stan­dard <utility>, next to std::pair.
10.
^ lib­st­dc++ has a for­ward dec­la­ra­tion for std::tu­ple in the stan­dard <type­_­traits>.
11.
^ MSVC has the full std::vec­tor def­i­ni­tion in <vector>, lib­st­dc++ has a small-ish full def­i­ni­tion in <bits/stl_vector.h> but the head­er is not her­met­ic and when all need­ed de­pen­den­cies are in­clud­ed as well the size is not much dif­fer­ent from the stan­dard head­er.
12.
^ libc++ 3.9 and up has a for­ward dec­la­ra­tion for std::vec­tor in the stan­dard <ios­fwd> (old­er ver­sions don’t).
13.
^ Nei­ther lib­st­dc++ nor MSVC STL im­ple­ment C++20 std::span yet.
14.
^ libc++ 7.0 has a friend for­ward dec­la­ra­tion for std::span in the <it­er­a­tor> head­er but that’s not enough to have the for­ward dec­la­ra­tion avail­able glob­al­ly in the std names­pace. Too bad, be­cause the <span> head­er is heavy.
15.
^ All stan­dard (un­ordered) (mul­ti)map/set im­ple­men­ta­tions have just the full def­i­ni­tion with no pos­si­bil­i­ty to for­ward-de­clare. Since these types are very rarely used di­rect­ly as func­tion pa­ram­e­ters or re­turn types, it’s not such a big prob­lem. Be­sides that, they tend to be rather heavy both at com­pile time and at run­time due to their ex­treme gener­ic­i­ty, so the less they get used the bet­ter 😉

Con­clu­sion

While the heav­ier map / set types don’t have for­ward dec­la­ra­tions, the ex­ist­ing for­ward dec­la­ra­tions can al­ready cov­er many use cas­es for li­braries that want to be both fast-to-com­pile and STL-friend­ly:

Too bad that the std::string_view and std::span types, while meant to be light­weight at run­time, are so com­pile-time heavy and im­pos­si­ble to for­ward-de­clare.

While it’s pos­si­ble to make use of the for­ward dec­la­ra­tions al­so for func­tions tak­ing/re­turn­ing vec­tors, ar­rays and strings by con­stant ref­er­ences, a bet­ter ap­proach is to make use of STL com­pat­i­bil­i­ty in the Con­tain­ers::Ar­rayView class­es — that’ll al­low more types than just std::vec­tor, std::ar­ray or std::string to be used and the com­pile-time im­pact is clear and con­sis­tent with this type. See the ar­ti­cle about STL-com­pat­i­ble ar­ray views for more in­for­ma­tion and de­tailed per­for­mance over­view.

I al­so didn’t look for for­ward dec­la­ra­tions of the std::unique_p­tr, std::op­tion­al or std::ref­er­ence_wrap­per types be­cause they’re very sim­ple and thus easy to re­place. See the Light­weight but still STL-com­pat­i­ble unique point­er post for more in­for­ma­tion.

Try them in your code

Cor­rade pro­vides the above for­ward dec­la­ra­tions in tiny Cor­rade/Util­i­ty/Stl­For­war­dAr­ray.h, Cor­rade/Util­i­ty/Stl­For­ward­String.h, Cor­rade/Util­i­ty/Stl­For­ward­Tu­ple.h and Cor­rade/Util­i­ty/Stl­For­ward­Vec­tor.h head­ers, sim­ply in­clude them in­stead of the stan­dard <array>, <string>, <tuple> or <vector> head­ers where de­sired. The en­gine is mov­ing to use them as well, but since it will cause many build er­rors due to sud­den­ly miss­ing #includes in us­er code, the move is done grad­u­al­ly to avoid sud­den­ly break­ing ev­ery­thing.

The Mag­num Sin­gles repos­i­to­ry con­tains these as well, as a her­metic­head­ers not de­pend­ing on Cor­rade’s in­ter­nals. Take them and bun­dle right in­to your project:

Li­brary LoC Pre­pro­cessed LoC De­scrip­tion
Cor­radeStl­For­war­dAr­ray.h 67 245516 See Cor­rade/Util­i­ty/Stl­For­war­dAr­ray.h docs
Cor­radeStl­For­ward­String.h 74 48 See Cor­rade/Util­i­ty/Stl­For­ward­String.h docs
Cor­radeStl­For­ward­Tu­ple.h 78 1616 See Cor­rade/Util­i­ty/Stl­For­ward­Tu­ple.h docs
Cor­radeStl­For­ward­Vec­tor.h 62 74116 See Cor­rade/Util­i­ty/Stl­For­ward­Vec­tor.h docs
16.
^ a b gath­ered us­ing Clang 7.0 and libc++, since GCC 8.2’s lib­st­dc++ doesn’t have a for­ward dec­la­ra­tion for std::ar­ray / std::vec­tor

Can we con­vince ven­dors to do this more?

While I think it’s pos­si­ble to add ad­di­tion­al for­ward dec­la­ra­tions to some STL im­ple­men­ta­tions, it might not be al­ways pos­si­ble to do so with­out break­ing ABI com­pat­i­bil­i­ty — even in Mag­num I had to break ABI com­pat­i­bil­i­ty a few times in the past in or­der to achieve that (and now I know what to avoid to make new types eas­i­ly for­ward-de­clar­able from the start).

The ide­al way would be to have the for­ward dec­la­ra­tions guar­an­teed by the stan­dard (ex­tend­ing <iosfwd> fur­ther, for ex­am­ple) so we don’t need to in­clude plat­form-spe­cif­ic in­ter­nal “bits”, but again this may cause an ABI break for many ven­dors and thus take years to im­ple­ment (like it hap­pened with std::string in C++11 where lib­st­dc++ could no longer have it copy-on-write — and the prob­lems it caused per­sist un­til to­day).

There’s al­so a slight chance that, due to com­plex­i­ties of std::al­lo­ca­tor and oth­er types used in de­fault tem­plate ar­gu­ments, adding for­ward dec­la­ra­tions to <iosfwd> would make it no longer light­weight. This re­al­ly de­pends on how the im­ple­men­ta­tions are done and what all needs to be known to for­ward-de­clare giv­en type.