Mar 28 2019 Forward-declaring STL container types

The prob­lem with tem­plates

Most con­tain­er types in the STL take op­tion­al tem­plate para­met­ers for an al­loc­at­or and oth­er things. For ex­ample, the full set of things needed for de­fin­ing the std::string type looks like this:

// <string>

namespace std {
    template<class> class allocator;
    template<class> class char_traits;

    template<class CharT,
        class Traits = char_traits<CharT>,
        class Allocator = allocator<CharT>
    > class basic_string {
        …
    };

    typedef basic_string<char> string;
    …
}

As with func­tions, the stand­ard man­dates that the de­fault value is spe­cified only once — either on a (for­ward) de­clar­a­tion or on the defin­i­tion. So a way to for­ward-de­clare these is put­ting the de­fault tem­plate para­met­er on a for­ward de­clar­a­tion and then have the defin­i­tion without. This is by the way the main reas­on Mag­num provides for­ward de­clar­a­tion head­ers such as Cor­rade/Con­tain­ers/Con­tain­ers.h in­stead of sug­gest­ing people to write the for­ward de­clar­a­tions on their own (it’s also much easi­er and less er­ror-prone when the type is a long typedef chain).

For the std::string case above, it would mean that the defin­i­tion has to be split in­to two parts — for ex­ample, with <iosfwd> con­tain­ing the for­ward de­clar­a­tion and all re­lated typedefs, and <string> just the defin­i­tion. For the ac­tu­al defin­i­tion we have to in­clude the for­ward de­clar­a­tion as well in or­der to get val­ues of the de­fault para­met­ers.

// <iosfwd>

namespace std {
    template<class> class allocator;
    template<class> class char_traits;

    template<class CharT,
        class = char_traits<CharT>,
        class = allocator<CharT>
    > class basic_string;

    typedef basic_string<char> string;
    …
}

// <string>

#include <iosfwd>

namespace std {
    template<class CharT,
        class Traits,
        class Allocator
    > class basic_string {
        …
    };

    …
}

So, ba­sic­ally, if we’re able to find a for­ward de­clar­a­tion of a STL con­tain­er type in­clud­ing the de­fault ar­gu­ments in some (even in­tern­al) STL head­er, the head­er is her­met­ic and sig­ni­fic­antly smal­ler than the cor­res­pond­ing stand­ard head­er, we won. Con­versely, if the type defin­i­tion con­tains the tem­plate de­fault para­met­ers, then we can be sure that no for­ward de­clar­a­tion is pos­sible.

De­tect­ing what STL we’re on

Be­cause we now wandered in­to the im­ple­ment­a­tion-spe­cif­ic ter­rit­ory, we need a way to de­tect what STL fla­vor is the code be­ing com­piled on. Then, for known STL im­ple­ment­a­tions with known for­ward de­clar­a­tion head­ers we in­clude the par­tic­u­lar head­er, and use the full type defin­i­tion in the stand­ard head­er oth­er­wise. That means our for­ward de­clar­a­tion wrap­per will al­ways work, but giv­ing us com­pil­a­tion time ad­vant­ages in some cases.

The clas­sic way to de­tect a STL vendor is to in­clude the <ciso646> head­er (which is defined to be empty on C++) and then check for either _LIBCPP_VERSION (defined on libc++, used by Clang mainly on ma­cOS and iOS), _CPPLIB_VER (defined on MS­VC STL, formerly Dinkum­ware) or __GLIBCXX__ (defined on GCC’s lib­stdc++). One thing to note is that on GCC be­fore ver­sion 6.1 the <ciso646> head­er doesn’t define the _LIBCPP_VERSION macro, so it’s needed to get it via some oth­er means. Be­gin­ning with C++20, there will be a new head­er, <ver­sion>, stand­ard­iz­ing this pro­cess.

If you use Cor­rade and in­clude any of its head­ers, you’ll get the de­tec­ted lib­rary ex­posed through one of the COR­RADE_TAR­GET_LIB­CXX, COR­RADE_TAR­GET_LIB­STDCXX or COR­RADE_TAR­GET_DINKUM­WARE mac­ros.

What you can ex­pect

The fol­low­ing table sum­mar­izes what com­mon STL con­tain­er types can be for­ward-de­clared on which im­ple­ment­a­tion. The left column shows pre­pro­cessed line count with GNU lib­stdc++ and C++11 (un­less said oth­er­wise), gathered us­ing the fol­low­ing com­mand line:

echo "#include <utility>" | gcc -std=c++11 -P -E -x c++ - | wc -l

The oth­er columns then show how many pre­pro­cessed lines is the cor­res­pond­ing for­ward de­clar­a­tion on a par­tic­u­lar im­ple­ment­a­tion, if ap­plic­able.

1.

^ Some STL im­ple­ment­a­tions have a for­ward de­clar­a­tion for std::pair, but you’ll need <utility> in most cases any­way for the std::move(), std::for­ward() and oth­er utils there’s no point in both­er­ing with a for­ward de­clar­a­tion — which wouldn’t be much smal­ler than the full defin­i­tion any­way. Sim­il­arly it goes for std::ini­tial­izer­_l­ist, the full defin­i­tion is also very tiny. Both these types don’t have any stand­ard de­fault tem­plate ar­gu­ment so these could be the­or­et­ic­ally safe to for­ward-de­clare how­ever noth­ing pre­vents STL im­ple­ment­a­tions from adding their own de­fault tem­plate ar­gu­ments.

2.

^ Both std::list and std::for­ward_l­ist have a full defin­i­tion in the stand­ard <list> / <forward_list> head­ers on all three im­ple­ment­a­tions. The libc++ im­ple­ment­a­tion has a for­ward de­clar­a­tion in the same file, but it would first need to be ex­trac­ted out­side to make it use­ful.

3.

^ lib­stdc++ has std::string, std::wstring, std::u16string and std::u32string for­ward-de­clared in <bits/string­fwd.h>.

4.

^ libc++ has std::string and std::wstring for­ward-de­clared in the stand­ard <ios­fwd>, un­for­tu­nately the std::u16string and std::u32string ty­pedefs are miss­ing.

5.

^ MS­VC STL has the full defin­i­tion of std::ba­sic_string in­clud­ing de­fault para­met­ers in <xstring>, which makes it im­possible to for­ward-de­clare.

6.

^ std::ba­sic_string_view has the full defin­i­tion dir­ectly in the <string_view> head­er on both libc++ and lib­stdc++, MS­VC 2017 has both a for­ward de­clar­a­tion and the full defin­i­tion in <xstring> — the de­clar­a­tion could be ex­trac­ted to a sep­ar­ate head­er to make this work.

7.

^ I couldn’t find any for­ward de­clar­a­tion for std::ar­ray in lib­stdc++. It how­ever doesn’t mean it hasn’t any — the type has no de­fault tem­plate para­met­ers so it should be pos­sible.

8.

^ libc++ has a for­ward de­clar­a­tion for std::ar­ray and std::tuple in <__tuple>.

9.

^ MS­VC STL has a for­ward de­clar­a­tion for std::ar­ray and std::tuple defined in the stand­ard <utility>, next to std::pair.

10.

^ lib­stdc++ from ver­sion 7 to 11 has a for­ward de­clar­a­tion for std::tuple in the stand­ard <type_traits> head­er, but it got re­moved in GCC 12. The oth­er for­ward de­clar­a­tion, ad­ded in 4.6 and avail­able also in GCC 12, is in <bits/stl_pair.h>. To avoid is­sues, in­clude the whole <utility> — it isn’t that much lar­ger than <type_traits>.

11.

^ MS­VC has the full std::vec­tor defin­i­tion in <vector>, lib­stdc++ has a small-ish full defin­i­tion in <bits/stl_vector.h> but the head­er is not her­met­ic and when all needed de­pend­en­cies are in­cluded as well the size is not much dif­fer­ent from the stand­ard head­er.

12.

^ libc++ 3.9 and up has a for­ward de­clar­a­tion for std::vec­tor in the stand­ard <ios­fwd> (older ver­sions don’t).

13.

^ libc++ 7.0 has a friend for­ward de­clar­a­tion for std::span in the <iter­at­or> head­er but that’s not enough to have the for­ward de­clar­a­tion avail­able glob­ally in the std namespace. Too bad, be­cause the <span> head­er is heavy.

14.

^ All stand­ard (un­ordered) (multi)map/set im­ple­ment­a­tions have just the full defin­i­tion with no pos­sib­il­ity to for­ward-de­clare. Since these types are very rarely used dir­ectly as func­tion para­met­ers or re­turn types, it’s not such a big prob­lem. Be­sides that, they tend to be rather heavy both at com­pile time and at runtime due to their ex­treme gen­er­i­city, so the less they get used the bet­ter 😉

Con­clu­sion

While the heav­ier map / set types don’t have for­ward de­clar­a­tions, the ex­ist­ing for­ward de­clar­a­tions can already cov­er many use cases for lib­rar­ies that want to be both fast-to-com­pile and STL-friendly:

• a STL-friendly func­tion over­load re­turn­ing a std::string in­stead of a cus­tom light­weight string type to avoid fur­ther cop­ies when passing the string to STL-ori­ented APIs (used for ex­ample by Util­ity::Dir­ect­ory::read() vs Util­ity::Dir­ect­ory::read­String())
• a func­tion over­load which takes a std::vec­tor by mut­able ref­er­ence (in­stead of e.g. a ref­er­ence to Con­tain­ers::Ar­ray) in or­der to fill it, to avoid un­ne­ces­sary cop­ies when a std::vec­tor is needed fur­ther
• func­tions tak­ing and re­turn­ing std::ar­rays / std::tuples could now reside in oth­er­wise STL-free head­ers in­stead of be­ing pushed out to avoid the 13k in­cluded lines

Too bad that the std::string_view and std::span types, while meant to be light­weight at runtime, are so com­pile-time heavy and im­possible to for­ward-de­clare.

While it’s pos­sible to make use of the for­ward de­clar­a­tions also for func­tions tak­ing/re­turn­ing vec­tors, ar­rays and strings by con­stant ref­er­ences, a bet­ter ap­proach is to make use of STL com­pat­ib­il­ity in the Con­tain­ers::Ar­rayView classes — that’ll al­low more types than just std::vec­tor, std::ar­ray or std::string to be used and the com­pile-time im­pact is clear and con­sist­ent with this type. See the art­icle about STL-com­pat­ible ar­ray views for more in­form­a­tion and de­tailed per­form­ance over­view.

I also didn’t look for for­ward de­clar­a­tions of the std::unique_ptr, std::op­tion­al or std::ref­er­en­ce_wrap­per types be­cause they’re very simple and thus easy to re­place. See the Light­weight but still STL-com­pat­ible unique point­er post for more in­form­a­tion.

Try them in your code

Cor­rade provides the above for­ward de­clar­a­tions in tiny Cor­rade/Util­ity/StlFor­wardAr­ray.h, Cor­rade/Util­ity/StlFor­ward­String.h, Cor­rade/Util­ity/StlFor­wardTuple.h and Cor­rade/Util­ity/StlFor­ward­Vec­tor.h head­ers, simply in­clude them in­stead of the stand­ard <array>, <string>, <tuple> or <vector> head­ers where de­sired. The en­gine is mov­ing to use them as well, but since it will cause many build er­rors due to sud­denly miss­ing #includes in user code, the move is done gradu­ally to avoid sud­denly break­ing everything.

The Mag­num Singles re­pos­it­ory con­tains these as well, as a her­metic­head­ers not de­pend­ing on Cor­rade’s in­tern­als. Take them and bundle right in­to your pro­ject:

16.

^ a b gathered us­ing Clang 7.0 and libc++, since GCC 8.2’s lib­stdc++ doesn’t have a for­ward de­clar­a­tion for std::ar­ray / std::vec­tor

Can we con­vince vendors to do this more?

While I think it’s pos­sible to add ad­di­tion­al for­ward de­clar­a­tions to some STL im­ple­ment­a­tions, it might not be al­ways pos­sible to do so without break­ing ABI com­pat­ib­il­ity — even in Mag­num I had to break ABI com­pat­ib­il­ity a few times in the past in or­der to achieve that (and now I know what to avoid to make new types eas­ily for­ward-de­clar­able from the start).

The ideal way would be to have the for­ward de­clar­a­tions guar­an­teed by the stand­ard (ex­tend­ing <iosfwd> fur­ther, for ex­ample) so we don’t need to in­clude plat­form-spe­cif­ic in­tern­al “bits”, but again this may cause an ABI break for many vendors and thus take years to im­ple­ment (like it happened with std::string in C++11 where lib­stdc++ could no longer have it copy-on-write — and the prob­lems it caused per­sist un­til today).

There’s also a slight chance that, due to com­plex­it­ies of std::al­loc­at­or and oth­er types used in de­fault tem­plate ar­gu­ments, adding for­ward de­clar­a­tions to <iosfwd> would make it no longer light­weight. This really de­pends on how the im­ple­ment­a­tions are done and what all needs to be known to for­ward-de­clare giv­en type.