linux gcc header cost benchmarked!

also hopefully fixing CI builds with gcc 4.7........ (%^#$ this -Winline warning from gcc 4.7!!!)

4 files changed, 51 insertions, 40 deletions

diff --git a/.gitignore b/.gitignore
index acbf7f0..74be3b2 100644
--- a/.gitignore
+++ b/.gitignore
@@ -30,4 +30,6 @@ build/*
 build-mingw/*
 examples/*/build/*
 examples/*/build-mingw/*
+scripts/*/build/*
+scripts/*/build-mingw/*
 *.pyc
diff --git a/doc/markdown/benchmarks.md b/doc/markdown/benchmarks.md
index 14c90a5..bae3459 100644
--- a/doc/markdown/benchmarks.md
+++ b/doc/markdown/benchmarks.md
@@ -2,7 +2,20 @@
 
 The benchmarks are done with [**this**](../../scripts/bench/bench.py) script using CMake.
 
-It should be noted that GCC performs much better on Unix - but even there the speedup ratios are similar (clang should be the similar).
+GCC performs much better on Unix - but even there the speedup ratios are similar (clang should be the similar).
+
+Compilers used:
+- Microsoft Visual Studio Community 2015 - Version 14.0.25431.01 Update 3
+- gcc 6.2.0 (x86_64-posix-seh-rev0, Built by MinGW-W64 project)
+- gcc 5.2.1 20151010 (Ubuntu 5.2.1-22ubuntu2)
+
+Environment used (Intel i7 3770k, 16g RAM)
+- Windows 7 - on an SSD
+- Ubuntu 15.10 in a VirtualBox VM - on a HDD
+
+**doctest** version: 1.1.0 (released on 2016.09.20)
+
+[**Catch**](https://github.com/philsquared/Catch) version: 1.5.6 (released on 2016.06.09)
 
 ## Cost of including the header
 
@@ -11,7 +24,7 @@ This is a benchmark that is relevant only to single header and header only frame
 The script generates 201 source files and in 200 of them makes a function in the form of ```int f135() { return 135; }``` and in ```main.cpp``` it forward declares all the 200 such dummy functions and accumulates their result to return from the ```main()``` function. This is done to ensure that all source files are built and that the linker doesn't remove/optimize anything.
 
 - **baseline** - how much time the source files need for a single threaded build with ```msbuild```/```mingw32-make``` 
-- **+ implementation** - only in ```main.cpp``` the header is included with a ```#define``` before it so the test runner gets instantiated:
+- **+ implement** - only in ```main.cpp``` the header is included with a ```#define``` before it so the test runner gets implemented:
 
     ```c++
 #define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN
@@ -22,21 +35,25 @@ The script generates 201 source files and in 200 of them makes a function in the
 
 ### doctest
 
-|                   | baseline | + implementation | + header everywhere | + disabled |
-|------------------------|----------|------------------|---------------------|------------|
-| MSVC Debug             | 5.9      | 7.1              | 8.3                 | 7.0        |
-| MSVC Release           | 5.4      | 6.9              | 8.7                 | 6.5        |
-| MinGW-w64 GCC Debug    | 9.4      | 11.7             | 14.4                | 11.1       |
-| MinGW-w64 GCC Release  | 9.6      | 12.3             | 14.9                | 11.4       |
+|                   | baseline | + implement | + header everywhere | + disabled |
+|------------------------|----------|-------------|---------------------|------------|
+| MSVC Debug             | 5.9      | 7.1         | 8.3                 | 7.0        |
+| MSVC Release           | 5.4      | 6.9         | 8.7                 | 6.5        |
+| MinGW-w64 GCC Debug    | 9.4      | 11.7        | 14.4                | 11.1       |
+| MinGW-w64 GCC Release  | 9.6      | 12.3        | 14.9                | 11.4       |
+| Linux GCC Debug        | 6.3      | 7.1         | 10.2                | 7.4        |
+| Linux GCC Release      | 6.5      | 8.4         | 10.8                | 7.8        |
 
 ### Catch
 
-|                   | baseline | + implementation | + header everywhere |                             |
-|------------------------|----------|------------------|---------------------|------------|
-| MSVC Debug             | 5.9      | 8.5              | 102                 |            |
-| MSVC Release           | 5.4      | 10.3             | 96                  |            |
-| MinGW-w64 GCC Debug    | 9.4      | 24.5             | 125                 |            |
-| MinGW-w64 GCC Release  | 9.6      | 18.4             | 113                 |            |
+|                   | baseline | + implement | + header everywhere |                             |
+|------------------------|----------|-------------|---------------------|------------|
+| MSVC Debug             | 5.9      | 8.5         | 102                 |            |
+| MSVC Release           | 5.4      | 10.3        | 96                  |            |
+| MinGW-w64 GCC Debug    | 9.4      | 24.5        | 125                 |            |
+| MinGW-w64 GCC Release  | 9.6      | 18.4        | 113                 |            |
+| Linux GCC Debug        | 6.3      | 10.4        | 59                  |            |
+| Linux GCC Release      | 6.5      | 14.1        | 64                  |            |
 
 ### Conclusion
 
@@ -44,15 +61,15 @@ So on a modern developer machine:
 
 #### doctest
 
-- instantiating the test runner in one source file costs ~1.5 seconds ```implementation - baseline```
-- the inclusion of ```doctest.h``` in one source file costs below 9ms ```(header_everywhere - implementation) / 200```
+- instantiating the test runner in one source file costs ~1.5 seconds ```implement - baseline```
+- the inclusion of ```doctest.h``` in one source file costs below 9ms ```(header_everywhere - implement) / 200```
 - including the library everywhere - but everything disabled - costs less than 2 seconds ```disabled - baseline```
 
 #### [Catch](https://github.com/philsquared/Catch)
 
-- instantiating the test runner in one source file costs ~5 second ```implementation - baseline```
+- instantiating the test runner in one source file costs ~5 second ```implement - baseline```
   (~12 seconds for MinGW-w64)
-- the inclusion of ```catch.hpp```  in one source file costs around 430ms ```(header_everywhere - implementation) / 200```
+- the inclusion of ```catch.hpp```  in one source file costs around 430ms ```(header_everywhere - implement) / 200```
   (below 280ms for MinGW-w64 which is really odd)
 
 ----------
@@ -63,26 +80,11 @@ So if ```doctest.h``` costs 8ms and ```catch.hpp``` costs 430ms on MSVC - then t
 
 The results are in seconds and are in **no way** intended to bash [**Catch**](https://github.com/philsquared/Catch) - the **doctest** framework wouldn't exist without it.
 
-The reason the **doctest** header is so light on compile times is because it forward declares everything and doesn't drag any standard headers in the source files (except for the source file where the test runner gets implemented). This was a key design decision.
+The reason the **doctest** header is so light on compile times is because it forward declares everything and doesn't drag any headers in the source files (except for the source file where the test runner gets implemented). This was a key design decision.
 
 ## Cost of an assertion macro
 
-trololo
-
-# Details
-
-Compilers used:
-
-- Microsoft Visual Studio Community 2015 - Version 14.0.25431.01 Update 3
-- gcc 6.2.0 (x86_64-posix-seh-rev0, Built by MinGW-W64 project)
-
-Environment used - Windows 7 on an SSD, Intel i7 3770k, 16g RAM
-
-Date: 2016.09.19
-
-**doctest** version: 1.1.0
-
-[**Catch**](https://github.com/philsquared/Catch) version: 1.5.6
+coming soon
 
 ---------------
 
diff --git a/examples/removing_doctest_options/main.cpp b/examples/removing_doctest_options/main.cpp
index 044fdf3..02750ec 100644
--- a/examples/removing_doctest_options/main.cpp
+++ b/examples/removing_doctest_options/main.cpp
@@ -2,6 +2,10 @@
 #define DOCTEST_CONFIG_IMPLEMENT
 #include "doctest.h"
 
+#if __GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ > 6)
+#pragma GCC diagnostic ignored "-Wzero-as-null-pointer-constant"
+#endif // > gcc 4.6
+
 #include <string>
 #include <vector>
 
@@ -15,12 +19,15 @@ public:
         for(; *argv_in; ++argv_in)
             if(!starts_with(*argv_in, "--dt-"))
                 vec.push_back(*argv_in);
-#if __GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ > 6)
-#pragma GCC diagnostic ignored "-Wzero-as-null-pointer-constant"
-#endif // > gcc 4.6
         vec.push_back(NULL);
     }
 
+// to fix gcc 4.7 "-Winline" warnings
+#if defined(__GNUC__) && !defined(__clang__)
+    __attribute__((noinline))
+#endif
+    ~dt_removed() {}
+    
     int          argc() { return static_cast<int>(vec.size()) - 1; }
     const char** argv() { return &vec[0]; } // Note: non-const char **:
 
diff --git a/examples/removing_doctest_options/test_output/removing_doctest_options.txt b/examples/removing_doctest_options/test_output/removing_doctest_options.txt
index fa7d82b..fe9e959 100644
--- a/examples/removing_doctest_options/test_output/removing_doctest_options.txt
+++ b/examples/removing_doctest_options/test_output/removing_doctest_options.txt
@@ -1,10 +1,10 @@
 [doctest] doctest version is "1.0.0"
 [doctest] run with "--help" for options
 ===============================================================================
-main.cpp(48)
+main.cpp(55)
 Fail
 
-main.cpp(48) FAILED! 
+main.cpp(55) FAILED! 
   REQUIRE( 0 )
 with expansion:
   REQUIRE( 0 )