Small correction to test comment.

Its easier to manage, I'm sticking with generating the single header so it wont matter, its easy to refactor back if desired.

Readme.md

@@ -21,7 +21,10 @@ These build up a code AST to then serialize with a file builder.
 
 ## Notes
 
-The project has reached a sort of *alpha* state, all the current functionality works for the test cases but it will most likely break in many other cases.
+The project has reached an *alpha* state, all the current functionality works for the test cases but it will most likely break in many other cases.
+
+Note: Do not trying to do any large generations with this (at least not without changing the serialization implementation).
+It does not resue any memory yet for dynamic strings and thus any signficant size memory will result in massive consumption. 
 
 The project has no external dependencies beyond:
 
@@ -36,13 +39,12 @@ The project has no external dependencies beyond:
 Dependencies for the project are wrapped within `GENCPP_ROLL_OWN_DEPENDENCIES` (Defining it will disable them).  
 The majority of the dependency's implementation was derived from the [c-zpl library](https://github.com/zpl-c/zpl).
 
-When gencpp is in a stable state, I will make a C variant with the same feature set.  
-A single-header version will also be generated for both.
-
 A `natvis` and `natstepfilter` are provided in the scripts directory.
 
 ***The editor and scanner have not been implemented yet. The scanner will come first, then the editor.***
 
+A C variant is hosted [here](https://github.com/Ed94/genc); I haven't gotten headwind on it, should be easier to make than this...
+
 ## Usage
 
 A metaprogram is built to generate files before the main program is built. We'll term runtime for this program as `gen_time`. The metaprogram's core implementation are within `gen.hpp` and `gen.cpp` in the project directory.
@@ -159,10 +161,12 @@ This method is setup where all the metaprogram's code are the within the same fi
 ### *WHAT IS NOT PROVIDED*
 
 * Lambdas
-* Vendor provided dynamic dispatch (virtuals) : `override` and `final` specifiers complicate the specifier parsing and serialization. (I'll problably end up adding in later)
+* Lang provided dynamic dispatch (virtuals) : `override` and `final` specifiers complicate the specifier parsing and serialization. (Its a todo)
+* Suffix specifiers for functions (Ex: void() const ). Same reason as virtual/override/final missing for now.
 * RTTI
 * Exceptions
 * Execution statement validation              : Execution expressions are defined using the untyped API.
+* Parsing support for module specifiers and attributes. (Its a todo)
 
 Keywords kept from "Modern C++":
 
@@ -286,6 +290,12 @@ Data Notes:
   * Memory within the buckets is not resused, so its inherently wasteful (most likely will give non-cached strings their own tailored allocator later)
 * Linked lists used children nodes on bodies, and parameters.
 * Its intended to generate the AST in one go and serialize after. The contructors and serializer are designed to be a "one pass, front to back" setup.
+* When benchmarking, the three most significant var to tune are:
+  * `Memory::Global_BlockSize` (found gen_dep.hpp) : Used by the GlobalAllocator for the size of each global arena.
+  * `SizePer_StringArena` (found in gen.hpp under the constants region) : Used by the string cache to store strings.
+  * `CodePool_NumBlocks` (found in gen.hpp under constants region) : Used by code pool to store ASTs.
+  * The default values can handled generating for a stirng up to a size of ~650 kbs (bottleneck is serialization).
+  * Increasing the values can generate files upwards of over a million lines without issue (the formatter will most likely run slower than it)
 
 Two generic templated containers are used throughout the library:
 
@@ -588,6 +598,8 @@ Editor and Scanner are disabled by default, use `GEN_FEATURE_EDITOR` and `GEN_FE
 
 ### Editor is for editing a series of files based on a set of requests provided to it
 
+**Note: Not implemented yet**
+
 * The purpose is to overrite a specific file, it places its contents in a buffer to scan.
 * Requests are populated using the following interface:
   * add : Add code.
@@ -610,6 +622,8 @@ It will on call add a request to the queue to run the refactor script on the fil
 
 ### Scanner allows the user to generate Code ASTs by reading files
 
+**Note: Not implemented yet**
+
 * The purpose is to grab definitions to generate metadata or generate new code from these definitions.
 * Requests are populated using the add( SymbolInfo, Policy ) function. The symbol info is the same as the one used for the editor. So is the case with Policy.
 
@@ -664,8 +678,12 @@ Names or Content fields are interned strings and thus showed be cached using `ge
 # TODO
 
 * Implement a context stack for the parsing, allows for accurate scope validation for the AST types.
-* Make a test suite thats covers some base cases and zpl containers (+ anything else suitable)
-* Finish support for module specifiers and standard/platform attributes.
+* Make a more robust test suite.
 * Generate a single-header library.
-* Improve the allocation strategy for strings in `AST::to_string`, `Parser::lex`, and `token_fmt_va`
+* Improve the allocation strategy for strings in `Builder`, `AST::to_string`, `Parser::lex`, all three can use some form of slab allocation strategy...
+  * Can most likely use a simple slag allocator.
 * May be in need of a better name, I found a few repos with this same one...
+* Support module and attribute parsing (Marked with TODOs for now..)
+* Suffix specifiers for functions (const, override, final)
+* Implement the Scanner
+* Implement the Editor

project/Readme.md

@@ -13,6 +13,9 @@ Feature Macros:
 * `GEN_FEATURE_EDITOR` : Defines the file editing features for changing definitions based on ASTs
 * `GEN_FEATURE_SCANNER` : Defines the file scanning features for generating ASTs
 
+`GEN_USE_RECURSIVE_AST_DUPLICATION` is available but its not well tested and should not need to be used.
+If constructing ASTs properly. There should be no modification of ASTs, and thus this would never become an issue.
+
 Due to the design of `gen.hpp` to support being written alongside runtime intended code (in the same file), all the code is wrapped in a `gen_time` `#ifdef` and then wrapped further in a `gen` namespace to avoid pollution of the global scope.
 
 *Note: Its possible with the scanner feature to support parsing runtime files that use "generic" macros or identifiers with certain patterns.  

project/gen.undef.macros.hpp

@@ -1,6 +1,5 @@
 #if gen_time
 // This undefines the macros used by the gen library but are not necessary for the user.
-// TODO : This is incomplete until all dependencies are brough in from ZPL into bloat.
 
 #undef GEN_ARCH_64_BIT
 #undef GEN_ARCH_32_BIT

scripts/build.ci.ps1

@@ -48,7 +48,7 @@ Push-location $path_gen
 # Run meta-program
 	$gencpp = Join-Path $path_gen_build gencpp.exe
 
-	Write-Host `nGenerating files -- Parsed...
+	Write-Host `nRunning tests...
 	& $gencpp
 
 # Format generated files

scripts/genccp.natstepfilter

@@ -24,4 +24,7 @@
 		<Name>gen::Code.*::to_string</Name>
 		<Action>NoStepInto</Action>
 	</Function>
+	<Function>
+		<Name>gen::String::operator .*</Name>
+	</Function>
 </StepFilter>

scripts/gencpp.natvis

@@ -5,6 +5,18 @@
 		<DisplayString>Data:{Data} Proc:{Proc}</DisplayString>
 	</Type>
 
+	<Type Name="gen::Pool">
+		<DisplayString>NumBlocks: {NumBlocks} TotalSize: {TotalSize}</DisplayString>
+		<Expand>
+			<LinkedListItems>
+				<Size>NumBlocks</Size>
+				<HeadPointer>FreeList</HeadPointer>
+				<NextPointer>FreeList</NextPointer>
+				<ValueNode>PhysicalStart</ValueNode>
+			</LinkedListItems>
+		</Expand>
+	</Type>
+
 	<Type Name="gen::Array&lt;*&gt;">
 		<DisplayString>Num:{((Header*)((char*)Data - sizeof(Header)))->Num},
 			Capacity:{((Header*)((char*)Data - sizeof(Header)))->Capacity}</DisplayString>
@@ -70,7 +82,8 @@
 			<Item Name="Content">ast->Content</Item>
 			<Item Name="Body">ast->Body</Item>
 			<Item Name="Parent">ast->Parent</Item>
-			<Item Name="ModuleFlags" Condition="ast->ModuleFlags != ModuleFlag::Invalid">ast->ModuleFlags</Item>
+			<Item Name="ModuleFlags" Condition="ast->ModuleFlags != ModuleFlag::Invalid">
+				ast->ModuleFlags</Item>
 			<Item Name="ArrSpecs" Condition="ast->ArrSpecs[0] &lt; 18">ast->ArrSpecs</Item>
 			<Item Name="Prev">ast->Prev</Item>
 			<Item Name="Next">ast->Next</Item>
@@ -365,8 +378,8 @@
 		<DisplayString Condition="ast != nullptr">{ast->Name} {ast->Type}</DisplayString>
 		<Expand>
 			<Item Name="Parent">ast->Parent</Item>
-			<Item Name="Prev">ast->Front</Item>
-			<Item Name="Next">ast->Back</Item>
+			<Item Name="Front">ast->Front</Item>
+			<Item Name="Back">ast->Back</Item>
 			<Item Name="NumEntries">ast->NumEntries</Item>
 		</Expand>
 	</Type>
@@ -665,4 +678,4 @@
 		<DisplayString>Current[ { Arr[Idx] } ] Idx:{ Idx }</DisplayString>
 	</Type>
 
-</AutoVisualizer>
+</AutoVisualizer>

test/Parsed/Array.Parsed.hpp

@@ -1,6 +1,11 @@
 #pragma once
 
 #if gen_time
+#define GEN_FEATURE_PARSING
+#define GEN_DEFINE_LIBRARY_CODE_CONSTANTS
+#define GEN_ENFORCE_STRONG_CODE_TYPES
+#define GEN_EXPOSE_BACKEND
+#define GEN_BENCHMARK
 #include "gen.hpp"
 
 using namespace gen;

test/Parsed/Buffer.Parsed.hpp

@@ -1,6 +1,11 @@
 #pragma once
 
 #if gen_time
+#define GEN_FEATURE_PARSING
+#define GEN_DEFINE_LIBRARY_CODE_CONSTANTS
+#define GEN_ENFORCE_STRONG_CODE_TYPES
+#define GEN_EXPOSE_BACKEND
+#define GEN_BENCHMARK
 #include "gen.hpp"
 
 using namespace gen;

test/Parsed/HashTable.Parsed.hpp

@@ -1,6 +1,11 @@
 #pragma once
 
 #if gen_time
+#define GEN_FEATURE_PARSING
+#define GEN_DEFINE_LIBRARY_CODE_CONSTANTS
+#define GEN_ENFORCE_STRONG_CODE_TYPES
+#define GEN_EXPOSE_BACKEND
+#define GEN_BENCHMARK
 #include "gen.hpp"
 #include "Array.Parsed.hpp"
 

test/Parsed/Ring.Parsed.hpp

@@ -1,6 +1,11 @@
 #pragma once
 
 #if gen_time
+#define GEN_FEATURE_PARSING
+#define GEN_DEFINE_LIBRARY_CODE_CONSTANTS
+#define GEN_ENFORCE_STRONG_CODE_TYPES
+#define GEN_EXPOSE_BACKEND
+#define GEN_BENCHMARK
 #include "gen.hpp"
 #include "Buffer.Parsed.hpp"
 

test/Parsed/Sanity.Parsed.hpp

@@ -1,5 +1,10 @@
 #pragma once
 #ifdef gen_time
+#define GEN_FEATURE_PARSING
+#define GEN_DEFINE_LIBRARY_CODE_CONSTANTS
+#define GEN_ENFORCE_STRONG_CODE_TYPES
+#define GEN_EXPOSE_BACKEND
+#define GEN_BENCHMARK
 #include "gen.hpp"
 
 using namespace gen;
@@ -286,7 +291,7 @@ u32 gen_sanity()
 
 		));
 
-		CodeUsingNamespace npspace_using = (CodeUsingNamespace) parse_using( code(
+		CodeUsing npspace_using = parse_using( code(
 			using namespace TestNamespace;
 		));
 

test/SOA.hpp

@@ -1,6 +1,9 @@
 #pragma once
 
 #ifdef gen_time
+#define GEN_FEATURE_PARSING
+#define GEN_DEFINE_LIBRARY_CODE_CONSTANTS
+#define GEN_ENFORCE_STRONG_CODE_TYPES
 #include "gen.hpp"
 using namespace gen;
 
@@ -11,7 +14,7 @@ Code gen_SOA( CodeStruct struct_def, s32 num_entries = 0 )
 	));
 
 	Code
-	soa_entry = { struct_def.raw()->duplicate() };
+	soa_entry = { struct_def.duplicate() };
 	soa_entry->Name = get_cached_string( name(Entry) );
 
 	constexpr s32 Num_Vars_Cap = 128;

test/Upfront/Array.Upfront.hpp

@@ -18,7 +18,7 @@ Code gen__array_base()
 
 	CodeFn grow_formula = def_function( name(array_grow_formula), def_param( t_uw, name(value)), t_uw
 		, def_execution( code( return 2 * value * 8; ) )
-		, def_specifiers( args( ESpecifier::Static_Member, ESpecifier::Inline ) )
+		, def_specifiers( args( ESpecifier::Static, ESpecifier::Inline ) )
 	);
 
 	return def_global_body( args( header, grow_formula ) );
@@ -29,9 +29,9 @@ Code gen__array( StrC type )
 	static CodeType t_allocator_info = def_type( name(AllocatorInfo) );
 	static Code     v_nullptr        = code_str(nullptr);
 
-	static CodeSpecifier spec_ct_member     = def_specifiers( 2, ESpecifier::Constexpr, ESpecifier::Static_Member );
-	static CodeSpecifier spec_static_inline = def_specifiers( 2, ESpecifier::Static_Member, ESpecifier::Inline );
-	static CodeSpecifier spec_static        = def_specifier( ESpecifier::Static_Member );
+	static CodeSpecifier spec_ct_member     = def_specifiers( 2, ESpecifier::Constexpr, ESpecifier::Static );
+	static CodeSpecifier spec_static_inline = def_specifiers( 2, ESpecifier::Static, ESpecifier::Inline );
+	static CodeSpecifier spec_static        = def_specifier( ESpecifier::Static );
 
 	static CodeUsing using_header    = def_using( name(Header), def_type( name(ArrayHeader) ) );
 	static CodeVar ct_grow_formula = def_variable( t_auto, name(grow_formula), untyped_str( code( & array_grow_formula )), spec_ct_member );

test/sanity.cpp

@@ -0,0 +1,93 @@
+// Testing to make sure backend of library is operating properly.
+
+#ifdef gen_time
+#define GEN_FEATURE_PARSING
+#define GEN_DEFINE_LIBRARY_CODE_CONSTANTS
+#define GEN_ENFORCE_STRONG_CODE_TYPES
+#define GEN_EXPOSE_BACKEND
+#define GEN_BENCHMARK
+#include "gen.hpp"
+
+void check_sanity()
+{
+	using namespace gen;
+	gen::init();
+	log_fmt("\ncheck_sanity:\n");
+
+	// Test string caching:
+	CodeType t_int_dupe = def_type( name(int) );
+
+	if ( t_int_dupe->Name != t_int->Name )
+		fatal("check_sanity: String caching failed!");
+
+
+	// Purposefully use an excessive amount of memory to make so the the memory backend doesn't break.
+	// This has been tested with num_iterations set to 15000000 (generates 15 million lines of code), the Global_BlockSize, along with CodePool_NumBlocks, and SizePer_StringArena
+	// must be adjusted to gigabytes(2), kilobytes(512), and gigabyte(1) for good performance without crashing.
+	/*
+		Typical usage (megabytes(10), kilobytes(4), megabytes(1), for 650000 (the limit of 10 meg partition buckets in global arena) )
+		Memory after builder:
+		Num Global Arenas       : 14 TotalSize: 146800640 !
+		Num Code Pools          : 794 TotalSize: 416284672 !
+		Num String Cache Arenas : 60 TotalSize: 62914560 !
+		Num String Cache        : 1300007
+
+		Memory usage to expect at 15 mil file:
+		Num Global Arenas       : 2 TotalSize: 4294967296 !
+		Num Code Pools          : 144 TotalSize: 9663676416 !
+		Num String Cache Arenas : 2 TotalSize: 2147483648 !
+		Num String Cache        : 30000025
+	*/
+	constexpr
+	s32 num_iterations = 650000;
+
+	Array<CodeTypedef> typedefs = Array<CodeTypedef>::init_reserve( Memory::GlobalAllocator, num_iterations * 2 );
+
+	s32 idx = num_iterations;
+	while( --idx )
+	{
+		// Stress testing string allocation
+		String type_name = String::fmt_buf( Memory::GlobalAllocator, "type_%d", idx );
+		String typedef_name = String::fmt_buf( Memory::GlobalAllocator, "typedef_%d", idx );
+
+		CodeTypedef type_as_int = def_typedef( type_name, t_int );
+		CodeType    type        = def_type( type_name );
+		CodeTypedef type_def    = def_typedef( typedef_name, type );
+
+		typedefs.append( type_as_int );
+		typedefs.append( type_def );
+	}
+
+	log_fmt("\nMemory before builder:\n");
+	log_fmt("Num Global Arenas       : %llu TotalSize: %llu !\n", Memory::Global_AllocatorBuckets.num(), Memory::Global_AllocatorBuckets.num() * Memory::Global_BucketSize);
+	log_fmt("Num Code Pools          : %llu TotalSize: %llu !\n", StaticData::CodePools.num(), StaticData::CodePools.num() * CodePool_NumBlocks * StaticData::CodePools.back().BlockSize);
+	log_fmt("Num String Cache Arenas : %llu TotalSize: %llu !\n", StaticData::StringArenas.num(), StaticData::StringArenas.num() * SizePer_StringArena);
+	log_fmt("Num String Cache        : %llu\n", StaticData::StringCache.Entries.num(), StaticData::StringCache);
+
+	Builder builder;
+	builder.open( "sanity.gen.hpp" );
+
+	idx = num_iterations;
+#ifdef GEN_BENCHMARK
+	u64 time_start     = time_rel_ms();
+#endif
+	while( --idx )
+	{
+		builder.print( typedefs[idx] );
+	}
+
+	builder.write();
+#ifdef GEN_BENCHMARK
+	log_fmt("\n\nBuilder finished writting. Time taken: %llu ms\n", time_rel_ms() - time_start);
+#endif
+
+	log_fmt("\nMemory after builder:\n");
+	log_fmt("Num Global Arenas       : %llu TotalSize: %llu !\n", Memory::Global_AllocatorBuckets.num(), Memory::Global_AllocatorBuckets.num() * Memory::Global_BucketSize);
+	log_fmt("Num Code Pools          : %llu TotalSize: %llu !\n", StaticData::CodePools.num(), StaticData::CodePools.num() * CodePool_NumBlocks * StaticData::CodePools.back().BlockSize);
+	log_fmt("Num String Cache Arenas : %llu TotalSize: %llu !\n", StaticData::StringArenas.num(), StaticData::StringArenas.num() * SizePer_StringArena);
+	log_fmt("Num String Cache        : %llu\n", StaticData::StringCache.Entries.num(), StaticData::StringCache);
+
+	log_fmt("\nSanity passed!\n");
+	gen::deinit();
+}
+#endif

test/test.Upfront.cpp

@@ -1,6 +1,9 @@
 #ifdef gen_time
-#define GEN_DEFINE_LIBRARY_CODE_CONSTANTS
 #define GEN_FEATURE_PARSING
+#define GEN_DEFINE_LIBRARY_CODE_CONSTANTS
+#define GEN_ENFORCE_STRONG_CODE_TYPES
+#define GEN_EXPOSE_BACKEND
+#define GEN_BENCHMARK
 #include "gen.cpp"
 #include "Upfront\Array.Upfront.hpp"
 #include "Upfront\Buffer.Upfront.hpp"
@@ -15,7 +18,6 @@ using namespace gen;
 
 int gen_main()
 {
-	Memory::setup();
 	gen::init();
 
 	gen_sanity_upfront();
@@ -35,7 +37,6 @@ int gen_main()
 	gen_ring_file();
 
 	gen::deinit();
-	Memory::cleanup();
 	return 0;
 }
 #endif

test/test.cpp

@@ -1,74 +1,25 @@
-#ifdef gen_time
-#define GEN_DEFINE_LIBRARY_CODE_CONSTANTS
-#define GEN_FEATURE_PARSING
-#include "Parsed\Array.Parsed.hpp"
-#include "Parsed\Buffer.Parsed.hpp"
-#include "Parsed\HashTable.Parsed.hpp"
-#include "Parsed\Ring.Parsed.hpp"
-#include "Parsed\Sanity.Parsed.hpp"
-#include "SOA.hpp"
 #include "gen.cpp"
+#include "sanity.cpp"
 
-using namespace gen;
-
-// TODO : Rewrite this to include both upfront and parsed testing.
-
+#if gen_time
 int gen_main()
 {
-	Memory::setup();
-	gen::init();
+	using namespace gen;
+	log_fmt("\ngen_time:");
 
-	gen_sanity();
+	check_sanity();
 
-	gen_array( u8 );
-	gen_array( sw );
-
-	gen_buffer( u8 );
-
-	gen_hashtable( u32 );
-
-	gen_ring( s16 );
-	gen_ring( uw );
-
-	gen_array_file();
-	gen_buffer_file();
-	gen_hashtable_file();
-	gen_ring_file();
-
-	Builder soa_test; soa_test.open( "SOA.gen.hpp" );
-
-	soa_test.print( parse_using( code(
-		using u16 = unsigned short;
-	)));
-
-	soa_test.print( def_include( txt_StrC("gen.hpp")));
-
-	soa_test.print( def_using_namespace( name(gen) ) );
-
-	soa_test.print( gen_SOA(
-		parse_struct( code(
-			struct TestStruct
-			{
-				u8  A;
-				u16 B;
-				u32 C;
-				u64 D;
-			};
-		))
-	));
-
-	soa_test.write();
-
-	gen::deinit();
-	Memory::cleanup();
 	return 0;
 }
 #endif
 
 
-#ifdef runtime
+// This only has to be done if symbol conflicts occur.
+#ifndef gen_time
 int main()
 {
+
+
 	return 0;
 }
 #endif

test/test.parsing.cpp

@@ -0,0 +1,75 @@
+#ifdef gen_time
+#define GEN_FEATURE_PARSING
+#define GEN_DEFINE_LIBRARY_CODE_CONSTANTS
+#define GEN_ENFORCE_STRONG_CODE_TYPES
+#define GEN_EXPOSE_BACKEND
+#define GEN_BENCHMARK
+#include "Parsed\Array.Parsed.hpp"
+#include "Parsed\Buffer.Parsed.hpp"
+#include "Parsed\HashTable.Parsed.hpp"
+#include "Parsed\Ring.Parsed.hpp"
+#include "Parsed\Sanity.Parsed.hpp"
+#include "SOA.hpp"
+#include "gen.cpp"
+
+using namespace gen;
+
+// TODO : Need to make a more robust test suite
+
+int gen_main()
+{
+	gen::init();
+
+	gen_sanity();
+
+	gen_array( u8 );
+	gen_array( sw );
+
+	gen_buffer( u8 );
+
+	gen_hashtable( u32 );
+
+	gen_ring( s16 );
+	gen_ring( uw );
+
+	gen_array_file();
+	gen_buffer_file();
+	gen_hashtable_file();
+	gen_ring_file();
+
+	Builder soa_test; soa_test.open( "SOA.gen.hpp" );
+
+	soa_test.print( parse_using( code(
+		using u16 = unsigned short;
+	)));
+
+	soa_test.print( def_include( txt_StrC("gen.hpp")));
+
+	soa_test.print( def_using_namespace( name(gen) ) );
+
+	soa_test.print( gen_SOA(
+		parse_struct( code(
+			struct TestStruct
+			{
+				u8  A;
+				u16 B;
+				u32 C;
+				u64 D;
+			};
+		))
+	));
+
+	soa_test.write();
+
+	gen::deinit();
+	return 0;
+}
+#endif
+
+
+#ifdef runtime
+int main()
+{
+	return 0;
+}
+#endif

test/upfront.hpp

@@ -0,0 +1,21 @@
+#ifdef gen_time
+#define GEN_FEATURE_PARSING
+#define GEN_DEFINE_LIBRARY_CODE_CONSTANTS
+#define GEN_ENFORCE_STRONG_CODE_TYPES
+#define GEN_EXPOSE_BACKEND
+#define GEN_BENCHMARK
+#include "gen.hpp"
+
+void check_upfront()
+{
+	using namespace gen;
+	log_fmt("\nupfront: ");
+	gen::init();
+
+
+
+	gen::deinit();
+	log_fmt("Passed!\n");
+}
+
+#endif