Alot (see description)

- Made a better global allocator for the process.
- Some small fixes to gen.hpp, removed clear_code_memory as I'm designing this library to for now never free any memory.
- Fixes to memory usage for cached strings
- Added missing verification for attributes in some upfront constructors. Added attribute param for def_type procedure.
- Started to use internal and global keywords in gen.cpp for associated definitions
- Progress toward getting the parsing constructors to support operator definitions.
- There was an *attempt* to get parse_type to support parsing function types. Its not tested yet....
  - Its not an nice setup, there is no validation of parameters, problably will add that in the future.

Readme.md

@@ -35,7 +35,7 @@ With the dependency code being under 10000 sloc. (Containers, Memory, String han
 Any dependencies from the zpl library will be exposed manually with using declarations into global scope.  
 They will be removed when the library is feature complete for version 1 (zero dependencies milestone).
 
-*Right now the upfront constructors are working to some extent based on testing*
+*Right now the constructors are working to some extent based on testing*
 
 ***The editor and scanner will NOT be implemented by version 1. They require alot code and the focus for version 1 is to have a robust constructor API and builder, witch a wide suite of usage examples in the tests for the project.***
 
@@ -156,15 +156,17 @@ If in your use case, you decide to have exclusive separation or partial separati
 
 ### *WHAT IS NOT PROVIDED*
 
-* Macro or template generation                : This library is to avoid those, adding support for them adds unnecessary complexity.
+* Macro or template generation                : This library is *currently* intended to avoid those, adding support for them adds unnecessary complexity.
+  * There may be an argument to support basic templates for substitution, to reduce symbol redundancy for the user, since debuggers tend to do well for them.
+  * Any sort of template complexity however to resolve if the subtiution is valid with templates would not be supported.
 * Vendor provided dynamic dispatch (virtuals) : `override` and `final` specifiers complicate the specifier serialization. (I'll problably end up adding in later)
 * RTTI
 * Exceptions
-* Execution statement validation              : Execution expressions are defined using the untyped string API.
+* Execution statement validation              : Execution expressions are defined using the untyped API.
 
 Keywords in from "Modern C++":
 
-* constexpr : Great to store compile-time constants, (easier to garantee when emitted from gentime)
+* constexpr : Great to store compile-time constants, (easier to guarantee when emitted from gentime)
 * consteval : Technically fine so long as templates are not used. Need to make sure to execute in moderation.
 * constinit : Better than constexpr at doing its job, however, its only c++ 20.
 * export    : Useful if c++ modules ever come around to actually being usable.
@@ -220,13 +222,13 @@ uw ArrS_Cap =
 (     AST_POD_Size
     - sizeof(AST*)         // Parent
     - sizeof(StringCached) // Name
-    - sizeof(CodeT) 	   // Type
+    - sizeof(CodeT)        // Type
     - sizeof(OperatorT)    // Op
     - sizeof(ModuleFlag)   // ModuleFlags
     - sizeof(AccessSpec)   // ParentAccess
-    - sizeof(u32) 		   // StaticIndex
+    - sizeof(u32)          // StaticIndex
-    - sizeof(bool) * 1     // DynamicEntries
+    - sizeof(bool)         // DynamicEntries
-    - sizeof(u8) * 2 )     // _Align_Pad
+    - sizeof(u8) * 3 )     // _Align_Pad
 / sizeof(AST*);
 ```
 
@@ -243,6 +245,10 @@ Data Notes:
 * Both AST and Code have member symbols but their data layout is enforced to be POD types.
 * This library treats memory failures as fatal.
 * Strings are stored in their own set of arenas. AST constructors use cached strings for names, and content.
+  * `StringArenas`, `StringMap`, `Allocator_StringArena`, and `Allocator_StringTable` are the associated containers or allocators.
+* Strings used for seralization and file buffers are not contained by those used for cached strings.
+  * They are currently using `Memory::GlobalAllocator`, which are tracked array of arenas that grows as needed (adds buckets when one runs out).
+  * Memory within the buckets is not resused, so its inherently wasteful (most likely will give non-cached strings their own tailored alloator later)
 
 ## There are three sets of interfaces for Code AST generation the library provides
 
@@ -258,10 +264,13 @@ The construction will fail and return InvalidCode otherwise.
 Interface :
 
 * def_attributes
+  * *This is preappened right before the function symbol, or placed after the class or struct keyword for any flavor of attributes used.*
+  * *Its up to the user to use the desired attribute formatting: `[[]]` (standard), `__declspec` (Microsoft), or `__attribute__` (GNU).*
 * def_comment
 * def_class
 * def_enum
-* def_execution       NOTE: This is equivalent to untyped_str, except that its intended for use only in execution scopes.
+* def_execution
+  * *This is equivalent to untyped_str, except that its intended for use only in execution scopes.*
 * def_extern_link
 * def_friend
 * def_function
@@ -287,7 +296,8 @@ Bodies:
 * def_enum_body
 * def_export_body
 * def_extern_link_body
-* def_function_body   NOTE: Use this for operator bodies as well.
+* def_function_body
+  * *Use this for operator bodies as well*
 * def_global_body
 * def_namespace_body
 * def_struct_body
@@ -328,8 +338,17 @@ Interface :
 * parse_using
 * parse_variable
 
-The parse API treats any execution scope definitions with no validation and are turned into untyped Code ASTs.  
+The lexing and parsing takes shortcuts from whats expected in the standard.
-This includes the assignmetn of variables; due to the library not yet supporting c/c++ expression parsing.
+
+* Numeric literals are not check for validity.
+* The parse API treats any execution scope definitions with no validation and are turned into untyped Code ASTs.
+  * *This includes the assignment of variables.*
+* Attributes ( `[[]]` (standard), `__declspec` (Microsoft), or `__attribute__` (GNU) )
+  * Assumed to *come before specifiers* (`const`, `constexpr`, `extern`, `static`, etc) for a function
+  * Or in the usual spot for class, structs, (*right after the declaration keyword*)
+  * typedefs have attributes with the type (`parse_type`)
+* As a general rule; if its not available from the upfront contructors, its not available in the parsing constructors.
+  * *Upfront constructors are not necessarily used in the parsing constructors, this is just a good metric to know what can be parsed.*
 
 Usage:
 
@@ -402,42 +421,45 @@ The following are provided predefined by the library as they are commonly used:
 * `spec_consteval`
 * `spec_constexpr`
 * `spec_constinit`
-* `spec_extern_linkage`
+* `spec_extern_linkage` (extern)
+* `spec_global` (global macro)
 * `spec_inline`
-* `spec_internal_linkage`
+* `spec_internal_linkage` (internal macro)
-* `spec_local_persist`
+* `spec_local_persist` (local_persist macro)
 * `spec_mutable`
 * `spec_ptr`
 * `spec_ref`
 * `spec_register`
 * `spec_rvalue`
-* `spec_static_member`
+* `spec_static_member` (static)
 * `spec_thread_local`
 * `spec_volatile`
 * `spec_type_signed`
 * `spec_type_unsigned`
 * `spec_type_short`
 * `spec_type_long`
-* `type_ns(void)`
+* `t_auto`
-* `type_ns(int)`
+* `t_void`
-* `type_ns(bool)`
+* `t_int`
-* `type_ns(char)`
+* `t_bool`
-* `type_ns(wchar_t)`
+* `t_char`
+* `t_wchar_t`
 
 Optionally the following may be defined if `GEN_DEFINE_LIBRARY_CODE_CONSTANTS` is defined
 
-* `type_ns( s8 )`
+* `t_b32`
-* `type_ns( s16 )`
+* `t_s8`
-* `type_ns( s32 )`
+* `t_s16`
-* `type_ns( s64 )`
+* `t_s32`
-* `type_ns( u8 )`
+* `t_s64`
-* `type_ns( u16 )`
+* `t_u8`
-* `type_ns( u32 )`
+* `t_u16`
-* `type_ns( u64 )`
+* `t_u32`
-* `type_ns( sw )`
+* `t_u64`
-* `type_ns( uw )`
+* `t_sw`
-* `type_ns( f32 )`
+* `t_uw`
-* `type_ns( f64 )`
+* `t_f32`
+* `t_f64`
 
 ## Extent of operator overload validation
 
@@ -551,5 +573,4 @@ Names or Content fields are interned strings and thus showed be cached using `ge
 * Make a test suite made up of collections based of the ZPL library templated colllection macros and the memory module.
 * Remove full ZPL dependency, move into Bloat header/source only what is used.
 * Generate a single-header library.
-* Generate a C-supported single-header library.
 * Actually get to version 1.

project/Bloat.cpp

@@ -6,33 +6,100 @@ namespace Memory
 {
 	using namespace zpl;
 
-	Arena Global_Arena {};
+	global AllocatorInfo GlobalAllocator;
+
+	global Array(Arena) Global_AllocatorBuckets;
+
+	void* Global_Allocator_Proc( void* allocator_data, AllocType type, sw size, sw alignment, void* old_memory, sw old_size, u64 flags )
+	{
+		Arena* last = & array_back( Global_AllocatorBuckets );
+
+		switch ( type )
+		{
+			case EAllocationALLOC:
+			{
+				if ( last->total_allocated + size > last->total_size )
+				{
+					Arena bucket;
+					arena_init_from_allocator( & bucket, heap(), BucketSize );
+
+					if ( bucket.physical_start == nullptr )
+						fatal( "Failed to create bucket for Global_AllocatorBuckets");
+
+					if ( ! array_append( Global_AllocatorBuckets, bucket ) )
+						fatal( "Failed to append bucket to Global_AllocatorBuckets");
+
+					last = & array_back( Global_AllocatorBuckets );
+				}
+
+				return alloc_align( arena_allocator( last), size, alignment );
+			}
+			case EAllocationFREE:
+			{
+				// Doesn't recycle.
+			}
+			case EAllocationFREE_ALL:
+			{
+				// Memory::cleanup instead.
+			}
+			case EAllocationRESIZE:
+			{
+				if ( last->total_allocated + size > last->total_size )
+				{
+					Arena bucket;
+					arena_init_from_allocator( & bucket, heap(), BucketSize );
+
+					if ( bucket.physical_start == nullptr )
+						fatal( "Failed to create bucket for Global_AllocatorBuckets");
+
+					if ( ! array_append( Global_AllocatorBuckets, bucket ) )
+						fatal( "Failed to append bucket to Global_AllocatorBuckets");
+
+					last = & array_back( Global_AllocatorBuckets );
+				}
+
+				void* result = alloc_align( arena_allocator( last), size, alignment );
+
+				if ( result != nullptr && old_memory != nullptr )
+				{
+					mem_copy( result, old_memory, size );
+				}
+
+				return result;
+			}
+		}
+
+		return nullptr;
+	}
 
 	void setup()
 	{
-		arena_init_from_allocator( & Global_Arena, heap(), Initial_Reserve );
+		GlobalAllocator = AllocatorInfo { & Global_Allocator_Proc, nullptr };
 
-		if ( Global_Arena.total_size == 0 )
+		if ( ! array_init_reserve( Global_AllocatorBuckets, heap(), 128 ) )
-		{
+			fatal( "Failed to reserve memory for Global_AllocatorBuckets");
-			assert_crash( "Failed to reserve memory for Tests:: Global_Arena" );
-		}
-	}
 
-	void resize( uw new_size )
+		Arena bucket;
-	{
+		arena_init_from_allocator( & bucket, heap(), BucketSize );
-		void* new_memory = resize( heap(), Global_Arena.physical_start, Global_Arena.total_size, new_size );
 
-		if ( new_memory == nullptr )
+		if ( bucket.physical_start == nullptr )
-		{
+			fatal( "Failed to create first bucket for Global_AllocatorBuckets");
-			fatal("Failed to resize global arena!");
-		}
 
-		Global_Arena.physical_start = new_memory;
+		array_append( Global_AllocatorBuckets, bucket );
-		Global_Arena.total_size     = new_size;
 	}
 
 	void cleanup()
 	{
-		arena_free( & Global_Arena);
+		s32 index = 0;
+		s32 left  = array_count( Global_AllocatorBuckets );
+		do
+		{
+			Arena* bucket = & Global_AllocatorBuckets[ index ];
+			arena_free( bucket );
+			index++;
+		}
+		while ( left--, left );
+
+		array_free( Global_AllocatorBuckets );
 	}
 }

project/Bloat.hpp

@@ -61,6 +61,7 @@ using zpl::EFileMode_WRITE;
 using zpl::EFileError_NONE;
 
 using zpl::alloc;
+using zpl::alloc_align;
 using zpl::arena_allocator;
 using zpl::arena_init_from_memory;
 using zpl::arena_init_from_allocator;
@@ -70,6 +71,8 @@ using zpl::str_fmt_buf;
 using zpl::char_first_occurence;
 using zpl::char_is_alpha;
 using zpl::char_is_alphanumeric;
+using zpl::char_is_digit;
+using zpl::char_is_hex_digit;
 using zpl::char_is_space;
 using zpl::crc32;
 using zpl::free_all;
@@ -564,16 +567,17 @@ char const* Msg_Invalid_Value = "INVALID VALUE PROVIDED";
 
 namespace Memory
 {
-	constexpr uw Initial_Reserve = megabytes(10);
+	// NOTE: This limits the size of the string that can be read from a file or generated to 10 megs.
+	// If you are generating a string larger than this, increase the size of the bucket here.
+	constexpr uw BucketSize = megabytes(10);
 
-	extern Arena Global_Arena;
+	// Global allocator used for data with process lifetime.
-	// #define g_allocator arena_allocator( & Memory::Global_Arena)
+	extern AllocatorInfo GlobalAllocator;
 
 	// Heap allocator is being used for now to isolate errors from being memory related (tech debt till ready to address)
-	#define g_allocator heap()
+	// #define g_allocator heap()
 
 	void setup();
-	void resize( uw new_size );
 	void cleanup();
 }
 

project/Readme.md

@@ -33,7 +33,6 @@ While getting fleshed out, all feature macros are defined on the top of the head
 These macros are:
 
 * `GEN_DEFINE_LIBRARY_CORE_CONSTANTS` : Optional typename codes as they are non-standard to C/C++ and not necessary to library usage
-* `GEN_FEATURE_INCREMENTAL` : Defines the incremental constructors
 * `GEN_FEATURE_PARSING` : Defines the parse constructors
 * `GEN_FEATURE_EDITOR` : Defines the file editing features for changing definitions based on ASTs
 * `GEN_FEATURE_SCANNER` : Defines the file scanning features for generating ASTs

project/gen.hpp

@@ -548,8 +548,8 @@ namespace gen
 			- sizeof(ModuleFlag)   // ModuleFlags
 			- sizeof(AccessSpec)   // ParentAccess
 			- sizeof(u32) 		   // StaticIndex
-			- sizeof(bool) * 1     // DynamicEntries
+			- sizeof(bool)         // DynamicEntries
-			- sizeof(u8) * 2 )     // _Align_Pad
+			- sizeof(u8) * 3 )     // _Align_Pad
 		/ sizeof(AST*);
 
 		constexpr static
@@ -713,13 +713,6 @@ namespace gen
 	// However on Windows at least, it doesn't need to occur as the OS will clean up after the process.
 	void deinit();
 
-	/*
-		Use this only if you know you generated the code you needed to a file.
-		And rather get rid of current code asts instead of growing the pool memory.
-		TODO: Need to put permanent ASTs into a separate set of memory. (I might just remove this tbh as it might be useless)
-	*/
-	void clear_code_memory();
-
 	// Used internally to retrive or make string allocations.
 	// Strings are stored in a series of string arenas of fixed size (SizePer_StringArena)
 	StringCached get_cached_string( StrC str );
@@ -787,7 +780,7 @@ namespace gen
 		, Code       attributes = NoCode
 		, ModuleFlag mflags     = ModuleFlag::None );
 
-	Code def_type   ( StrC name, Code arrayexpr = NoCode, Code specifiers = NoCode );
+	Code def_type   ( StrC name, Code arrayexpr = NoCode, Code specifiers = NoCode, Code attributes = NoCode );
 	Code def_typedef( StrC name, Code type, Code attributes = NoCode, ModuleFlag mflags = ModuleFlag::None );
 
 	Code def_union( StrC name, Code body, Code attributes = NoCode, ModuleFlag mflags = ModuleFlag::None );
@@ -830,6 +823,7 @@ namespace gen
 #	ifdef GEN_FEATURE_PARSING
 	Code parse_class       ( StrC class_def     );
 	Code parse_enum        ( StrC enum_def      );
+	Code parse_export_body ( StrC export_def    );
 	Code parse_extern_link ( StrC exten_link_def);
 	Code parse_friend      ( StrC friend_def    );
 	Code parse_function    ( StrC fn_def        );
@@ -1030,7 +1024,7 @@ namespace gen
 
 namespace gen
 {
-	// These constexprs are used for allocation heavior of data structurs
+	// These constexprs are used for allocation behavior of data structures
 	// or string handling while constructing or serializing.
 	// Change them to suit your needs.
 

test/NonParsed/Memory.NonParsed.hpp

@@ -0,0 +1 @@
+// 

test/Parsed/Sanity.Parsed.hpp

@@ -107,7 +107,6 @@ u32 gen_sanity()
 	gen_sanity_file.print_fmt("\n");
 
 	// Function
-	if (0)
 	{
 		Code fwd = parse_function( code(
 			void test_function();
@@ -128,7 +127,6 @@ u32 gen_sanity()
 	gen_sanity_file.print_fmt("\n");
 
 	// Namespace
-	if (0)
 	{
 		Code def = parse_namespace( code(
 			namespace TestNamespace
@@ -144,10 +142,9 @@ u32 gen_sanity()
 	gen_sanity_file.print_fmt("\n");
 
 	// Operator
-
 	if (0)
 	{
-		Code bitflagtest = parse_class( code(
+		Code bitflagtest = parse_enum( code(
 			enum class EBitFlagTest : u8
 			{
 				A = 1 << 0,